Learning

Malware Analysis Explore the concepts, tools, and techniques to analyze and investigate Windows malware

By Monnappa K A

www.packt.com

FOR SALE IN INDIA ONLY

Learning Malware Analysis

Explore the concepts, tools, and techniques to analyze and investigate Windows malware

Monnappa K A

BIRMINGHAM - MUMBAI

Learning Malware Analysis Copyright © 2018 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Commissioning Editor: Gebin George Acquisition Editor: Shrilekha Inani Content Development Editor: Sharon Raj Technical Editor: Prashant Chaudhari Copy Editor: Safis Editing Project Coordinator: Virginia Dias Proofreader: Safis Editing Indexer: Aishwarya Gangawane Graphics: Tom Scaria Production Coordinator: Nilesh Mohite First published: June 2018 Production reference: 1290618 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-78839-250-1

www.packtpub.com

To my beloved wife, for standing by me throughout the journey. Without her, it would have been impossible to complete this project. To my parents, and in-laws for their continued support and encouragement. To my dog, for staying awake with me during the sleepless nights.

mapt.io

Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.

Why subscribe? Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals Improve your learning with Skill Plans built especially for you Get a free eBook or video every month Mapt is fully searchable Copy and paste, print, and bookmark content

PacktPub.com Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details. At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.

Contributors About the author Monnappa K A works for Cisco Systems as an information security investigator focusing on threat intelligence and the investigation of advanced cyber attacks. He is a member of the Black Hat review board, the creator of Limon Linux sandbox, the winner of the Volatility plugin contest 2016, and the co-founder of the Cysinfo cybersecurity research community. He has presented and conducted training sessions at various security conferences including Black Hat, FIRST, OPCDE, and DSCI. He regularly conducts training at the Black Hat Security Conference in USA, Asia, and Europe. I would like to extend my gratitude to Daniel Cuthbert and Dr. Michael Spreitzenbarth for taking time out of their busy schedule to review the book. Thanks to Sharon Raj, Prashant Chaudhari, Shrilekha Inani, and the rest of the Packt team for their support. Thanks to Michael Scheck, Chris Fry, Scott Heider, and my coworkers at Cisco CSIRT for their encouragement. Thanks to Michael Hale Ligh, Andrew Case, Jamie Levy, Aaron Walters, Matt Suiche, Ilfak Guilfanov, and Lenny Zeltser who have inspired and motivated me with their work. Thanks to Sajan Shetty, Vijay Sharma, Gavin Reid, Levi Gundert, Joanna Kretowicz, Marta Strzelec, Venkatesh Murthy, Amit Malik, and Ashwin Patil for their unending support. Thanks to the authors of other books, websites, blogs, and tools, which have contributed to my knowledge, and therefore this book.

About the reviewers Daniel Cuthbert is the Global Head of Security Research in Banco Santander. In his 20+ years' career on both the offensive and defensive side, he's seen the evolution of hacking from small groups of curious minds to the organized criminal networks and nation states we see today. He sits on the Black Hat Review Board and is the co-author of the OWASP Testing Guide (2003) and OWASP Application Security Verification Standard (ASVS). Dr. Michael Spreitzenbarth has been freelancing in the IT security sector for several years after finishing his diploma thesis with his major topic being mobile phone forensics. In 2013, he finished his PhD in the field of Android forensics and mobile malware analysis. Then, he started working at an internationally operating CERT and in an internal RED team. He deals daily with the security of mobile systems, forensic analysis of smartphones, and suspicious mobile applications, as well as the investigation of security-related incidents and simulating cybersecurity attacks.

Packt is searching for authors like you If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.

Table of Contents Preface Chapter 1: Introduction to Malware Analysis 1. What Is Malware? 2. What Is Malware Analysis? 3. Why Malware Analysis? 4. Types Of Malware Analysis 5. Setting Up The Lab Environment

5.1 Lab Requirements 5.2 Overview Of Lab Architecture 5.3 Setting Up And Configuring Linux VM 5.4 Setting Up And Configuring Windows VM

6. Malware Sources Summary Chapter 2: Static Analysis 1. Determining the File Type

1.1 Identifying File Type Using Manual Method 1.2 Identifying File Type Using Tools 1.3 Determining File Type Using Python

2. Fingerprinting the Malware

2.1 Generating Cryptographic Hash Using Tools 2.2 Determining Cryptographic Hash in Python

3. Multiple Anti-Virus Scanning

3.1 Scanning the Suspect Binary with VirusTotal 3.2 Querying Hash Values Using VirusTotal Public API

4. Extracting Strings

4.1 String Extraction Using Tools 4.2 Decoding Obfuscated Strings Using FLOSS

5. Determining File Obfuscation

5.1 Packers and Cryptors 5.2 Detecting File Obfuscation Using Exeinfo PE

6. Inspecting PE Header Information

6.1 Inspecting File Dependencies and Imports 6.2 Inspecting Exports 6.3 Examining PE Section Table And Sections 6.4 Examining the Compilation Timestamp 6.5 Examining PE Resources

7. Comparing And Classifying The Malware

1 7 7 9 9 10 11 12 12 14 21 24 25 27 27 28 29 29 31 31 32 32 33 34 36 37 39 40 41 43 44 45 49 50 53 54 56

Table of Contents

7.1 Classifying Malware Using Fuzzy Hashing 7.2 Classifying Malware Using Import Hash 7.3 Classifying Malware Using Section Hash 7.4 Classifying Malware Using YARA 7.4.1 Installing YARA 7.4.2 YARA Rule Basics 7.4.3 Running YARA 7.4.4 Applications of YARA

Summary Chapter 3: Dynamic Analysis 1. Lab Environment Overview 2. System And Network Monitoring 3. Dynamic Analysis (Monitoring) Tools

3.1 Process Inspection with Process Hacker 3.2 Determining System Interaction with Process Monitor 3.3 Logging System Activities Using Noriben 3.4 Capturing Network Traffic With Wireshark 3.5 Simulating Services with INetSim

4. Dynamic Analysis Steps 5. Putting it All Together: Analyzing a Malware Executable 5.1 Static Analysis of the Sample 5.2 Dynamic Analysis of the Sample

6. Dynamic-Link Library (DLL) Analysis 6.1 Why Attackers Use DLLs 6.2 Analyzing the DLL Using rundll32.exe

6.2.1 Working of rundll32.exe 6.2.2 Launching the DLL Using rundll32.exe

Example 1 – Analyzing a DLL With No Exports Example 2 – Analyzing a DLL Containing Exports Example 3 – Analyzing a DLL Accepting Export Arguments

6.3 Analyzing a DLL with Process Checks

Summary Chapter 4: Assembly Language and Disassembly Primer 1. Computer Basics 1.1 Memory

1.1.1 How Data Resides In Memory

1.2 CPU

1.2.1 Machine Language

1.3 Program Basics

1.3.1 Program Compilation 1.3.2 Program On Disk 1.3.3 Program In Memory 1.3.4 Program Disassembly (From Machine code To Assembly code)

2. CPU Registers

2.1 General-Purpose Registers

[ ii ]

57 59 60 61 61 62 63 64 69 71 72 73 73 74 75 76 78 79 82 82 83 85 88 90 91 91 92 92 94 95 96 98 99 100 101 102 102 102 103 103 103 105 108 109 109

Table of Contents

2.2 Instruction Pointer (EIP) 2.3 EFLAGS Register

3. Data Transfer Instructions

3.1 Moving a Constant Into Register 3.2 Moving Values From Register To Register 3.3 Moving Values From Memory To Registers 3.4 Moving Values From Registers To Memory 3.5 Disassembly Challenge 3.6 Disassembly Solution

4. Arithmetic Operations

4.1 Disassembly Challenge 4.2 Disassembly Solution

5. Bitwise Operations 6. Branching And Conditionals 6.1 Unconditional Jumps 6.2 Conditional Jumps 6.3 If Statement 6.4 If-Else Statement 6.5 If-Elseif-Else Statement 6.6 Disassembly Challenge 6.7 Disassembly Solution

7. Loops

7.1 Disassembly Challenge 7.2 Disassembly Solution

8. Functions

8.1 Stack 8.2 Calling Function 8.3 Returning From Function 8.4 Function Parameters And Return Values

9. Arrays And Strings

9.1 Disassembly Challenge 9.2 Disassembly Solution 9.3 Strings

9.3.1 String Instructions 9.3.2 Moving From Memory To Memory (movsx) 9.3.3 Repeat Instructions (rep) 9.3.4 Storing Value From Register to Memory (stosx) 9.3.5 Loading From Memory to Register (lodsx) 9.3.6 Scanning Memory (scasx) 9.3.7 Comparing Values in Memory (cmpsx)

10. Structures 11. x64 Architecture

11.1 Analyzing 32-bit Executable On 64-bit Windows

12. Additional Resources 13. Summary

[ iii ]

110 110 110 110 111 111 113 114 114 116 117 118 120 121 122 122 123 124 125 126 126 129 131 132 134 134 136 136 136 142 143 144 148 149 149 150 151 151 151 151 152 153 155 156 156

Table of Contents

Chapter 5: Disassembly Using IDA 1. Code Analysis Tools 2. Static Code Analysis (Disassembly) Using IDA 2.1 Loading Binary in IDA 2.2 Exploring IDA Displays

2.2.1 Disassembly Window 2.2.2 Functions Window 2.2.3 Output Window 2.2.4 Hex View Window 2.2.5 Structures Window 2.2.6 Imports Window 2.2.7 Exports Window 2.2.8 Strings Window 2.2.9 Segments Window

2.3 Improving Disassembly Using IDA 2.3.1 Renaming Locations 2.3.2 Commenting in IDA 2.3.3 IDA Database 2.3.4 Formatting Operands 2.3.5 Navigating Locations 2.3.6 Cross-References 2.3.7 Listing All Cross-References 2.3.8 Proximity View And Graphs

3. Disassembling Windows API 3.1 Understanding Windows API

3.1.1 ANSI and Unicode API Functions 3.1.2 Extended API Functions

3.2 Windows API 32-Bit and 64-Bit Comparison

4. Patching Binary Using IDA 4.1 Patching Program Bytes 4.2 Patching Instructions

5. IDA Scripting and Plugins 5.1 Executing IDA Scripts 5.2 IDAPython

5.2.1 Checking The Presence Of CreateFile API 5.2.2 Code Cross-References to CreateFile Using IDAPython

5.3 IDA Plugins

6. Summary Chapter 6: Debugging Malicious Binaries 1. General Debugging Concepts

1.1 Launching And Attaching To Processes 1.2 Controlling Process Execution 1.3 Interrupting a Program with Breakpoints 1.4 Tracing Program Execution

2. Debugging a Binary Using x64dbg

2.1 Launching a New Process in x64dbg

[ iv ]

157 157 158 159 161 161 163 164 164 164 164 165 165 165 166 168 169 170 172 172 173 176 177 179 180 185 185 185 188 189 191 192 192 193 194 195 196 196 197 198 198 199 200 201 201 202

Table of Contents

2.2 Attaching to an Existing Process Using x64dbg 2.3 x64dbg Debugger Interface 2.4 Controlling Process Execution Using x64dbg 2.5 Setting a Breakpoint in x64dbg 2.6 Debugging 32-bit Malware 2.7 Debugging 64-bit Malware 2.8 Debugging a Malicious DLL Using x64dbg

2.8.1 Using rundll32.exe to Debug the DLL in x64dbg 2.8.2 Debugging a DLL in a Specific Process

2.9 Tracing Execution in x64dbg 2.9.1 Instruction Tracing 2.9.2 Function Tracing

2.10 Patching in x64dbg

3. Debugging a Binary Using IDA

3.1 Launching a New Process in IDA 3.2 Attaching to an Existing Process Using IDA 3.3 IDA's Debugger Interface 3.4 Controlling Process Execution Using IDA 3.5 Setting a Breakpoint in IDA 3.6 Debugging Malware Executables 3.7 Debugging a Malicious DLL Using IDA 3.7.1 Debugging a DLL in a Specific Process

3.8 Tracing Execution Using IDA 3.9 Debugger Scripting Using IDAPython

3.9.1 Example – Determining Files Accessed by Malware

4. Debugging a .NET Application Summary Chapter 7: Malware Functionalities and Persistence 1. Malware Functionalities 1.1 Downloader 1.2 Dropper

1.2.1 Reversing a 64-bit Dropper

1.3 Keylogger

1.3.1 Keylogger Using GetAsyncKeyState() 1.3.2 Keylogger Using SetWindowsHookEx()

1.4 Malware Replication Via Removable Media 1.5 Malware Command and Control (C2) 1.5.1 HTTP Command and Control 1.5.2 Custom Command and Control

1.6 PowerShell-Based Execution

1.6.1 PowerShell Command Basics 1.6.2 PowerShell Scripts And Execution Policy 1.6.2 Analyzing PowerShell Commands/Scripts 1.6.3 How Attackers Use PowerShell

2. Malware Persistence Methods

[v]

203 204 208 208 209 210 213 214 215 216 218 219 220 221 222 222 223 226 226 228 229 231 232 235 237 239 241 243 243 243 245 247 247 248 249 250 255 255 259 262 263 264 265 266 268

Table of Contents

2.1 Running the Registry Key 2.2 Scheduled Tasks 2.3 Startup Folder 2.4 Winlogon Registry Entries 2.5 Image File Execution Options 2.6 Accessibility Programs 2.7 AppInit_DLLs 2.8 DLL Search Order Hijacking 2.9 COM hijacking 2.10 Service

Summary Chapter 8: Code Injection and Hooking 1. Virtual Memory

1.1 Process Memory Components (User Space) 1.2 Kernel Memory Contents (Kernel Space)

2. User Mode And Kernel Mode 2.1 Windows API Call Flow

3. Code Injection Techniques

3.1 Remote DLL Injection 3.2 DLL Injection Using APC (APC Injection) 3.3 DLL Injection Using SetWindowsHookEx() 3.4 DLL Injection Using The Application Compatibility Shim 3.4.1 Creating A Shim 3.4.2 Shim Artifacts 3.4.3 How Attackers Use Shims 3.4.4 Analyzing The Shim Database

3.5 Remote Executable/Shellcode Injection 3.6 Hollow Process Injection (Process Hollowing)

4. Hooking Techniques

4.1 IAT Hooking 4.2 Inline Hooking (Inline Patching) 4.3 In-memory Patching Using Shim

5. Additional Resources Summary Chapter 9: Malware Obfuscation Techniques 1. Simple Encoding 1.1 Caesar Cipher

1.1.1 Working Of Caesar Cipher 1.1.2 Decrypting Caesar Cipher In Python

1.2 Base64 Encoding

1.2.1 Translating Data To Base64 1.2.2 Encoding And Decoding Base64 1.2.3 Decoding Custom Base64 1.2.4 Identifying Base64

[ vi ]

268 269 270 271 272 273 275 276 278 281 286 287 288 291 292 293 295 297 299 302 304 306 307 312 313 314 315 317 322 322 324 326 330 331 333 335 335 335 337 338 338 339 341 344

Table of Contents

1.3 XOR Encoding

1.3.1 Single Byte XOR 1.3.2 Finding XOR Key Through Brute-Force 1.3.3 NULL Ignoring XOR Encoding 1.3.4 Multi-byte XOR Encoding 1.3.5 Identifying XOR Encoding

2. Malware Encryption

2.1 Identifying Crypto Signatures Using Signsrch 2.2 Detecting Crypto Constants Using FindCrypt2 2.3 Detecting Crypto Signatures Using YARA 2.4 Decrypting In Python

3. Custom Encoding/Encryption 4. Malware Unpacking 4.1 Manual Unpacking

4.1.1 Identifying The OEP 4.1.2 Dumping Process Memory With Scylla 4.1.3 Fixing The Import Table

4.2 Automated Unpacking

Summary Chapter 10: Hunting Malware Using Memory Forensics 1. Memory Forensics Steps 2. Memory Acquisition 2.1 Memory Acquisition Using DumpIt

3. Volatility Overview

3.1 Installing Volatility

3.1.1 Volatility Standalone Executable 3.1.2 Volatility Source Package

3.2 Using Volatility

4. Enumerating Processes 4.1 Process Overview

4.1.1 Examining the _EPROCESS Structure 4.1.2 Understanding ActiveProcessLinks

4.2 Listing Processes Using psscan

4.2.1 Direct Kernel Object Manipulation (DKOM) 4.2.2 Understanding Pool Tag Scanning

4.3 Determining Process Relationships 4.4 Process Listing Using psxview

5. Listing Process Handles 6. Listing DLLs

6.1 Detecting a Hidden DLL Using ldrmodules

7. Dumping an Executable and DLL 8. Listing Network Connections and Sockets 9. Inspecting Registry 10. Investigating Service [ vii ]

345 346 349 350 352 354 355 355 359 359 361 362 367 368 368 372 373 374 377 379 380 380 381 384 384 384 385 386 388 389 390 394 396 397 398 401 402 404 406 410 411 413 415 417

Table of Contents

11. Extracting Command History Summary Chapter 11: Detecting Advanced Malware Using Memory Forensics 1. Detecting Code Injection 1.1 Getting VAD Information 1.2 Detecting Injected Code Using VAD 1.3 Dumping The Process Memory Region 1.4 Detecting Injected Code Using malfind

2. Investigating Hollow Process Injection 2.1 Hollow Process Injection Steps 2.2 Detecting Hollow Process Injection 2.3 Hollow Process Injection Variations

3. Detecting API Hooks 4. Kernel Mode Rootkits 5. Listing Kernel Modules

5.1 Listing Kernel Modules Using driverscan

6. I/O Processing

6.1 The Role Of The Device Driver 6.2 The Role Of The I/O Manager 6.3 Communicating With The Device Driver 6.4 I/O Requests To Layered Drivers

7. Displaying Device Trees 8. Detecting Kernel Space Hooking 8.1 Detecting SSDT Hooking 8.2 Detecting IDT Hooking 8.3 Identifying Inline Kernel Hooks 8.4 Detecting IRP Function Hooks

9. Kernel Callbacks And Timers Summary Other Books You May Enjoy Index

419 421 423 424 425 427 429 430 431 431 433 435 438 439 440 443 444 447 454 455 457 461 464 464 467 468 470 473 479 481 485

[ viii ]

Preface The advancement of the computer and internet technology has changed our lives, and it has revolutionized the way the organizations conduct businesses. However, technology evolution and digitization has given rise to cybercriminal activities. The growing threat of cyberattacks on critical infrastructure, data centers, and private/public, defence, energy, government, and financial sectors poses a unique challenge for everyone from an individual to large corporations. These cyberattacks make use of malicious software (also known as Malware ) for financial theft, espionage, sabotage, intellectual property theft, and political motives. With adversaries becoming sophisticated and carrying out advanced malware attacks, detecting and responding to such intrusions is critical for cybersecurity professionals. Malware analysis has become a must-have skill for fighting advanced malware and targeted attacks. Malware analysis requires a well-balanced knowledge of many different skills and subjects. In other words, learning malware analysis demands time and requires patience. This book teaches the concepts, tools, and techniques to understand the behavior and characteristics of Windows malware using malware analysis. This book starts by introducing you to basic concepts of malware analysis. It then gradually progresses deep into more advanced concepts of code analysis and memory forensics. To help you understand the concepts better, various real-world malware samples, infected memory images, and visual diagrams are used in the examples throughout the book. In addition to this, enough information is given to help you understand the required concepts, and wherever possible, references to additional resources are provided for further reading. If you are new to the field of malware analysis, this book should help you get started, or if you are experienced in this field, this book will help enhance your knowledge further. Although you are learning malware analysis to perform a forensic investigation, to respond to an incident, or for fun, this book enables you to accomplish your goals.

Who this book is for If you're an incident responder, cybersecurity investigator, system administrator, malware analyst, forensic practitioner, student, or a curious security professional interested in learning or enhancing your malware analysis skills, then this book is for you.

Learning

Malware Analysis Malware analysis and memory forensics are powerful analysis and investigation techniques used in reverse engineering, digital forensics, and incident response. With adversaries becoming sophisticated and carrying out advanced malware attacks on critical infrastructures, data centers, and private and public organizations, detecting, responding to, and investigating such intrusions is critical to information security professionals. Malware analysis and memory forensics have become must-have skills to fight advanced malware, targeted attacks, and security breaches. This book teaches you the concepts, techniques, and tools to understand

the behavior and characteristics of malware through malware analysis. It also teaches you techniques to investigate and hunt malware using memory forensics. This book introduces you to the basics of malware analysis, and then gradually progresses into the more advanced concepts of code analysis and memory forensics. It uses real-world malware samples, infected memory images, and visual diagrams to help you gain a better understanding of the subject and to equip you with the skills required to analyze, investigate, and respond to malware-related incidents.

Things you will learn: •

Create a safe and isolated lab environment for malware analysis

•

Reverse-engineer various malware functionalities

•

Extract the metadata associated with malware

•

Reverse engineer and decode common encoding/encryption algorithms

•

Determine malware's interaction with the system

•

Perform different code injection and hooking techniques

•

Perform code analysis using IDA Pro and x64dbg

•

Investigate and hunt malware using memory forensics

www.packt.com

FOR SALE IN INDIA ONLY