Skip to main content
Reverse engineering (RE) is the process of understanding a program’s behaviour without access to its source code. In a security context this means recovering logic, finding hidden functionality, identifying vulnerabilities, or understanding malware.

​
Common targets

Malware analysis

Understand what a suspicious binary does: C2 communication, persistence mechanisms, evasion tricks.

CTF challenges

Find hidden flags by reversing licence checks, custom crypto, obfuscated logic, or game mechanics.

Vulnerability research

Identify unsafe functions, integer overflows, or logic flaws in closed-source software.

Interoperability

Understand proprietary protocols or file formats to build compatible tooling.

​
Reversing methodology

1

Static overview

Run file, strings, and checksec first. Identify the architecture, format (ELF/PE/Mach-O), packing (UPX, custom), and enabled mitigations.
file ./target
strings ./target | grep -i flag
checksec --file=./target
2

Disassembly / decompilation

Load the binary into Ghidra, IDA, or Binary Ninja. Rename functions and variables as you understand them. Start from main or any well-known entry point.
3

Dynamic analysis

Run the binary under GDB, x64dbg, or Frida. Set breakpoints on interesting functions, inspect registers and memory at key points, and trace system calls with strace/ltrace.
4

Focus on interesting logic

Follow string references, xrefs to crypto constants (S-boxes, magic numbers like 0x61C88647), network functions, or comparison instructions that look like key or flag checks.
5

Patch or script

Use Ghidra scripts, IDA Python, or Frida hooks to automate analysis, patch out anti-debug checks, or dump decrypted payloads at runtime.

​
Key concepts

​
Calling conventions

Understanding how arguments are passed is essential to reading disassembly:
ConventionArgumentsReturn valueUsed on
x86 cdeclStack (right-to-left)EAXLinux/Windows 32-bit
x86-64 System VRDI, RSI, RDX, RCX, R8, R9RAXLinux/macOS 64-bit
x86-64 WindowsRCX, RDX, R8, R9RAXWindows 64-bit
ARM64 (AAPCS64)X0–X7X0Linux/macOS ARM64

​
Recognising common patterns

  • String comparisons: look for strcmp, memcmp, or XOR loops near conditional jumps.
  • Crypto constants: AES S-box starts with 0x63, SHA-256 uses specific round constants, RC4 has a 256-byte key-scheduling loop.
  • Anti-debug: IsDebuggerPresent, PTRACE_TRACEME self-check, timing checks with RDTSC, or exception-based tricks.
  • Packers: a short first section that allocates RWX memory, writes data into it, and jumps — the payload is unpacked at runtime.

​
Anti-analysis techniques in malware

Real-world malware uses many tricks to hinder analysis:
Many stealers abort execution on specific locale or keyboard layouts (commonly CIS countries) to avoid analysing researcher machines. The API chain is GetKeyboardLayout → GetLocaleInfoA/W → compare against a block-list.
Malware scans for Defender’s emulator exports (MpVmp32Entry, VFS_Open, ThrdMgr_GetCurrentThreadHandle, etc.). If found, it sleeps for 10–30 minutes before continuing.
A CLI switch (e.g., /i:--type=renderer mimicking Chromium) must be present or the loader exits immediately, preventing sandbox auto-execution.
Legitimate processes (RegAsm.exe, MSBuild.exe) are launched suspended, their image unmapped, and a malicious PE written in its place. The payload never touches disk in plain form.

​
Language-specific tips

Use dnSpy or ILSpy to decompile MSIL back to C#. For debugging, enable DebuggableAttribute in the assembly and attach dnSpy to the IIS/process.
// Instrument with dnSpy to log values at runtime
File.AppendAllText(@"C:\temp\debug.txt", "value: " + variable + "\n");

​
Quick tool reference

ToolCategoryNotes
GhidraDisassembler / decompilerFree, scriptable, great for large binaries
IDA Pro / FreeDisassembler / decompilerIndustry standard; free version covers x86/x64
Binary NinjaDisassembler / decompilerStrong API, good for automation
x64dbg / x32dbgDebugger (Windows)Plugin ecosystem, ScyllaHide for anti-anti-debug
GDB + pwndbg/GEFDebugger (Linux)Heap-aware, ROP-aware
FridaDynamic instrumentationInject JS hooks into any process, cross-platform
radare2 / CutterMulti-toolCLI + GUI; good for shellcode analysis
Detect-It-Easy (DIE)Packer/compiler IDRecognises packers, compilers, protectors

Build docs developers (and LLMs) love

Get started for freeTalk to us