Skip to main content

​
Overview

The /crash-analysis command provides autonomous root-cause analysis for C/C++ crashes. It uses deterministic record-replay debugging (rr), function tracing, and coverage analysis to identify the exact cause of security bugs.

​
Syntax

/crash-analysis <bug-tracker-url> <git-repo-url>

​
Parameters

​
bug-tracker-url
string
required
URL to the bug tracker report (e.g., Trac, GitHub Issues, Bugzilla)
​
git-repo-url
string
required
URL to the Git repository containing the vulnerable code

​
What It Does

  1. Fetches bug report from the provided URL
  2. Clones repository from Git URL
  3. Reads README to determine build process
  4. Rebuilds with instrumentation (AddressSanitizer + debug symbols)
  5. Reproduces the crash using inputs from bug report
  6. Generates execution traces with function-level granularity
  7. Collects coverage data using gcov
  8. Records with rr for deterministic replay
  9. Performs root-cause analysis with validation loop
  10. Produces confirmed hypothesis with evidence

​
Workflow Agents

The crash analysis workflow orchestrates multiple specialized agents:

​
Main Orchestrator

  • crash-analysis-agent: Coordinates the entire workflow

​
Analysis Agents

  • crash-analyzer-agent: Performs deep root-cause analysis using rr traces
  • crash-analyzer-checker-agent: Validates analysis rigorously

​
Data Collection Agents

  • function-trace-generator-agent: Creates function execution traces
  • coverage-analysis-generator-agent: Generates gcov coverage data

​
Examples

​
Analyze FFmpeg Crash

/crash-analysis https://trac.ffmpeg.org/ticket/11234 https://github.com/FFmpeg/FFmpeg.git
Analyzes a reported crash in FFmpeg.

​
Analyze ImageMagick Vulnerability

/crash-analysis https://github.com/ImageMagick/ImageMagick/issues/5678 https://github.com/ImageMagick/ImageMagick.git
Investigates a security bug in ImageMagick.

​
Analyze OpenSSL Issue

/crash-analysis https://github.com/openssl/openssl/issues/9012 https://github.com/openssl/openssl.git
Analyzes a reported issue in OpenSSL.

​
Prerequisites

​
Required Tools

​
rr
tool
required
Record-replay debugger for deterministic debugging
apt install rr  # Debian/Ubuntu
# or build from source: https://github.com/rr-debugger/rr
​
gcc/clang
tool
required
Compiler with AddressSanitizer support
apt install gcc g++ clang
​
gdb
tool
required
GNU Debugger for debugging rr traces
apt install gdb
​
gcov
tool
required
Code coverage tool (bundled with gcc)
# Usually installed with gcc
apt install gcc

​
Analysis Features

​
Deterministic Replay with rr

rr records program execution and allows perfect replay: 
# Record the crash
rr record ./vulnerable_program < crash_input

# Replay infinitely many times
rr replay
Benefits:
  • Exact reproduction of crash every time
  • Reverse execution to find root cause
  • No Heisenbugs (observer effect eliminated)
  • Shareable traces for collaboration

​
Function Tracing

Instruments code to log all function calls: 
# Compile with instrumentation
gcc -finstrument-functions program.c -o program

# Generate trace
./program > trace.json
Visualize in Perfetto for execution flow analysis.

​
Coverage Analysis

Collects line-level coverage data: 
# Compile with coverage
gcc --coverage program.c -o program

# Run and collect coverage
./program
gcov program.c
Identifies which lines executed during crash.

​
Hypothesis-Validation Loop

The analysis follows a rigorous validation process:
  1. Analyzer forms hypothesis about root cause
  2. Checker validates hypothesis against evidence
  3. If rejected, analyzer revises with feedback
  4. If confirmed, produces final report
This ensures high-quality, evidence-backed analysis.

​
Output Structure

./crash-analysis-<timestamp>/
├── rr-trace/                        # Deterministic replay recording
│   └── (shareable for debugging)
├── traces/                          # Function execution traces
│   ├── trace.json
│   └── trace.perfetto              # Visualize in Perfetto
├── gcov/                            # Code coverage data
│   ├── program.c.gcov
│   └── coverage-summary.txt
├── root-cause-hypothesis-v1.md      # Initial analysis
├── root-cause-hypothesis-v2.md      # Revised analysis (if needed)
├── root-cause-hypothesis-confirmed.md  # Final validated analysis
└── crash-analysis-report.md         # Human-readable summary

​
Root Cause Report Format

# Root Cause Analysis: CVE-2024-XXXX

## Summary
Heap buffer overflow in image parsing function.

## Vulnerability Details
**Type**: Heap Buffer Overflow  
**Location**: src/image.c:142  
**Function**: `parse_image_header()`  
**CWE**: CWE-787 (Out-of-bounds Write)

## Root Cause
The function allocates a fixed-size buffer but reads user-controlled
size from file header without validation.

```c
char *buffer = malloc(1024);  // Fixed size
size_t len = read_size_from_header(file);  // Attacker controlled
memcpy(buffer, data, len);  // No bounds check!

​
Attack Vector

  1. Attacker creates malicious image with header size > 1024
  2. Application reads size: 4096 bytes
  3. memcpy writes beyond buffer bounds
  4. Heap corruption occurs

​
Evidence

  • rr trace: Crash at memcpy+0x42
  • ASAN report: Heap-buffer-overflow write
  • Coverage: Line 142 executed
  • Function trace: parse_image_header called from main

​
Impact

  • Remote code execution via heap corruption
  • Denial of service via crash
  • Information disclosure via memory leaks

​
Recommendations

  1. Validate size from header: if (len > 1024) return ERROR;
  2. Use safe memory functions: memcpy_s(buffer, sizeof(buffer), data, len)
  3. Add fuzzing to test suite
  4. Enable ASAN in CI/CD pipeline

## Use Cases

- Security bug analysis
- Crash debugging for complex applications
- Root cause identification for CVEs
- Exploit development research
- Fuzzing campaign analysis
- Bug bounty investigation

## Advanced Features

### Shareable rr Traces

rr traces can be packaged and shared:

```bash
# Package trace
tar czf crash-trace.tar.gz crash-analysis-*/rr-trace/

# Share with team or security researchers
# Recipients can replay exact crash

​
Reverse Execution

Step backwards through program execution: 
rr replay
(rr) reverse-continue  # Run backwards to previous breakpoint
(rr) reverse-step      # Step backwards
(rr) reverse-next      # Step over backwards

​
Integration with Fuzzing

Combine with /fuzz for comprehensive analysis: 
# 1. Fuzz the binary
/fuzz /path/to/binary --duration 3600

# 2. Analyze crash
/crash-analysis <bug-url> <repo-url>

​
Workflow Integration

/fuzz -> /crash-analysis -> /exploit
   |            |              |
   v            v              v
 Find        Understand     Develop
 crashes     root cause     exploits

/fuzz

Find crashes through fuzzing

/exploit

Generate exploits from analysis

/validate

Validate exploitability

/patch

Generate fixes for vulnerabilities

​
Skills Referenced

The crash analysis workflow uses skills from .claude/skills/crash-analysis/:
  • rr-debugger: Deterministic record-replay debugging
  • function-tracing: Function instrumentation with -finstrument-functions
  • gcov-coverage: Code coverage collection
  • line-execution-checker: Fast line execution queries

​
Notes

  • Requires Linux (rr only supports Linux)
  • Works best with C/C++ applications
  • rr traces can be large (hundreds of MBs)
  • Analysis follows hypothesis-validation loop for accuracy
  • Produces shareable, reproducible crash traces
  • For security research and authorized testing only

Build docs developers (and LLMs) love

Get started for freeTalk to us