Skip to main content
The Function Trace Generator agent creates detailed function-level execution traces for C/C++ programs using compiler instrumentation. These traces are essential for crash analysis and understanding program behavior.

​
Purpose

Generate function-level execution traces that show:
  • Function entry and exit events
  • Call stack depth
  • Execution timeline
  • Thread-level information

​
Invocation

Invoked by the crash-analysis-agent as part of the crash analysis workflow. Receives:
  • Code repository path
  • Working directory path
  • Crashing example program and build instructions
Creates: traces/ subdirectory in working directory

​
Workflow

1

Build Instrumentation Library

cd .claude/skills/crash-analysis/function-tracing/

# Build the trace library
gcc -c -fPIC trace_instrument.c -o trace_instrument.o
gcc -shared trace_instrument.o -o libtrace.so -ldl -lpthread

# Build Perfetto converter
g++ -O3 -std=c++17 trace_to_perfetto.cpp -o trace_to_perfetto
2

Rebuild Target with Instrumentation

Add instrumentation flags to the build:
  • Add -finstrument-functions -g to CFLAGS
  • Add -L<path-to-libtrace> -ltrace -ldl -lpthread to LDFLAGS
Autotools:
./configure CFLAGS="-finstrument-functions -g" \
            LDFLAGS="-L/path/to/libtrace -ltrace -ldl -lpthread"
CMake:
cmake -DCMAKE_C_FLAGS="-finstrument-functions -g" \
      -DCMAKE_EXE_LINKER_FLAGS="-L/path/to/libtrace -ltrace -ldl -lpthread" ..
Makefile:
make CFLAGS="-finstrument-functions -g" \
     LDFLAGS="-L/path/to/libtrace -ltrace -ldl -lpthread"
3

Run Crashing Program

export LD_LIBRARY_PATH=/path/to/libtrace:$LD_LIBRARY_PATH
<crashing-command>
# Creates trace_<tid>.log files
4

Convert to Perfetto Format (Optional)

./trace_to_perfetto trace_*.log -o traces/trace.json
# Can be viewed at ui.perfetto.dev
5

Move Trace Files

Copy all trace files to the traces/ subdirectory in working directory

​
Instrumentation Details

The -finstrument-functions flag causes the compiler to insert calls to: 
void __cyg_profile_func_enter(void *this_fn, void *call_site);
void __cyg_profile_func_exit(void *this_fn, void *call_site);
These are implemented by libtrace.so to log:
  • Function address
  • Timestamp (nanosecond precision)
  • Thread ID
  • Entry/exit event type

​
Trace File Format

Raw trace files (trace_<tid>.log) contain: 
[0] [1.000000000]  [ENTRY] main
[1] [1.000050000] . [ENTRY] some_function
[2] [1.000100000] .. [ENTRY] helper_function
[3] [1.000150000] .. [EXIT]  helper_function
[4] [1.000200000] . [EXIT]  some_function
[5] [1.000250000]  [EXIT]  main
  • [N]: Event sequence number
  • [timestamp]: Nanoseconds since start
  • Dots: Call depth visualization
  • [ENTRY/EXIT]: Event type
  • Function name: Resolved from debug symbols

​
Perfetto Format

The Perfetto JSON format enables visualization at ui.perfetto.dev: 
{
  "traceEvents": [
    {
      "name": "main",
      "cat": "function",
      "ph": "B",
      "ts": 1000000000,
      "pid": 12345,
      "tid": 12345
    },
    {
      "name": "main",
      "cat": "function",
      "ph": "E",
      "ts": 1000250000,
      "pid": 12345,
      "tid": 12345
    }
  ]
}

​
Validation

After generating traces, validate: Example validation: 
# Check trace files exist
ls traces/trace_*.log

# Check for function events
head -50 traces/trace_*.log

# Should see lines like:
# [0] [1.000000000]  [ENTRY] main
# [1] [1.000050000] . [ENTRY] some_function

​
Usage in Crash Analysis

The crash-analyzer-agent uses function traces to:
  1. Verify execution path: Confirm hypothesized functions were actually called
  2. Track control flow: Follow execution from entry to crash
  3. Identify missing functions: Detect functions that should have been called but weren’t
  4. Correlate with coverage: Cross-reference with gcov data

​
Performance Impact

Function instrumentation adds significant overhead:
  • 10-100x slowdown typical
  • Large trace files (MB-GB for complex programs)
  • Memory overhead for buffering
Recommendations:
  • Use only for crash reproduction, not production
  • Limit trace duration to necessary execution
  • Consider filtering high-frequency functions if needed

​
Troubleshooting

  • Check LD_LIBRARY_PATH includes libtrace.so directory
  • Verify program actually executed (didn’t fail immediately)
  • Check write permissions in current directory
  • Program may have crashed before trace buffer flushed
  • Increase buffer size in trace_instrument.c
  • Add explicit flush before crash-prone code
  • Missing debug symbols (-g flag)
  • Stripped binary
  • Use addr2line or nm to resolve manually
  • Filter out high-frequency functions
  • Limit tracing to specific code sections
  • Use sampling instead of full instrumentation

​
Output Structure

<working-dir>/traces/
├── trace_12345.log          # Raw trace (thread 12345)
├── trace_12346.log          # Raw trace (thread 12346)
└── trace.json               # Perfetto format (all threads)

Crash Analysis

Main crash analysis orchestrator

Coverage Analyzer

Complementary coverage data generation

Build docs developers (and LLMs) love

Get started for freeTalk to us