Debugging Unikraft Applications

Overview

Debugging unikernels requires specialized techniques due to their unique architecture. This guide covers:

GDB remote debugging with GDB stub
QEMU’s built-in GDB support
Debug logging and tracing
Crash analysis
Performance profiling

Debug Build Configuration

Before debugging, ensure your unikernel is built with debug symbols.

Using kraft CLI

# Debug build (default includes debug symbols)
kraft build

# The .dbg image contains full debug information
ls build/*.dbg

Using menuconfig

Open configuration

make menuconfig

Disable optimizations

Build Options  --->
  Optimization level  --->
    (X) No optimizations (-O0)

Enable maximum debug symbols

Build Options  --->
  Debug information level  --->
    (X) Level 3 (-g3)

Keep frame pointers

Build Options  --->
  [X] Keep stack frame pointers

Enable debug features

Library Configuration  --->
  [X] ukdebug: Debugging features  --->
    [X] Enable debug messages
    [X] Enable assertions (UK_BUGON)
    [X] Print stack trace on crash

GDB Remote Debugging (Recommended)

Unikraft includes a native GDB stub that allows full debugging capabilities.

Prerequisites

Enable GDB stub in configuration

Library Configuration  --->
  [X] ukdebug: Debugging features  --->
    [X] uklibparam: Kernel parameter parser
    [X] GDB stub support

Rebuild with debug configuration

make clean
make -j$(nproc)

Setting Up GDB Debugging

Start QEMU with GDB stub

Launch your unikernel with serial console connected to TCP:

qemu-system-x86_64 \
  -kernel build/app-myapp_qemu-x86_64 \
  -cpu max \
  -nographic \
  -serial tcp::1235,server,nowait \
  -append "build/app-myapp_qemu-x86_64 debug.gdbcon=0 --"

Parameters explained:

-serial tcp::1235,server,nowait: Creates TCP socket for GDB on port 1235
debug.gdbcon=0: Tells GDB stub to use console 0 (COM1)
First parameter must be the kernel name (Unikraft convention)

Start GDB in another terminal

Use the .dbg image which contains symbols:

gdb build/app-myapp_qemu-x86_64.dbg

For cross-compiled builds, use the appropriate GDB:

aarch64-linux-gnu-gdb build/app-myapp_qemu-arm64.dbg

Connect to the GDB stub

Inside GDB:

(gdb) target remote localhost:1235
Remote debugging using localhost:1235
0x0000000000120e02 in gdb_entry (ctx=<optimized out>) at /unikraft/lib/ukdebug/gdbstub.c:1247

You’re now connected and execution is paused.

Start debugging

(gdb) break main
Breakpoint 1 at 0x123456: file main.c, line 42.

(gdb) continue
Continuing.

Common GDB Commands

# Set breakpoint at function
(gdb) break main
(gdb) break myfunction

# Set breakpoint at file:line
(gdb) break main.c:42

# List breakpoints
(gdb) info breakpoints

# Delete breakpoint
(gdb) delete 1

# Conditional breakpoint
(gdb) break main.c:42 if x == 5

Advanced GDB Usage

Watch Variables

# Watch for writes
(gdb) watch variable_name

# Watch for reads
(gdb) rwatch variable_name

# Watch for reads or writes
(gdb) awatch variable_name

Core Dumps

# Generate core dump
(gdb) generate-core-file myapp.core

# Analyze later
gdb build/app-myapp_qemu-x86_64.dbg myapp.core

Remote Debugging Script

Create a .gdbinit file:

.gdbinit

# Connect automatically
target remote localhost:1235

# Load symbols
symbol-file build/app-myapp_qemu-x86_64.dbg

# Set breakpoints
break main
break error_handler

# Continue to main
continue

Then run:

gdb -x .gdbinit

Debugging with Two Consoles

Separate kernel output from GDB:

Configure two serial ports

Platform Configuration  --->
  KVM guest  --->
    [*] Serial console on COM1
    [*] Serial console on COM2

Launch QEMU with two serial devices

qemu-system-x86_64 \
  -kernel build/app-myapp_qemu-x86_64 \
  -cpu max \
  -nographic \
  -serial stdio \
  -serial tcp::1235,server,nowait \
  -append "build/app-myapp_qemu-x86_64 debug.gdbcon=1 --"

Now COM1 shows kernel output (stdio) and COM2 connects to GDB (TCP:1235).

QEMU Built-in GDB Stub

QEMU has its own GDB stub for low-level debugging.

Basic QEMU GDB Usage

Start QEMU with GDB server

qemu-system-x86_64 \
  -kernel build/app-myapp_qemu-x86_64 \
  -s \
  -S \
  -nographic

Options:

-s: Enable GDB server on localhost:1234
-S: Pause execution at startup

Connect GDB

gdb build/app-myapp_qemu-x86_64.dbg

(gdb) target remote localhost:1234
(gdb) break main
(gdb) continue

QEMU vs Unikraft GDB Stub

Feature	QEMU GDB Stub	Unikraft GDB Stub
Setup	Simple	Requires configuration
Access	Full system	Kernel context only
Boot debugging	Yes	No
Exception handling	Limited	Full support
Production use	No	Possible
Performance	Slower	Faster

Use QEMU GDB stub for early boot issues. Use Unikraft GDB stub for application debugging.

Debugging the GDB Stub Itself

For GDB stub development:

# Terminal 1: QEMU with dual GDB support
qemu-system-x86_64 \
  -s -S \
  -kernel build/app-myapp_qemu-x86_64 \
  -serial tcp::1235,server,nowait \
  -append "build/app-myapp_qemu-x86_64 debug.gdbcon=0 --"

# Terminal 2: GDB for QEMU stub (system level)
gdb build/app-myapp_qemu-x86_64.dbg
(gdb) target remote localhost:1234
(gdb) set remotetimeout 1000000
(gdb) break uk_gdb_stub_init
(gdb) continue

# Terminal 3: GDB for Unikraft stub (application level)
gdb build/app-myapp_qemu-x86_64.dbg
(gdb) target remote localhost:1235

Debug Logging

Unikraft provides comprehensive logging facilities.

Enable Debug Messages

Library Configuration  --->
  ukdebug: Debugging features  --->
    [X] Enable debug messages
    Print level  --->
      (X) Show all types of messages

Log Levels

Unikraft supports multiple log levels:

#include <uk/print.h>

uk_pr_debug("Debug message: %d\n", value);    // Debug
uk_pr_info("Info message\n");                 // Info
uk_pr_warn("Warning: %s\n", msg);             // Warning
uk_pr_err("Error occurred\n");                // Error
uk_pr_crit("Critical error\n");               // Critical

Runtime Log Control

Control verbosity via kernel parameters:

kraft run . -- --verbose

With QEMU:

qemu-system-x86_64 \
  -kernel build/app-myapp_qemu-x86_64 \
  -append "build/app-myapp_qemu-x86_64 -- --verbose" \
  -nographic

Capture Logs

kraft run . 2>&1 | tee debug.log

Assertions and Crash Handling

Enable Assertions

Library Configuration  --->
  ukdebug: Debugging features  --->
    [X] Enable assertions (UK_BUGON)
    [X] Print stack trace on crash

Using Assertions

#include <uk/assert.h>

UK_ASSERT(pointer != NULL);
UK_BUGON(condition);  // Crashes if condition is true

if (error) {
    UK_CRASH("Fatal error: %s\n", error_msg);
}

Stack Trace on Crash

When enabled, crashes show full stack trace:

[CRIT] <uk/assert.h>: Assertion failed: pointer != NULL
Stack trace:
#0  0x00000000001234ab in main+0x42
#1  0x00000000001156cd in _ukplat_entry+0x15
...

Memory Debugging

Debugging Allocator

Enable debug allocator:

Library Configuration  --->
  ukalloc: Memory allocators  --->
    Default memory allocator  --->
      (X) Debug allocator
    [X] Enable memory allocation statistics

Detect Memory Leaks

The debug allocator tracks allocations:

#include <uk/alloc.h>

uk_alloc_stats_print(uk_alloc_get_default());

Address Sanitizer (Future)

ASAN support is in development. Track progress in the Unikraft repository.

Performance Profiling

Using perf with KVM

# Record performance data
sudo perf kvm --host record \
  qemu-system-x86_64 \
  -kernel build/app-myapp_kvm-x86_64 \
  -enable-kvm \
  ...

# Analyze results
sudo perf report

CPU Time Profiling

Use QEMU’s built-in profiler:

qemu-system-x86_64 \
  -kernel build/app-myapp_qemu-x86_64 \
  -d cpu,exec \
  -D qemu-trace.log \
  -nographic

Function Call Tracing

Enable function tracing:

Build Options  --->
  [X] Enable function tracing (-pg)

Generate call graph with gprof:

gprof build/app-myapp_qemu-x86_64.dbg > callgraph.txt

Common Debugging Scenarios

Debugging Crashes

Build with debug symbols

make menuconfig
# Set optimization to -O0
# Set debug level to -g3
make clean && make

Run with GDB

# Terminal 1
qemu-system-x86_64 \
  -kernel build/app_qemu-x86_64 \
  -serial tcp::1235,server,nowait \
  -append "build/app_qemu-x86_64 debug.gdbcon=0 --" \
  -nographic

# Terminal 2
gdb build/app_qemu-x86_64.dbg
(gdb) target remote localhost:1235
(gdb) continue

Analyze crash

When crash occurs:

(gdb) backtrace
(gdb) info registers
(gdb) x/20x $rsp
(gdb) list

Debugging Network Issues

# Enable network debugging
qemu-system-x86_64 \
  -kernel build/app_qemu-x86_64 \
  -netdev user,id=net0,hostfwd=tcp::8080-:8080 \
  -device virtio-net-pci,netdev=net0 \
  -object filter-dump,id=net0,netdev=net0,file=network.pcap \
  -nographic

# Analyze with Wireshark
wireshark network.pcap

Debugging Boot Issues

Use QEMU GDB stub for early boot:

qemu-system-x86_64 \
  -kernel build/app_qemu-x86_64 \
  -s -S \
  -nographic

gdb build/app_qemu-x86_64.dbg
(gdb) target remote localhost:1234
(gdb) break _ukplat_entry
(gdb) continue

Debugging Infinite Loops

# Connect with GDB
(gdb) target remote localhost:1235

# Interrupt execution
(gdb) Ctrl-C

# Check where it's stuck
(gdb) backtrace
(gdb) info threads

Debugging Tools

GDB Enhanced (GEF)

Install GEF for better GDB experience:

bash -c "$(curl -fsSL https://gef.blah.cat/sh)"

Features:

Colored output
Memory view
Register display
Heap analysis

QEMU Monitor

Access QEMU monitor:

qemu-system-x86_64 \
  -kernel build/app_qemu-x86_64 \
  -monitor telnet::45454,server,nowait \
  -nographic

# In another terminal
telnet localhost 45454

Useful monitor commands:

(qemu) info registers
(qemu) info mem
(qemu) info tlb
(qemu) x/10x 0x100000

Best Practices

1. Always Use Debug Builds for Development

# Development
kraft build  # Includes debug symbols by default

# Production
kraft build --no-debug

2. Keep Debug and Release Configurations

# Save debug config
cp .config configs/debug.config

# Save release config
cp .config configs/release.config

3. Use Separate Consoles

Always separate GDB from application output when possible.

4. Enable Stack Traces

Always build with frame pointers and stack trace support:

[X] Keep stack frame pointers
[X] Print stack trace on crash

5. Version Control Debug Scripts

Save your GDB scripts and QEMU commands:

# .gdbinit
target remote localhost:1235
break main
continue

# run-debug.sh
#!/bin/bash
qemu-system-x86_64 \
  -kernel build/app_qemu-x86_64 \
  -serial tcp::1235,server,nowait \
  -append "build/app_qemu-x86_64 debug.gdbcon=0 --" \
  -nographic

Troubleshooting Debugging

GDB Can’t Connect

Problem: “Connection refused” Solution:

# Check if QEMU is listening
netstat -tlnp | grep 1235

# Verify serial configuration
qemu-system-x86_64 ... -serial tcp::1235,server,nowait ...

No Debug Symbols

Problem: “No debugging symbols found” Solution:

# Ensure using .dbg image
gdb build/app_qemu-x86_64.dbg

# Verify symbols exist
file build/app_qemu-x86_64.dbg
nm build/app_qemu-x86_64.dbg | grep main

GDB Timeout

Problem: “Remote connection timeout” Solution:

(gdb) set remotetimeout 1000000

Breakpoints Not Hit

Problem: Breakpoints not triggering Solution:

# Check if code is optimized out
(gdb) disassemble main

# Rebuild with -O0
make menuconfig  # Select "No optimizations"
make clean && make

Next Steps

Building Applications

Build debug-enabled unikernels

Configuration

Configure debug options

Running Unikernels

Run with debug flags

ukdebug Library

Explore ukdebug source code

Getting Started

Core Concepts

Guides

Platform Support

Architecture

​Overview

​Debug Build Configuration

​Using kraft CLI

​Using menuconfig

​GDB Remote Debugging (Recommended)

​Prerequisites

​Setting Up GDB Debugging

​Common GDB Commands

​Advanced GDB Usage

​Watch Variables

​Core Dumps

​Remote Debugging Script

​Debugging with Two Consoles

​QEMU Built-in GDB Stub

​Basic QEMU GDB Usage

​QEMU vs Unikraft GDB Stub

​Debugging the GDB Stub Itself

​Debug Logging

​Enable Debug Messages

​Log Levels

​Runtime Log Control

​Capture Logs

​Assertions and Crash Handling

​Enable Assertions

​Using Assertions

​Stack Trace on Crash

​Memory Debugging

​Debugging Allocator

​Detect Memory Leaks

​Address Sanitizer (Future)

​Performance Profiling

​Using perf with KVM

​CPU Time Profiling

​Function Call Tracing

​Common Debugging Scenarios

​Debugging Crashes

​Debugging Network Issues

​Debugging Boot Issues

​Debugging Infinite Loops

​Debugging Tools

​GDB Enhanced (GEF)

​QEMU Monitor

​Best Practices

​1. Always Use Debug Builds for Development

​2. Keep Debug and Release Configurations

​3. Use Separate Consoles

​4. Enable Stack Traces

​5. Version Control Debug Scripts

​Troubleshooting Debugging

​GDB Can’t Connect

​No Debug Symbols

​GDB Timeout

​Breakpoints Not Hit

​Next Steps

Building Applications

Configuration

Running Unikernels

ukdebug Library

Build docs developers (and LLMs) love

Overview

Debug Build Configuration

Using kraft CLI

Using menuconfig

GDB Remote Debugging (Recommended)

Prerequisites

Setting Up GDB Debugging

Common GDB Commands

Advanced GDB Usage

Watch Variables

Core Dumps

Remote Debugging Script

Debugging with Two Consoles

QEMU Built-in GDB Stub

Basic QEMU GDB Usage

QEMU vs Unikraft GDB Stub

Debugging the GDB Stub Itself

Debug Logging

Enable Debug Messages

Log Levels

Runtime Log Control

Capture Logs

Assertions and Crash Handling

Enable Assertions

Using Assertions

Stack Trace on Crash

Memory Debugging

Debugging Allocator

Detect Memory Leaks

Address Sanitizer (Future)

Performance Profiling

Using perf with KVM

CPU Time Profiling

Function Call Tracing

Common Debugging Scenarios

Debugging Crashes

Debugging Network Issues

Debugging Boot Issues

Debugging Infinite Loops

Debugging Tools

GDB Enhanced (GEF)

QEMU Monitor

Best Practices

1. Always Use Debug Builds for Development

2. Keep Debug and Release Configurations

3. Use Separate Consoles

4. Enable Stack Traces

5. Version Control Debug Scripts

Troubleshooting Debugging

GDB Can’t Connect

No Debug Symbols

GDB Timeout

Breakpoints Not Hit

Next Steps