Debugging Aeolos

Debugging Overview

Aeolos provides multiple debugging mechanisms to help you diagnose issues and understand kernel behavior:

Serial output - Real-time kernel logs via COM1
Kernel logging system - Structured logging with color-coded levels
Symbol table - Function names in stack traces
QEMU debugging - GDB integration for low-level debugging

Serial Output

How It Works

Aeolos initializes COM1 (serial port 0x3F8) at 9600 baud and sends all kernel log output to it. This is implemented in kernel/dev/serial/serial.c:4-15:

void serial_init()
{
    uint16_t divisor = 115200 / 9600;
    uint8_t divlow = divisor & 0xFF;
    uint8_t divhigh = divisor >> 8;
    port_outb(0x3fb, 0x80);
    port_outb(0x3f8, divlow);
    port_outb(0x3f9, divhigh);
    port_outb(0x3fb, 0x03);
    port_outb(0x3fa, 0xC7);
    port_outb(0x3fc, 0x03);
}

Every call to klog() automatically sends output to the serial port via serial_send() in kernel/lib/klog.c:23.

Capturing Serial Output in QEMU

To File
To Terminal
To TCP Socket

Redirect serial output to a file:

qemu-system-x86_64 -cdrom os.iso \
  -m 128 -smp 4 \
  -serial file:serial.log

Monitor in real-time:

tail -f serial.log

Output directly to your terminal:

qemu-system-x86_64 -cdrom os.iso \
  -m 128 -smp 4 \
  -serial stdio

Using -serial stdio can interfere with QEMU’s monitor console.

Stream to a TCP socket:

qemu-system-x86_64 -cdrom os.iso \
  -m 128 -smp 4 \
  -serial tcp::4444,server,nowait

Connect with netcat:

nc localhost 4444

Kernel Logging System

Log Levels

Aeolos uses color-coded log levels defined in kernel/lib/klog.c:180-195:

void klog(loglevel_t lvl, const char* s, ...)
{
    lock_wait(&log_lock);
    switch (lvl) {
    case LOG_SUCCESS:
        puts("\033[32;1m[OKAY]  \033[0m");
        break;
    case LOG_WARN:
        puts("\033[33m[WARN]  \033[0m");
        break;
    case LOG_ERROR:
        puts("\033[31;1m[ERROR] \033[0m");
        break;
    default:
        puts("\033[34;1m[INFO]  \033[0m");
    }
    // ...
}

LOG_INFO

General information messages (blue)

LOG_SUCCESS

Successful operations (green)

LOG_WARN

Warning messages (yellow)

LOG_ERROR

Error conditions (red)

Using the Logger

The klog() function supports printf-style formatting:

klog(LOG_INFO, "Initializing subsystem\n");
klog(LOG_SUCCESS, "CPU %d initialized\n", cpu_id);
klog(LOG_WARN, "Memory low: %x bytes free\n", free_mem);
klog(LOG_ERROR, "Failed to mount filesystem\n");

Supported Format Specifiers

From kernel/lib/klog.c:106-140:

%d - Signed decimal integer
%x - Hexadecimal with 0x prefix
%s - String
%b - Boolean (“true” or “false”)
%% - Literal percent sign

Ring Buffer

All log output is stored in a ring buffer (kernel/lib/klog.c:11-13):

static uint8_t log_buff[KLOG_BUFF_LEN];
static uint16_t log_start = 0;
static uint16_t log_end = 0;

This preserves recent logs even if the screen scrolls, and powers the on-screen log display.

Kernel Panics and Stack Traces

Panic Behavior

When a critical error occurs, kernel_panic() is called from kernel/sys/panic.c:58-82:

_Noreturn void kernel_panic(const char* s, ...)
{
    asm volatile("cli");
    lock_wait(&panic_lock);

    // stop other cores
    apic_timer_set_handler(halt);

    // wait for some time for cores to stop
    pit_wait(10);

    // now print error information
    klog_puts("\033[31m[PANIC] \033[37;1m");
    va_list args;
    va_start(args, s);
    klog_vprintf(s, args);
    va_end(args);
    klog_puts("\033[0m");
    do_stacktrace();
    klog_show_now();

    // halt this core
    while (true)
        asm volatile("hlt");
}

The panic handler:

Disables interrupts
Halts all other CPU cores
Prints the panic message in red
Generates a stack trace
Displays the log buffer
Halts the system

The -no-reboot -no-shutdown QEMU flags keep the VM running after a panic so you can examine the output.

Stack Traces

Stack traces use the kernel symbol table to resolve addresses to function names (kernel/sys/panic.c:32-56):

static void do_stacktrace()
{
    uint64_t* rbp_val = 0;
    asm volatile("movq %%rbp, %0" : "=rm"(rbp_val));

    klog_printf("\nStack Trace:\n");
    if ((uint64_t)rbp_val <= MEM_VIRT_OFFSET) {
        klog_printf("\n \t<optimised out>");
        return;
    }
    for (int i = 0;; i++) {
        klog_printf(" \t%d: ", i);
        uint64_t func_addr = *(rbp_val + 1);
        const char* func_name = symtab_get_func(func_addr);
        klog_printf("\t%x", func_addr);
        if (!func_name) {
            klog_printf(" (Unknown Function)");
            break;
        }
        klog_printf(" (%s)\n", func_name);
        rbp_val = (uint64_t*)*rbp_val;
    }
}

The -fno-omit-frame-pointer compiler flag ensures stack traces work correctly by preserving the frame pointer in %rbp.

Symbol Table Generation

The kernel build process generates symbols twice (from kernel/Makefile:46-60):

Link kernel without symbols
Extract symbols using gensym script
Compile symbols as C code
Link symbols into final kernel

This provides function name resolution in stack traces without external tools.

GDB Debugging with QEMU

Starting a Debug Session

Launch QEMU with GDB server:

qemu-system-x86_64 -cdrom os.iso \
  -m 128 -smp 4 \
  -s -S

-s: Start GDB server on TCP port 1234
-S: Pause execution at start

In another terminal, connect GDB:

gdb kernel/kernel.elf
(gdb) target remote :1234
(gdb) continue

Common GDB Commands

# Break at function
break kmain

# Break at address
break *0xffffffff80100000

# Conditional breakpoint
break vmm_map if addr == 0

# List breakpoints
info breakpoints

Debugging Tips

Debug early boot code

The kernel is loaded at virtual address 0xffffffff80000000 (higher half). Set breakpoints relative to this:

break *0xffffffff80100000

Inspect page tables

# Print CR3 (page table root)
print/x $cr3

# Examine PML4 table
x/512gx $cr3

Debug SMP initialization

QEMU’s -smp 4 flag enables 4 CPUs. Use GDB’s thread commands:

info threads
thread 2
where

View kernel structures

# Print structure
print *(struct task*)0xffffffff80300000

# Pretty-print
set print pretty on

QEMU Monitor

Access QEMU’s monitor console with Ctrl+Alt+2 (return with Ctrl+Alt+1). Useful monitor commands:

# Show CPU state
info registers

# Show memory mappings
info mem

# Show page table
info tlb

# Show interrupt vectors
info pic
info irq

# Save VM state
savevm snapshot1

# Load VM state
loadvm snapshot1

Debugging Techniques

Adding Debug Output

Add temporary debug logging:

klog(LOG_INFO, "[DEBUG] Function called with param=%x\n", param);
klog(LOG_INFO, "[DEBUG] State: ptr=%x, count=%d\n", ptr, count);

Asserting Conditions

Use kernel panic for assertions:

if (ptr == NULL)
    kernel_panic("Null pointer in function %s\n", __func__);

Tracing Execution Flow

Add entry/exit logging:

void my_function(int arg)
{
    klog(LOG_INFO, ">>> Entering %s with arg=%d\n", __func__, arg);
    // function body
    klog(LOG_INFO, "<<< Exiting %s\n", __func__);
}

Memory Debugging

Log memory operations:

void* ptr = kmalloc(size);
klog(LOG_INFO, "kmalloc(%d) = %x\n", size, ptr);

Common Issues

Triple Fault

QEMU exits immediately with “Triple fault” - usually indicates:

Invalid page tables
Bad IDT/GDT setup
Stack overflow

Debug with:

qemu-system-x86_64 -cdrom os.iso -d int,cpu_reset -no-reboot

Page Fault Loop

Repeating page faults indicate recursive page fault handling. Check:

IDT exception handlers are mapped correctly
Kernel stack is valid
Exception handler doesn’t trigger another fault

No Serial Output

If you don’t see serial output:

Ensure serial port is initialized before first log
Check QEMU serial flag: -serial stdio
Verify serial_init() is called in kernel initialization

Deadlock

System hangs with no output:

Add debug logs before suspected deadlock
Use QEMU monitor: info registers shows if CPUs are halted
Check lock ordering in kernel code

Next Steps

Setup

Set up your development environment

Contributing

Learn how to contribute to Aeolos

Getting Started

Architecture

Kernel Subsystems

Drivers

Development

Debugging Aeolos

Debugging Overview

Serial Output

How It Works

Capturing Serial Output in QEMU

Kernel Logging System

Log Levels

LOG_INFO

LOG_SUCCESS

LOG_WARN

LOG_ERROR

Using the Logger

Supported Format Specifiers

Ring Buffer

Kernel Panics and Stack Traces

Panic Behavior

Stack Traces

Symbol Table Generation

GDB Debugging with QEMU

Starting a Debug Session

Common GDB Commands

Debugging Tips

QEMU Monitor

Debugging Techniques

Adding Debug Output

Asserting Conditions

Tracing Execution Flow

Memory Debugging

Common Issues

Next Steps

Setup

Contributing

Build docs developers (and LLMs) love

Getting Started

Architecture

Kernel Subsystems

Drivers

Development

​Debugging Overview

​Serial Output

​How It Works

​Capturing Serial Output in QEMU

​Kernel Logging System

​Log Levels

LOG_INFO

LOG_SUCCESS

LOG_WARN

LOG_ERROR

​Using the Logger

​Supported Format Specifiers

​Ring Buffer

​Kernel Panics and Stack Traces

​Panic Behavior

​Stack Traces

​Symbol Table Generation

​GDB Debugging with QEMU

​Starting a Debug Session

​Common GDB Commands

​Debugging Tips

​QEMU Monitor

​Debugging Techniques

​Adding Debug Output

​Asserting Conditions

​Tracing Execution Flow

​Memory Debugging

​Common Issues

​Next Steps

Setup

Contributing

Build docs developers (and LLMs) love

Debugging Overview

Serial Output

How It Works

Capturing Serial Output in QEMU

Kernel Logging System

Log Levels

Using the Logger

Supported Format Specifiers

Ring Buffer

Kernel Panics and Stack Traces

Panic Behavior

Stack Traces

Symbol Table Generation

GDB Debugging with QEMU

Starting a Debug Session

Common GDB Commands

Debugging Tips

QEMU Monitor

Debugging Techniques

Adding Debug Output

Asserting Conditions

Tracing Execution Flow

Memory Debugging

Common Issues

Next Steps