Skip to main content
The Linux kernel ships a comprehensive suite of debugging and analysis tools. Most are enabled via Kconfig options and impose overhead only when active — production builds leave them out.
Always test patches with CONFIG_PREEMPT, CONFIG_DEBUG_PREEMPT, CONFIG_SLUB_DEBUG, CONFIG_DEBUG_PAGEALLOC, CONFIG_DEBUG_MUTEXES, CONFIG_DEBUG_SPINLOCK, CONFIG_DEBUG_ATOMIC_SLEEP, CONFIG_PROVE_RCU, and CONFIG_DEBUG_OBJECTS_RCU_HEAD all enabled simultaneously.

​
Memory error detection

​
KASAN — Kernel Address Sanitizer

KASAN is a dynamic memory safety error detector that finds out-of-bounds and use-after-free bugs.

Generic KASAN

CONFIG_KASAN_GENERIC. Supported on x86_64, arm, arm64, powerpc, riscv, s390, xtensa, loongarch. Significant performance and memory overhead. Best for debugging.

Software Tag-Based

CONFIG_KASAN_SW_TAGS. arm64 only. Moderate memory overhead — suitable for testing on real workloads on memory-restricted devices.

Hardware Tag-Based

CONFIG_KASAN_HW_TAGS. Requires arm64 with MTE (Memory Tagging Extension). Low overhead — intended for in-field use or security mitigation.
Compiler requirements:
  • Generic KASAN: GCC ≥ 8.3.0 or any supported Clang
  • Software Tag-Based: GCC ≥ 11 or any supported Clang
  • Hardware Tag-Based: GCC ≥ 10 or Clang ≥ 12
Enable and build: 
# Enable via menuconfig: Kernel hacking -> Memory Debugging -> KASAN
make menuconfig

# Or set directly
scripts/config --enable CONFIG_KASAN
scripts/config --enable CONFIG_KASAN_GENERIC

make LLVM=1
A typical KASAN report for a heap out-of-bounds access: 
==================================================================
BUG: KASAN: slab-out-of-bounds in kmalloc_oob_right+0x6b/0x70
Write of size 1 at addr ffff8881b7bda066 by task insmod/2176

CPU: 0 PID: 2176 Comm: insmod Not tainted 6.8.0 #1
Call Trace:
 kasan_report+0x14c/0x190
 __asan_store1+0x4b/0x50
 kmalloc_oob_right+0x6b/0x70
 kmalloc_tests_init+0x18/0x110
 do_one_initcall+0x87/0x240

Allocated by task 2176:
 kmalloc_oob_right+0x33/0x70
 kmalloc_tests_init+0x18/0x110
==================================================================

​
KMEMLEAK — Kernel Memory Leak Detector

KMEMLEAK detects possible kernel memory leaks using a tracing garbage-collection approach. Orphaned objects are reported via debugfs rather than freed. 
# Enable
scripts/config --enable CONFIG_DEBUG_KMEMLEAK
make && boot the kernel

# Mount debugfs if not already mounted
mount -t debugfs nodev /sys/kernel/debug/

# View current leak suspects
cat /sys/kernel/debug/kmemleak

# Trigger an immediate scan (default scan runs every 600 seconds)
echo scan > /sys/kernel/debug/kmemleak

# Clear the current list
echo clear > /sys/kernel/debug/kmemleak

# Disable kmemleak (irreversible for this boot)
echo off > /sys/kernel/debug/kmemleak

# Dump info about a specific object
echo dump=0xffff8881b7bda000 > /sys/kernel/debug/kmemleak
Boot parameter: kmemleak=off to disable at boot, kmemleak=on when CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF is set. KMEMLEAK scans memory every 10 minutes by default. Note that objects at the beginning of the list may cause subsequent objects to be reported, so address reports from the top down.

​
KFENCE — Kernel Electric Fence

CONFIG_KFENCE provides a low-overhead sampling-based memory safety detector suitable for production use. It randomly samples allocations and places them in a guarded region to detect out-of-bounds and use-after-free bugs with near-zero overhead on the normal code path.

​
Concurrency and locking

​
lockdep — Lock Dependency Validator

lockdep is the kernel’s lock dependency validator. It tracks lock acquisition order and detects potential deadlocks before they occur in production. 
# Enable lockdep
scripts/config --enable CONFIG_PROVE_LOCKING
scripts/config --enable CONFIG_DEBUG_LOCKDEP
scripts/config --enable CONFIG_LOCK_STAT
A lockdep report for a deadlock: 
======================================================
WARNING: possible circular locking dependency detected
------------------------------------------------------
kworker/0:1/18 is trying to acquire lock:
 (&dev->mutex){+.+.}, at: device_lock+0x18/0x20

but task is already holding lock:
 (&bus->subsys.rwsem){++++}, at: bus_for_each_dev+0x4b/0xa0

the existing dependency chain (in reverse order) is:
-> #1 (&bus->subsys.rwsem):
       lock_acquire+0xb7/0x1f0
       ...
-> #0 (&dev->mutex):
       lock_acquire+0xb7/0x1f0
       ...
lockdep also validates spinlock/mutex nesting (spinlocks cannot be acquired while holding a mutex in interrupt context), RCU lock ordering, and seqlock correctness.

​
KCSAN — Kernel Concurrency Sanitizer

KCSAN is a dynamic data race detector that uses compile-time instrumentation and a watchpoint-based sampling approach. 
scripts/config --enable CONFIG_KCSAN
# GCC >= 11 or Clang >= 11 required
make LLVM=1
A typical KCSAN data race report: 
==================================================================
BUG: KCSAN: data-race in test_kernel_read / test_kernel_write

write to 0xffffffffc009a628 of 8 bytes by task 487 on cpu 0:
 test_kernel_write+0x1d/0x30
 access_thread+0x89/0xd0
 kthread+0x23e/0x260

read to 0xffffffffc009a628 of 8 bytes by task 488 on cpu 6:
 test_kernel_read+0x10/0x20
 access_thread+0x89/0xd0
 kthread+0x23e/0x260

value changed: 0x00000000000009a6 -> 0x00000000000009b2
==================================================================
KCSAN reports show the two threads racing, their stack traces, and the observed value change. Configure behavior via lib/Kconfig.kcsan. Annotate intentional data races with data_race() or READ_ONCE()/WRITE_ONCE() to suppress false positives: 
/* Intentional race — reader does not need to see the latest value */
if (data_race(foo->state) == STATE_IDLE)
	return;

/* Use READ_ONCE/WRITE_ONCE for properly-annotated races */
old = READ_ONCE(shared_var);
WRITE_ONCE(shared_var, new_val);

​
Tracing and profiling

​
ftrace — Function Tracer

ftrace is the kernel’s built-in tracing framework, accessible through the tracefs filesystem. 
# Mount tracefs
mount -t tracefs nodev /sys/kernel/tracing
cd /sys/kernel/tracing

# List available tracers
cat available_tracers
# output: blk function_graph function nop ...

# Enable function tracer
echo function > current_tracer
echo 1 > tracing_on
sleep 1
echo 0 > tracing_on
cat trace | head -50

# Trace a specific function
echo schedule > set_ftrace_filter
echo function > current_tracer
echo 1 > tracing_on

# Function graph tracer (shows call/return with timing)
echo function_graph > current_tracer
echo do_sys_open > set_graph_function

# Trace events
echo 1 > events/sched/sched_switch/enable
echo 1 > events/irq/irq_handler_entry/enable
Enable via CONFIG_FTRACE, CONFIG_FUNCTION_TRACER, CONFIG_FUNCTION_GRAPH_TRACER.

​
perf

perf is the kernel’s performance analysis tool, built on the kernel’s perf_events subsystem. 
# CPU cycles profiling
perf stat ./my_benchmark

# Record a profile
perf record -g ./my_program
perf report

# System-wide profiling
perf top

# Trace specific kernel events
perf trace -e 'syscalls:sys_enter_*' ./my_program

# Count hardware performance counters
perf stat -e cache-misses,cache-references,instructions,cycles ./my_program

# Kernel function latency
perf probe --add 'schedule'
perf record -e probe:schedule -ag
perf report
Enable via CONFIG_PERF_EVENTS, CONFIG_HW_PERF_EVENTS.

​
Interactive debugging

​
KGDB — Kernel GNU Debugger

KGDB allows debugging a running kernel with GDB over a serial connection or network. 
# Kernel configuration
scripts/config --enable CONFIG_KGDB
scripts/config --enable CONFIG_KGDB_SERIAL_CONSOLE
scripts/config --enable CONFIG_DEBUG_INFO
scripts/config --enable CONFIG_FRAME_POINTER
Boot the target kernel with: 
kgdboc=ttyS0,115200 kgdbwait
On the host, connect GDB: 
gdb vmlinux
(gdb) set remotebaud 115200
(gdb) target remote /dev/ttyS0
(gdb) bt        # backtrace
(gdb) info registers
(gdb) p *current  # print current task_struct
For QEMU-based development: 
# Start QEMU with GDB stub
qemu-system-x86_64 \
  -kernel arch/x86/boot/bzImage \
  -s -S \
  -append "kgdboc=kbd kgdbwait console=ttyS0"

# On host
gdb vmlinux
(gdb) target remote :1234
(gdb) continue

​
Dynamic debug

Dynamic debug (CONFIG_DYNAMIC_DEBUG) allows enabling and disabling individual pr_debug() and dev_dbg() messages at runtime without recompilation. 
# Enable all debug messages in a file
echo 'file drivers/usb/core/urb.c +p' > /sys/kernel/debug/dynamic_debug/control

# Enable debug messages in a module
echo 'module usb_storage +p' > /sys/kernel/debug/dynamic_debug/control

# Enable a specific function's debug messages
echo 'func usb_submit_urb +p' > /sys/kernel/debug/dynamic_debug/control

# Enable all debug messages containing a string
echo 'format "urb" +p' > /sys/kernel/debug/dynamic_debug/control

# Disable all
echo '* -p' > /sys/kernel/debug/dynamic_debug/control

# View current state
cat /sys/kernel/debug/dynamic_debug/control
Flags you can set/clear:
  • p — print the message
  • f — include function name in output
  • l — include line number
  • m — include module name
  • t — include thread ID
In your driver code: 
#include <linux/printk.h>

/* pr_debug is compiled out without CONFIG_DYNAMIC_DEBUG */
pr_debug("Processing request id=%u len=%zu\n", id, len);

/* dev_dbg includes device context and is also dynamically controllable */
dev_dbg(dev, "Received packet: type=%x len=%u\n", type, len);

/* print_hex_dump_debug for binary data */
print_hex_dump_debug("buf: ", DUMP_PREFIX_OFFSET, 16, 1, buf, len, true);

​
BUG() and WARN() macros

Do not add new code that uses BUG() or BUG_ON(). These halt the kernel immediately and unconditionally. Use WARN*() instead, which prints a backtrace and continues.
/*
 * BUG() — halts kernel, dumps registers. Use only for truly
 * unrecoverable situations. New code should not use this.
 */
BUG();
BUG_ON(ptr == NULL);

/*
 * WARN_ON_ONCE() — preferred. Prints backtrace once per condition,
 * regardless of how many times it triggers. Avoids log flooding.
 */
WARN_ON_ONCE(refcount < 0);

/*
 * WARN_ON() — prints backtrace every time the condition is true.
 * Use only when each occurrence is meaningful.
 */
WARN_ON(list_empty(&expected_nonempty_list));

/*
 * WARN() — like WARN_ON() but with a format string.
 */
WARN(size > MAX_SIZE, "size %zu exceeds limit %zu\n", size, MAX_SIZE);

/*
 * BUILD_BUG_ON() — compile-time assertion, zero runtime cost.
 * Fails the build if the condition is true.
 */
BUILD_BUG_ON(sizeof(struct my_header) != 64);
BUILD_BUG_ON(ARRAY_SIZE(handler_table) != NUM_EVENTS);
WARN*() is for unexpected conditions only. Do not use it for:
  • Conditions expected during normal operation
  • Conditions triggered by user-space input
  • Pre/post-condition assertions

​
Panic settings

The kernel’s panic behavior is controlled via /proc/sys/kernel/: 
# Automatically reboot after N seconds following a panic (0 = no reboot)
echo 5 > /proc/sys/kernel/panic

# Panic on oops (useful for catching bugs that would otherwise continue)
echo 1 > /proc/sys/kernel/panic_on_oops

# Panic on WARN (panic_on_warn=1 at boot)
# Enables: any WARN*() fires causes a panic
# Useful for catching unexpected conditions on test systems

# Print oops info before reboot
echo 1 > /proc/sys/kernel/oops_all_cpu_backtrace
As boot parameters: 
panic=5 panic_on_oops=1 oops=panic
For development VMs, set panic=1 and panic_on_oops=1. For debugging a specific issue, panic_on_warn=1 ensures that any WARN*() produces a full crash dump you can analyze with crash or kdb.

​
Additional tools

sparse

Static analysis tool for C. Checks for context (interrupt vs. process), lock balance, __user/__kernel pointer misuse, and endianness. Run with make C=1 or make C=2.

UBSAN

CONFIG_UBSAN — Undefined Behavior Sanitizer. Detects signed integer overflow, shift-out-of-bounds, misaligned accesses, and other undefined behavior at runtime.

KCOV

CONFIG_KCOV — kernel code coverage for use with fuzzing tools like syzkaller. Exposes coverage information to user-space for guided fuzzing.

checkpatch.pl

scripts/checkpatch.pl — style checker for patches and source files. Required before submission. Run with --strict for additional checks.

KMSAN

CONFIG_KMSAN — Kernel Memory Sanitizer. Detects uses of uninitialized memory. Clang-only. Finds a class of bugs that KASAN misses.

kunit

CONFIG_KUNIT — unit testing framework for kernel code. Tests run in-kernel via kunit_tool or as loadable modules. Supports test fixtures and assertions.

Build docs developers (and LLMs) love

Get started for freeTalk to us