Debugging

Effective debugging is essential for developing and troubleshooting EVerest charging stations. This guide covers logging, MQTT inspection, debugging tools, and common issues.

Logging System

EVerest uses a comprehensive logging system with multiple severity levels.

Log Levels

Use the appropriate EVLOG macro for different message types:

#include <everest/logging.hpp>

EVLOG_debug << "Detailed debugging: " << variable_value;
EVLOG_info << "Informational message: Module started";
EVLOG_warning << "Warning: Unusual condition detected";
EVLOG_error << "Error: Operation failed: " << error_message;

debug
info
warning
error

Verbose information for development and troubleshooting:

EVLOG_debug << "Setting simulated isolation resistance to " 
            << resistance_ohm << " Ohm via MQTT";

General operational messages:

EVLOG_info << "Starting firmware update";
EVLOG_info << "Module initialized successfully";

Potentially problematic situations:

EVLOG_warning << "Signing certificate is missing in FirmwareUpdateRequest";
EVLOG_warning << "Attempting to add a network without an interface";

Errors and failures:

EVLOG_error << "Failed to open serial port " << config.serial_port;
EVLOG_error << "Could not create KVS table: " << sqlite3_errmsg(db);

Configuring Log Output

Log configuration is set in logging.ini:

[loggers]
keys=root

[handlers]
keys=consoleHandler,fileHandler

[formatters]
keys=standardFormatter

[logger_root]
level=DEBUG
handlers=consoleHandler,fileHandler

[handler_consoleHandler]
class=StreamHandler
level=INFO
formatter=standardFormatter
args=(sys.stdout,)

[handler_fileHandler]
class=handlers.RotatingFileHandler
level=DEBUG
formatter=standardFormatter
args=('everest.log', 'a', 10485760, 5)

[formatter_standardFormatter]
format=%(asctime)s [%(name)s] [%(levelname)s] %(message)s

Runtime Log Level Control

Change log levels at runtime via environment variables:

# Set global log level
export EVEREST_LOG_LEVEL=debug

# Set module-specific log level
export EVEREST_MODULE_evse_manager_LOG_LEVEL=debug

# Run EVerest with verbose logging
./everest-core/build/dist/bin/manager --conf config.yaml

Session Logging

Many modules support detailed session logging:

# In module configuration
config_module:
  session_logging: true
  session_logging_path: /var/log/everest/sessions
  session_logging_xml: true  # Log full HLC XML messages

This creates detailed logs for each charging session including:

Protocol messages (ISO15118, SLAC, etc.)
State transitions
Measurement values
Error conditions

MQTT Inspection

EVerest uses MQTT for inter-module communication. Inspecting MQTT traffic is crucial for debugging.

Connecting to MQTT

By default, EVerest runs an internal MQTT broker on localhost:1883. Subscribe to all topics:

# Using mosquitto_sub
mosquitto_sub -h localhost -p 1883 -t 'everest/#' -v

# Using mqttui (interactive)
mqttui --broker localhost:1883

Important MQTT Topics

Module Status

Monitor module lifecycle:

mosquitto_sub -t 'everest/ready' -v

Shows when modules become ready.

Interface Commands

Watch commands between modules:

mosquitto_sub -t 'everest/+/+/cmd/#' -v

Interface Results

See command responses:

mosquitto_sub -t 'everest/+/+/res/#' -v

Variables

Monitor published variables:

mosquitto_sub -t 'everest/+/+/var/#' -v

Error Messages

Track raised errors:

mosquitto_sub -t 'everest_errors/#' -v

Sending Test Commands

You can send commands to modules via MQTT for testing:

# Enable charging on connector 1
mosquitto_pub -t 'everest/evse_manager/evse/cmd/enable' \
  -m '{"connector_id": 1}'

# Set simulated values in test modules
mosquitto_pub -t 'everest_external/isolation_monitor/set_resistance' \
  -m '{"resistance_ohm": 1000}'

MQTT Tools

mosquitto_sub

Command-line MQTT subscriber

mosquitto_sub -h localhost -t 'topic' -v

mosquitto_pub

Command-line MQTT publisher

mosquitto_pub -h localhost -t 'topic' -m 'message'

mqttui

Interactive TUI for MQTT

cargo install mqttui
mqttui --broker localhost:1883

MQTT Explorer

GUI MQTT clientDownload from http://mqtt-explorer.com/

Debug Tools

GDB Debugging

Debug modules with GDB:

# Build with debug symbols
cd everest-core/build
cmake -DCMAKE_BUILD_TYPE=Debug ..
make -j$(nproc)

# Run manager under GDB
gdb --args ./dist/bin/manager --conf ../config.yaml

# Common GDB commands
(gdb) break MyModule::init     # Set breakpoint
(gdb) run                      # Start execution
(gdb) continue                 # Continue after breakpoint
(gdb) print variable_name      # Print variable
(gdb) backtrace               # Show call stack

Attaching to Running Process

# Find process ID
ps aux | grep manager

# Attach GDB
gdb -p <pid>

# Set breakpoint and continue
(gdb) break module::ready
(gdb) continue

Valgrind for Memory Issues

Detect memory leaks and errors:

valgrind --leak-check=full --show-leak-kinds=all \
  ./dist/bin/manager --conf ../config.yaml

AddressSanitizer

Build with AddressSanitizer for runtime memory error detection:

cmake -DCMAKE_BUILD_TYPE=Debug \
      -DCMAKE_CXX_FLAGS="-fsanitize=address -fno-omit-frame-pointer" \
      ..
make -j$(nproc)

./dist/bin/manager --conf ../config.yaml

Packet Capture

The PacketSniffer module captures network traffic for protocol debugging.

Enabling Packet Capture

active_modules:
  packet_sniffer:
    module: PacketSniffer
    config_module:
      device: "eth0"  # Network interface
      session_logging_path: "/var/log/everest/pcap"

This creates .pcap files for each charging session that can be analyzed with Wireshark.

Analyzing with Wireshark

# Open capture file
wireshark session_12345.pcap

# Filter for ISO15118 (typically TCP port 15118)
tcp.port == 15118

# Filter for SLAC (EtherType 0x88E1)
eth.type == 0x88e1

Common Issues

Module fails to start

Symptoms: Module not appearing in MQTT topics, manager crashesDebug steps:

Check manager logs for errors
Verify manifest.yaml is valid
Ensure required interfaces are provided by other modules
Check config.yaml for correct module/interface mapping

# Validate manifest
ev-cli module create MyModule --diff

# Check manager output
./manager --conf config.yaml --log-level debug

Interface connection errors

Symptoms: “Required interface not satisfied” errorsDebug steps:

Verify module provides the required interface
Check config.yaml module connections
Ensure module names match in config

# config.yaml
active_modules:
  consumer_module:
    module: MyConsumer
    connections:
      storage:  # requirement ID
        module_id: store_module
        implementation_id: main
  store_module:
    module: Store  # Must provide 'kvs' interface

No MQTT messages appearing

Symptoms: mosquitto_sub shows no messagesDebug steps:

Verify MQTT broker is running: ps aux | grep mosquitto
Check broker port: netstat -tlnp | grep 1883
Verify EVerest is connected: check manager logs
Check topic filter is correct (use # for wildcard)

# Test MQTT broker directly
mosquitto_pub -t 'test' -m 'hello'
mosquitto_sub -t 'test' -v

Charging session not starting

Symptoms: Car connected but no chargingDebug steps:

Check EVSE state: mosquitto_sub -t 'everest/evse_manager/+/var/session_event'
Verify authorization: check Auth module logs
Review power path: check energy manager limits
Inspect protocol: enable session logging for ISO15118/SLAC details

# Enable detailed session logging
config_module:
  session_logging: true
  session_logging_xml: true

Segmentation faults

Symptoms: Crash with “Segmentation fault (core dumped)”Debug steps:

Enable core dumps: ulimit -c unlimited
Run with GDB to get backtrace
Check for null pointer dereferences
Use AddressSanitizer or Valgrind

# Get backtrace from core dump
gdb ./dist/bin/manager core
(gdb) backtrace

Performance Debugging

Profiling with perf

# Record performance data
perf record -g ./dist/bin/manager --conf ../config.yaml

# Analyze
perf report

CPU Usage

# Monitor per-thread CPU usage
top -H -p $(pgrep manager)

# Get detailed thread info
ps -eLf | grep manager

Best Practices

Log Strategically

Use appropriate log levels. Debug logs should provide context, not spam.

Monitor MQTT Early

Start mosquitto_sub before debugging to capture initial messages.

Enable Session Logging

Always enable session logging when debugging charging issues.

Use Debug Builds

Debug builds provide better backtraces and variable inspection.

Isolate Issues

Use minimal configurations to isolate problems.

Next Steps

Testing

Write tests to catch issues early

Development Tools

Master ev-cli and other development tools

Logging Reference

Complete logging API reference

Error Handling

Learn the error framework

Get Started

Core Concepts

Modules

Development

Deployment

Community

Logging System

Log Levels

Configuring Log Output

Runtime Log Level Control

Session Logging

MQTT Inspection

Connecting to MQTT

Important MQTT Topics

Sending Test Commands

MQTT Tools

mosquitto_sub

mosquitto_pub

mqttui

MQTT Explorer

Debug Tools

GDB Debugging

Attaching to Running Process

Valgrind for Memory Issues

AddressSanitizer

Packet Capture

Enabling Packet Capture

Analyzing with Wireshark

Common Issues

Performance Debugging

Profiling with perf

CPU Usage

Best Practices

Next Steps

Testing

Development Tools

Logging Reference

Error Handling

Build docs developers (and LLMs) love

Get Started

Core Concepts

Modules

Development

Deployment

Community

​Logging System

​Log Levels

​Configuring Log Output

​Runtime Log Level Control

​Session Logging

​MQTT Inspection

​Connecting to MQTT

​Important MQTT Topics

​Sending Test Commands

​MQTT Tools

mosquitto_sub

mosquitto_pub

mqttui

MQTT Explorer

​Debug Tools

​GDB Debugging

​Attaching to Running Process

​Valgrind for Memory Issues

​AddressSanitizer

​Packet Capture

​Enabling Packet Capture

​Analyzing with Wireshark

​Common Issues

​Performance Debugging

​Profiling with perf

​CPU Usage

​Best Practices

​Next Steps

Testing

Development Tools

Logging Reference

Error Handling

Build docs developers (and LLMs) love

Logging System

Log Levels

Configuring Log Output

Runtime Log Level Control

Session Logging

MQTT Inspection

Connecting to MQTT

Important MQTT Topics

Sending Test Commands

MQTT Tools

Debug Tools

GDB Debugging

Attaching to Running Process

Valgrind for Memory Issues

AddressSanitizer

Packet Capture

Enabling Packet Capture

Analyzing with Wireshark

Common Issues

Performance Debugging

Profiling with perf

CPU Usage

Best Practices

Next Steps