fdctl configure command sets up your host system so Firedancer can run correctly. This step must be performed after each system reboot to reconfigure kernel parameters, huge pages, and networking devices.
Overview
The configure command performs several critical system setup tasks:- hugetlbfs - Reserves huge (2MiB) and gigantic (1GiB) pages for Firedancer
- sysctl - Sets required kernel parameters
- hyperthreads - Checks hyperthreaded pairs for critical CPU cores
- bonding - Prepares bonded network devices for XDP networking
- ethtool-channels - Configures network device channel count
- ethtool-offloads - Disables incompatible offload features
- ethtool-loopback - Disables tx-udp-segmentation on loopback
Configure Modes
You run configure withfdctl configure <mode> <stage>... where mode is one of:
check
Verifies if each stage is already configured. Never requires privileges and makes no changes to the system. Exits with an error if stages are not properly configured.
init
Configures the provided stages if they are not already configured. Only requires privileges if it needs to actually change something.
all to configure everything:
Configuration Stages
hugetlbfs
Reserves huge (2MiB) and gigantic (1GiB) memory pages from the Linux kernel for Firedancer. Almost all memory in Firedancer is allocated from these pages for performance reasons. This is a two-step process:- Increases the number of huge and gigantic pages in the kernel by modifying
/sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages - Assigns pages to Firedancer by creating hugetlbfs mounts at
/mnt/.fd/.gigantic/and/mnt/.fd/.huge
fini mode unmounts the filesystems and returns pages to the kernel global pool, but will not decrease the global pool size.
sysctl
Configures kernel parameters for optimal Firedancer performance. The stage only increases values to meet minimums and will not decrease them if the minimum is already met.| Sysctl | Minimum | Required | Description |
|---|---|---|---|
vm.max_map_count | 1000000 | Yes | Agave accounts database requires mapping many files |
fs.file-max | 1024000 | Yes | Agave accounts database requires opening many files |
fs.nr_open | 1024000 | Yes | Agave accounts database requires opening many files |
net.ipv4.conf.lo.rp_filter | 2 | Yes | Required for QUIC over loopback |
net.ipv4.conf.lo.accept_local | 1 | Yes | Required for QUIC over loopback |
net.core.bpf_jit_enable | 1 | No | Makes BPF networking faster |
kernel.numa_balancing | 0 | No | Firedancer manages NUMA, rebalancing hurts performance |
Non-required sysctls produce warnings if not set correctly, but configuration will proceed normally.
CAP_SYS_ADMIN
The fini mode does nothing - kernel parameters are never reduced or changed back.
hyperthreads
Firedancer has two critical tiles that must run serially on dedicated cores:- pack - Schedules transactions for execution when leader
- poh - Performs repeated SHA256 hashes and stamps blocks when leader
When using
auto layout, Firedancer ensures no other tiles run on the hyperthread pairs. With manual layouts, you must verify this yourself.bonding
Validator hosts commonly use bonded network devices for increased bandwidth and fault tolerance. Common Linux distributions use bonding driver defaults incompatible with XDP mode on some network drivers. Firedancer reconfigures bonding parameters to tolerate brief downtime when configuring XDP by increasing timeout parameters in/sys/class/net/bond0/bonding/ to five seconds:
miimodedowndelaypeer_notif_delay
ethtool-channels
Configures receive side scaling (RSS) to improve network performance. Eachnet tile serves one network queue, so this stage modifies the network device to steer all Firedancer packets to the proper queues.
Three modes are available (set in your configuration):
simple mode (default)
simple mode (default)
Modifies the combined channel count to match the number of
net tiles. Works with all network devices but reduces queue count system-wide, potentially impacting non-Firedancer traffic.Similar to running:dedicated mode
dedicated mode
Reserves a dedicated hardware queue for each net tile. More advanced and may not work with all devices. Modifies the RXFH indirection table and installs ntuple rules to direct Firedancer traffic to dedicated queues while sharding other traffic across the rest. Better performance for both Firedancer and non-Firedancer traffic.
auto mode
auto mode
Attempts dedicated mode first, automatically falls back to simple mode if any failure occurs.
ethtool-offloads
XDP is incompatible with certain network device features that must be disabled:rx-udp-gro-forwarding- GRE segmentation offload
ethtool-loopback
XDP is incompatible with localhost UDP traffic usingtx-udp-segmentation. This feature must be disabled when connecting Agave clients to Firedancer over loopback or when using Frankendancer.
Runs the command:
Complete Initialization
To initialize all stages at once:It is strongly recommended to run the configure command automatically when your system boots. All stages need to be reconfigured after each reboot.