Net Tile Internals

​

Overview

• Bypasses large parts of the kernel’s routing code
• Avoids expensive context switches
• Offloads most data copies to network hardware (“zero copy” I/O)

• Passes incoming packets down to app tiles
• Routes outgoing packets to the right network interface
• Wakes up the kernel ~20k times per second to do batches of RX and TX

​

XDP Modes

​

Driver Mode (drv)

• Achieves best performance
• Not available on all drivers
• Less stable due to frequent kernel changes
• Performance target: ~20 million packets per second

​

Socket Buffer Mode (skb)

• Uses Linux network stack’s struct sk_buff
• Works everywhere
• Performance penalty compared to driver mode

​

XDP Lifecycle

​

Persistent Configuration

• Disables problematic ethtool offloads (GRO, UDP segmentation)
• Configures NIC to steer all Firedancer traffic to a single queue per net tile
• One XDP socket required per NIC channel

$ sudo fdctl configure init all --config config.toml

​

Ephemeral Configuration

• Installs an XDP program
• Deploys various supporting eBPF maps
• Creates AF_XDP sockets and XDP metadata rings
• Binds UMEM regions to AF_XDP sockets
• Creates an rtnetlink socket to monitor for config changes

​

Topology

​

TX Links

• Metadata ring is an mcache
• “mcache publish marks data freed” pattern (assumes TSO memory model)
• Data region is a dcache
• Chunk addressing with base

​

RX Links

• 2 net tiles
• 3 quic tiles
• 1 shred tile

• net:0 → quic
• net:0 → shred
• net:1 → quic
• net:1 → shred

​

UMEM Region

• 4K aligned memory region
• Subdivided into 2048 byte frames
• Each frame carries one Ethernet packet

• Firedancer app tiles (read-only)
• Firedancer net tiles (read-write)
• The Linux kernel (read-write)
• PCIe network devices (read-write via IOMMU)

​

RX Lifecycle

    ┌──────┐     ┌──────┐
    │ FILL ┼─────►  RX  │
    └─▲────┘     └────┬─┘
      │               │
      │  ┌────────┐   │
      │  │ MCACHE │   │
      ├──┼  QUIC  ◄───┤
      │  └────────┘   │
      │               │
      │  ┌────────┐   │
      │  │ MCACHE │   │
      └──┼ SHRED  ◄───┘
         └────────┘

​

1. FILL Ring

• One FILL ring per XDP socket
• Consists of 64-bit aligned pointers relative to UMEM start
• Allocated by kernel, read-write mapped to userspace
• If empty, incoming packets get dropped

​

2. RX Ring

• One RX ring per XDP socket
• Contains struct xdp_desc entries
• Net tile consumes descriptors and either:
  • Frees buffer immediately (returns to FILL ring)
  • Forwards to mcache ring

​

3. MCACHE Ring

• One mcache ring per downstream tile interested in traffic
• Number of UMEM buffers in ‘MCACHE’ state equals depths of all mcache RX rings
• RX mcache ring does not backpressure (too slow app tiles get overridden)
• Net tile produces Tango descriptors (fd_frag_meta_t) to mcache

​

Frag Metadata Schema

static inline void const *
rx_pkt_laddr( void * base,
              ulong  chunk,
              ulong  ctl ) {
  return (void *)( (ulong)base + (chunk<<6) + ctl );
}

​

TX Lifecycle

┌──────┐      ┌──────┐
│ FREE ┼──────►  TX  │
└─▲────┘      └────┬─┘
  │                │
  │ ┌────────────┐ │
  └─┼ COMPLETION ◄─┘
    └────────────┘

​

1. MCACHE Rings (TX)

• All net tiles listen for new outgoing packets on all TX mcaches
• Take turns according to packet’s load balancing hash

​

2. FREE, TX Rings

• Allocates UMEM TX frame from tx_free ring
• Copies packet payload into frame
• Enters frame into TX ring

​

3. Completion Ring

• Moves it to completion ring
• Net tile moves completed frames back to free ring

​

Loopback

• Testing and development
• Local traffic that Agave sends to itself (e.g., leader votes to own TPU socket)

​

Security Protections

• seccomp system call filtering
• User namespaces
• Dropped capabilities

• UMEM regions and RX mcaches: mapped read-only to app tiles
• Prevents malicious app tile from corrupting unrelated traffic
• Allows app tile to eavesdrop on any incoming packets

• Net tile read-only maps TX mcaches and dcaches from app tiles
• Each app tile can only access its own TX link
• Net tile speculatively copies TX packets but checks for overruns

​

Known Limitations

• No IPv6 support (practically all Solana traffic uses IPv4 as of Feb 2025)
• Does not yet use SO_PREFERRED_BUSY_POLL
• Supports only one external network interface (plus loopback)
• fdctl does not yet configure IRQ affinity or disable NIC interrupts
• Cannot share network interface with other AF_XDP apps
• Only supports simple route tables
• Performance detriment to other apps using Linux networking on shared interfaces
• RX mcaches scale as O(n*m) where n=net tiles, m=app tiles (could be O(max(n,m)))

​

Development Tools

​

fddev pktgen

$ fddev pktgen

• Runs single net tile + special pktgen tile
• Generates ~10 million 64-byte Ethernet frames per second
• Packets are not routable (cannot harm IP networks)

ip neigh add FAKE_DST_IP dev DEVICE lladdr 00:00:00:00:00:00