AF_XDP Networking Internals

​

What is AF_XDP?

• Redirects packets from/to shared memory buffers
• Hardware-agnostic (unlike DPDK)
• Can share NIC with Linux networking stack
• Allows deployment in existing, heterogeneous networks
• Shared memory region called “UMEM”

​

XDP (eXpress Data Path)

• Runs before tc and netfilter
• eBPF is JIT-compiled bytecode
• Some hardware/driver combinations offload eBPF to NICs
• Not to be confused with Solana BPF (sBPF)

​

Packet Flow Architecture

​

RX Block Diagram

   ┌─────┐  ┌────────┐  ┌─────┐ XDP_PASS ┌─────────┐
   │ NIC ├──> Driver ├──> XDP ├──────────> sk_buff │
   └─────┘  └────────┘  └─┬───┘          └─────────┘
                          │
                          │ XDP_REDIRECT
                          │
                       ┌──▼───────┐      ┌─────────┐
                       │ XSK/UMEM ├──────> fd_aio  │
                       └──────────┘      └─────────┘

1. NIC receives packet via hardware
2. Driver passes packet to XDP facility
3. XDP eBPF program inspects packet:
   • XDP_PASS: Continue to Linux stack (sk_buff, ip_rcv, tc, netfilter)
   • XDP_REDIRECT: Copy to XSK UMEM, allocate RX queue entry
4. fd_aio backend provides async I/O interface

​

TX Block Diagram

   ┌────────┐  ┌──────────┐  ┌────────┐  ┌─────┐
   │ fd_aio ├──> XSK/UMEM ├──> Driver ├──> NIC │
   └────────┘  └──────────┘  └────────┘  └─────┘

1. fd_aio: Userspace delivers packets to XSK/UMEM buffers
2. Kernel: Forwards packets to NIC driver
3. Driver: NIC transmits packets

​

Memory Management

​

UMEM Overview

• Allocated from userspace (recommended: huge pages via fd_util shmem/wksp APIs)
• Divided into equally sized frames
• Each frame owned by either userspace or kernel at any time
• All frames initially owned by userspace

/* FD_XSK_UMEM_ALIGN: byte alignment requirement (Linux 4.18+) */
#define FD_XSK_UMEM_ALIGN (4096UL)

​

Ring Buffers

Data flow:
(U->K) = userspace-to-kernel
(K->U) = kernel-to-userspace

FILL (U->K):        Free frames provided to kernel
                    Kernel may populate with RX packet data

RX (K->U):          Kernel passes filled frames back to userspace

TX (U->K):          TX frames sent by userspace to kernel

COMPLETION (K->U):  Kernel returns processed TX frames to userspace

​

XDP Ring Structure

​

Ring Descriptor

struct __attribute__((aligned(64UL))) fd_xdp_ring {
  /* mmap() params for munmap() during leave */
  void *  mem;      /* Start of shared descriptor ring mmap region */
  ulong   map_sz;   /* Size of shared descriptor ring mmap region */
  
  /* Pointers to opaque XSK ring structure
     (layout is unstable, queried via getsockopt(SOL_XDP, XDP_MMAP_OFFSETS)) */
  union {
    void *            ptr;          /* Opaque pointer */
    struct xdp_desc * packet_ring;  /* For RX, TX rings */
    ulong *           frame_ring;   /* For FILL, COMPLETION rings */
  };
  uint *  flags;       /* Points to flags in shared descriptor ring */
  uint *  prod;        /* Points to producer seq in shared ring */
  uint *  cons;        /* Points to consumer seq in shared ring */
  
  /* Managed by fd_xsk_t */
  uint    depth;       /* Capacity of ring in number of entries */
  uint    cached_prod; /* Cached value of *prod */
  uint    cached_cons; /* Cached value of *cons */
};

​

Ring Operations

#define FD_XDP_RING_ROLE_PROD 0
#define FD_XDP_RING_ROLE_CONS 1

static inline int
fd_xdp_ring_empty( fd_xdp_ring_t * ring, uint role ) {
  if( role == FD_XDP_RING_ROLE_PROD ) {
    /* Userspace is producer (fill, tx) */
    if( FD_UNLIKELY( ring->cached_prod == ring->cached_cons ) ) 
      return 1;
    ring->cached_cons = FD_VOLATILE_CONST( *ring->cons );
  } else {
    /* Userspace is consumer (rx, completion) */
    if( FD_LIKELY( ring->cached_cons < ring->cached_prod ) ) 
      return 0;
    ring->cached_prod = FD_VOLATILE_CONST( *ring->prod );
  }
  return ring->cached_prod == ring->cached_cons;
}

• Caches producer/consumer sequence numbers
• Only refreshes from shared memory when potentially empty/full
• Reduces expensive memory fence operations

​

XSK Socket Structure

struct fd_xsk {
  /* Informational */
  uint if_idx;                    /* Net device index */
  uint if_queue_id;               /* Net device combined queue index */
  long log_suppress_until_ns;     /* Suppress log messages until time */

  /* Kernel descriptor of XSK rings
     from getsockopt(SOL_XDP, XDP_MMAP_OFFSETS) */
  struct xdp_mmap_offsets offsets;

  /* AF_XDP socket file descriptor */
  int xsk_fd;

  /* XSK ring descriptors */
  fd_xdp_ring_t ring_rx;   /* Receive ring */
  fd_xdp_ring_t ring_tx;   /* Transmit ring */
  fd_xdp_ring_t ring_fr;   /* Fill ring */
  fd_xdp_ring_t ring_cr;   /* Completion ring */
};

​

XSK Parameters

struct fd_xsk_params {
  /* Ring depths */
  ulong fr_depth;  /* Fill ring depth */
  ulong rx_depth;  /* RX ring depth */
  ulong tx_depth;  /* TX ring depth */
  ulong cr_depth;  /* Completion ring depth */

  /* UMEM configuration */
  void * umem_addr;  /* Pointer to UMEM in local address space */
  ulong frame_sz;    /* Frame size in UMEM ring buffers */
  ulong umem_sz;     /* Total UMEM size (contiguous, aligned) */

  /* Network interface */
  uint if_idx;       /* Linux interface index */
  uint if_queue_id;  /* Interface queue index */

  /* XDP flags */
  uint bind_flags;   /* sockaddr_xdp.sxdp_flags (e.g., XDP_ZEROCOPY) */
  
  /* Core dump configuration */
  int core_dump;     /* Include xsk memory in core dumps */
};

​

Important Flags

• Driver mode (fast path)
• XDP support in NIC driver before sk_buff allocation
• Best performance
• Hardware/driver dependent

• Socket buffer mode (fallback)
• Uses Linux sk_buff for memory management
• Works everywhere
• Slower than driver mode

• Enables zero-copy I/O
• UMEM directly accessible to NIC via DMA
• Requires driver mode
• Eliminates software packet copies

​

Zero-Copy I/O

1. PCIe device initiates DMA write to DRAM
2. Device driver signals new packet arrival
3. Kernel XDP passes pointer through to net tile
4. Net tile identifies recipient app tile
5. Net tile passes pointer via mcache to app tile
6. App tile reads packet data directly from UMEM

​

IOMMU Protection

• Firedancer app tiles: read-only
• Firedancer net tiles: read-write
• Linux kernel: read-write
• PCIe network devices: read-write (via IOMMU)

• Provides memory protection for DMA
• Prevents NIC from accessing arbitrary memory
• Allows zero-copy while maintaining security

​

Supported Drivers

Driver    NIC Model              Notes
------    ---------              -----
ixgbe     Intel X540             Stable, widely available
i40e      Intel X710 series      Good performance
ice       Intel E800 series      Latest generation

• Driver mode support varies by driver
• Zero-copy support varies by driver
• Performance characteristics differ

​

Netlink Integration

​

Configuration Sources

​

Netbase Shared Memory

• Interface configurations
• Routing information
• Neighbor (ARP) entries

​

ARP Handling

1. App tile needs to send to IP address
2. Net tile checks neighbor table for MAC address
3. If not found, net tile notifies netlink tile
4. Netlink tile requests neighbor solicitation via netlink
5. Kernel broadcasts ARP request
6. ARP response received, kernel updates neighbor table
7. Netlink tile updates netbase shared memory
8. Net tile can now send packet

​

Security Considerations

​

Required Capabilities

• CAP_SYS_ADMIN - System administration
• CAP_NET_RAW - Raw network access

​

Isolation

• Separate process with own sandbox
• Only net tiles can communicate with it
• Net tiles have read-only access to netbase
• Malicious netlink tile can compromise net tiles, but not vice versa

​

XDP Program

• Redirects Firedancer traffic to AF_XDP sockets
• Passes other traffic to Linux stack (XDP_PASS)
• Unloaded when Firedancer exits

​

Limitations

​

Monitoring

​

Interface Sharing

• Firedancer cannot share an interface with other AF_XDP apps
• May cause performance impact to other apps using Linux networking on shared interfaces
• Only one external network interface supported (plus loopback)

​

IPv6

• Practically all Solana traffic uses IPv4 (as of Feb 2025)
• IPv6 would be more expensive (lower MTU, mandatory UDP checksums, longer addresses)
• Adding IPv6 support would be straightforward if needed

​

Performance Characteristics

​

Target Performance

• RX throughput: ~20 million packets per second
• Kernel wake-ups: ~20,000 times per second
• Busy polling: Net tile never sleeps

​

Optimizations

​

Future Improvements

• SO_PREFERRED_BUSY_POLL not yet used
• IRQ affinity not yet configured
• NIC interrupts not yet disabled
• Could reduce RX mcache scaling from O(n*m) to O(max(n,m))
• tx_free ring probably obsolete (could move COMPLETION→TX directly)