Skip to main content
Retis tracks packets as they flow through the Linux networking stack, allowing you to reconstruct their complete journey. This is essential for understanding how packets are processed, modified, and forwarded.
Packet tracking is not a built-in feature of the Linux kernel. Retis implements tracking mechanisms that work across most scenarios, though they cannot be 100% foolproof in all edge cases.

​
How Tracking Works

Retis uses multiple complementary approaches to track packets:

​
1. Core SKB Tracking

The Retis core generates unique identifiers by tracking the data part of socket buffers. The tracking identifier includes:
  • The socket buffer data pointer
  • The socket buffer structure address
This allows tracking of:
  • Original packets
  • Cloned packets (shallow copies that share data)
  • Modified packets (e.g., after NAT transformation)
The core tracking logic is used by the filtering system to continue tracking packets even after they’re modified. For implementation details, see the source code.

​
2. skb-tracking Collector

The skb-tracking collector retrieves the core tracking information and reports it in events: 
$ retis collect -c skb-tracking
Without enabling this collector, tracking information won’t be reported in events and can’t be used during post-processing with the sort command.
Probes for tracking packets are always installed by the Retis core, regardless of which collectors are enabled.

​
3. OVS Upcall Tracking

The ovs collector includes special tracking for OpenVSwitch upcalls. This allows following a packet when it’s:
  • Sent to the OVS userspace daemon (ovs-vswitchd)
  • Processed by OpenFlow tables
  • Re-injected back into the kernel
Enable OVS tracking with: 
$ retis collect -c ovs --ovs-track

​
Tracking and Filtering

Tracking works seamlessly with filtering. When you apply a filter:
  1. Packets matching the filter are tracked
  2. Even if the packet is later modified (e.g., by NAT)
  3. And no longer matches the original filter
  4. It continues to be reported because it’s tracked
Example scenario: 
# Filter for packets to port 8080
$ retis collect -f 'tcp port 8080'
If a packet to port 8080 is NATed to port 9090:
  • The initial packet matches and tracking begins
  • After NAT, the packet has destination port 9090
  • The packet still generates events because it’s being tracked
  • You see the complete journey including the NAT transformation

​
Using the sort Command

The sort command uses tracking information to reconstruct packet journeys by grouping and reordering events that belong to the same packet.

​
Basic Usage

# Collect events with tracking
$ retis collect -c skb-tracking -o retis.data

# Sort and display packet journeys
$ retis sort

​
Example Output

Here’s an example showing a UDP packet being dropped by Netfilter: 
$ retis collect --allow-system-changes -p ip_local_deliver \
        --nft-verdicts drop -f 'udp port 8080' -o --print

$ retis sort

136852156905 (3) [swapper/3] 0 [k] ip_local_deliver #1fdd03636dffff889641716940 (skb ffff889642506300) n 0
  172.16.42.1.39677 > 172.16.42.2.8080 tos 0x0 ttl 64 id 17803 off 0 [DF] len 31 proto UDP (17) len 3
  ns 0x1/4026531833 if 2 (eth0) rxif 2
  ct_state NEW status 0x100 udp orig [172.16.42.1.39677 > 172.16.42.2.8080] reply [172.16.42.2.8080 > 172.16.42.1.39677] zone 0 mark 0
  ↳ 136852318052 (3) [swapper/3] 0 [k] __nft_trace_packet #1fdd03636dffff889641716940 (skb ffff889642506300) n 1
      172.16.42.1.39677 > 172.16.42.2.8080 tos 0x0 ttl 64 id 17803 off 0 [DF] len 31 proto UDP (17) len 3
      ns 0x1/4026531833 if 2 (eth0) rxif 2
      table firewalld (4) chain filter_INPUT_POLICIES (124) handle 284 drop
      ct_state NEW status 0x100 udp orig [172.16.42.1.39677 > 172.16.42.2.8080] reply [172.16.42.2.8080 > 172.16.42.1.39677] zone 0 mark 0
  ↳ 136852333279 (3) [swapper/3] 0 [tp] skb:kfree_skb #1fdd03636dffff889641716940 (skb ffff889642506300) n 2 drop (reason NETFILTER_DROP)
      172.16.42.1.39677 > 172.16.42.2.8080 tos 0x0 ttl 64 id 17803 off 0 [DF] len 31 proto UDP (17) len 3
      ns 0x1/4026531833 if 2 (eth0) rxif 2
      ct_state NEW status 0x100 udp orig [172.16.42.1.39677 > 172.16.42.2.8080] reply [172.16.42.2.8080 > 172.16.42.1.39677] zone 0 mark 0

​
Understanding the Output

Key elements in sorted output:
The tracking ID appears in each event:
#1fdd03636dffff889641716940
This unique identifier is the same across all events for this packet, showing they’re part of the same journey.

​
OVS Tracking Example

When tracking packets through OpenVSwitch with userspace upcalls: 
$ retis -p generic collect -f "icmp" -c ovs --ovs-track -o
$ retis sort
Example output showing an OVS upcall: 
202388857516033 [handler7] 3215286/3215259 [tp] openvswitch:ovs_dp_upcall #b81253f4ce4bffff977beedbe580 (skb 18446629158226620928) n 5
  if 181 (p2_l) rxif 181 172.200.0.3 > 172.200.0.2 ttl 64 tos 0x0 id 58112 off 0 len 84 proto ICMP (1) type 0 code 0
  upcall (miss) port 3644007146 cpu 7

  + 202388857543026 [handler7] 3215286/3215259 [kr] queue_userspace_packet #b81253f4ce4bffff977beedbe580 (skb 18446629158226620928) n 6
    if 181 (p2_l) rxif 181 172.200.0.3 > 172.200.0.2 ttl 64 tos 0x0 id 58112 off 0 len 84 proto ICMP (1) type 0 code 0
    upcall_enqueue (miss) (7/202388857516033) q 2809249329 ret 0

  + 202388857658575 [handler9] 3215302/3215259 [u] dpif_recv:recv_upcall (ovs-vswitchd) #b81253f4ce4bffff977beedbe580 (skb 18446629158226620928) n 8
    upcall_recv q 2809249329 pkt_size 98
  + 202388857762836 [handler9] 3215302/3215259 [u] dpif_netlink_operate__:op_flow_put (ovs-vswitchd) #b81253f4ce4bffff977beedbe580 (skb 18446629158226620928) n 9
    flow_put q 2809249329 ts 202388857658575 (0)
  + 202388857771230 [handler9] 3215302/3215259 [u] dpif_netlink_operate__:op_flow_execute (ovs-vswitchd) #b81253f4ce4bffff977beedbe580 (skb 18446629158226620928) n 10
    flow_exec q 2809249329 ts 202388857658575 (0)

  + 202388857827572 [handler9] 3215302/3215259 [tp] openvswitch:ovs_do_execute_action #b81253f4ce4bffff977beedbe580 (skb 18446629158226620928) n 11
    if 181 (p2_l) 172.200.0.3 > 172.200.0.2 ttl 64 tos 0x0 id 58112 off 0 len 84 proto ICMP (1) type 0 code 0
    exec oport 2 q 2809249329
Notice the queue identifier q 2809249329 appears across:
  • The upcall enqueue event (kernel)
  • The upcall receive event (userspace)
  • The flow operations (userspace)
  • The action execute event (kernel, after re-injection)
This demonstrates how Retis tracks the packet’s journey from kernel to userspace and back.

​
Best Practices

Always Enable skb-tracking

Include the skb-tracking collector when you plan to use sort for post-processing:
$ retis collect -c skb-tracking -o

Use Filters Wisely

Start with broad filters, then narrow down. Tracking ensures you see the complete journey even if the packet changes:
$ retis collect -f 'tcp port 80' -c skb-tracking -o

Enable OVS Tracking

When working with OpenVSwitch, always use --ovs-track:
$ retis collect -c ovs --ovs-track -o

Store Events for Analysis

Always save events to a file for later analysis with sort and other commands:
$ retis collect -o retis.data
$ retis sort retis.data

​
Limitations

While Retis tracking is robust, some scenarios can be challenging:
  • Packet fragmentation and reassembly may create new socket buffers
  • Deep packet cloning in some edge cases
  • Hardware offload paths where packets bypass normal kernel paths
  • XDP programs that modify packets before skb allocation
If you encounter tracking issues, please report them to help improve Retis.

​
Next Steps

Post-Processing

Learn more about the sort command and other post-processing tools

Profiles

Use profiles to quickly configure tracking for common scenarios

Build docs developers (and LLMs) love

Get started for freeTalk to us