Skip to main content
The skb-tracking collector reports tracking information (unique IDs) that can be used at post-processing time to reconstruct in-kernel packet flows.

​
Overview

The skb-tracking collector enables packet tracking through the Linux networking stack by reporting unique identifiers and socket buffer addresses in events. This allows you to follow a packet’s journey, even after transformations like NAT or cloning.
The skb-tracking collector does not track skbs in the kernel itself—that’s done by Retis core. This collector only reports the tracking information in events.

​
What Data is Retrieved

The skb-tracking collector retrieves:
  • Tracking ID: A unique identifier generated by Retis’s core tracking logic
  • SKB address: The socket buffer’s memory address
This information helps distinguish between:
  • The same packet at different points in the stack
  • Different packets entirely
  • Clones of the same packet (they share a tracking ID but have different SKB addresses)

​
Probe Installation

The skb-tracking collector does not install any probes. It relies on Retis core’s built-in skb tracking, which is always active.
Probes for tracking packets are automatically installed by the core tracking system.

​
Command-Line Options

The skb-tracking collector has no specific command-line options.

​
Event Sections Produced

The skb-tracking collector produces the skb-tracking event section. See skb-tracking event documentation for detailed format.

​
Event Format

#{tracking_id} (skb {skb_address})
Example: 
#b81253ea5defffff977be5ec6f80 (skb 18446629157470561024)
Where:
  • tracking_id: Unique identifier for the packet’s data buffer
  • skb_address: Memory address of the socket buffer structure

​
Usage Examples

​
Basic Tracking

Enable tracking to follow packets: 
retis collect -c skb,skb-tracking

​
Track and Sort

Collect events then group by packet: 
# Collect events to file
retis collect -c skb,skb-tracking -o events.json

# Sort events by packet flow
retis sort events.json
The sort command uses tracking IDs to group events that belong to the same packet.

​
Track Drops

Follow packets until they’re dropped: 
retis collect -c skb,skb-tracking,skb-drop -o events.json
retis sort events.json

​
Track Through OVS

Follow packets through OpenVSwitch including upcalls: 
retis collect -c skb,skb-tracking,ovs --ovs-track -o events.json
retis sort events.json

​
With Filtering

Track only specific traffic: 
# Track TCP traffic on port 443
retis collect -c skb,skb-tracking -f 'tcp port 443' -o events.json

# Track specific namespace
retis collect -c skb,skb-tracking -m 'sk_buff.dev.nd_net.net.ns.inum == 4026531840'

​
Example Output

​
Raw Events

Without sorting, events show tracking information: 
202388856790511 [ping] 3215414 [tp] net:net_dev_queue #b81253ea5defffff977be5ec6f80 (skb 18446629157470561024) n 0
  if 178 (p1_r) 172.200.0.2 > 172.200.0.3 ttl 64 tos 0x0 id 22378 off 0 [DF] len 84 proto ICMP (1) type 8 code 0

202388856802883 [ping] 3215414 [k] skb_scrub_packet #b81253ea5defffff977be5ec6f80 (skb 18446629157470561024) n 1
  if 178 (p1_r) 172.200.0.2 > 172.200.0.3 ttl 64 tos 0x0 id 22378 off 0 [DF] len 84 proto ICMP (1) type 8 code 0
Notice both events share the same tracking ID (#b81253ea5defffff977be5ec6f80) and SKB address.

​
Sorted Events

After running retis sort, events are grouped by packet: 
202388856790511 [ping] 3215414 [tp] net:net_dev_queue #b81253ea5defffff977be5ec6f80 (skb 18446629157470561024) n 0
  if 178 (p1_r) 172.200.0.2 > 172.200.0.3 ttl 64 tos 0x0 id 22378 off 0 [DF] len 84 proto ICMP (1) type 8 code 0
  + 202388856802883 [ping] 3215414 [k] skb_scrub_packet #b81253ea5defffff977be5ec6f80 (skb 18446629157470561024) n 1
    if 178 (p1_r) 172.200.0.2 > 172.200.0.3 ttl 64 tos 0x0 id 22378 off 0 [DF] len 84 proto ICMP (1) type 8 code 0
  + 202388856809633 [ping] 3215414 [tp] net:netif_rx #b81253ea5defffff977be5ec6f80 (skb 18446629157470561024) n 2
    if 179 (p1_l) 172.200.0.2 > 172.200.0.3 ttl 64 tos 0x0 id 22378 off 0 [DF] len 84 proto ICMP (1) type 8 code 0
Events are now indented (+) under the first event, showing they’re all part of the same packet’s journey.

​
How Tracking Works

​
Core Tracking

Retis’s core implements packet tracking by monitoring the data part of socket buffers:
  1. Data Buffer Tracking: The unique ID is based on the skb’s data buffer pointer
  2. Clone Detection: Cloned skbs share the same data buffer, so they get the same tracking ID
  3. Transformation Handling: Even after NAT or other modifications, the tracking ID persists

​
Limitations

Packet tracking cannot be 100% foolproof because:
  • It’s not a built-in kernel feature
  • Some transformations may break tracking (though rare)
  • Memory reuse can theoretically cause ID collisions (very rare)
If you encounter tracking issues, please report them as bugs. The tracking implementation is continuously improved.

​
Implementation Details

Full implementation details can be found in the source code: retis/src/core/tracking/skb_tracking.rs

​
Integration with Other Collectors

​
skb

Essential combination for seeing packet content: 
retis collect -c skb,skb-tracking --skb-sections all -o events.json
retis sort events.json
Shows packet content at each point in its journey.

​
skb-drop

Track packets to their drop point: 
retis collect -c skb,skb-tracking,skb-drop -o events.json
retis sort events.json
See the full path a packet took before being dropped.

​
ovs

Track packets through OpenVSwitch: 
retis collect -c skb,skb-tracking,ovs --ovs-track -o events.json
retis sort events.json
The OVS collector adds additional upcall tracking that integrates with skb tracking.

​
ct

Track conntrack state changes: 
retis collect -c skb,skb-tracking,ct -o events.json
retis sort events.json
See how conntrack state evolves as a packet moves through the stack.

​
nft

Track packets through Netfilter: 
retis collect -c skb,skb-tracking,nft --allow-system-changes -o events.json
retis sort events.json
See which firewall rules a packet traverses.

​
Post-Processing with Sort

The primary use of skb-tracking is with the sort post-processing command:

​
Basic Sorting

retis collect -c skb,skb-tracking -o events.json
retis sort events.json

​
Sorting with Output File

retis sort events.json -o sorted.json

​
Advanced Sorting

The sort command:
  1. Groups events by tracking ID
  2. Orders events chronologically within each group
  3. Indents related events under the first event
  4. Handles special cases like OVS upcalls

​
Use Cases

​
Debugging Packet Loss

Find where packets disappear: 
retis collect -c skb,skb-tracking,skb-drop -f 'host 10.0.0.1' -o events.json
retis sort events.json

​
Understanding Packet Path

See the full journey of a packet: 
retis collect -c skb,skb-tracking,dev -p 'net:*' -o events.json
retis sort events.json

​
NAT Troubleshooting

Track packets through NAT: 
retis collect -c skb,skb-tracking,ct -f 'tcp port 80' -o events.json
retis sort events.json
The tracking ID persists across NAT transformations.

​
Clone Investigation

See when and where packets are cloned: 
retis collect -c skb,skb-tracking --skb-sections dataref -o events.json
retis sort events.json
Clones share tracking IDs but have different SKB addresses.

​
Cross-Namespace Tracking

retis collect -c skb,skb-tracking,ns -o events.json
retis sort events.json
Track packets as they move between network namespaces.

​
Technical Details

​
Kernel Types

The skb-tracking collector activates when these types appear in probe arguments:
  • struct sk_buff *

​
Tracking ID Format

The tracking ID is displayed in hexadecimal format: 
#b81253ea5defffff977be5ec6f80
It’s derived from:
  • SKB data buffer pointer
  • Additional entropy for uniqueness

​
SKB Address

The SKB address is the actual kernel memory address of the struct sk_buff: 
skb 18446629157470561024
Multiple SKBs (clones) can share the same tracking ID if they reference the same data.

​
Source Code References

  • Core tracking: retis/src/core/tracking/skb_tracking.rs
  • Collector: retis/src/collect/collector/skb_tracking/
  • Sort command: retis/src/process/sort/

​
Best Practices

  1. Always use with sort: The real value comes from post-processing
  2. Save to file: Use -o to enable post-processing
  3. Combine with skb: See packet content at each point
  4. Use filtering: Track specific flows to reduce noise
  5. Enable related collectors: Add context with ct, nft, ovs, etc.

​
Performance Considerations

  • Core overhead: Tracking is always active in core (minimal overhead)
  • Collector overhead: Only adds tracking ID to events (negligible)
  • Post-processing: Sorting is fast and scales well
  • Storage: Each event adds ~24 bytes for tracking info

​
Troubleshooting

​
Events Not Grouped

If events aren’t grouping after sort:
  1. Verify skb-tracking collector was enabled during collection
  2. Check that events actually share the same packet (filter might be too broad)
  3. Ensure tracking IDs are present in events

​
Unexpected Tracking IDs

If tracking IDs change unexpectedly:
  1. This might indicate packet transformation that breaks tracking
  2. Report as a bug with reproduction steps
  3. Check if data buffer was reallocated (rare)

​
Missing Tracking Info

If tracking info is missing:
  1. Verify collector was enabled: retis collect -c skb-tracking
  2. Check that probes have struct sk_buff * arguments
  3. Ensure core tracking didn’t fail (check logs)

​
See Also

Build docs developers (and LLMs) love

Get started for freeTalk to us