Skip to main content
The ovs collector provides insights into the OpenVSwitch datapath, tracking packets through both kernel and userspace components.

​
Overview

The ovs collector is essential for debugging OpenVSwitch behavior. It tracks packets through the OVS datapath, including slow-path upcalls to the userspace daemon and fast-path processing in the kernel.
The ovs collector is the most complex collector in Retis, tracking packets across kernel-userspace boundaries and maintaining correlation between events.

​
What Data is Retrieved

The ovs collector retrieves:
  • Upcalls: Packets sent to userspace for processing
  • Flow lookups: Kernel datapath flow table queries
  • Flow operations: Flow installation and modification from userspace
  • Action execution: OVS actions applied to packets
  • USDT events: Userspace daemon events (optional)
  • Flow information: Datapath and OpenFlow flow details (optional)

​
Probe Installation

The ovs collector automatically installs multiple probes: kernel tracepoints, kretprobes, and optionally USDT probes for userspace tracking.
Probes installed:
  • openvswitch:ovs_dp_upcall (tracepoint): Upcall events
  • queue_userspace_packet (kretprobe): Upcall enqueueing
  • ovs_dp_process_packet (kretprobe): Flow lookup results
  • openvswitch:ovs_do_execute_action (tracepoint): Action execution
  • USDT probes in ovs-vswitchd (optional with --ovs-track)

​
Prerequisites

​
OpenVSwitch Module

The ovs collector requires the OpenVSwitch kernel module: 
# Check if module is loaded
lsmod | grep openvswitch

# Load module if needed
modprobe openvswitch

​
USDT Support (Optional)

For userspace tracking with --ovs-track, ovs-vswitchd must be compiled with USDT support. See OVS USDT documentation for compilation instructions. Check USDT support: 
# Find ovs-vswitchd binary
which ovs-vswitchd

# Check for USDT probes
readelf -n $(which ovs-vswitchd) | grep dpif

​
Flow Enrichment Requirements (Optional)

For --ovs-enrich-flows:
  • OpenVSwitch >= 3.4
  • Access to OVS unixctl socket (typically /var/run/openvswitch/ovs-vswitchd.*.ctl)

​
Command-Line Options

​
--ovs-track

​
--ovs-track
boolean
default:"false"
Enable OpenVSwitch upcall tracking using USDT probes.When enabled:
  • Attaches to USDT probes in ovs-vswitchd
  • Tracks packets through userspace processing
  • Correlates userspace events with kernel events
  • Generates queue identifiers for upcall tracking
Requirements:
  • ovs-vswitchd compiled with USDT support
  • Retis must be in the same PID namespace as ovs-vswitchd

​
--ovs-enrich-flows

​
--ovs-enrich-flows
boolean
default:"false"
Enable datapath flow enrichment by querying the OVS daemon.When enabled:
  • Queries OVS for flow details using the unixctl interface
  • Adds ovs-detrace event sections
  • Shows datapath flows and OpenFlow flows
  • Rate-limited to prevent daemon overload
Requirements:
  • OpenVSwitch >= 3.4
  • Access to OVS control socket

​
--ovs-enrich-rate

​
--ovs-enrich-rate
integer
default:"20"
Rate limit for flow enrichment queries (requests per second).Controls how many flow lookup requests are sent to the OVS daemon per second.Considerations:
  • Higher rates provide more information but increase daemon load
  • Lower rates reduce load but may miss some flows
  • Default of 20 req/s is conservative for most deployments

​
How OpenVSwitch Works

Understanding OVS architecture helps interpret collector output:

​
Kernel Datapath

  1. Fast Path: Packets matching existing flows are processed in kernel
  2. Flow Table: Kernel maintains a cache of datapath flows
  3. Actions: Each flow has associated actions (output, modify, etc.)

​
Userspace Daemon

  1. OpenFlow Tables: Complex rule processing in ovs-vswitchd
  2. Upcalls: Packets not matching kernel flows are sent to userspace
  3. Flow Installation: Daemon installs flows in kernel for future packets

​
Packet Flow

Packet arrives
    ↓
Kernel flow lookup
    ↓
Match? → Yes → Execute actions (fast path) → Done
    ↓
    No
    ↓
Upcall to userspace (slow path)
    ↓
ovs-vswitchd processes (OpenFlow rules)
    ↓
Flow put (install in kernel)
    ↓
Flow exec (execute actions on packet)
    ↓
Packet continues

​
Event Types

The ovs collector generates several event types:

​
upcall

Packet sent to userspace (no kernel flow match): 
upcall (miss) port {port_id} cpu {cpu}

​
upcall_enqueue

Packet fragment enqueued to netlink socket: 
upcall_enqueue (miss) ({pid}/{timestamp}) q {queue_id} ret {result}
Note: Large packets may generate multiple upcall_enqueue events.

​
upcall_recv

USDT probe when ovs-vswitchd receives packet: 
upcall_recv q {queue_id} pkt_size {size}

​
flow_put

USDT probe when daemon installs a flow: 
flow_put q {queue_id} ts {timestamp} ({offset})

​
flow_exec

USDT probe when daemon executes actions: 
flow_exec q {queue_id} ts {timestamp} ({offset})

​
flow_tbl_lookup

Kernel flow table lookup result: 
flow_tbl_lookup hit ufid {ufid} masks {mask_count} cache {cache_count}
or 
flow_tbl_lookup miss

​
action_execute

Kernel executing an action: 
exec oport {output_port} q {queue_id}
See OVS Actions for action types.

​
Usage Examples

​
Basic OVS Monitoring

Monitor OVS datapath: 
retis collect -c skb,ovs

​
With Userspace Tracking

Track through userspace: 
retis collect -c skb,ovs --ovs-track

​
With Flow Enrichment

Query flow details: 
retis collect -c skb,ovs --ovs-track --ovs-enrich-flows

​
Full Tracking

Complete packet tracking: 
retis collect -c skb,skb-tracking,ovs --ovs-track -o events.json
retis sort events.json

​
With Filtering

Track specific traffic: 
# ICMP traffic
retis collect -c skb,ovs --ovs-track -f 'icmp' -o events.json
retis sort events.json

# Specific host
retis collect -c skb,ovs --ovs-track -f 'host 10.0.0.1'

​
Using Profiles

retis -p generic collect -c ovs --ovs-track

​
Example Output

From the OVS documentation, here’s a complete example of packet flow:

​
Fast Path

202388857835660 [handler9] 3215302/3215259 [tp] net:net_dev_queue #b81253f4ce4bffff977beedbe580 (skb 18446629158226620928) n 12
  if 179 (p1_l) 172.200.0.3 > 172.200.0.2 ttl 64 tos 0x0 id 58112 off 0 len 84 proto ICMP (1) type 0 code 0
Packet processed in kernel (fast path), no upcall needed.

​
Slow Path (Upcall)

# Upcall triggered
202388857516033 [handler7] 3215286/3215259 [tp] openvswitch:ovs_dp_upcall #b81253f4ce4bffff977beedbe580 (skb 18446629158226620928) n 5
  if 181 (p2_l) rxif 181 172.200.0.3 > 172.200.0.2 ttl 64 tos 0x0 id 58112 off 0 len 84 proto ICMP (1) type 0 code 0
  upcall (miss) port 3644007146 cpu 7

# Enqueue to userspace
+ 202388857543026 [handler7] 3215286/3215259 [kr] queue_userspace_packet #b81253f4ce4bffff977beedbe580 (skb 18446629158226620928) n 6
  if 181 (p2_l) rxif 181 172.200.0.3 > 172.200.0.2 ttl 64 tos 0x0 id 58112 off 0 len 84 proto ICMP (1) type 0 code 0
  upcall_enqueue (miss) (7/202388857516033) q 2809249329 ret 0
Notice the queue identifier q 2809249329 - this links kernel and userspace events.

​
Userspace Processing

# Userspace receives packet
+ 202388857658575 [handler9] 3215302/3215259 [u] dpif_recv:recv_upcall (ovs-vswitchd) #b81253f4ce4bffff977beedbe580 (skb 18446629158226620928) n 8
  upcall_recv q 2809249329 pkt_size 98

# Install flow
+ 202388857762836 [handler9] 3215302/3215259 [u] dpif_netlink_operate__:op_flow_put (ovs-vswitchd) #b81253f4ce4bffff977beedbe580 (skb 18446629158226620928) n 9
  flow_put q 2809249329 ts 202388857658575 (0)

# Execute actions
+ 202388857771230 [handler9] 3215302/3215259 [u] dpif_netlink_operate__:op_flow_execute (ovs-vswitchd) #b81253f4ce4bffff977beedbe580 (skb 18446629158226620928) n 10
  flow_exec q 2809249329 ts 202388857658575 (0)
All events share the same queue identifier, showing they’re part of the same upcall.

​
Action Execution

# Packet re-injected, action executed
+ 202388857827572 [handler9] 3215302/3215259 [tp] openvswitch:ovs_do_execute_action #b81253f4ce4bffff977beedbe580 (skb 18446629158226620928) n 11
  if 181 (p2_l) 172.200.0.3 > 172.200.0.2 ttl 64 tos 0x0 id 58112 off 0 len 84 proto ICMP (1) type 0 code 0
  exec oport 2 q 2809249329
Packet output to port 2, still linked by queue identifier.

​
OVS Tracking

The --ovs-track flag enables sophisticated packet tracking:

​
Queue Identifiers

Generated by Retis to correlate events: 
q 2809249329
Links:
  • Kernel upcall events
  • Userspace processing events
  • Re-injection and action execution

​
Cross-Boundary Tracking

Retis tracks packets even when:
  • Sent to userspace (upcall)
  • Processed by ovs-vswitchd
  • Re-injected into kernel
The same tracking ID and queue identifier appear in all related events.

​
Flow Enrichment

With --ovs-enrich-flows, Retis queries OVS for flow details:

​
dpctl/get

Shows datapath flow for a UFID: 
ovs-appctl dpctl/get {ufid}

​
ofproto/detrace

Shows OpenFlow flows (OVS 3.4+): 
ovs-appctl ofproto/detrace {ufid}

​
Usage

This information is added as ovs-detrace event sections and combined with flow_tbl_lookup events by retis sort.

​
OVS Actions

Actions reported in action_execute events:
  • OVS_ACTION_ATTR_OUTPUT: Output to port
  • OVS_ACTION_ATTR_USERSPACE: Send to userspace
  • OVS_ACTION_ATTR_SET: Set field
  • OVS_ACTION_ATTR_PUSH_VLAN: Push VLAN tag
  • OVS_ACTION_ATTR_POP_VLAN: Pop VLAN tag
  • OVS_ACTION_ATTR_RECIRC: Recirculate
  • OVS_ACTION_ATTR_HASH: Calculate hash
  • OVS_ACTION_ATTR_PUSH_MPLS: Push MPLS label
  • OVS_ACTION_ATTR_POP_MPLS: Pop MPLS label
  • And more…
See openvswitch uapi header for complete list.

​
Integration with Other Collectors

​
skb

Essential for seeing packet content: 
retis collect -c skb,ovs --ovs-track --skb-sections all

​
skb-tracking

Complete packet flow: 
retis collect -c skb,skb-tracking,ovs --ovs-track -o events.json
retis sort events.json
Shows packet journey through entire network stack and OVS.

​
skb-drop

Debug OVS drops: 
retis collect -c skb,skb-drop,ovs --ovs-track
See OVS-specific drop reasons.

​
ct (Conntrack)

Track conntrack through OVS: 
retis collect -c skb,ct,ovs --ovs-track

​
nft (Netfilter)

Firewall and OVS interaction: 
retis collect -c skb,nft,ovs --ovs-track --allow-system-changes

​
dev

Interface tracking: 
retis collect -c skb,dev,ovs --ovs-track

​
Use Cases

​
Debug Missing Flows

retis collect -c skb,ovs --ovs-track -f 'host 10.0.0.1' -o events.json
retis sort events.json
See if packets are upcalling instead of using fast path.

​
Performance Analysis

retis collect -c skb,ovs --ovs-track --ovs-enrich-flows
Identify frequent upcalls that should be cached.

​
Flow Installation Issues

retis collect -c skb,ovs --ovs-track -o events.json
retis sort events.json
Verify flows are installed after upcalls.

​
Action Debugging

retis collect -c skb,ovs --ovs-track
See what actions are executed on packets.

​
OpenFlow Troubleshooting

retis collect -c skb,ovs --ovs-track --ovs-enrich-flows -f 'tcp port 80'
Map datapath flows to OpenFlow rules.

​
Technical Details

​
Kernel Types

The ovs collector activates when these types appear in probe arguments:
  • struct sk_buff *

​
USDT Probes

When --ovs-track is enabled, attaches to:
  • dpif_recv:recv_upcall: Upcall receipt
  • dpif_netlink_operate__:op_flow_put: Flow installation
  • dpif_netlink_operate__:op_flow_execute: Action execution

​
Tracking Maps

The collector maintains BPF maps for:
  • In-flight upcalls
  • In-flight flow operations
  • Upcall batches
  • Queue ID generation

​
Garbage Collection

A background thread runs every 5 seconds to clean up old tracking entries (older than 60 seconds).

​
Source Code References

  • Collector: retis/src/collect/collector/ovs/ovs.rs
  • Flow enrichment: retis/src/collect/collector/ovs/flow_info.rs
  • eBPF hooks: retis/src/collect/collector/ovs/bpf/
  • Event factory: retis/src/collect/collector/ovs/bpf.rs

​
Best Practices

  1. Use --ovs-track for debugging: Essential for understanding upcall flow
  2. Combine with skb-tracking: See complete packet journey
  3. Filter traffic: OVS can generate many events
  4. Save to file: Use post-processing with retis sort
  5. Check USDT support: Verify before using --ovs-track
  6. Consider rate limits: Adjust --ovs-enrich-rate based on load
  7. Use flow enrichment selectively: Only when you need OpenFlow details

​
Performance Considerations

  • Multiple probes: Several kernel and USDT probes
  • Tracking overhead: Maps and GC thread
  • USDT impact: Userspace probes have overhead
  • Enrichment cost: Flow queries add daemon load
  • Production use: Test performance impact before deployment

​
Troubleshooting

​
No OVS Events

If OVS events aren’t appearing:
  1. Check module is loaded: 
    lsmod | grep openvswitch
  2. Verify traffic is going through OVS: 
    ovs-dpctl show
  3. Ensure collector is enabled: 
    retis collect -c ovs

​
USDT Probes Fail

If --ovs-track fails:
  1. Check USDT support: 
    readelf -n $(which ovs-vswitchd) | grep dpif
  2. Verify Retis is in same namespace: 
    ps aux | grep ovs-vswitchd
  3. See OVS USDT documentation

​
Flow Enrichment Fails

If --ovs-enrich-flows fails:
  1. Check OVS version: 
    ovs-vswitchd --version
  2. Verify control socket: 
    ls /var/run/openvswitch/*.ctl
  3. Test manually: 
    ovs-appctl dpctl/get {ufid}

​
Missing Queue IDs

If events aren’t correlated:
  1. Ensure --ovs-track is enabled
  2. Check USDT probes are working
  3. Verify tracking maps aren’t full (check logs)

​
See Also

Build docs developers (and LLMs) love

Get started for freeTalk to us