UVC Streaming

Overview

UVC (USB Video Class) is a standardized protocol for transmitting video data over USB. The ESP32_USB_STREAM library implements a UVC host driver that allows ESP32-S2 and ESP32-S3 to receive video streams from USB cameras.

The library supports MJPEG format by default, which provides a good balance between image quality and bandwidth requirements.

How UVC Streaming Works

The library creates internal FreeRTOS tasks to handle USB data transfers from the camera’s streaming pipe. When a complete frame is assembled, it triggers a user-defined callback function.

Frame Formats and Resolutions

Supported Frame Formats

The library supports multiple UVC frame formats:

Format	Description	Use Case
`UVC_FRAME_FORMAT_MJPEG`	Motion-JPEG (default)	Most common, good compression
`UVC_FRAME_FORMAT_YUYV`	YUV 4:2:2 encoding	Uncompressed, higher bandwidth
`UVC_FRAME_FORMAT_RGB`	24-bit RGB	Direct color format
`UVC_FRAME_FORMAT_H264`	H.264 encoded	Advanced compression
`UVC_FRAME_FORMAT_GRAY8`	8-bit grayscale	Low bandwidth

Resolution Configuration

You can specify exact resolutions or use FRAME_RESOLUTION_ANY to let the library negotiate with the camera:

// Request any available resolution
usb->uvcConfiguration(
    FRAME_RESOLUTION_ANY,     // width
    FRAME_RESOLUTION_ANY,     // height
    FRAME_INTERVAL_FPS_15,    // frame interval
    55 * 1024,                // transfer buffer size
    _xferBufferA, 
    _xferBufferB,
    55 * 1024,                // frame buffer size
    _frameBuffer
);

// Request specific resolution (e.g., 640x480)
usb->uvcConfiguration(
    640,                      // width
    480,                      // height  
    FRAME_INTERVAL_FPS_30,    // 30 fps
    55 * 1024, 
    _xferBufferA, 
    _xferBufferB,
    55 * 1024, 
    _frameBuffer
);

Frame Intervals (FPS)

The library provides predefined frame intervals:

Constant	FPS	Frame Interval (100ns units)
`FRAME_INTERVAL_FPS_5`	5 fps	2,000,000
`FRAME_INTERVAL_FPS_10`	10 fps	1,000,000
`FRAME_INTERVAL_FPS_15`	15 fps	666,666
`FRAME_INTERVAL_FPS_20`	20 fps	500,000
`FRAME_INTERVAL_FPS_30`	25 fps	400,000

Higher frame rates require more bandwidth and processing power. Ensure your ESP32 has sufficient resources.

Double Buffering System

The library implements a double-buffering mechanism to ensure smooth video streaming without frame drops:

Buffer A receives data

While Buffer A is being filled with incoming USB data packets, Buffer B processes the previous frame.

Buffers swap

When Buffer A is full, the buffers swap roles seamlessly.

Frame assembly

The complete frame is assembled in the frame buffer and passed to your callback.

Buffer Size Requirements

// Both transfer buffers must be larger than one complete frame
uint8_t *_xferBufferA = (uint8_t *)malloc(55 * 1024);  // 55KB
uint8_t *_xferBufferB = (uint8_t *)malloc(55 * 1024);  // 55KB
uint8_t *_frameBuffer = (uint8_t *)malloc(55 * 1024);  // 55KB

For higher resolutions (e.g., 720p or 1080p), you may need to allocate larger buffers (100KB or more).

Frame Callbacks and Processing

The frame callback is executed when a complete frame is ready:

static void onCameraFrameCallback(uvc_frame_t *frame, void *user_ptr)
{
    // Frame information
    Serial.printf("Frame format: %d\n", frame->frame_format);
    Serial.printf("Sequence: %u\n", frame->sequence);
    Serial.printf("Width: %u\n", frame->width);
    Serial.printf("Height: %u\n", frame->height);
    Serial.printf("Data size: %u bytes\n", frame->data_bytes);
    
    // Access frame data
    uint8_t *imageData = (uint8_t *)frame->data;
    
    // Process the frame (save to SD, send over WiFi, etc.)
    // IMPORTANT: Don't block for too long in this callback!
}

// Register the callback
usb->uvcCamRegisterCb(&onCameraFrameCallback, NULL);

Frame Structure

The uvc_frame_t structure contains:

Field	Type	Description
`data`	`void*`	Pointer to image data
`data_bytes`	`size_t`	Size of image data in bytes
`width`	`uint32_t`	Frame width in pixels
`height`	`uint32_t`	Frame height in pixels
`frame_format`	`enum`	Pixel format (MJPEG, YUYV, etc.)
`sequence`	`uint32_t`	Frame sequence number
`capture_time`	`struct timeval`	Capture timestamp

Do not call any UVC functions from within the frame callback. This can cause deadlocks. Process the frame data quickly and return.

Suspend and Resume Capabilities

You can temporarily pause video streaming without disconnecting the camera:

// Suspend streaming (camera stays connected)
usb->uvcCamSuspend(NULL);

// Resume streaming
usb->uvcCamResume(NULL);

Dynamic Frame Configuration

You can change resolution or frame rate while suspended:

// Suspend the stream
usb->uvcCamSuspend(NULL);

// Change to different resolution and frame rate
usb->uvcCamFrameReset(1280, 720, FRAME_INTERVAL_FPS_30);

// Resume with new settings
usb->uvcCamResume(NULL);

Frame configuration changes only take effect after calling uvcCamResume().

Querying Camera Capabilities

Get Available Frame Sizes

// Get the number of available frame sizes
size_t frame_list_size = 0;
size_t current_index = 0;
usb->uvcCamGetFrameListSize(&frame_list_size, &current_index);

// Allocate array and get frame size list
uvc_frame_size_t *frameList = new uvc_frame_size_t[frame_list_size];
usb->uvcCamGetFrameSize(frameList);

// Print available frame sizes
for (size_t i = 0; i < frame_list_size; i++) {
    Serial.printf("Frame %d: %dx%d\n", 
                  i, 
                  frameList[i].width, 
                  frameList[i].height);
    Serial.printf("  Interval: %u - %u (step: %u)\n",
                  frameList[i].interval_min,
                  frameList[i].interval_max,
                  frameList[i].interval_step);
}

Transfer Modes

UVC supports two transfer types:

Isochronous Transfer

Default mode for most cameras
Guaranteed bandwidth
Time-sensitive delivery
May have occasional data loss

Bulk Transfer

Higher bandwidth potential
Guaranteed delivery
No time guarantees
Better for high-resolution streams

Example: Complete UVC Setup

#include <Arduino.h>
#include "USB_STREAM.h"

static void onCameraFrameCallback(uvc_frame_t *frame, void *user_ptr)
{
    Serial.printf("Frame %u: %ux%u, %u bytes\n",
                  frame->sequence,
                  frame->width,
                  frame->height,
                  frame->data_bytes);
}

void setup()
{
    Serial.begin(115200);
    
    // Create USB_STREAM instance
    USB_STREAM *usb = new USB_STREAM();
    
    // Allocate buffers (55KB each for typical MJPEG frames)
    uint8_t *xferBufferA = (uint8_t *)malloc(55 * 1024);
    uint8_t *xferBufferB = (uint8_t *)malloc(55 * 1024);
    uint8_t *frameBuffer = (uint8_t *)malloc(55 * 1024);
    
    // Configure UVC with 15 fps at any resolution
    usb->uvcConfiguration(
        FRAME_RESOLUTION_ANY,
        FRAME_RESOLUTION_ANY,
        FRAME_INTERVAL_FPS_15,
        55 * 1024, xferBufferA, xferBufferB,
        55 * 1024, frameBuffer
    );
    
    // Register callback
    usb->uvcCamRegisterCb(&onCameraFrameCallback, NULL);
    
    // Start streaming
    usb->start();
    
    // Wait for camera connection (1 second timeout)
    usb->connectWait(1000);
    
    Serial.println("UVC streaming started!");
}

void loop()
{
    vTaskDelay(pdMS_TO_TICKS(100));
}

Best Practices

Memory Management

Always check malloc() returns non-NULL before use
Allocate buffers large enough for your target resolution
Free buffers when streaming is stopped to prevent memory leaks

Callback Performance

Keep callback functions short and fast
Avoid blocking operations (Serial.print, delay, etc.)
Use queues to pass frame data to other tasks if needed
Never call USB_STREAM functions from within callbacks

Frame Rate Optimization

Start with lower frame rates (5-15 fps) and increase if stable
Monitor free heap memory during operation
Consider reducing resolution if experiencing frame drops
Use bulk transfer mode for high-resolution streams if supported

UAC Streaming

Learn about USB audio streaming

Hardware Requirements

ESP32 SoC and USB wiring requirements

Get Started

Core Concepts

Guides

Examples

Overview

How UVC Streaming Works

Frame Formats and Resolutions

Supported Frame Formats

Resolution Configuration

Frame Intervals (FPS)

Double Buffering System

Buffer Size Requirements

Frame Callbacks and Processing

Frame Structure

Suspend and Resume Capabilities

Dynamic Frame Configuration

Querying Camera Capabilities

Get Available Frame Sizes

Transfer Modes

Isochronous Transfer

Bulk Transfer

Example: Complete UVC Setup

Best Practices

UAC Streaming

Hardware Requirements

Build docs developers (and LLMs) love

Get Started

Core Concepts

Guides

Examples

​Overview

​How UVC Streaming Works

​Frame Formats and Resolutions

​Supported Frame Formats

​Resolution Configuration

​Frame Intervals (FPS)

​Double Buffering System

​Buffer Size Requirements

​Frame Callbacks and Processing

​Frame Structure

​Suspend and Resume Capabilities

​Dynamic Frame Configuration

​Querying Camera Capabilities

​Get Available Frame Sizes

​Transfer Modes

Isochronous Transfer

Bulk Transfer

​Example: Complete UVC Setup

​Best Practices

​Related Topics

UAC Streaming

Hardware Requirements

Build docs developers (and LLMs) love

Overview

How UVC Streaming Works

Frame Formats and Resolutions

Supported Frame Formats

Resolution Configuration

Frame Intervals (FPS)

Double Buffering System

Buffer Size Requirements

Frame Callbacks and Processing

Frame Structure

Suspend and Resume Capabilities

Dynamic Frame Configuration

Querying Camera Capabilities

Get Available Frame Sizes

Transfer Modes

Example: Complete UVC Setup

Best Practices

Related Topics