libswscale - FFmpeg

libswscale implements highly optimized color conversion and image scaling routines for video processing. It provides efficient software scaling with support for various pixel formats and scaling algorithms.

Purpose and Capabilities

libswscale provides:

Video Scaling: Resize video frames to different resolutions
Color Space Conversion: Convert between pixel formats (RGB, YUV, etc.)
Bit Depth Conversion: Convert between different bit depths
Interlacing Support: Handle interlaced video
Optimized Algorithms: Multiple scaling algorithms for quality/speed tradeoffs
SIMD Optimization: CPU-specific optimizations (SSE, AVX, NEON)

Key Concepts

SwsContext

The main structure for scaling and conversion operations.

typedef struct SwsContext SwsContext;

Maintains:

Source and destination image parameters
Scaling algorithm configuration
Conversion tables and lookup data
Internal buffers for multi-pass operations

Pixel Formats

FFmpeg supports hundreds of pixel formats:

YUV Formats: YUV420P, YUV422P, YUV444P, NV12, NV21
RGB Formats: RGB24, BGR24, RGBA, BGRA, RGB48
Packed vs Planar: Data organization
Bit Depths: 8-bit, 10-bit, 12-bit, 16-bit

Scaling Algorithms

Different algorithms offer quality/speed tradeoffs:

Algorithm	Quality	Speed	Use Case
Fast Bilinear	Low	Fastest	Real-time previews
Bilinear	Medium	Fast	General purpose
Bicubic	High	Medium	High quality scaling
Lanczos	Highest	Slowest	Professional quality
Point	N/A	Very Fast	Nearest neighbor (no interpolation)

Core Functions

Context Management

sws_getContext()

Allocates and initializes a SwsContext.

SwsContext *sws_getContext(int srcW, int srcH, enum AVPixelFormat srcFormat,
                          int dstW, int dstH, enum AVPixelFormat dstFormat,
                          int flags, SwsFilter *srcFilter,
                          SwsFilter *dstFilter, const double *param);

Parameters:

srcW, srcH: Source width and height
srcFormat: Source pixel format
dstW, dstH: Destination width and height
dstFormat: Destination pixel format
flags: Scaling algorithm flags
srcFilter, dstFilter: Custom filters (usually NULL)
param: Algorithm-specific parameters (usually NULL)

Returns: Allocated context or NULL on failure Example:

SwsContext *sws_ctx = sws_getContext(
    1920, 1080, AV_PIX_FMT_YUV420P,  // Source: 1080p YUV
    1280, 720, AV_PIX_FMT_RGB24,      // Dest: 720p RGB
    SWS_BILINEAR,                      // Algorithm
    NULL, NULL, NULL                   // Filters and params
);

if (!sws_ctx) {
    fprintf(stderr, "Cannot create scaling context\n");
    return -1;
}

sws_getCachedContext()

Reuses or reallocates a scaling context.

SwsContext *sws_getCachedContext(SwsContext *context,
                                int srcW, int srcH, enum AVPixelFormat srcFormat,
                                int dstW, int dstH, enum AVPixelFormat dstFormat,
                                int flags, SwsFilter *srcFilter,
                                SwsFilter *dstFilter, const double *param);

Example:

// Reuse context with potentially different parameters
sws_ctx = sws_getCachedContext(sws_ctx,
                               1920, 1080, AV_PIX_FMT_YUV420P,
                               1280, 720, AV_PIX_FMT_RGB24,
                               SWS_BICUBIC, NULL, NULL, NULL);

sws_freeContext()

Frees a SwsContext.

void sws_freeContext(SwsContext *swsContext);

Example:

sws_freeContext(sws_ctx);
sws_ctx = NULL;

sws_free_context()

Modern API to free context (sets pointer to NULL).

void sws_free_context(SwsContext **ctx);

Modern Frame API

sws_scale_frame()

Scales an AVFrame (recommended modern API).

int sws_scale_frame(SwsContext *c, AVFrame *dst, const AVFrame *src);

Parameters:

c: SwsContext (can be NULL for automatic context creation)
dst: Destination frame
src: Source frame

Returns: 0 on success, negative error code on failure Example:

AVFrame *src_frame = av_frame_alloc();
AVFrame *dst_frame = av_frame_alloc();

// Configure destination frame
dst_frame->format = AV_PIX_FMT_RGB24;
dst_frame->width = 1280;
dst_frame->height = 720;
av_frame_get_buffer(dst_frame, 0);

// Scale
SwsContext *sws_ctx = NULL; // Auto-allocated
int ret = sws_scale_frame(sws_ctx, dst_frame, src_frame);
if (ret < 0) {
    fprintf(stderr, "Scaling failed\n");
}

av_frame_free(&src_frame);
av_frame_free(&dst_frame);
sws_free_context(&sws_ctx);

sws_frame_setup()

Prepares a scaling context from frame parameters.

int sws_frame_setup(SwsContext *ctx, const AVFrame *dst, const AVFrame *src);

sws_is_noop()

Checks if scaling would be a no-op.

int sws_is_noop(const AVFrame *dst, const AVFrame *src);

Returns: 1 if no conversion needed, 0 otherwise

Legacy Scaling API

sws_scale()

Scales image data (legacy API, still widely used).

int sws_scale(SwsContext *c,
             const uint8_t *const srcSlice[], const int srcStride[],
             int srcSliceY, int srcSliceH,
             uint8_t *const dst[], const int dstStride[]);

Parameters:

c: SwsContext
srcSlice: Source data planes
srcStride: Source line sizes
srcSliceY: Starting Y position in source
srcSliceH: Height of source slice
dst: Destination data planes
dstStride: Destination line sizes

Returns: Height of output slice Example:

SwsContext *sws_ctx = sws_getContext(
    src_width, src_height, AV_PIX_FMT_YUV420P,
    dst_width, dst_height, AV_PIX_FMT_RGB24,
    SWS_BILINEAR, NULL, NULL, NULL
);

// Allocate destination buffer
int dst_linesize = dst_width * 3; // RGB24 = 3 bytes per pixel
uint8_t *dst_data = av_malloc(dst_linesize * dst_height);

// Scale
const uint8_t *src_data[4] = {src_frame->data[0], src_frame->data[1], 
                              src_frame->data[2], NULL};
int src_linesize[4] = {src_frame->linesize[0], src_frame->linesize[1],
                      src_frame->linesize[2], 0};

uint8_t *dst_planes[4] = {dst_data, NULL, NULL, NULL};
int dst_linesize_arr[4] = {dst_linesize, 0, 0, 0};

int output_height = sws_scale(sws_ctx, src_data, src_linesize,
                             0, src_height, dst_planes, dst_linesize_arr);

printf("Scaled to height: %d\n", output_height);

av_free(dst_data);
sws_freeContext(sws_ctx);

Format Testing

sws_test_format()

Tests if a pixel format is supported.

int sws_test_format(enum AVPixelFormat format, int output);

Parameters:

format: Pixel format to test
output: 1 for output support, 0 for input support

Returns: 1 if supported, 0 otherwise Example:

if (sws_test_format(AV_PIX_FMT_YUV420P, 0)) {
    printf("YUV420P supported as input\n");
}

if (sws_test_format(AV_PIX_FMT_RGB24, 1)) {
    printf("RGB24 supported as output\n");
}

sws_isSupportedInput()

Checks if format is supported as input.

int sws_isSupportedInput(enum AVPixelFormat pix_fmt);

sws_isSupportedOutput()

Checks if format is supported as output.

int sws_isSupportedOutput(enum AVPixelFormat pix_fmt);

Color Space Functions

sws_getCoefficients()

Gets color space conversion coefficients.

const int *sws_getCoefficients(int colorspace);

Example:

const int *coeffs = sws_getCoefficients(SWS_CS_ITU709);
// Use coefficients for custom conversions

sws_setColorspaceDetails()

Sets color space conversion parameters.

int sws_setColorspaceDetails(SwsContext *c, const int inv_table[4],
                            int srcRange, const int table[4],
                            int dstRange, int brightness,
                            int contrast, int saturation);

Parameters:

inv_table: Inverse color conversion matrix for input
srcRange: Input color range (0=MPEG, 1=JPEG)
table: Color conversion matrix for output
dstRange: Output color range
brightness, contrast, saturation: Adjustments

Scaling Flags

Algorithm Flags

SWS_FAST_BILINEAR  // Fast bilinear
SWS_BILINEAR       // Bilinear
SWS_BICUBIC        // Bicubic (2-tap B-spline)
SWS_X              // Experimental
SWS_POINT          // Nearest neighbor
SWS_AREA           // Area averaging
SWS_BICUBLIN       // Bicubic for luma, bilinear for chroma
SWS_GAUSS          // Gaussian
SWS_SINC           // Sinc
SWS_LANCZOS        // Lanczos (3-tap sinc)
SWS_SPLINE         // Natural bicubic spline

Additional Flags

SWS_FULL_CHR_H_INT   // Full chroma horizontal interpolation
SWS_FULL_CHR_H_INP   // Full chroma horizontal input
SWS_ACCURATE_RND     // Accurate rounding
SWS_BITEXACT         // Bit-exact output
SWS_PRINT_INFO       // Print scaling info

Example:

int flags = SWS_BICUBIC | SWS_FULL_CHR_H_INT | SWS_ACCURATE_RND;
SwsContext *sws_ctx = sws_getContext(src_w, src_h, src_fmt,
                                    dst_w, dst_h, dst_fmt,
                                    flags, NULL, NULL, NULL);

Usage Patterns

Simple Scaling

#include <libswscale/swscale.h>
#include <libavutil/frame.h>

int scale_frame(AVFrame *src_frame, int dst_width, int dst_height) {
    // Create destination frame
    AVFrame *dst_frame = av_frame_alloc();
    dst_frame->format = src_frame->format;
    dst_frame->width = dst_width;
    dst_frame->height = dst_height;
    
    if (av_frame_get_buffer(dst_frame, 0) < 0) {
        fprintf(stderr, "Cannot allocate frame buffer\n");
        av_frame_free(&dst_frame);
        return -1;
    }
    
    // Create scaling context
    SwsContext *sws_ctx = sws_getContext(
        src_frame->width, src_frame->height, src_frame->format,
        dst_width, dst_height, src_frame->format,
        SWS_BILINEAR, NULL, NULL, NULL
    );
    
    if (!sws_ctx) {
        fprintf(stderr, "Cannot create scaling context\n");
        av_frame_free(&dst_frame);
        return -1;
    }
    
    // Scale
    sws_scale(sws_ctx,
             (const uint8_t *const *)src_frame->data, src_frame->linesize,
             0, src_frame->height,
             dst_frame->data, dst_frame->linesize);
    
    printf("Scaled from %dx%d to %dx%d\n",
           src_frame->width, src_frame->height,
           dst_frame->width, dst_frame->height);
    
    // Cleanup
    sws_freeContext(sws_ctx);
    av_frame_free(&dst_frame);
    return 0;
}

Color Space Conversion

int convert_yuv_to_rgb(AVFrame *yuv_frame) {
    // Allocate RGB frame
    AVFrame *rgb_frame = av_frame_alloc();
    rgb_frame->format = AV_PIX_FMT_RGB24;
    rgb_frame->width = yuv_frame->width;
    rgb_frame->height = yuv_frame->height;
    av_frame_get_buffer(rgb_frame, 0);
    
    // Create conversion context
    SwsContext *sws_ctx = sws_getContext(
        yuv_frame->width, yuv_frame->height, AV_PIX_FMT_YUV420P,
        rgb_frame->width, rgb_frame->height, AV_PIX_FMT_RGB24,
        SWS_BILINEAR, NULL, NULL, NULL
    );
    
    // Set color space (BT.709 for HD)
    const int *inv_table = sws_getCoefficients(SWS_CS_ITU709);
    const int *table = sws_getCoefficients(SWS_CS_DEFAULT);
    
    sws_setColorspaceDetails(sws_ctx,
                            inv_table, 0,  // Input: limited range
                            table, 1,      // Output: full range
                            0, 1 << 16, 1 << 16);  // No adjustments
    
    // Convert
    sws_scale(sws_ctx,
             (const uint8_t *const *)yuv_frame->data, yuv_frame->linesize,
             0, yuv_frame->height,
             rgb_frame->data, rgb_frame->linesize);
    
    // Process RGB data
    printf("Converted to RGB: %d bytes per line\n", rgb_frame->linesize[0]);
    
    sws_freeContext(sws_ctx);
    av_frame_free(&rgb_frame);
    return 0;
}

Batch Processing with Cached Context

int process_video_frames(AVFrame **frames, int frame_count) {
    SwsContext *sws_ctx = NULL;
    AVFrame *scaled_frame = av_frame_alloc();
    
    // Configure output format
    scaled_frame->format = AV_PIX_FMT_RGB24;
    scaled_frame->width = 1280;
    scaled_frame->height = 720;
    av_frame_get_buffer(scaled_frame, 0);
    
    for (int i = 0; i < frame_count; i++) {
        AVFrame *frame = frames[i];
        
        // Reuse or create context
        sws_ctx = sws_getCachedContext(sws_ctx,
                                      frame->width, frame->height, frame->format,
                                      scaled_frame->width, scaled_frame->height,
                                      scaled_frame->format,
                                      SWS_BICUBIC, NULL, NULL, NULL);
        
        if (!sws_ctx) {
            fprintf(stderr, "Cannot create scaling context for frame %d\n", i);
            continue;
        }
        
        // Scale frame
        sws_scale(sws_ctx,
                 (const uint8_t *const *)frame->data, frame->linesize,
                 0, frame->height,
                 scaled_frame->data, scaled_frame->linesize);
        
        // Process scaled frame
        printf("Processed frame %d\n", i);
    }
    
    av_frame_free(&scaled_frame);
    sws_freeContext(sws_ctx);
    return 0;
}

Thumbnail Generation

int create_thumbnail(AVFrame *src_frame, int thumb_width, int thumb_height,
                    const char *output_file) {
    // Create thumbnail frame
    AVFrame *thumb_frame = av_frame_alloc();
    thumb_frame->format = AV_PIX_FMT_RGB24;
    thumb_frame->width = thumb_width;
    thumb_frame->height = thumb_height;
    av_frame_get_buffer(thumb_frame, 0);
    
    // Use high-quality scaling for thumbnails
    SwsContext *sws_ctx = sws_getContext(
        src_frame->width, src_frame->height, src_frame->format,
        thumb_width, thumb_height, AV_PIX_FMT_RGB24,
        SWS_LANCZOS | SWS_ACCURATE_RND,  // Best quality
        NULL, NULL, NULL
    );
    
    if (!sws_ctx) {
        av_frame_free(&thumb_frame);
        return -1;
    }
    
    // Scale
    sws_scale(sws_ctx,
             (const uint8_t *const *)src_frame->data, src_frame->linesize,
             0, src_frame->height,
             thumb_frame->data, thumb_frame->linesize);
    
    // Save thumbnail (implementation not shown)
    save_rgb_image(thumb_frame, output_file);
    
    sws_freeContext(sws_ctx);
    av_frame_free(&thumb_frame);
    return 0;
}

Performance Optimization

Choose Appropriate Algorithm

Use SWS_FAST_BILINEAR for real-time processing, SWS_BICUBIC for general use, and SWS_LANCZOS for best quality offline processing.

Reuse Contexts

Use sws_getCachedContext() when processing multiple frames with similar parameters to avoid reinitialization overhead.

Minimize Format Conversions

Keep video in native format as long as possible. Only convert when necessary (e.g., for display or encoding).

SIMD Utilization

libswscale automatically uses CPU-specific optimizations. Ensure your FFmpeg build includes SIMD support for your platform.

Common Pixel Formats

YUV Formats

Format	Description	Subsampling	Planes
YUV420P	Planar 4:2:0	2x2	3
YUV422P	Planar 4:2:2	2x1	3
YUV444P	Planar 4:4:4	None	3
NV12	Semi-planar 4:2:0	2x2	2
NV21	Semi-planar 4:2:0	2x2	2

RGB Formats

Format	Description	Bits/Pixel	Alpha
RGB24	Packed RGB	24	No
BGR24	Packed BGR	24	No
RGBA	Packed RGBA	32	Yes
BGRA	Packed BGRA	32	Yes
RGB48LE	16-bit RGB	48	No

Best Practices

Check Format Support

Use sws_test_format() to verify format support before creating contexts.

Handle Color Spaces

Use appropriate color space conversion matrices (BT.601, BT.709, BT.2020) based on your source material.

Maintain Aspect Ratio

Calculate destination dimensions to maintain aspect ratio: dst_h = dst_w * src_h / src_w

Use Modern API

Prefer sws_scale_frame() for new code as it’s simpler and handles frame metadata automatically.

Color Space Constants

SWS_CS_ITU709      // ITU-R BT.709 (HD)
SWS_CS_FCC         // FCC
SWS_CS_ITU601      // ITU-R BT.601 (SD)
SWS_CS_ITU624      // ITU-R BT.624
SWS_CS_SMPTE170M   // SMPTE 170M
SWS_CS_SMPTE240M   // SMPTE 240M
SWS_CS_DEFAULT     // Default (BT.601)
SWS_CS_BT2020      // ITU-R BT.2020 (UHD)

Core Libraries

API Examples

​Purpose and Capabilities

​Key Concepts

​SwsContext

​Pixel Formats

​Scaling Algorithms

​Core Functions

​Context Management

​sws_getContext()

​sws_getCachedContext()

​sws_freeContext()

​sws_free_context()

​Modern Frame API

​sws_scale_frame()

​sws_frame_setup()

​sws_is_noop()

​Legacy Scaling API

​sws_scale()

​Format Testing

​sws_test_format()

​sws_isSupportedInput()

​sws_isSupportedOutput()

​Color Space Functions

​sws_getCoefficients()

​sws_setColorspaceDetails()

​Scaling Flags

​Algorithm Flags

​Additional Flags

​Usage Patterns

​Simple Scaling

​Color Space Conversion

​Batch Processing with Cached Context

​Thumbnail Generation

​Performance Optimization

​Common Pixel Formats

​YUV Formats

​RGB Formats

​Best Practices

​Color Space Constants

​See Also

Build docs developers (and LLMs) love

Purpose and Capabilities

Key Concepts

SwsContext

Pixel Formats

Scaling Algorithms

Core Functions

Context Management

sws_getContext()

sws_getCachedContext()

sws_freeContext()

sws_free_context()

Modern Frame API

sws_scale_frame()

sws_frame_setup()

sws_is_noop()

Legacy Scaling API

sws_scale()

Format Testing

sws_test_format()

sws_isSupportedInput()

sws_isSupportedOutput()

Color Space Functions

sws_getCoefficients()

sws_setColorspaceDetails()

Scaling Flags

Algorithm Flags

Additional Flags

Usage Patterns

Simple Scaling

Color Space Conversion

Batch Processing with Cached Context

Thumbnail Generation

Performance Optimization

Common Pixel Formats

YUV Formats

RGB Formats

Best Practices

Color Space Constants

See Also