Performance Tuning - Camera Workflow

Overview

Camera Workflow is designed for high-throughput parallel processing. This guide covers performance optimization techniques, resource management, and tuning guidelines for different hardware configurations.

Quick Wins

1. Enable Hardware Acceleration

Dramatic speedup for video conversions (macOS):

media-converter /source /dest --video-acceleration

Impact: 3-5x faster video encoding, 70-80% less CPU usage

2. Optimize Worker Count

Balance parallelism with system resources:

# Auto-detect optimal workers (CPU cores - 2)
media-converter /source /dest

# Manual override
media-converter /source /dest --jobs 8

Guidelines:

Image-heavy batches: jobs = CPU cores (images are less CPU-intensive)
Video-heavy batches: jobs = CPU cores / 2 (videos saturate CPU)
Mixed batches: jobs = CPU cores - 2 (default, balanced)

3. Use Adaptive Workers for Videos

Automatically scale video concurrency based on system load:

media-converter /source /dest \
  --adaptive-workers \
  --adaptive-workers-max 6

Impact: Prevents system overload while maximizing throughput

Hardware-Specific Tuning

Desktop Workstation (16+ cores, 32+ GB RAM)

Goal: Maximum throughput

~/.media-converter.yaml

max_jobs: 14                    # Leave 2 cores for system
video_acceleration: true         # Use GPU encoding
photo_format: avif              # Best compression
photo_quality_avif: 75          # Lower quality = faster encoding

adaptive_workers:
  enabled: true
  min: 4
  max: 12
  cpu_high: 90.0
  cpu_low: 65.0
  mem_low_percent: 15.0

Expected Performance:

Images: 20-40 files/second (JPEG → AVIF)
Videos: 2-6 concurrent conversions
Bottleneck: Disk I/O or GPU encoder

Laptop (4-8 cores, 8-16 GB RAM)

Goal: Balance performance with thermal/battery constraints

~/.media-converter.yaml

max_jobs: 4                      # Conservative parallelism
video_acceleration: true         # Reduce CPU load
photo_quality_avif: 80          # Better quality at slightly slower speed

adaptive_workers:
  enabled: true
  min: 1
  max: 3
  cpu_high: 75.0                # Back off earlier to prevent throttling
  cpu_low: 45.0
  mem_low_percent: 25.0         # More aggressive memory protection

Expected Performance:

Images: 8-15 files/second
Videos: 1-3 concurrent conversions
Bottleneck: Thermal throttling

Server (32+ cores, 64+ GB RAM, No GPU)

Goal: Maximize software encoding throughput

~/.media-converter.yaml

max_jobs: 30                     # High parallelism for images
video_acceleration: false        # No GPU available
video_codec: h265
video_crf: 28

adaptive_workers:
  enabled: true
  min: 8
  max: 16                       # More workers than desktop (no GPU)
  cpu_high: 85.0
  cpu_low: 60.0
  mem_low_percent: 20.0
  interval_seconds: 5           # Less frequent checks

Expected Performance:

Images: 40-80 files/second
Videos: 8-16 concurrent conversions
Bottleneck: CPU, possibly disk I/O

Format-Specific Optimization

AVIF (Best Compression, Slower)

Characteristics:

Encoding: 2-3x slower than WebP
File size: 30-50% smaller than WebP
Best for: Archival, web delivery, bandwidth-limited scenarios

Tuning:

photo_format: avif
photo_quality_avif: 75          # Lower quality = faster encoding
max_jobs: 8                     # AVIF benefits from parallelism

Speed vs Quality:

Quality	Speed	File Size	Visual Quality
70	1.0x	100%	Excellent
80	0.7x	130%	Near-lossless
90	0.4x	180%	Lossless-like

WebP (Balanced)

Characteristics:

Encoding: Fast (baseline for comparison)
File size: Good compression (60-70% of JPEG)
Best for: General use, compatibility with older browsers

Tuning:

photo_format: webp
photo_quality_webp: 85
max_jobs: 12                    # WebP scales well with parallelism

Video Codec Comparison

Codec	Encoding Speed	Compression	Compatibility	Best Use Case
H.264 (libx264)	1.5x	Good	Universal	Compatibility
H.265 (libx265)	1.0x	Excellent	Modern devices	Balance
H.265 (VideoToolbox)	5.0x	Excellent	Modern devices	Speed (macOS)
AV1 (libaom-av1)	0.1x	Best	Limited	Archival only

AV1 encoding is 5-10x slower than H.265 software encoding. Only use for final archival encodes where maximum compression is critical.

Disk I/O Optimization

Storage Types

Performance Hierarchy:

NVMe SSD: 3000-7000 MB/s (best for working directory)
SATA SSD: 500-600 MB/s (good for source or destination)
HDD (7200 RPM): 150-200 MB/s (acceptable for archival destination)
Network (Gigabit): 100-125 MB/s (bottleneck risk)
Network (100 Mbps): 10-12 MB/s (severe bottleneck)

Optimal Storage Configuration

Best Performance:

# Source and destination on separate fast SSDs
media-converter /mnt/ssd1/source /mnt/ssd2/dest --jobs 12

Budget Friendly:

# Source on HDD, destination on SSD (write-heavy workload)
media-converter /mnt/hdd/source /mnt/ssd/dest --jobs 8

Network Storage (use with caution):

# Reduce parallelism to avoid saturating network
media-converter /nfs/source /nfs/dest --jobs 4

Temporary File Location

Conversions write to <output>.tmp before atomic rename. Ensure destination has:

Free space: 50% of source directory size (safety margin)
Write speed: Sufficient for concurrent conversions
Low latency: Local/direct-attached storage preferred

Memory Management

Memory Requirements

Per-Worker Estimates:

Image conversion: 100-300 MB (JPEG/HEIC)
Image conversion: 500-800 MB (RAW files, 50+ MP)
Video conversion (hardware): 200-400 MB
Video conversion (software): 500-1000 MB

Calculation:

Required RAM = (max_jobs × avg_worker_memory) + 2 GB (system overhead)

Example (8 jobs, mixed workload):
  = (8 × 600 MB) + 2 GB
  = 4.8 GB + 2 GB
  = 6.8 GB minimum

Adaptive Worker Memory Protection

Prevent OOM conditions with adaptive workers:

adaptive_workers:
  enabled: true
  mem_low_percent: 20.0         # Reduce workers when < 20% memory free

Behavior:

Monitors available memory every 3 seconds
Reduces video workers if free memory drops below threshold
Prevents new workers from starting (doesn’t kill existing)

Image conversions are not managed by adaptive workers. Set --jobs conservatively if processing large RAW files.

Monitoring Performance

Real-Time Metrics

The converter logs progress and resource adjustments:

Adaptive workers: increasing video concurrency to 4 (CPU 52.3%, free memory 42.1%)
📸 Processing images: [████████████████████] 1847/1847 (100%)
📹 vacation.mov: [██████████████░░░░░░░░] 68.4% (2.3x, ETA: 00:42)

Performance Bottleneck Detection

CPU Bottleneck:

Symptoms:
  - CPU at 95-100% sustained
  - Video encoding speed < 1.0x
  - Adaptive workers frequently reducing concurrency

Solutions:
  - Enable hardware acceleration
  - Reduce --jobs count
  - Lower video quality (increase CRF)

Memory Bottleneck:

Symptoms:
  - Frequent adaptive worker reductions
  - System swap usage increasing
  - Conversion timeouts

Solutions:
  - Reduce --jobs count
  - Enable adaptive workers with higher mem_low_percent
  - Process in smaller batches

Disk I/O Bottleneck:

Symptoms:
  - CPU usage < 70%
  - Low encoding speed despite available resources
  - iostat shows disk at 100% utilization

Solutions:
  - Move source/destination to faster storage
  - Reduce parallelism (--jobs) to avoid I/O contention
  - Use separate physical drives for source and destination

Benchmark Examples

Test Dataset

1000 JPEG images (12-24 MP, 3-8 MB each)
50 videos (1080p-4K, 100-500 MB each)
Total: ~25 GB

Results (M2 MacBook Pro, 10 cores, 16 GB RAM)

Configuration 1: Defaults

media-converter /source /dest

Jobs: 8 (auto-detected)
Video acceleration: Enabled
Adaptive workers: Disabled
Time: 18 minutes
Peak CPU: 95%
Peak Memory: 8 GB

Configuration 2: Optimized

media-converter /source /dest \
  --jobs 10 \
  --adaptive-workers \
  --adaptive-workers-max 6

Jobs: 10 (images)
Video acceleration: Enabled
Adaptive workers: 2-6 (dynamic)
Time: 14 minutes
Peak CPU: 85%
Peak Memory: 7 GB

Configuration 3: Quality Priority

media-converter /source /dest \
  --video-acceleration=false \
  --video-crf 20 \
  --photo-quality-avif 85

Jobs: 8
Video acceleration: Disabled
Time: 52 minutes
Peak CPU: 100%
Peak Memory: 12 GB
Output size: 15% larger, visually superior

Advanced Tuning

CPU Affinity (Linux)

Pin conversions to specific CPU cores:

taskset -c 0-11 media-converter /source /dest --jobs 12

Useful for NUMA systems or isolating workloads.

Process Priority

Reduce system impact of long conversions:

# Linux/macOS
nice -n 10 media-converter /source /dest

# Windows (PowerShell)
Start-Process media-converter -ArgumentList "/source","/dest" -Priority BelowNormal

I/O Scheduling (Linux)

Prioritize interactive system responsiveness:

ionice -c 3 media-converter /source /dest

Configuration Templates

Maximum Speed (Desktop)

~/.media-converter.yaml

max_jobs: 12
video_acceleration: true
photo_format: webp              # Faster than AVIF
photo_quality_webp: 80
video_codec: h265
video_crf: 30                   # Lower quality = faster

adaptive_workers:
  enabled: true
  min: 4
  max: 10
  cpu_high: 95.0
  cpu_low: 70.0

Maximum Quality (Archival)

~/.media-converter.yaml

max_jobs: 6                     # Slower but higher quality
video_acceleration: false       # Software encoding for CRF control
photo_format: avif
photo_quality_avif: 90
video_codec: h265
video_crf: 18                   # Near-lossless

adaptive_workers:
  enabled: false                # Fixed concurrency for consistency

Balanced (Recommended)

~/.media-converter.yaml

max_jobs: 8
video_acceleration: true
photo_format: avif
photo_quality_avif: 80
video_codec: h265
video_crf: 28

adaptive_workers:
  enabled: true
  min: 2
  max: 6
  cpu_high: 80.0
  cpu_low: 50.0
  mem_low_percent: 20.0

Troubleshooting

Slow Conversion Speed

Check hardware acceleration status:

# Look for this in logs:
"Using hardware acceleration: VideoToolbox HEVC"

Monitor resource usage:

# macOS
top -l 1 | grep media-converter

# Linux
htop

Test with single worker:
```
media-converter /source /dest --jobs 1
```
If single-threaded is slow, issue is encoding efficiency, not parallelism.

System Unresponsive During Conversion

Reduce resource consumption:

nice -n 10 media-converter /source /dest \
  --jobs 4 \
  --adaptive-workers \
  --adaptive-workers-max 3 \
  --adaptive-workers-cpu-high 70.0

Out of Memory Errors

Solutions:

Enable adaptive workers with aggressive memory protection:

adaptive_workers:
  enabled: true
  mem_low_percent: 30.0

Reduce parallelism:
```
media-converter /source /dest --jobs 4
```

Process in batches:

# Split source into subdirectories and process sequentially
for dir in /source/batch_*; do
  media-converter "$dir" /dest
done

Adaptive Workers

Automatic resource-based concurrency scaling

Hardware Acceleration

GPU-based video encoding for efficiency

Getting Started

CLI Reference

Core Features

Raycast Extension

Advanced

Development

​Overview

​Quick Wins

​1. Enable Hardware Acceleration

​2. Optimize Worker Count

​3. Use Adaptive Workers for Videos

​Hardware-Specific Tuning

​Desktop Workstation (16+ cores, 32+ GB RAM)

​Laptop (4-8 cores, 8-16 GB RAM)

​Server (32+ cores, 64+ GB RAM, No GPU)

​Format-Specific Optimization

​AVIF (Best Compression, Slower)

​WebP (Balanced)

​Video Codec Comparison

​Disk I/O Optimization

​Storage Types

​Optimal Storage Configuration

​Temporary File Location

​Memory Management

​Memory Requirements

​Adaptive Worker Memory Protection

​Monitoring Performance

​Real-Time Metrics

​Performance Bottleneck Detection

​Benchmark Examples

​Test Dataset

​Results (M2 MacBook Pro, 10 cores, 16 GB RAM)

​Advanced Tuning

​CPU Affinity (Linux)

​Process Priority

​I/O Scheduling (Linux)

​Configuration Templates

​Maximum Speed (Desktop)

​Maximum Quality (Archival)

​Balanced (Recommended)

​Troubleshooting

​Slow Conversion Speed

​System Unresponsive During Conversion

​Out of Memory Errors

​Related Configuration

Adaptive Workers

Hardware Acceleration

Build docs developers (and LLMs) love

Overview

Quick Wins

1. Enable Hardware Acceleration

2. Optimize Worker Count

3. Use Adaptive Workers for Videos

Hardware-Specific Tuning

Desktop Workstation (16+ cores, 32+ GB RAM)

Laptop (4-8 cores, 8-16 GB RAM)

Server (32+ cores, 64+ GB RAM, No GPU)

Format-Specific Optimization

AVIF (Best Compression, Slower)

WebP (Balanced)

Video Codec Comparison

Disk I/O Optimization

Storage Types

Optimal Storage Configuration

Temporary File Location

Memory Management

Memory Requirements

Adaptive Worker Memory Protection

Monitoring Performance

Real-Time Metrics

Performance Bottleneck Detection

Benchmark Examples

Test Dataset

Results (M2 MacBook Pro, 10 cores, 16 GB RAM)

Advanced Tuning

CPU Affinity (Linux)

Process Priority

I/O Scheduling (Linux)

Configuration Templates

Maximum Speed (Desktop)

Maximum Quality (Archival)

Balanced (Recommended)

Troubleshooting

Slow Conversion Speed

System Unresponsive During Conversion

Out of Memory Errors

Related Configuration