Skip to main content

​
Overview

Adaptive worker management automatically adjusts the number of concurrent video conversions based on real-time CPU and memory usage. This prevents system overload during intensive encoding operations while maximizing throughput when resources are available.
Adaptive workers only affect video conversions. Image conversions use the standard --jobs setting.

​
How It Works

The adaptive system continuously monitors system resources and adjusts worker limits dynamically:
  1. Resource Monitor - Samples CPU and memory every 3 seconds (configurable)
  2. Threshold Evaluation - Compares metrics against configured thresholds
  3. Worker Adjustment - Increases or decreases concurrent video workers
  4. Graceful Scaling - Never interrupts in-flight conversions

​
Scaling Logic

Scale Down (immediate, after 1 measurement):
  • CPU usage ≥ cpu_high threshold (default: 80%)
  • Available memory ≤ mem_low_percent threshold (default: 20%)
  • Reduces workers by 1, down to min limit
Scale Up (conservative, after 2 consecutive measurements):
  • CPU usage ≤ cpu_low threshold (default: 50%)
  • Available memory ≥ mem_low_percent + 5% (default: 25%)
  • Increases workers by 1, up to max limit
The system uses streak counters to prevent oscillation. Scale-down is aggressive (1 sample) to protect system stability, while scale-up is conservative (2 samples) to ensure sustained capacity.

​
Configuration

​
Command-Line Flags

​
--adaptive-workers
boolean
default:"false"
Enable adaptive worker management for video conversions
​
--adaptive-workers-min
integer
default:"1"
Minimum concurrent video conversions (floor limit)
​
--adaptive-workers-max
integer
default:"6"
Maximum concurrent video conversions (ceiling limit)
​
--adaptive-workers-cpu-high
float
default:"80.0"
CPU utilization percentage threshold to reduce workers (0-100)
​
--adaptive-workers-cpu-low
float
default:"50.0"
CPU utilization percentage threshold to increase workers (0-100)
​
--adaptive-workers-mem-low
float
default:"20.0"
Minimum available memory percentage before reducing workers (0-100)
​
--adaptive-workers-interval
integer
default:"3"
Seconds between resource monitoring checks

​
YAML Configuration

~/.media-converter.yaml
adaptive_workers:
  enabled: true
  min: 2
  max: 8
  cpu_high: 85.0
  cpu_low: 60.0
  mem_low_percent: 15.0
  interval_seconds: 5

​
Platform-Specific Monitoring

Resource monitoring uses lightweight OS-specific implementations:

​
macOS (Darwin)

CPU Monitoring (resource_darwin.go:13-38):
  • Reads system load average via sysctl -n vm.loadavg
  • Normalizes by CPU count: load / NumCPU() * 100
Memory Monitoring (resource_darwin.go:40-104):
  • Parses vm_stat output for free, inactive, and speculative pages
  • Calculates available memory: (free + inactive + speculative) * pageSize
  • Reads total memory from sysctl -n hw.memsize

​
Linux

CPU Monitoring (resource_linux.go:14-36):
  • Reads 1-minute load average from /proc/loadavg
  • Normalizes by CPU count: load / NumCPU() * 100
Memory Monitoring (resource_linux.go:38-79):
  • Parses /proc/meminfo for MemTotal and MemAvailable
  • Calculates percentage: (MemAvailable / MemTotal) * 100
If resource metrics fail (unsupported platform, permission errors), the system logs a warning once and continues with fixed worker limits.

​
Examples

​
Conservative Server Configuration

Ideal for shared servers or systems running other services: 
media-converter /source /dest \
  --adaptive-workers \
  --adaptive-workers-min 1 \
  --adaptive-workers-max 3 \
  --adaptive-workers-cpu-high 70.0 \
  --adaptive-workers-cpu-low 40.0 \
  --adaptive-workers-mem-low 25.0

​
Aggressive Desktop Configuration

Maximize throughput on dedicated workstations: 
media-converter /source /dest \
  --adaptive-workers \
  --adaptive-workers-min 3 \
  --adaptive-workers-max 12 \
  --adaptive-workers-cpu-high 90.0 \
  --adaptive-workers-cpu-low 65.0 \
  --adaptive-workers-mem-low 10.0
Good default for most scenarios:
~/.media-converter.yaml
adaptive_workers:
  enabled: true
  min: 2
  max: 6
  cpu_high: 80.0
  cpu_low: 50.0
  mem_low_percent: 20.0
  interval_seconds: 3

​
Monitoring Output

The converter logs worker adjustments in real-time: 
Adaptive workers enabled: starting with 2 concurrent video conversions (min=2, max=6)
Adaptive workers: increasing video concurrency to 3 (CPU 45.2%, free memory 38.1%)
Adaptive workers: reducing video concurrency to 2 (CPU 82.5%, free memory 22.3%)
Worker adjustments appear as WARN level messages to ensure visibility during batch processing.

​
Performance Considerations

​
When to Enable

Good Use Cases:
  • Large video batches with varying file sizes/complexities
  • Shared systems where other processes may compete for resources
  • Systems with limited memory relative to video file sizes
  • Long-running conversions where system load fluctuates
Not Recommended:
  • Small batches (< 10 videos) where overhead exceeds benefit
  • Dedicated conversion servers with predictable workloads
  • When using hardware acceleration (already resource-efficient)

​
Tuning Guidelines

  1. Set realistic bounds - max should not exceed physical cores
  2. Leave headroom - Set cpu_high to 80-85% for system responsiveness
  3. Monitor first - Run with defaults, then adjust based on observed behavior
  4. Balance thresholds - Keep cpu_low ≤ 60% of cpu_high to prevent oscillation

​
Implementation Details

The adaptive system consists of three components (adaptive_workers.go):

​
ResourceMonitor (adaptive_workers.go:23-106)

Periodically samples system metrics:
  • Runs in background goroutine with configurable interval
  • Emits ResourceSnapshot structs on channel
  • Logs failures once per metric type to avoid spam
  • Drops samples if consumer is behind (non-blocking)

​
AdaptiveLimiter (adaptive_limiter.go:7-79)

Thread-safe semaphore with dynamic limit adjustment:
  • Acquire() - Blocks until worker slot available
  • Release() - Frees worker slot and signals waiters
  • SetLimit(n) - Updates concurrency limit without interrupting active workers
  • Uses sync.Cond for efficient wait/wake coordination

​
Adaptive Controller (adaptive_workers.go:110-187)

Decision engine that adjusts limiter based on snapshots:
  • Maintains highStreak and lowStreak counters
  • Applies hysteresis to prevent rapid oscillation
  • Respects configured min/max bounds
  • Logs all limit changes with current metrics
The limiter never interrupts in-flight conversions. When reducing the limit, it simply prevents new workers from starting until active count drops below the new threshold.

​
Troubleshooting

​
Metrics Unavailable Warnings

Adaptive workers: CPU metrics unavailable (exec: "sysctl": not found)
Cause: Platform not supported or system utilities missing Solution: Adaptive mode falls back to fixed limit (max). Install required utilities or disable adaptive mode.

​
No Worker Adjustments

Check:
  1. Verify --adaptive-workers flag is enabled
  2. Ensure thresholds are reachable (e.g., cpu_low < cpu_high)
  3. Confirm resource metrics are being collected (no warnings in logs)
  4. Increase logging verbosity to see snapshot values

​
Excessive Oscillation

Symptoms: Workers constantly increasing/decreasing Solutions:
  • Widen threshold gap: Reduce cpu_low or increase cpu_high
  • Increase interval_seconds for less frequent adjustments
  • Raise min to prevent dropping too low

Performance Tuning

General performance optimization settings

Hardware Acceleration

GPU-based video encoding for efficiency

Build docs developers (and LLMs) love

Get started for freeTalk to us