Performance Benchmarks

​

Quick Comparison: Moonshine vs Whisper

• Moonshine Medium Streaming achieves better accuracy than Whisper Large v3 with 6x fewer parameters
• Moonshine models are 10-100x faster than equivalent Whisper models for real-time speech
• Moonshine Tiny Streaming runs in 34ms on a MacBook Pro, enabling sub-200ms total latency for voice interfaces
• All Moonshine models run efficiently on Raspberry Pi 5, while Whisper Large v3 cannot run on this platform

​

Understanding the Metrics

​

Word Error Rate (WER)

​

Latency (ms)

• Streaming models do most work while the user is talking
• Non-streaming models must process the entire segment after speech ends
• For responsive voice interfaces, target latency below 200ms

​

Compute Percentage

• 20% means the model uses 1/5 of CPU time, leaving 80% for your application
• 100% means the model uses all available CPU just to keep up with real-time audio
• Values over 100% mean the model cannot process audio in real-time

​

Benchmark Methodology

​

Test Setup

1. Loading a .wav audio file
2. Feeding it in chunks to the model (simulating real-time streaming)
3. Measuring absolute processing time
4. Calculating percentage of audio duration
5. Computing average response latency

​

Running Benchmarks

cd core
mkdir build
cd build
cmake ..
cmake --build . --config Release
./benchmark

./benchmark --model-path /path/to/model --model-arch 5

​

Adjusting Update Frequency

./benchmark --transcription-interval 0.3

​

Python Benchmark Script

• Automatically downloads models
• Evaluates both Moonshine and Whisper models
• Provides detailed latency and compute cost comparisons

​

Whisper Comparison Methodology

• Speech must be responded to quickly once a user completes a phrase
• Phrases range from 1-10 seconds in duration
• Latency on individual segments matters more than overall throughput

• Test file: two_cities.wav (mix of short and long phrases)
• Moonshine models: Tiny, Base, Tiny/Small/Medium Streaming
• Whisper models: Tiny, Base, Small, Large v3
• Comparison: Moonshine Medium Streaming vs Whisper Large v3 (both achieve sub-8% WER)
• VAD: Moonshine VAD segmenter splits audio into phrases
• Platform: CPU only (using faster-whisper for best cross-platform performance)

1. Response Latency: Time from phrase completion (VAD detection) to transcribed text
   • Whisper: Full transcription time for each segment
   • Moonshine Streaming: Minimal time (most work done during speech)
2. Compute Cost: Total audio processing time as percentage of audio duration
   • Inverse of the Real-Time Factor (RTF) metric
   • Reflects actual CPU load for real-time applications

​

Why Not Whisper for Live Speech?

​

Fixed 30-Second Input Window

• Voice interface phrases are typically 1-10 seconds
• Remaining 20+ seconds are zero padding
• Wasted computation encoding empty input
• Increased latency even on high-end hardware

​

No Caching

• Voice interfaces need to display feedback while the user talks
• This requires repeated transcription calls as speech continues
• Whisper starts from scratch each time, repeating work on unchanged audio
• Moonshine caches encoder output and decoder state for dramatic speedup

​

Poor Multilingual Support

• 82 languages listed
• Only 33 languages achieve sub-20% WER (usable quality)
• For Base model (common on edge devices): only 5 languages under 20% WER
• Languages like Korean and Japanese have poor accuracy despite large markets

​

Fragmented Edge Support

• Desktop-focused frameworks with mature ecosystems
• Inconsistent interfaces and capabilities across iOS, Android, Raspberry Pi
• Difficult to build applications that run on multiple platforms

​

Platform-Specific Notes

​

MacBook Pro

​

Linux x86

​

Raspberry Pi 5

• All models run efficiently on the device
• Tiny Streaming achieves 237ms latency (suitable for most voice interfaces)
• Whisper Tiny is 24x slower (5,863ms vs 237ms)
• Whisper Large v3 cannot run on this platform

​

Custom Benchmarking

python scripts/run-benchmarks.py --wav_path /path/to/your/audio.wav