Speech & Audio Overview

​

Core Features

​

Speech to Text (STT)

• Multilingual support: 96+ languages with automatic language detection
• Streaming transcription: Real-time audio processing with committed and non-committed results
• Word-level timestamps: Detailed timing information when verbose mode is enabled
• Audio format: Requires 16kHz mono audio as Float32Array

​

Text to Speech (TTS)

• Multiple voices: English (US and GB) with customizable voice embeddings
• Speed control: Adjustable speech rate for different use cases
• Streaming mode: Start playback before full synthesis completes
• Audio output: Returns 22kHz mono audio as Float32Array

​

Voice Activity Detection (VAD)

• Segment detection: Identifies start and end times of speech activity
• Low latency: Fast on-device processing for real-time applications
• Audio format: Requires 16kHz mono audio as Float32Array
• Timestamp precision: Returns segments in seconds

​

Audio Format Requirements

• Sample rate: 16kHz (except TTS output which is 22kHz)
• Channels: Mono (single channel)
• Data type: Float32Array with values normalized between -1.0 and 1.0
• Buffer format: Contiguous samples in time order

​

Common Use Cases

​

Best Practices

​

Audio Preprocessing

• Always resample audio to 16kHz before processing
• Convert stereo audio to mono by averaging channels
• Normalize audio samples to the range [-1.0, 1.0]
• Remove DC offset and apply appropriate filtering

​

Memory Management

• Reuse Float32Array buffers when possible to reduce allocations
• Process audio in chunks for long recordings
• Clean up resources when components unmount
• Monitor download progress for large model files

​

Performance Optimization

• Use streaming modes for real-time requirements
• Batch audio processing when latency is not critical
• Leverage VAD to process only speech segments
• Cache models to avoid repeated downloads

​

Model Downloads

import { useSpeechToText } from 'react-native-executorch';

const { isReady, downloadProgress } = useSpeechToText({
  model: {
    isMultilingual: true,
    encoderSource: 'https://example.com/whisper-encoder.pte',
    decoderSource: 'https://example.com/whisper-decoder.pte',
    tokenizerSource: 'https://example.com/whisper-tokenizer.json',
  },
});

// Monitor download progress
console.log(`Download: ${(downloadProgress * 100).toFixed(0)}%`);

​

Error Handling

const { error, isReady } = useSpeechToText({ model });

if (error) {
  console.error('Failed to load model:', error.message);
  // Handle specific error codes
  switch (error.code) {
    case 'MODULE_NOT_LOADED':
      // Model not ready yet
      break;
    case 'DOWNLOAD_INTERRUPTED':
      // Network issue during download
      break;
    // ... handle other cases
  }
}

​

Next Steps