Model Configurations

Overview

Omnilingual ASR uses several configuration classes to define model architecture, beam search, and streaming behavior.

Wav2Vec2LlamaConfig

Top-level configuration combining Wav2Vec2 encoder and Llama decoder settings.

Structure

wav2vec2_asr_config

Wav2Vec2AsrConfig

required

Wav2Vec2 configuration for encoder frontend and encoder.

llama_config

LLaMAConfig

required

Llama configuration for decoder.

beam_search_config

Wav2Vec2LlamaBeamSearchConfig

default:"Wav2Vec2LlamaBeamSearchConfig()"

Beam search configuration for LLM-ASR decoding.

streaming_config

Wav2Vec2LlamaStreamingConfig

default:"Wav2Vec2LlamaStreamingConfig()"

Streaming configuration for >30s transcriptions.

encoder_stacking

int

default:"1"

Number of audio embedding frames to stack before decoder.

frozen_encoder

int

default:"1"

Encoder freeze setting:

0: Frozen
1: Unfrozen
N > 1: Unfrozen every N calls

lang_embeddings_p

float

default:"0.0"

Probability of dropping language embeddings during training.

language_column_name

str

default:"lang"

Column name containing language information in batch metadata.

context_text_only

bool

default:"False"

Adapts model input syntax for text-only context (instead of audio+text).

n_special_tokens

int

default:"0"

Number of additional special tokens to allocate in vocab-embedding mapping.

model_type

ModelType

default:"ModelType.LLM_ASR"

High-level model type:

ModelType.LLM_ASR: Standard ASR
ModelType.LLM_ASR_LID: ASR with language ID
ModelType.ZERO_SHOT: Zero-shot with context

n_context_examples

int

default:"0"

Number of context examples for zero-shot model.

Vocabulary Parameters

unk_idx

int

default:"3"

Index of UNK token in vocabulary.

bos_idx

int

default:"0"

Index of BOS (beginning of sequence) token.

eos_idx

int

default:"2"

Index of EOS (end of sequence) token.

pad_idx

int

default:"1"

Index of PAD token. Must align with Llama’s pad_idx.

Example

from fairseq2.runtime.config_registry import get_config
from omnilingual_asr.models.wav2vec2_llama.config import Wav2Vec2LlamaConfig

# Load predefined config
config = get_config(Wav2Vec2LlamaConfig, "7b")

print(f"Model type: {config.model_type}")
print(f"Encoder stacking: {config.encoder_stacking}")
print(f"Lang embeddings p: {config.lang_embeddings_p}")
print(f"Vocab size: {config.llama_config.vocab_size}")

Wav2Vec2LlamaBeamSearchConfig

Configuration for beam search decoding.

Parameters

nbest

int

default:"5"

Size of the beam (number of hypotheses to maintain).

length_norm

bool

default:"False"

Whether to apply length normalization when computing hypothesis scores.

compression_window

int

default:"100"

Window size for early stopping detection. If the last N tokens compress at a ratio greater than compression_threshold, decoding stops.

compression_threshold

float

default:"4.0"

Compression ratio threshold for early stopping (used with compression_window).

Example

from omnilingual_asr.models.wav2vec2_llama.config import Wav2Vec2LlamaBeamSearchConfig
from omnilingual_asr.models.inference import ASRInferencePipeline

# Custom beam search config
beam_config = Wav2Vec2LlamaBeamSearchConfig(
    nbest=10,  # Larger beam
    length_norm=True,  # Enable length normalization
    compression_window=50,
    compression_threshold=3.0
)

# Use with pipeline
pipeline = ASRInferencePipeline(
    "omniASR_LLM_7B",
    beam_search_config=beam_config
)

Early Stopping

Early stopping prevents infinite loops on bad audio:

During decoding, track the last compression_window tokens
Compute compression ratio (e.g., using zlib)
If ratio > compression_threshold, stop decoding

Example: Repetitive output like “the the the the…” compresses well and triggers early stopping.

Wav2Vec2LlamaStreamingConfig

Configuration for streaming mode (unlimited audio length).

Parameters

is_streaming

bool

default:"False"

Enable streaming mode for >30s audio.

segment_secs

float

default:"15.0"

Duration of each audio segment in seconds.

sample_rate

int

default:"16000"

Audio sample rate in Hz.

n_context_segments

int

default:"1"

Number of context segments to maintain (in addition to current segment).

text_tokenizer

str

default:""

Name of text tokenizer for streaming mode.

min_audio_ms

int

default:"25"

Minimum audio length in milliseconds. Shorter segments are dropped.

Example

from omnilingual_asr.models.wav2vec2_llama.config import Wav2Vec2LlamaStreamingConfig

# Streaming config
streaming_config = Wav2Vec2LlamaStreamingConfig(
    is_streaming=True,
    segment_secs=20.0,  # 20-second segments
    sample_rate=16000,
    n_context_segments=2,  # Keep 2 previous segments as context
    text_tokenizer="omniASR_tokenizer_v2",
    min_audio_ms=50  # Drop segments <50ms
)

# Note: Streaming models are loaded with pre-configured streaming settings
# This is typically not set manually
from fairseq2.models.hub import load_model
model = load_model("omniASR_LLM_7B_Unlimited")
print(model.streaming_config.is_streaming)  # True

Streaming Processing

Segmentation: Audio is split into segment_secs chunks
Context: Previous n_context_segments are maintained for continuity
Transcription: Each segment is transcribed with context
Concatenation: Segment transcriptions are concatenated into final output

Wav2Vec2LlamaSpecialTokens

Special token indices allocated beyond vocabulary size.

Properties

All special tokens are allocated as vocab_size + offset:

Default/LID Syntax

lid_marker

int

Language ID marker token: vocab_size + 0

Streaming Syntax

streaming_lang

int

Streaming language token: vocab_size + 0

last_segment

int

Last segment marker: vocab_size + 1

regular_segment

int

Regular segment marker: vocab_size + 2

Context Syntax (Zero-Shot)

context_start

int

Context start marker: vocab_size + 0

context_end

int

Context end marker: vocab_size + 1

context_example_start

int

Context example start marker: vocab_size + 2

context_example_end

int

Context example end marker: vocab_size + 3

context_bos

int

Context BOS token: vocab_size + 4

context_eos

int

Context EOS token: vocab_size + 5

Example

from omnilingual_asr.models.wav2vec2_llama.config import Wav2Vec2LlamaSpecialTokens

vocab_size = 10000
special_tokens = Wav2Vec2LlamaSpecialTokens(vocab_size)

print(f"LID marker: {special_tokens.lid_marker}")  # 10000
print(f"Context start: {special_tokens.context_start}")  # 10000
print(f"Context BOS: {special_tokens.context_bos}")  # 10004

ModelType Enum

Defines the high-level model variant.

from enum import Enum

class ModelType(Enum):
    LLM_ASR = 1      # Standard encoder-decoder ASR
    LLM_ASR_LID = 2  # ASR with language identification
    ZERO_SHOT = 3    # Zero-shot with context examples

Usage

from omnilingual_asr.models.wav2vec2_llama.config import ModelType
from fairseq2.models.hub import load_model

model = load_model("omniASR_LLM_7B")
if model.model_type == ModelType.LLM_ASR_LID:
    print("Model supports language conditioning")
elif model.model_type == ModelType.ZERO_SHOT:
    print("Model requires context examples")

Predefined Configurations

Omnilingual ASR provides predefined configurations:

Config Name	Size	Type	Special Features
`300m`	300M	LLM_ASR_LID	Language conditioning
`1b`	1B	LLM_ASR_LID	Language conditioning
`3b`	3B	LLM_ASR_LID	Language conditioning
`7b`	7B	LLM_ASR_LID	Language conditioning
`7b_zs`	7B	ZERO_SHOT	Zero-shot, 10 context examples
`7b_unlimited_v2`	7B	LLM_ASR_LID	Streaming enabled

Loading Configs

from fairseq2.runtime.config_registry import get_config
from omnilingual_asr.models.wav2vec2_llama.config import Wav2Vec2LlamaConfig

# Load 7B config
config_7b = get_config(Wav2Vec2LlamaConfig, "7b")

# Load zero-shot config
config_zs = get_config(Wav2Vec2LlamaConfig, "7b_zs")
print(f"Context examples: {config_zs.n_context_examples}")  # 10

# Load streaming config  
config_stream = get_config(Wav2Vec2LlamaConfig, "7b_unlimited_v2")
print(f"Streaming: {config_stream.streaming_config.is_streaming}")  # True

Configuration Validation

Configs are validated in __post_init__:

# Vocabulary size must match
assert wav2vec2_asr_config.target_vocab_size == llama_config.vocab_size

# PAD index must match
assert pad_idx == llama_config.pad_idx

Complete Example

from fairseq2.models.llama import LLaMAConfig
from fairseq2.models.wav2vec2.asr import Wav2Vec2AsrConfig
from omnilingual_asr.models.wav2vec2_llama.config import (
    Wav2Vec2LlamaConfig,
    Wav2Vec2LlamaBeamSearchConfig,
    Wav2Vec2LlamaStreamingConfig,
    ModelType
)

# Create custom config
wav2vec2_config = Wav2Vec2AsrConfig(
    target_vocab_size=10000,
    # ... other wav2vec2 settings
)

llama_config = LLaMAConfig(
    model_dim=4096,
    vocab_size=10000,
    num_layers=12,
    num_attn_heads=32,
    # ... other llama settings
)

beam_search_config = Wav2Vec2LlamaBeamSearchConfig(
    nbest=5,
    length_norm=True
)

streaming_config = Wav2Vec2LlamaStreamingConfig(
    is_streaming=False
)

config = Wav2Vec2LlamaConfig(
    wav2vec2_asr_config=wav2vec2_config,
    llama_config=llama_config,
    beam_search_config=beam_search_config,
    streaming_config=streaming_config,
    model_type=ModelType.LLM_ASR_LID,
    encoder_stacking=1,
    lang_embeddings_p=0.5,
    n_special_tokens=1
)

print(f"Config created: {config.model_type}")

Source Reference

See implementation at src/omnilingual_asr/models/wav2vec2_llama/config.py

Core API

Datasets

Models

Overview

Wav2Vec2LlamaConfig

Structure

Vocabulary Parameters

Example

Wav2Vec2LlamaBeamSearchConfig

Parameters

Example

Early Stopping

Wav2Vec2LlamaStreamingConfig

Parameters

Example

Streaming Processing

Wav2Vec2LlamaSpecialTokens

Properties

Default/LID Syntax

Streaming Syntax

Context Syntax (Zero-Shot)

Example

ModelType Enum

Usage

Predefined Configurations

Loading Configs

Configuration Validation

Complete Example

Source Reference

Build docs developers (and LLMs) love

Core API

Datasets

Models

​Overview

​Wav2Vec2LlamaConfig

​Structure

​Vocabulary Parameters

​Example

​Wav2Vec2LlamaBeamSearchConfig

​Parameters

​Example

​Early Stopping

​Wav2Vec2LlamaStreamingConfig

​Parameters

​Example

​Streaming Processing

​Wav2Vec2LlamaSpecialTokens

​Properties

​Default/LID Syntax

​Streaming Syntax

​Context Syntax (Zero-Shot)

​Example

​ModelType Enum

​Usage

​Predefined Configurations

​Loading Configs

​Configuration Validation

​Complete Example

​Source Reference

Build docs developers (and LLMs) love

Overview

Wav2Vec2LlamaConfig

Structure

Vocabulary Parameters

Example

Wav2Vec2LlamaBeamSearchConfig

Parameters

Example

Early Stopping

Wav2Vec2LlamaStreamingConfig

Parameters

Example

Streaming Processing

Wav2Vec2LlamaSpecialTokens

Properties

Default/LID Syntax

Streaming Syntax

Context Syntax (Zero-Shot)

Example

ModelType Enum

Usage

Predefined Configurations

Loading Configs

Configuration Validation

Complete Example

Source Reference