Performance Optimization

​

FlashAttention 2

​

Installation

pip install -U flash-attn --no-build-isolation

MAX_JOBS=4 pip install -U flash-attn --no-build-isolation

​

Hardware Requirements

• GPU Architecture: Ampere (A100, A10) or newer (H100, L40)
• Compute Capability: 8.0+ (Ampere) or 9.0+ (Hopper)
• Data Types: Only works with torch.float16 or torch.bfloat16

​

Usage

import torch
from qwen_tts import Qwen3TTSModel

model = Qwen3TTSModel.from_pretrained(
    "Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice",
    device_map="cuda:0",
    dtype=torch.bfloat16,  # Required: bfloat16 or float16
    attn_implementation="flash_attention_2",  # Enable FlashAttention 2
)

​

Performance Impact

​

Batch Processing

​

Batch Generation Example

import torch
from qwen_tts import Qwen3TTSModel

model = Qwen3TTSModel.from_pretrained(
    "Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice",
    device_map="cuda:0",
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)

# Batch inference
wavs, sr = model.generate_custom_voice(
    text=[
        "其实我真的有发现，我是一个特别善于观察别人情绪的人。",
        "She said she would be here by noon.",
        "The quick brown fox jumps over the lazy dog.",
    ],
    language=["Chinese", "English", "English"],
    speaker=["Vivian", "Ryan", "Aiden"],
    instruct=["", "Very happy.", "Speak slowly and clearly."]
)

# Save outputs
import soundfile as sf
for i, wav in enumerate(wavs):
    sf.write(f"output_{i}.wav", wav, sr)

​

Batch Size Guidelines

​

Throughput Gains

​

Memory Optimization

​

Data Type Selection

​

bfloat16 (Recommended)

model = Qwen3TTSModel.from_pretrained(
    "Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice",
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)

• Compatible with FlashAttention 2
• Better numerical stability than float16
• Native support on modern GPUs (Ampere, Hopper)
• Minimal quality loss

• Requires Ampere or newer GPUs

​

float16

model = Qwen3TTSModel.from_pretrained(
    "Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice",
    dtype=torch.float16,
    attn_implementation="flash_attention_2",
)

• Compatible with FlashAttention 2
• Wide GPU support (Pascal, Turing, Ampere, Hopper)
• Lower memory usage

• Less numerical stability than bfloat16
• Potential for overflow/underflow in some cases

​

float32

model = Qwen3TTSModel.from_pretrained(
    "Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice",
    dtype=torch.float32,
)

• Maximum numerical precision
• Universal GPU support

• 2x memory usage vs. float16/bfloat16
• Slower inference
• Cannot use FlashAttention 2

​

Memory Comparison

​

Gradient Checkpointing (Training)

# In sft_12hz.py, the Accelerator already uses mixed precision
accelerator = Accelerator(
    gradient_accumulation_steps=4,
    mixed_precision="bf16",  # Uses bfloat16 for training
    log_with="tensorboard"
)

• Standard training: ~12-16GB for 1.7B model
• With gradient accumulation (4 steps): Effective batch size 4x larger with same memory

​

GPU Selection

​

Consumer GPUs

​

Data Center GPUs

​

Cloud GPU Recommendations

​

Streaming vs Non-Streaming

​

Non-Streaming (Default)

wavs, sr = model.generate_custom_voice(
    text="The quick brown fox jumps over the lazy dog.",
    language="English",
    speaker="Ryan",
)
# wavs[0] contains the complete audio

• Simpler to use
• Better for batch processing
• Easier error handling

• Higher latency (wait for complete generation)
• Not suitable for real-time interaction

​

Streaming

• First audio packet: Output immediately after a single character input
• End-to-end latency: As low as 97ms
• Real-time factor: Can generate audio faster than real-time playback

• Real-time conversation systems
• Interactive voice response (IVR)
• Live audiobook narration
• Low-latency TTS applications

​

Generation Parameters

​

max_new_tokens

wavs, sr = model.generate_custom_voice(
    text="Your text here",
    language="English",
    speaker="Ryan",
    max_new_tokens=2048,  # Maximum audio length in tokens
)

• Default: 2048 (recommended)
• Lower values: Faster, but may truncate long audio
• Higher values: Allow longer generation, but slower

​

Sampling Parameters

wavs, sr = model.generate_custom_voice(
    text="Your text here",
    language="English",
    speaker="Ryan",
    temperature=0.7,  # Higher = more random/varied
    top_p=0.9,        # Nucleus sampling
    top_k=50,         # Top-k sampling
)

• temperature: 0.6-0.8 for natural variation, 0.3-0.5 for consistent output
• top_p: 0.9-0.95 for balanced quality
• top_k: 50 (default is usually good)

​

Evaluation Benchmark

model.generate_custom_voice(
    text=test_text,
    language="auto",  # or explicit language
    dtype=torch.bfloat16,
    max_new_tokens=2048,
    # All other parameters at defaults
)

• Seed-TTS test-zh: 0.77 WER (1.7B model)
• Seed-TTS test-en: 1.24 WER (1.7B model)

​

Performance Checklist

• Install FlashAttention 2
• Use dtype=torch.bfloat16 (or torch.float16)
• Enable attn_implementation="flash_attention_2"
• Use batch processing when possible
• Select appropriate GPU with sufficient VRAM
• Tune batch size based on GPU memory
• Use streaming for low-latency applications
• Set max_new_tokens=2048 for standard use cases
• Profile your specific workload to find optimal settings

​

Troubleshooting

​

Out of Memory (OOM)

1. Reduce batch size
2. Switch to smaller model (1.7B → 0.6B)
3. Use bfloat16/float16 instead of float32
4. Enable FlashAttention 2
5. Free up GPU memory from other processes

​

Slow Generation

1. Install FlashAttention 2
2. Enable attn_implementation="flash_attention_2"
3. Use bfloat16 instead of float32
4. Increase batch size for throughput
5. Consider vLLM-Omni for production workloads

​

Quality Issues

1. Avoid too-low temperature (less than 0.3)
2. Use bfloat16 for better stability
3. Ensure input text is clean and well-formatted
4. Check that language is correctly specified

​

Next Steps

• Deploy with vLLM integration for production
• Use DashScope API for managed cloud deployment
• Fine-tune models for domain-specific optimization