Skip to main content
The Stims audio system provides pooled microphone access and audio analysis for reactive visualizations. It minimizes permission prompts and stream allocations by reusing a single MediaStream across multiple toys.

​
Audio Architecture

The audio system is built around the concept of pooled microphone access where a single microphone stream is shared across multiple toy instances:

​
Audio Handle Interface

The audio service provides an AudioHandle that includes all resources needed for audio-reactive toys: 
export type AudioHandle = {
  analyser: FrequencyAnalyser;
  listener: THREE.AudioListener;
  audio: THREE.Audio | THREE.PositionalAudio;
  stream?: MediaStream;
  release: () => void;
};

​
AudioHandle Components

​
Acquiring Audio

Use acquireAudioHandle() to get audio resources for your toy: 
import { acquireAudioHandle } from './services/audio-service';

export async function start(options?: ToyStartOptions): Promise<ToyInstance> {
  // Request audio with default settings (pooled microphone)
  const audioHandle = await acquireAudioHandle();
  
  // Use the analyser in your animation loop
  function animate() {
    const freqData = audioHandle.analyser.getFrequencyData();
    // Update visualization based on audio...
  }
  
  return {
    dispose() {
      // Always release audio handle
      audioHandle.release();
    },
  };
}

​
Audio Init Options

export interface AudioInitOptions {
  /**
   * Reuse the pooled microphone stream (default: true)
   */
  reuseMicrophone?: boolean;
  
  /**
   * MediaStream constraints for getUserMedia()
   */
  constraints?: MediaStreamConstraints;
  
  /**
   * Existing stream to use instead of requesting a new one
   */
  stream?: MediaStream;
  
  /**
   * Custom initAudio implementation for testing
   */
  initAudioImpl?: typeof initAudio;
  
  /**
   * Stop the stream when release() is called (default: false with pooling)
   */
  teardownOnRelease?: boolean;
}

​
Microphone Pooling

The audio service maintains a global pool to avoid repeated permission prompts: 
type AudioPoolEntry = {
  stream: MediaStream;
  users: number;
};

let pooledStream: AudioPoolEntry | null = null;
let streamPromise: Promise<MediaStream | null> | null = null;

​
Pool Lifecycle

  1. First acquisition: Prompts user for microphone permission via getUserMedia()
  2. Subsequent acquisitions: Reuses the same stream, incrementing the user count
  3. Release: Decrements the user count; stream remains active for next toy
  4. Reset: Optionally stops all tracks when navigating back to library
By default, the microphone stream stays active even when no toys are using it. This provides instant audio when loading the next toy, but you can call resetAudioPool({ stopStreams: true }) to fully stop it.

​
Stream Management

The audio service provides several functions for managing the stream lifecycle:

​
getOrCreateStream()

async function getOrCreateStream(constraints?: MediaStreamConstraints) {
  if (pooledStream?.stream) return pooledStream.stream;
  if (streamPromise) return streamPromise;

  streamPromise = navigator.mediaDevices
    ?.getUserMedia(constraints ?? DEFAULT_MICROPHONE_CONSTRAINTS)
    .catch((error) => {
      streamPromise = null;
      throw error;
    });

  const stream = await streamPromise;
  if (!stream) return null;

  pooledStream = { stream, users: 0 };
  return stream;
}
Ensures only one getUserMedia() call is active at a time, preventing race conditions.

​
prewarmMicrophone()

export async function prewarmMicrophone(
  constraints?: MediaStreamConstraints
): Promise<PermissionState> {
  const permission = await getMicrophonePermissionState();
  if (permission !== 'granted') return permission;

  await getOrCreateStream(constraints);
  return permission;
}
Preloads the microphone stream before a toy starts. Use this during app initialization to hide microphone latency: 
// In app.ts bootstrap
await prewarmMicrophone();

​
resetAudioPool()

export async function resetAudioPool({ stopStreams = true } = {}) {
  if (stopStreams) {
    stopPooledStream();
    return;
  }

  pooledStream = null;
  streamPromise = null;
}
Resets the audio pool state. Call with stopStreams: true to fully stop microphone tracks (e.g., when navigating to the library).

​
Audio Handle Lifecycle

The acquireAudioHandle() function manages the complete lifecycle: 
export async function acquireAudioHandle(
  options: AudioInitOptions = {},
): Promise<AudioHandle> {
  const {
    reuseMicrophone = true,
    initAudioImpl = initAudio,
    teardownOnRelease = false,
    ...audioOptions
  } = options;

  let stream: MediaStream | null = audioOptions.stream ?? null;
  let pooledEntry: AudioPoolEntry | null = null;

  // Get or reuse stream
  if (reuseMicrophone && !stream) {
    stream = await getOrCreateStream(audioOptions.constraints);
    if (stream && pooledStream) {
      pooledStream.users += 1;
      pooledEntry = pooledStream;
    }
  }

  // Initialize audio components
  let audio: Awaited<ReturnType<typeof initAudioImpl>>;
  try {
    audio = await initAudioImpl({
      ...audioOptions,
      stream: stream ?? audioOptions.stream,
      stopStreamOnCleanup: !reuseMicrophone,
    });
  } catch (error) {
    // Rollback pool user count on error
    if (pooledEntry) {
      pooledEntry.users = Math.max(0, pooledEntry.users - 1);
      if (pooledEntry.users === 0) {
        stopPooledStream();
      }
    }
    throw error;
  }

  // Create release function
  const release = () => {
    audio.cleanup?.();

    if (reuseMicrophone && pooledStream && stream === pooledStream.stream) {
      pooledStream.users = Math.max(0, pooledStream.users - 1);

      if (pooledStream.users === 0 && teardownOnRelease) {
        stopPooledStream();
      }
    }
  };

  return {
    analyser: audio.analyser,
    listener: audio.listener,
    audio: audio.audio,
    stream: audio.stream,
    release,
  };
}

​
Error Handling

If audio initialization fails after acquiring a pooled stream, the user count is automatically rolled back to prevent pool corruption.

​
Microphone Permission Flow

The audio system integrates with the UI permission flow:

​
Usage Patterns

​
Standard Audio-Reactive Toy

import { acquireAudioHandle } from '../core/services/audio-service';
import type { ToyInstance, ToyStartOptions } from '../core/toy-interface';

export async function start(options?: ToyStartOptions): Promise<ToyInstance> {
  // Get pooled audio
  const audioHandle = await acquireAudioHandle();
  
  // Set up scene with audio listener
  const camera = new THREE.PerspectiveCamera();
  camera.add(audioHandle.listener);
  
  function animate() {
    const analyser = audioHandle.analyser;
    
    // Get frequency bins
    const freqData = analyser.getFrequencyData();
    
    // Or use convenience methods
    const bass = analyser.getBass();
    const mid = analyser.getMid();
    const treble = analyser.getTreble();
    
    // Update visualization...
  }
  
  return {
    dispose() {
      audioHandle.release();
    },
  };
}

​
Custom Stream (No Pooling)

// For specialized audio toys that need their own stream
const audioHandle = await acquireAudioHandle({
  reuseMicrophone: false,
  teardownOnRelease: true,
  constraints: {
    audio: {
      echoCancellation: false,
      noiseSuppression: false,
      autoGainControl: false,
    },
  },
});

​
Prewarming for Instant Start

// In loader or app bootstrap
import { prewarmMicrophone } from '../core/services/audio-service';

// Before loading an audio toy
const permission = await prewarmMicrophone();

if (permission === 'granted') {
  // Stream is ready, toy will start instantly
  await loadToy(slug);
} else {
  // Show permission UI
  showMicrophonePrompt();
}

​
Audio Service Integration Points

ModulePurposeIntegration
microphone-flow.tsUI permission buttonsWires buttons to prewarmMicrophone() and acquireAudioHandle()
web-toy.tsToy runtime helpersOptionally requests audio via acquireAudioHandle()
loader.tsToy lifecycleCalls resetAudioPool({ stopStreams: true }) on navigate to library
app.tsApp bootstrapCan call prewarmMicrophone() after capability preflight

​
Best Practices

Always release audio handles - Call audioHandle.release() in your toy’s dispose() method to prevent memory leaks and pool corruption.
Don’t stop pooled streams manually - Let the audio service manage stream lifecycle. Stopping a pooled stream will break other toys.

​
Do’s and Don’ts

✅ Do:
  • Use default pooling (reuseMicrophone: true) for standard toys
  • Call release() in your dispose() method
  • Use prewarmMicrophone() to hide latency
  • Check permission state before acquiring audio
❌ Don’t:
  • Manually call stream.getTracks().forEach(t => t.stop())
  • Acquire multiple handles in the same toy without releasing
  • Forget to release handles on disposal
  • Bypass the audio service for microphone access

​
Debugging Audio Issues

​
Next Steps

Rendering

Learn about renderer pooling and WebGL/WebGPU backend selection

Architecture Overview

Return to the architecture overview

Build docs developers (and LLMs) love

Get started for freeTalk to us