Skip to main content

​
Overview

The createNavaiMobileVoiceSession function creates a voice session that coordinates WebRTC transport, backend client communication, and real-time event handling. It provides a high-level interface for managing voice agent sessions.

​
Usage

import {
  createNavaiMobileVoiceSession,
  createReactNativeWebRtcTransport,
  createNavaiMobileBackendClient
} from '@navai/voice-mobile';
import { mediaDevices, RTCPeerConnection } from 'react-native-webrtc';

const session = createNavaiMobileVoiceSession({
  transport: createReactNativeWebRtcTransport({
    globals: { mediaDevices, RTCPeerConnection },
    model: 'gpt-4o-realtime-preview',
    remoteAudioTrackVolume: 10
  }),
  backendClient: createNavaiMobileBackendClient({
    apiBaseUrl: 'https://api.myapp.com'
  }),
  onRealtimeEvent: (event) => console.log('Event:', event),
  onRealtimeError: (error) => console.error('Error:', error)
});

// Start the session
const response = await session.start({
  model: 'gpt-4o-realtime-preview',
  voice: 'alloy'
});

console.log('Backend functions:', response.backendFunctions);

// Send events
await session.sendRealtimeEvent({
  type: 'session.update',
  session: { instructions: 'You are a helpful assistant.' }
});

// Execute backend function
const result = await session.executeBackendFunction({
  functionName: 'get_weather',
  payload: { location: 'San Francisco' }
});

// Stop the session
await session.stop();

​
Options

​
transport
NavaiRealtimeTransport
required
WebRTC transport instance. Create with createReactNativeWebRtcTransport.
transport: createReactNativeWebRtcTransport({
  globals: { mediaDevices, RTCPeerConnection }
})
​
backendClient
NavaiMobileBackendClient
Backend client instance. Create with createNavaiMobileBackendClient.If not provided, a client will be created using backendClientOptions.
backendClient: createNavaiMobileBackendClient({
  apiBaseUrl: 'https://api.myapp.com'
})
​
backendClientOptions
CreateNavaiMobileBackendClientOptions
Options for creating a backend client if backendClient is not provided.
backendClientOptions: {
  apiBaseUrl: 'https://api.myapp.com',
  env: { NAVAI_API_URL: 'https://api.myapp.com' }
}
​
onRealtimeEvent
(event: unknown) => void
Callback for real-time events from the OpenAI Realtime API.
onRealtimeEvent: (event) => {
  console.log('Event type:', event.type);
}
​
onRealtimeError
(error: unknown) => void
Callback for transport and real-time errors.
onRealtimeError: (error) => {
  console.error('Realtime error:', error);
}

​
Return Value

Returns NavaiMobileVoiceSession object:
​
start
(input?: StartNavaiMobileVoiceSessionInput) => Promise<StartNavaiMobileVoiceSessionResult>
Start the voice session.Throws error if session is already connecting or connected, or if connection fails.
​
stop
() => Promise<void>
Stop the voice session and disconnect transport. Safe to call multiple times.
​
listBackendFunctions
(forceReload?: boolean) => Promise<BackendFunctionsResult>
List available backend functions.
​
executeBackendFunction
(input: ExecuteNavaiBackendFunctionInput) => Promise<unknown>
Execute a backend function.Returns the function result. Throws error if execution fails.
​
sendRealtimeEvent
(event: unknown) => Promise<void>
Send a real-time event to the OpenAI Realtime API.
await session.sendRealtimeEvent({
  type: 'session.update',
  session: { instructions: 'New instructions' }
});
Throws error if transport doesn’t support sending events.
​
getSnapshot
() => NavaiMobileVoiceSessionSnapshot
Get current session snapshot.

​
Examples

​
Basic Setup

import {
  createNavaiMobileVoiceSession,
  createReactNativeWebRtcTransport,
  createNavaiMobileBackendClient
} from '@navai/voice-mobile';
import { mediaDevices, RTCPeerConnection } from 'react-native-webrtc';

const session = createNavaiMobileVoiceSession({
  transport: createReactNativeWebRtcTransport({
    globals: { mediaDevices, RTCPeerConnection }
  }),
  backendClient: createNavaiMobileBackendClient({
    apiBaseUrl: process.env.NAVAI_API_URL
  })
});

const response = await session.start();
console.log('Session started with', response.backendFunctions.length, 'functions');

​
With Event Handlers

const session = createNavaiMobileVoiceSession({
  transport,
  backendClient,
  onRealtimeEvent: (event) => {
    console.log('[Event]', event.type);
    
    if (event.type === 'conversation.item.created') {
      console.log('New conversation item:', event.item);
    }
    
    if (event.type === 'response.audio_transcript.done') {
      console.log('Transcript:', event.transcript);
    }
  },
  onRealtimeError: (error) => {
    console.error('[Error]', error);
    // Handle error (e.g., show error message to user)
  }
});

​
Custom Voice Configuration

const response = await session.start({
  model: 'gpt-4o-realtime-preview',
  voice: 'shimmer',
  instructions: 'You are a helpful fitness coach. Be encouraging and motivational.',
  language: 'en'
});

​
Preloading Backend Functions

// Preload functions during start (default)
const response = await session.start({
  preloadBackendFunctions: true
});

console.log('Functions:', response.backendFunctions);

// Or load functions separately
const { functions, warnings } = await session.listBackendFunctions(true);
console.log('Available functions:', functions.map(f => f.name));

​
Executing Backend Functions

try {
  const weatherResult = await session.executeBackendFunction({
    functionName: 'get_weather',
    payload: {
      location: 'San Francisco, CA',
      units: 'fahrenheit'
    }
  });
  
  console.log('Weather:', weatherResult);
} catch (error) {
  console.error('Function execution failed:', error.message);
}

​
Sending Events

// Update session instructions
await session.sendRealtimeEvent({
  type: 'session.update',
  session: {
    type: 'realtime',
    instructions: 'You are now a Spanish tutor.',
    tools: [...],
    tool_choice: 'auto'
  }
});

// Create a user message
await session.sendRealtimeEvent({
  type: 'conversation.item.create',
  item: {
    type: 'message',
    role: 'user',
    content: [{ type: 'text', text: 'Hello!' }]
  }
});

// Trigger response
await session.sendRealtimeEvent({
  type: 'response.create'
});

​
Session Lifecycle Management

const session = createNavaiMobileVoiceSession({ transport, backendClient });

try {
  // Start session
  const response = await session.start();
  console.log('Started with warnings:', response.warnings);
  
  // Check state
  const snapshot = session.getSnapshot();
  console.log('State:', snapshot.state); // 'connected'
  console.log('Transport:', snapshot.transportState); // 'connected'
  
  // Use session...
  
} catch (error) {
  console.error('Session failed:', error);
} finally {
  // Clean up
  await session.stop();
  console.log('Session stopped');
}

​
Integration with Agent Runtime

import {
  createNavaiMobileVoiceSession,
  createNavaiMobileAgentRuntime,
  loadNavaiFunctions
} from '@navai/voice-mobile';

const session = createNavaiMobileVoiceSession({ transport, backendClient });

// Start session and get backend functions
const response = await session.start();

// Load frontend functions
const functionsRegistry = await loadNavaiFunctions(functionModuleLoaders);

// Create agent runtime
const agentRuntime = createNavaiMobileAgentRuntime({
  navigate: (path) => navigation.navigate(path),
  routes,
  functionsRegistry,
  backendFunctions: response.backendFunctions,
  executeBackendFunction: session.executeBackendFunction
});

// Configure session with agent runtime
await session.sendRealtimeEvent({
  type: 'session.update',
  session: {
    type: 'realtime',
    instructions: agentRuntime.session.instructions,
    tools: agentRuntime.session.tools,
    tool_choice: 'auto'
  }
});

​
Session States

  • idle: Session not started
  • connecting: Starting session (creating secret, connecting transport, loading functions)
  • connected: Session active and ready
  • error: Session failed to start or encountered error

​
Error Handling

try {
  await session.start();
} catch (error) {
  console.error('Failed to start session:', error.message);
  
  const snapshot = session.getSnapshot();
  console.log('Session state:', snapshot.state); // 'error'
  console.log('Transport state:', snapshot.transportState);
}

// Sending events
try {
  await session.sendRealtimeEvent({ type: 'session.update', ... });
} catch (error) {
  console.error('Failed to send event:', error.message);
}

// Backend functions
try {
  const result = await session.executeBackendFunction({
    functionName: 'unknown_function',
    payload: null
  });
} catch (error) {
  console.error('Function execution failed:', error.message);
}

Build docs developers (and LLMs) love

Get started for freeTalk to us