316 lines
7.7 KiB
Markdown
316 lines
7.7 KiB
Markdown
# OpenAI Realtime API Configuration Options
|
|
|
|
This document lists all available configuration parameters for the OpenAI Realtime API session.
|
|
|
|
## Core Parameters
|
|
|
|
### `model`
|
|
- **Type**: `string`
|
|
- **Default**: `'gpt-4o-realtime-preview-2024-12-17'`
|
|
- **Options**: `'gpt-realtime'`, `'gpt-4o-realtime-preview-2024-12-17'`
|
|
- **Description**: The Realtime model to use. `gpt-realtime` is the latest production model.
|
|
|
|
### `voice`
|
|
- **Type**: `string`
|
|
- **Default**: `'alloy'`
|
|
- **Options**: `'alloy'`, `'echo'`, `'fable'`, `'onyx'`, `'nova'`, `'shimmer'`
|
|
- **Description**: Voice to use for AI speech generation
|
|
|
|
### `modalities`
|
|
- **Type**: `array of strings`
|
|
- **Default**: `['text', 'audio']`
|
|
- **Options**: `['text']`, `['audio']`, `['text', 'audio']`
|
|
- **Description**: Response modalities to use. Set to `['text']` for text-only mode with separate TTS.
|
|
|
|
### `temperature`
|
|
- **Type**: `float`
|
|
- **Default**: `0.8`
|
|
- **Range**: `0.6` to `1.2`
|
|
- **Description**: Controls response randomness. Lower values = more deterministic, higher values = more creative.
|
|
|
|
### `max_response_output_tokens`
|
|
- **Type**: `integer`
|
|
- **Default**: Not specified
|
|
- **Example**: `2048`
|
|
- **Description**: Maximum number of tokens in the model's response.
|
|
|
|
### `instructions`
|
|
- **Type**: `string`
|
|
- **Default**: None
|
|
- **Description**: System instructions for the model (like system messages in Chat API)
|
|
|
|
---
|
|
|
|
## Audio Configuration
|
|
|
|
### `input_audio_format`
|
|
- **Type**: `string`
|
|
- **Default**: `'pcm16'`
|
|
- **Options**: `'pcm16'`, `'g711_ulaw'`, `'g711_alaw'`
|
|
- **Description**: Format for input audio
|
|
|
|
### `output_audio_format`
|
|
- **Type**: `string`
|
|
- **Default**: `'pcm16'`
|
|
- **Options**: `'pcm16'`, `'g711_ulaw'`, `'g711_alaw'`
|
|
- **Description**: Format for output audio
|
|
|
|
### `input_audio_transcription`
|
|
- **Type**: `object`
|
|
- **Default**: None
|
|
- **Properties**:
|
|
- `model`: Transcription model (e.g., `'gpt-4o-transcribe'`)
|
|
- **Description**: Configuration for transcribing input audio
|
|
- **Example**:
|
|
```json
|
|
{
|
|
"model": "gpt-4o-transcribe"
|
|
}
|
|
```
|
|
|
|
### `input_audio_noise_reduction` (New in 2025)
|
|
- **Type**: `object | null`
|
|
- **Default**: `null`
|
|
- **Description**: Noise reduction configuration for input audio
|
|
- **Note**: Using noise reduction can impact latency, especially with semantic VAD turn detection
|
|
- **Status**: Recently added parameter, check latest API docs for configuration details
|
|
|
|
---
|
|
|
|
## Turn Detection Configuration
|
|
|
|
Controls when the AI should start/stop responding based on voice activity.
|
|
|
|
### `turn_detection`
|
|
- **Type**: `object | null`
|
|
- **Default**: Server VAD mode
|
|
- **Description**: Configuration for turn detection
|
|
|
|
#### Server VAD Mode (Default)
|
|
```json
|
|
{
|
|
"type": "server_vad",
|
|
"threshold": 0.5,
|
|
"prefix_padding_ms": 300,
|
|
"silence_duration_ms": 500,
|
|
"create_response": true,
|
|
"interrupt_response": true
|
|
}
|
|
```
|
|
|
|
**Parameters**:
|
|
- `threshold` (float, 0.0-1.0): Voice activity detection threshold. Higher values (e.g., 0.7) work better in noisy environments.
|
|
- `prefix_padding_ms` (integer): Audio to include before speech starts (ms)
|
|
- `silence_duration_ms` (integer): Silence duration to wait before considering speech ended (ms)
|
|
- `create_response` (boolean): Automatically create response when turn detected
|
|
- `interrupt_response` (boolean): Allow user to interrupt AI responses
|
|
|
|
#### Semantic VAD Mode
|
|
```json
|
|
{
|
|
"type": "semantic_vad",
|
|
"eagerness": "auto",
|
|
"create_response": true,
|
|
"interrupt_response": true
|
|
}
|
|
```
|
|
|
|
**Parameters**:
|
|
- `eagerness` (string): `'low'`, `'medium'`, `'high'`, or `'auto'`. Controls how quickly the model responds.
|
|
- `create_response` (boolean): Automatically create response
|
|
- `interrupt_response` (boolean): Allow interruptions
|
|
|
|
**Note**: Semantic VAD can have higher latency, especially with noise reduction enabled.
|
|
|
|
#### Disable Turn Detection
|
|
```json
|
|
{
|
|
"type": "none"
|
|
}
|
|
```
|
|
Use this if you want manual control over when responses are generated.
|
|
|
|
---
|
|
|
|
## Tool Configuration
|
|
|
|
### `tools`
|
|
- **Type**: `array of objects`
|
|
- **Default**: `[]`
|
|
- **Description**: Tools available for the model to use
|
|
|
|
#### Remote MCP Server Tool
|
|
```json
|
|
{
|
|
"type": "mcp",
|
|
"server_label": "my-mcp-server",
|
|
"server_url": "https://example.com/mcp",
|
|
"authorization": "Bearer token", // Optional
|
|
"require_approval": "never" // or "always"
|
|
}
|
|
```
|
|
|
|
**Parameters**:
|
|
- `type`: Must be `'mcp'` for MCP servers
|
|
- `server_label`: Friendly name for the server
|
|
- `server_url`: HTTPS URL of the MCP server
|
|
- `authorization`: Optional auth header value
|
|
- `require_approval`: `'never'` for auto-execution, `'always'` for manual approval
|
|
|
|
#### Function Tool (Client-Side)
|
|
```json
|
|
{
|
|
"type": "function",
|
|
"name": "function_name",
|
|
"description": "What the function does",
|
|
"parameters": {
|
|
"type": "object",
|
|
"properties": {
|
|
"param1": {
|
|
"type": "string",
|
|
"description": "Parameter description"
|
|
}
|
|
},
|
|
"required": ["param1"]
|
|
}
|
|
}
|
|
```
|
|
|
|
### `tool_choice`
|
|
- **Type**: `string`
|
|
- **Default**: `'auto'`
|
|
- **Options**: `'auto'`, `'required'`, `'none'`
|
|
- **Description**: Controls when tools are used
|
|
- `'auto'`: AI decides when to use tools
|
|
- `'required'`: AI must use a tool
|
|
- `'none'`: Tools disabled for this response
|
|
|
|
---
|
|
|
|
## Current Configuration (app/api/session/route.ts)
|
|
|
|
```typescript
|
|
{
|
|
model: 'gpt-realtime',
|
|
voice: 'shimmer',
|
|
input_audio_transcription: {
|
|
model: 'gpt-4o-transcribe',
|
|
},
|
|
tools: [
|
|
{
|
|
type: 'mcp',
|
|
server_label: 'local-mcp',
|
|
server_url: 'https://mohameds-macbook-pro.tail8fe838.ts.net/mcp?password=dev-password',
|
|
require_approval: 'never',
|
|
},
|
|
],
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Recommended Settings for Different Use Cases
|
|
|
|
### High Quality Conversation (Default)
|
|
```json
|
|
{
|
|
"model": "gpt-realtime",
|
|
"temperature": 0.8,
|
|
"turn_detection": {
|
|
"type": "server_vad",
|
|
"threshold": 0.5
|
|
}
|
|
}
|
|
```
|
|
|
|
### Noisy Environment
|
|
```json
|
|
{
|
|
"model": "gpt-realtime",
|
|
"input_audio_noise_reduction": {},
|
|
"turn_detection": {
|
|
"type": "server_vad",
|
|
"threshold": 0.7, // Higher threshold for noisy environments
|
|
"silence_duration_ms": 700 // Wait longer for silence
|
|
}
|
|
}
|
|
```
|
|
|
|
### More Deterministic Responses
|
|
```json
|
|
{
|
|
"model": "gpt-realtime",
|
|
"temperature": 0.6,
|
|
"max_response_output_tokens": 1024
|
|
}
|
|
```
|
|
|
|
### Fast, Responsive Conversation
|
|
```json
|
|
{
|
|
"model": "gpt-realtime",
|
|
"turn_detection": {
|
|
"type": "semantic_vad",
|
|
"eagerness": "high"
|
|
}
|
|
}
|
|
```
|
|
|
|
### Manual Control (No Auto-Response)
|
|
```json
|
|
{
|
|
"model": "gpt-realtime",
|
|
"turn_detection": {
|
|
"type": "none"
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Notes on Noise Reduction
|
|
|
|
**Availability**: `input_audio_noise_reduction` is a newly added parameter (2025).
|
|
|
|
**Pros**:
|
|
- Improves recognition in noisy environments
|
|
- Better handling of background noise
|
|
|
|
**Cons**:
|
|
- Can increase latency, especially with semantic VAD
|
|
- May affect audio quality in some cases
|
|
|
|
**Recommendation**: Test with and without noise reduction to see if the latency trade-off is worth it for your use case.
|
|
|
|
---
|
|
|
|
## Client-Side Audio Processing
|
|
|
|
In addition to server-side configuration, consider client-side audio processing:
|
|
|
|
1. **Browser WebRTC**: Modern browsers include:
|
|
- Echo cancellation
|
|
- Noise suppression
|
|
- Automatic gain control
|
|
|
|
2. **getUserMedia constraints**:
|
|
```javascript
|
|
navigator.mediaDevices.getUserMedia({
|
|
audio: {
|
|
echoCancellation: true,
|
|
noiseSuppression: true,
|
|
autoGainControl: true
|
|
}
|
|
});
|
|
```
|
|
|
|
These are currently **not configured** in the app but can be added in `use-webrtc-voice.ts` for additional audio quality improvements.
|
|
|
|
---
|
|
|
|
## References
|
|
|
|
- [OpenAI Realtime API Documentation](https://platform.openai.com/docs/guides/realtime)
|
|
- [OpenAI Realtime API Reference](https://platform.openai.com/docs/api-reference/realtime)
|
|
- [LiveKit OpenAI Plugin Docs](https://docs.livekit.io/agents/integrations/realtime/openai/)
|