Initial monorepo setup with mcp-server and web packages
This commit is contained in:
Mohamed Boudra
@@ -0,0 +1,315 @@
|
||||
# OpenAI Realtime API Configuration Options
|
||||
|
||||
This document lists all available configuration parameters for the OpenAI Realtime API session.
|
||||
|
||||
## Core Parameters
|
||||
|
||||
### `model`
|
||||
- **Type**: `string`
|
||||
- **Default**: `'gpt-4o-realtime-preview-2024-12-17'`
|
||||
- **Options**: `'gpt-realtime'`, `'gpt-4o-realtime-preview-2024-12-17'`
|
||||
- **Description**: The Realtime model to use. `gpt-realtime` is the latest production model.
|
||||
|
||||
### `voice`
|
||||
- **Type**: `string`
|
||||
- **Default**: `'alloy'`
|
||||
- **Options**: `'alloy'`, `'echo'`, `'fable'`, `'onyx'`, `'nova'`, `'shimmer'`
|
||||
- **Description**: Voice to use for AI speech generation
|
||||
|
||||
### `modalities`
|
||||
- **Type**: `array of strings`
|
||||
- **Default**: `['text', 'audio']`
|
||||
- **Options**: `['text']`, `['audio']`, `['text', 'audio']`
|
||||
- **Description**: Response modalities to use. Set to `['text']` for text-only mode with separate TTS.
|
||||
|
||||
### `temperature`
|
||||
- **Type**: `float`
|
||||
- **Default**: `0.8`
|
||||
- **Range**: `0.6` to `1.2`
|
||||
- **Description**: Controls response randomness. Lower values = more deterministic, higher values = more creative.
|
||||
|
||||
### `max_response_output_tokens`
|
||||
- **Type**: `integer`
|
||||
- **Default**: Not specified
|
||||
- **Example**: `2048`
|
||||
- **Description**: Maximum number of tokens in the model's response.
|
||||
|
||||
### `instructions`
|
||||
- **Type**: `string`
|
||||
- **Default**: None
|
||||
- **Description**: System instructions for the model (like system messages in Chat API)
|
||||
|
||||
---
|
||||
|
||||
## Audio Configuration
|
||||
|
||||
### `input_audio_format`
|
||||
- **Type**: `string`
|
||||
- **Default**: `'pcm16'`
|
||||
- **Options**: `'pcm16'`, `'g711_ulaw'`, `'g711_alaw'`
|
||||
- **Description**: Format for input audio
|
||||
|
||||
### `output_audio_format`
|
||||
- **Type**: `string`
|
||||
- **Default**: `'pcm16'`
|
||||
- **Options**: `'pcm16'`, `'g711_ulaw'`, `'g711_alaw'`
|
||||
- **Description**: Format for output audio
|
||||
|
||||
### `input_audio_transcription`
|
||||
- **Type**: `object`
|
||||
- **Default**: None
|
||||
- **Properties**:
|
||||
- `model`: Transcription model (e.g., `'gpt-4o-transcribe'`)
|
||||
- **Description**: Configuration for transcribing input audio
|
||||
- **Example**:
|
||||
```json
|
||||
{
|
||||
"model": "gpt-4o-transcribe"
|
||||
}
|
||||
```
|
||||
|
||||
### `input_audio_noise_reduction` (New in 2025)
|
||||
- **Type**: `object | null`
|
||||
- **Default**: `null`
|
||||
- **Description**: Noise reduction configuration for input audio
|
||||
- **Note**: Using noise reduction can impact latency, especially with semantic VAD turn detection
|
||||
- **Status**: Recently added parameter, check latest API docs for configuration details
|
||||
|
||||
---
|
||||
|
||||
## Turn Detection Configuration
|
||||
|
||||
Controls when the AI should start/stop responding based on voice activity.
|
||||
|
||||
### `turn_detection`
|
||||
- **Type**: `object | null`
|
||||
- **Default**: Server VAD mode
|
||||
- **Description**: Configuration for turn detection
|
||||
|
||||
#### Server VAD Mode (Default)
|
||||
```json
|
||||
{
|
||||
"type": "server_vad",
|
||||
"threshold": 0.5,
|
||||
"prefix_padding_ms": 300,
|
||||
"silence_duration_ms": 500,
|
||||
"create_response": true,
|
||||
"interrupt_response": true
|
||||
}
|
||||
```
|
||||
|
||||
**Parameters**:
|
||||
- `threshold` (float, 0.0-1.0): Voice activity detection threshold. Higher values (e.g., 0.7) work better in noisy environments.
|
||||
- `prefix_padding_ms` (integer): Audio to include before speech starts (ms)
|
||||
- `silence_duration_ms` (integer): Silence duration to wait before considering speech ended (ms)
|
||||
- `create_response` (boolean): Automatically create response when turn detected
|
||||
- `interrupt_response` (boolean): Allow user to interrupt AI responses
|
||||
|
||||
#### Semantic VAD Mode
|
||||
```json
|
||||
{
|
||||
"type": "semantic_vad",
|
||||
"eagerness": "auto",
|
||||
"create_response": true,
|
||||
"interrupt_response": true
|
||||
}
|
||||
```
|
||||
|
||||
**Parameters**:
|
||||
- `eagerness` (string): `'low'`, `'medium'`, `'high'`, or `'auto'`. Controls how quickly the model responds.
|
||||
- `create_response` (boolean): Automatically create response
|
||||
- `interrupt_response` (boolean): Allow interruptions
|
||||
|
||||
**Note**: Semantic VAD can have higher latency, especially with noise reduction enabled.
|
||||
|
||||
#### Disable Turn Detection
|
||||
```json
|
||||
{
|
||||
"type": "none"
|
||||
}
|
||||
```
|
||||
Use this if you want manual control over when responses are generated.
|
||||
|
||||
---
|
||||
|
||||
## Tool Configuration
|
||||
|
||||
### `tools`
|
||||
- **Type**: `array of objects`
|
||||
- **Default**: `[]`
|
||||
- **Description**: Tools available for the model to use
|
||||
|
||||
#### Remote MCP Server Tool
|
||||
```json
|
||||
{
|
||||
"type": "mcp",
|
||||
"server_label": "my-mcp-server",
|
||||
"server_url": "https://example.com/mcp",
|
||||
"authorization": "Bearer token", // Optional
|
||||
"require_approval": "never" // or "always"
|
||||
}
|
||||
```
|
||||
|
||||
**Parameters**:
|
||||
- `type`: Must be `'mcp'` for MCP servers
|
||||
- `server_label`: Friendly name for the server
|
||||
- `server_url`: HTTPS URL of the MCP server
|
||||
- `authorization`: Optional auth header value
|
||||
- `require_approval`: `'never'` for auto-execution, `'always'` for manual approval
|
||||
|
||||
#### Function Tool (Client-Side)
|
||||
```json
|
||||
{
|
||||
"type": "function",
|
||||
"name": "function_name",
|
||||
"description": "What the function does",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"param1": {
|
||||
"type": "string",
|
||||
"description": "Parameter description"
|
||||
}
|
||||
},
|
||||
"required": ["param1"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### `tool_choice`
|
||||
- **Type**: `string`
|
||||
- **Default**: `'auto'`
|
||||
- **Options**: `'auto'`, `'required'`, `'none'`
|
||||
- **Description**: Controls when tools are used
|
||||
- `'auto'`: AI decides when to use tools
|
||||
- `'required'`: AI must use a tool
|
||||
- `'none'`: Tools disabled for this response
|
||||
|
||||
---
|
||||
|
||||
## Current Configuration (app/api/session/route.ts)
|
||||
|
||||
```typescript
|
||||
{
|
||||
model: 'gpt-realtime',
|
||||
voice: 'shimmer',
|
||||
input_audio_transcription: {
|
||||
model: 'gpt-4o-transcribe',
|
||||
},
|
||||
tools: [
|
||||
{
|
||||
type: 'mcp',
|
||||
server_label: 'local-mcp',
|
||||
server_url: 'https://mohameds-macbook-pro.tail8fe838.ts.net/mcp?password=dev-password',
|
||||
require_approval: 'never',
|
||||
},
|
||||
],
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Recommended Settings for Different Use Cases
|
||||
|
||||
### High Quality Conversation (Default)
|
||||
```json
|
||||
{
|
||||
"model": "gpt-realtime",
|
||||
"temperature": 0.8,
|
||||
"turn_detection": {
|
||||
"type": "server_vad",
|
||||
"threshold": 0.5
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Noisy Environment
|
||||
```json
|
||||
{
|
||||
"model": "gpt-realtime",
|
||||
"input_audio_noise_reduction": {},
|
||||
"turn_detection": {
|
||||
"type": "server_vad",
|
||||
"threshold": 0.7, // Higher threshold for noisy environments
|
||||
"silence_duration_ms": 700 // Wait longer for silence
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### More Deterministic Responses
|
||||
```json
|
||||
{
|
||||
"model": "gpt-realtime",
|
||||
"temperature": 0.6,
|
||||
"max_response_output_tokens": 1024
|
||||
}
|
||||
```
|
||||
|
||||
### Fast, Responsive Conversation
|
||||
```json
|
||||
{
|
||||
"model": "gpt-realtime",
|
||||
"turn_detection": {
|
||||
"type": "semantic_vad",
|
||||
"eagerness": "high"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Manual Control (No Auto-Response)
|
||||
```json
|
||||
{
|
||||
"model": "gpt-realtime",
|
||||
"turn_detection": {
|
||||
"type": "none"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Notes on Noise Reduction
|
||||
|
||||
**Availability**: `input_audio_noise_reduction` is a newly added parameter (2025).
|
||||
|
||||
**Pros**:
|
||||
- Improves recognition in noisy environments
|
||||
- Better handling of background noise
|
||||
|
||||
**Cons**:
|
||||
- Can increase latency, especially with semantic VAD
|
||||
- May affect audio quality in some cases
|
||||
|
||||
**Recommendation**: Test with and without noise reduction to see if the latency trade-off is worth it for your use case.
|
||||
|
||||
---
|
||||
|
||||
## Client-Side Audio Processing
|
||||
|
||||
In addition to server-side configuration, consider client-side audio processing:
|
||||
|
||||
1. **Browser WebRTC**: Modern browsers include:
|
||||
- Echo cancellation
|
||||
- Noise suppression
|
||||
- Automatic gain control
|
||||
|
||||
2. **getUserMedia constraints**:
|
||||
```javascript
|
||||
navigator.mediaDevices.getUserMedia({
|
||||
audio: {
|
||||
echoCancellation: true,
|
||||
noiseSuppression: true,
|
||||
autoGainControl: true
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
These are currently **not configured** in the app but can be added in `use-webrtc-voice.ts` for additional audio quality improvements.
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [OpenAI Realtime API Documentation](https://platform.openai.com/docs/guides/realtime)
|
||||
- [OpenAI Realtime API Reference](https://platform.openai.com/docs/api-reference/realtime)
|
||||
- [LiveKit OpenAI Plugin Docs](https://docs.livekit.io/agents/integrations/realtime/openai/)
|
||||
Reference in New Issue
Block a user