jb/paseo

Files

T

Mohamed Boudra 40b199e329 Based on the git diff showing extensive package-lock.json changes and modifications to the agent system prompt and configuration, here's the commit message:

```
feat: add AI SDK dependencies and refine agent voice interaction config

- Add @ai-sdk packages for enhanced AI capabilities
- Install Deepgram SDK, livekit-client, and werift for voice/WebRTC
- Expand agent system prompt with multi-tool workflows and Claude Code interaction patterns
- Adjust VAD timing (1.2s silence threshold) and turn detection for better voice UX
- Switch TTS to inworld/inworld-tts-1 with Olivia voice
- Configure interruption timing (0.2s detection, no word requirement)
- Add comprehensive guidelines for conversational responses vs structured output
- Document plan mode triggers and Claude response type handling

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
```

2025-10-19 15:41:42 +02:00

4.8 KiB

Raw Blame History

Audio Processing Architecture

Overview

This application uses browser-side WebRTC audio processing with WebSocket transport. We do NOT use WebRTC for peer-to-peer connections.

Why This Architecture?

Browser-Side WebRTC Processing

✅ Echo cancellation - Removes feedback from speakers
✅ Noise suppression - Filters background noise
✅ Auto gain control - Normalizes volume levels
✅ No server-side processing needed

WebSocket Transport

✅ Simple and reliable
✅ Works with existing infrastructure
✅ No NAT traversal issues
✅ Easy to debug

No Server-Side WebRTC

❌ Server doesn't need to be a WebRTC peer
❌ No complex signaling protocol needed
❌ No audio encoding/decoding on server
❌ Simpler architecture

Audio Flow

Voice Input (STT)

Browser: getUserMedia() with audio constraints
Browser: MediaRecorder captures processed audio
Browser → Server: WebSocket (base64-encoded audio)
Server: OpenAI Whisper transcription
Server: LLM processing
Server: Generate TTS response

Voice Output (TTS)

Server: OpenAI TTS generates MP3
Server → Browser: WebSocket (base64-encoded MP3)
Browser: HTMLAudioElement playback

Audio Constraints

The browser applies these constraints to audio capture:

{
  echoCancellation: true,    // Remove echo
  noiseSuppression: true,    // Remove noise
  autoGainControl: true,     // Normalize volume
  sampleRate: 16000,         // Optimal for speech
  channelCount: 1,           // Mono
}

These constraints are applied via the Web Audio API in getUserMedia(), which uses the browser's built-in audio processing pipeline (same as WebRTC uses internally).

Audio Format

Input Format: WebM/Opus (preferred) or browser's best available codec
Sample Rate: 16kHz (optimal for speech recognition)
Channels: Mono (sufficient for voice)
Output Format: MP3 (from OpenAI TTS)

Browser Compatibility

✅ Chrome/Edge - Full support
✅ Firefox - Full support
✅ Safari - Full support (iOS requires https)
✅ Mobile browsers - Supported

Testing Audio Processing

To verify echo cancellation and noise suppression work:

Play music or make noise near your microphone
Start recording
Speak while noise is present
Stop recording and check transcription
Background noise should be filtered out

Performance

Latency: 2-4 seconds total
- STT: 1-2s (Whisper API)
- LLM: 0.5-1s (OpenAI GPT)
- TTS: 0.5-1s (OpenAI TTS)
Bandwidth: ~10-20 KB/s for audio (both directions)
Audio Quality: Excellent for speech (16kHz mono)

Implementation Details

Browser Side (`src/ui/lib/audio-capture.ts`)

The createAudioRecorder() function:

Requests microphone access with WebRTC audio constraints
Creates a MediaRecorder to capture processed audio
Collects audio chunks during recording
Returns a Blob when recording stops

Key features:

Logs actual applied audio settings for debugging
Prefers WebM/Opus codec (best for speech)
Falls back to other supported codecs if needed
Stops all audio tracks on completion

Server Side (`src/server/websocket-server.ts`)

The WebSocket server:

Receives base64-encoded audio chunks
Buffers chunks until recording completes
Passes complete audio to STT handler
Broadcasts transcription and TTS response