paseo/packages/voice-assistant/VOICE_IMPLEMENTATION_PLAN.md

Voice Implementation Plan (Phases 6 & 7)

Research Summary

OpenAI Whisper API (STT)

Key Findings:

• No Native Streaming Support: OpenAI Whisper API does NOT support real-time streaming transcription
• Batch Processing Only: Must send complete audio files after recording stops
• Supported Formats: flac, m4a, mp3, mp4, mpeg, mpga, oga, ogg, wav, webm
• File Size Limit: 25 MB maximum
• API Endpoint: /v1/audio/transcriptions
• Strategy: Push-to-talk (click to record, click to stop, then transcribe)

Alternative Considered:

• OpenAI Realtime API provides true streaming STT+LLM+TTS in one WebSocket connection
• However, it's a different architecture (conversational AI vs tool-calling assistant)
• For this phase, we'll use Whisper API with push-to-talk model

OpenAI TTS API

Key Findings:

• Streaming Supported: Yes, via chunk transfer encoding
• Supported Formats: mp3, opus, aac, flac, wav, pcm
• Available Voices: alloy, echo, fable, onyx, nova, shimmer
• Quality Models:
  • tts-1: Faster, lower latency, normal quality
  • tts-1-hd: Higher quality, slower
• API Endpoint: /v1/audio/speech
• Default Format: MP3

Browser Audio APIs

MediaRecorder API (Audio Capture):

• Widely Supported: Chrome, Firefox, Safari, Edge (since 2021)
• Format Support Varies:
  • Chrome/Opera: audio/webm;codecs=opus
  • Firefox: audio/ogg;codecs=opus (also supports webm since Firefox 63)
  • Need to check MediaRecorder.isTypeSupported() at runtime
• Recommended Approach: Capture as WebM/Opus, convert if needed
• Events: dataavailable for chunks, stop for complete recording

Web Audio API (Audio Playback):

• Challenge: decodeAudioData() requires complete files, not chunks
• Solution Options:
  1. Use <audio> element with blob URLs (simpler, works with MP3)
  2. Use MediaSource Extensions for true streaming
  3. Buffer complete audio before playback
• Recommended: Option 1 for simplicity - buffer complete TTS response, then play

Architecture Decisions

STT Flow (Push-to-Talk Model)

User clicks microphone button
  ↓
Browser captures audio via MediaRecorder
  ↓
Audio buffered in browser as WebM/Opus
  ↓
User clicks stop button
  ↓
Complete audio sent to server via WebSocket (base64 encoded)
  ↓
Server saves as temporary file (WebM)
  ↓
Server converts to MP3 if needed (using FFmpeg or accept WebM directly)
  ↓
Server calls OpenAI Whisper API with audio file
  ↓
Transcript returned and fed to orchestrator
  ↓
processUserMessage() handles as normal text input

TTS Flow (Buffer-and-Play Model)

Orchestrator generates LLM response (streaming text)
  ↓
Server buffers complete response text
  ↓
Server calls OpenAI TTS API with complete text
  ↓
TTS returns audio (MP3 format, streamed from OpenAI)
  ↓
Server buffers complete audio
  ↓
Server sends complete audio to browser via WebSocket (base64 encoded)
  ↓
Browser creates Blob URL from audio data
  ↓
Browser plays using <audio> element
  ↓
Queue management: only play next when current finishes

Rationale for Buffer-and-Play:

• Simpler implementation (no streaming audio chunks)
• Avoids "clicks" between chunks
• Works reliably across all browsers
• TTS API is fast enough that latency is acceptable
• Can be upgraded to true streaming later if needed

Implementation Plan

Phase 6A: STT Integration

1. Server-side STT Module (src/server/agent/stt-openai.ts)

export interface STTConfig {
  apiKey: string;
  model?: 'whisper-1';
}

export interface TranscriptionResult {
  text: string;
  language?: string;
  duration?: number;
}

export function initializeSTT(config: STTConfig): void;
export async function transcribeAudio(audioBuffer: Buffer, format: string): Promise<TranscriptionResult>;

Implementation Notes:

• Use openai package (already installed)
• Accept audio buffer and format (webm, mp3, etc.)
• Call /v1/audio/transcriptions endpoint
• Return transcript text
• Handle errors gracefully

2. Update WebSocket Message Types (src/server/types.ts)

export interface WebSocketMessage {
  type: 'activity_log' | 'status' | 'webrtc_signal' | 'ping' | 'pong'
       | 'user_message' | 'assistant_chunk'
       | 'audio_chunk' | 'audio_output' | 'recording_state' | 'transcription_result';
  payload: unknown;
}

export interface AudioChunkPayload {
  audio: string; // base64 encoded audio data
  format: string; // 'webm', 'ogg', etc.
  isLast: boolean; // true when recording stopped
}

export interface RecordingStatePayload {
  isRecording: boolean;
}

export interface TranscriptionResultPayload {
  text: string;
  duration?: number;
}

3. Add Audio Handling to WebSocket Server (src/server/websocket-server.ts)

• Add handler for audio_chunk messages
• Buffer audio chunks until isLast: true
• Call STT transcription
• Send transcription_result back to client
• Feed transcript to orchestrator

4. Wire STT to Orchestrator (src/server/agent/orchestrator.ts)

• Add processAudioInput() function that wraps processUserMessage()
• Add activity log entries for STT events

Phase 6B: Browser Audio Capture

1. Audio Capture Utility (src/ui/lib/audio-capture.ts)

export interface AudioCaptureConfig {
  mimeType?: string;
  audioBitsPerSecond?: number;
}

export interface AudioRecorder {
  start(): Promise<void>;
  stop(): Promise<Blob>;
  isRecording(): boolean;
  getSupportedMimeType(): string | null;
}

export function createAudioRecorder(config?: AudioCaptureConfig): AudioRecorder;

Implementation Notes:

• Request microphone permission
• Detect supported mime type (prefer webm/opus)
• Use MediaRecorder API
• Buffer chunks in array
• Return complete Blob on stop

2. Voice Controls Component (src/ui/components/VoiceControls.tsx)

interface VoiceControlsProps {
  onAudioRecorded: (audio: Blob, format: string) => void;
  isProcessing: boolean;
}

UI Elements:

• Microphone button (toggle recording)
• Recording indicator (red dot animation)
• Processing state (spinning loader)
• Visual feedback (waveform or simple animation)

3. Update Main App (src/ui/App.tsx)

• Add voice controls section
• Handle audio_chunk sending
• Display transcription results
• Show recording/processing states

Phase 7A: TTS Integration

1. Server-side TTS Module (src/server/agent/tts-openai.ts)

export interface TTSConfig {
  apiKey: string;
  model?: 'tts-1' | 'tts-1-hd';
  voice?: 'alloy' | 'echo' | 'fable' | 'onyx' | 'nova' | 'shimmer';
  responseFormat?: 'mp3' | 'opus' | 'aac' | 'flac' | 'wav' | 'pcm';
}

export interface SpeechResult {
  audio: Buffer;
  format: string;
}

export function initializeTTS(config: TTSConfig): void;
export async function synthesizeSpeech(text: string): Promise<SpeechResult>;

Implementation Notes:

• Use openai package streaming support
• Default to tts-1 model with alloy voice
• Stream from OpenAI and buffer complete audio
• Return as Buffer with format info

2. Update WebSocket Message Types

export interface AudioOutputPayload {
  audio: string; // base64 encoded audio data (complete)
  format: string; // 'mp3'
  id: string; // unique ID for queue management
}

3. Wire TTS to Orchestrator (src/server/agent/orchestrator.ts)

• Modify processUserMessage() to optionally synthesize response
• Add enableTTS: boolean parameter
• After complete assistant response, call TTS
• Send audio_output message to WebSocket clients
• Keep text display alongside audio

Phase 7B: Browser Audio Playback

1. Audio Playback Utility (src/ui/lib/audio-playback.ts)

export interface AudioPlayer {
  play(audioData: Blob): Promise<void>;
  stop(): void;
  isPlaying(): boolean;
}

export function createAudioPlayer(): AudioPlayer;

Implementation Notes:

• Create blob URLs from audio data
• Use <audio> element for playback
• Clean up blob URLs after playback
• Simple queue: play next when current finishes

2. Update Voice Controls

• Add speaker indicator (playing state)
• Add stop playback button
• Visual feedback (audio wave or icon animation)

3. Update Main App

• Handle audio_output messages
• Queue audio playback
• Show playing state in UI

Testing Strategy

Unit Tests (Optional for now)

• Test audio format detection
• Test buffer encoding/decoding
• Test queue management

Manual Integration Tests

Test 1: Basic STT

1. Open UI, ensure WebSocket connected
2. Click microphone button
3. Speak: "list terminals"
4. Click stop recording
5. Verify: Transcription appears in activity log
6. Verify: Tool execution happens
7. Verify: Response displayed

Test 2: Basic TTS

1. Send text message: "what terminals are available?"
2. Verify: Text response streams as normal
3. Verify: Audio playback starts automatically
4. Verify: Audio matches text response
5. Verify: Text remains visible during playback

Test 3: Full Voice Conversation

1. Record: "create a terminal called voice-test in /tmp"
2. Verify: Transcription → tool call → response → TTS playback
3. Record: "list all terminals"
4. Verify: Second interaction works correctly
5. Verify: Terminal was actually created in tmux

Test 4: Error Handling

1. Try recording without microphone permission
2. Try sending empty audio
3. Try sending very long audio (>1 minute)
4. Verify: Graceful error messages

Test 5: Audio Format Compatibility

1. Test in Chrome (webm/opus)
2. Test in Firefox (ogg/opus)
3. Test in Safari (if possible)
4. Verify: Format detection works correctly

Environment Configuration

Add to .env (or .env.example):

# OpenAI API Configuration
OPENAI_API_KEY=sk-...

# STT Configuration (optional overrides)
WHISPER_MODEL=whisper-1

# TTS Configuration (optional overrides)
TTS_MODEL=tts-1
TTS_VOICE=alloy
TTS_FORMAT=mp3

Dependencies (Already Installed)

• openai: ^4.20.0 ✓
• ws: ^8.14.2 ✓
• uuid: ^9.0.1 ✓

No additional dependencies needed!

Implementation Order

1. STT Server Module - Core transcription logic
2. Browser Audio Capture - Get microphone working
3. WebSocket Audio Messages - Wire STT to UI
4. STT Integration Test - Verify speech-to-text works end-to-end
5. TTS Server Module - Core synthesis logic
6. Browser Audio Playback - Get speaker working
7. TTS Integration Test - Verify text-to-speech works end-to-end
8. Polish & Error Handling - Edge cases, loading states, errors
9. End-to-End Voice Test - Full conversation flow

Future Enhancements (Out of Scope)

• Real-time streaming STT (requires different API or third-party service)
• Voice Activity Detection (VAD) for automatic recording stop
• Speaker diarization (multiple speakers)
• Custom wake words
• Interrupt playback to speak
• Noise cancellation
• Audio visualization (waveforms)
• OpenAI Realtime API migration (full duplex voice)

Success Criteria

✅ User can record voice and get accurate transcription ✅ Transcription feeds into existing text chat flow ✅ Assistant responses are spoken via TTS ✅ Audio quality is acceptable (clear, natural) ✅ Latency is acceptable (< 3 seconds for STT, < 2 seconds for TTS) ✅ Error handling is graceful ✅ Works in Chrome and Firefox ✅ No new dependencies required ✅ Code is maintainable and well-structured