paseo/packages/voice-assistant/WEBRTC_IMPLEMENTATION_PLAN.md

WebRTC Audio Streaming Implementation Plan

Date: 2025-10-19 Phase: 8 - WebRTC Migration Status: Planning Complete SUPERSEDED - See Note Below

---

⚠️ ARCHITECTURAL PIVOT - OCTOBER 2025

This implementation plan has been superseded by a simpler architecture.

We initially planned to use server-side WebRTC (via werift) to handle bidirectional audio streaming. However, we realized that server-side WebRTC is not necessary for our use case.

What We Actually Need

1. Browser-side WebRTC audio processing - Echo cancellation, noise suppression, auto gain control
2. Simple WebSocket transport - Send/receive audio between browser and server

Why Server-Side WebRTC is Unnecessary

• The server doesn't need to be a WebRTC peer
• We only need the browser's audio processing features (available via getUserMedia())
• WebSocket provides simple, reliable transport for audio data
• No need for complex signaling, ICE, STUN, or TURN servers

Current Architecture (Implemented)

Browser                           Server
  ↓                                 ↓
[getUserMedia] → WebRTC Audio Processing → [MediaRecorder]
  ↓                                           ↓
[Base64 Encode] → WebSocket → [Decode] → STT → LLM → TTS
                                                ↓
[Audio Element] ← WebSocket ← [Base64 Encode] ← [MP3]

Key Benefits:

• Simple architecture (no server-side WebRTC complexity)
• Browser handles all audio processing (echo cancel, noise suppress, AGC)
• WebSocket transport is reliable and easy to debug
• No NAT traversal issues
• Works perfectly for our use case

See Updated Documentation

For current audio processing architecture, see:

• /Users/moboudra/dev/voice-dev/packages/voice-assistant/AUDIO_PROCESSING.md

---

Original Plan (For Historical Reference)

Executive Summary

This document outlines the plan to migrate from WebSocket-based audio transport to WebRTC for real-time bidirectional audio streaming. The implementation uses a dual-connection architecture:

1. WebSocket - Control channel for signaling, text messages, tool notifications, and status updates
2. WebRTC - Media channel for low-latency bidirectional audio streaming

---

Research Findings

Node.js WebRTC Library Evaluation

After extensive research (avoiding wrtc due to native compilation issues), here are the viable options:

Option 1: werift-webrtc (RECOMMENDED)

• Repository: https://github.com/shinyoshiaki/werift-webrtc
• Status: Actively maintained (latest: 0.22.2, 3 months ago)
• Type: Pure TypeScript/JavaScript implementation
• Pros:
  • No native dependencies (no compilation issues)
  • Full TypeScript support
  • Includes ICE/DTLS/SCTP/RTP/SRTP
  • Works in Node.js
  • Active development
• Cons:
  • Still maturing (some features incomplete)
  • ICE restart not yet available
  • TURN ICE TLS/TCP not supported
  • No simulcast (send) support yet
• Verdict: Best choice for our use case (simple peer-to-peer audio)

Option 2: simple-peer

• Repository: https://github.com/feross/simple-peer
• Status: Popular but unmaintained (last update 2021)
• Type: Browser + Node.js wrapper around WebRTC
• Pros:
  • Simple API
  • Well-documented
  • 312 projects using it
• Cons:
  • Requires wrtc for Node.js (which we're avoiding)
  • Not updated in 4 years
  • Would need @roamhq/wrtc fork
• Verdict: Not suitable (requires wrtc)

Option 3: Alternative Approach - Signaling Only

• Concept: Browser handles both WebRTC peers
• Architecture: Server acts as signaling relay only
• Pros:
  • No Node.js WebRTC library needed
  • Server complexity stays low
  • Browser WebRTC is mature and reliable
• Cons:
  • Cannot process audio server-side (defeats our purpose)
  • Cannot wire to STT/TTS on server
• Verdict: Not applicable to our use case

Recommended Approach: werift-webrtc

We will use werift-webrtc for server-side WebRTC peer connection management. This gives us:

1. Pure JavaScript implementation (no native compilation)
2. Full control over audio streams on the server
3. Ability to wire WebRTC audio tracks to STT/TTS pipelines
4. TypeScript support for better development experience

---

Architecture Design

Current Architecture (WebSocket Audio)

Browser                           Server
  ↓                                 ↓
[MediaRecorder] → WebSocket → [Buffer] → STT → LLM → TTS → [Buffer] → WebSocket → [Audio Element]
  ↑                                                                                      ↓
[Microphone]                                                                        [Speakers]

Issues:

• High latency (base64 encoding/decoding)
• Buffering required (push-to-talk)
• Not real-time
• Inefficient for audio transport

New Architecture (WebRTC + WebSocket)

Browser                                Server
  ↓                                      ↓
[RTCPeerConnection] ←─────────────→ [werift.RTCPeerConnection]
  ↑                   WebRTC Audio        ↓
  ↑                                       ↓
[Microphone]                         [Audio Track] → STT Stream → LLM
                                          ↑
                                     [Audio Track] ← TTS Stream
  ↓                                       ↓
[Speakers]          ←─────────────────────┘

WebSocket (signaling only)
  ↓                                      ↓
[WebSocket] ←──── SDP/ICE ────→ [WebSocket Server]
            ←─── Messages ────→
            ←── Tool Logs ────→

Benefits:

• Real-time audio streaming (low latency)
• No base64 encoding overhead
• Native browser media handling
• Continuous streaming (no push-to-talk requirement)
• Efficient bandwidth usage

---

Implementation Plan

Phase 1: Backend Setup

Task 1.1: Install werift-webrtc

cd /Users/moboudra/dev/voice-dev/packages/voice-assistant
npm install werift

Task 1.2: Create WebRTC Server Module

File: src/server/webrtc/peer-connection.ts

import { RTCPeerConnection, RTCSessionDescription, RTCIceCandidate } from 'werift';

export interface WebRTCPeerConfig {
  iceServers?: Array<{
    urls: string | string[];
    username?: string;
    credential?: string;
  }>;
}

export interface WebRTCPeer {
  id: string;
  connection: RTCPeerConnection;
  onAudioTrack: (track: MediaStreamTrack) => void;
  addAudioTrack: (track: MediaStreamTrack) => void;
  close: () => void;
}

export function createWebRTCPeer(config?: WebRTCPeerConfig): WebRTCPeer;

Responsibilities:

• Create RTCPeerConnection with STUN/TURN configuration
• Handle ICE candidate gathering
• Manage audio track addition/reception
• Connection state management
• Clean shutdown

Task 1.3: Create WebRTC Manager

File: src/server/webrtc/manager.ts

import type { VoiceAssistantWebSocketServer } from '../websocket-server.js';
import type { WebRTCPeer } from './peer-connection.js';

export interface WebRTCManager {
  createPeerForClient(clientId: string): Promise<WebRTCPeer>;
  handleOffer(clientId: string, offer: RTCSessionDescriptionInit): Promise<RTCSessionDescriptionInit>;
  handleAnswer(clientId: string, answer: RTCSessionDescriptionInit): Promise<void>;
  handleIceCandidate(clientId: string, candidate: RTCIceCandidateInit): Promise<void>;
  removePeer(clientId: string): void;
}

export function createWebRTCManager(wsServer: VoiceAssistantWebSocketServer): WebRTCManager;

Responsibilities:

• Manage multiple peer connections (one per client)
• Coordinate signaling via WebSocket
• Track connection state
• Clean up disconnected peers

Task 1.4: Integrate Signaling with WebSocket

File: src/server/websocket-server.ts (modifications)

Add handlers for:

• webrtc_offer - Client sends SDP offer
• webrtc_answer - Client sends SDP answer
• webrtc_ice_candidate - Client sends ICE candidate

Add methods:

• sendWebRTCSignal(clientId, signal) - Send signaling to specific client

Task 1.5: Wire Audio Input to STT

File: src/server/audio/stream-processor.ts (new)

export interface AudioStreamProcessor {
  processInputStream(track: MediaStreamTrack, onTranscript: (text: string) => void): void;
  stop(): void;
}

export function createAudioStreamProcessor(): AudioStreamProcessor;

Responsibilities:

• Receive WebRTC audio track
• Convert to format suitable for streaming STT
• Buffer audio chunks
• Feed to STT API (may need streaming STT service)
• Emit transcripts as they arrive

Challenge: OpenAI Whisper doesn't support streaming. Options:

1. Option A: Buffer audio chunks (e.g., 3-second segments) and transcribe incrementally
2. Option B: Use streaming STT service (Deepgram, AssemblyAI, Google Speech-to-Text)
3. Option C: Keep hybrid model: WebRTC for TTS output, WebSocket for STT input

Recommendation: Start with Option A (buffered chunks), evaluate Option B if latency is too high.

Task 1.6: Wire TTS Output to WebRTC

File: src/server/audio/tts-stream.ts (new)

export interface TTSAudioStream {
  sendToTrack(audioBuffer: Buffer, track: MediaStreamTrack): Promise<void>;
  close(): void;
}

export function createTTSAudioStream(): TTSAudioStream;

Responsibilities:

• Receive TTS audio buffer from OpenAI
• Convert to WebRTC audio frames (RTP packets)
• Send via WebRTC audio track to client
• Handle backpressure and buffering

---

Phase 2: Frontend Setup

Task 2.1: Create WebRTC Client Module

File: src/ui/lib/webrtc-client.ts

export interface WebRTCClient {
  initialize(): Promise<void>;
  createOffer(): Promise<RTCSessionDescriptionInit>;
  handleAnswer(answer: RTCSessionDescriptionInit): Promise<void>;
  handleIceCandidate(candidate: RTCIceCandidateInit): Promise<void>;
  addLocalAudioTrack(stream: MediaStream): void;
  onRemoteTrack(callback: (track: MediaStreamTrack) => void): void;
  close(): void;
}

export function createWebRTCClient(
  onIceCandidate: (candidate: RTCIceCandidate) => void,
  onConnectionStateChange: (state: RTCPeerConnectionState) => void
): WebRTCClient;

Responsibilities:

• Create browser RTCPeerConnection
• Manage local media stream (microphone)
• Handle remote audio track (TTS output)
• Emit ICE candidates for signaling
• Track connection state

Task 2.2: Integrate Signaling with WebSocket Hook

File: src/ui/hooks/useWebSocket.ts (modifications)

Add message handlers:

• webrtc_offer - Server sends offer (if server initiates)
• webrtc_answer - Server sends answer
• webrtc_ice_candidate - Server sends ICE candidate

Add methods:

• sendWebRTCSignal(type, data) - Send signaling to server

Task 2.3: Create WebRTC Connection Hook

File: src/ui/hooks/useWebRTC.ts (new)

export interface UseWebRTCResult {
  connectionState: RTCPeerConnectionState;
  localStream: MediaStream | null;
  remoteStream: MediaStream | null;
  startConnection: () => Promise<void>;
  stopConnection: () => void;
  error: string | null;
}

export function useWebRTC(wsClient: WebSocketClient): UseWebRTCResult;

Responsibilities:

• Manage WebRTC client lifecycle
• Coordinate with WebSocket for signaling
• Provide connection state to UI
• Handle errors and reconnection

Task 2.4: Update VoiceControls Component

File: src/ui/components/VoiceControls.tsx (modifications)

Changes:

• Remove createAudioRecorder() usage
• Remove push-to-talk button logic
• Add connection toggle: "Connect Voice" / "Disconnect Voice"
• Display connection state
• Use WebRTC local/remote streams instead of MediaRecorder

New UI states:

• Disconnected
• Connecting
• Connected (streaming)
• Error

Task 2.5: Handle Remote Audio Playback

File: src/ui/lib/webrtc-playback.ts (new)

export function playRemoteAudioTrack(track: MediaStreamTrack): HTMLAudioElement;

Responsibilities:

• Create Audio element for remote track
• Attach MediaStream to Audio element
• Auto-play TTS audio as it arrives
• Clean up on disconnect

---

Phase 3: Integration and Testing

Task 3.1: Connection Flow Testing

Test Case 1: Establish WebRTC Connection

1. Open browser UI
2. Click "Connect Voice"
3. Grant microphone permission
4. Verify: WebSocket signaling messages exchanged
5. Verify: ICE candidates gathered and exchanged
6. Verify: WebRTC connection state = "connected"
7. Verify: No errors in console

Expected Latency: < 2 seconds to establish connection

Task 3.2: Audio Input Testing

Test Case 2: Microphone to Server

1. Establish WebRTC connection
2. Speak into microphone: "list terminals"
3. Verify: Audio track received on server
4. Verify: Audio data flowing (check logs)
5. Verify: STT processing triggered
6. Verify: Transcript appears in activity log
7. Verify: LLM response generated

Expected Latency: < 1 second for STT (with buffered chunks)

Task 3.3: Audio Output Testing

Test Case 3: TTS to Browser

1. Establish WebRTC connection
2. Speak: "what terminals are available?"
3. Verify: LLM response generated
4. Verify: TTS audio synthesized
5. Verify: Audio sent via WebRTC track
6. Verify: Audio plays in browser automatically
7. Verify: Clear, natural speech quality

Expected Latency: < 1 second for TTS

Task 3.4: Full Conversation Testing

Test Case 4: Continuous Voice Conversation

1. Establish connection
2. Speak: "create a terminal called test in /tmp"
3. Wait for response (spoken)
4. Speak: "list all terminals"
5. Verify: Both interactions work smoothly
6. Verify: No audio cutoff or overlap
7. Verify: Terminal actually created in tmux

Expected Total Round-Trip: < 3 seconds

Task 3.5: Error Handling Testing

Test Case 5: Connection Failures

1. Deny microphone permission → graceful error
2. Kill WebSocket connection → WebRTC reconnect attempt
3. ICE gathering fails → show connection error
4. Send invalid SDP → handle error gracefully

Test Case 6: Audio Processing Failures

1. STT API error → show error, allow retry
2. TTS API error → text-only fallback
3. Audio codec mismatch → log warning, attempt recovery

---

WebRTC Message Types

Updated src/server/types.ts

export interface WebSocketMessage {
  type: 'activity_log' | 'status' | 'ping' | 'pong' | 'user_message' | 'assistant_chunk'
       | 'audio_chunk' | 'audio_output' | 'recording_state' | 'transcription_result'
       // New WebRTC signaling types:
       | 'webrtc_offer' | 'webrtc_answer' | 'webrtc_ice_candidate';
  payload: unknown;
}

export interface WebRTCOfferPayload {
  sdp: string;
  type: 'offer';
}

export interface WebRTCAnswerPayload {
  sdp: string;
  type: 'answer';
}

export interface WebRTCIceCandidatePayload {
  candidate: string;
  sdpMLineIndex: number | null;
  sdpMid: string | null;
}

---

Signaling Flow Diagram

Browser                                    Server
  |                                          |
  |--- Connect to WebSocket --------------->|
  |<-- WebSocket connected -----------------|
  |                                          |
  |--- Click "Connect Voice" --------------->|
  |                                          |
  |--- getUserMedia (microphone) ---------->|
  |<-- MediaStream acquired -----------------|
  |                                          |
  |--- Create RTCPeerConnection ------------>|
  |--- Add audio track --------------------->|
  |--- createOffer() ----------------------->|
  |                                          |
  |--- WS: webrtc_offer -------------------->|
  |                   SDP offer              |
  |                                          |--- Create RTCPeerConnection
  |                                          |--- setRemoteDescription(offer)
  |                                          |--- createAnswer()
  |                                          |
  |<-- WS: webrtc_answer --------------------|
  |          SDP answer                      |
  |                                          |
  |--- setRemoteDescription(answer) -------->|
  |                                          |
  |<-- WS: webrtc_ice_candidate -------------|
  |--- WS: webrtc_ice_candidate ------------>|
  |<-- WS: webrtc_ice_candidate -------------|
  |--- WS: webrtc_ice_candidate ------------>|
  |          (multiple ICE exchanges)        |
  |                                          |
  |========= WebRTC Connection Established ==|
  |                                          |
  |--- Audio streaming ---------------------->|--- Audio track received
  |                                          |--- STT processing
  |                                          |--- LLM response
  |                                          |--- TTS synthesis
  |<-- Audio streaming ----------------------|--- Send via audio track
  |                                          |

---

Configuration

Environment Variables

# Existing
OPENAI_API_KEY=sk-...
TTS_VOICE=alloy
TTS_MODEL=tts-1

# New (optional)
WEBRTC_STUN_SERVER=stun:stun.l.google.com:19302
WEBRTC_TURN_SERVER=             # Optional TURN server
WEBRTC_TURN_USERNAME=           # Optional
WEBRTC_TURN_CREDENTIAL=         # Optional

Default STUN/TURN Configuration

const defaultIceServers = [
  { urls: 'stun:stun.l.google.com:19302' },
  { urls: 'stun:stun1.l.google.com:19302' },
];

For most peer-to-peer scenarios, STUN is sufficient. TURN is only needed for restrictive NAT/firewall environments.

---

Migration Strategy

Hybrid Approach (Recommended)

To minimize risk, implement WebRTC in parallel with existing WebSocket audio:

Phase 8A: WebRTC for TTS Only

• Keep WebSocket for STT input (existing push-to-talk)
• Use WebRTC for TTS output (lower latency playback)
• Test and validate

Phase 8B: WebRTC for STT (Buffered Chunks)

• Add WebRTC audio input
• Buffer 2-3 second chunks
• Send chunks to Whisper API
• Compare latency with WebSocket approach

Phase 8C: Full WebRTC (Optional Streaming STT)

• Evaluate streaming STT services (Deepgram, AssemblyAI)
• If beneficial, replace Whisper with streaming STT
• Remove WebSocket audio code

Phase 8D: Cleanup

• Remove old WebSocket audio handlers
• Remove MediaRecorder code
• Remove audio-capture.ts and audio-playback.ts
• Update documentation

---

STT Streaming Strategy

Challenge: OpenAI Whisper is Not Streaming

Current State: Whisper requires complete audio files.

Options for WebRTC Audio Input:

1. Option A: Buffered Chunks (Simple)

   • Buffer WebRTC audio track for N seconds (e.g., 3 seconds)
   • Send buffer to Whisper API
   • Repeat for continuous transcription
   • Pros: Works with existing Whisper API, simple
   • Cons: Still has buffering delay, not true real-time

2. Option B: Streaming STT Service (Advanced)

   • Replace Whisper with streaming-capable service:
     • Deepgram: WebSocket streaming, excellent accuracy
     • AssemblyAI: Universal Streaming model
     • Google Speech-to-Text: Streaming recognition
     • Azure Speech: Real-time transcription
   • Pros: True real-time transcription, lower latency
   • Cons: Additional service, potential cost increase

3. Option C: Hybrid Model (Conservative)

   • Use WebRTC for TTS output only (low-latency playback)
   • Keep WebSocket for STT input (push-to-talk)
   • Pros: Simple migration, reuses existing STT
   • Cons: Doesn't achieve full real-time input

Recommendation: Start with Option C (hybrid), then evaluate Option A (buffered chunks). Consider Option B (streaming STT) if latency is critical.

Deepgram Integration (If Pursuing Option B)

npm install @deepgram/sdk

import { createClient } from '@deepgram/sdk';

const deepgram = createClient(process.env.DEEPGRAM_API_KEY);

const connection = deepgram.listen.live({
  model: 'nova-2',
  language: 'en-US',
  smart_format: true,
});

connection.on('transcript', (data) => {
  console.log('Transcript:', data.channel.alternatives[0].transcript);
});

// Send audio chunks from WebRTC track
audioTrack.on('data', (chunk) => {
  connection.send(chunk);
});

Note: The Deepgram SDK is already installed in the project (@deepgram/sdk: ^3.4.0).

---

Dependencies

New Dependencies

{
  "dependencies": {
    "werift": "^0.22.2"
  }
}

Existing Dependencies (No Changes)

• openai: ^4.20.0
• ws: ^8.14.2
• @deepgram/sdk: ^3.4.0 (already installed, may use for streaming STT)
• uuid: ^9.0.1

---

File Structure

src/
├── server/
│   ├── webrtc/
│   │   ├── peer-connection.ts      # NEW: WebRTC peer wrapper
│   │   ├── manager.ts               # NEW: Peer manager
│   │   └── signaling.ts             # NEW: Signaling coordination
│   ├── audio/
│   │   ├── stream-processor.ts      # NEW: Audio input processing
│   │   └── tts-stream.ts            # NEW: TTS to WebRTC track
│   ├── websocket-server.ts          # MODIFIED: Add WebRTC signaling
│   ├── types.ts                     # MODIFIED: Add WebRTC types
│   └── index.ts                     # MODIFIED: Initialize WebRTC
├── ui/
│   ├── lib/
│   │   ├── webrtc-client.ts         # NEW: Browser WebRTC client
│   │   └── webrtc-playback.ts       # NEW: Remote audio playback
│   ├── hooks/
│   │   ├── useWebRTC.ts             # NEW: WebRTC connection hook
│   │   └── useWebSocket.ts          # MODIFIED: Add signaling
│   ├── components/
│   │   └── VoiceControls.tsx        # MODIFIED: WebRTC UI
│   └── App.tsx                      # MODIFIED: Integrate WebRTC

---

Performance Expectations

Latency Comparison

Metric	WebSocket (Current)	WebRTC (Target)
STT Latency	1-3 seconds	0.5-2 seconds
TTS Latency	1-2 seconds	0.5-1 second
Total Round-Trip	3-5 seconds	1-3 seconds
Audio Quality	Good	Excellent
Bandwidth Efficiency	Low (base64)	High (binary)

Expected Improvements

• 50-60% reduction in total latency
• 30-40% reduction in bandwidth usage
• Improved audio quality (no encoding overhead)
• True streaming capability (with streaming STT)

---

Known Limitations and Risks

Technical Risks

1. werift Maturity: Library is still maturing

   • Mitigation: Extensive testing, fallback to WebSocket

2. Browser Compatibility: WebRTC support varies

   • Mitigation: Feature detection, graceful degradation

3. NAT/Firewall Issues: May require TURN server

   • Mitigation: Provide TURN configuration option

4. STT Streaming: OpenAI Whisper doesn't stream

   • Mitigation: Hybrid approach or alternative STT service

Operational Risks

1. TURN Server Costs: If needed for production

   • Mitigation: Use free STUN initially, add TURN only if needed

2. Increased Complexity: Two connection types to manage

   • Mitigation: Clear separation of concerns, good documentation

3. Debugging Difficulty: WebRTC is complex to debug

   • Mitigation: Extensive logging, use chrome://webrtc-internals/

---

Success Criteria

• ✅ WebRTC connection established reliably between browser and server
• ✅ Microphone audio streams to server via WebRTC in real-time
• ✅ TTS audio streams from server to browser via WebRTC in real-time
• ✅ Total latency < 3 seconds for full voice round-trip
• ✅ Audio quality is excellent (no artifacts, clear speech)
• ✅ WebSocket still handles text messages and tool notifications
• ✅ Connection is stable (no disconnects or audio dropouts)
• ✅ Error handling is graceful (reconnection, fallback)
• ✅ Works in Chrome and Firefox
• ✅ No regression in existing text chat functionality
• ✅ Code is maintainable and well-documented

---

Implementation Timeline

Conservative Estimate (Phased Approach)

• Phase 8A (WebRTC TTS Only): 2-3 days

  • Install werift, setup basic peer connection
  • Implement signaling
  • Wire TTS to WebRTC output
  • Test playback

• Phase 8B (WebRTC STT Buffered): 2-3 days

  • Implement audio input capture
  • Buffer and send chunks to Whisper
  • Test transcription accuracy

• Phase 8C (Streaming STT - Optional): 1-2 days

  • Integrate Deepgram or similar
  • Wire WebRTC audio to streaming STT
  • Test real-time transcription

• Phase 8D (Cleanup): 1 day

  • Remove old WebSocket audio code
  • Update documentation
  • Final testing

Total: 6-9 days

Aggressive Estimate (Full Implementation)

• All Phases: 4-5 days
  • Parallel development of client/server
  • Accept some technical debt
  • Minimal testing between phases

---

Next Steps

1. Confirm Approach: Verify architecture decisions
2. Install Dependencies: npm install werift
3. Start with Phase 8A: Implement TTS via WebRTC first
4. Iterative Testing: Test each phase before proceeding
5. Document Learnings: Update this plan as we learn

---

References

• werift-webrtc: https://github.com/shinyoshiaki/werift-webrtc
• WebRTC API (MDN): https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API
• Deepgram Streaming: https://developers.deepgram.com/docs/streaming
• WebRTC Signaling: https://blog.logrocket.com/webrtc-signaling-websocket-node-js/

---

End of Plan