paseo/packages/voice-assistant/IMPLEMENTATION_PLAN.md

Voice Assistant - Implementation Plan

Project Overview

Goal: Build a voice-controlled terminal assistant that runs locally as a single Express service.

Architecture:

• Single Express process serving everything
• Vite React UI bundled and served from Express
• WebSocket for control messages (activity log, status updates)
• Agent logic (STT/TTS/LLM pipeline using APIs)
• Daemon logic (terminal control using tmux) - same process, direct function calls
• Audio streaming solution (Phase 8)

Technology Stack

Server

• Runtime: Node.js with TypeScript
• Framework: Express
• WebSocket: ws library
• Terminal Control: tmux (via child_process)
• APIs:
  • OpenAI (GPT-4 Turbo for LLM, TTS)
  • Deepgram (Streaming STT)

Client

• Framework: React 18
• Build Tool: Vite
• Styling: CSS (custom)
• WebSocket: Native WebSocket API

Package Structure

packages/voice-assistant/
├── package.json
├── tsconfig.json              # UI TypeScript config
├── tsconfig.server.json       # Server TypeScript config
├── tsconfig.node.json         # Vite config TypeScript
├── vite.config.ts             # Vite bundler config
├── .env.example
├── .gitignore
├── IMPLEMENTATION_PLAN.md     # This file
│
├── src/
│   ├── server/
│   │   ├── index.ts                    # ✅ Express + HTTP server entry point
│   │   ├── types.ts                    # ✅ Shared server types
│   │   ├── websocket-server.ts         # ✅ WebSocket server
│   │   │
│   │   ├── agent/
│   │   │   ├── orchestrator.ts         # Agent pipeline coordinator
│   │   │   ├── stt-deepgram.ts         # Deepgram streaming STT
│   │   │   ├── llm-openai.ts           # OpenAI GPT-4 with tools
│   │   │   ├── tts-openai.ts           # OpenAI TTS (stream)
│   │   │   └── system-prompt.ts        # Load from agent-prompt.md
│   │   │
│   │   └── daemon/
│   │       ├── terminal-manager.ts     # Terminal control API
│   │       ├── tmux.ts                 # tmux primitives (from mcp-server)
│   │       └── tool-definitions.ts     # LLM tool schemas
│   │
│   └── ui/
│       ├── index.html                  # ✅ HTML entry point
│       ├── main.tsx                    # ✅ React root
│       ├── App.tsx                     # ✅ Main app component
│       ├── App.css                     # ✅ App styles
│       ├── index.css                   # ✅ Global styles
│       │
│       ├── components/
│       │   ├── VoiceInterface.tsx      # Main voice UI
│       │   ├── ActivityLog.tsx         # Conversation log
│       │   ├── ConnectionIndicator.tsx # Connection status
│       │   └── AudioVisualizer.tsx     # Audio waveform
│       │
│       ├── hooks/
│       │   ├── useWebSocket.ts         # ✅ WebSocket client
│       │   └── useWebRTC.ts            # WebRTC client (Phase 8)
│       │
│       └── lib/
│           └── audio-utils.ts          # Audio processing utilities
│
└── dist/
    ├── server/    # Compiled TypeScript
    └── ui/        # Built Vite app

---

Implementation Phases

✅ Phase 1: Foundation (COMPLETED)

Tasks:

1. ✅ Create package structure and configuration files
2. ✅ Set up package.json with dependencies and scripts
3. ✅ Create TypeScript configurations (tsconfig.json, tsconfig.server.json)
4. ✅ Create Vite configuration (vite.config.ts)
5. ✅ Create basic Express server (src/server/index.ts)
6. ✅ Create basic Vite React app (src/ui/)
7. ✅ Install dependencies and test dev servers

Key Files Created:

• src/server/index.ts - Express server with production static file serving
• src/server/types.ts - Shared TypeScript types
• src/ui/App.tsx - Main React component
• src/ui/index.html - HTML entry point
• vite.config.ts - Vite build configuration

Development Workflow:

npm run dev        # Runs both servers (concurrently)
npm run dev:server # Express on port 3000
npm run dev:ui     # Vite on port 5173

Production Build:

npm run build      # Builds UI + server
npm start          # Serves from dist/

---

✅ Phase 2: WebSocket Communication (COMPLETED)

Tasks:

1. ✅ Add WebSocket server (websocket-server.ts)
2. ✅ Add WebSocket client hook (useWebSocket.ts)
3. ✅ Test WebSocket ping/pong communication

Key Features:

• WebSocket server on /ws path
• Client-side useWebSocket hook with auto-reconnect
• Message handler registration system
• Ping/pong messaging for connection testing
• Activity log UI component
• Real-time connection status indicator

Key Files Created:

• src/server/websocket-server.ts - VoiceAssistantWebSocketServer class
• src/ui/hooks/useWebSocket.ts - Client WebSocket hook
• Updated src/ui/App.tsx - WebSocket integration & activity log

WebSocket Message Types:

interface WebSocketMessage {
  type: 'activity_log' | 'status' | 'webrtc_signal' | 'ping' | 'pong';
  payload: unknown;
}

Testing:

1. Start both servers: npm run dev
2. Open http://localhost:5173
3. Click "Send Ping" button
4. See "Received pong from server" in activity log

---

⏳ Phase 3: Terminal Control (Daemon)

Tasks:

1. ⏳ Copy tmux.ts from mcp-server package
2. ⏳ Create terminal-manager.ts with tool functions
3. ⏳ Create tool-definitions.ts for LLM schemas
4. ⏳ Test terminal operations (list, create, capture)

Objectives:

• Reuse existing tmux primitives from mcp-server
• Create high-level terminal management API
• Define LLM tool schemas for terminal operations
• Test terminal creation, command execution, output capture

Terminal Tools (7 core operations):

// Tool functions to implement in terminal-manager.ts
1. listTerminals() → TerminalInfo[]
2. createTerminal(name, workingDirectory, initialCommand?)
3. captureTerminal(terminalId, lines?, wait?)
4. sendText(terminalId, text, pressEnter?, return_output?)
5. sendKeys(terminalId, keys, repeat?, return_output?)
6. renameTerminal(terminalId, name)
7. killTerminal(terminalId)

Terminal Model:

• Default tmux session: voice-dev
• Terminal = tmux window (single pane)
• Terminal ID = window ID (format: @123)
• Working directory set on creation

---

⏳ Phase 4: LLM Integration

Tasks:

1. ⏳ Copy agent-prompt.md from agent-python
2. ⏳ Create system-prompt.ts to load prompt file
3. ⏳ Create llm-openai.ts (GPT-4 with function calling)
4. ⏳ Test LLM integration with tool calls

Objectives:

• Reuse existing system prompt from Python agent
• Implement OpenAI GPT-4 Turbo with function calling
• Define terminal tools as OpenAI functions
• Test: "create a terminal called test" → executes createTerminal()

LLM Configuration:

{
  model: "gpt-4-turbo",
  tools: terminalToolDefinitions,
  tool_choice: "auto",
  stream: true
}

---

⏳ Phase 5: Agent Orchestrator

Tasks:

1. ⏳ Create agent orchestrator.ts
2. ⏳ Wire text input → LLM → tool execution → response
3. ⏳ Emit activity log events to WebSocket
4. ⏳ Test end-to-end text-based commands

Objectives:

• Create main agent loop
• Handle: user input → LLM → tool calls → terminal → response
• Broadcast activity to WebSocket clients
• Test complete text-based interaction flow

Agent Loop:

1. Receive user input (text/transcript)
2. Add to conversation context
3. Call LLM with tools
4. If tool calls:
   a. Execute each tool via terminal-manager
   b. Add results to context
   c. Call LLM again with results
5. Return final response
6. Broadcast all events to WebSocket

---

⏳ Phase 6: Speech-to-Text (Deepgram)

Tasks:

1. ⏳ Add Deepgram STT integration (stt-deepgram.ts)
2. ⏳ Test STT with audio input

Objectives:

• Implement Deepgram streaming STT
• Handle audio chunks from client
• Return transcript fragments in real-time
• Detect end-of-utterance

Deepgram Configuration:

{
  model: "nova-2",
  language: "en",
  smart_format: true,
  punctuate: true,
  interim_results: true
}

---

⏳ Phase 7: Text-to-Speech (OpenAI)

Tasks:

1. ⏳ Add OpenAI TTS integration (tts-openai.ts)
2. ⏳ Test TTS with text input

Objectives:

• Implement OpenAI TTS API
• Convert text responses to audio
• Stream audio chunks to client
• Handle playback in browser

TTS Configuration:

{
  model: "tts-1",
  voice: "alloy",
  response_format: "pcm"  // or "opus" for streaming
}

---

⏳ Phase 8: Audio Streaming

Tasks:

1. ⏳ Research WebRTC alternatives for audio streaming
2. ⏳ Implement audio streaming solution
3. ⏳ Test end-to-end voice interaction

Objectives:

• Implement bidirectional audio streaming
• Browser → Server: Microphone input
• Server → Browser: TTS output
• Consider: WebRTC (browser native) or WebSocket audio

Note:

• wrtc package removed due to native compilation issues
• Alternative options:
  1. WebRTC (browser-to-browser via signaling) - Most robust
  2. WebSocket audio streaming - Simpler, less efficient
  3. simple-peer - WebRTC wrapper library
  4. MediaRecorder API - Send audio chunks via WebSocket

---

⏳ Phase 9: UI Polish

Tasks:

1. ⏳ Create ActivityLog component (proper component)
2. ⏳ Create ConnectionIndicator component
3. ⏳ Add mute/unmute controls
4. ⏳ Polish UI and test full application

Objectives:

• Extract activity log into reusable component
• Add microphone mute/unmute button
• Add visual audio level indicator
• Improve styling and UX
• End-to-end testing

---

Environment Variables

.env file (copy from .env.example):

# OpenAI API Key (for GPT-4 and TTS)
OPENAI_API_KEY=sk-...

# Deepgram API Key (for streaming STT)
DEEPGRAM_API_KEY=...

# Server Configuration
PORT=3000
NODE_ENV=development

---

Development Commands

# Install dependencies
npm install

# Development (runs both servers)
npm run dev

# Development (individual servers)
npm run dev:server   # Express on port 3000
npm run dev:ui       # Vite on port 5173

# Build
npm run build:ui     # Build Vite UI → dist/ui/
npm run build:server # Compile TS → dist/server/
npm run build        # Build both

# Production
npm start            # Serve from dist/

# Type checking
npm run typecheck    # Check both UI and server types

---

Testing Strategy

Phase 1-2 (Foundation + WebSocket)

• ✅ Manual: Start servers, check endpoints
• ✅ Manual: Open UI, test ping/pong

Phase 3 (Terminal Control)

• Manual: Call terminal-manager functions from server
• Manual: Verify tmux windows created/destroyed
• Manual: Check command execution and output capture

Phase 4 (LLM)

• Manual: Send text prompts, check tool calls
• Manual: Verify tool execution results in context

Phase 5 (Orchestrator)

• Manual: Send "create a terminal called test"
• Verify: LLM calls createTerminal tool
• Verify: Activity log shows tool call and result

Phase 6-7 (STT/TTS)

• Manual: Record audio, check transcript
• Manual: Send text, check audio playback

Phase 8 (Audio Streaming)

• Manual: Speak into microphone
• Verify: Real-time transcription
• Verify: Agent response plays back

Phase 9 (Full E2E)

• Manual: Complete voice interaction loop
• Test: Various terminal commands via voice
• Test: Error handling and edge cases

---

Key Decisions & Trade-offs

1. Single Process Architecture

• Decision: Run agent + daemon in same process
• Why: Simpler V1, avoid IPC complexity
• Future: Can split into microservices later

2. OpenAI All-in-One

• Decision: Use OpenAI for both LLM and TTS, Deepgram for STT
• Why: Simpler integration, fewer providers
• Note: Deepgram required because OpenAI Whisper doesn't support streaming

3. Removed wrtc Package

• Decision: Defer WebRTC server implementation to Phase 8
• Why: wrtc has native compilation issues
• Alternative: Will use browser-native WebRTC or WebSocket audio

4. Auto-Execute Terminal Commands

• Decision: No permission prompts in V1
• Why: Faster development, simpler UX
• Future: Add permission system in V2

5. Reuse Existing Code

• Decision: Copy tmux.ts and agent-prompt.md from existing packages
• Why: Proven, tested code
• Benefit: Consistent terminal model across packages

---

Current Progress

✅ Completed: 10/33 tasks (30%)

• Phase 1: Foundation (7 tasks)
• Phase 2: WebSocket (3 tasks)

⏳ In Progress: Phase 3 (Terminal Control)

📋 Remaining: 23 tasks

• Phase 3: 4 tasks
• Phase 4: 4 tasks
• Phase 5: 4 tasks
• Phase 6: 2 tasks
• Phase 7: 2 tasks
• Phase 8: 3 tasks
• Phase 9: 4 tasks

---

Next Steps

1. Phase 3: Copy tmux.ts from mcp-server
2. Phase 3: Create terminal-manager.ts with 7 core tools
3. Phase 3: Define OpenAI function schemas for tools
4. Phase 3: Test terminal operations manually
5. Phase 4: Copy agent-prompt.md and set up LLM

---

References

• Existing Packages:

  • packages/mcp-server/src/tmux.ts - Terminal control primitives
  • packages/agent-python/agent-prompt.md - System prompt
  • packages/web/app/hooks/use-livekit-voice.ts - LiveKit reference

• APIs:

  • OpenAI API Docs
  • Deepgram API Docs
  • WebRTC API Docs

---

Notes

• This is a V1 implementation focused on getting a working prototype
• Prioritize simplicity over optimization
• Each phase should be independently testable
• WebSocket infrastructure is ready for real-time updates
• Audio streaming is the most complex remaining piece