paseo/.claude/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

Voice-controlled terminal assistant using OpenAI's Realtime API (via Whisper STT, GPT-4, and TTS). Single-service architecture combining Express server, React UI, and tmux-based terminal management.

Monorepo Structure

This is an npm workspace monorepo:

• Root: Workspace configuration and shared TypeScript config
• packages/voice-assistant: Backend server (Express + WebSocket + Vite UI)
• packages/voice-mobile: React Native mobile app (Expo)

Development Commands

All commands run from the repository root:

# Development (starts Express server with integrated Vite middleware)
npm run dev

# Build everything
npm run build

# Type checking
npm run typecheck

# Production
npm run start

Voice Assistant Package Commands

From packages/voice-assistant/:

# Development
npm run dev              # Watch mode: Express server + Vite HMR (port 6767)
npm run dev:server       # Express server only (port 6767)

# Building
npm run build            # Build both UI and server
npm run build:ui         # Vite build → dist/ui
npm run build:server     # TypeScript → dist/server

# Production
npm start                # Run built server

# Type checking
npm run typecheck        # Check both UI and server

Voice Mobile Package Commands

From packages/voice-mobile/:

# Development - IMPORTANT: Must run in a TTY for interactive QR code
npx expo start           # Start Expo dev server (Metro bundler on port 8081)

# Running in tmux (recommended for proper TTY)
tmux new-session -d -s expo -c /Users/moboudra/dev/voice-dev/packages/voice-mobile "npx expo start"
tmux attach -t expo      # Attach to see QR code and interact

# View expo session output without attaching
tmux capture-pane -t expo -p | tail -30

# Building
npx expo prebuild        # Generate native iOS/Android projects
eas build --platform ios # Build for iOS (requires EAS)
eas build --platform android # Build for Android (requires EAS)

# Type checking
npm run typecheck        # Check TypeScript

Why tmux for Expo:

• Expo requires a TTY to display the interactive QR code and menu
• Running without a TTY results in errors or missing output
• tmux provides a persistent TTY session that survives disconnects

Architecture

Overall Structure

Single service with three main subsystems:

1. Server Layer (src/server/):

   • Express HTTP server (port 3000)
   • WebSocket server for bidirectional communication
   • Vite middleware in development, static files in production

2. Agent Layer (src/server/agent/):

   • orchestrator.ts: Main conversation manager
   • llm-openai.ts: GPT-4 streaming with tool execution
   • stt-openai.ts: Whisper transcription
   • tts-openai.ts: OpenAI TTS synthesis
   • system-prompt.ts: Agent instructions (voice-first UX focused)

3. Daemon Layer (src/server/daemon/):

   • terminal-manager.ts: High-level terminal operations
   • tmux.ts: Low-level tmux integration
   • tool-definitions.ts: OpenAI function schemas for LLM

4. UI Layer (src/ui/):

   • React 18 + Vite
   • WebSocket client for real-time communication
   • Audio capture and playback handling

Key Architecture Decisions

Terminal Model:

• Uses tmux session "voice-dev" with windows-as-terminals
• Each terminal = one tmux window with single pane
• Terminal ID = tmux window ID (format: @123)
• Terminal names must be unique

Voice-First Design:

• All responses designed for TTS output
• STT errors handled intelligently (see agent-prompt.md)
• No visual feedback assumptions

Streaming Architecture:

• Audio: Browser � WebSocket � Whisper STT � LLM � TTS � WebSocket � Browser
• Text chunks streamed for real-time response
• Tool calls logged as activity events

Data Flow

1. Voice Input: Browser captures audio � sends chunks via WebSocket � STT transcribes � orchestrator processes
2. LLM Processing: Orchestrator streams GPT-4 responses � executes tool calls � accumulates response
3. Voice Output: Response � TTS synthesis � base64 audio � WebSocket � browser playback

TypeScript Configuration

Three separate configs:

1. tsconfig.base.json: Root workspace defaults (ES2020, ESNext modules)
2. packages/voice-assistant/tsconfig.json: UI build config (React JSX, bundler mode, noEmit)
3. packages/voice-assistant/tsconfig.server.json: Server build config (NodeNext modules, outputs to dist/server)

Module System:

• Server: ES modules with .js extensions in imports (NodeNext resolution)
• UI: Vite bundler mode (no file extensions needed)

Environment Variables

Required in packages/voice-assistant/.env:

OPENAI_API_KEY=sk-...     # Required for GPT-4, Whisper, and TTS

# Optional TTS configuration
TTS_VOICE=alloy           # alloy|echo|fable|onyx|nova|shimmer
TTS_MODEL=tts-1           # tts-1 (fast) | tts-1-hd (quality)

# Server config
PORT=3000
NODE_ENV=development

Terminal Control (MCP-like Tools)

The LLM has access to terminal operations via function calling:

• list_terminals(): Get all terminals with status
• create_terminal(name, workingDirectory, initialCommand?): Create new terminal
• send_text(terminalId, text, pressEnter, captureOutput?): Execute commands
• capture_terminal(terminalId, lines?, wait?): Read terminal output
• kill_terminal(terminalId): Remove terminal

See src/server/daemon/tool-definitions.ts for complete schemas.

WebSocket Protocol

Message types (see src/server/types.ts):

Client � Server:

• user_message: Text input
• audio_chunk: Audio data (base64, includes format and isLast flag)
• recording_state: Recording started/stopped
• ping: Keepalive

Server � Client:

• activity_log: Conversation events (transcript, tool calls, errors)
• assistant_chunk: Streaming response text
• audio_output: Complete TTS audio (base64 MP3)
• transcription_result: STT output
• status: Connection state
• pong: Keepalive response

Agent System Prompt

Located in agent-prompt.md (32KB) - loaded by system-prompt.ts.

Critical aspects:

• Voice-first interaction guidelines
• STT error handling strategies
• Terminal operation best practices
• Response brevity requirements (user is on phone)

Testing

Currently manual testing:

1. Start dev server: npm run dev
2. Open http://localhost:3000
3. Test WebSocket connection (status indicator)
4. Test voice recording/playback
5. Test terminal operations via /api/test-terminal

Test scripts:

• test-llm.sh: Direct LLM streaming test
• test-orchestrator.js: Conversation flow test

File Organization

packages/voice-assistant/
�� src/
   �� server/
      �� agent/          # LLM, STT, TTS, orchestration
      �� daemon/         # Terminal management + tool schemas
      �� index.ts        # Express + WebSocket server
      �� websocket-server.ts
      �� types.ts        # Shared server types
   �� ui/
       �� components/     # React components
       �� hooks/          # WebSocket, audio hooks
       �� lib/            # Audio capture/playback utilities
       �� App.tsx
       �� main.tsx
�� dist/                   # Build output (gitignored)
�� agent-prompt.md         # System prompt source
�� package.json
�� tsconfig.json           # UI TypeScript config
�� tsconfig.server.json    # Server TypeScript config
�� vite.config.ts

Common Tasks

Adding a new terminal tool:

1. Add function schema to daemon/tool-definitions.ts
2. Implement handler in agent/llm-openai.ts executeTool function
3. If needed, add low-level tmux operation to daemon/tmux.ts

Modifying agent behavior:

1. Edit agent-prompt.md for instructions
2. Restart server (prompt loaded on startup)

Changing audio formats:

1. STT: Modify agent/stt-openai.ts (currently accepts webm/ogg/mp3)
2. TTS: Change TTS_MODEL env var or tts-openai.ts config

WebSocket message types:

1. Add type to types.ts WebSocketMessage union
2. Add handler in websocket-server.ts or client hooks

Known Patterns

• No barrel files: Direct imports only (no index.ts exports)
• Function declarations: Always use function not arrow functions for exports
• Error handling: Let errors bubble to Express/WebSocket error handlers
• Tool execution: Synchronous within LLM streaming loop (tools modify conversation state)
• Conversation cleanup: Auto-cleanup runs every 10 minutes (60 min TTL)
• Never run dev servers: User will test manually - don't start servers unless explicitly requested