paseo/.claude/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

Voice-controlled terminal assistant using OpenAI's Realtime API (via Whisper STT, GPT-4, and TTS). Backend server with Expo app for all platforms (iOS, Android, and Web).

Monorepo Structure

This is an npm workspace monorepo:

• Root: Workspace configuration and shared TypeScript config
• packages/server: Backend server (Express + WebSocket API)
• packages/app: Cross-platform app (Expo - iOS, Android, Web)

Development Commands

All commands run from the repository root:

# Development
npm run dev              # Start backend server (port 6767)
npm run dev:app          # Start Expo dev server

# Build everything
npm run build

# Type checking
npm run typecheck

# Production
npm run start            # Start backend server

# Platform-specific
npm run android          # Run on Android
npm run ios              # Run on iOS
npm run web              # Run web version

Server Package Commands

From packages/server/:

# Development
npm run dev              # Watch mode: Express server (port 6767)

# Building
npm run build            # TypeScript → dist/server

# Production
npm start                # Run built server

# Type checking
npm run typecheck        # Check server TypeScript

App Package Commands

From packages/app/:

# Development - IMPORTANT: Must run in a TTY for interactive QR code
npx expo start           # Start Expo dev server (Metro bundler on port 8081)

# Running in tmux (recommended for proper TTY)
tmux new-session -d -s expo -c /Users/moboudra/dev/voice-dev/packages/app "npx expo start"
tmux attach -t expo      # Attach to see QR code and interact

# View expo session output without attaching
tmux capture-pane -t expo -p | tail -30

# Building
npx expo prebuild        # Generate native iOS/Android projects
eas build --platform ios # Build for iOS (requires EAS)
eas build --platform android # Build for Android (requires EAS)

# Type checking
npm run typecheck        # Check TypeScript

Why tmux for Expo:

• Expo requires a TTY to display the interactive QR code and menu
• Running without a TTY results in errors or missing output
• tmux provides a persistent TTY session that survives disconnects

Architecture

Overall Structure

Two-package architecture:

1. Server Package (packages/server/src/server/):

   • Express HTTP server (port 6767)

   • WebSocket server for bidirectional communication

   • No UI (API only)

   • Agent Layer (agent/):

     • orchestrator.ts: Main conversation manager
     • llm-openai.ts: GPT-4 streaming with tool execution
     • stt-openai.ts: Whisper transcription
     • tts-openai.ts: OpenAI TTS synthesis
     • system-prompt.ts: Agent instructions (voice-first UX focused)

   • Daemon Layer (daemon/):

     • terminal-manager.ts: High-level terminal operations
     • tmux.ts: Low-level tmux integration
     • tool-definitions.ts: OpenAI function schemas for LLM

2. App Package (packages/app/):

   • Expo (React Native)
   • Cross-platform: iOS, Android, Web
   • WebSocket client for real-time communication
   • Audio capture and playback handling

Key Architecture Decisions

Terminal Model:

• Uses tmux session "voice-dev" with windows-as-terminals
• Each terminal = one tmux window with single pane
• Terminal ID = tmux window ID (format: @123)
• Terminal names must be unique

Voice-First Design:

• All responses designed for TTS output
• STT errors handled intelligently (see agent-prompt.md)
• No visual feedback assumptions

Streaming Architecture:

• Audio: App � WebSocket � Whisper STT � LLM � TTS � WebSocket � App
• Text chunks streamed for real-time response
• Tool calls logged as activity events

Data Flow

1. Voice Input: App captures audio � sends chunks via WebSocket � STT transcribes � orchestrator processes
2. LLM Processing: Orchestrator streams GPT-4 responses � executes tool calls � accumulates response
3. Voice Output: Response � TTS synthesis � base64 audio � WebSocket � app playback

TypeScript Configuration

Two separate configs:

1. packages/server/tsconfig.server.json: Server build config (NodeNext modules, outputs to dist/server)
2. packages/app/tsconfig.json: App config (extends Expo base, includes path mappings to server types)

Module System:

• Server: ES modules with .js extensions in imports (NodeNext resolution)
• App: Expo bundler (imports server types via @server/* path alias)

Environment Variables

Required in packages/server/.env:

OPENAI_API_KEY=sk-...     # Required for GPT-4, Whisper, and TTS

# Optional TTS configuration
TTS_VOICE=alloy           # alloy|echo|fable|onyx|nova|shimmer
TTS_MODEL=tts-1           # tts-1 (fast) | tts-1-hd (quality)

# Server config
PORT=3000
NODE_ENV=development

Terminal Control (MCP-like Tools)

The LLM has access to terminal operations via function calling:

• list_terminals(): Get all terminals with status
• create_terminal(name, workingDirectory, initialCommand?): Create new terminal
• send_text(terminalId, text, pressEnter, captureOutput?): Execute commands
• capture_terminal(terminalId, lines?, wait?): Read terminal output
• kill_terminal(terminalId): Remove terminal

See src/server/daemon/tool-definitions.ts for complete schemas.

WebSocket Protocol

Message types (see src/server/types.ts):

Client � Server:

• user_message: Text input
• audio_chunk: Audio data (base64, includes format and isLast flag)
• recording_state: Recording started/stopped
• ping: Keepalive

Server � Client:

• activity_log: Conversation events (transcript, tool calls, errors)
• assistant_chunk: Streaming response text
• audio_output: Complete TTS audio (base64 MP3)
• transcription_result: STT output
• status: Connection state
• pong: Keepalive response

Agent System Prompt

Located in agent-prompt.md (32KB) - loaded by system-prompt.ts.

Critical aspects:

• Voice-first interaction guidelines
• STT error handling strategies
• Terminal operation best practices
• Response brevity requirements (user is on phone)

Testing

Currently manual testing:

1. Start backend server: npm run dev
2. Start Expo app: npm run dev:app
3. Test WebSocket connection in app
4. Test voice recording/playback
5. Test terminal operations via API endpoints

Test scripts:

• test-llm.sh: Direct LLM streaming test
• test-orchestrator.js: Conversation flow test

File Organization

packages/voice-assistant/
�� src/
   �� server/
      �� agent/          # LLM, STT, TTS, orchestration
      �� daemon/         # Terminal management + tool schemas
      �� index.ts        # Express + WebSocket server
      �� websocket-server.ts
      �� types.ts        # Shared server types
   �� ui/
       �� components/     # React components
       �� hooks/          # WebSocket, audio hooks
       �� lib/            # Audio capture/playback utilities
       �� App.tsx
       �� main.tsx
�� dist/                   # Build output (gitignored)
�� agent-prompt.md         # System prompt source
�� package.json
�� tsconfig.json           # UI TypeScript config
�� tsconfig.server.json    # Server TypeScript config
�� vite.config.ts

Common Tasks

Adding a new terminal tool:

1. Add function schema to daemon/tool-definitions.ts
2. Implement handler in agent/llm-openai.ts executeTool function
3. If needed, add low-level tmux operation to daemon/tmux.ts

Modifying agent behavior:

1. Edit agent-prompt.md for instructions
2. Restart server (prompt loaded on startup)

Changing audio formats:

1. STT: Modify agent/stt-openai.ts (currently accepts webm/ogg/mp3)
2. TTS: Change TTS_MODEL env var or tts-openai.ts config

WebSocket message types:

1. Add type to types.ts WebSocketMessage union
2. Add handler in websocket-server.ts or client hooks

Known Patterns

• No barrel files: Direct imports only (no index.ts exports)
• Function declarations: Always use function not arrow functions for exports
• Error handling: Let errors bubble to Express/WebSocket error handlers
• Tool execution: Synchronous within LLM streaming loop (tools modify conversation state)
• Conversation cleanup: Auto-cleanup runs every 10 minutes (60 min TTL)
• Never run dev servers: User will test manually - don't start servers unless explicitly requested