paseo/.claude/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

Voice-controlled terminal assistant using OpenAI's Realtime API (via Whisper STT, GPT-4, and TTS). Single-service architecture combining Express server, React UI, and tmux-based terminal management.

## Monorepo Structure

This is an npm workspace monorepo:

- **Root**: Workspace configuration and shared TypeScript config
- **packages/voice-assistant**: Main application (only active package currently)

## Development Commands

All commands run from the repository root:

```bash
# Development (starts Express server with integrated Vite middleware)
npm run dev

# Build everything
npm run build

# Type checking
npm run typecheck

# Production
npm run start
```

### Voice Assistant Package Commands

From `packages/voice-assistant/`:

```bash
# Development
npm run dev              # Watch mode: Express server + Vite HMR
npm run dev:server       # Express server only (port 3000)

# Building
npm run build            # Build both UI and server
npm run build:ui         # Vite build � dist/ui
npm run build:server     # TypeScript � dist/server

# Production
npm start                # Run built server

# Type checking
npm run typecheck        # Check both UI and server
```

## Architecture

### Overall Structure

Single service with three main subsystems:

1. **Server Layer** (`src/server/`):

   - Express HTTP server (port 3000)
   - WebSocket server for bidirectional communication
   - Vite middleware in development, static files in production

2. **Agent Layer** (`src/server/agent/`):

   - `orchestrator.ts`: Main conversation manager
   - `llm-openai.ts`: GPT-4 streaming with tool execution
   - `stt-openai.ts`: Whisper transcription
   - `tts-openai.ts`: OpenAI TTS synthesis
   - `system-prompt.ts`: Agent instructions (voice-first UX focused)

3. **Daemon Layer** (`src/server/daemon/`):

   - `terminal-manager.ts`: High-level terminal operations
   - `tmux.ts`: Low-level tmux integration
   - `tool-definitions.ts`: OpenAI function schemas for LLM

4. **UI Layer** (`src/ui/`):
   - React 18 + Vite
   - WebSocket client for real-time communication
   - Audio capture and playback handling

### Key Architecture Decisions

**Terminal Model**:

- Uses tmux session "voice-dev" with windows-as-terminals
- Each terminal = one tmux window with single pane
- Terminal ID = tmux window ID (format: `@123`)
- Terminal names must be unique

**Voice-First Design**:

- All responses designed for TTS output
- STT errors handled intelligently (see `agent-prompt.md`)
- No visual feedback assumptions

**Streaming Architecture**:

- Audio: Browser � WebSocket � Whisper STT � LLM � TTS � WebSocket � Browser
- Text chunks streamed for real-time response
- Tool calls logged as activity events

### Data Flow

1. **Voice Input**: Browser captures audio � sends chunks via WebSocket � STT transcribes � orchestrator processes
2. **LLM Processing**: Orchestrator streams GPT-4 responses � executes tool calls � accumulates response
3. **Voice Output**: Response � TTS synthesis � base64 audio � WebSocket � browser playback

## TypeScript Configuration

**Three separate configs:**

1. `tsconfig.base.json`: Root workspace defaults (ES2020, ESNext modules)
2. `packages/voice-assistant/tsconfig.json`: UI build config (React JSX, bundler mode, noEmit)
3. `packages/voice-assistant/tsconfig.server.json`: Server build config (NodeNext modules, outputs to dist/server)

**Module System**:

- Server: ES modules with `.js` extensions in imports (NodeNext resolution)
- UI: Vite bundler mode (no file extensions needed)

## Environment Variables

Required in `packages/voice-assistant/.env`:

```bash
OPENAI_API_KEY=sk-...     # Required for GPT-4, Whisper, and TTS

# Optional TTS configuration
TTS_VOICE=alloy           # alloy|echo|fable|onyx|nova|shimmer
TTS_MODEL=tts-1           # tts-1 (fast) | tts-1-hd (quality)

# Server config
PORT=3000
NODE_ENV=development
```

## Terminal Control (MCP-like Tools)

The LLM has access to terminal operations via function calling:

- `list_terminals()`: Get all terminals with status
- `create_terminal(name, workingDirectory, initialCommand?)`: Create new terminal
- `send_text(terminalId, text, pressEnter, captureOutput?)`: Execute commands
- `capture_terminal(terminalId, lines?, wait?)`: Read terminal output
- `kill_terminal(terminalId)`: Remove terminal

See `src/server/daemon/tool-definitions.ts` for complete schemas.

## WebSocket Protocol

Message types (see `src/server/types.ts`):

**Client � Server**:

- `user_message`: Text input
- `audio_chunk`: Audio data (base64, includes format and isLast flag)
- `recording_state`: Recording started/stopped
- `ping`: Keepalive

**Server � Client**:

- `activity_log`: Conversation events (transcript, tool calls, errors)
- `assistant_chunk`: Streaming response text
- `audio_output`: Complete TTS audio (base64 MP3)
- `transcription_result`: STT output
- `status`: Connection state
- `pong`: Keepalive response

## Agent System Prompt

Located in `agent-prompt.md` (32KB) - loaded by `system-prompt.ts`.

**Critical aspects**:

- Voice-first interaction guidelines
- STT error handling strategies
- Terminal operation best practices
- Response brevity requirements (user is on phone)

## Testing

Currently manual testing:

1. Start dev server: `npm run dev`
2. Open http://localhost:3000
3. Test WebSocket connection (status indicator)
4. Test voice recording/playback
5. Test terminal operations via `/api/test-terminal`

Test scripts:

- `test-llm.sh`: Direct LLM streaming test
- `test-orchestrator.js`: Conversation flow test

## File Organization

```
packages/voice-assistant/
 src/
    server/
       agent/          # LLM, STT, TTS, orchestration
       daemon/         # Terminal management + tool schemas
       index.ts        # Express + WebSocket server
       websocket-server.ts
       types.ts        # Shared server types
    ui/
        components/     # React components
        hooks/          # WebSocket, audio hooks
        lib/            # Audio capture/playback utilities
        App.tsx
        main.tsx
 dist/                   # Build output (gitignored)
 agent-prompt.md         # System prompt source
 package.json
 tsconfig.json           # UI TypeScript config
 tsconfig.server.json    # Server TypeScript config
 vite.config.ts
```

## Common Tasks

**Adding a new terminal tool:**

1. Add function schema to `daemon/tool-definitions.ts`
2. Implement handler in `agent/llm-openai.ts` executeTool function
3. If needed, add low-level tmux operation to `daemon/tmux.ts`

**Modifying agent behavior:**

1. Edit `agent-prompt.md` for instructions
2. Restart server (prompt loaded on startup)

**Changing audio formats:**

1. STT: Modify `agent/stt-openai.ts` (currently accepts webm/ogg/mp3)
2. TTS: Change TTS_MODEL env var or `tts-openai.ts` config

**WebSocket message types:**

1. Add type to `types.ts` WebSocketMessage union
2. Add handler in `websocket-server.ts` or client hooks

## Known Patterns

- **No barrel files**: Direct imports only (no index.ts exports)
- **Function declarations**: Always use `function` not arrow functions for exports
- **Error handling**: Let errors bubble to Express/WebSocket error handlers
- **Tool execution**: Synchronous within LLM streaming loop (tools modify conversation state)
- **Conversation cleanup**: Auto-cleanup runs every 10 minutes (60 min TTL)
- never run dev server, user will test manually