# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

Voice-controlled terminal assistant using OpenAI's Realtime API (via Whisper STT, GPT-4, and TTS). Backend server with Expo app for all platforms (iOS, Android, and Web).

## Monorepo Structure

This is an npm workspace monorepo:

- **Root**: Workspace configuration and shared TypeScript config
- **packages/server**: Backend server (Express + WebSocket API)
- **packages/app**: Cross-platform app (Expo - iOS, Android, Web)

## Development Commands

All commands run from the repository root:

```bash
# Development
npm run dev              # Start backend server (port 6767)
npm run dev:app          # Start Expo dev server

# Build everything
npm run build

# Type checking
npm run typecheck

# Production
npm run start            # Start backend server

# Platform-specific
npm run android          # Run on Android
npm run ios              # Run on iOS
npm run web              # Run web version
```

### Server Package Commands

From `packages/server/`:

```bash
# Development
npm run dev              # Watch mode: Express server (port 6767)

# Building
npm run build            # TypeScript → dist/server

# Production
npm start                # Run built server

# Type checking
npm run typecheck        # Check server TypeScript
```

### App Package Commands

From `packages/app/`:

```bash
# Development - IMPORTANT: Must run in a TTY for interactive QR code
npx expo start           # Start Expo dev server (Metro bundler on port 8081)

# Running in tmux (recommended for proper TTY)
tmux new-session -d -s expo -c /Users/moboudra/dev/voice-dev/packages/app "npx expo start"
tmux attach -t expo      # Attach to see QR code and interact

# View expo session output without attaching
tmux capture-pane -t expo -p | tail -30

# Building
npx expo prebuild        # Generate native iOS/Android projects
eas build --platform ios # Build for iOS (requires EAS)
eas build --platform android # Build for Android (requires EAS)

# Type checking
npm run typecheck        # Check TypeScript
```

**Why tmux for Expo:**
- Expo requires a TTY to display the interactive QR code and menu
- Running without a TTY results in errors or missing output
- tmux provides a persistent TTY session that survives disconnects

## Architecture

### Overall Structure

Two-package architecture:

1. **Server Package** (`packages/server/src/server/`):

   - Express HTTP server (port 6767)
   - WebSocket server for bidirectional communication
   - No UI (API only)

   - **Agent Layer** (`agent/`):
     - `orchestrator.ts`: Main conversation manager
     - `llm-openai.ts`: GPT-4 streaming with tool execution
     - `stt-openai.ts`: Whisper transcription
     - `tts-openai.ts`: OpenAI TTS synthesis
     - `system-prompt.ts`: Agent instructions (voice-first UX focused)

   - **Daemon Layer** (`daemon/`):
     - `terminal-manager.ts`: High-level terminal operations
     - `tmux.ts`: Low-level tmux integration
     - `tool-definitions.ts`: OpenAI function schemas for LLM

2. **App Package** (`packages/app/`):
   - Expo (React Native)
   - Cross-platform: iOS, Android, Web
   - WebSocket client for real-time communication
   - Audio capture and playback handling

### Key Architecture Decisions

**Terminal Model**:

- Uses tmux session "voice-dev" with windows-as-terminals
- Each terminal = one tmux window with single pane
- Terminal ID = tmux window ID (format: `@123`)
- Terminal names must be unique

**Voice-First Design**:

- All responses designed for TTS output
- STT errors handled intelligently (see `agent-prompt.md`)
- No visual feedback assumptions

**Streaming Architecture**:

- Audio: App � WebSocket � Whisper STT � LLM � TTS � WebSocket � App
- Text chunks streamed for real-time response
- Tool calls logged as activity events

### Data Flow

1. **Voice Input**: App captures audio � sends chunks via WebSocket � STT transcribes � orchestrator processes
2. **LLM Processing**: Orchestrator streams GPT-4 responses � executes tool calls � accumulates response
3. **Voice Output**: Response � TTS synthesis � base64 audio � WebSocket � app playback

## TypeScript Configuration

**Two separate configs:**

1. `packages/server/tsconfig.server.json`: Server build config (NodeNext modules, outputs to dist/server)
2. `packages/app/tsconfig.json`: App config (extends Expo base, includes path mappings to server types)

**Module System**:

- Server: ES modules with `.js` extensions in imports (NodeNext resolution)
- App: Expo bundler (imports server types via `@server/*` path alias)

## Environment Variables

Required in `packages/server/.env`:

```bash
OPENAI_API_KEY=sk-...     # Required for GPT-4, Whisper, and TTS

# Optional TTS configuration
TTS_VOICE=alloy           # alloy|echo|fable|onyx|nova|shimmer
TTS_MODEL=tts-1           # tts-1 (fast) | tts-1-hd (quality)

# Server config
PORT=3000
NODE_ENV=development
```

## Terminal Control (MCP-like Tools)

The LLM has access to terminal operations via function calling:

- `list_terminals()`: Get all terminals with status
- `create_terminal(name, workingDirectory, initialCommand?)`: Create new terminal
- `send_text(terminalId, text, pressEnter, captureOutput?)`: Execute commands
- `capture_terminal(terminalId, lines?, wait?)`: Read terminal output
- `kill_terminal(terminalId)`: Remove terminal

See `src/server/daemon/tool-definitions.ts` for complete schemas.

## WebSocket Protocol

Message types (see `src/server/types.ts`):

**Client � Server**:

- `user_message`: Text input
- `audio_chunk`: Audio data (base64, includes format and isLast flag)
- `recording_state`: Recording started/stopped
- `ping`: Keepalive

**Server � Client**:

- `activity_log`: Conversation events (transcript, tool calls, errors)
- `assistant_chunk`: Streaming response text
- `audio_output`: Complete TTS audio (base64 MP3)
- `transcription_result`: STT output
- `status`: Connection state
- `pong`: Keepalive response

## Agent System Prompt

Located in `agent-prompt.md` (32KB) - loaded by `system-prompt.ts`.

**Critical aspects**:

- Voice-first interaction guidelines
- STT error handling strategies
- Terminal operation best practices
- Response brevity requirements (user is on phone)

## Testing

Currently manual testing:

1. Start backend server: `npm run dev`
2. Start Expo app: `npm run dev:app`
3. Test WebSocket connection in app
4. Test voice recording/playback
5. Test terminal operations via API endpoints

Test scripts:

- `test-llm.sh`: Direct LLM streaming test
- `test-orchestrator.js`: Conversation flow test

## File Organization

```
packages/voice-assistant/
   src/
      server/
         agent/          # LLM, STT, TTS, orchestration
         daemon/         # Terminal management + tool schemas
         index.ts        # Express + WebSocket server
         websocket-server.ts
         types.ts        # Shared server types
      ui/
          components/     # React components
          hooks/          # WebSocket, audio hooks
          lib/            # Audio capture/playback utilities
          App.tsx
          main.tsx
   dist/                   # Build output (gitignored)
   agent-prompt.md         # System prompt source
   package.json
   tsconfig.json           # UI TypeScript config
   tsconfig.server.json    # Server TypeScript config
   vite.config.ts
```

## Common Tasks

**Adding a new terminal tool:**

1. Add function schema to `daemon/tool-definitions.ts`
2. Implement handler in `agent/llm-openai.ts` executeTool function
3. If needed, add low-level tmux operation to `daemon/tmux.ts`

**Modifying agent behavior:**

1. Edit `agent-prompt.md` for instructions
2. Restart server (prompt loaded on startup)

**Changing audio formats:**

1. STT: Modify `agent/stt-openai.ts` (currently accepts webm/ogg/mp3)
2. TTS: Change TTS_MODEL env var or `tts-openai.ts` config

**WebSocket message types:**

1. Add type to `types.ts` WebSocketMessage union
2. Add handler in `websocket-server.ts` or client hooks

## Known Patterns

- **No barrel files**: Direct imports only (no index.ts exports)
- **Function declarations**: Always use `function` not arrow functions for exports
- **Error handling**: Let errors bubble to Express/WebSocket error handlers
- **Tool execution**: Synchronous within LLM streaming loop (tools modify conversation state)
- **Conversation cleanup**: Auto-cleanup runs every 10 minutes (60 min TTL)
- **Never run dev servers**: User will test manually - don't start servers unless explicitly requested