286 lines
8.4 KiB
Markdown
286 lines
8.4 KiB
Markdown
# CLAUDE.md
|
|
|
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
|
|
## Project Overview
|
|
|
|
Voice-controlled terminal assistant using OpenAI's Realtime API (via Whisper STT, GPT-4, and TTS). Backend server with Expo app for all platforms (iOS, Android, and Web).
|
|
|
|
## Monorepo Structure
|
|
|
|
This is an npm workspace monorepo:
|
|
|
|
- **Root**: Workspace configuration and shared TypeScript config
|
|
- **packages/server**: Backend server (Express + WebSocket API)
|
|
- **packages/app**: Cross-platform app (Expo - iOS, Android, Web)
|
|
|
|
## Development Commands
|
|
|
|
All commands run from the repository root:
|
|
|
|
```bash
|
|
# Development
|
|
npm run dev # Start backend server (port 6767)
|
|
npm run dev:app # Start Expo dev server
|
|
|
|
# Build everything
|
|
npm run build
|
|
|
|
# Type checking
|
|
npm run typecheck
|
|
|
|
# Production
|
|
npm run start # Start backend server
|
|
|
|
# Platform-specific
|
|
npm run android # Run on Android
|
|
npm run ios # Run on iOS
|
|
npm run web # Run web version
|
|
```
|
|
|
|
### Server Package Commands
|
|
|
|
From `packages/server/`:
|
|
|
|
```bash
|
|
# Development
|
|
npm run dev # Watch mode: Express server (port 6767)
|
|
|
|
# Building
|
|
npm run build # TypeScript → dist/server
|
|
|
|
# Production
|
|
npm start # Run built server
|
|
|
|
# Type checking
|
|
npm run typecheck # Check server TypeScript
|
|
```
|
|
|
|
### App Package Commands
|
|
|
|
From `packages/app/`:
|
|
|
|
```bash
|
|
# Development - IMPORTANT: Must run in a TTY for interactive QR code
|
|
npx expo start # Start Expo dev server (Metro bundler on port 8081)
|
|
|
|
# Running in tmux (recommended for proper TTY)
|
|
tmux new-session -d -s expo -c /Users/moboudra/dev/voice-dev/packages/app "npx expo start"
|
|
tmux attach -t expo # Attach to see QR code and interact
|
|
|
|
# View expo session output without attaching
|
|
tmux capture-pane -t expo -p | tail -30
|
|
|
|
# Building
|
|
npx expo prebuild # Generate native iOS/Android projects
|
|
eas build --platform ios # Build for iOS (requires EAS)
|
|
eas build --platform android # Build for Android (requires EAS)
|
|
|
|
# Type checking
|
|
npm run typecheck # Check TypeScript
|
|
```
|
|
|
|
**Why tmux for Expo:**
|
|
- Expo requires a TTY to display the interactive QR code and menu
|
|
- Running without a TTY results in errors or missing output
|
|
- tmux provides a persistent TTY session that survives disconnects
|
|
|
|
## Architecture
|
|
|
|
### Overall Structure
|
|
|
|
Two-package architecture:
|
|
|
|
1. **Server Package** (`packages/server/src/server/`):
|
|
|
|
- Express HTTP server (port 6767)
|
|
- WebSocket server for bidirectional communication
|
|
- No UI (API only)
|
|
|
|
- **Agent Layer** (`agent/`):
|
|
- `orchestrator.ts`: Main conversation manager
|
|
- `llm-openai.ts`: GPT-4 streaming with tool execution
|
|
- `stt-openai.ts`: Whisper transcription
|
|
- `tts-openai.ts`: OpenAI TTS synthesis
|
|
- `system-prompt.ts`: Agent instructions (voice-first UX focused)
|
|
|
|
- **Daemon Layer** (`daemon/`):
|
|
- `terminal-manager.ts`: High-level terminal operations
|
|
- `tmux.ts`: Low-level tmux integration
|
|
- `tool-definitions.ts`: OpenAI function schemas for LLM
|
|
|
|
2. **App Package** (`packages/app/`):
|
|
- Expo (React Native)
|
|
- Cross-platform: iOS, Android, Web
|
|
- WebSocket client for real-time communication
|
|
- Audio capture and playback handling
|
|
|
|
### Key Architecture Decisions
|
|
|
|
**Terminal Model**:
|
|
|
|
- Uses tmux session "voice-dev" with windows-as-terminals
|
|
- Each terminal = one tmux window with single pane
|
|
- Terminal ID = tmux window ID (format: `@123`)
|
|
- Terminal names must be unique
|
|
|
|
**Voice-First Design**:
|
|
|
|
- All responses designed for TTS output
|
|
- STT errors handled intelligently (see `agent-prompt.md`)
|
|
- No visual feedback assumptions
|
|
|
|
**Streaming Architecture**:
|
|
|
|
- Audio: App � WebSocket � Whisper STT � LLM � TTS � WebSocket � App
|
|
- Text chunks streamed for real-time response
|
|
- Tool calls logged as activity events
|
|
|
|
### Data Flow
|
|
|
|
1. **Voice Input**: App captures audio � sends chunks via WebSocket � STT transcribes � orchestrator processes
|
|
2. **LLM Processing**: Orchestrator streams GPT-4 responses � executes tool calls � accumulates response
|
|
3. **Voice Output**: Response � TTS synthesis � base64 audio � WebSocket � app playback
|
|
|
|
## TypeScript Configuration
|
|
|
|
**Two separate configs:**
|
|
|
|
1. `packages/server/tsconfig.server.json`: Server build config (NodeNext modules, outputs to dist/server)
|
|
2. `packages/app/tsconfig.json`: App config (extends Expo base, includes path mappings to server types)
|
|
|
|
**Module System**:
|
|
|
|
- Server: ES modules with `.js` extensions in imports (NodeNext resolution)
|
|
- App: Expo bundler (imports server types via `@server/*` path alias)
|
|
|
|
## Environment Variables
|
|
|
|
Required in `packages/server/.env`:
|
|
|
|
```bash
|
|
OPENAI_API_KEY=sk-... # Required for GPT-4, Whisper, and TTS
|
|
|
|
# Optional TTS configuration
|
|
TTS_VOICE=alloy # alloy|echo|fable|onyx|nova|shimmer
|
|
TTS_MODEL=tts-1 # tts-1 (fast) | tts-1-hd (quality)
|
|
|
|
# Server config
|
|
PORT=3000
|
|
NODE_ENV=development
|
|
```
|
|
|
|
## Terminal Control (MCP-like Tools)
|
|
|
|
The LLM has access to terminal operations via function calling:
|
|
|
|
- `list_terminals()`: Get all terminals with status
|
|
- `create_terminal(name, workingDirectory, initialCommand?)`: Create new terminal
|
|
- `send_text(terminalId, text, pressEnter, captureOutput?)`: Execute commands
|
|
- `capture_terminal(terminalId, lines?, wait?)`: Read terminal output
|
|
- `kill_terminal(terminalId)`: Remove terminal
|
|
|
|
See `src/server/daemon/tool-definitions.ts` for complete schemas.
|
|
|
|
## WebSocket Protocol
|
|
|
|
Message types (see `src/server/types.ts`):
|
|
|
|
**Client � Server**:
|
|
|
|
- `user_message`: Text input
|
|
- `audio_chunk`: Audio data (base64, includes format and isLast flag)
|
|
- `recording_state`: Recording started/stopped
|
|
- `ping`: Keepalive
|
|
|
|
**Server � Client**:
|
|
|
|
- `activity_log`: Conversation events (transcript, tool calls, errors)
|
|
- `assistant_chunk`: Streaming response text
|
|
- `audio_output`: Complete TTS audio (base64 MP3)
|
|
- `transcription_result`: STT output
|
|
- `status`: Connection state
|
|
- `pong`: Keepalive response
|
|
|
|
## Agent System Prompt
|
|
|
|
Located in `agent-prompt.md` (32KB) - loaded by `system-prompt.ts`.
|
|
|
|
**Critical aspects**:
|
|
|
|
- Voice-first interaction guidelines
|
|
- STT error handling strategies
|
|
- Terminal operation best practices
|
|
- Response brevity requirements (user is on phone)
|
|
|
|
## Testing
|
|
|
|
Currently manual testing:
|
|
|
|
1. Start backend server: `npm run dev`
|
|
2. Start Expo app: `npm run dev:app`
|
|
3. Test WebSocket connection in app
|
|
4. Test voice recording/playback
|
|
5. Test terminal operations via API endpoints
|
|
|
|
Test scripts:
|
|
|
|
- `test-llm.sh`: Direct LLM streaming test
|
|
- `test-orchestrator.js`: Conversation flow test
|
|
|
|
## File Organization
|
|
|
|
```
|
|
packages/voice-assistant/
|
|
src/
|
|
server/
|
|
agent/ # LLM, STT, TTS, orchestration
|
|
daemon/ # Terminal management + tool schemas
|
|
index.ts # Express + WebSocket server
|
|
websocket-server.ts
|
|
types.ts # Shared server types
|
|
ui/
|
|
components/ # React components
|
|
hooks/ # WebSocket, audio hooks
|
|
lib/ # Audio capture/playback utilities
|
|
App.tsx
|
|
main.tsx
|
|
dist/ # Build output (gitignored)
|
|
agent-prompt.md # System prompt source
|
|
package.json
|
|
tsconfig.json # UI TypeScript config
|
|
tsconfig.server.json # Server TypeScript config
|
|
vite.config.ts
|
|
```
|
|
|
|
## Common Tasks
|
|
|
|
**Adding a new terminal tool:**
|
|
|
|
1. Add function schema to `daemon/tool-definitions.ts`
|
|
2. Implement handler in `agent/llm-openai.ts` executeTool function
|
|
3. If needed, add low-level tmux operation to `daemon/tmux.ts`
|
|
|
|
**Modifying agent behavior:**
|
|
|
|
1. Edit `agent-prompt.md` for instructions
|
|
2. Restart server (prompt loaded on startup)
|
|
|
|
**Changing audio formats:**
|
|
|
|
1. STT: Modify `agent/stt-openai.ts` (currently accepts webm/ogg/mp3)
|
|
2. TTS: Change TTS_MODEL env var or `tts-openai.ts` config
|
|
|
|
**WebSocket message types:**
|
|
|
|
1. Add type to `types.ts` WebSocketMessage union
|
|
2. Add handler in `websocket-server.ts` or client hooks
|
|
|
|
## Known Patterns
|
|
|
|
- **No barrel files**: Direct imports only (no index.ts exports)
|
|
- **Function declarations**: Always use `function` not arrow functions for exports
|
|
- **Error handling**: Let errors bubble to Express/WebSocket error handlers
|
|
- **Tool execution**: Synchronous within LLM streaming loop (tools modify conversation state)
|
|
- **Conversation cleanup**: Auto-cleanup runs every 10 minutes (60 min TTL)
|
|
- **Never run dev servers**: User will test manually - don't start servers unless explicitly requested |