jb/paseo
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: remove README.md and mcp-server README.md, add CLAUDE.md for project guidance

Browse Source 
Delete the existing README.md and mcp-server README.md files to streamline documentation. Introduce CLAUDE.md, which provides comprehensive guidance for the voice-dev project, including architecture, setup instructions, and component interactions.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
This commit is contained in:
Mohamed Boudra
2025-10-14 11:57:50 +02:00
parent b70c0cfc73
commit 1bd3852659
3 changed files with 301 additions and 361 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/CLAUDE.md
+301
View File
@@ -0,0 +1,301 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview
Voice-dev is a voice-controlled development environment consisting of three interconnected services:
1. **Web UI** (Next.js) - LiveKit-based voice interface for initiating voice chats
2. **Voice Agent** (Python/LiveKit) - Coordinates TTS/STT/LLM with MCP tool access
3. **MCP Server** (Node.js/TypeScript) - Provides tmux control tools to the voice agent
All three services must be running for the system to function. The web UI connects users to LiveKit rooms where the voice agent (running on the user's machine) joins and provides voice interaction with terminal control capabilities via MCP.
## Architecture
### Component Flow
```
User Browser (Web UI)
↓ LiveKit WebRTC
LiveKit Cloud Infrastructure
↓ LiveKit Protocol
Voice Agent (Python)
↓ MCP HTTP
MCP Server (tmux tools)
↓ tmux commands
Terminal Environment
```
### Key Technologies
- **LiveKit**: Voice infrastructure (WebRTC, STT/TTS/LLM coordination)
- **MCP (Model Context Protocol)**: Tool interface between agent and terminal
- **tmux**: Terminal multiplexer for executing commands
- **Next.js 15**: Web UI framework
- **Python 3.10+**: Voice agent runtime
- **TypeScript/Node.js**: MCP server runtime
## Development Setup
### Prerequisites
- Node.js (for web and MCP server)
- Python 3.10+ with uv package manager
- tmux installed and running
- LiveKit account with API credentials
### Initial Setup
```bash
# Install root dependencies
npm install
# Install web dependencies
cd packages/web && npm install
# Install MCP server dependencies
cd packages/mcp-server && npm install
# Install agent dependencies (uses uv)
cd packages/agent-python
# Dependencies managed by uv, see pyproject.toml
```
### Environment Configuration
**Web UI** (`packages/web/.env.local`):
```
OPENAI_API_KEY=sk-... # For legacy OpenAI Realtime API support
AUTH_PASSWORD=... # Password to access the web UI
MCP_SERVER_URL=... # Optional, for connecting to MCP server
```
**Voice Agent** (`packages/agent-python/.env`):
```
LIVEKIT_URL=wss://...
LIVEKIT_API_KEY=...
LIVEKIT_API_SECRET=...
MCP_SERVER_URL=http://localhost:6767 # Local MCP server
```
**MCP Server**: Configured via command-line arguments (see below)
## Running the Services
### Start All Services (Development)
You need three terminal sessions/tmux panes:
```bash
# Terminal 1: MCP Server
cd packages/mcp-server
npm run dev # Starts HTTP MCP server on port 6767 with password 'dev-password'
# Terminal 2: Voice Agent
cd packages/agent-python
uv run python agent.py dev # Starts LiveKit agent worker
# Terminal 3: Web UI
cd packages/web
npm run dev # Starts Next.js on http://localhost:3000
```
### Alternative: Use root scripts
```bash
# From repository root:
npm run dev # Starts web UI only
npm run dev:agent # Starts voice agent
npm run dev:mcp # Starts MCP server
```
### Testing Individual Components
```bash
# Test MCP server
cd packages/mcp-server
npm run typecheck
npm run build
# Test web UI
cd packages/web
npm run build
npm run typecheck
# No explicit tests for agent currently
```
## Key File Locations
### Web UI (`packages/web/`)
- `app/page.tsx` - Entry point (renders VoiceClient)
- `app/voice-client.tsx` - Main voice interface component
- `app/hooks/use-livekit-voice.ts` - LiveKit connection management
- `app/api/session/route.ts` - Generates LiveKit participant tokens
- `app/components/` - UI components (mute button, volume bar, activity log, etc.)
### Voice Agent (`packages/agent-python/`)
- `agent.py` - Complete agent implementation (entrypoint, system prompt, MCP setup)
- `pyproject.toml` - Python dependencies (uses uv)
- `.env` - LiveKit and MCP configuration
### MCP Server (`packages/mcp-server/`)
- `src/index.ts` - MCP server setup, tool/resource definitions
- `src/tmux.ts` - tmux interaction primitives
- `src/http-server.ts` - HTTP transport for MCP (enables remote agent access)
## MCP Server Details
### Available MCP Tools
The MCP server exposes these tools to the voice agent:
- **list** - List tmux sessions/windows/panes (hierarchical)
- **capture-pane** - Capture terminal output from a pane
- **create-session/create-window/split-pane** - Create new tmux resources
- **rename-window** - Rename a window
- **kill** - Kill sessions/windows/panes
- **send-keys** - Send special keys (Enter, Escape, Ctrl-C, etc.)
- **send-text** - Type text into a pane (primary way to run shell commands)
- **execute-shell-command** - Run a command synchronously with timeout
### MCP Server Modes
The server supports two transport modes:
1. **stdio** (default): For local MCP clients
```bash
npm run dev:stdio
```
2. **HTTP**: For remote agents (used by LiveKit agent)
```bash
npm run dev # HTTP on port 6767 with password 'dev-password'
# Or:
tsx src/index.ts --http --password <password> --port <port>
```
## Voice Agent System Prompt
The agent's system prompt is embedded in `packages/agent-python/agent.py`. It instructs the agent to:
- Acknowledge requests verbally before acting (critical for voice UX)
- Report tool execution results back to the user
- Handle voice-to-text errors gracefully
- Be concise for mobile users
- Work with Claude Code running in tmux sessions
- Understand tmux/git/gh CLI workflows
The prompt is comprehensive (~500 lines) and defines the agent's personality and behavior patterns.
## Common Development Tasks
### Adding a New MCP Tool
1. Define the tool in `packages/mcp-server/src/index.ts`:
```typescript
server.tool(
"tool-name",
"Description",
{
/* schema */
},
async (params) => {
// Implementation
}
);
```
2. Add underlying tmux primitives to `src/tmux.ts` if needed
3. Rebuild and restart the MCP server
### Modifying the Voice Agent Behavior
1. Edit `SYSTEM_PROMPT` in `packages/agent-python/agent.py`
2. Restart the agent (uv run python agent.py dev)
### Updating Web UI Components
1. Components are in `packages/web/app/components/`
2. Hooks for LiveKit integration in `packages/web/app/hooks/`
3. Use `npm run dev` in packages/web for hot reload
### TypeScript Changes
Run typecheck across all packages:
```bash
npm run typecheck # Root command, checks all workspaces
```
## Important Notes
### LiveKit Inference vs OpenAI Realtime API
The system recently migrated from OpenAI's Realtime API to LiveKit. The codebase contains legacy references to OpenAI's API (e.g., in README.md, some hooks). The current implementation uses:
- **STT**: AssemblyAI Universal Streaming
- **LLM**: OpenAI GPT-4.1-mini (via LiveKit Inference)
- **TTS**: Cartesia Sonic 2
LiveKit Inference handles API key management for all providers, so individual provider keys are not needed in the agent config.
### MCP HTTP Transport Authentication
The MCP server in HTTP mode requires a password parameter. The agent must include this in the `MCP_SERVER_URL`:
```
MCP_SERVER_URL=http://dev-password@localhost:6767
```
### Mobile Development
The web UI requires HTTPS for microphone access on mobile devices. Use Cloudflare Tunnel, ngrok, or Tailscale for mobile testing (see packages/web/README.md).
### Workspace Structure
This is an npm workspace (see root `package.json`). All packages share the same `node_modules` at the root. Python agent uses separate virtual environment managed by uv.
## Build and Release
```bash
# Build all packages
npm run build
# Build specific packages
npm run build:web
npm run build:mcp
npm run build:agent # No-op currently
# MCP server release
cd packages/mcp-server
npm run check-release # Dry run
npm run release # Publish to npm
```
## Debugging Tips
- **MCP Server**: Check HTTP server logs for incoming tool calls
- **Voice Agent**: Look for console output showing MCP server connection status
- **Web UI**: Check browser console for LiveKit connection events
- **LiveKit Connection**: Verify `LIVEKIT_URL`, `LIVEKIT_API_KEY`, `LIVEKIT_API_SECRET` are correct
- **MCP Connection**: Verify `MCP_SERVER_URL` includes password and is accessible from agent
## Documentation Files
Several markdown files document specific aspects:
- `packages/web/README.md` - Web UI setup and usage
- `packages/web/IMPLEMENTATION_PLAN.md` - Original OpenAI Realtime API implementation details
- `packages/web/MCP_INTEGRATION.md` - MCP integration research and decisions
- `packages/web/AUDIO_VISUALIZATION_PLAN.md` - Audio visualization feature planning
README.md
-267
View File
@@ -1,267 +0,0 @@
# Voice Dev
Voice-controlled development environment powered by LiveKit and OpenAI Realtime API.
## Packages
- **`@voice-dev/agent`** - LiveKit agent with OpenAI Realtime API
- Located in `packages/agent/`
- Handles voice conversations via LiveKit
- Integrates OpenAI Realtime API
- Manages agent lifecycle
- **`@voice-dev/mcp-server`** - MCP server for terminal control
- Located in `packages/mcp-server/`
- Provides terminal session management via MCP protocol
- Can run as HTTP server or stdio mode
- **`@voice-dev/web`** - Next.js web interface
- Located in `packages/web/`
- Voice interaction via LiveKit Cloud
- Password authentication
- Agent activity transparency (debug panel)
- Real-time status display
## Quick Start
For detailed setup instructions, see [QUICKSTART.md](./QUICKSTART.md).
**You need TWO terminals:**
```bash
# Terminal 1: Start the LiveKit agent
npm run dev:agent
# Terminal 2: Start the web app
npm run dev
```
Then open http://localhost:3000 and click "Start Voice Chat".
For more info:
- [QUICKSTART.md](./QUICKSTART.md) - Step-by-step setup guide
- [MIGRATION.md](./MIGRATION.md) - Details about LiveKit migration
### Other Commands
```bash
# Install all dependencies
npm install
# Run MCP server in dev mode
npm run dev:mcp
# Build everything
npm run build
# Type check all packages
npm run typecheck
```
## Development
### LiveKit Agent
```bash
# Development
npm run dev:agent # Start agent in dev mode
# Production
npm run build:agent # Build TypeScript
npm run start:agent # Run built agent
```
### Web App
```bash
# Development
npm run dev # Start Next.js dev server (port 3000)
# Production
npm run build:web # Build for production
npm run start # Start production server
```
### MCP Server
```bash
# Development
npm run dev:mcp # Start with HTTP server and dev-password
# Production
npm run build:mcp # Build TypeScript
npm run start:mcp # Run built server
```
## Environment Variables
Create `.env.local` in both `packages/web/` and `packages/agent/`:
```bash
# OpenAI
OPENAI_API_KEY=sk-your-api-key-here
# LiveKit Cloud
LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your-api-key
LIVEKIT_API_SECRET=your-api-secret
# Optional
AUTH_PASSWORD=your-secure-password-here # For web auth
MCP_SERVER_URL=https://your-mcp-server-url # For MCP integration
```
See `packages/web/.env.local.example` for template.
## Project Structure
```
voice-dev/
├── packages/
│ ├── agent/ # LiveKit Agent Package
│ │ ├── src/
│ │ │ └── agent.ts # Agent implementation
│ │ ├── dist/ # Compiled output
│ │ ├── package.json
│ │ └── tsconfig.json
│ │
│ ├── mcp-server/ # MCP Server Package
│ │ ├── src/
│ │ │ ├── index.ts # CLI entry point
│ │ │ ├── http-server.ts # HTTP server mode
│ │ │ └── tmux.ts # Terminal operations
│ │ ├── build/ # Compiled output
│ │ ├── package.json
│ │ └── tsconfig.json
│ │
│ └── web/ # Web App Package
│ ├── app/
│ │ ├── api/ # API routes (token generation)
│ │ ├── components/ # React components
│ │ ├── hooks/ # Custom hooks (LiveKit)
│ │ ├── types/ # TypeScript types
│ │ └── voice-client.tsx # Main voice UI
│ ├── package.json
│ └── next.config.js
│
├── package.json # Root workspace config
├── tsconfig.base.json # Shared TypeScript config
├── README.md # This file
├── QUICKSTART.md # Setup guide
└── MIGRATION.md # LiveKit migration details
```
## Features
### LiveKit Agent
- OpenAI Realtime API integration
- Voice-to-voice conversation
- Low-latency audio processing
- Automatic room management
- System prompt customization
- Tool/function calling support
### Web App
- Real-time voice interaction via LiveKit
- LiveKit React SDK components
- Live audio level visualization
- Device selection
- Mute/unmute controls
- Password authentication
- Agent activity transparency (debug log panel)
- Real-time agent status display
- Automatic reconnection
### MCP Server
- Full terminal session control
- List sessions, windows, and panes
- Send keystrokes and text
- Create and kill sessions/windows/panes
- Capture pane output
- HTTP server mode with password auth
- stdio mode for Claude Desktop integration
## Deployment
### Web App (Vercel)
Deploy from the web package directory:
```bash
# Deploy to Vercel
cd ~/dev/voice-dev/packages/web
vercel --prod
# Set environment variables
vercel env add OPENAI_API_KEY production
vercel env add AUTH_PASSWORD production
vercel env add MCP_SERVER_URL production
```
### MCP Server
The MCP server typically runs locally:
```bash
# Build and run
npm run build:mcp
npm run start:mcp -- --http --password your-password
```
## Scripts Reference
### Root Level
| Script | Description |
|--------|-------------|
| `npm run dev` | Start web app dev server |
| `npm run dev:agent` | Start LiveKit agent dev server |
| `npm run dev:mcp` | Start MCP server dev server |
| `npm run build` | Build all packages |
| `npm run build:agent` | Build agent only |
| `npm run build:mcp` | Build MCP server only |
| `npm run build:web` | Build web app only |
| `npm run typecheck` | Type check all packages |
| `npm run start` | Start web app production server |
| `npm run start:agent` | Start agent production |
| `npm run start:mcp` | Start MCP server production |
### Package Level
Run commands in specific packages:
```bash
# Run command in web package
npm run <script> --workspace=web
# Run command in mcp-server package
npm run <script> --workspace=mcp-server
```
## Tech Stack
- **Architecture**: npm workspaces
- **Web**: Next.js 15, React 19, TypeScript, Tailwind CSS, LiveKit React SDK
- **Agent**: LiveKit Agents Framework, OpenAI Realtime API, TypeScript
- **MCP Server**: TypeScript, Express, MCP SDK
- **Voice**: LiveKit Cloud, OpenAI Realtime API, WebRTC
- **Build**: TypeScript compiler, Next.js build
## Architecture
```
┌─────────────┐ ┌──────────────┐ ┌─────────────┐
│ Browser │◄───────►│ LiveKit Cloud│◄───────►│ Agent │
│ (React UI) │ WebRTC │ (Server) │ WebRTC │ (Node.js) │
└─────────────┘ └──────────────┘ └─────────────┘
│
▼
┌─────────────┐
│ OpenAI │
│ Realtime API│
└─────────────┘
```
## License
MIT
packages/mcp-server/README.md
-94
View File
@@ -1,94 +0,0 @@
# Tmux MCP Server
Model Context Protocol server that enables Claude Desktop to interact with and view tmux session content. This integration allows AI assistants to read from, control, and observe your terminal sessions.
## Features
- List and search tmux sessions
- View and navigate tmux windows and panes
- Capture and expose terminal content from any pane
- Execute commands in tmux panes and retrieve results (use it at your own risk ⚠️)
- Create new tmux sessions and windows
- Split panes horizontally or vertically with customizable sizes
- Kill tmux sessions, windows, and panes
Check out this short video to get excited!
</br>
[![youtube video](http://i.ytimg.com/vi/3W0pqRF1RS0/hqdefault.jpg)](https://www.youtube.com/watch?v=3W0pqRF1RS0)
## Prerequisites
- Node.js
- tmux installed and running
## Usage
### Configure Claude Desktop
Add this MCP server to your Claude Desktop configuration:
```json
"mcpServers": {
"tmux": {
"command": "npx",
"args": ["-y", "tmux-mcp"]
}
}
```
### MCP server options
You can optionally specify the command line shell you are using, if unspecified it defaults to `bash`
```json
"mcpServers": {
"tmux": {
"command": "npx",
"args": ["-y", "tmux-mcp", "--shell-type=fish"]
}
}
```
The MCP server needs to know the shell only when executing commands, to properly read its exit status.
## Available Resources
- `tmux://sessions` - List all tmux sessions
- `tmux://pane/{paneId}` - View content of a specific tmux pane
## Available Tools
### Hierarchy & Information
- `list` - List tmux sessions, windows, and panes with flexible scoping
- `scope="all"` - Full nested hierarchy (sessions → windows → panes)
- `scope="sessions"` - List all sessions
- `scope="session"` + `target="$0"` - List windows in session
- `scope="window"` + `target="@1"` - List panes in window
- `scope="pane"` + `target="%2"` - Get pane details
- `capture-pane` - Capture content from a tmux pane
### Session & Layout Management
- `create-session` - Create a new tmux session
- `create-window` - Create a new window in a tmux session
- `split-pane` - Split a tmux pane horizontally or vertically with optional size
- `kill` - Kill a session, window, or pane by scope and target
- `scope="session"` + `target="$0"` - Kill session
- `scope="window"` + `target="@1"` - Kill window
- `scope="pane"` + `target="%2"` - Kill pane
### Command Execution & Interaction
- `execute-shell-command` - Execute a shell command synchronously (default 30s timeout)
- Returns output, exit code, and status immediately
- Use for: ls, grep, npm test, quick commands
- **For long-running commands:** Use `send-text` with `pressEnter=true` instead, then monitor with `capture-pane`
- `send-keys` - Send special keys or key combinations (raw pass-through)
- Use for: TUI navigation, control sequences
- Examples: "Up", "Down", "C-c", "M-x", "Enter", "C-b d"
- `send-text` - Type text character-by-character with literal interpretation
- Uses `-l` flag per character (prevents special key interpretation)
- Use for: REPLs, forms, interactive apps, long-running commands
- Optional `pressEnter` parameter
- **For long-running commands:** Set `pressEnter=true`, then use `capture-pane` to monitor output