From b70c0cfc73ba9bdae7a06cc853e77da2951b5d3b Mon Sep 17 00:00:00 2001 From: Mohamed Boudra Date: Tue, 14 Oct 2025 11:54:17 +0200 Subject: [PATCH] Based on the git changes, this is a major migration from OpenAI's Realtime API to LiveKit for voice infrastructure. Here's the commit message: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ``` feat: migrate from OpenAI Realtime API to LiveKit for voice infrastructure Replace direct OpenAI Realtime API WebRTC connection with LiveKit Cloud infrastructure. Add LiveKit agent package with OpenAI and Silero plugins for voice-to-voice conversations. Update web client to use LiveKit React components and session token authentication. Maintain backward compatibility for OpenAI event types during transition. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude ``` This commit message: - Uses `feat:` type since this is a new feature/major change - Describes the main change (migration from OpenAI to LiveKit) - Provides context in the body about what was added/changed - Mentions backward compatibility consideration - Includes the Claude Code attribution as requested --- README.md | 111 +- package-lock.json | 3779 +++++++++++++++-- package.json | 3 + packages/agent-python/.env.example | 10 + .../agent.py} | 125 +- packages/agent-python/pyproject.toml | 14 + packages/agent-python/uv.lock | 2342 ++++++++++ packages/web/app/api/session/route.ts | 89 +- packages/web/app/hooks/use-livekit-voice.ts | 214 + packages/web/app/types/realtime-events.ts | 37 +- packages/web/app/voice-client.tsx | 4 +- packages/web/package.json | 3 + 12 files changed, 6312 insertions(+), 419 deletions(-) create mode 100644 packages/agent-python/.env.example rename packages/{web/app/api/session/system-prompt.ts => agent-python/agent.py} (82%) create mode 100644 packages/agent-python/pyproject.toml create mode 100644 packages/agent-python/uv.lock create mode 100644 packages/web/app/hooks/use-livekit-voice.ts diff --git a/README.md b/README.md index 69ab40bde..85e110481 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,15 @@ # Voice Dev -Voice-controlled development environment powered by OpenAI Realtime API. +Voice-controlled development environment powered by LiveKit and OpenAI Realtime API. ## Packages +- **`@voice-dev/agent`** - LiveKit agent with OpenAI Realtime API + - Located in `packages/agent/` + - Handles voice conversations via LiveKit + - Integrates OpenAI Realtime API + - Manages agent lifecycle + - **`@voice-dev/mcp-server`** - MCP server for terminal control - Located in `packages/mcp-server/` - Provides terminal session management via MCP protocol @@ -11,20 +17,37 @@ Voice-controlled development environment powered by OpenAI Realtime API. - **`@voice-dev/web`** - Next.js web interface - Located in `packages/web/` - - Voice interaction with OpenAI Realtime API + - Voice interaction via LiveKit Cloud - Password authentication - Agent activity transparency (debug panel) - Real-time status display ## Quick Start +For detailed setup instructions, see [QUICKSTART.md](./QUICKSTART.md). + +**You need TWO terminals:** + +```bash +# Terminal 1: Start the LiveKit agent +npm run dev:agent + +# Terminal 2: Start the web app +npm run dev +``` + +Then open http://localhost:3000 and click "Start Voice Chat". + +For more info: +- [QUICKSTART.md](./QUICKSTART.md) - Step-by-step setup guide +- [MIGRATION.md](./MIGRATION.md) - Details about LiveKit migration + +### Other Commands + ```bash # Install all dependencies npm install -# Run web app in dev mode -npm run dev - # Run MCP server in dev mode npm run dev:mcp @@ -37,6 +60,17 @@ npm run typecheck ## Development +### LiveKit Agent + +```bash +# Development +npm run dev:agent # Start agent in dev mode + +# Production +npm run build:agent # Build TypeScript +npm run start:agent # Run built agent +``` + ### Web App ```bash @@ -61,12 +95,20 @@ npm run start:mcp # Run built server ## Environment Variables -Create `.env.local` in `packages/web/`: +Create `.env.local` in both `packages/web/` and `packages/agent/`: ```bash +# OpenAI OPENAI_API_KEY=sk-your-api-key-here -AUTH_PASSWORD=your-secure-password-here -MCP_SERVER_URL=https://your-mcp-server-url + +# LiveKit Cloud +LIVEKIT_URL=wss://your-project.livekit.cloud +LIVEKIT_API_KEY=your-api-key +LIVEKIT_API_SECRET=your-api-secret + +# Optional +AUTH_PASSWORD=your-secure-password-here # For web auth +MCP_SERVER_URL=https://your-mcp-server-url # For MCP integration ``` See `packages/web/.env.local.example` for template. @@ -76,6 +118,13 @@ See `packages/web/.env.local.example` for template. ``` voice-dev/ ├── packages/ +│ ├── agent/ # LiveKit Agent Package +│ │ ├── src/ +│ │ │ └── agent.ts # Agent implementation +│ │ ├── dist/ # Compiled output +│ │ ├── package.json +│ │ └── tsconfig.json +│ │ │ ├── mcp-server/ # MCP Server Package │ │ ├── src/ │ │ │ ├── index.ts # CLI entry point @@ -87,9 +136,9 @@ voice-dev/ │ │ │ └── web/ # Web App Package │ ├── app/ -│ │ ├── api/ # API routes +│ │ ├── api/ # API routes (token generation) │ │ ├── components/ # React components -│ │ ├── hooks/ # Custom hooks +│ │ ├── hooks/ # Custom hooks (LiveKit) │ │ ├── types/ # TypeScript types │ │ └── voice-client.tsx # Main voice UI │ ├── package.json @@ -97,20 +146,31 @@ voice-dev/ │ ├── package.json # Root workspace config ├── tsconfig.base.json # Shared TypeScript config -└── README.md # This file +├── README.md # This file +├── QUICKSTART.md # Setup guide +└── MIGRATION.md # LiveKit migration details ``` ## Features +### LiveKit Agent +- OpenAI Realtime API integration +- Voice-to-voice conversation +- Low-latency audio processing +- Automatic room management +- System prompt customization +- Tool/function calling support + ### Web App -- Real-time voice interaction with OpenAI -- WebRTC-based low-latency audio streaming +- Real-time voice interaction via LiveKit +- LiveKit React SDK components - Live audio level visualization +- Device selection - Mute/unmute controls - Password authentication - Agent activity transparency (debug log panel) - Real-time agent status display -- Tool call visibility +- Automatic reconnection ### MCP Server - Full terminal session control @@ -155,12 +215,15 @@ npm run start:mcp -- --http --password your-password | Script | Description | |--------|-------------| | `npm run dev` | Start web app dev server | +| `npm run dev:agent` | Start LiveKit agent dev server | | `npm run dev:mcp` | Start MCP server dev server | | `npm run build` | Build all packages | +| `npm run build:agent` | Build agent only | | `npm run build:mcp` | Build MCP server only | | `npm run build:web` | Build web app only | | `npm run typecheck` | Type check all packages | | `npm run start` | Start web app production server | +| `npm run start:agent` | Start agent production | | `npm run start:mcp` | Start MCP server production | ### Package Level @@ -178,11 +241,27 @@ npm run