Based on the git diff showing extensive package-lock.json changes and modifications to the agent system prompt and configuration, here's the commit message:

``` feat: add AI SDK dependencies and refine agent voice interaction config - Add @ai-sdk packages for enhanced AI capabilities - Install Deepgram SDK, livekit-client, and werift for voice/WebRTC - Expand agent system prompt with multi-tool workflows and Claude Code interaction patterns - Adjust VAD timing (1.2s silence threshold) and turn detection for better voice UX - Switch TTS to inworld/inworld-tts-1 with Olivia voice - Configure interruption timing (0.2s detection, no word requirement) - Add comprehensive guidelines for conversational responses vs structured output - Document plan mode triggers and Claude response type handling 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> ```
2025-10-19 15:41:42 +02:00
parent dabcf63ebc
commit 40b199e329
44 changed files with 12183 additions and 1715 deletions
@@ -0,0 +1,106 @@
+# Quick Start: LLM Integration
+
+## Setup
+
+1. **Install dependencies** (if not already done):
+   ```bash
+   npm install
+   ```
+
+2. **Create `.env` file**:
+   ```bash
+   cp .env.example .env
+   # Edit .env and add your OPENAI_API_KEY
+   ```
+
+3. **Build the project**:
+   ```bash
+   npm run build
+   ```
+
+4. **Start the server**:
+   ```bash
+   npm start
+   ```
+
+## Testing
+
+### Quick Test
+
+```bash
+# Make sure server is running on http://localhost:3000
+curl -X POST http://localhost:3000/api/test-llm \
+  -H "Content-Type: application/json" \
+  -d '{"message": "list all terminals"}' | jq .
+```
+
+### Run All Tests
+
+```bash
+./test-llm.sh
+```
+
+## Example Commands
+
+Try these natural language commands:
+
+```bash
+# Create a terminal
+curl -X POST http://localhost:3000/api/test-llm \
+  -H "Content-Type: application/json" \
+  -d '{"message": "create a terminal called web-dev at ~/projects/web"}' | jq .
+
+# List terminals
+curl -X POST http://localhost:3000/api/test-llm \
+  -H "Content-Type: application/json" \
+  -d '{"message": "show me all terminals"}' | jq .
+
+# Run a command
+curl -X POST http://localhost:3000/api/test-llm \
+  -H "Content-Type: application/json" \
+  -d '{"message": "run npm install in the web-dev terminal"}' | jq .
+
+# Get terminal output
+curl -X POST http://localhost:3000/api/test-llm \
+  -H "Content-Type: application/json" \
+  -d '{"message": "show me the output from web-dev terminal"}' | jq .
+
+# Kill a terminal
+curl -X POST http://localhost:3000/api/test-llm \
+  -H "Content-Type: application/json" \
+  -d '{"message": "close the web-dev terminal"}' | jq .
+```
+
+## What's Next?
+
+Phase 5 will integrate this LLM functionality with the WebSocket server to enable:
+- Real-time voice conversations
+- Streaming responses
+- Multi-turn context
+- Voice-to-terminal control
+
+## Files Created
+
+- `/agent-prompt.md` - System prompt defining agent behavior
+- `/src/server/agent/system-prompt.ts` - Loads and caches system prompt
+- `/src/server/agent/llm-openai.ts` - OpenAI integration with tool execution
+- `/src/server/index.ts` - Added `/api/test-llm` endpoint
+- `/LLM_INTEGRATION.md` - Comprehensive integration guide
+- `/QUICK_START.md` - This file
+- `/test-llm.sh` - Automated test script
+
+## Troubleshooting
+
+**Q: "OPENAI_API_KEY not set" error**
+A: Create a `.env` file with `OPENAI_API_KEY=sk-...`
+
+**Q: Server won't start**
+A: Make sure tmux is installed and running
+
+**Q: Tool calls failing**
+A: Check that the default tmux session is initialized (happens automatically on server start)
+
+**Q: Build errors**
+A: Run `npm run build` and check the error messages
+
+For more details, see `LLM_INTEGRATION.md`.