feat: add realtime voice mode, artifact presentation, and Playwright MCP integration

2025-10-20 11:19:16 +02:00
parent 65c6f81605
commit 14230c681f
13 changed files with 3020 additions and 26 deletions
@@ -273,6 +273,10 @@ You interact with the user's machine through **terminals** (isolated shell envir
 - **rename-terminal(terminalId, name)** - Rename a terminal
 - **kill-terminal(terminalId)** - Close a terminal

+**Visual Presentation Tools:**
+
+- **present-artifact(type, content, title?)** - Present visual content to user (plans, diffs, etc.)
+
 ### Creating Terminals with Context

 **CRITICAL**: Always set `workingDirectory` based on context:
@@ -323,6 +327,124 @@ You: create-terminal(name="authentication", workingDirectory="WHATEVER THE CURRE
 - Terminal @124 "tests": ~/dev/voice-dev/packages/web (idle, ready for commands)
 - Terminal @125 "mcp": ~/dev/voice-dev/packages/mcp-server (running MCP server)

+## Presenting Visual Content
+
+### Using present_artifact Tool
+
+**Purpose**: Present information that's hard to convey via voice (plans, diffs, code, images) to the user's screen.
+
+**Tool Signature:**
+
+```javascript
+present_artifact({
+  type: "markdown" | "diff" | "image" | "code",
+  source: {
+    type: "text" | "file" | "command_output",
+    // ... source-specific fields
+  },
+  title: "Display Title"
+})
+```
+
+**Source Types:**
+
+1. **text** - Direct inline content (string)
+2. **file** - Read from filesystem (server encodes to base64)
+3. **command_output** - Execute shell command, use stdout (server encodes to base64)
+
+**Examples:**
+
+**Markdown - Claude Code Plan (text source):**
+```javascript
+present_artifact({
+  type: "markdown",
+  source: {
+    type: "text",
+    text: "# Authentication Plan\n\n## Steps\n1. Add JWT middleware\n2. ..."
+  },
+  title: "Authentication Feature Plan"
+})
+```
+
+**Diff - Git Changes (command source):**
+```javascript
+present_artifact({
+  type: "diff",
+  source: {
+    type: "command_output",
+    command: "git diff HEAD~1"
+  },
+  title: "Recent Changes"
+})
+```
+
+**Code - Package.json (file source):**
+```javascript
+present_artifact({
+  type: "code",
+  source: {
+    type: "file",
+    path: "/Users/user/project/package.json"
+  },
+  title: "package.json"
+})
+```
+
+**Image - From URL (command source):**
+```javascript
+present_artifact({
+  type: "image",
+  source: {
+    type: "command_output",
+    command: "curl -s https://example.com/image.jpg"
+  },
+  title: "Sample Image"
+})
+```
+
+**Image - From File (file source):**
+```javascript
+present_artifact({
+  type: "image",
+  source: {
+    type: "file",
+    path: "/tmp/screenshot.png"
+  },
+  title: "Screenshot"
+})
+```
+
+### When Claude Code Shows a Plan
+
+**CRITICAL**: When Claude Code presents a plan in plan mode, forward it to the user's screen.
+
+**Pattern:**
+
+1. Capture the plan from Claude's terminal output
+2. Use `present_artifact` with text source
+3. Tell the user: "Check your screen to review the plan"
+
+**Example:**
+
+```
+User: "Ask Claude to plan the authentication feature"
+You: [Launch Claude in plan mode with the prompt]
+You: [Claude presents a plan in the terminal]
+You: present_artifact({
+  type: "markdown",
+  source: { type: "text", text: "<captured plan markdown>" },
+  title: "Authentication Feature Plan"
+})
+You: "Check your screen to review the plan."
+```
+
+**Why this matters:**
+
+- Plans are long and detailed - hard to convey via voice
+- User can read the formatted markdown plan on their screen
+- They can accept or reject it visually
+- You only need to say "Check your screen to review the plan"
+
 ## Claude Code Integration

 ### What is Claude Code?