docs: add orchestrator agent specification
This commit is contained in:
Mohamed Boudra
@@ -0,0 +1,181 @@
|
|||||||
|
# Voice Dev: Orchestrator + Agent Chat Architecture Spec
|
||||||
|
|
||||||
|
## Paradigm
|
||||||
|
|
||||||
|
### Two Interaction Models
|
||||||
|
|
||||||
|
**1. Orchestrator (Realtime Voice)**
|
||||||
|
- Top-level conversational agent
|
||||||
|
- Controls everything: creates agents, checks status, manages projects
|
||||||
|
- Realtime voice mode with TTS
|
||||||
|
- App-level session (not project-specific)
|
||||||
|
|
||||||
|
**2. Agent Chat (Text + Voice Notes)**
|
||||||
|
- Direct interaction with individual coding agents
|
||||||
|
- Text messages + voice notes (transcribed to text)
|
||||||
|
- No realtime, no TTS
|
||||||
|
- Per-agent conversation history
|
||||||
|
|
||||||
|
### Key Rules
|
||||||
|
|
||||||
|
1. **Realtime always routes to orchestrator** - regardless of which view you're in
|
||||||
|
2. **Non-realtime routes based on view**:
|
||||||
|
- Orchestrator view → orchestrator session
|
||||||
|
- Agent view → specific agent
|
||||||
|
3. **Input area is identical everywhere** - same UI, different routing
|
||||||
|
4. **TTS is implied by message type** - no flags needed
|
||||||
|
|
||||||
|
## Message Architecture
|
||||||
|
|
||||||
|
### Separation of Concerns
|
||||||
|
|
||||||
|
**Realtime Messages (Orchestrator Only)**
|
||||||
|
- `audio_chunk` (realtime) → triggers TTS response
|
||||||
|
- Implies: speech-to-speech interaction
|
||||||
|
- Server handles with full orchestrator Session
|
||||||
|
|
||||||
|
**Non-Realtime Messages**
|
||||||
|
- `user_text` (orchestrator) → text to orchestrator, no TTS
|
||||||
|
- `user_audio_note` (orchestrator) → transcribe → text to orchestrator, no TTS
|
||||||
|
- `send_agent_message` (agent) → text to agent via ACP, no TTS
|
||||||
|
- `send_agent_audio` (agent) → transcribe → text to agent, no TTS
|
||||||
|
|
||||||
|
**Outbound (Existing)**
|
||||||
|
- Orchestrator: `assistant_chunk`, `activity_log`, `audio_output` (TTS)
|
||||||
|
- Agents: `agent_update` (session notifications from ACP)
|
||||||
|
|
||||||
|
## Message Types
|
||||||
|
|
||||||
|
### New Inbound Types
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// To agent (text)
|
||||||
|
{
|
||||||
|
type: "send_agent_message",
|
||||||
|
agentId: string,
|
||||||
|
text: string
|
||||||
|
}
|
||||||
|
|
||||||
|
// To agent (voice note)
|
||||||
|
{
|
||||||
|
type: "send_agent_audio",
|
||||||
|
agentId: string,
|
||||||
|
audio: string, // base64
|
||||||
|
format: string,
|
||||||
|
isLast: boolean
|
||||||
|
}
|
||||||
|
|
||||||
|
// Separate audio_chunk into realtime context
|
||||||
|
{
|
||||||
|
type: "realtime_audio_chunk", // rename from audio_chunk
|
||||||
|
audio: string,
|
||||||
|
format: string,
|
||||||
|
isLast: boolean
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Existing Types (Clarified)
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// To orchestrator (text)
|
||||||
|
{
|
||||||
|
type: "user_text",
|
||||||
|
text: string
|
||||||
|
// NO disableTTS flag - never has TTS in non-realtime context
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## UI Architecture
|
||||||
|
|
||||||
|
### Unified Input Area
|
||||||
|
|
||||||
|
**Same everywhere, routes differently:**
|
||||||
|
- Text input
|
||||||
|
- Voice note button (push-to-talk, sends when released)
|
||||||
|
- Realtime toggle
|
||||||
|
|
||||||
|
**Routing Logic:**
|
||||||
|
```
|
||||||
|
if (realtimeMode) {
|
||||||
|
→ orchestrator (speech-to-speech)
|
||||||
|
} else if (viewMode === "agent") {
|
||||||
|
→ agent[activeAgentId] (text only)
|
||||||
|
} else {
|
||||||
|
→ orchestrator (text only)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Views
|
||||||
|
|
||||||
|
**Orchestrator View (index.tsx)**
|
||||||
|
- Current chat UI (unchanged)
|
||||||
|
- Conversation with orchestrator
|
||||||
|
- Realtime UI when enabled
|
||||||
|
|
||||||
|
**Agent View (agent-stream-view.tsx)**
|
||||||
|
- Enhanced to match orchestrator chat quality
|
||||||
|
- User messages + agent responses (parsed from session updates)
|
||||||
|
- Tool calls, thoughts, plans (inline activity)
|
||||||
|
- Same input area (routed to agent)
|
||||||
|
- Shows realtime UI if enabled (but messages still go to orchestrator)
|
||||||
|
|
||||||
|
### Navigation
|
||||||
|
|
||||||
|
- Header: connection status, conversation selector, settings, "New Agent" button
|
||||||
|
- ActiveProcesses: horizontal list of agents (click to open agent view)
|
||||||
|
- "Back to Chat" button when in agent view
|
||||||
|
|
||||||
|
## Implementation Phases
|
||||||
|
|
||||||
|
### Phase 1: Message Routing Foundation
|
||||||
|
|
||||||
|
**Server**
|
||||||
|
- Rename `audio_chunk` → `realtime_audio_chunk` for clarity
|
||||||
|
- Add `send_agent_message` and `send_agent_audio` message types
|
||||||
|
- Route non-realtime messages to `agentManager.sendPrompt()`
|
||||||
|
- Handle audio transcription for agent audio notes
|
||||||
|
|
||||||
|
**App**
|
||||||
|
- Add message routing logic based on `viewMode` and `isRealtimeMode`
|
||||||
|
- Update WebSocket to handle new message types
|
||||||
|
- Add per-agent message state: `Map<agentId, MessageEntry[]>`
|
||||||
|
|
||||||
|
### Phase 2: Agent Chat Interface
|
||||||
|
|
||||||
|
**App**
|
||||||
|
- Parse `agent_update` messages into chat messages
|
||||||
|
- Extract text chunks (user_message_chunk, agent_message_chunk)
|
||||||
|
- Extract thoughts (agent_thought_chunk)
|
||||||
|
- Merge tool calls with updates
|
||||||
|
- Enhance agent-stream-view to chat quality:
|
||||||
|
- Render user messages
|
||||||
|
- Render agent responses (streaming)
|
||||||
|
- Render activity inline (thoughts, tools, plans)
|
||||||
|
- Remove its own input area (uses shared one)
|
||||||
|
|
||||||
|
### Phase 3: Agent Creation
|
||||||
|
|
||||||
|
**Server**
|
||||||
|
- Ensure `createAgent` supports mode selection (already exists)
|
||||||
|
|
||||||
|
**App**
|
||||||
|
- Create agent creation modal
|
||||||
|
- Directory input/picker
|
||||||
|
- Mode selector dropdown
|
||||||
|
- Validate directory exists (optional)
|
||||||
|
- "New Agent" button in header
|
||||||
|
- Auto-switch to agent view after creation
|
||||||
|
|
||||||
|
### Phase 4: Polish
|
||||||
|
|
||||||
|
**App**
|
||||||
|
- Ensure realtime UI shows in both views
|
||||||
|
- Clear visual indication of routing (subtle badge/label?)
|
||||||
|
- Smooth transitions between views
|
||||||
|
- Handle edge cases (agent killed while viewing, etc.)
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
1. Should we show a subtle indicator in the input area showing where messages will go?
|
||||||
|
2. How to handle agent errors during chat (failed, killed)?
|
||||||
|
3. Should agent creation modal remember last used directory?
|
||||||
Reference in New Issue
Block a user