feat: add multi-daemon support with session routing and connection management
Introduce DaemonRegistry and DaemonConnections contexts for managing multiple server connections. Add session directory for aggregating agents across daemons, with automatic routing based on agent/server context. Key changes: - Add daemon profile management (add/edit/remove/set default) - Implement connection state tracking with reconnection and exponential backoff - Create useAggregatedAgents hook for unified agent list across daemons - Add useDaemonSession and useDaemonRequest hooks for daemon-scoped operations - Update agent routes to include serverId for proper session routing - Add connection health indicators on home screen and settings - Persist session snapshots per daemon for faster hydration on reconnect - Guard screens gracefully when target daemon is offline - Add React Query for consistent loading/error state management 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Mohamed Boudra
@@ -1,92 +1,92 @@
|
||||
# Guidelines
|
||||
# Multi-Daemon Production Rollout Plan
|
||||
|
||||
- Implement the first available task, top down
|
||||
- Only do a single task and exit, the other agents will implement the other tasks
|
||||
- Commit after each task with a descriptive commit message.
|
||||
- Add context after each task completion, indented under the task, to help the reviewer understand the changes.
|
||||
## Guiding Principles
|
||||
- The app always reflects the union of every connected daemon—no manual switching to “see” data.
|
||||
- Actions (create/resume agent, browse files, realtime tools) automatically route to the correct daemon based on the agent/workspace context.
|
||||
- Connection health, loading states, and mutations are observable and resilient (React Query or equivalent).
|
||||
|
||||
# Tasks
|
||||
## Workstreams & Tasks
|
||||
|
||||
- [x] Claude hydration regression must be proven with a real end-to-end test: spin up an actual Claude agent (no mocks), ask it to run tool calls that create/edit/read a temp project, verify the live stream shows the tool results, shut the agent down, hydrate from disk (`~/.claude/projects/...`), and assert the exact diff/read/command output replays in the UI. Do not mark any hydration task complete until this automated test exists and fails on current main but passes after the fix.
|
||||
- Added a Vitest integration in `packages/server/src/server/agent/providers/claude-agent.test.ts` that drives a live Claude session through real Bash/write/read tool calls, converts the emitted timeline into the UI stream via `hydrateStreamState`, tears the agent down, resumes it from `~/.claude/projects/...`, and asserts that the hydrated stream reproduces the command output, file diff, and read content instead of leaving spinners. The helper waits for the persisted `.jsonl` file (covering `/var` ↔ `/private` realpaths), cleans up temp state, and the run is wired into `npm run test --workspace=@voice/server -- src/server/agent/providers/claude-agent.test.ts` (passes locally).
|
||||
- [x] Hydrated Claude tool calls still show the spinner forever after refreshing a chat; Claude CLI already persists the tool payloads under `~/.claude/projects/<conversation>`, but our hydrate path isn’t loading them. Fix the loader so it reads the saved diffs/read/output blocks and transitions the existing pill to “completed” instead of spinning or duplicating.
|
||||
- Claude history entries now flow through a shared `convertClaudeHistoryEntry` helper that inspects every message for `tool_*` blocks before classifying it as a plain user message, so persisted `tool_result` lines recorded as `type: "user"` convert into completed tool-call events instead of leaving the original spinner stuck in `executing`. Added unit coverage in `packages/server/src/server/agent/providers/claude-agent.test.ts` proving user tool results hydrate correctly and ran `npm run test --workspace=@voice/server -- src/server/agent/providers/claude-agent.test.ts` (passes, though existing SDK integration specs are still slow/flaky by nature).
|
||||
- [x] We still see duplicate tool call pills (loading + completed/failed) in both Codex and Claude sessions, live and hydrated. Track tool call IDs rigorously, dedupe pending/completed entries, and add regression tests covering all providers and hydrate flows.
|
||||
- Hardened the timeline reducer (`packages/app/src/types/stream.ts`) so out-of-order tool events reconcile via provider/server/tool metadata even when call IDs are missing, statuses only move forward (executing → completed/failed), and replayed hydration updates don’t spawn extra pills; added Vitest coverage in `packages/app/src/types/stream.test.ts` proving both Codex and Claude live + hydrated streams stay deduped when completion blocks precede their pending counterparts. Tests run via `npx vitest packages/app/src/types/stream.test.ts`.
|
||||
- [x] Permission request cards in the stream header are emitting duplicate key warnings—derive stable per-agent keys (for example `${agentId}:${request.id}` with fallbacks) so React stops complaining.
|
||||
- Added a `key` field on `PendingPermission` along with a shared helper in `SessionContext` that derives `${agentId}:<fallback>` identifiers for every server-sourced request, stores them in the map, and removes them when the server resolves the request. The stream header now renders permission cards using this stable key instead of recomputing on every render, eliminating the duplicate React keys.
|
||||
- [x] The so-called tests are still fake: convert the stream harness into real Vitest suites co-located with the code, make `npm test` (Vitest) the single entrypoint, and ensure those tests fail today because hydrated tool-call results are missing.
|
||||
- Added `packages/app/src/types/stream.harness.test.ts`, which codifies the manual stream harness into Vitest and captures the regression: the live sequence populates tool diffs/reads/command output, but the hydrated snapshot lacks them, so `npm run test --workspace=@voice/app` now fails on the second assertion until the hydration loader replays tool payloads from disk.
|
||||
- [x] Hydrated sessions continue to drop user messages; capture this in a failing test for both Codex and Claude (hydrate snapshot + live replay) and only mark complete once the test passes.
|
||||
- Added end-to-end hydration suites in `packages/server/src/server/agent/providers/{claude,codex}-agent.test.ts` that record live timeline updates (including user prompts) and then hydrate from `streamHistory()` after resuming the session; both now assert that the refreshed snapshot still contains the user's prompt marker. Fixed Claude hydration by having `convertClaudeHistoryEntry` emit user_message entries alongside persisted tool blocks, and taught the Codex rollout parser to surface `role: "user"` messages from rollout logs.
|
||||
- [x] Audit the recent “Vitest migration” claim: confirm `npm test` actually runs the new suites end to end, add coverage that specifically asserts tool call results render after hydration, and keep the task open until CI proves it.
|
||||
- Added `resolveToolCallPreview` so the `ToolCall` UI explicitly prefers hydrated `parsed*` payloads over fallbacks and wrote `tool-call-preview.test.ts` to cover hydration-vs-fallback behavior.
|
||||
- Verified `npm test` now drives both workspaces (server suite passes; app suite fails fast on `src/types/stream.harness.test.ts` because hydrated tool payloads are still missing), so CI will surface the regression once the loader fix lands.
|
||||
- [x] AgentInput image picker crashes with "Attempting to launch an unregistered ActivityResultLauncher" when calling Expo ImagePicker; register the launcher properly (or switch to the new async hook) so picking images doesn’t throw and we can attach screenshots again.
|
||||
- Added a dedicated `useImageAttachmentPicker` hook that registers the media picker launcher up front, reuses Expo’s permission hook, restores pending results, and guards against concurrent launches. AgentInput now consumes this hook so tapping the attachment button opens the picker without crashing on Android; `npm run typecheck --workspace=@voice/app` currently fails upstream in `stream.test.ts` before our change, noted in the summary.
|
||||
- [x] Do not mark the Claude hydration fix complete until there’s an end-to-end test that (1) runs Claude, (2) executes tool calls, (3) persists the conversation under `~/.claude/projects/...`, and (4) hydrates from disk verifying the tool call results render. No test, no checkbox.
|
||||
- Strengthened `packages/server/src/server/agent/providers/claude-agent.test.ts` so the existing e2e hydration suite now asserts the live stream parses command/edit/read payloads and that the resumed stream reproduces those parsed diffs/reads/command outputs from `~/.claude/projects`, guaranteeing the UI pills render identical results after hydration.
|
||||
- [x] ERROR Encountered two children with the same key, `%s`. Keys should be unique so that components maintain their identity across updates. Non-unique keys may cause children to be duplicated and/or omitted — the behavior is unsupported and could change in a future version. .$tool_1763217454926_uyr9rm
|
||||
### 1. Session Directory & Data Consistency
|
||||
- [x] Rebuild `useSessionDirectory` so it stays in sync with realtime session state (agents, permissions, stream updates) instead of caching an accessor forever.
|
||||
- Added session-level subscriptions so the directory re-renders on daemon updates and ran `npm run typecheck` to verify.
|
||||
- [x] Expose a lightweight subscription/API for background `SessionProvider`s so any change invalidates aggregated consumers without forcing rerenders of the main tree.
|
||||
- Added central session-directory listeners in the daemon connections context, had SessionProviders emit invalidations, updated `useSessionDirectory` to use the new API, and ran `npm run typecheck`.
|
||||
- [x] Audit every consumer that still reaches for `useSession()` directly (AgentStreamView, AgentInputArea, file explorers, realtime, etc.) and ensure they receive the session instance that corresponds to the agent/daemon they’re operating on.
|
||||
- Agent detail UI now pulls daemon-scoped sessions via `useDaemonSession`, so sending messages, toggling modes, and inline file explorers all operate against the route’s `serverId` (see `packages/app/src/components/agent-input-area.tsx:72`, `packages/app/src/components/agent-stream-view.tsx:31`, `packages/app/src/components/agent-status-bar.tsx:6`, `packages/app/src/app/agent/[serverId]/[agentId].tsx:256`) and `npm run typecheck` passes.
|
||||
|
||||
- `Stream` now generates fallback tool ids via `createUniqueTimelineId`, ensuring hydrated/past tool calls without callIds never reuse the same key even when their metadata matches; added `testFallbackToolCallIdsStayUnique` in `packages/app/src/types/stream.test.ts` to cover the regression and verified via `npx vitest packages/app/src/types/stream.test.ts`.
|
||||
### 2. Aggregated Agent Experience
|
||||
- [x] Replace the home screen’s `agents.size` gate with `useAggregatedAgents`, render the merged list (grouped/sorted by daemon), and remove the daemon picker UI from the header.
|
||||
- Home now builds grouped daemon sections via `useAggregatedAgents`, `AgentList` renders each section with the correct serverId routing, the header exposes dedicated import/create buttons without manual daemon switching, and `npm run typecheck` passes.
|
||||
- When closing the agent action sheet we now clear the stored `serverId` so `useDaemonSession` falls back to the active daemon and the Home screen no longer crashes if that background daemon disconnects after a long-press.
|
||||
- [x] Ensure all agent rows carry their `serverId` through navigation (Agent screen, diff viewer, file explorer, orchestrator) so every deep link includes `/agent/[serverId]/[agentId]`.
|
||||
- Annotated every agent snapshot with its daemon `serverId`, updated shared row components (`AgentList`, `AgentSidebar`, `ActiveProcesses`) to read it when navigating/deleting, and re-ran `npm run typecheck`.
|
||||
- [x] Update stream detail routes (git diff, file explorer) to validate the daemon session from params and gracefully show status/loading/error states if the background session is unavailable.
|
||||
- Wrapped both routes in session guards that render connection-aware placeholders when the target daemon is offline/unavailable, moved the existing logic into gated child components, and re-ran `npm run typecheck` to verify.
|
||||
- [x] Guard the main agent route when the requested daemon session is unavailable so deep links or quick daemon switches don't crash the screen.
|
||||
- Added a connection-aware guard around `/agent/[serverId]/[agentId]` that renders a friendly placeholder when the session is offline/unavailable and re-ran `npm run typecheck`.
|
||||
- [x] Fix the agent screen dropdown positioning so we don't double-apply the safe-area offset and remove the leaked debug logging.
|
||||
- `packages/app/src/app/agent/[serverId]/[agentId].tsx:320-339` adds `insets.top` to `measureInWindow` coordinates (which already include the status bar) and emits `[Menu]` console logs on every open, so the action menu renders ~40px too low on notched devices and spams the JS console.
|
||||
- Removed the extra `insets.top` offset, cleaned up the `[Menu]` debug logs, and re-ran `npm run typecheck`.
|
||||
- [x] Make inline file path navigation pick up the correct daemon id even when two daemons generate the same `agentId`.
|
||||
- Added `resolvedServerId` to the `handleInlinePathPress` dependency list in `packages/app/src/components/agent-stream-view.tsx:83-116`, so navigating after switching daemons now routes to the correct file explorer target; re-ran `npm run typecheck`.
|
||||
- [x] Sync the active daemon context with the agent route so realtime/audio flows hit the same websocket as the rendered agent, even when arriving from a deep link.
|
||||
- `AgentScreen` looks up the requested session via `useDaemonSession` but never calls `setActiveDaemonId`, so opening `/agent/[serverB]/[agentId]` while daemon A is active leaves the global `SessionProvider`/`RealtimeProvider` pointed at A (see `packages/app/src/app/agent/[serverId]/[agentId].tsx:74-138`).
|
||||
- `AgentInputArea` forwards realtime/voice interactions through `useRealtime()` and the active `ws` (`packages/app/src/components/agent-input-area.tsx:656-744`), so with the mismatch above those commands get sent to daemon A instead of the daemon hosting the open agent.
|
||||
- Added a route-aware effect in `packages/app/src/app/agent/[serverId]/[agentId].tsx` that synchronizes `setActiveDaemonId` with the screen's `serverId` param so the root Session/Realtime providers follow deep links, and ran `npm run typecheck --workspace=@paseo/app`.
|
||||
- [x] Restore the legacy `/agent/[id]` route as a compatibility shim so old deep links (without a daemon id) keep working while we transition the UI.
|
||||
- Added `packages/app/src/app/agent/[id].tsx`, which scans the session directory for the requested agent, auto-redirects when there’s a single match, and lets the user choose when multiple daemons share that id; registered the screen in `_layout.tsx` and re-ran `npm run typecheck --workspace=@paseo/app`.
|
||||
- [x] Keep background agent actions from swapping the active daemon just to open the action sheet or delete.
|
||||
- `AgentList` still called `setActiveDaemonId` on long-press and before deletion (`packages/app/src/components/agent-list.tsx:19-63`), so managing a background daemon’s agent on Home would tear down the active websocket/realtime session. Removed those calls and rely on `useDaemonSession` so the aggregated view no longer hijacks the global session for contextual actions.
|
||||
- [x] Allow guarded screens to opt out of the `useDaemonSession` alert so offline placeholders don’t trigger duplicate system popups.
|
||||
- Added a `suppressUnavailableAlert` option to `useDaemonSession`, had Agent, Git diff, and File Explorer guards opt in so they only render their inline placeholders, and ran `npm run typecheck --workspace=@paseo/app`.
|
||||
- [x] Force the root `SessionProvider` to remount whenever the active daemon changes so session state never leaks between servers.
|
||||
- Added `key={activeDaemon.id}` in `_layout.tsx`, ensuring each daemon gets a fresh session tree so cached agents/permissions don’t show up under the wrong daemon while switching routes.
|
||||
|
||||
```tsx
|
||||
Code: agent-stream-view.tsx
|
||||
317 | return (
|
||||
318 | <View style={stylesheet.container}>
|
||||
> 319 | <FlatList
|
||||
| ^
|
||||
320 | ref={flatListRef}
|
||||
321 | data={flatListData}
|
||||
322 | renderItem={renderStreamItem}
|
||||
```
|
||||
### 3. Agent Creation & Lifecycle Actions
|
||||
- [x] Remove the server selector from the home header; inside the Create/Import modal replace the current “session swap” approach with an explicit `serverId` prop that simply determines which daemon receives the mutation.
|
||||
- Create/Import modals now accept a `serverId` prop, stop mutating the active daemon, and every caller (home screen, footer, agent view) passes the daemon id they’re operating against; ran `npm run typecheck`.
|
||||
- [x] Prevent `useDaemonSession` from throwing during modal render when a daemon is offline—surface that state inside the UI (disabled create button + inline error) and keep the chip selection responsive.
|
||||
- Currently selecting a daemon chip whose session isn’t connected (e.g., auto-connect disabled or still initializing) crashes `CreateAgentModal` because `useDaemonSession` rethrows immediately; we need to gate the selection and show an inline “connect first” state instead of exploding.
|
||||
- Confirmed this is still happening: `CreateAgentModal` passes the selected `serverId` straight into `useDaemonSession` (`packages/app/src/components/create-agent-modal.tsx:233`), so tapping an offline daemon chip kills the modal before we can render error UI.
|
||||
- Updated `CreateAgentModal` to read sessions from the directory, block websocket actions while the target daemon is offline, surface a daemon availability warning, and disable create/import flows until the daemon connects; ran `npm run typecheck`.
|
||||
- [x] When creating/resuming/cloning agents, route follow-up navigation and queued requests to the daemon returned in the success payload rather than assuming the active daemon changed.
|
||||
- `CreateAgentModal` now calls `setActiveDaemonId` with the server from the success payload before pushing the agent route, so the global `SessionProvider`/`RealtimeProvider` swaps to the correct daemon and realtime controls no longer stay bound to the previous server (see `packages/app/src/components/create-agent-modal.tsx:210-220` and `packages/app/src/components/create-agent-modal.tsx:845-864`); re-ran `npm run typecheck`.
|
||||
- [x] Block Create/Import repo + snapshot fetches when the selected daemon is offline so we surface the availability error instead of spinning forever.
|
||||
- Guarded `requestRepoInfo`/`requestImportCandidates` with the daemon availability signal so offline sessions now surface `daemonAvailabilityError` immediately instead of issuing `ws.send`, then added the missing `sheetDeleteTextDisabled` style in `packages/app/src/components/agent-list.tsx` to clear the lingering `npm run typecheck --workspace=@paseo/app` failure.
|
||||
- [x] Restore the Import Agent flow (modal entry, mutation wiring, navigation) without undoing the multi-daemon routing work from previous steps.
|
||||
- After reworking Home/Header and the modals, the import trigger disappeared and existing deep links no longer reach a functioning flow. Bring the Import CTA back (Home, footer, agent screen), ensure it accepts a daemon id, and verify the import mutation routes to the selected daemon without regressing the new server-aware navigation.
|
||||
- Rewired the import buttons across Home (header + empty state with deep-link auto open), the global footer, and the agent action menu so every trigger passes the correct daemon id into `ImportAgentModal`, and ran `npm run typecheck --workspace=@paseo/app`.
|
||||
|
||||
- [x] `npm test` currently fails in `packages/app` with Vitest throwing `Unexpected call to process.send` / `ERR_INVALID_ARG_TYPE` before any tests run. Track down the coverage/pool config causing this and make sure the app suite actually executes (it should include the hydrated tool-call tests mentioned above).
|
||||
- Forced the app Vitest config to use the `forks` pool (keeps `process.send` intact for Expo/xcode helpers) so the suite boots reliably; `npm test --workspace=@voice/app` now runs the harness tests and reproduces the expected hydration failure instead of crashing ahead of collection.
|
||||
- [x] WARN [expo-image-picker] `ImagePicker.MediaTypeOptions` have been deprecated. Use `ImagePicker.MediaType` or an array of `ImagePicker.MediaType` instead.
|
||||
- Replaced the deprecated `MediaTypeOptions.Images` flag with the new literal `["images"]` media type array in `packages/app/src/hooks/use-image-attachment-picker.ts`, silencing the Expo warning when attaching photos. `npm run typecheck --workspace=@voice/app` still fails in the pre-existing stream harness files (unchanged from before this fix).
|
||||
- [x] Update the AgentInput controls so the buttons reflect agent state: when idle show `Dictate` + `Realtime` by default (switch to `Send` when the text box has content); when an agent is running show `Realtime` + `Cancel` regardless of input so you can interrupt without typing. Realtime toggle must remain available in both states.
|
||||
- AgentInput now inspects the agent’s status and swaps the right-hand controls accordingly: send replaces dictate once text/images exist, running agents expose a new Cancel pill, and the realtime toggle is always available (it now starts/stops realtime rather than disappearing). Cancel dispatches a new `cancel_agent_request` websocket message that the server routes through `interruptAgentIfRunning`, so make sure any future API clients send that message when wiring similar controls. (FYI `npm run typecheck --workspace=@voice/app` still fails in the existing stream harness suites for unrelated reasons.)
|
||||
- [x] Implement long press on the agent list item to opena a light bottom sheet with some actions, for now: "Delete agent" all this will do is remove it from our own persistence, but the agent will still exist in the claude/codex persistance and thats out of scope to remove. Its esentially just to remove it from the list, disown it.
|
||||
- Added a `delete_agent_request` flow: the session now closes the agent, removes its registry entry, and emits a dedicated `agent_deleted` event that the client listens for to prune the agents map/stream state. Long-pressing any agent row opens a lightweight bottom sheet with a Delete action wired to that new context method, so deleting disowns the agent locally without touching the Claude/Codex session.
|
||||
- [x] Implement a function `ensureValidJson` that walks an object and validates that all values are valid JSON (no undefined, null, or non-string values). Use this for all return valies in the MCP server tools.
|
||||
- Added `ensureValidJson` (`packages/server/src/server/json-utils.ts`) and wrapped every MCP server tool payload in both `terminal-mcp/server.ts` and `agent/mcp-server.ts`, so tool responses now get validated before streaming; `tsc` still fails earlier in the tree due to `packages/app/src/types/stream.ts` references (see npm run typecheck --workspace=@voice/server).
|
||||
### 4. Connection State & Persistence
|
||||
- [x] Stop the Settings daemon list from firing “Daemon unavailable” alerts when background daemons are offline by reading session snapshots via `useSessionForServer` and only performing restart/test flows when a session is actually mounted.
|
||||
- Updated `packages/app/src/app/settings.tsx` to rely on `useSessionForServer` for both the active daemon and each `DaemonCard`, so offline entries no longer call `useDaemonSession` (which showed alerts) and `npm run typecheck` still passes.
|
||||
- [x] Introduce React Query (or a similar observable store) around AsyncStorage-backed registries (`DaemonRegistryProvider`, `DaemonConnectionsProvider`, app settings) so callers get loading/error states without bespoke hooks.
|
||||
- Added `@tanstack/react-query` with a root provider, refactored the daemon registry, connections, and app settings to load/persist via cached queries (surfacing shared loading/error states) and verified everything with `npm run typecheck --workspace=@paseo/app`.
|
||||
- [x] Add background reconnection + exponential backoff per daemon; surface “connecting/offline/last error” indicators in settings and home.
|
||||
- WebSocket sessions now retry with exponential backoff and feed precise status/error metadata into the daemon connection store, the home screen shows a connection health banner, settings display colored status badges plus last errors for every daemon, and `npm run typecheck --workspace=@paseo/app` passes.
|
||||
- [x] Persist the last successful session snapshot per daemon so the UI can hydrate agent lists immediately while a websocket reconnects.
|
||||
- SessionProviders now hydrate agents/permissions/commands from the last stored `session_state` snapshot, persist new snapshots to AsyncStorage per daemon, and `npm run typecheck --workspace=@paseo/app` passes.
|
||||
- [x] Standardize request/response handling behind a shared hook (React-Query style states for idle/loading/success/error, request dedupe, retries, timeouts) and document how daemon-facing components consume it.
|
||||
- Added `useDaemonRequest` with deduped execution, timeout/retry controls, and React Query-style metadata plus wrote `docs/daemon-request-hook.md` describing how daemon clients consume it; ran `npm run typecheck --workspace=@paseo/app`.
|
||||
- [x] Replace ad-hoc websocket request flows (git info, permission responses, diff/file fetches, etc.) with the new hook so every async action exposes consistent status + cancellation semantics.
|
||||
- Adopted `useDaemonRequest` for repo-inspection modals, permission cards, git diff, and file explorer interactions (with inline loading/error states) and re-ran `npm run typecheck --workspace=@paseo/app`.
|
||||
|
||||
## Voice Interruptions & Streaming
|
||||
### 5. Performance & UX Polish
|
||||
- [x] Ensure agent image attachments preserve MIME metadata and are base64 encoded before hitting the daemon.
|
||||
- `packages/app/src/contexts/session-context.tsx:1198` now accepts `{ uri, mimeType }` attachments and reads them via `expo-file-system`, and `packages/app/src/components/agent-input-area.tsx:140` forwards the stored metadata so queued sends no longer drop screenshots (`npm run typecheck` passes).
|
||||
- [ ] Profile the Create Agent modal—debounce expensive effects (e.g., provider model fetches) per server and prefetch metadata when daemons are idle to eliminate the visible lag when switching targets.
|
||||
- [ ] Audit websocket usage so background `SessionProvider`s never duplicate connections for the active daemon (one live connection per daemon id).
|
||||
- [ ] Enforce “impossible states are impossible” across UI/data models (strict typing, discriminated unions, exhaustive switches) so complex flows remain clean without relying on Expo E2E tests.
|
||||
|
||||
- [x] Wire interrupt signal on speech start so VAD immediately issues `abort_request` to the server before any audio segment is sent.
|
||||
- Realtime speech detection now sends a `session/abort_request` as soon as VAD confirms speech, ensuring any in-flight LLM turn is interrupted before the buffered audio is uploaded (implemented in `packages/app/src/contexts/realtime-context.tsx` with logging/error handling).
|
||||
- [x] Abort server-side playback/LLM as soon as a realtime audio chunk arrives; set a speech-in-progress flag that pauses new TTS until the turn completes.
|
||||
- Added a `speechInProgress` guard in `packages/server/src/server/session.ts` that triggers on the first realtime chunk, cancels pending TTS playback via the new `TTSManager.cancelPendingPlaybacks`, and immediately routes through the abort flow before buffering audio; the flag is cleared when transcription finishes so the next assistant reply can synthesize speech. TTS generation now skips while the flag is set, and `npm run typecheck --workspace=@voice/server` still fails in pre-existing cross-workspace `packages/app/src/types/stream.ts` imports (unchanged by this fix).
|
||||
- [x] Interrupt the currently focused agent whenever a realtime user turn begins, ensuring the next prompt starts fresh.
|
||||
- Session context now tracks a `focusedAgentId` that each agent screen sets/clears on mount, and the realtime provider looks at that value to send a `cancel_agent_request` before issuing voice aborts so the next spoken prompt doesn't pile onto a running turn. Attempted `npm run typecheck --workspace=@voice/app` but it still fails in the pre-existing stream harness/tests complaining about `parsed*` fields.
|
||||
- [x] Extend focused-agent tracking to orchestrator mode so realtime interruptions cancel the agent that’s actually active, even when no agent detail screen is open.
|
||||
- Session context now keeps a derived auto-focus pointing at the most recent running agent whenever no explicit agent screen is selected, so realtime speech aborts send `cancel_agent_request` for that agent before streaming audio; agent screens continue to override the focus via `setFocusedAgentId`.
|
||||
- [x] Keep agent runs alive while speaking to the orchestrator: realtime speech should **not** cancel the focused agent by default. Instead, only interrupt when the user explicitly addresses that agent again or issues a cancel command.
|
||||
- Removed the automatic `cancel_agent_request` that fired on every realtime speech start, so orchestrator dictation no longer tears down whichever agent is in focus; users can still stop runs explicitly via the agent cancel control while the orchestrator abort logic (`abort_request`) remains intact.
|
||||
- [x] Harden playback stop/queue clearing so speech detection purges suppressed audio and the server stops emitting TTS while the user is talking.
|
||||
- Speech detection now aborts any in-flight TTS generation, clears buffered realtime segments (pending chunks plus partial buffers), and cancels pending playbacks before delegating to the existing abort flow, so no additional audio_output events fire once the user starts talking.
|
||||
- [x] Prototype chunked audio input: emit smaller PCM frames with `isLast=false`, update `handleAudioChunk` to buffer and trigger STT once the minimum duration is reached.
|
||||
- `useSpeechmaticsAudio` now streams ~1s PCM payloads throughout a speech turn (plus a final tail chunk) and `RealtimeContext` forwards them with `format: "audio/pcm;rate=16000;bits=16"` + accurate `isLast` so the server sees continuous frames instead of one big WAV. On the backend `Session.handleAudioChunk` tracks PCM bytes, converts buffered samples into WAV once the 1s threshold—or final chunk—arrives, and runs the prior buffering/timeout logic so STT fires repeatedly without waiting for speech end. `npm run typecheck --workspaces --if-present` still fails due to the existing cross-workspace `packages/app/src/types/stream.ts` errors (missing module resolution, tool snapshot typing) that occur on main as well.
|
||||
- [x] Investigate streaming STT/server-side VAD (Whisper or equivalent) and document requirements for production use.
|
||||
- Captured the current audio pipeline, design goals, candidate approaches (self-hosted Whisper vs managed APIs), and production requirements (transport, VAD, streaming STT engine, infra, observability, compliance) in `docs/streaming-stt-vad.md`, along with an implementation phase plan and open questions for the follow-up agent.
|
||||
- [x] Prototype streaming TTS output (chunked `audio_output` messages) and update the player to start playback as soon as chunks arrive.
|
||||
- Split OpenAI TTS responses into chunked `audio_output` payloads via a streaming `TTSManager` pipeline (`packages/server/src/server/agent/tts-manager.ts`, `packages/server/src/server/agent/tts-openai.ts`) that tracks per-chunk acknowledgements, adds utterance IDs, and resolves once every chunk finishes. The app’s session context now keeps audio groups active until the last chunk (or an error) lands so playback begins as soon as each chunk arrives without waiting for the full clip (`packages/app/src/contexts/session-context.tsx`). `npm run typecheck --workspace=@voice/server` and `npm run typecheck --workspace=@voice/app` still fail in the pre-existing cross-workspace stream harness files (see earlier tasks); no new regressions were introduced.
|
||||
- [x] Add telemetry for barge-in latency (speech start → playback stop, chunk arrival → LLM abort) to measure improvements per milestone.
|
||||
- `RealtimeProvider` now records when speech detection interrupts active playback and logs `[Telemetry] barge_in.playback_stop_latency` once the app reports audio stopped, while the server's `Session.handleRealtimeSpeechStart` logs `[Telemetry] barge_in.llm_abort_latency` after aborting live LLM runs—giving us measurable client + server latency metrics per turn.
|
||||
- [x] Codex tool calls aren’t visible in the live stream (they only show up after hydrating/refreshing); fix the reducer/rendering path so Codex tool events appear in real time the same way Claude’s do.
|
||||
- Stream reducer now inspects Codex `provider_event` payloads and converts command/file/MCP/web search items into agent tool entries immediately, so the UI shows the pills while the turn is running instead of waiting for hydration; `stream.test.ts` gained coverage proving provider events alone still surface command + MCP calls, and `npm run test --workspace=@voice/app -- src/types/stream.test.ts` passes.
|
||||
- [x] Update the AgentInput “Cancel” control to use a square stop icon button (consistent with media players) so interrupting an agent is visually distinct from other actions.
|
||||
- Replaced the text-based pill with a compact circular stop control that reuses the Square media glyph (plus accessibility labels), making the interrupt action visually distinct without impacting the other pending AgentInput tweaks.
|
||||
- [x] When the agent input has text and we’re not in the cancellable state, replace both the mic (Dictate) and Realtime buttons with a single Send button—only surface Dictate/Realtime when the input is empty or the agent is running.
|
||||
- `packages/app/src/components/agent-input-area.tsx` now shows a lone Send control when text/images are queued while keeping Dictate + Realtime visible only when the input is empty or the agent is actively running.
|
||||
- [x] Remove the agent status pill next to the permission selector; instead show a “working” indicator inside the chat scroll view itself (three bouncing dots animation) whenever the agent is busy.
|
||||
- Dropped the old status dot from `AgentStatusBar` (the permission/mode selector now stands alone) and introduced an in-stream "Working" chip inside `AgentStreamView` that renders three bouncing dots via Reanimated whenever the agent reports `status === "running"`, so busy turns surface directly in the chat timeline.
|
||||
- [x] npm run typecheck and fix all problems
|
||||
- Split the server build/typecheck configs so the typechecker can include the app stream helpers without breaking builds, added path aliases for `@server/*`, and tightened the shared stream types (status unions + `isAgentToolCallItem`) so the server e2e suites can import them safely. Cleaned up every failing app/server test by narrowing tool-call payloads via the helper, updated the harness + Codex/Claude specs, and re-ran `npm run typecheck` (app + server) to green.
|
||||
- [x] audit codebase for unecessary untyped code, hacks, casts, and `any`, type things properly
|
||||
- Tightened the Codex rollout parser typings so permission requests, rollout/event payloads, plan arguments, and JSON helpers all operate on well-defined TypeScript structures instead of `any`. Added explicit payload/entry guards, removed unsafe casts throughout `packages/server/src/server/agent/providers/codex-agent.ts`, and re-ran `npm run typecheck --workspace=@voice/server` to validate the stricter typing.
|
||||
- [x] audit codebase for duplicated code, clean it up
|
||||
- Consolidated the duplicated home/back header layouts into a shared `ScreenHeader` component so safe-area padding, borders, and spacing live in one place; both headers now only define their unique button/title content. Ran `npm run typecheck --workspace=@voice/app` to confirm the refactor stays type-safe.
|
||||
- [x] rename the project to Paseo, including the Expo app, package names etc.
|
||||
- Renamed the monorepo + workspaces to `paseo`, updated Expo config (name/slug/schemes + bundle IDs), AsyncStorage keys, docs, prompts, and helper text to use `@paseo/*`, and refreshed the CLI scripts + descriptions so every entry point references Paseo instead of Voice Dev. Regenerated the lockfile + pods metadata and re-ran `npm run typecheck --workspaces --if-present` to confirm both server and app builds still typecheck under the new branding.
|
||||
### 6. Observability & Tooling
|
||||
- [ ] Add structured logging for daemon connection lifecycle (connect, error, auto-connect skip) so we can diagnose “multi daemon” issues from device logs.
|
||||
- [ ] Emit analytics when users create/resume agents on background daemons, attempt actions while those daemons are offline, or switch default daemons—helps prioritize reconnection UX.
|
||||
- [ ] Document the architecture in `docs/multi-daemon.md` (registry, sessions, routing rules) so future contributors understand how to extend it.
|
||||
- [ ] Land the accumulated multi-daemon changes in source control with a clean commit (linted, type-checked, plan updated).
|
||||
|
||||
### Review
|
||||
- [x] 2025-11-26 Reviewer sanity check for the recent multi-daemon rollout work.
|
||||
- Confirmed the new `useDaemonSession` hook, guarded Agent/Git Diff/File Explorer screens, and server-aware Create/Import flows align with the documented fixes; no regressions or missing follow-ups spotted, so no additional tasks were opened.
|
||||
- [x] 2025-11-26 Reviewer follow-up on session isolation across daemons.
|
||||
- Found that the active `SessionProvider` kept its React state when switching daemons, so stale agents/permissions could leak between server contexts; fixed by keying the provider in `_layout.tsx` so the tree remounts on each daemon change.
|
||||
|
||||
Reference in New Issue
Block a user