jb/paseo
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

auto-focus orchestrator agent during realtime

Browse Source
This commit is contained in:
Mohamed Boudra
2025-11-15 19:18:39 +01:00
parent 7284efd0fe
commit 24a245bd17
2 changed files with 38 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plan.md
+3
View File
@@ -60,6 +60,9 @@
- Added a `speechInProgress` guard in `packages/server/src/server/session.ts` that triggers on the first realtime chunk, cancels pending TTS playback via the new `TTSManager.cancelPendingPlaybacks`, and immediately routes through the abort flow before buffering audio; the flag is cleared when transcription finishes so the next assistant reply can synthesize speech. TTS generation now skips while the flag is set, and `npm run typecheck --workspace=@voice-dev/server` still fails in pre-existing cross-workspace `packages/app/src/types/stream.ts` imports (unchanged by this fix).
- [x] Interrupt the currently focused agent whenever a realtime user turn begins, ensuring the next prompt starts fresh.
- Session context now tracks a `focusedAgentId` that each agent screen sets/clears on mount, and the realtime provider looks at that value to send a `cancel_agent_request` before issuing voice aborts so the next spoken prompt doesn't pile onto a running turn. Attempted `npm run typecheck --workspace=@voice-dev/app` but it still fails in the pre-existing stream harness/tests complaining about `parsed*` fields.
- [x] Extend focused-agent tracking to orchestrator mode so realtime interruptions cancel the agent that’s actually active, even when no agent detail screen is open.
- Session context now keeps a derived auto-focus pointing at the most recent running agent whenever no explicit agent screen is selected, so realtime speech aborts send `cancel_agent_request` for that agent before streaming audio; agent screens continue to override the focus via `setFocusedAgentId`.
- [ ] Keep agent runs alive while speaking to the orchestrator: realtime speech should **not** cancel the focused agent by default. Instead, only interrupt when the user explicitly addresses that agent again or issues a cancel command.
- [x] Harden playback stop/queue clearing so speech detection purges suppressed audio and the server stops emitting TTS while the user is talking.
- Speech detection now aborts any in-flight TTS generation, clears buffered realtime segments (pending chunks plus partial buffers), and cancels pending playbacks before delegating to the existing abort flow, so no additional audio_output events fire once the user starts talking.
- [x] Prototype chunked audio input: emit smaller PCM frames with `isLast=false`, update `handleAudioChunk` to buffer and trigger STT once the minimum duration is reached.