jb/paseo
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

Interrupt focused agent before realtime turns

Browse Source
This commit is contained in:
Mohamed Boudra
2025-11-15 18:35:13 +01:00
parent 377fa196f8
commit 8202985bad
4 changed files with 40 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plan.md
+2 -1
View File
@@ -58,7 +58,8 @@
- Realtime speech detection now sends a `session/abort_request` as soon as VAD confirms speech, ensuring any in-flight LLM turn is interrupted before the buffered audio is uploaded (implemented in `packages/app/src/contexts/realtime-context.tsx` with logging/error handling).
- [x] Abort server-side playback/LLM as soon as a realtime audio chunk arrives; set a speech-in-progress flag that pauses new TTS until the turn completes.
- Added a `speechInProgress` guard in `packages/server/src/server/session.ts` that triggers on the first realtime chunk, cancels pending TTS playback via the new `TTSManager.cancelPendingPlaybacks`, and immediately routes through the abort flow before buffering audio; the flag is cleared when transcription finishes so the next assistant reply can synthesize speech. TTS generation now skips while the flag is set, and `npm run typecheck --workspace=@voice-dev/server` still fails in pre-existing cross-workspace `packages/app/src/types/stream.ts` imports (unchanged by this fix).
- [ ] Interrupt the currently focused agent whenever a realtime user turn begins, ensuring the next prompt starts fresh.
- [x] Interrupt the currently focused agent whenever a realtime user turn begins, ensuring the next prompt starts fresh.
- Session context now tracks a `focusedAgentId` that each agent screen sets/clears on mount, and the realtime provider looks at that value to send a `cancel_agent_request` before issuing voice aborts so the next spoken prompt doesn't pile onto a running turn. Attempted `npm run typecheck --workspace=@voice-dev/app` but it still fails in the pre-existing stream harness/tests complaining about `parsed*` fields.
- [ ] Harden playback stop/queue clearing so speech detection purges suppressed audio and the server stops emitting TTS while the user is talking.
- [ ] Prototype chunked audio input: emit smaller PCM frames with `isLast=false`, update `handleAudioChunk` to buffer and trigger STT once the minimum duration is reached.
- [ ] Investigate streaming STT/server-side VAD (Whisper or equivalent) and document requirements for production use.