jb/paseo
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

Stop emitting TTS after speech detection

Browse Source
This commit is contained in:
Mohamed Boudra
2025-11-15 18:41:31 +01:00
parent 8202985bad
commit 104aae95da
2 changed files with 20 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plan.md
+2 -1
View File
@@ -60,7 +60,8 @@
- Added a `speechInProgress` guard in `packages/server/src/server/session.ts` that triggers on the first realtime chunk, cancels pending TTS playback via the new `TTSManager.cancelPendingPlaybacks`, and immediately routes through the abort flow before buffering audio; the flag is cleared when transcription finishes so the next assistant reply can synthesize speech. TTS generation now skips while the flag is set, and `npm run typecheck --workspace=@voice-dev/server` still fails in pre-existing cross-workspace `packages/app/src/types/stream.ts` imports (unchanged by this fix).
- [x] Interrupt the currently focused agent whenever a realtime user turn begins, ensuring the next prompt starts fresh.
- Session context now tracks a `focusedAgentId` that each agent screen sets/clears on mount, and the realtime provider looks at that value to send a `cancel_agent_request` before issuing voice aborts so the next spoken prompt doesn't pile onto a running turn. Attempted `npm run typecheck --workspace=@voice-dev/app` but it still fails in the pre-existing stream harness/tests complaining about `parsed*` fields.
- [ ] Harden playback stop/queue clearing so speech detection purges suppressed audio and the server stops emitting TTS while the user is talking.
- [x] Harden playback stop/queue clearing so speech detection purges suppressed audio and the server stops emitting TTS while the user is talking.
- Speech detection now aborts any in-flight TTS generation, clears buffered realtime segments (pending chunks plus partial buffers), and cancels pending playbacks before delegating to the existing abort flow, so no additional audio_output events fire once the user starts talking.
- [ ] Prototype chunked audio input: emit smaller PCM frames with `isLast=false`, update `handleAudioChunk` to buffer and trigger STT once the minimum duration is reached.
- [ ] Investigate streaming STT/server-side VAD (Whisper or equivalent) and document requirements for production use.
- [ ] Prototype streaming TTS output (chunked `audio_output` messages) and update the player to start playback as soon as chunks arrive.