jb/paseo
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

Document streaming STT requirements

Browse Source
This commit is contained in:
Mohamed Boudra
2025-11-15 18:54:37 +01:00
parent 104aae95da
commit 561e164b02
1 changed files with 4 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plan.md
+4 -2
View File
@@ -62,7 +62,9 @@
- Session context now tracks a `focusedAgentId` that each agent screen sets/clears on mount, and the realtime provider looks at that value to send a `cancel_agent_request` before issuing voice aborts so the next spoken prompt doesn't pile onto a running turn. Attempted `npm run typecheck --workspace=@voice-dev/app` but it still fails in the pre-existing stream harness/tests complaining about `parsed*` fields.
- [x] Harden playback stop/queue clearing so speech detection purges suppressed audio and the server stops emitting TTS while the user is talking.
- Speech detection now aborts any in-flight TTS generation, clears buffered realtime segments (pending chunks plus partial buffers), and cancels pending playbacks before delegating to the existing abort flow, so no additional audio_output events fire once the user starts talking.
- [ ] Prototype chunked audio input: emit smaller PCM frames with `isLast=false`, update `handleAudioChunk` to buffer and trigger STT once the minimum duration is reached.
- [ ] Investigate streaming STT/server-side VAD (Whisper or equivalent) and document requirements for production use.
- [x] Prototype chunked audio input: emit smaller PCM frames with `isLast=false`, update `handleAudioChunk` to buffer and trigger STT once the minimum duration is reached.
- `useSpeechmaticsAudio` now streams ~1s PCM payloads throughout a speech turn (plus a final tail chunk) and `RealtimeContext` forwards them with `format: "audio/pcm;rate=16000;bits=16"` + accurate `isLast` so the server sees continuous frames instead of one big WAV. On the backend `Session.handleAudioChunk` tracks PCM bytes, converts buffered samples into WAV once the 1s threshold—or final chunk—arrives, and runs the prior buffering/timeout logic so STT fires repeatedly without waiting for speech end. `npm run typecheck --workspaces --if-present` still fails due to the existing cross-workspace `packages/app/src/types/stream.ts` errors (missing module resolution, tool snapshot typing) that occur on main as well.
- [x] Investigate streaming STT/server-side VAD (Whisper or equivalent) and document requirements for production use.
- Captured the current audio pipeline, design goals, candidate approaches (self-hosted Whisper vs managed APIs), and production requirements (transport, VAD, streaming STT engine, infra, observability, compliance) in `docs/streaming-stt-vad.md`, along with an implementation phase plan and open questions for the follow-up agent.
- [ ] Prototype streaming TTS output (chunked `audio_output` messages) and update the player to start playback as soon as chunks arrive.
- [ ] Add telemetry for barge-in latency (speech start → playback stop, chunk arrival → LLM abort) to measure improvements per milestone.