From 561e164b02beb33c0c622045ce9ebfc38f3ad39e Mon Sep 17 00:00:00 2001 From: Mohamed Boudra Date: Sat, 15 Nov 2025 18:54:37 +0100 Subject: [PATCH] Document streaming STT requirements --- plan.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/plan.md b/plan.md index 9c65cfbfb..03a706cd9 100644 --- a/plan.md +++ b/plan.md @@ -62,7 +62,9 @@ - Session context now tracks a `focusedAgentId` that each agent screen sets/clears on mount, and the realtime provider looks at that value to send a `cancel_agent_request` before issuing voice aborts so the next spoken prompt doesn't pile onto a running turn. Attempted `npm run typecheck --workspace=@voice-dev/app` but it still fails in the pre-existing stream harness/tests complaining about `parsed*` fields. - [x] Harden playback stop/queue clearing so speech detection purges suppressed audio and the server stops emitting TTS while the user is talking. - Speech detection now aborts any in-flight TTS generation, clears buffered realtime segments (pending chunks plus partial buffers), and cancels pending playbacks before delegating to the existing abort flow, so no additional audio_output events fire once the user starts talking. -- [ ] Prototype chunked audio input: emit smaller PCM frames with `isLast=false`, update `handleAudioChunk` to buffer and trigger STT once the minimum duration is reached. -- [ ] Investigate streaming STT/server-side VAD (Whisper or equivalent) and document requirements for production use. +- [x] Prototype chunked audio input: emit smaller PCM frames with `isLast=false`, update `handleAudioChunk` to buffer and trigger STT once the minimum duration is reached. + - `useSpeechmaticsAudio` now streams ~1s PCM payloads throughout a speech turn (plus a final tail chunk) and `RealtimeContext` forwards them with `format: "audio/pcm;rate=16000;bits=16"` + accurate `isLast` so the server sees continuous frames instead of one big WAV. On the backend `Session.handleAudioChunk` tracks PCM bytes, converts buffered samples into WAV once the 1s threshold—or final chunk—arrives, and runs the prior buffering/timeout logic so STT fires repeatedly without waiting for speech end. `npm run typecheck --workspaces --if-present` still fails due to the existing cross-workspace `packages/app/src/types/stream.ts` errors (missing module resolution, tool snapshot typing) that occur on main as well. +- [x] Investigate streaming STT/server-side VAD (Whisper or equivalent) and document requirements for production use. + - Captured the current audio pipeline, design goals, candidate approaches (self-hosted Whisper vs managed APIs), and production requirements (transport, VAD, streaming STT engine, infra, observability, compliance) in `docs/streaming-stt-vad.md`, along with an implementation phase plan and open questions for the follow-up agent. - [ ] Prototype streaming TTS output (chunked `audio_output` messages) and update the player to start playback as soon as chunks arrive. - [ ] Add telemetry for barge-in latency (speech start → playback stop, chunk arrival → LLM abort) to measure improvements per milestone.