jb/paseo
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

Prototype streaming TTS chunks

Browse Source
This commit is contained in:
Mohamed Boudra
2025-11-15 19:06:53 +01:00
parent 561e164b02
commit 7284efd0fe
5 changed files with 163 additions and 87 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plan.md
+6 -1
View File
@@ -66,5 +66,10 @@
- `useSpeechmaticsAudio` now streams ~1s PCM payloads throughout a speech turn (plus a final tail chunk) and `RealtimeContext` forwards them with `format: "audio/pcm;rate=16000;bits=16"` + accurate `isLast` so the server sees continuous frames instead of one big WAV. On the backend `Session.handleAudioChunk` tracks PCM bytes, converts buffered samples into WAV once the 1s threshold—or final chunk—arrives, and runs the prior buffering/timeout logic so STT fires repeatedly without waiting for speech end. `npm run typecheck --workspaces --if-present` still fails due to the existing cross-workspace `packages/app/src/types/stream.ts` errors (missing module resolution, tool snapshot typing) that occur on main as well.
- [x] Investigate streaming STT/server-side VAD (Whisper or equivalent) and document requirements for production use.
- Captured the current audio pipeline, design goals, candidate approaches (self-hosted Whisper vs managed APIs), and production requirements (transport, VAD, streaming STT engine, infra, observability, compliance) in `docs/streaming-stt-vad.md`, along with an implementation phase plan and open questions for the follow-up agent.
- [ ] Prototype streaming TTS output (chunked `audio_output` messages) and update the player to start playback as soon as chunks arrive.
- [x] Prototype streaming TTS output (chunked `audio_output` messages) and update the player to start playback as soon as chunks arrive.
- Split OpenAI TTS responses into chunked `audio_output` payloads via a streaming `TTSManager` pipeline (`packages/server/src/server/agent/tts-manager.ts`, `packages/server/src/server/agent/tts-openai.ts`) that tracks per-chunk acknowledgements, adds utterance IDs, and resolves once every chunk finishes. The app’s session context now keeps audio groups active until the last chunk (or an error) lands so playback begins as soon as each chunk arrives without waiting for the full clip (`packages/app/src/contexts/session-context.tsx`). `npm run typecheck --workspace=@voice-dev/server` and `npm run typecheck --workspace=@voice-dev/app` still fail in the pre-existing cross-workspace stream harness files (see earlier tasks); no new regressions were introduced.
- [ ] Add telemetry for barge-in latency (speech start → playback stop, chunk arrival → LLM abort) to measure improvements per milestone.
- [ ] Codex tool calls aren’t visible in the live stream (they only show up after hydrating/refreshing); fix the reducer/rendering path so Codex tool events appear in real time the same way Claude’s do.
- [ ] Update the AgentInput “Cancel” control to use a square stop icon button (consistent with media players) so interrupting an agent is visually distinct from other actions.
- [ ] When the agent input has text and we’re not in the cancellable state, replace both the mic (Dictate) and Realtime buttons with a single Send button—only surface Dictate/Realtime when the input is empty or the agent is running.
- [ ] Remove the agent status pill next to the permission selector; instead show a “working” indicator inside the chat scroll view itself (three bouncing dots animation) whenever the agent is busy.