feat(voice): implement turn-detection and real-time audio streaming

docs/VOICE_TURN_DETECTION_PLAN.md

@@ -0,0 +1,581 @@
+# Local Voice Turn Detection Plan
+
+## Goal
+
+For local realtime voice, the client should stream PCM continuously to the server. The server should own turn detection using bundled Sherpa + Silero VAD. Client-side logic stays only for meter/activation visuals and is not authoritative for turn boundaries.
+
+The server should start buffering immediately so we do not clip the beginning of speech.
+
+## Scope
+
+This plan is intentionally local-only for now.
+
+- Turn detection: local only
+- STT: local only
+- TTS: unchanged
+- Client VAD: removed as an authority, retained only as UI metering/activation
+
+## Design Summary
+
+### Boundaries
+
+- Client
+  - Captures PCM
+  - Streams fixed-size chunks continuously during voice mode
+  - Shows volume meter / activation visuals
+  - Does not decide turn start or turn end
+- Server turn detection
+  - Consumes live PCM stream
+  - Runs local Silero VAD through Sherpa
+  - Emits normalized events:
+    - `speech_started`
+    - `speech_stopped`
+- Server voice turn controller
+  - Owns barge-in
+  - Owns rolling pre-speech buffer
+  - Owns utterance lifecycle
+  - Feeds utterance audio into the existing STT path
+- Server STT
+  - Transcribes utterance audio selected by turn detection
+
+## Core Principle
+
+Do not hide VAD inside STT.
+
+If STT owns VAD, the abstraction gets coupled again. Turn detection and transcription are different responsibilities and should be modeled separately.
+
+Bad shape:
+
+```ts
+interface SpeechToTextProvider {
+  createSession(): {
+    appendPcm16(chunk: Buffer): void;
+    commit(): void;
+    on(event: "speech_started" | "speech_stopped" | "transcript", handler: unknown): unknown;
+  };
+}
+```
+
+Better shape:
+
+```ts
+interface TurnDetectionProvider {
+  createSession(input: {
+    logger: pino.Logger;
+  }): TurnDetectionSession;
+}
+
+interface TurnDetectionSession {
+  requiredSampleRate: number;
+  connect(): Promise<void>;
+  appendPcm16(chunk: Buffer): void;
+  close(): void;
+
+  on(event: "speech_started", handler: () => void): unknown;
+  on(event: "speech_stopped", handler: () => void): unknown;
+  on(event: "error", handler: (err: unknown) => void): unknown;
+}
+```
+
+And keep STT separate:
+
+```ts
+interface SpeechToTextProvider {
+  id: "local" | "openai" | (string & {});
+  createSession(input: {
+    logger: pino.Logger;
+    language?: string;
+    prompt?: string;
+  }): StreamingTranscriptionSession;
+}
+```
+
+## Config Shape
+
+Even in a local-only rollout, the boundary should be explicit now so it does not need a second refactor later.
+
+Current shape:
+
+```ts
+type RequestedSpeechProviders = {
+  dictationStt: RequestedSpeechProvider;
+  voiceStt: RequestedSpeechProvider;
+  voiceTts: RequestedSpeechProvider;
+};
+```
+
+Proposed shape:
+
+```ts
+type RequestedSpeechProviders = {
+  dictationStt: RequestedSpeechProvider;
+  voiceTurnDetection: RequestedSpeechProvider;
+  voiceStt: RequestedSpeechProvider;
+  voiceTts: RequestedSpeechProvider;
+};
+```
+
+For the first implementation, resolution can still force local:
+
+```ts
+const providers = {
+  dictationStt: { provider: "local", explicit: false, enabled: true },
+  voiceTurnDetection: { provider: "local", explicit: false, enabled: true },
+  voiceStt: { provider: "local", explicit: false, enabled: true },
+  voiceTts: { provider: "local", explicit: false, enabled: true },
+};
+```
+
+## Runtime Shape
+
+### 1. Local Turn Detection Provider
+
+Add a local provider parallel to local STT/TTS runtime wiring.
+
+Suggested files:
+
+- `packages/server/src/server/speech/turn-detection-provider.ts`
+- `packages/server/src/server/speech/providers/local/sherpa/silero-vad-session.ts`
+- `packages/server/src/server/speech/providers/local/sherpa/silero-vad-provider.ts`
+
+Expanded local runtime output:
+
+```ts
+interface InitializedLocalSpeech {
+  sttService: SpeechToTextProvider | null;
+  ttsService: TextToSpeechProvider | null;
+  dictationSttService: SpeechToTextProvider | null;
+  turnDetectionService: TurnDetectionProvider | null;
+  cleanup: () => void;
+}
+```
+
+Provider construction in local runtime:
+
+```ts
+if (
+  providers.voiceTurnDetection.enabled !== false &&
+  providers.voiceTurnDetection.provider === "local"
+) {
+  turnDetectionService = new SherpaSileroTurnDetectionProvider(
+    {
+      modelPath: bundledSileroModelPath,
+      sampleRate: 16000,
+    },
+    logger
+  );
+}
+```
+
+### 2. Voice Turn Controller
+
+This should sit in the server session layer and own the utterance state machine.
+
+Suggested file:
+
+- `packages/server/src/server/voice/voice-turn-controller.ts`
+
+Responsibilities:
+
+- receive continuous PCM chunks
+- write chunks into a rolling pre-speech buffer
+- feed chunks into detector
+- on `speech_started`
+  - perform barge-in
+  - start a new utterance
+  - flush pre-speech buffer into utterance
+- while speech is active
+  - append live chunks into utterance
+- on `speech_stopped`
+  - finalize utterance into STT
+
+Public shape:
+
+```ts
+interface VoiceTurnController {
+  start(): Promise<void>;
+  stop(): Promise<void>;
+  appendClientChunk(input: {
+    audioBase64: string;
+    format: string;
+  }): Promise<void>;
+}
+```
+
+Internal state should be explicit:
+
+```ts
+type VoiceInputState =
+  | { status: "idle" }
+  | {
+      status: "listening";
+      rollingPrefixBytes: number;
+    }
+  | {
+      status: "capturing";
+      utteranceId: string;
+      rollingPrefixBytes: number;
+      utteranceBytes: number;
+    };
+```
+
+That is better than several booleans like `speechInProgress`, `hasStarted`, or `awaitingStop`.
+
+## Immediate Buffering Strategy
+
+Because buffering starts immediately, the detector does not decide whether audio is kept at all. It decides when buffered audio becomes an utterance.
+
+### Rolling Prefix Buffer
+
+Maintain a small ring buffer of recent PCM before speech start.
+
+Suggested defaults:
+
+- sample rate: `16000`
+- mono PCM16
+- prefix duration: `300ms` to `500ms`
+
+Example utility:
+
+```ts
+interface RollingPcmBuffer {
+  append(chunk: Buffer): void;
+  drain(): Buffer;
+  clear(): void;
+}
+```
+
+Example implementation shape:
+
+```ts
+class FixedDurationPcmRingBuffer {
+  private chunks: Buffer[] = [];
+  private totalBytes = 0;
+
+  constructor(private readonly maxBytes: number) {}
+
+  append(chunk: Buffer): void {
+    if (chunk.length === 0) {
+      return;
+    }
+
+    this.chunks.push(chunk);
+    this.totalBytes += chunk.length;
+
+    while (this.totalBytes > this.maxBytes && this.chunks.length > 0) {
+      const removed = this.chunks.shift()!;
+      this.totalBytes -= removed.length;
+    }
+  }
+
+  drain(): Buffer {
+    return Buffer.concat(this.chunks);
+  }
+
+  clear(): void {
+    this.chunks = [];
+    this.totalBytes = 0;
+  }
+}
+```
+
+### Flow
+
+Per incoming chunk:
+
+```ts
+prefixBuffer.append(chunk);
+detector.appendPcm16(chunk);
+
+if (state.status === "capturing") {
+  utteranceChunks.push(chunk);
+}
+```
+
+On speech start:
+
+```ts
+async function onSpeechStarted(): Promise<void> {
+  await interruptPlaybackAndAgentRun();
+
+  const prefix = prefixBuffer.drain();
+  const utteranceChunks = prefix.length > 0 ? [prefix] : [];
+
+  state = {
+    status: "capturing",
+    utteranceId: uuidv4(),
+    rollingPrefixBytes: prefix.length,
+    utteranceBytes: prefix.length,
+  };
+
+  activeUtteranceChunks = utteranceChunks;
+}
+```
+
+On speech stop:
+
+```ts
+async function onSpeechStopped(): Promise<void> {
+  if (state.status !== "capturing") {
+    return;
+  }
+
+  const pcm16 = Buffer.concat(activeUtteranceChunks);
+
+  await submitUtterance({
+    pcm16,
+    sampleRate: 16000,
+    format: "audio/pcm;rate=16000;bits=16",
+  });
+
+  activeUtteranceChunks = [];
+  state = {
+    status: "listening",
+    rollingPrefixBytes: 0,
+  };
+}
+```
+
+## STT Integration
+
+Do not make the turn controller talk to STT one chunk at a time unless there is a strong need.
+
+For the first version, the simplest shape is:
+
+- detector runs on the continuous stream
+- utterance audio is accumulated in memory
+- on stop, the utterance goes through the existing internal voice dictation path
+
+That minimizes refactor risk and keeps the first migration focused on authority and layering.
+
+Boundary:
+
+```ts
+interface VoiceUtteranceSink {
+  submitUtterance(input: {
+    pcm16: Buffer;
+    sampleRate: number;
+    format: string;
+  }): Promise<void>;
+}
+```
+
+Then `session.ts` can implement that sink using the existing voice dictation internals.
+
+## Session Refactor
+
+Current problem:
+
+- raw `voice_audio_chunk` handling triggers speech-start behavior too early
+- the client remains the authority for `isLast`
+
+Target shape:
+
+```ts
+private async handleAudioChunk(
+  msg: Extract<SessionInboundMessage, { type: "voice_audio_chunk" }>
+): Promise<void> {
+  if (this.isVoiceMode) {
+    await this.voiceTurnController.appendClientChunk({
+      audioBase64: msg.audio,
+      format: msg.format || "audio/pcm;rate=16000;bits=16",
+    });
+    return;
+  }
+
+  // existing non-voice path remains
+}
+```
+
+And the controller calls back into the session:
+
+```ts
+interface VoiceTurnControllerCallbacks {
+  onSpeechStarted(): Promise<void>;
+  onUtteranceReady(input: {
+    pcm16: Buffer;
+    sampleRate: number;
+    format: string;
+  }): Promise<void>;
+  onError(error: Error): void;
+}
+```
+
+Session remains responsible for:
+
+- aborting agent/TTS
+- submitting transcript text to the agent
+- emitting logs
+
+The controller is responsible for:
+
+- audio and detector state
+- prefix buffering
+- utterance assembly
+
+## Client Refactor
+
+The app should stop using `SpeechSegmenter` as authoritative segmentation in voice mode.
+
+Instead, voice mode uses a continuous chunk sender:
+
+```ts
+interface ContinuousVoiceUploader {
+  appendPcmChunk(chunk: Uint8Array): void;
+  flush(): Promise<void>;
+}
+```
+
+In `voice-runtime.ts`, the path becomes:
+
+```ts
+handleCapturePcm(chunk) {
+  if (!state.snapshot.isVoiceMode || state.snapshot.isMuted) {
+    return;
+  }
+
+  continuousUploader.appendPcmChunk(chunk);
+}
+```
+
+And UI activation remains independent:
+
+```ts
+handleCaptureVolume(level) {
+  const nowMs = Date.now();
+  publishDisplayVolume(level, nowMs);
+
+  const isActive = level > DISPLAY_ACTIVE_THRESHOLD;
+  patchTelemetry((prev) => ({
+    ...prev,
+    isSpeaking: isActive,
+  }));
+}
+```
+
+The client should no longer use a “segmenter” abstraction for voice mode. The concept is now wrong for that layer.
+
+## Bundling Silero
+
+Silero VAD should be treated as a bundled runtime asset, not a user-downloaded speech model.
+
+Implications:
+
+- not listed in `paseo speech download`
+- not part of user-facing missing-model readiness
+- shipped with `packages/server`
+
+Example:
+
+```ts
+function resolveBundledSileroVadModelPath(): string {
+  return fileURLToPath(
+    new URL("./assets/silero_vad.onnx", import.meta.url)
+  );
+}
+```
+
+That keeps the model an implementation detail of local realtime voice infrastructure.
+
+## What Not To Do In V1
+
+- No semantic turn detection
+- No mixed authorities
+- No client-side fallback turn detection
+- No detector logic buried inside dictation manager
+- No detector-specific branching spread through session logic
+
+If local VAD is the authority, then only local VAD decides `speech_started` and `speech_stopped`.
+
+## Verification Plan
+
+### Unit Tests
+
+`silero-vad-session.test.ts`
+
+- emits `speech_started` after speech frames
+- emits `speech_stopped` after silence frames
+- does not emit duplicate starts/stops
+
+`fixed-duration-pcm-ring-buffer.test.ts`
+
+- retains only configured prefix window
+- drains prefix in correct order
+
+`voice-turn-controller.test.ts`
+
+- buffers before speech start
+- includes prefix in utterance on start
+- finalizes one utterance on stop
+- does not barge in on silence-only chunks
+
+### Integration Tests
+
+`session.voice-turn-control.test.ts`
+
+- continuous voice chunks do not immediately interrupt agent output
+- detector start triggers barge-in
+- detector stop triggers one finalized transcript submission
+
+`voice-local-agent.e2e.test.ts`
+
+- local voice mode works with continuous client upload
+- leading words are preserved because prefix buffering is included
+
+### Regression
+
+- dictation stream tests remain unchanged
+- TTS playback tests remain unchanged
+- app voice UI tests still verify meter/activation behavior
+
+## Ordered Implementation Plan
+
+1. Add server-side turn detection interfaces and config type.
+2. Add bundled local Silero provider via Sherpa.
+3. Add rolling prefix buffer utility.
+4. Add `VoiceTurnController`.
+5. Refactor `session.ts` voice path to use controller callbacks instead of client `isLast`.
+6. Replace app voice segmentation with continuous PCM upload.
+7. Keep app meter/activation visuals only.
+8. Add tests bottom-up.
+9. Run `npm run typecheck`.
+
+## Suggested Canonical Types
+
+```ts
+interface DetectedVoiceUtterance {
+  pcm16: Buffer;
+  sampleRate: number;
+  startedAt: number;
+  endedAt: number;
+}
+```
+
+```ts
+interface VoiceTurnControllerCallbacks {
+  onSpeechStarted(): Promise<void>;
+  onUtteranceReady(utterance: DetectedVoiceUtterance): Promise<void>;
+  onError(error: Error): void;
+}
+```
+
+```ts
+interface TurnDetectionProvider {
+  id: "local" | "openai" | (string & {});
+  createSession(input: {
+    logger: pino.Logger;
+  }): TurnDetectionSession;
+}
+```
+
+## Recommendation
+
+For v1, optimize for correctness and clean layering rather than minimum diff size:
+
+- continuous upload from client
+- bundled local Silero detector on server
+- rolling prefix buffer
+- detector-owned turn boundaries
+- existing STT pipeline reused as utterance sink
+
+That keeps responsibilities clear and avoids inventing a leaky “smart STT session” abstraction.

packages/app/e2e/workspace-draft-tab-retarget.spec.ts

@@ -1,6 +1,6 @@
 import { test, expect } from "./fixtures";
 import type { Page } from "@playwright/test";
-import { createAgentInRepo } from "./helpers/app";
+import { createAgentInRepo, ensureHostSelected, gotoHome } from "./helpers/app";
 import { createTempGitRepo } from "./helpers/workspace";
 import {
   ensureWorkspaceAgentPaneVisible,
@@ -9,6 +9,7 @@ import {
   waitForWorkspaceTabsVisible,
 } from "./helpers/workspace-tabs";
 import { switchWorkspaceViaSidebar } from "./helpers/workspace-ui";
+import { buildHostWorkspaceRouteWithOpenIntent } from "@/utils/host-routes";
 
 async function expectComposerFocused(page: Page) {
   const composer = page.getByRole("textbox", { name: "Message agent..." }).first();
@@ -123,3 +124,142 @@ test("workspace agent tab switch focuses composer on desktop web", async ({ page
     await repo.cleanup();
   }
 });
+
+test("workspace draft tabs keep separate prompt state", async ({ page }) => {
+  const serverId = process.env.E2E_SERVER_ID;
+  if (!serverId) {
+    throw new Error("E2E_SERVER_ID is not set.");
+  }
+
+  const repo = await createTempGitRepo("paseo-e2e-draft-isolation-");
+  const seedPrompt = `seed isolation ${Date.now()}`;
+  const firstDraftPrompt = `first draft ${Date.now()}`;
+  const secondDraftPrompt = `second draft ${Date.now()}`;
+
+  try {
+    await createAgentInRepo(page, { directory: repo.path, prompt: seedPrompt });
+
+    await switchWorkspaceViaSidebar({ page, serverId, targetWorkspacePath: repo.path });
+    await waitForWorkspaceTabsVisible(page);
+    await ensureWorkspaceAgentPaneVisible(page);
+
+    const beforeFirstDraftIds = await getWorkspaceTabTestIds(page);
+    await page.getByTestId("workspace-new-agent-tab").first().click();
+    await expectComposerFocused(page);
+
+    const withFirstDraftIds = await getWorkspaceTabTestIds(page);
+    const firstDraftTabTestId = withFirstDraftIds.find((id) => !beforeFirstDraftIds.includes(id));
+    if (!firstDraftTabTestId) {
+      throw new Error("Expected first draft tab to be created.");
+    }
+
+    const composer = page.getByRole("textbox", { name: "Message agent..." }).first();
+    await composer.fill(firstDraftPrompt);
+
+    const beforeSecondDraftIds = await getWorkspaceTabTestIds(page);
+    await page.getByTestId("workspace-new-agent-tab").first().click();
+    await expectComposerFocused(page);
+
+    const withSecondDraftIds = await getWorkspaceTabTestIds(page);
+    const secondDraftTabTestId = withSecondDraftIds.find((id) => !beforeSecondDraftIds.includes(id));
+    if (!secondDraftTabTestId) {
+      throw new Error("Expected second draft tab to be created.");
+    }
+
+    await composer.fill(secondDraftPrompt);
+
+    await page.getByTestId(firstDraftTabTestId).first().click();
+    await expect(composer).toHaveValue(firstDraftPrompt);
+
+    await page.getByTestId(secondDraftTabTestId).first().click();
+    await expect(composer).toHaveValue(secondDraftPrompt);
+  } finally {
+    await repo.cleanup();
+  }
+});
+
+test("workspace draft promotion does not steal focus from another tab", async ({ page }) => {
+  const serverId = process.env.E2E_SERVER_ID;
+  if (!serverId) {
+    throw new Error("E2E_SERVER_ID is not set.");
+  }
+
+  const repo = await createTempGitRepo("paseo-e2e-draft-no-focus-steal-");
+  const seedPrompt = `seed focus ${Date.now()}`;
+  const createPrompt = `background create ${Date.now()}`;
+
+  try {
+    await createAgentInRepo(page, { directory: repo.path, prompt: seedPrompt });
+
+    await switchWorkspaceViaSidebar({ page, serverId, targetWorkspacePath: repo.path });
+    await waitForWorkspaceTabsVisible(page);
+    await ensureWorkspaceAgentPaneVisible(page);
+
+    const beforeDraftIds = await getWorkspaceTabTestIds(page);
+    const firstAgentTabTestId = beforeDraftIds.find((id) => id.startsWith("workspace-tab-agent_"));
+    if (!firstAgentTabTestId) {
+      throw new Error("Expected an existing agent tab.");
+    }
+
+    await page.getByTestId("workspace-new-agent-tab").first().click();
+    await expect(page.getByRole("textbox", { name: "Message agent..." }).first()).toBeEditable();
+
+    const withDraftIds = await getWorkspaceTabTestIds(page);
+    const draftTabTestId = withDraftIds.find((id) => !beforeDraftIds.includes(id));
+    if (!draftTabTestId) {
+      throw new Error("Expected a draft tab to be created.");
+    }
+
+    const draftId = draftTabTestId.replace("workspace-tab-", "");
+    const draftCloseButton = page.getByTestId(`workspace-draft-close-${draftId}`).first();
+    const composer = page.getByRole("textbox", { name: "Message agent..." }).first();
+    await composer.fill(createPrompt);
+    await composer.press("Enter");
+    await page.getByTestId(firstAgentTabTestId).first().click();
+
+    await expect(page.getByText(seedPrompt, { exact: true }).first()).toBeVisible({
+      timeout: 30_000,
+    });
+    await expect(draftCloseButton).not.toBeVisible({ timeout: 30_000 });
+    await expect(page.getByText(createPrompt, { exact: true }).first()).not.toBeVisible();
+    await expect(page.getByText(seedPrompt, { exact: true }).first()).toBeVisible();
+  } finally {
+    await repo.cleanup();
+  }
+});
+
+test("workspace open intent creates exactly one draft tab in an empty workspace", async ({ page }) => {
+  const serverId = process.env.E2E_SERVER_ID;
+  if (!serverId) {
+    throw new Error("E2E_SERVER_ID is not set.");
+  }
+
+  const repo = await createTempGitRepo("paseo-e2e-draft-open-intent-");
+
+  try {
+    await gotoHome(page);
+    await ensureHostSelected(page);
+
+    await page.goto(
+      buildHostWorkspaceRouteWithOpenIntent(serverId, repo.path, {
+        kind: "draft",
+        draftId: "new",
+      })
+    );
+
+    await waitForWorkspaceTabsVisible(page);
+    await ensureWorkspaceAgentPaneVisible(page);
+    await expect(page.getByRole("textbox", { name: "Message agent..." }).first()).toBeEditable();
+
+    const tabIds = await getWorkspaceTabTestIds(page);
+    expect(tabIds).toHaveLength(1);
+    expect(tabIds[0]?.startsWith("workspace-tab-draft_")).toBe(true);
+
+    const tabLabel = page
+      .locator('[data-testid^="workspace-tab-"]:not([data-testid^="workspace-tab-context-"])')
+      .first();
+    await expect(tabLabel).toContainText("New Agent");
+  } finally {
+    await repo.cleanup();
+  }
+});

packages/app/src/components/tool-call-sheet.tsx

@@ -109,6 +109,7 @@ export function ToolCallSheetProvider({ children }: ToolCallSheetProviderProps)
         ref={bottomSheetRef}
         snapPoints={snapPoints}
         index={0}
+        stackBehavior="replace"
         enableDynamicSizing={false}
         onChange={handleSheetChange}
         backdropComponent={renderBackdrop}

packages/app/src/contexts/session-context.tsx

@@ -676,7 +676,7 @@ function SessionProviderInternal({
       return;
     }
 
-    return voiceRuntime.registerSession({
+  return voiceRuntime.registerSession({
       serverId,
       setVoiceMode: async (enabled, agentId) => {
         if (!client) {
@@ -684,11 +684,17 @@ function SessionProviderInternal({
         }
         await client.setVoiceMode(enabled, agentId);
       },
-      sendVoiceAudioChunk: async (audioData, mimeType, isLast) => {
+      sendVoiceAudioChunk: async (audioData, mimeType) => {
         if (!client) {
           throw new Error("Daemon unavailable");
         }
-        await client.sendVoiceAudioChunk(audioData, mimeType, isLast);
+        await client.sendVoiceAudioChunk(audioData, mimeType);
+      },
+      audioPlayed: async (chunkId) => {
+        if (!client) {
+          throw new Error("Daemon unavailable");
+        }
+        await client.audioPlayed(chunkId);
       },
       abortRequest: async () => {
         if (!client) {
@@ -1236,6 +1242,11 @@ function SessionProviderInternal({
       }
 
       const payload: AudioOutputPayload = message.payload;
+      if (payload.isVoiceMode && voiceRuntime) {
+        voiceRuntime.handleAudioOutput(serverId, payload);
+        return;
+      }
+
       const playbackGroupId = payload.groupId ?? payload.id;
       const chunkIndex = payload.chunkIndex ?? 0;
       const isFinalChunk = payload.isLastChunk ?? true;
@@ -1421,14 +1432,22 @@ function SessionProviderInternal({
       if (message.type !== "transcription_result") return;
 
       const transcriptText = message.payload.text.trim();
+      voiceRuntime?.onTranscriptionResult(serverId, transcriptText);
       if (!transcriptText) {
-        voiceRuntime?.onTranscriptionResult(serverId, transcriptText);
         return;
       }
 
       setCurrentAssistantMessage(serverId, "");
     });
 
+    const unsubVoiceInputState = client.on("voice_input_state", (message) => {
+      if (message.type !== "voice_input_state") return;
+      voiceRuntime?.onServerSpeechStateChanged(
+        serverId,
+        message.payload.isSpeaking
+      );
+    });
+
     const unsubAgentDeleted = client.on("agent_deleted", (message) => {
       if (message.type !== "agent_deleted") {
         return;
@@ -1537,6 +1556,7 @@ function SessionProviderInternal({
       unsubActivity();
       unsubChunk();
       unsubTranscription();
+      unsubVoiceInputState();
       unsubAgentDeleted();
       unsubAgentArchived();
     };

packages/app/src/screens/workspace/workspace-desktop-tabs-row.tsx

@@ -17,6 +17,7 @@ import {
   deriveWorkspaceTabPresentation,
   WorkspaceTabIcon,
 } from "@/screens/workspace/workspace-tab-presentation";
+import { buildWorkspaceTabMenuEntries } from "@/screens/workspace/workspace-tab-menu";
 import type { WorkspaceTabDescriptor } from "@/screens/workspace/workspace-tabs-types";
 import { encodeFilePathForPathSegment } from "@/utils/host-routes";
 import type { Agent } from "@/stores/session-store";
@@ -163,9 +164,19 @@ export function WorkspaceDesktopTabsRow({
                 : presentation.label;
 
             const contextMenuTestId = `workspace-tab-context-${tab.key}`;
-            const isFirstTab = index === 0;
+            const menuEntries = buildWorkspaceTabMenuEntries({
-            const isLastTab = index === tabs.length - 1;
+              surface: "desktop",
-            const isOnlyTab = tabs.length <= 1;
+              tab,
+              index,
+              tabCount: tabs.length,
+              menuTestIDBase: contextMenuTestId,
+              onCopyResumeCommand,
+              onCopyAgentId,
+              onCloseTab,
+              onCloseTabsBefore: onCloseTabsToLeft,
+              onCloseTabsAfter: onCloseTabsToRight,
+              onCloseOtherTabs,
+            });
 
             return (
               <ContextMenu key={tab.key}>
@@ -285,64 +296,21 @@ export function WorkspaceDesktopTabsRow({
                 </Tooltip>
 
                 <ContextMenuContent align="start" width={DROPDOWN_WIDTH} testID={contextMenuTestId}>
-                  {tab.kind === "agent" ? (
+                  {menuEntries.map((entry) =>
-                    <>
+                    entry.kind === "separator" ? (
+                      <ContextMenuSeparator key={entry.key} />
+                    ) : (
                       <ContextMenuItem
-                        testID={`${contextMenuTestId}-copy-resume-command`}
+                        key={entry.key}
-                        onSelect={() => {
+                        testID={entry.testID}
-                          void onCopyResumeCommand(tab.agentId);
+                        disabled={entry.disabled}
-                        }}
+                        destructive={entry.destructive}
+                        onSelect={entry.onSelect}
                       >
-                        Copy resume command
+                        {entry.label}
                       </ContextMenuItem>
-                      <ContextMenuItem
+                    )
-                        testID={`${contextMenuTestId}-copy-agent-id`}
+                  )}
-                        onSelect={() => {
-                          void onCopyAgentId(tab.agentId);
-                        }}
-                      >
-                        Copy agent id
-                      </ContextMenuItem>
-                    </>
-                  ) : null}
-
-                  <ContextMenuSeparator />
-
-                  <ContextMenuItem
-                    testID={`${contextMenuTestId}-close-left`}
-                    disabled={isFirstTab}
-                    onSelect={() => {
-                      void onCloseTabsToLeft(tab.tabId);
-                    }}
-                  >
-                    Close to the left
-                  </ContextMenuItem>
-                  <ContextMenuItem
-                    testID={`${contextMenuTestId}-close-right`}
-                    disabled={isLastTab}
-                    onSelect={() => {
-                      void onCloseTabsToRight(tab.tabId);
-                    }}
-                  >
-                    Close to the right
-                  </ContextMenuItem>
-                  <ContextMenuItem
-                    testID={`${contextMenuTestId}-close-others`}
-                    disabled={isOnlyTab}
-                    onSelect={() => {
-                      void onCloseOtherTabs(tab.tabId);
-                    }}
-                  >
-                    Close other tabs
-                  </ContextMenuItem>
-                  <ContextMenuItem
-                    testID={`${contextMenuTestId}-close`}
-                    onSelect={() => {
-                      void onCloseTab(tab.tabId);
-                    }}
-                  >
-                    Close
-                  </ContextMenuItem>
                 </ContextMenuContent>
               </ContextMenu>
             );

packages/app/src/screens/workspace/workspace-screen.tsx

@@ -31,6 +31,7 @@ import {
   DropdownMenu,
   DropdownMenuContent,
   DropdownMenuItem,
+  DropdownMenuSeparator,
   DropdownMenuTrigger,
 } from "@/components/ui/dropdown-menu";
 import {
@@ -84,6 +85,7 @@ import {
   WorkspaceTabIcon,
   WorkspaceTabOptionRow,
 } from "@/screens/workspace/workspace-tab-presentation";
+import { buildWorkspaceTabMenuEntries } from "@/screens/workspace/workspace-tab-menu";
 import type { WorkspaceTabDescriptor } from "@/screens/workspace/workspace-tabs-types";
 import {
   resolveWorkspaceHeader,
@@ -143,6 +145,7 @@ function buildOpenIntentKey(input: {
 }
 
 type MobileWorkspaceTabSwitcherProps = {
+  tabs: WorkspaceTabDescriptor[];
   activeTabKey: string;
   activeTabLabel: string;
   activeTabPresentation: WorkspaceTabPresentation | null;
@@ -151,9 +154,16 @@ type MobileWorkspaceTabSwitcherProps = {
   tabPresentationsByKey: Map<string, WorkspaceTabPresentation>;
   onSelectSwitcherTab: (key: string) => void;
   onSelectNewTabOption: (key: typeof NEW_TAB_AGENT_OPTION_ID) => void;
+  onCopyResumeCommand: (agentId: string) => Promise<void> | void;
+  onCopyAgentId: (agentId: string) => Promise<void> | void;
+  onCloseTab: (tabId: string) => Promise<void> | void;
+  onCloseTabsAbove: (tabId: string) => Promise<void> | void;
+  onCloseTabsBelow: (tabId: string) => Promise<void> | void;
+  onCloseOtherTabs: (tabId: string) => Promise<void> | void;
 };
 
 const MobileWorkspaceTabSwitcher = memo(function MobileWorkspaceTabSwitcher({
+  tabs,
   activeTabKey,
   activeTabLabel,
   activeTabPresentation,
@@ -162,15 +172,29 @@ const MobileWorkspaceTabSwitcher = memo(function MobileWorkspaceTabSwitcher({
   tabPresentationsByKey,
   onSelectSwitcherTab,
   onSelectNewTabOption,
+  onCopyResumeCommand,
+  onCopyAgentId,
+  onCloseTab,
+  onCloseTabsAbove,
+  onCloseTabsBelow,
+  onCloseOtherTabs,
 }: MobileWorkspaceTabSwitcherProps) {
   const { theme } = useUnistyles();
   const [isOpen, setIsOpen] = useState(false);
   const anchorRef = useRef<View>(null);
+  const tabIndexByKey = useMemo(() => {
+    const map = new Map<string, number>();
+    tabs.forEach((tab, index) => {
+      map.set(tab.key, index);
+    });
+    return map;
+  }, [tabs]);
 
   return (
     <View style={styles.mobileTabsRow} testID="workspace-tabs-row">
       <Pressable
         ref={anchorRef}
+        testID="workspace-tab-switcher-trigger"
         style={({ hovered, pressed }) => [
           styles.switcherTrigger,
           (hovered || pressed || isOpen) && styles.switcherTriggerActive,
@@ -242,12 +266,71 @@ const MobileWorkspaceTabSwitcher = memo(function MobileWorkspaceTabSwitcher({
           const presentation =
             tabPresentationsByKey.get(option.id) ??
             deriveWorkspaceTabPresentation({ tab });
+          const tabIndex = tabIndexByKey.get(tab.key) ?? -1;
+          const menuTestIDBase = `workspace-tab-menu-${tab.key}`;
+          const menuEntries =
+            tabIndex < 0
+              ? []
+              : buildWorkspaceTabMenuEntries({
+                  surface: "mobile",
+                  tab,
+                  index: tabIndex,
+                  tabCount: tabs.length,
+                  menuTestIDBase,
+                  onCopyResumeCommand,
+                  onCopyAgentId,
+                  onCloseTab,
+                  onCloseTabsBefore: onCloseTabsAbove,
+                  onCloseTabsAfter: onCloseTabsBelow,
+                  onCloseOtherTabs,
+                });
           return (
             <WorkspaceTabOptionRow
               presentation={presentation}
               selected={selected}
               active={active}
               onPress={onPress}
+              trailingAccessory={
+                <DropdownMenu>
+                  <DropdownMenuTrigger
+                    testID={`${menuTestIDBase}-trigger`}
+                    accessibilityRole="button"
+                    accessibilityLabel={`Open menu for ${presentation.label}`}
+                    hitSlop={8}
+                    style={({ open, pressed }) => [
+                      styles.mobileTabMenuTrigger,
+                      (open || pressed) && styles.mobileTabMenuTriggerActive,
+                    ]}
+                  >
+                    <Ellipsis
+                      size={theme.iconSize.sm}
+                      color={theme.colors.foregroundMuted}
+                    />
+                  </DropdownMenuTrigger>
+                  <DropdownMenuContent
+                    side="bottom"
+                    align="end"
+                    width={220}
+                    testID={menuTestIDBase}
+                  >
+                    {menuEntries.map((entry) =>
+                      entry.kind === "separator" ? (
+                        <DropdownMenuSeparator key={entry.key} />
+                      ) : (
+                        <DropdownMenuItem
+                          key={entry.key}
+                          testID={entry.testID}
+                          disabled={entry.disabled}
+                          destructive={entry.destructive}
+                          onSelect={entry.onSelect}
+                        >
+                          {entry.label}
+                        </DropdownMenuItem>
+                      )
+                    )}
+                  </DropdownMenuContent>
+                </DropdownMenu>
+              }
             />
           );
         }}
@@ -587,6 +670,29 @@ function WorkspaceScreenContent({
     [normalizedServerId, normalizedWorkspaceId, openIntent]
   );
 
+  const openWorkspaceDraftTab = useCallback(
+    (input?: { draftId?: string; focus?: boolean }) => {
+      if (!normalizedServerId || !normalizedWorkspaceId) {
+        return null;
+      }
+
+      const tabId = openDraftTab({
+        serverId: normalizedServerId,
+        workspaceId: normalizedWorkspaceId,
+        draftId: trimNonEmpty(input?.draftId) ?? generateDraftId(),
+      });
+      if (tabId && input?.focus !== false) {
+        focusTab({
+          serverId: normalizedServerId,
+          workspaceId: normalizedWorkspaceId,
+          tabId,
+        });
+      }
+      return tabId;
+    },
+    [focusTab, normalizedServerId, normalizedWorkspaceId, openDraftTab]
+  );
+
   useEffect(() => {
     if (!currentOpenIntentKey) {
       if (resolvedOpenIntentKey !== null) {
@@ -619,24 +725,11 @@ function WorkspaceScreenContent({
 
     if (openIntent.kind === "draft") {
       const draftId = openIntent.draftId.trim();
-      const tabId =
+      const tabId = openWorkspaceDraftTab({
-        draftId === "new"
+        ...(draftId === "new" ? {} : { draftId }),
-          ? openDraftTab({
+        focus: true,
-              serverId: normalizedServerId,
+      });
-              workspaceId: normalizedWorkspaceId,
-              draftId: generateDraftId(),
-            })
-          : openDraftTab({
-              serverId: normalizedServerId,
-              workspaceId: normalizedWorkspaceId,
-              draftId,
-            });
       if (tabId) {
-        focusTab({
-          serverId: normalizedServerId,
-          workspaceId: normalizedWorkspaceId,
-          tabId,
-        });
         setResolvedOpenIntentKey(intentKey);
       }
       return;
@@ -653,18 +746,12 @@ function WorkspaceScreenContent({
             : { kind: "file", path: openIntent.path },
     });
     if (tabId) {
-      focusTab({
-        serverId: normalizedServerId,
-        workspaceId: normalizedWorkspaceId,
-        tabId,
-      });
       setResolvedOpenIntentKey(intentKey);
     }
   }, [
     currentOpenIntentKey,
-    focusTab,
-    openDraftTab,
     openIntent,
+    openWorkspaceDraftTab,
     openOrFocusTab,
     persistenceKey,
     normalizedServerId,
@@ -821,6 +908,10 @@ function WorkspaceScreenContent({
     if (!persistenceKey) {
       return;
     }
+    if (openIntent) {
+      emptyWorkspaceSeedRef.current = null;
+      return;
+    }
     if (workspaceAgents.length > 0 || terminals.length > 0) {
       emptyWorkspaceSeedRef.current = null;
       return;
@@ -834,20 +925,12 @@ function WorkspaceScreenContent({
       return;
     }
     emptyWorkspaceSeedRef.current = workspaceKey;
-    const draftId = generateDraftId();
+    openWorkspaceDraftTab();
-    const tabId = openDraftTab({
-      serverId: normalizedServerId,
-      workspaceId: normalizedWorkspaceId,
-      draftId,
-    });
-    if (tabId) {
-      navigateToTabId(tabId);
-    }
   }, [
-    navigateToTabId,
     normalizedServerId,
     normalizedWorkspaceId,
-    openDraftTab,
+    openIntent,
+    openWorkspaceDraftTab,
     persistenceKey,
     terminals.length,
     tabs.length,
@@ -924,19 +1007,8 @@ function WorkspaceScreenContent({
   }, [activeTabPresentation]);
 
   const handleCreateDraftTab = useCallback(() => {
-    if (!normalizedServerId || !normalizedWorkspaceId) {
+    openWorkspaceDraftTab();
-      return;
+  }, [openWorkspaceDraftTab]);
-    }
-    const draftId = generateDraftId();
-    const tabId = openDraftTab({
-      serverId: normalizedServerId,
-      workspaceId: normalizedWorkspaceId,
-      draftId,
-    });
-    if (tabId) {
-      navigateToTabId(tabId);
-    }
-  }, [navigateToTabId, normalizedServerId, normalizedWorkspaceId, openDraftTab]);
 
   const handleCreateTerminal = useCallback(() => {
     if (createTerminalMutation.isPending) {
@@ -1375,29 +1447,23 @@ function WorkspaceScreenContent({
     }
 
     if (target.kind === "draft") {
+      const tabId = activeTabId ?? target.draftId;
       return (
         <WorkspaceDraftAgentTab
+          key={`${normalizedServerId}:${normalizedWorkspaceId}:${tabId}`}
           serverId={normalizedServerId}
           workspaceId={normalizedWorkspaceId}
-          tabId={activeTabId ?? target.draftId}
+          tabId={tabId}
           draftId={target.draftId}
           onOpenWorkspaceFile={handleOpenFileFromChat}
           onCreated={(agentSnapshot) => {
-            const tabId = activeTabId ?? target.draftId;
             const normalized = normalizeAgentSnapshot(agentSnapshot, normalizedServerId);
-            const nextTabId = retargetWorkspaceTab({
+            retargetWorkspaceTab({
               serverId: normalizedServerId,
               workspaceId: normalizedWorkspaceId,
               tabId,
               target: { kind: "agent", agentId: agentSnapshot.id },
             });
-            if (nextTabId) {
-              focusTab({
-                serverId: normalizedServerId,
-                workspaceId: normalizedWorkspaceId,
-                tabId: nextTabId,
-              });
-            }
             useSessionStore.getState().setAgents(normalizedServerId, (prev) => {
               const next = new Map(prev);
               next.set(agentSnapshot.id, normalized);
@@ -1635,6 +1701,7 @@ function WorkspaceScreenContent({
 
           {isMobile ? (
             <MobileWorkspaceTabSwitcher
+              tabs={tabs}
               activeTabKey={activeTabKey}
               activeTabLabel={activeTabLabel}
               activeTabPresentation={activeTabPresentation}
@@ -1643,6 +1710,12 @@ function WorkspaceScreenContent({
               tabPresentationsByKey={tabPresentationsByKey}
               onSelectSwitcherTab={handleSelectSwitcherTab}
               onSelectNewTabOption={handleSelectNewTabOption}
+              onCopyResumeCommand={handleCopyResumeCommand}
+              onCopyAgentId={handleCopyAgentId}
+              onCloseTab={handleCloseTabById}
+              onCloseTabsAbove={handleCloseTabsToLeft}
+              onCloseTabsBelow={handleCloseTabsToRight}
+              onCloseOtherTabs={handleCloseOtherTabs}
             />
           ) : (
             <WorkspaceDesktopTabsRow
@@ -1868,6 +1941,16 @@ const styles = StyleSheet.create((theme) => ({
     color: theme.colors.foreground,
     fontSize: theme.fontSize.sm,
   },
+  mobileTabMenuTrigger: {
+    width: 28,
+    height: 28,
+    borderRadius: theme.borderRadius.md,
+    alignItems: "center",
+    justifyContent: "center",
+  },
+  mobileTabMenuTriggerActive: {
+    backgroundColor: theme.colors.surface2,
+  },
   tabsContainer: {
     borderBottomWidth: 1,
     borderBottomColor: theme.colors.border,

packages/app/src/screens/workspace/workspace-tab-menu.test.ts

@@ -0,0 +1,107 @@
+import { describe, expect, it, vi } from "vitest";
+import { buildWorkspaceTabMenuEntries } from "@/screens/workspace/workspace-tab-menu";
+import type { WorkspaceTabDescriptor } from "@/screens/workspace/workspace-tabs-types";
+
+function createAgentTab(): WorkspaceTabDescriptor {
+  return {
+    key: "agent_123",
+    tabId: "agent_123",
+    kind: "agent",
+    agentId: "agent-123",
+    provider: "codex",
+    label: "Agent 123",
+    subtitle: "/repo",
+    titleState: "ready",
+  };
+}
+
+describe("buildWorkspaceTabMenuEntries", () => {
+  it("uses desktop tab ordering labels for desktop menus", () => {
+    const onCopyResumeCommand = vi.fn();
+    const onCopyAgentId = vi.fn();
+    const onCloseTab = vi.fn();
+    const onCloseTabsBefore = vi.fn();
+    const onCloseTabsAfter = vi.fn();
+    const onCloseOtherTabs = vi.fn();
+
+    const entries = buildWorkspaceTabMenuEntries({
+      surface: "desktop",
+      tab: createAgentTab(),
+      index: 1,
+      tabCount: 3,
+      menuTestIDBase: "workspace-tab-context-agent_123",
+      onCopyResumeCommand,
+      onCopyAgentId,
+      onCloseTab,
+      onCloseTabsBefore,
+      onCloseTabsAfter,
+      onCloseOtherTabs,
+    });
+
+    expect(
+      entries.filter((entry) => entry.kind === "item").map((entry) => entry.label)
+    ).toEqual([
+      "Copy resume command",
+      "Copy agent id",
+      "Close to the left",
+      "Close to the right",
+      "Close other tabs",
+      "Close",
+    ]);
+  });
+
+  it("uses stacked ordering labels for mobile menus", () => {
+    const entries = buildWorkspaceTabMenuEntries({
+      surface: "mobile",
+      tab: createAgentTab(),
+      index: 1,
+      tabCount: 3,
+      menuTestIDBase: "workspace-tab-menu-agent_123",
+      onCopyResumeCommand: vi.fn(),
+      onCopyAgentId: vi.fn(),
+      onCloseTab: vi.fn(),
+      onCloseTabsBefore: vi.fn(),
+      onCloseTabsAfter: vi.fn(),
+      onCloseOtherTabs: vi.fn(),
+    });
+
+    expect(
+      entries.filter((entry) => entry.kind === "item").map((entry) => entry.label)
+    ).toEqual([
+      "Copy resume command",
+      "Copy agent id",
+      "Close tabs above",
+      "Close tabs below",
+      "Close other tabs",
+      "Close",
+    ]);
+  });
+
+  it("omits agent copy actions for non-agent tabs", () => {
+    const entries = buildWorkspaceTabMenuEntries({
+      surface: "mobile",
+      tab: {
+        key: "draft_123",
+        tabId: "draft_123",
+        kind: "draft",
+        draftId: "draft_123",
+        label: "New agent",
+        subtitle: "/repo",
+      },
+      index: 0,
+      tabCount: 1,
+      menuTestIDBase: "workspace-tab-menu-draft_123",
+      onCopyResumeCommand: vi.fn(),
+      onCopyAgentId: vi.fn(),
+      onCloseTab: vi.fn(),
+      onCloseTabsBefore: vi.fn(),
+      onCloseTabsAfter: vi.fn(),
+      onCloseOtherTabs: vi.fn(),
+    });
+
+    expect(entries.some((entry) => entry.kind === "item" && entry.label === "Copy agent id")).toBe(
+      false
+    );
+    expect(entries.some((entry) => entry.kind === "separator")).toBe(false);
+  });
+});

packages/app/src/screens/workspace/workspace-tab-menu.ts

@@ -0,0 +1,137 @@
+import type { WorkspaceTabDescriptor } from "@/screens/workspace/workspace-tabs-types";
+
+export type WorkspaceTabMenuSurface = "desktop" | "mobile";
+
+export type WorkspaceTabMenuEntry =
+  | {
+      kind: "item";
+      key: string;
+      label: string;
+      disabled?: boolean;
+      destructive?: boolean;
+      testID: string;
+      onSelect: () => void;
+    }
+  | {
+      kind: "separator";
+      key: string;
+    };
+
+interface BuildWorkspaceTabMenuEntriesInput {
+  surface: WorkspaceTabMenuSurface;
+  tab: WorkspaceTabDescriptor;
+  index: number;
+  tabCount: number;
+  menuTestIDBase: string;
+  onCopyResumeCommand: (agentId: string) => Promise<void> | void;
+  onCopyAgentId: (agentId: string) => Promise<void> | void;
+  onCloseTab: (tabId: string) => Promise<void> | void;
+  onCloseTabsBefore: (tabId: string) => Promise<void> | void;
+  onCloseTabsAfter: (tabId: string) => Promise<void> | void;
+  onCloseOtherTabs: (tabId: string) => Promise<void> | void;
+}
+
+function buildCloseBeforeLabel(surface: WorkspaceTabMenuSurface): string {
+  return surface === "mobile" ? "Close tabs above" : "Close to the left";
+}
+
+function buildCloseAfterLabel(surface: WorkspaceTabMenuSurface): string {
+  return surface === "mobile" ? "Close tabs below" : "Close to the right";
+}
+
+function buildCloseBeforeTestIDSuffix(surface: WorkspaceTabMenuSurface): string {
+  return surface === "mobile" ? "close-above" : "close-left";
+}
+
+function buildCloseAfterTestIDSuffix(surface: WorkspaceTabMenuSurface): string {
+  return surface === "mobile" ? "close-below" : "close-right";
+}
+
+export function buildWorkspaceTabMenuEntries(
+  input: BuildWorkspaceTabMenuEntriesInput
+): WorkspaceTabMenuEntry[] {
+  const {
+    surface,
+    tab,
+    index,
+    tabCount,
+    menuTestIDBase,
+    onCopyResumeCommand,
+    onCopyAgentId,
+    onCloseTab,
+    onCloseTabsBefore,
+    onCloseTabsAfter,
+    onCloseOtherTabs,
+  } = input;
+  const isFirstTab = index === 0;
+  const isLastTab = index === tabCount - 1;
+  const isOnlyTab = tabCount <= 1;
+  const entries: WorkspaceTabMenuEntry[] = [];
+
+  if (tab.kind === "agent") {
+    entries.push({
+      kind: "item",
+      key: "copy-resume-command",
+      label: "Copy resume command",
+      testID: `${menuTestIDBase}-copy-resume-command`,
+      onSelect: () => {
+        void onCopyResumeCommand(tab.agentId);
+      },
+    });
+    entries.push({
+      kind: "item",
+      key: "copy-agent-id",
+      label: "Copy agent id",
+      testID: `${menuTestIDBase}-copy-agent-id`,
+      onSelect: () => {
+        void onCopyAgentId(tab.agentId);
+      },
+    });
+    entries.push({
+      kind: "separator",
+      key: "copy-separator",
+    });
+  }
+
+  entries.push({
+    kind: "item",
+    key: "close-before",
+    label: buildCloseBeforeLabel(surface),
+    disabled: isFirstTab,
+    testID: `${menuTestIDBase}-${buildCloseBeforeTestIDSuffix(surface)}`,
+    onSelect: () => {
+      void onCloseTabsBefore(tab.tabId);
+    },
+  });
+  entries.push({
+    kind: "item",
+    key: "close-after",
+    label: buildCloseAfterLabel(surface),
+    disabled: isLastTab,
+    testID: `${menuTestIDBase}-${buildCloseAfterTestIDSuffix(surface)}`,
+    onSelect: () => {
+      void onCloseTabsAfter(tab.tabId);
+    },
+  });
+  entries.push({
+    kind: "item",
+    key: "close-others",
+    label: "Close other tabs",
+    disabled: isOnlyTab,
+    testID: `${menuTestIDBase}-close-others`,
+    onSelect: () => {
+      void onCloseOtherTabs(tab.tabId);
+    },
+  });
+  entries.push({
+    kind: "item",
+    key: "close",
+    label: "Close",
+    testID: `${menuTestIDBase}-close`,
+    onSelect: () => {
+      void onCloseTab(tab.tabId);
+    },
+  });
+
+  return entries;
+}

packages/app/src/screens/workspace/workspace-tab-model.test.ts

@@ -104,6 +104,7 @@ describe("deriveWorkspaceTabModel", () => {
     });
 
     expect(model.tabs.map((tab) => tab.descriptor.kind)).toEqual(["draft", "agent", "file"]);
+    expect(model.tabs[0]?.descriptor.label).toBe("New Agent");
   });
 
   it("applies stored order and appends newly-derived tabs deterministically", () => {

packages/app/src/screens/workspace/workspace-tab-model.ts

@@ -169,8 +169,8 @@ export function deriveWorkspaceTabModel(input: {
           tabId: tab.tabId,
           kind: "draft",
           draftId: tab.target.draftId,
-          label: "Draft",
+          label: "New Agent",
-          subtitle: "Draft",
+          subtitle: "New Agent",
         },
       });
       continue;

packages/app/src/screens/workspace/workspace-tab-presentation.tsx

@@ -1,4 +1,4 @@
-import type { ReactElement } from "react";
+import type { ReactElement, ReactNode } from "react";
 import { Pressable, Text, View } from "react-native";
 import { Bot, Check, FileText, Pencil, Terminal } from "lucide-react-native";
 import { StyleSheet, useUnistyles } from "react-native-unistyles";
@@ -124,6 +124,7 @@ type WorkspaceTabOptionRowProps = {
   selected: boolean;
   active: boolean;
   onPress: () => void;
+  trailingAccessory?: ReactNode;
 };
 
 export function WorkspaceTabOptionRow({
@@ -131,30 +132,36 @@ export function WorkspaceTabOptionRow({
   selected,
   active,
   onPress,
+  trailingAccessory,
 }: WorkspaceTabOptionRowProps): ReactElement {
   const { theme } = useUnistyles();
   return (
-    <Pressable
+    <View style={[styles.optionRow, active && styles.optionRowActive]}>
-      onPress={onPress}
+      <Pressable
-      style={({ hovered = false, pressed }) => [
+        onPress={onPress}
-        styles.optionRow,
+        style={({ hovered = false, pressed }) => [
-        (hovered || pressed || active) && styles.optionRowActive,
+          styles.optionMainPressable,
-      ]}
+          (hovered || pressed || active) && styles.optionRowActive,
-    >
+        ]}
-      <View style={styles.optionLeadingSlot}>
+      >
-        <WorkspaceTabIcon presentation={presentation} active={selected || active} />
+        <View style={styles.optionLeadingSlot}>
-      </View>
+          <WorkspaceTabIcon presentation={presentation} active={selected || active} />
-      <View style={styles.optionContent}>
+        </View>
-        <Text numberOfLines={1} style={styles.optionLabel}>
+        <View style={styles.optionContent}>
-          {presentation.titleState === "loading" ? "Loading..." : presentation.label}
+          <Text numberOfLines={1} style={styles.optionLabel}>
-        </Text>
+            {presentation.titleState === "loading" ? "Loading..." : presentation.label}
-      </View>
+          </Text>
+        </View>
+      </Pressable>
       {selected ? (
         <View style={styles.optionTrailingSlot}>
           <Check size={16} color={theme.colors.foregroundMuted} />
         </View>
       ) : null}
-    </Pressable>
+      {trailingAccessory ? (
+        <View style={styles.optionTrailingAccessorySlot}>{trailingAccessory}</View>
+      ) : null}
+    </View>
   );
 }
 
@@ -178,12 +185,21 @@ const styles = StyleSheet.create((theme) => ({
     alignItems: "center",
     minHeight: 36,
     gap: theme.spacing[2],
-    paddingHorizontal: theme.spacing[3],
+    paddingHorizontal: theme.spacing[1],
-    paddingVertical: theme.spacing[2],
+    paddingVertical: theme.spacing[1],
     borderRadius: 0,
     marginHorizontal: theme.spacing[1],
     marginBottom: theme.spacing[1],
   },
+  optionMainPressable: {
+    flex: 1,
+    minWidth: 0,
+    flexDirection: "row",
+    alignItems: "center",
+    gap: theme.spacing[2],
+    paddingHorizontal: theme.spacing[2],
+    paddingVertical: theme.spacing[2],
+  },
   optionRowActive: {
     backgroundColor: theme.colors.surface1,
   },
@@ -204,6 +220,9 @@ const styles = StyleSheet.create((theme) => ({
     width: 16,
     alignItems: "center",
     justifyContent: "center",
-    marginLeft: "auto",
+  },
+  optionTrailingAccessorySlot: {
+    alignItems: "center",
+    justifyContent: "center",
   },
 }));

packages/app/src/stores/workspace-tabs-store.test.ts

@@ -137,6 +137,35 @@ describe("workspace-tabs-store retargetTab", () => {
     expect(order).toEqual([draftTabId]);
   });
 
+  it("retargeting a background draft keeps the currently focused tab focused", () => {
+    const draftTabId = "draft_background";
+    const key = buildWorkspaceTabPersistenceKey({ serverId: SERVER_ID, workspaceId: WORKSPACE_ID });
+    expect(key).toBeTruthy();
+    const workspaceKey = key as string;
+
+    useWorkspaceTabsStore.getState().openDraftTab({
+      serverId: SERVER_ID,
+      workspaceId: WORKSPACE_ID,
+      draftId: draftTabId,
+    });
+    const focusedFileTabId = useWorkspaceTabsStore.getState().openOrFocusTab({
+      serverId: SERVER_ID,
+      workspaceId: WORKSPACE_ID,
+      target: { kind: "file", path: "/repo/worktree/src/index.ts" },
+    });
+
+    useWorkspaceTabsStore.getState().retargetTab({
+      serverId: SERVER_ID,
+      workspaceId: WORKSPACE_ID,
+      tabId: draftTabId,
+      target: { kind: "agent", agentId: "created-agent" },
+    });
+
+    expect(useWorkspaceTabsStore.getState().focusedTabIdByWorkspace[workspaceKey]).toBe(
+      focusedFileTabId
+    );
+  });
+
   it("openOrFocusTab re-focuses an existing file tab after the workspace focus changed", () => {
     const key = buildWorkspaceTabPersistenceKey({ serverId: SERVER_ID, workspaceId: WORKSPACE_ID });
     expect(key).toBeTruthy();

packages/app/src/voice/audio-engine.native.ts

@@ -1,19 +1,9 @@
-import {
+import { Audio } from "expo-av";
-  addExpoTwoWayAudioEventListener,
+import * as FileSystem from "expo-file-system";
-  getMicrophonePermissionsAsync,
-  initialize,
-  playPCMData,
-  requestMicrophonePermissionsAsync,
-  resumePlayback,
-  stopPlayback,
-  tearDown,
-  toggleRecording,
-} from "@boudra/expo-two-way-audio";
 import {
   THINKING_TONE_REPEAT_GAP_MS,
 } from "@/utils/thinking-tone";
 import {
-  THINKING_TONE_NATIVE_PCM_BASE64,
   THINKING_TONE_NATIVE_PCM_DURATION_MS,
 } from "@/utils/thinking-tone.native-pcm";
 import type {
@@ -21,7 +11,9 @@ import type {
   AudioEngineCallbacks,
   AudioPlaybackSource,
 } from "@/voice/audio-engine-types";
-import { Buffer } from "buffer";
+
+// Use expo-av for playback instead of expo-two-way-audio (temporary test)
+const USE_EXPO_AV_PLAYBACK = true;
 
 interface QueuedAudio {
   audio: AudioPlaybackSource;
@@ -38,53 +30,6 @@ interface AudioEngineTraceOptions {
   traceLabel?: string;
 }
 
-function resamplePcm16(
-  pcm: Uint8Array,
-  fromRate: number,
-  toRate: number
-): Uint8Array {
-  if (fromRate === toRate) {
-    return pcm;
-  }
-
-  const inputSamples = Math.floor(pcm.length / 2);
-  const outputSamples = Math.floor((inputSamples * toRate) / fromRate);
-  const output = new Uint8Array(outputSamples * 2);
-  const ratio = fromRate / toRate;
-
-  const readInt16 = (sampleIndex: number): number => {
-    const offset = sampleIndex * 2;
-    if (offset + 1 >= pcm.length) {
-      return 0;
-    }
-    const lo = pcm[offset]!;
-    const hi = pcm[offset + 1]!;
-    let value = (hi << 8) | lo;
-    if (value & 0x8000) {
-      value -= 0x10000;
-    }
-    return value;
-  };
-
-  const writeInt16 = (sampleIndex: number, value: number): void => {
-    const clamped = Math.max(-32768, Math.min(32767, Math.round(value)));
-    const offset = sampleIndex * 2;
-    output[offset] = clamped & 0xff;
-    output[offset + 1] = (clamped >> 8) & 0xff;
-  };
-
-  for (let i = 0; i < outputSamples; i += 1) {
-    const sourceIndex = i * ratio;
-    const i0 = Math.floor(sourceIndex);
-    const frac = sourceIndex - i0;
-    const s0 = readInt16(i0);
-    const s1 = readInt16(Math.min(inputSamples - 1, i0 + 1));
-    writeInt16(i, s0 + (s1 - s0) * frac);
-  }
-
-  return output;
-}
-
 function parsePcmSampleRate(mimeType: string): number | null {
   const match = /rate=(\d+)/i.exec(mimeType);
   if (!match) {
@@ -117,6 +62,7 @@ export function createAudioEngine(
     };
     thinkingTone: CuePcm | null;
     destroyed: boolean;
+    mockVolumeInterval: ReturnType<typeof setInterval> | null;
   } = {
     initialized: false,
     captureActive: false,
@@ -132,41 +78,58 @@ export function createAudioEngine(
     },
     thinkingTone: null,
     destroyed: false,
+    mockVolumeInterval: null,
   };
 
-  const microphoneSubscription = addExpoTwoWayAudioEventListener(
+  // Only subscribe to native events if not mocking
-    "onMicrophoneData",
+  let microphoneSubscription: { remove(): void };
-    (event) => {
+  let volumeSubscription: { remove(): void };
-      if (!refs.captureActive || refs.muted) {
-        return;
-      }
-      callbacks.onCaptureData(event.data);
-    }
-  );
 
-  const volumeSubscription = addExpoTwoWayAudioEventListener(
+  if (MOCK_MODE) {
-    "onInputVolumeLevelData",
+    microphoneSubscription = { remove() {} };
-    (event) => {
+    volumeSubscription = { remove() {} };
-      if (!refs.captureActive) {
+  } else {
-        return;
+    const native = require("@boudra/expo-two-way-audio");
+    microphoneSubscription = native.addExpoTwoWayAudioEventListener(
+      "onMicrophoneData",
+      (event: any) => {
+        if (!refs.captureActive || refs.muted) {
+          return;
+        }
+        callbacks.onCaptureData(event.data);
       }
-      const level = refs.muted ? 0 : event.data;
+    );
-      callbacks.onVolumeLevel(level);
+    volumeSubscription = native.addExpoTwoWayAudioEventListener(
-    }
+      "onInputVolumeLevelData",
-  );
+      (event: any) => {
+        if (!refs.captureActive) {
+          return;
+        }
+        const level = refs.muted ? 0 : event.data;
+        callbacks.onVolumeLevel(level);
+      }
+    );
+  }
 
   async function ensureInitialized(): Promise<void> {
     if (refs.initialized) {
       return;
     }
-    await initialize();
+    if (!MOCK_MODE) {
+      const native = require("@boudra/expo-two-way-audio");
+      await native.initialize();
+    }
     refs.initialized = true;
   }
 
   async function ensureMicrophonePermission(): Promise<void> {
-    let permission = await getMicrophonePermissionsAsync().catch(() => null);
+    if (MOCK_MODE) {
+      return;
+    }
+    const native = require("@boudra/expo-two-way-audio");
+    let permission = await native.getMicrophonePermissionsAsync().catch(() => null);
     if (!permission?.granted) {
-      permission = await requestMicrophonePermissionsAsync().catch(() => null);
+      permission = await native.requestMicrophonePermissionsAsync().catch(() => null);
     }
     if (!permission?.granted) {
       throw new Error(
@@ -179,9 +142,8 @@ export function createAudioEngine(
     if (refs.thinkingTone) {
       return refs.thinkingTone;
     }
-    const pcm16k = Buffer.from(THINKING_TONE_NATIVE_PCM_BASE64, "base64");
     const durationMs = THINKING_TONE_NATIVE_PCM_DURATION_MS;
-    refs.thinkingTone = { pcm16k, durationMs };
+    refs.thinkingTone = { pcm16k: new Uint8Array(0), durationMs };
     return refs.thinkingTone;
   }
 
@@ -194,7 +156,6 @@ export function createAudioEngine(
 
   async function playAudio(audio: AudioPlaybackSource): Promise<number> {
     await ensureInitialized();
-    resumePlayback();
 
     return await new Promise<number>(async (resolve, reject) => {
       refs.activePlayback = { resolve, reject, settled: false };
@@ -203,10 +164,16 @@ export function createAudioEngine(
         const arrayBuffer = await audio.arrayBuffer();
         const pcm = new Uint8Array(arrayBuffer);
         const inputRate = parsePcmSampleRate(audio.type || "") ?? 24000;
-        const pcm16k = resamplePcm16(pcm, inputRate, 16000);
+        const durationSec = pcm.length / 2 / inputRate;
-        const durationSec = pcm16k.length / 2 / 16000;
+
+        console.log(`[AudioEngine.native] playPCMData: MOCK_MODE=${MOCK_MODE} inputRate=${inputRate} inputBytes=${pcm.length} durationSec=${durationSec.toFixed(3)}`);
+
+        if (!MOCK_MODE) {
+          const native = require("@boudra/expo-two-way-audio");
+          native.resumePlayback();
+          native.playPCMData(pcm);
+        }
 
-        playPCMData(pcm16k);
         clearPlaybackTimeout();
         refs.playbackTimeout = setTimeout(() => {
           clearPlaybackTimeout();
@@ -248,6 +215,13 @@ export function createAudioEngine(
     refs.processingQueue = false;
   }
 
+  function nativeStopPlayback(): void {
+    if (!MOCK_MODE) {
+      const native = require("@boudra/expo-two-way-audio");
+      native.stopPlayback();
+    }
+  }
+
   function stopLooping(): void {
     refs.looping.active = false;
     refs.looping.token += 1;
@@ -255,7 +229,7 @@ export function createAudioEngine(
       clearTimeout(refs.looping.timeout);
       refs.looping.timeout = null;
     }
-    stopPlayback();
+    nativeStopPlayback();
   }
 
   return {
@@ -273,14 +247,22 @@ export function createAudioEngine(
       this.stop();
       this.clearQueue();
       if (refs.captureActive) {
-        toggleRecording(false);
+        if (!MOCK_MODE) {
+          const native = require("@boudra/expo-two-way-audio");
+          native.toggleRecording(false);
+        }
+        if (refs.mockVolumeInterval) {
+          clearInterval(refs.mockVolumeInterval);
+          refs.mockVolumeInterval = null;
+        }
         refs.captureActive = false;
       }
       clearPlaybackTimeout();
       refs.muted = false;
       callbacks.onVolumeLevel(0);
-      if (refs.initialized) {
+      if (refs.initialized && !MOCK_MODE) {
-        tearDown();
+        const native = require("@boudra/expo-two-way-audio");
+        native.tearDown();
         refs.initialized = false;
       }
       microphoneSubscription.remove();
@@ -295,7 +277,22 @@ export function createAudioEngine(
       try {
         await ensureMicrophonePermission();
         await ensureInitialized();
-        toggleRecording(true);
+
+        if (MOCK_MODE) {
+          // Emit fake volume levels and empty PCM data periodically
+          refs.mockVolumeInterval = setInterval(() => {
+            if (!refs.captureActive) {
+              return;
+            }
+            const fakeVolume = refs.muted ? 0 : 0.15 + Math.random() * 0.5;
+            callbacks.onVolumeLevel(fakeVolume);
+            // Send empty PCM chunk to keep the pipeline alive
+            callbacks.onCaptureData(new Uint8Array(3200));
+          }, 100);
+        } else {
+          const native = require("@boudra/expo-two-way-audio");
+          native.toggleRecording(true);
+        }
         refs.captureActive = true;
       } catch (error) {
         const wrapped = error instanceof Error ? error : new Error(String(error));
@@ -306,7 +303,15 @@ export function createAudioEngine(
 
     async stopCapture() {
       if (refs.captureActive) {
-        toggleRecording(false);
+        if (MOCK_MODE) {
+          if (refs.mockVolumeInterval) {
+            clearInterval(refs.mockVolumeInterval);
+            refs.mockVolumeInterval = null;
+          }
+        } else {
+          const native = require("@boudra/expo-two-way-audio");
+          native.toggleRecording(false);
+        }
       }
       refs.captureActive = false;
       refs.muted = false;
@@ -335,7 +340,7 @@ export function createAudioEngine(
     },
 
     stop() {
-      stopPlayback();
+      nativeStopPlayback();
       clearPlaybackTimeout();
       const active = refs.activePlayback;
       refs.activePlayback = null;
@@ -368,20 +373,17 @@ export function createAudioEngine(
       void (async () => {
         try {
           await ensureInitialized();
-          const cue =
+          const cue = await ensureThinkingTone();
-            audio.byteLength > 0
-              ? {
-                  pcm16k: audio,
-                  durationMs: (audio.byteLength / 2 / 16000) * 1000,
-                }
-              : await ensureThinkingTone();
 
           const loop = () => {
             if (!refs.looping.active || refs.looping.token !== token) {
               return;
             }
-            resumePlayback();
+            if (!MOCK_MODE) {
-            playPCMData(cue.pcm16k);
+              const native = require("@boudra/expo-two-way-audio");
+              native.resumePlayback();
+              native.playPCMData(cue.pcm16k);
+            }
             refs.looping.timeout = setTimeout(
               loop,
               cue.durationMs + (gapMs || THINKING_TONE_REPEAT_GAP_MS)

packages/app/src/voice/speech-segmenter.test.ts

@@ -1,272 +0,0 @@
-import { describe, expect, it, vi } from "vitest";
-
-import { SpeechSegmenter } from "./speech-segmenter";
-
-const mkPcmBytes = (length: number): Uint8Array => {
-  const bytes = new Uint8Array(length);
-  for (let i = 0; i < length; i++) bytes[i] = i % 255;
-  return bytes;
-};
-
-describe("SpeechSegmenter", () => {
-  it("streams continuously and flushes at stop", () => {
-    const onAudioSegment = vi.fn();
-    const segmenter = new SpeechSegmenter(
-      {
-        enableContinuousStreaming: true,
-        volumeThreshold: 0.3,
-        silenceDurationMs: 2000,
-        speechConfirmationMs: 300,
-        detectionGracePeriodMs: 200,
-        minChunkDurationMs: 100, // ensure we only flush on stop
-        pcmSampleRate: 1000,
-      },
-      { onAudioSegment }
-    );
-
-    segmenter.pushPcmChunk(mkPcmBytes(50));
-    segmenter.pushPcmChunk(mkPcmBytes(50));
-    expect(onAudioSegment).not.toHaveBeenCalled();
-
-    segmenter.stop(Date.now());
-
-    expect(onAudioSegment).toHaveBeenCalledTimes(1);
-    const lastCall = onAudioSegment.mock.calls.at(-1)![0];
-    expect(lastCall.isLast).toBe(true);
-    expect(lastCall.audioData.length).toBeGreaterThan(0);
-  });
-
-  it("detects speech, calls onSpeechStart, and flushes on speech end", () => {
-    const onAudioSegment = vi.fn();
-    const onSpeechStart = vi.fn();
-    const onSpeechEnd = vi.fn();
-    const detectingChanges: boolean[] = [];
-    const speakingChanges: boolean[] = [];
-
-    const segmenter = new SpeechSegmenter(
-      {
-        enableContinuousStreaming: false,
-        volumeThreshold: 0.3,
-        silenceDurationMs: 2000,
-        speechConfirmationMs: 300,
-        detectionGracePeriodMs: 200,
-        minChunkDurationMs: 100,
-        pcmSampleRate: 1000,
-      },
-      {
-        onAudioSegment,
-        onSpeechStart,
-        onSpeechEnd,
-        onDetectingChange: (v) => detectingChanges.push(v),
-        onSpeakingChange: (v) => speakingChanges.push(v),
-      }
-    );
-
-    const t0 = 10_000;
-
-    // Start detection.
-    segmenter.pushVolumeLevel(0.5, t0);
-    segmenter.pushPcmChunk(mkPcmBytes(50));
-    expect(detectingChanges).toEqual([true]);
-    expect(speakingChanges).toEqual([true]);
-    expect(onSpeechStart).not.toHaveBeenCalled();
-
-    // Confirm speech.
-    segmenter.pushVolumeLevel(0.5, t0 + 300);
-    segmenter.pushPcmChunk(mkPcmBytes(50));
-    expect(onSpeechStart).toHaveBeenCalledTimes(1);
-    expect(detectingChanges).toEqual([true, false]);
-    expect(speakingChanges).toEqual([true]);
-
-    // End speech after silence.
-    segmenter.pushVolumeLevel(0.0, t0 + 301);
-    segmenter.pushPcmChunk(mkPcmBytes(50));
-    segmenter.pushVolumeLevel(0.0, t0 + 301 + 2000);
-    segmenter.pushVolumeLevel(0.0, t0 + 301 + 2000 + 2000);
-
-    expect(onSpeechEnd).toHaveBeenCalledTimes(1);
-    expect(detectingChanges).toEqual([true, false, true, false]);
-    expect(speakingChanges).toEqual([true, false]);
-    expect(onAudioSegment).toHaveBeenCalled();
-    const lastCall = onAudioSegment.mock.calls.at(-1)![0];
-    expect(lastCall.isLast).toBe(true);
-    expect(lastCall.audioData.length).toBeGreaterThan(0);
-  });
-
-  it("keeps detection alive across a brief pause and confirms short utterances", () => {
-    const onSpeechStart = vi.fn();
-    const detectingChanges: boolean[] = [];
-
-    const segmenter = new SpeechSegmenter(
-      {
-        enableContinuousStreaming: false,
-        volumeThreshold: 0.18,
-        silenceDurationMs: 1400,
-        speechConfirmationMs: 120,
-        detectionGracePeriodMs: 700,
-        minChunkDurationMs: 100,
-        pcmSampleRate: 1000,
-      },
-      {
-        onSpeechStart,
-        onDetectingChange: (v) => detectingChanges.push(v),
-      }
-    );
-
-    const t0 = 20_000;
-
-    segmenter.pushVolumeLevel(0.4, t0);
-    segmenter.pushPcmChunk(mkPcmBytes(20));
-    segmenter.pushVolumeLevel(0.0, t0 + 80);
-    segmenter.pushPcmChunk(mkPcmBytes(20));
-    segmenter.pushVolumeLevel(0.4, t0 + 140);
-    segmenter.pushPcmChunk(mkPcmBytes(20));
-
-    expect(detectingChanges).toContain(true);
-    expect(onSpeechStart).toHaveBeenCalledTimes(1);
-  });
-
-  it("keeps utterance open after fade-out and resumes speaking during silence grace", () => {
-    const onAudioSegment = vi.fn();
-    const onSpeechStart = vi.fn();
-    const onSpeechEnd = vi.fn();
-    const speakingChanges: boolean[] = [];
-
-    const segmenter = new SpeechSegmenter(
-      {
-        enableContinuousStreaming: false,
-        volumeThreshold: 0.2,
-        silenceDurationMs: 1500,
-        speechConfirmationMs: 120,
-        detectionGracePeriodMs: 400,
-        minChunkDurationMs: 1000, // keep everything buffered until final flush
-        pcmSampleRate: 1000,
-      },
-      {
-        onAudioSegment,
-        onSpeechStart,
-        onSpeechEnd,
-        onSpeakingChange: (v) => speakingChanges.push(v),
-      }
-    );
-
-    const t0 = 30_000;
-
-    // Start + confirm speaking.
-    segmenter.pushVolumeLevel(0.6, t0);
-    segmenter.pushPcmChunk(mkPcmBytes(200));
-    segmenter.pushVolumeLevel(0.6, t0 + 130);
-    segmenter.pushPcmChunk(mkPcmBytes(200));
-    expect(onSpeechStart).toHaveBeenCalledTimes(1);
-
-    // Drop below threshold: meter should fade, but turn remains open.
-    segmenter.pushVolumeLevel(0.0, t0 + 200);
-    segmenter.pushPcmChunk(mkPcmBytes(200));
-    segmenter.pushVolumeLevel(0.0, t0 + 500);
-    segmenter.pushPcmChunk(mkPcmBytes(200));
-
-    // Speak again before silence timeout expires: continue same utterance.
-    segmenter.pushVolumeLevel(0.6, t0 + 900);
-    segmenter.pushPcmChunk(mkPcmBytes(200));
-    expect(onSpeechStart).toHaveBeenCalledTimes(1);
-    expect(onSpeechEnd).not.toHaveBeenCalled();
-
-    // End for real.
-    segmenter.pushVolumeLevel(0.0, t0 + 1000);
-    segmenter.pushPcmChunk(mkPcmBytes(200));
-    segmenter.pushVolumeLevel(0.0, t0 + 2600);
-    segmenter.pushVolumeLevel(0.0, t0 + 4200);
-
-    expect(speakingChanges).toEqual([true, false, true, false]);
-    expect(onSpeechEnd).toHaveBeenCalledTimes(1);
-    const finalCalls = onAudioSegment.mock.calls.filter((call) => call[0].isLast);
-    expect(finalCalls).toHaveLength(1);
-  });
-
-  it("keeps confirmed speech active above release threshold even when below start threshold", () => {
-    const onSpeechStart = vi.fn();
-    const onSpeechEnd = vi.fn();
-    const speakingChanges: boolean[] = [];
-
-    const segmenter = new SpeechSegmenter(
-      {
-        enableContinuousStreaming: false,
-        volumeThreshold: 0.3,
-        volumeReleaseThreshold: 0.1,
-        silenceDurationMs: 1200,
-        speechConfirmationMs: 120,
-        detectionGracePeriodMs: 200,
-        minChunkDurationMs: 1000,
-        pcmSampleRate: 1000,
-      },
-      {
-        onSpeechStart,
-        onSpeechEnd,
-        onSpeakingChange: (v) => speakingChanges.push(v),
-      }
-    );
-
-    const t0 = 40_000;
-
-    // Confirm speech.
-    segmenter.pushVolumeLevel(0.4, t0);
-    segmenter.pushPcmChunk(mkPcmBytes(50));
-    segmenter.pushVolumeLevel(0.4, t0 + 130);
-    segmenter.pushPcmChunk(mkPcmBytes(50));
-    expect(onSpeechStart).toHaveBeenCalledTimes(1);
-
-    // Stay between start and release thresholds for longer than silence timeout:
-    // should still be considered speaking.
-    segmenter.pushVolumeLevel(0.15, t0 + 400);
-    segmenter.pushPcmChunk(mkPcmBytes(50));
-    segmenter.pushVolumeLevel(0.15, t0 + 1000);
-    segmenter.pushPcmChunk(mkPcmBytes(50));
-    segmenter.pushVolumeLevel(0.15, t0 + 2000);
-    segmenter.pushPcmChunk(mkPcmBytes(50));
-
-    expect(onSpeechEnd).not.toHaveBeenCalled();
-    expect(speakingChanges.at(-1)).toBe(true);
-
-    // Now actually drop below release threshold and allow silence timeout.
-    segmenter.pushVolumeLevel(0.0, t0 + 2001);
-    segmenter.pushPcmChunk(mkPcmBytes(50));
-    segmenter.pushVolumeLevel(0.0, t0 + 3405);
-    segmenter.pushVolumeLevel(0.0, t0 + 4706);
-
-    expect(onSpeechEnd).toHaveBeenCalledTimes(1);
-    expect(speakingChanges.at(-1)).toBe(false);
-  });
-
-  it("ignores brief confirmed dips below release threshold without flip-flopping speaking", () => {
-    const speakingChanges: boolean[] = [];
-
-    const segmenter = new SpeechSegmenter(
-      {
-        enableContinuousStreaming: false,
-        volumeThreshold: 0.2,
-        volumeReleaseThreshold: 0.08,
-        confirmedDropGracePeriodMs: 250,
-        silenceDurationMs: 1500,
-        speechConfirmationMs: 120,
-        detectionGracePeriodMs: 200,
-        minChunkDurationMs: 1000,
-        pcmSampleRate: 1000,
-      },
-      {
-        onSpeakingChange: (v) => speakingChanges.push(v),
-      }
-    );
-
-    const t0 = 50_000;
-    segmenter.pushVolumeLevel(0.4, t0);
-    segmenter.pushVolumeLevel(0.4, t0 + 130); // confirm speaking
-
-    // Two short dips below release threshold; both shorter than debounce.
-    segmenter.pushVolumeLevel(0.05, t0 + 300);
-    segmenter.pushVolumeLevel(0.09, t0 + 500);
-    segmenter.pushVolumeLevel(0.05, t0 + 620);
-    segmenter.pushVolumeLevel(0.1, t0 + 760);
-
-    expect(speakingChanges).toEqual([true]);
-  });
-});

packages/app/src/voice/speech-segmenter.ts

@@ -1,338 +0,0 @@
-import { Buffer } from "buffer";
-
-export type SpeechSegment = { audioData: string; isLast: boolean };
-
-export type SpeechSegmenterConfig = {
-  enableContinuousStreaming: boolean;
-  volumeThreshold: number; // 0..1
-  volumeReleaseThreshold?: number; // 0..1; lower threshold used to keep active speech from dropping out
-  confirmedDropGracePeriodMs?: number; // ms dip debounce before leaving confirmed speaking
-  silenceDurationMs: number;
-  speechConfirmationMs: number;
-  detectionGracePeriodMs: number;
-  minChunkDurationMs?: number;
-  pcmSampleRate?: number;
-  pcmChannels?: number;
-  pcmBitsPerSample?: number;
-};
-
-export type SpeechSegmenterCallbacks = {
-  onAudioSegment?: (segment: SpeechSegment) => void;
-  onSpeechStart?: () => void;
-  onSpeechEnd?: () => void;
-  onDetectingChange?: (isDetecting: boolean) => void;
-  onSpeakingChange?: (isSpeaking: boolean) => void;
-};
-
-function uint8ArrayToBase64(bytes: Uint8Array): string {
-  // NOTE: This is performance-sensitive during continuous streaming.
-  // Buffer-backed base64 is significantly faster than manual string building.
-  try {
-    return Buffer.from(bytes.buffer, bytes.byteOffset, bytes.byteLength).toString("base64");
-  } catch {
-    let binary = "";
-    for (let i = 0; i < bytes.length; i++) {
-      binary += String.fromCharCode(bytes[i]);
-    }
-    // eslint-disable-next-line no-restricted-globals
-    return btoa(binary);
-  }
-}
-
-function concatenateUint8Arrays(arrays: Uint8Array[]): Uint8Array {
-  const totalLength = arrays.reduce((sum, arr) => sum + arr.length, 0);
-  const result = new Uint8Array(totalLength);
-  let offset = 0;
-  for (const arr of arrays) {
-    result.set(arr, offset);
-    offset += arr.length;
-  }
-  return result;
-}
-
-export class SpeechSegmenter {
-  private config: Required<Omit<SpeechSegmenterConfig, "minChunkDurationMs" | "pcmSampleRate" | "pcmChannels" | "pcmBitsPerSample">> &
-    Required<Pick<SpeechSegmenterConfig, "minChunkDurationMs" | "pcmSampleRate" | "pcmChannels" | "pcmBitsPerSample">>;
-  private callbacks: SpeechSegmenterCallbacks;
-
-  private audioBuffer: Uint8Array[] = [];
-  private bufferedBytes = 0;
-
-  private speechDetectionStartMs: number | null = null;
-  private detectionSilenceStartMs: number | null = null;
-  private confirmedDropStartMs: number | null = null;
-  private silenceStartMs: number | null = null;
-  private isSpeaking = false;
-  private isDetecting = false;
-  private speechConfirmed = false;
-
-  private pcmBytesPerMs: number;
-  private minChunkBytes: number;
-
-  constructor(config: SpeechSegmenterConfig, callbacks: SpeechSegmenterCallbacks = {}) {
-    const volumeReleaseThreshold = this.resolveVolumeReleaseThreshold(
-      config.volumeThreshold,
-      config.volumeReleaseThreshold
-    );
-    this.config = {
-      ...config,
-      volumeReleaseThreshold,
-      confirmedDropGracePeriodMs: config.confirmedDropGracePeriodMs ?? 250,
-      minChunkDurationMs: config.minChunkDurationMs ?? 1000,
-      pcmSampleRate: config.pcmSampleRate ?? 16000,
-      pcmChannels: config.pcmChannels ?? 1,
-      pcmBitsPerSample: config.pcmBitsPerSample ?? 16,
-    };
-    this.callbacks = callbacks;
-    this.pcmBytesPerMs = 0;
-    this.minChunkBytes = 0;
-    this.recomputeDerived();
-  }
-
-  updateConfig(next: SpeechSegmenterConfig): void {
-    const volumeReleaseThreshold = this.resolveVolumeReleaseThreshold(
-      next.volumeThreshold,
-      next.volumeReleaseThreshold
-    );
-    this.config = {
-      ...next,
-      volumeReleaseThreshold,
-      confirmedDropGracePeriodMs: next.confirmedDropGracePeriodMs ?? this.config.confirmedDropGracePeriodMs,
-      minChunkDurationMs: next.minChunkDurationMs ?? this.config.minChunkDurationMs,
-      pcmSampleRate: next.pcmSampleRate ?? this.config.pcmSampleRate,
-      pcmChannels: next.pcmChannels ?? this.config.pcmChannels,
-      pcmBitsPerSample: next.pcmBitsPerSample ?? this.config.pcmBitsPerSample,
-    };
-    this.recomputeDerived();
-  }
-
-  setCallbacks(callbacks: SpeechSegmenterCallbacks): void {
-    this.callbacks = callbacks;
-  }
-
-  getSpeechDetectionStartMs(): number | null {
-    return this.speechDetectionStartMs;
-  }
-
-  getIsSpeaking(): boolean {
-    return this.isSpeaking;
-  }
-
-  getIsDetecting(): boolean {
-    return this.isDetecting;
-  }
-
-  getVolumeReleaseThreshold(): number {
-    return this.config.volumeReleaseThreshold;
-  }
-
-  reset(): void {
-    this.audioBuffer = [];
-    this.bufferedBytes = 0;
-    this.speechDetectionStartMs = null;
-    this.detectionSilenceStartMs = null;
-    this.confirmedDropStartMs = null;
-    this.silenceStartMs = null;
-    this.isSpeaking = false;
-    this.isDetecting = false;
-    this.speechConfirmed = false;
-  }
-
-  flush(isLast: boolean): void {
-    if (this.audioBuffer.length === 0) {
-      this.bufferedBytes = 0;
-      return;
-    }
-
-    const combinedBinary = concatenateUint8Arrays(this.audioBuffer);
-    const pcmBase64 = uint8ArrayToBase64(combinedBinary);
-    this.callbacks.onAudioSegment?.({ audioData: pcmBase64, isLast });
-
-    this.audioBuffer = [];
-    this.bufferedBytes = 0;
-  }
-
-  /**
-   * Called when the input stream emits a PCM16 chunk (Uint8Array of bytes).
-   * In non-continuous mode, chunk buffering is gated by the VAD state.
-   */
-  pushPcmChunk(chunk: Uint8Array): void {
-    if (chunk.length === 0) {
-      return;
-    }
-
-    if (this.config.enableContinuousStreaming) {
-      this.audioBuffer.push(chunk);
-      this.bufferedBytes += chunk.length;
-      if (this.bufferedBytes >= this.minChunkBytes) {
-        this.flush(false);
-      }
-      return;
-    }
-
-    // Buffer if we're detecting or speaking. When speech is confirmed, we also
-    // stream partial chunks to reduce latency.
-    if (this.speechDetectionStartMs !== null || this.isSpeaking) {
-      this.audioBuffer.push(chunk);
-      this.bufferedBytes += chunk.length;
-
-      if (this.speechConfirmed && this.bufferedBytes >= this.minChunkBytes) {
-        this.flush(false);
-      }
-    }
-  }
-
-  /**
-   * Called with a normalized volume level (0..1). Drives VAD transitions.
-   */
-  pushVolumeLevel(volume: number, nowMs: number): void {
-    if (this.config.enableContinuousStreaming) {
-      return;
-    }
-
-    const speechDetectedForStart = volume > this.config.volumeThreshold;
-    const speechDetectedForContinue = volume > this.config.volumeReleaseThreshold;
-
-    if (!this.speechConfirmed) {
-      if (speechDetectedForStart) {
-        // Initial detection phase.
-        this.confirmedDropStartMs = null;
-        if (this.speechDetectionStartMs === null) {
-          this.speechDetectionStartMs = nowMs;
-          this.detectionSilenceStartMs = null;
-          this.audioBuffer = [];
-          this.bufferedBytes = 0;
-          this.setDetecting(true);
-          this.setSpeaking(true);
-          return;
-        }
-
-        // Volume is back above threshold - reset grace period.
-        this.detectionSilenceStartMs = null;
-        this.setSpeaking(true);
-
-        const speechDuration = nowMs - this.speechDetectionStartMs;
-        if (speechDuration >= this.config.speechConfirmationMs) {
-          this.speechConfirmed = true;
-          this.silenceStartMs = null;
-          this.setDetecting(false);
-          this.callbacks.onSpeechStart?.();
-        }
-        return;
-      }
-
-      if (this.speechDetectionStartMs !== null) {
-        // Volume dropped during detection phase - apply grace period.
-        if (this.detectionSilenceStartMs === null) {
-          this.detectionSilenceStartMs = nowMs;
-          this.setSpeaking(false);
-          return;
-        }
-
-        const graceDuration = nowMs - this.detectionSilenceStartMs;
-        if (graceDuration >= this.config.detectionGracePeriodMs) {
-          // Cancel detection.
-          this.speechDetectionStartMs = null;
-          this.detectionSilenceStartMs = null;
-          this.confirmedDropStartMs = null;
-          this.audioBuffer = [];
-          this.bufferedBytes = 0;
-          this.setDetecting(false);
-          this.setSpeaking(false);
-        }
-      }
-      return;
-    }
-
-    if (speechDetectedForContinue) {
-      // Continuing confirmed speech (or resuming during silence grace).
-      this.confirmedDropStartMs = null;
-      this.silenceStartMs = null;
-      this.setDetecting(false);
-      this.setSpeaking(true);
-      return;
-    }
-
-    if (this.confirmedDropStartMs === null) {
-      this.confirmedDropStartMs = nowMs;
-      return;
-    }
-
-    const confirmedDropDuration = nowMs - this.confirmedDropStartMs;
-    if (confirmedDropDuration < this.config.confirmedDropGracePeriodMs) {
-      return;
-    }
-
-    // Potential speech END while utterance remains open.
-    if (this.silenceStartMs === null) {
-      this.silenceStartMs = nowMs;
-      this.setDetecting(true);
-      this.setSpeaking(false);
-      return;
-    }
-
-    const silenceDuration = nowMs - this.silenceStartMs;
-    if (silenceDuration >= this.config.silenceDurationMs) {
-      // Speech END confirmed.
-      this.speechConfirmed = false;
-      this.speechDetectionStartMs = null;
-      this.detectionSilenceStartMs = null;
-      this.confirmedDropStartMs = null;
-      this.silenceStartMs = null;
-      this.setDetecting(false);
-      this.setSpeaking(false);
-      this.callbacks.onSpeechEnd?.();
-      if (this.audioBuffer.length === 0) {
-        // Important: downstream expects an explicit segment boundary marker.
-        this.callbacks.onAudioSegment?.({ audioData: "", isLast: true });
-      } else {
-        this.flush(true);
-      }
-    }
-  }
-
-  stop(nowMs: number): void {
-    void nowMs;
-    if (this.config.enableContinuousStreaming) {
-      this.flush(true);
-    }
-    // Ensure observers see state clear transitions.
-    this.setDetecting(false);
-    this.setSpeaking(false);
-    this.reset();
-  }
-
-  private setDetecting(next: boolean): void {
-    if (this.isDetecting === next) {
-      return;
-    }
-    this.isDetecting = next;
-    this.callbacks.onDetectingChange?.(next);
-  }
-
-  private setSpeaking(next: boolean): void {
-    if (this.isSpeaking === next) {
-      return;
-    }
-    this.isSpeaking = next;
-    this.callbacks.onSpeakingChange?.(next);
-  }
-
-  private recomputeDerived(): void {
-    this.pcmBytesPerMs =
-      (this.config.pcmSampleRate *
-        this.config.pcmChannels *
-        (this.config.pcmBitsPerSample / 8)) /
-      1000;
-    this.minChunkBytes = Math.round(this.pcmBytesPerMs * this.config.minChunkDurationMs);
-  }
-
-  private resolveVolumeReleaseThreshold(
-    volumeThreshold: number,
-    configured?: number
-  ): number {
-    const defaultReleaseThreshold = volumeThreshold * 0.4;
-    const raw = configured ?? defaultReleaseThreshold;
-    return Math.max(0, Math.min(raw, volumeThreshold));
-  }
-}

packages/app/src/voice/voice-runtime.test.ts

@@ -30,11 +30,30 @@ function createSessionAdapter(serverId = "server-1"): VoiceSessionAdapter {
     serverId,
     setVoiceMode: vi.fn().mockResolvedValue(undefined),
     sendVoiceAudioChunk: vi.fn().mockResolvedValue(undefined),
+    audioPlayed: vi.fn().mockResolvedValue(undefined),
     abortRequest: vi.fn().mockResolvedValue(undefined),
     setAssistantAudioPlaying: vi.fn(),
   };
 }
 
+function createAudioPayload(args: {
+  id: string;
+  groupId: string;
+  chunkIndex: number;
+  isLastChunk: boolean;
+  isVoiceMode?: boolean;
+}) {
+  return {
+    id: args.id,
+    groupId: args.groupId,
+    chunkIndex: args.chunkIndex,
+    isLastChunk: args.isLastChunk,
+    isVoiceMode: args.isVoiceMode ?? true,
+    format: "pcm",
+    audio: Buffer.from(`chunk-${args.chunkIndex}`).toString("base64"),
+  } as const;
+}
+
 function createServerInfo(): DaemonServerInfo {
   return {
     serverId: "server-1",
@@ -91,37 +110,21 @@ describe("voice runtime", () => {
     });
   });
 
-  it("transitions from capturing to submitting to waiting on the final chunk", async () => {
+  it("streams continuous PCM chunks and waits after receiving a transcript", async () => {
     const adapter = createSessionAdapter();
     const { runtime } = createRuntime();
     runtime.registerSession(adapter);
 
     await runtime.startVoice("server-1", "agent-1");
-    runtime.handleCaptureVolume(0.4);
-    await vi.advanceTimersByTimeAsync(
-      REALTIME_VOICE_VAD_CONFIG.speechConfirmationMs + 10
-    );
-    runtime.handleCaptureVolume(0.4);
     runtime.handleCapturePcm(new Uint8Array(32000));
-    expect(runtime.getSnapshot().phase).toBe("capturing");
+    expect(runtime.getSnapshot().phase).toBe("listening");
-
-    runtime.handleCapturePcm(new Uint8Array(4000));
-    runtime.handleCaptureVolume(0);
-    await vi.advanceTimersByTimeAsync(
-      REALTIME_VOICE_VAD_CONFIG.confirmedDropGracePeriodMs + 10
-    );
-    runtime.handleCaptureVolume(0);
-    await vi.advanceTimersByTimeAsync(
-      REALTIME_VOICE_VAD_CONFIG.silenceDurationMs + 10
-    );
-    runtime.handleCaptureVolume(0);
-    await Promise.resolve();
 
     expect(adapter.sendVoiceAudioChunk).toHaveBeenLastCalledWith(
       expect.any(String),
-      "audio/pcm;rate=16000;bits=16",
+      "audio/pcm;rate=16000;bits=16"
-      true
     );
+
+    runtime.onTranscriptionResult("server-1", "hello");
     expect(runtime.getSnapshot().phase).toBe("waiting");
   });
 
@@ -138,6 +141,110 @@ describe("voice runtime", () => {
     expect(adapter.setAssistantAudioPlaying).toHaveBeenCalledWith(true);
   });
 
+  it("starts playback on the first chunk before isLastChunk arrives", async () => {
+    const adapter = createSessionAdapter();
+    let resolvePlay!: (duration: number) => void;
+    const engine = createAudioEngineMock();
+    vi.mocked(engine.play).mockImplementation(
+      () =>
+        new Promise<number>((resolve) => {
+          resolvePlay = resolve;
+        })
+    );
+    const { runtime } = createRuntime({ engine });
+    runtime.registerSession(adapter);
+
+    await runtime.startVoice("server-1", "agent-1");
+    runtime.onTurnEvent("server-1", "agent-1", "turn_started");
+
+    runtime.handleAudioOutput(
+      "server-1",
+      createAudioPayload({
+        id: "chunk-0",
+        groupId: "group-1",
+        chunkIndex: 0,
+        isLastChunk: false,
+      })
+    );
+
+    await vi.waitFor(() => {
+      expect(engine.play).toHaveBeenCalledTimes(1);
+      expect(runtime.getSnapshot().phase).toBe("playing");
+      expect(adapter.audioPlayed).not.toHaveBeenCalled();
+    });
+
+    runtime.handleAudioOutput(
+      "server-1",
+      createAudioPayload({
+        id: "chunk-1",
+        groupId: "group-1",
+        chunkIndex: 1,
+        isLastChunk: true,
+      })
+    );
+
+    expect(engine.play).toHaveBeenCalledTimes(1);
+    resolvePlay(0.1);
+
+    await vi.waitFor(() => {
+      expect(adapter.audioPlayed).toHaveBeenCalledWith("chunk-0");
+      expect(engine.play).toHaveBeenCalledTimes(2);
+    });
+  });
+
+  it("acknowledges chunks only after they are consumed and finishes after the final chunk", async () => {
+    const adapter = createSessionAdapter();
+    const playResolvers: Array<(duration: number) => void> = [];
+    const engine = createAudioEngineMock();
+    vi.mocked(engine.play).mockImplementation(
+      () =>
+        new Promise<number>((resolve) => {
+          playResolvers.push(resolve);
+        })
+    );
+    const { runtime } = createRuntime({ engine });
+    runtime.registerSession(adapter);
+
+    await runtime.startVoice("server-1", "agent-1");
+    runtime.onTurnEvent("server-1", "agent-1", "turn_started");
+
+    runtime.handleAudioOutput(
+      "server-1",
+      createAudioPayload({
+        id: "chunk-0",
+        groupId: "group-1",
+        chunkIndex: 0,
+        isLastChunk: false,
+      })
+    );
+    runtime.handleAudioOutput(
+      "server-1",
+      createAudioPayload({
+        id: "chunk-1",
+        groupId: "group-1",
+        chunkIndex: 1,
+        isLastChunk: true,
+      })
+    );
+
+    await vi.waitFor(() => {
+      expect(engine.play).toHaveBeenCalledTimes(1);
+    });
+    expect(adapter.audioPlayed).not.toHaveBeenCalled();
+
+    playResolvers.shift()!(0.1);
+    await vi.waitFor(() => {
+      expect(adapter.audioPlayed).toHaveBeenCalledWith("chunk-0");
+      expect(engine.play).toHaveBeenCalledTimes(2);
+    });
+
+    playResolvers.shift()!(0.1);
+    await vi.waitFor(() => {
+      expect(adapter.audioPlayed).toHaveBeenCalledWith("chunk-1");
+      expect(runtime.getSnapshot().phase).toBe("waiting");
+    });
+  });
+
   it("returns to waiting after assistant playback when the turn is still active", async () => {
     const adapter = createSessionAdapter();
     const { runtime, engine } = createRuntime();
@@ -152,6 +259,18 @@ describe("voice runtime", () => {
     expect(engine.playLooping).toHaveBeenCalled();
   });
 
+  it("starts the thinking tone when an agent turn begins before playback", async () => {
+    const adapter = createSessionAdapter();
+    const { runtime, engine } = createRuntime();
+    runtime.registerSession(adapter);
+
+    await runtime.startVoice("server-1", "agent-1");
+    runtime.onTurnEvent("server-1", "agent-1", "turn_started");
+
+    expect(runtime.getSnapshot().phase).toBe("waiting");
+    expect(engine.playLooping).toHaveBeenCalled();
+  });
+
   it("returns to listening after assistant playback once the turn is complete", async () => {
     const adapter = createSessionAdapter();
     const { runtime, engine } = createRuntime();
@@ -164,10 +283,11 @@ describe("voice runtime", () => {
     runtime.onAssistantAudioFinished("server-1");
 
     expect(runtime.getSnapshot().phase).toBe("listening");
-    expect(engine.playLooping).not.toHaveBeenCalled();
+    expect(engine.playLooping).toHaveBeenCalledTimes(1);
+    expect(engine.stopLooping).toHaveBeenCalled();
   });
 
-  it("interrupts the active turn on barge-in after the grace period", async () => {
+  it("keeps local volume alone non-authoritative for playback interruption", async () => {
     const adapter = createSessionAdapter();
     const { runtime, engine } = createRuntime();
     runtime.registerSession(adapter);
@@ -177,16 +297,46 @@ describe("voice runtime", () => {
     runtime.onAssistantAudioStarted("server-1");
 
     runtime.handleCaptureVolume(0.5);
-    await vi.advanceTimersByTimeAsync(
+    expect(runtime.getTelemetrySnapshot().isSpeaking).toBe(false);
-      REALTIME_VOICE_VAD_CONFIG.speechConfirmationMs + 10
+    expect(adapter.abortRequest).not.toHaveBeenCalled();
-    );
+    expect(engine.stop).not.toHaveBeenCalled();
-    await vi.advanceTimersByTimeAsync(
+  });
-      REALTIME_VOICE_VAD_CONFIG.interruptGracePeriodMs
+
-    );
+  it("keeps the meter white state driven by server speech detection", async () => {
+    const adapter = createSessionAdapter();
+    const { runtime } = createRuntime();
+    runtime.registerSession(adapter);
+
+    await runtime.startVoice("server-1", "agent-1");
+    runtime.handleCaptureVolume(0.7);
+
+    expect(runtime.getTelemetrySnapshot()).toMatchObject({
+      volume: expect.any(Number),
+      isDetecting: true,
+      isSpeaking: false,
+    });
+
+    runtime.onServerSpeechStateChanged("server-1", true);
+    expect(runtime.getTelemetrySnapshot().isSpeaking).toBe(true);
+
+    runtime.onServerSpeechStateChanged("server-1", false);
+    expect(runtime.getTelemetrySnapshot().isSpeaking).toBe(false);
+  });
+
+  it("stops assistant playback immediately when server speech starts", async () => {
+    const adapter = createSessionAdapter();
+    const { runtime, engine } = createRuntime();
+    runtime.registerSession(adapter);
+
+    await runtime.startVoice("server-1", "agent-1");
+    runtime.onAssistantAudioStarted("server-1");
+
+    runtime.onServerSpeechStateChanged("server-1", true);
 
-    expect(adapter.abortRequest).toHaveBeenCalled();
     expect(engine.stop).toHaveBeenCalled();
-    expect(runtime.getSnapshot().phase).toBe("capturing");
+    expect(engine.clearQueue).toHaveBeenCalled();
+    expect(adapter.setAssistantAudioPlaying).toHaveBeenCalledWith(false);
+    expect(runtime.getTelemetrySnapshot().isSpeaking).toBe(true);
   });
 
   it("authoritatively stops and suppresses later voice audio", async () => {

packages/app/src/voice/voice-runtime.ts

@@ -1,9 +1,12 @@
-import type { AgentStreamEventPayload } from "@server/shared/messages";
+import { Buffer } from "buffer";
+import type {
+  AgentStreamEventPayload,
+  SessionOutboundMessage,
+} from "@server/shared/messages";
 import { resolveVoiceUnavailableMessage } from "@/utils/server-info-capabilities";
 import type { DaemonServerInfo } from "@/stores/session-store";
 import type { AudioEngine } from "@/voice/audio-engine-types";
 import { REALTIME_VOICE_VAD_CONFIG } from "@/voice/realtime-voice-config";
-import { SpeechSegmenter } from "@/voice/speech-segmenter";
 
 const PCM_MIME_TYPE = "audio/pcm;rate=16000;bits=16";
 const KEEP_AWAKE_TAG = "paseo:voice";
@@ -22,7 +25,6 @@ export type VoiceRuntimePhase =
   | "disabled"
   | "starting"
   | "listening"
-  | "capturing"
   | "submitting"
   | "waiting"
   | "playing"
@@ -47,11 +49,8 @@ export interface VoiceRuntimeTelemetrySnapshot {
 export interface VoiceSessionAdapter {
   serverId: string;
   setVoiceMode(enabled: boolean, agentId?: string): Promise<void>;
-  sendVoiceAudioChunk(
+  sendVoiceAudioChunk(audioData: string, mimeType: string): Promise<void>;
-    audioData: string,
+  audioPlayed(chunkId: string): Promise<void>;
-    mimeType: string,
-    isLast: boolean
-  ): Promise<void>;
   abortRequest(): Promise<void>;
   setAssistantAudioPlaying(isPlaying: boolean): void;
 }
@@ -68,16 +67,51 @@ interface RuntimeSessionState {
   connected: boolean;
 }
 
+interface ContinuousVoiceUploader {
+  reset(): void;
+  pushPcmChunk(chunk: Uint8Array): void;
+}
+
 interface RuntimeState {
   snapshot: VoiceRuntimeSnapshot;
   telemetry: VoiceRuntimeTelemetrySnapshot;
   turnInProgress: boolean;
+  serverSpeechDetected: boolean;
   transportReady: boolean;
   generation: number;
-  speechInterruptTimer: ReturnType<typeof setTimeout> | null;
-  speechInterruptSent: boolean;
   segmentDurationTimer: ReturnType<typeof setInterval> | null;
   lastDisplayVolumePublishMs: number;
+  localSpeechStartedAt: number | null;
+}
+
+type AudioOutputPayload = Extract<
+  SessionOutboundMessage,
+  { type: "audio_output" }
+>["payload"];
+
+type StreamingPlaybackChunk = {
+  id: string;
+  chunkIndex: number;
+  source: { arrayBuffer(): Promise<ArrayBuffer>; size: number; type: string };
+};
+
+type StreamingPlaybackGroup = {
+  groupId: string;
+  isVoiceMode: boolean;
+  shouldPlay: boolean;
+  chunks: Map<number, StreamingPlaybackChunk>;
+  nextChunkToPlay: number;
+  finalChunkIndex: number | null;
+  started: boolean;
+  ackedChunkIds: Set<string>;
+};
+
+interface RuntimePlaybackState {
+  groups: Map<string, StreamingPlaybackGroup>;
+  orderedGroupIds: string[];
+  activeGroupId: string | null;
+  processing: boolean;
+  generation: number;
 }
 
 const INITIAL_SNAPSHOT: VoiceRuntimeSnapshot = {
@@ -131,6 +165,7 @@ export interface VoiceRuntime {
   updateSessionConnection(serverId: string, connected: boolean): void;
   handleCapturePcm(chunk: Uint8Array): void;
   handleCaptureVolume(level: number): void;
+  handleAudioOutput(serverId: string, payload: AudioOutputPayload): void;
   startVoice(serverId: string, agentId: string): Promise<void>;
   stopVoice(): Promise<void>;
   destroy(): Promise<void>;
@@ -140,6 +175,7 @@ export interface VoiceRuntime {
   onAssistantAudioStarted(serverId: string): void;
   onAssistantAudioFinished(serverId: string): void;
   onTranscriptionResult(serverId: string, text: string): void;
+  onServerSpeechStateChanged(serverId: string, isSpeaking: boolean): void;
   onTurnEvent(serverId: string, agentId: string, eventType: TurnEventType): void;
 }
 
@@ -151,12 +187,19 @@ export function createVoiceRuntime(deps: VoiceRuntimeDeps): VoiceRuntime {
     snapshot: INITIAL_SNAPSHOT,
     telemetry: INITIAL_TELEMETRY,
     turnInProgress: false,
+    serverSpeechDetected: false,
     transportReady: false,
     generation: 0,
-    speechInterruptTimer: null,
-    speechInterruptSent: false,
     segmentDurationTimer: null,
     lastDisplayVolumePublishMs: 0,
+    localSpeechStartedAt: null,
+  };
+  const playback: RuntimePlaybackState = {
+    groups: new Map(),
+    orderedGroupIds: [],
+    activeGroupId: null,
+    processing: false,
+    generation: 0,
   };
 
   function emit(): void {
@@ -213,10 +256,151 @@ export function createVoiceRuntime(deps: VoiceRuntimeDeps): VoiceRuntime {
     return sessions.get(state.snapshot.activeServerId) ?? null;
   }
 
-  function clearSpeechInterruptTimer(): void {
+  function decodeAudioChunk(base64: string): Uint8Array {
-    if (state.speechInterruptTimer) {
+    return Buffer.from(base64, "base64");
-      clearTimeout(state.speechInterruptTimer);
+  }
-      state.speechInterruptTimer = null;
+
+  function toPlaybackSource(
+    bytes: Uint8Array,
+    format: string
+  ): { arrayBuffer(): Promise<ArrayBuffer>; size: number; type: string } {
+    const mimeType =
+      format === "pcm"
+        ? "audio/pcm;rate=24000;bits=16"
+        : format === "mp3"
+          ? "audio/mpeg"
+          : `audio/${format}`;
+
+    return {
+      size: bytes.byteLength,
+      type: mimeType,
+      async arrayBuffer() {
+        return Uint8Array.from(bytes).buffer;
+      },
+    };
+  }
+
+  function resetPlaybackState(): void {
+    const hadGroups = playback.groups.size;
+    playback.generation += 1;
+    playback.groups.clear();
+    playback.orderedGroupIds = [];
+    playback.activeGroupId = null;
+    playback.processing = false;
+    if (hadGroups > 0) {
+      console.log(`[VoiceRuntime] resetPlaybackState: cleared ${hadGroups} groups, new gen=${playback.generation}`);
+    }
+  }
+
+  function activateNextPlaybackGroup(): void {
+    while (playback.orderedGroupIds.length > 0) {
+      const groupId = playback.orderedGroupIds[0]!;
+      if (playback.groups.has(groupId)) {
+        playback.activeGroupId = groupId;
+        return;
+      }
+      playback.orderedGroupIds.shift();
+    }
+    playback.activeGroupId = null;
+  }
+
+  async function acknowledgeChunk(chunkId: string): Promise<void> {
+    const activeSession = getActiveSession();
+    if (!activeSession) {
+      return;
+    }
+    await activeSession.adapter.audioPlayed(chunkId);
+  }
+
+  async function processPlaybackQueue(serverId: string): Promise<void> {
+    if (playback.processing) {
+      return;
+    }
+
+    playback.processing = true;
+    const generation = playback.generation;
+    console.log(`[VoiceRuntime] processPlaybackQueue start gen=${generation} activeGroup=${playback.activeGroupId}`);
+    try {
+      while (playback.activeGroupId) {
+        if (generation !== playback.generation) {
+          console.log(`[VoiceRuntime] processPlaybackQueue abort: generation changed ${generation} -> ${playback.generation}`);
+          return;
+        }
+
+        const group = playback.groups.get(playback.activeGroupId);
+        if (!group) {
+          activateNextPlaybackGroup();
+          continue;
+        }
+
+        const nextChunk = group.chunks.get(group.nextChunkToPlay);
+        if (!nextChunk) {
+          if (
+            group.finalChunkIndex !== null &&
+            group.nextChunkToPlay > group.finalChunkIndex
+          ) {
+            console.log(`[VoiceRuntime] group=${group.groupId} complete, played=${group.started} chunks=${group.nextChunkToPlay}`);
+            playback.groups.delete(group.groupId);
+            if (playback.orderedGroupIds[0] === group.groupId) {
+              playback.orderedGroupIds.shift();
+            } else {
+              playback.orderedGroupIds = playback.orderedGroupIds.filter(
+                (value) => value !== group.groupId
+              );
+            }
+            if (group.started && group.isVoiceMode) {
+              api.onAssistantAudioFinished(serverId);
+            }
+            activateNextPlaybackGroup();
+            continue;
+          }
+          console.log(`[VoiceRuntime] group=${group.groupId} waiting for chunk=${group.nextChunkToPlay} (finalChunkIndex=${group.finalChunkIndex})`);
+          return;
+        }
+
+        group.chunks.delete(group.nextChunkToPlay);
+
+        if (group.shouldPlay && !group.started && group.isVoiceMode) {
+          group.started = true;
+          console.log(`[VoiceRuntime] group=${group.groupId} first play starting at chunk=${group.nextChunkToPlay}`);
+          api.onAssistantAudioStarted(serverId);
+        }
+
+        const playStart = Date.now();
+        try {
+          if (group.shouldPlay) {
+            await deps.engine.play(nextChunk.source);
+            console.log(`[VoiceRuntime] played chunk=${group.nextChunkToPlay} id=${nextChunk.id} took=${Date.now() - playStart}ms`);
+          } else {
+            console.log(`[VoiceRuntime] SKIPPED chunk=${group.nextChunkToPlay} id=${nextChunk.id} shouldPlay=false`);
+          }
+        } catch (error) {
+          if (generation !== playback.generation) {
+            console.log(`[VoiceRuntime] play error + generation changed, aborting`);
+            return;
+          }
+          console.error(`[VoiceRuntime] play error chunk=${group.nextChunkToPlay} took=${Date.now() - playStart}ms:`, error);
+        }
+
+        if (generation !== playback.generation) {
+          console.log(`[VoiceRuntime] post-play generation changed ${generation} -> ${playback.generation}`);
+          return;
+        }
+
+        if (!group.ackedChunkIds.has(nextChunk.id)) {
+          group.ackedChunkIds.add(nextChunk.id);
+          void acknowledgeChunk(nextChunk.id).catch((error) => {
+            console.warn("[VoiceRuntime] Failed to confirm audio playback:", error);
+          });
+        }
+
+        group.nextChunkToPlay += 1;
+      }
+    } finally {
+      if (generation === playback.generation) {
+        playback.processing = false;
+      }
+      console.log(`[VoiceRuntime] processPlaybackQueue exit gen=${generation} currentGen=${playback.generation}`);
     }
   }
 
@@ -227,9 +411,7 @@ export function createVoiceRuntime(deps: VoiceRuntimeDeps): VoiceRuntime {
     }
   }
 
-  function reconcileSegmentDurationTimer(
+  function reconcileSegmentDurationTimer(): void {
-    segmenter: SpeechSegmenter
-  ): void {
     if (!state.telemetry.isDetecting && !state.telemetry.isSpeaking) {
       clearSegmentDurationTimer();
       patchTelemetry((prev) => ({ ...prev, segmentDuration: 0 }));
@@ -241,7 +423,7 @@ export function createVoiceRuntime(deps: VoiceRuntimeDeps): VoiceRuntime {
     }
 
     state.segmentDurationTimer = setInterval(() => {
-      const startedAt = segmenter.getSpeechDetectionStartMs();
+      const startedAt = state.localSpeechStartedAt;
       patchTelemetry((prev) => ({
         ...prev,
         segmentDuration: startedAt ? Date.now() - startedAt : 0,
@@ -262,10 +444,9 @@ export function createVoiceRuntime(deps: VoiceRuntimeDeps): VoiceRuntime {
     deps.engine.stopLooping();
   }
 
-  function resetSegmenter(segmenter: SpeechSegmenter): void {
+  function resetCaptureTelemetry(): void {
-    segmenter.reset();
-    clearSpeechInterruptTimer();
     clearSegmentDurationTimer();
+    state.localSpeechStartedAt = null;
     patchTelemetry({ ...INITIAL_TELEMETRY });
   }
 
@@ -277,134 +458,34 @@ export function createVoiceRuntime(deps: VoiceRuntimeDeps): VoiceRuntime {
     deps.engine.playLooping(new Uint8Array(0), THINKING_TONE_REPEAT_GAP_MS);
   }
 
-  async function interruptActiveTurn(): Promise<void> {
+  const uploader: ContinuousVoiceUploader = {
-    const activeSession = getActiveSession();
+    reset() {},
-    if (!activeSession) {
+    pushPcmChunk(chunk) {
-      return;
+      const activeSession = getActiveSession();
-    }
+      if (
+        !activeSession ||
+        !state.transportReady ||
+        !state.snapshot.isVoiceMode ||
+        chunk.byteLength === 0
+      ) {
+        return;
+      }
 
-    stopCue();
+      void activeSession.adapter
-    deps.engine.stop();
+        .sendVoiceAudioChunk(Buffer.from(chunk).toString("base64"), PCM_MIME_TYPE)
-    deps.engine.clearQueue();
+        .catch((error) => {
-    activeSession.adapter.setAssistantAudioPlaying(false);
+          console.error("[VoiceRuntime] Failed to send audio chunk:", error);
-    patchSnapshot((prev) => ({ ...prev, phase: "capturing" }));
+        });
-
-    try {
-      await activeSession.adapter.abortRequest();
-    } catch (error) {
-      console.error("[VoiceRuntime] Failed to abort active turn:", error);
-    }
-  }
-
-  const segmenter = new SpeechSegmenter(
-    {
-      enableContinuousStreaming: false,
-      volumeThreshold: REALTIME_VOICE_VAD_CONFIG.volumeThreshold,
-      confirmedDropGracePeriodMs:
-        REALTIME_VOICE_VAD_CONFIG.confirmedDropGracePeriodMs,
-      silenceDurationMs: REALTIME_VOICE_VAD_CONFIG.silenceDurationMs,
-      speechConfirmationMs: REALTIME_VOICE_VAD_CONFIG.speechConfirmationMs,
-      detectionGracePeriodMs: REALTIME_VOICE_VAD_CONFIG.detectionGracePeriodMs,
     },
-    {
+  };
-      onAudioSegment: ({ audioData, isLast }) => {
-        const activeSession = getActiveSession();
-        if (!activeSession || !state.transportReady || !state.snapshot.isVoiceMode) {
-          return;
-        }
-
-        const generation = state.generation;
-        patchSnapshot((prev) => ({
-          ...prev,
-          phase: isLast ? "submitting" : "capturing",
-        }));
-        if (isLast) {
-          state.turnInProgress = true;
-        }
-
-        void activeSession.adapter
-          .sendVoiceAudioChunk(audioData, PCM_MIME_TYPE, isLast)
-          .then(() => {
-            if (
-              !isLast ||
-              generation !== state.generation ||
-              !state.snapshot.isVoiceMode ||
-              state.snapshot.activeServerId !== activeSession.adapter.serverId ||
-              state.snapshot.phase !== "submitting"
-            ) {
-              return;
-            }
-            patchSnapshot((prev) => ({ ...prev, phase: "waiting" }));
-            reconcileCue();
-          })
-          .catch((error) => {
-            console.error("[VoiceRuntime] Failed to send audio segment:", error);
-          });
-      },
-      onSpeechStart: () => {
-        stopCue();
-      },
-      onSpeechEnd: () => {
-        clearSpeechInterruptTimer();
-        state.speechInterruptSent = false;
-        reconcileCue();
-      },
-      onDetectingChange: (isDetecting) => {
-        const previous = state.telemetry;
-        patchTelemetry((prev) => ({ ...prev, isDetecting }));
-        if (previous.isDetecting !== isDetecting) {
-        }
-        if (!previous.isDetecting && isDetecting) {
-          stopCue();
-        }
-        reconcileSegmentDurationTimer(segmenter);
-        reconcileCue();
-      },
-      onSpeakingChange: (isSpeaking) => {
-        const previous = state.telemetry;
-        patchTelemetry((prev) => ({ ...prev, isSpeaking }));
-        if (previous.isSpeaking !== isSpeaking) {
-        }
-
-        if (!previous.isSpeaking && isSpeaking) {
-          stopCue();
-          if (
-            state.snapshot.phase === "waiting" ||
-            state.snapshot.phase === "playing"
-          ) {
-            clearSpeechInterruptTimer();
-            state.speechInterruptTimer = setTimeout(() => {
-              state.speechInterruptTimer = null;
-              if (
-                state.speechInterruptSent ||
-                !state.snapshot.isVoiceMode ||
-                !state.telemetry.isSpeaking
-              ) {
-                return;
-              }
-              state.speechInterruptSent = true;
-              void interruptActiveTurn();
-            }, REALTIME_VOICE_VAD_CONFIG.interruptGracePeriodMs);
-          }
-        }
-
-        if (!isSpeaking) {
-          clearSpeechInterruptTimer();
-          state.speechInterruptSent = false;
-        }
-
-        reconcileSegmentDurationTimer(segmenter);
-        reconcileCue();
-      },
-    }
-  );
 
   function resetToDisabledState(): void {
     state.transportReady = false;
     state.turnInProgress = false;
-    state.speechInterruptSent = false;
+    state.serverSpeechDetected = false;
     state.lastDisplayVolumePublishMs = 0;
-    resetSegmenter(segmenter);
+    uploader.reset();
+    resetCaptureTelemetry();
     patchSnapshot({ ...INITIAL_SNAPSHOT });
   }
 
@@ -435,6 +516,8 @@ export function createVoiceRuntime(deps: VoiceRuntimeDeps): VoiceRuntime {
 
   async function performLocalStop(): Promise<void> {
     stopCue();
+    uploader.reset();
+    resetPlaybackState();
     deps.engine.stop();
     deps.engine.clearQueue();
     await deps.engine.stopCapture().catch(() => undefined);
@@ -466,7 +549,7 @@ export function createVoiceRuntime(deps: VoiceRuntimeDeps): VoiceRuntime {
     }
   }
 
-  return {
+  const api: VoiceRuntime = {
     subscribe(listener) {
       listeners.add(listener);
       return () => {
@@ -524,7 +607,7 @@ export function createVoiceRuntime(deps: VoiceRuntimeDeps): VoiceRuntime {
       if (!state.snapshot.isVoiceMode || state.snapshot.isMuted) {
         return;
       }
-      segmenter.pushPcmChunk(chunk);
+      uploader.pushPcmChunk(chunk);
     },
 
     handleCaptureVolume(level) {
@@ -532,9 +615,81 @@ export function createVoiceRuntime(deps: VoiceRuntimeDeps): VoiceRuntime {
       const displayLevel = state.snapshot.isMuted ? 0 : level;
       publishDisplayVolume(displayLevel, nowMs);
       if (!state.snapshot.isVoiceMode || state.snapshot.isMuted) {
+        state.localSpeechStartedAt = null;
+        patchTelemetry((prev) => ({
+          ...prev,
+          isDetecting: false,
+          isSpeaking: false,
+          segmentDuration: 0,
+        }));
         return;
       }
-      segmenter.pushVolumeLevel(level, nowMs);
+
+      const isActive = level > REALTIME_VOICE_VAD_CONFIG.volumeThreshold;
+      if (isActive && state.localSpeechStartedAt === null) {
+        state.localSpeechStartedAt = nowMs;
+      }
+      if (!isActive) {
+        state.localSpeechStartedAt = null;
+      }
+
+      patchTelemetry((prev) => ({
+        ...prev,
+        isDetecting: isActive,
+        isSpeaking: state.serverSpeechDetected,
+      }));
+      reconcileSegmentDurationTimer();
+      if (!isActive) {
+        patchTelemetry((prev) => ({ ...prev, segmentDuration: 0 }));
+      }
+      reconcileCue();
+    },
+
+    handleAudioOutput(serverId, payload) {
+      if (
+        serverId !== state.snapshot.activeServerId ||
+        !state.snapshot.isVoiceMode ||
+        !payload.isVoiceMode
+      ) {
+        console.log(`[VoiceRuntime] audio_output DROPPED: activeServer=${state.snapshot.activeServerId} serverId=${serverId} isVoiceMode=${state.snapshot.isVoiceMode} payloadVoice=${payload.isVoiceMode}`);
+        return;
+      }
+
+      const groupId = payload.groupId ?? payload.id;
+      const chunkIndex = payload.chunkIndex ?? 0;
+      console.log(`[VoiceRuntime] audio_output groupId=${groupId} chunk=${chunkIndex} isLast=${payload.isLastChunk} size=${payload.audio.length} format=${payload.format}`);
+
+      let group = playback.groups.get(groupId);
+      if (!group) {
+        const shouldPlay = api.shouldPlayVoiceAudio(serverId);
+        console.log(`[VoiceRuntime] new group=${groupId} shouldPlay=${shouldPlay} phase=${state.snapshot.phase} isDetecting=${state.telemetry.isDetecting} isSpeaking=${state.telemetry.isSpeaking}`);
+        group = {
+          groupId,
+          isVoiceMode: payload.isVoiceMode,
+          shouldPlay: api.shouldPlayVoiceAudio(serverId),
+          chunks: new Map(),
+          nextChunkToPlay: 0,
+          finalChunkIndex: null,
+          started: false,
+          ackedChunkIds: new Set(),
+        };
+        playback.groups.set(groupId, group);
+        playback.orderedGroupIds.push(groupId);
+        if (!playback.activeGroupId) {
+          playback.activeGroupId = groupId;
+        }
+      }
+
+      group.chunks.set(chunkIndex, {
+        id: payload.id,
+        chunkIndex,
+        source: toPlaybackSource(decodeAudioChunk(payload.audio), payload.format),
+      });
+      if (payload.isLastChunk) {
+        group.finalChunkIndex = chunkIndex;
+      }
+
+      void processPlaybackQueue(serverId);
     },
 
     async startVoice(serverId, agentId) {
@@ -594,7 +749,8 @@ export function createVoiceRuntime(deps: VoiceRuntimeDeps): VoiceRuntime {
 
         state.transportReady = true;
         state.turnInProgress = false;
-        resetSegmenter(segmenter);
+        uploader.reset();
+        resetCaptureTelemetry();
         patchSnapshot((prev) => ({
           ...prev,
           isVoiceMode: true,
@@ -619,8 +775,10 @@ export function createVoiceRuntime(deps: VoiceRuntimeDeps): VoiceRuntime {
       }));
 
       try {
-        state.transportReady = false;
         stopCue();
+        uploader.reset();
+        state.transportReady = false;
+        resetPlaybackState();
         deps.engine.stop();
         deps.engine.clearQueue();
         activeSession?.adapter.setAssistantAudioPlaying(false);
@@ -647,7 +805,8 @@ export function createVoiceRuntime(deps: VoiceRuntimeDeps): VoiceRuntime {
     toggleMute() {
       const nextMuted = deps.engine.toggleMute();
       if (nextMuted) {
-        resetSegmenter(segmenter);
+        uploader.reset();
+        resetCaptureTelemetry();
         patchSnapshot((prev) => ({
           ...prev,
           isMuted: true,
@@ -672,9 +831,7 @@ export function createVoiceRuntime(deps: VoiceRuntimeDeps): VoiceRuntime {
         state.snapshot.isVoiceMode &&
         state.snapshot.activeServerId === serverId &&
         state.snapshot.phase !== "stopping" &&
-        state.snapshot.phase !== "disabled" &&
+        state.snapshot.phase !== "disabled"
-        !state.telemetry.isDetecting &&
-        !state.telemetry.isSpeaking
       );
     },
 
@@ -701,8 +858,6 @@ export function createVoiceRuntime(deps: VoiceRuntimeDeps): VoiceRuntime {
       }
 
       if (state.turnInProgress) {
-        patchSnapshot((prev) => ({ ...prev, phase: "waiting" }));
-        reconcileCue();
         return;
       }
 
@@ -719,6 +874,9 @@ export function createVoiceRuntime(deps: VoiceRuntimeDeps): VoiceRuntime {
       }
 
       if (text.trim()) {
+        state.turnInProgress = true;
+        patchSnapshot((prev) => ({ ...prev, phase: "waiting" }));
+        reconcileCue();
         return;
       }
 
@@ -727,6 +885,30 @@ export function createVoiceRuntime(deps: VoiceRuntimeDeps): VoiceRuntime {
       stopCue();
     },
 
+    onServerSpeechStateChanged(serverId, isSpeaking) {
+      if (
+        serverId !== state.snapshot.activeServerId ||
+        !state.snapshot.isVoiceMode
+      ) {
+        return;
+      }
+
+      console.log(`[VoiceRuntime] onServerSpeechStateChanged isSpeaking=${isSpeaking} phase=${state.snapshot.phase}`);
+      state.serverSpeechDetected = isSpeaking;
+      if (isSpeaking) {
+        resetPlaybackState();
+        deps.engine.stop();
+        deps.engine.clearQueue();
+        getActiveSession()?.adapter.setAssistantAudioPlaying(false);
+      }
+      patchTelemetry((prev) => ({
+        ...prev,
+        isSpeaking,
+      }));
+      reconcileSegmentDurationTimer();
+      reconcileCue();
+    },
+
     onTurnEvent(serverId, agentId, eventType) {
       if (
         !state.snapshot.isVoiceMode ||
@@ -738,6 +920,10 @@ export function createVoiceRuntime(deps: VoiceRuntimeDeps): VoiceRuntime {
 
       if (eventType === "turn_started") {
         state.turnInProgress = true;
+        if (state.snapshot.phase !== "playing") {
+          patchSnapshot((prev) => ({ ...prev, phase: "waiting" }));
+          reconcileCue();
+        }
         return;
       }
 
@@ -748,4 +934,6 @@ export function createVoiceRuntime(deps: VoiceRuntimeDeps): VoiceRuntime {
       stopCue();
     },
   };
+
+  return api;
 }

packages/server/package.json

@@ -11,6 +11,7 @@
     "dist/server",
     "dist/src",
     "dist/scripts",
+    "src/server/speech/providers/local/sherpa/assets",
     "README.md",
     ".env.example",
     "agent-prompt.md"
@@ -35,7 +36,7 @@
     "dev": "NODE_ENV=development tsx scripts/dev-runner.ts",
     "dev:tsx": "NODE_ENV=development tsx watch --ignore '**/*.timestamp-*' src/server/index.ts",
     "build": "node -e \"require('node:fs').rmSync('dist',{ recursive: true, force: true })\" && npm run build:lib && npm run build:scripts",
-    "build:lib": "tsc -p tsconfig.server.json --incremental false",
+    "build:lib": "tsc -p tsconfig.server.json --incremental false && node -e \"const fs=require('node:fs'); fs.mkdirSync('dist/server/server/speech/providers/local/sherpa/assets',{recursive:true}); fs.copyFileSync('src/server/speech/providers/local/sherpa/assets/silero_vad.onnx','dist/server/server/speech/providers/local/sherpa/assets/silero_vad.onnx');\"",
     "build:scripts": "tsc -p tsconfig.scripts.json --incremental false && node -e \"const fs=require('node:fs'); fs.mkdirSync('dist/scripts',{recursive:true}); fs.copyFileSync('scripts/mcp-stdio-socket-bridge-cli.mjs','dist/scripts/mcp-stdio-socket-bridge-cli.mjs');\"",
     "prepack": "npm run build",
     "start": "NODE_ENV=production node dist/server/server/index.js",

packages/server/src/client/daemon-client.ts

@@ -1648,7 +1648,11 @@ export class DaemonClient {
     return response
   }
 
-  async sendVoiceAudioChunk(audio: string, format: string, isLast: boolean): Promise<void> {
+  async sendVoiceAudioChunk(
+    audio: string,
+    format: string,
+    isLast = false
+  ): Promise<void> {
     this.sendSessionMessage({ type: 'voice_audio_chunk', audio, format, isLast })
   }
 

packages/server/src/server/agent/tts-manager.test.ts

@@ -1,4 +1,4 @@
-import { describe, expect, it } from "vitest";
+import { describe, expect, it, vi } from "vitest";
 import pino from "pino";
 import { Readable } from "node:stream";
 
@@ -74,7 +74,143 @@ describe("TTSManager", () => {
     );
 
     expect(calls.length).toBeGreaterThan(1);
-    expect(calls.every((text) => text.length <= 400)).toBe(true);
+    expect(calls.every((text) => text.length <= 260)).toBe(true);
+    expect(calls[0]!.length).toBeLessThanOrEqual(120);
+    expect(calls.slice(1).some((text) => text.length > calls[0]!.length)).toBe(true);
+  });
+
+  it("prefetches the next segment before current playback completes", async () => {
+    const started: string[] = [];
+    const gateResolvers = new Map<string, () => void>();
+    const tts: TextToSpeechProvider = {
+      async synthesizeSpeech(text: string): Promise<{ stream: Readable; format: string }> {
+        started.push(text);
+        await new Promise<void>((resolve) => {
+          gateResolvers.set(text, resolve);
+        });
+        return {
+          stream: Readable.from([Buffer.from(text)]),
+          format: "pcm;rate=24000",
+        };
+      },
+    };
+
+    const manager = new TTSManager("s1", pino({ level: "silent" }), tts);
+    const abort = new AbortController();
+    const segments = [
+      "One sentence that is long enough to stand alone in the first voice chunk.",
+      "Two sentence that is also long enough to require a separate synthesized group.",
+      "Three sentence that should only be synthesized after playback of the first group starts because it carries extra detail about the lookahead pipeline and should exceed the later packing target on its own.",
+    ];
+    const text = segments.join(" ");
+    const emittedGroups: string[] = [];
+    let firstChunkId: string | null = null;
+    let releaseSecondAck: (() => void) | null = null;
+
+    const task = manager.generateAndWaitForPlayback(
+      text,
+      (msg) => {
+        if (msg.type !== "audio_output") {
+          return;
+        }
+
+        if (!emittedGroups.includes(msg.payload.groupId!)) {
+          emittedGroups.push(msg.payload.groupId!);
+        }
+
+        if (emittedGroups.length === 1) {
+          firstChunkId = msg.payload.id;
+          return;
+        }
+
+        releaseSecondAck?.();
+        manager.confirmAudioPlayed(msg.payload.id);
+      },
+      abort.signal,
+      true
+    );
+
+    expect(started).toEqual([segments[0], segments[1]]);
+
+    gateResolvers.get(segments[0])?.();
+    await vi.waitFor(() => {
+      expect(firstChunkId).not.toBeNull();
+    });
+
+    expect(started).toEqual(segments);
+    expect(emittedGroups).toHaveLength(1);
+
+    const secondGroupPlayed = new Promise<void>((resolve) => {
+      releaseSecondAck = resolve;
+    });
+
+    manager.confirmAudioPlayed(firstChunkId!);
+    gateResolvers.get(segments[1])?.();
+
+    await vi.waitFor(() => {
+      expect(started).toEqual(segments);
+      expect(emittedGroups).toHaveLength(2);
+    });
+
+    gateResolvers.get(segments[2])?.();
+    await secondGroupPlayed;
+    await task;
+  });
+
+  it("destroys prefetched streams after abort", async () => {
+    const destroyed: string[] = [];
+    const tts: TextToSpeechProvider = {
+      async synthesizeSpeech(text: string): Promise<{ stream: Readable; format: string }> {
+        const stream = new Readable({
+          read() {
+            this.push(Buffer.from(text));
+            this.push(null);
+          },
+        });
+        const destroySpy = vi.spyOn(stream, "destroy").mockImplementation(function (
+          this: Readable,
+          error?: Error
+        ) {
+          destroyed.push(text);
+          return Readable.prototype.destroy.call(this, error);
+        });
+        void destroySpy;
+        return {
+          stream,
+          format: "pcm;rate=24000",
+        };
+      },
+    };
+
+    const manager = new TTSManager("s1", pino({ level: "silent" }), tts);
+    const abort = new AbortController();
+    let firstChunkId: string | null = null;
+
+    const task = manager.generateAndWaitForPlayback(
+      [
+        "First sentence that is long enough to stand alone in the first synthesized group.",
+        "Second sentence that should be prefetched and then discarded after abort is requested.",
+        "Third sentence that should also be cleaned up if it was prefetched before the abort.",
+      ].join(" "),
+      (msg) => {
+        if (msg.type === "audio_output" && firstChunkId === null) {
+          firstChunkId = msg.payload.id;
+        }
+      },
+      abort.signal,
+      true
+    );
+
+    await vi.waitFor(() => {
+      expect(firstChunkId).not.toBeNull();
+    });
+
+    abort.abort();
+
+    await task;
+    await vi.waitFor(() => {
+      expect(destroyed.length).toBeGreaterThanOrEqual(2);
+    });
   });
 
   it("does not emit unhandled rejections when stream iteration fails", async () => {
@@ -122,4 +258,31 @@ describe("TTSManager", () => {
       process.off("unhandledRejection", onUnhandled);
     }
   });
+
+  it("ignores late confirmations for recently closed audio IDs", async () => {
+    const logger = pino({ level: "silent" });
+    const warnSpy = vi.spyOn(logger, "warn");
+    const manager = new TTSManager("s1", logger, new FakeTts());
+    const abort = new AbortController();
+    const emitted: SessionOutboundMessage[] = [];
+
+    await manager.generateAndWaitForPlayback(
+      "hello",
+      (msg) => {
+        emitted.push(msg);
+        if (msg.type === "audio_output") {
+          manager.confirmAudioPlayed(msg.payload.id);
+        }
+      },
+      abort.signal,
+      true
+    );
+
+    const firstAudio = emitted.find((msg) => msg.type === "audio_output");
+    expect(firstAudio?.type).toBe("audio_output");
+
+    manager.confirmAudioPlayed((firstAudio as Extract<SessionOutboundMessage, { type: "audio_output" }>).payload.id);
+
+    expect(warnSpy).not.toHaveBeenCalled();
+  });
 });

packages/server/src/server/agent/tts-manager.ts

@@ -1,4 +1,5 @@
 import type pino from "pino";
+import type { Readable } from "node:stream";
 import { v4 as uuidv4 } from "uuid";
 import type { TextToSpeechProvider } from "../speech/speech-provider.js";
 import { toResolver, type Resolvable } from "../speech/provider-resolver.js";
@@ -11,80 +12,114 @@ interface PendingPlayback {
   streamEnded: boolean;
 }
 
-const MAX_TTS_SEGMENT_CHARS = 400;
+type TtsSegment = {
+  index: number;
+  text: string;
+};
 
-function splitTextForTts(text: string, maxChars: number): string[] {
+type PreparedTtsSegment = TtsSegment & {
+  format: string;
+  stream: Readable;
+};
+
+type PreparedSegmentResult =
+  | { kind: "prepared"; prepared: PreparedTtsSegment }
+  | { kind: "aborted" }
+  | { kind: "error"; error: unknown };
+
+const MAX_TTS_SEGMENT_CHARS = 260;
+const TTS_PREFETCH_SEGMENTS = 2;
+const CLOSED_AUDIO_ID_TTL_MS = 10_000;
+
+function splitOversizedFragment(fragment: string, maxChars: number): string[] {
+  const trimmed = fragment.trim();
+  if (!trimmed) {
+    return [];
+  }
+
+  if (trimmed.length <= maxChars) {
+    return [trimmed];
+  }
+
+  const clauseChunks = trimmed.split(/(?<=[,;:])\s+/);
+  if (clauseChunks.length > 1) {
+    const parts: string[] = [];
+    let current = "";
+
+    const pushCurrent = () => {
+      const value = current.trim();
+      if (value) {
+        parts.push(value);
+      }
+      current = "";
+    };
+
+    for (const clause of clauseChunks) {
+      const clauseText = clause.trim();
+      if (!clauseText) {
+        continue;
+      }
+
+      if (clauseText.length > maxChars) {
+        pushCurrent();
+        parts.push(...splitOversizedFragment(clauseText, maxChars));
+        continue;
+      }
+
+      if (!current) {
+        current = clauseText;
+        continue;
+      }
+
+      const candidate = `${current} ${clauseText}`;
+      if (candidate.length <= maxChars) {
+        current = candidate;
+        continue;
+      }
+
+      pushCurrent();
+      current = clauseText;
+    }
+
+    pushCurrent();
+    if (parts.length > 1 || parts[0] !== trimmed) {
+      return parts;
+    }
+  }
+
+  const parts: string[] = [];
+  let remaining = trimmed;
+  while (remaining.length > maxChars) {
+    let idx = remaining.lastIndexOf(" ", maxChars);
+    if (idx < Math.floor(maxChars * 0.5)) {
+      idx = maxChars;
+    }
+    parts.push(remaining.slice(0, idx).trim());
+    remaining = remaining.slice(idx).trim();
+  }
+  if (remaining.length > 0) {
+    parts.push(remaining);
+  }
+  return parts;
+}
+
+function splitTextForTts(text: string): TtsSegment[] {
   const normalized = text.trim().replace(/\s+/g, " ");
   if (!normalized) {
     throw new Error("Cannot synthesize empty text");
   }
 
-  if (normalized.length <= maxChars) {
+  const sentences = normalized.split(/(?<=[.!?])\s+/);
-    return [normalized];
+  const parts: TtsSegment[] = [];
-  }
+  let segmentIndex = 0;
 
-  const parts: string[] = [];
+  for (const sentence of sentences) {
-  const sentenceChunks = normalized.split(/(?<=[.!?])\s+/);
+    const fragments = splitOversizedFragment(sentence, MAX_TTS_SEGMENT_CHARS);
-
-  let current = "";
-  const pushCurrent = () => {
-    const trimmed = current.trim();
-    if (trimmed.length > 0) {
-      parts.push(trimmed);
-    }
-    current = "";
-  };
-
-  const appendFragment = (fragment: string) => {
-    const trimmed = fragment.trim();
-    if (!trimmed) {
-      return;
-    }
-
-    if (!current) {
-      current = trimmed;
-      return;
-    }
-
-    const candidate = `${current} ${trimmed}`;
-    if (candidate.length <= maxChars) {
-      current = candidate;
-      return;
-    }
-
-    pushCurrent();
-    current = trimmed;
-  };
-
-  const splitLargeFragment = (fragment: string): string[] => {
-    const trimmed = fragment.trim();
-    if (trimmed.length <= maxChars) {
-      return [trimmed];
-    }
-
-    const out: string[] = [];
-    let remaining = trimmed;
-    while (remaining.length > maxChars) {
-      let idx = remaining.lastIndexOf(" ", maxChars);
-      if (idx < Math.floor(maxChars * 0.5)) {
-        idx = maxChars;
-      }
-      out.push(remaining.slice(0, idx).trim());
-      remaining = remaining.slice(idx).trim();
-    }
-    if (remaining.length > 0) {
-      out.push(remaining);
-    }
-    return out;
-  };
-
-  for (const sentence of sentenceChunks) {
-    const fragments = splitLargeFragment(sentence);
     for (const fragment of fragments) {
-      appendFragment(fragment);
+      parts.push({ index: segmentIndex, text: fragment });
+      segmentIndex += 1;
     }
   }
-  pushCurrent();
 
   return parts;
 }
@@ -95,6 +130,7 @@ function splitTextForTts(text: string, maxChars: number): string[] {
  */
 export class TTSManager {
   private pendingPlaybacks: Map<string, PendingPlayback> = new Map();
+  private readonly recentlyClosedAudioIds: Map<string, number> = new Map();
   private readonly logger: pino.Logger;
   private readonly resolveTts: () => TextToSpeechProvider | null;
 
@@ -117,6 +153,7 @@ export class TTSManager {
     abortSignal: AbortSignal,
     isVoiceMode: boolean
   ): Promise<void> {
+    const ttsStartMs = Date.now();
     this.logger.info(
       {
         isVoiceMode,
@@ -126,45 +163,176 @@ export class TTSManager {
       "TTS input text"
     );
 
-    const segments = splitTextForTts(text, MAX_TTS_SEGMENT_CHARS);
+    const segments = splitTextForTts(text);
-    for (const segment of segments) {
+    this.logger.info(
-      if (abortSignal.aborted) {
+      { segmentCount: segments.length, segments: segments.map((s) => ({ index: s.index, chars: s.text.length, text: s.text.slice(0, 80) })) },
-        this.logger.debug("Aborted before generating segmented audio");
+      `TTS split into ${segments.length} segment(s)`
-        return;
+    );
-      }
 
-      await this.generateSegmentAndWaitForPlayback(
+    const inflight = new Map<number, Promise<PreparedSegmentResult>>();
-        segment,
+    let nextSegmentToSchedule = 0;
-        emitMessage,
+
-        abortSignal,
+    const scheduleNextSegments = () => {
-        isVoiceMode
+      while (
+        nextSegmentToSchedule < segments.length &&
+        inflight.size < TTS_PREFETCH_SEGMENTS
+      ) {
+        const segment = segments[nextSegmentToSchedule]!;
+        inflight.set(segment.index, this.scheduleSegmentSynthesis(segment, abortSignal));
+        nextSegmentToSchedule += 1;
+      }
+    };
+
+    scheduleNextSegments();
+
+    try {
+      for (const segment of segments) {
+        if (abortSignal.aborted) {
+          this.logger.debug("Aborted before emitting segmented audio");
+          return;
+        }
+
+        const synthWaitStart = Date.now();
+        const result = await inflight.get(segment.index)!;
+        const synthWaitMs = Date.now() - synthWaitStart;
+        inflight.delete(segment.index);
+        scheduleNextSegments();
+
+        if (result.kind === "aborted") {
+          return;
+        }
+
+        if (result.kind === "error") {
+          throw result.error;
+        }
+
+        this.logger.info(
+          { segmentIndex: segment.index, synthWaitMs, totalElapsedMs: Date.now() - ttsStartMs, chars: segment.text.length },
+          `TTS segment ${segment.index} synthesis ready (waited ${synthWaitMs}ms, total ${Date.now() - ttsStartMs}ms)`
+        );
+
+        const emitStart = Date.now();
+        await this.emitPreparedSegment({
+          prepared: result.prepared,
+          emitMessage,
+          abortSignal,
+          isVoiceMode,
+        });
+        this.logger.info(
+          { segmentIndex: segment.index, emitAndPlayMs: Date.now() - emitStart, totalElapsedMs: Date.now() - ttsStartMs },
+          `TTS segment ${segment.index} playback confirmed (emit+play ${Date.now() - emitStart}ms, total ${Date.now() - ttsStartMs}ms)`
+        );
+
+        scheduleNextSegments();
+      }
+    } finally {
+      this.cleanupPrefetchedSegments(inflight);
+      this.logger.info(
+        { totalMs: Date.now() - ttsStartMs },
+        `TTS generateAndWaitForPlayback done (${Date.now() - ttsStartMs}ms)`
       );
     }
   }
 
-  private async generateSegmentAndWaitForPlayback(
+  private async synthesizeSegment(
-    text: string,
+    segment: TtsSegment,
-    emitMessage: (msg: SessionOutboundMessage) => void,
+    abortSignal: AbortSignal
-    abortSignal: AbortSignal,
+  ): Promise<PreparedTtsSegment> {
-    isVoiceMode: boolean
+    const resolveStart = Date.now();
-  ): Promise<void> {
     const tts = this.resolveTts();
     if (!tts) {
       throw new Error("TTS not configured");
     }
+    const resolveMs = Date.now() - resolveStart;
 
     if (abortSignal.aborted) {
-      this.logger.debug("Aborted before generating audio");
+      throw new Error("TTS synthesis aborted");
+    }
+
+    const synthStart = Date.now();
+    const { stream, format } = await tts.synthesizeSpeech(segment.text);
+    this.logger.info(
+      { segmentIndex: segment.index, resolveMs, synthMs: Date.now() - synthStart, chars: segment.text.length },
+      `TTS segment ${segment.index} synthesized (resolve=${resolveMs}ms, synth=${Date.now() - synthStart}ms, ${segment.text.length} chars)`
+    );
+
+    if (abortSignal.aborted) {
+      this.destroySpeechStream(stream);
+      throw new Error("TTS synthesis aborted");
+    }
+
+    return {
+      ...segment,
+      stream,
+      format,
+    };
+  }
+
+  private scheduleSegmentSynthesis(
+    segment: TtsSegment,
+    abortSignal: AbortSignal
+  ): Promise<PreparedSegmentResult> {
+    return this.synthesizeSegment(segment, abortSignal).then(
+      (prepared) => {
+        if (abortSignal.aborted) {
+          this.destroySpeechStream(prepared.stream);
+          return { kind: "aborted" };
+        }
+        return { kind: "prepared", prepared };
+      },
+      (error) => {
+        if (abortSignal.aborted) {
+          return { kind: "aborted" };
+        }
+        return { kind: "error", error };
+      }
+    );
+  }
+
+  private cleanupPrefetchedSegments(
+    inflight: Map<number, Promise<PreparedSegmentResult>>
+  ): void {
+    if (inflight.size === 0) {
       return;
     }
 
-    // Generate TTS audio stream
+    for (const pending of inflight.values()) {
-    const { stream, format } = await tts.synthesizeSpeech(text);
+      void pending.then((result) => {
-
+        if (result.kind === "prepared") {
-    if (abortSignal.aborted) {
+          this.destroySpeechStream(result.prepared.stream);
-      this.logger.debug("Aborted after generating audio");
+        }
-      return;
+      });
     }
+  }
+
+  private destroySpeechStream(stream: Readable): void {
+    if (typeof stream.destroy === "function" && !stream.destroyed) {
+      stream.destroy();
+    }
+  }
+
+  private pruneRecentlyClosedAudioIds(now: number): void {
+    for (const [audioId, expiresAt] of this.recentlyClosedAudioIds.entries()) {
+      if (expiresAt <= now) {
+        this.recentlyClosedAudioIds.delete(audioId);
+      }
+    }
+  }
+
+  private rememberClosedAudioId(audioId: string): void {
+    const now = Date.now();
+    this.pruneRecentlyClosedAudioIds(now);
+    this.recentlyClosedAudioIds.set(audioId, now + CLOSED_AUDIO_ID_TTL_MS);
+  }
+
+  private async emitPreparedSegment(params: {
+    prepared: PreparedTtsSegment;
+    emitMessage: (msg: SessionOutboundMessage) => void;
+    abortSignal: AbortSignal;
+    isVoiceMode: boolean;
+  }): Promise<void> {
+    const { prepared, emitMessage, abortSignal, isVoiceMode } = params;
+    const { stream, format, text } = prepared;
 
     const audioId = uuidv4();
     let playbackResolve!: () => void;
@@ -185,72 +353,53 @@ export class TTSManager {
     this.pendingPlaybacks.set(audioId, pendingPlayback);
 
     let onAbort: (() => void) | undefined;
-    const destroyStream = () => {
-      if (typeof stream.destroy === "function" && !stream.destroyed) {
-        stream.destroy();
-      }
-    };
 
     onAbort = () => {
       this.logger.debug("Aborted while waiting for playback");
       pendingPlayback.streamEnded = true;
       pendingPlayback.pendingChunks = 0;
       this.pendingPlaybacks.delete(audioId);
+      this.rememberClosedAudioId(audioId);
       playbackResolve();
-      destroyStream();
+      this.destroySpeechStream(stream);
     };
 
     abortSignal.addEventListener("abort", onAbort, { once: true });
 
     try {
-      const iterator = stream[Symbol.asyncIterator]();
+      const buffers: Buffer[] = [];
-      let chunkIndex = 0;
+      for await (const chunk of stream) {
-      let current = await iterator.next();
+        if (abortSignal.aborted) {
-
+          this.logger.debug("Aborted during stream collection");
-      if (!current.done) {
+          break;
-        let next = await iterator.next();
-
-        while (true) {
-          if (abortSignal.aborted) {
-            this.logger.debug("Aborted during stream emission");
-            break;
-          }
-
-          const chunkBuffer = Buffer.isBuffer(current.value)
-            ? current.value
-            : Buffer.from(current.value);
-
-          const chunkId = `${audioId}:${chunkIndex}`;
-          pendingPlayback.pendingChunks += 1;
-
-          emitMessage({
-            type: "audio_output",
-            payload: {
-              id: chunkId,
-              groupId: audioId,
-              chunkIndex,
-              isLastChunk: next.done,
-              audio: chunkBuffer.toString("base64"),
-              format,
-              isVoiceMode,
-            },
-          });
-
-          chunkIndex += 1;
-
-          if (next.done) {
-            break;
-          }
-
-          current = next;
-          next = await iterator.next();
         }
+        buffers.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk));
+      }
+
+      if (!abortSignal.aborted && buffers.length > 0) {
+        const fullBuffer = Buffer.concat(buffers);
+        const chunkId = `${audioId}:0`;
+        pendingPlayback.pendingChunks = 1;
+
+        emitMessage({
+          type: "audio_output",
+          payload: {
+            id: chunkId,
+            groupId: audioId,
+            chunkIndex: 0,
+            isLastChunk: true,
+            audio: fullBuffer.toString("base64"),
+            format,
+            isVoiceMode,
+          },
+        });
       }
 
       pendingPlayback.streamEnded = true;
 
       if (pendingPlayback.pendingChunks === 0) {
         this.pendingPlaybacks.delete(audioId);
+        this.rememberClosedAudioId(audioId);
         playbackResolve();
       }
 
@@ -267,14 +416,14 @@ export class TTSManager {
       if (onAbort) {
         abortSignal.removeEventListener("abort", onAbort);
       }
-      destroyStream();
+      this.destroySpeechStream(stream);
     }
 
     if (abortSignal.aborted) {
       return;
     }
 
-    this.logger.debug({ audioId }, "Audio playback confirmed");
+    this.logger.debug({ audioId, textLength: text.length }, "Audio playback confirmed");
   }
 
   /**
@@ -288,6 +437,13 @@ export class TTSManager {
     const pending = this.pendingPlaybacks.get(audioId);
 
     if (!pending) {
+      const now = Date.now();
+      this.pruneRecentlyClosedAudioIds(now);
+      const expiresAt = this.recentlyClosedAudioIds.get(audioId);
+      if (expiresAt && expiresAt > now) {
+        this.logger.debug({ chunkId }, "Ignoring late confirmation for recently closed audio ID");
+        return;
+      }
       this.logger.warn({ chunkId }, "Received confirmation for unknown audio ID");
       return;
     }
@@ -297,6 +453,7 @@ export class TTSManager {
     if (pending.pendingChunks === 0 && pending.streamEnded) {
       pending.resolve();
       this.pendingPlaybacks.delete(audioId);
+      this.rememberClosedAudioId(audioId);
     }
   }
 
@@ -316,6 +473,7 @@ export class TTSManager {
     for (const [audioId, pending] of this.pendingPlaybacks.entries()) {
       pending.resolve();
       this.pendingPlaybacks.delete(audioId);
+      this.rememberClosedAudioId(audioId);
       this.logger.debug({ audioId }, "Cleared pending playback");
     }
   }
@@ -328,6 +486,7 @@ export class TTSManager {
     for (const [audioId, pending] of this.pendingPlaybacks.entries()) {
       pending.reject(new Error("Session closed"));
       this.pendingPlaybacks.delete(audioId);
+      this.rememberClosedAudioId(audioId);
     }
   }
 }

packages/server/src/server/bootstrap.ts

@@ -512,6 +512,7 @@ export async function createPaseoDaemon(
     },
   });
   const {
+    resolveVoiceTurnDetection,
     resolveVoiceStt,
     resolveVoiceTts,
     resolveDictationStt,
@@ -536,7 +537,7 @@ export async function createPaseoDaemon(
     config.paseoHome,
     createInMemoryAgentMcpTransport,
     { allowedOrigins, allowedHosts: config.allowedHosts },
-    { stt: resolveVoiceStt, tts: resolveVoiceTts },
+    { turnDetection: resolveVoiceTurnDetection, stt: resolveVoiceStt, tts: resolveVoiceTts },
     terminalManager,
     {
       voiceAgentMcpStdio: {

packages/server/src/server/persisted-config.test.ts

@@ -107,3 +107,19 @@ describe("PersistedConfigSchema logging config", () => {
     expect(result.success).toBe(false);
   });
 });
+
+describe("PersistedConfigSchema voice mode config", () => {
+  test("accepts a dedicated turn detection provider", () => {
+    const parsed = PersistedConfigSchema.parse({
+      features: {
+        voiceMode: {
+          turnDetection: {
+            provider: "local",
+          },
+        },
+      },
+    });
+
+    expect(parsed.features?.voiceMode?.turnDetection?.provider).toBe("local");
+  });
+});

packages/server/src/server/persisted-config.ts

@@ -94,6 +94,12 @@ const FeatureVoiceModeSchema = z
       })
       .strict()
       .optional(),
+    turnDetection: z
+      .object({
+        provider: SpeechProviderIdSchema.optional(),
+      })
+      .strict()
+      .optional(),
     tts: z
       .object({
         provider: SpeechProviderIdSchema.optional(),

packages/server/src/server/session.ts

@@ -43,12 +43,17 @@ import {
 import { TTSManager } from './agent/tts-manager.js'
 import { STTManager } from './agent/stt-manager.js'
 import type { SpeechToTextProvider, TextToSpeechProvider } from './speech/speech-provider.js'
+import type { TurnDetectionProvider } from './speech/turn-detection-provider.js'
 import { maybePersistTtsDebugAudio } from './agent/tts-debug.js'
 import { isPaseoDictationDebugEnabled } from './agent/recordings-debug.js'
 import {
   DictationStreamManager,
   type DictationStreamOutboundMessage,
 } from './dictation/dictation-stream-manager.js'
+import {
+  createVoiceTurnController,
+  type VoiceTurnController,
+} from './voice/voice-turn-controller.js'
 import { buildConfigOverrides, buildSessionConfig, extractTimestamps } from './persistence-hooks.js'
 import { experimental_createMCPClient } from 'ai'
 import type { Transport } from '@modelcontextprotocol/sdk/shared/transport.js'
@@ -159,7 +164,7 @@ import {
   listLocalSpeechModels,
   type LocalSpeechModelId,
 } from './speech/providers/local/models.js'
-import type { Resolvable } from './speech/provider-resolver.js'
+import { toResolver, type Resolvable } from './speech/provider-resolver.js'
 import type { SpeechReadinessSnapshot, SpeechReadinessState } from './speech/speech-runtime.js'
 import type pino from 'pino'
 import { resolveClientMessageId } from './client-message-id.js'
@@ -331,8 +336,6 @@ const PCM_BITS_PER_SAMPLE = 16
 const PCM_BYTES_PER_MS = (PCM_SAMPLE_RATE * PCM_CHANNELS * (PCM_BITS_PER_SAMPLE / 8)) / 1000
 const MIN_STREAMING_SEGMENT_DURATION_MS = 1000
 const MIN_STREAMING_SEGMENT_BYTES = Math.round(PCM_BYTES_PER_MS * MIN_STREAMING_SEGMENT_DURATION_MS)
-const VOICE_MODE_INACTIVITY_FLUSH_MS = 4500
-const VOICE_INTERNAL_DICTATION_ID_PREFIX = '__voice_turn__:'
 const SAFE_GIT_REF_PATTERN = /^[A-Za-z0-9._\/-]+$/
 const AgentIdSchema = z.string().uuid()
 const VOICE_MCP_SERVER_NAME = 'paseo_voice'
@@ -380,6 +383,7 @@ export type SessionOptions = {
   terminalManager: TerminalManager | null
   voice?: {
     voiceAgentMcpStdio?: VoiceMcpStdioConfig | null
+    turnDetection?: Resolvable<TurnDetectionProvider | null>
   }
   voiceBridge?: {
     registerVoiceSpeakHandler?: (agentId: string, handler: VoiceSpeakHandler) => void
@@ -528,26 +532,13 @@ export class Session {
   private speechInProgress = false
 
   private readonly dictationStreamManager: DictationStreamManager
-  private readonly voiceStreamManager: DictationStreamManager
+  private readonly resolveVoiceTurnDetection: () => TurnDetectionProvider | null
+  private voiceTurnController: VoiceTurnController | null = null
 
   // Audio buffering for interruption handling
   private pendingAudioSegments: Array<{ audio: Buffer; format: string }> = []
   private bufferTimeout: NodeJS.Timeout | null = null
-  private voiceModeInactivityTimeout: NodeJS.Timeout | null = null
   private audioBuffer: AudioBufferState | null = null
-  private activeVoiceDictationId: string | null = null
-  private activeVoiceDictationFormat: string | null = null
-  private activeVoiceDictationNextSeq = 0
-  private activeVoiceDictationStartPromise: Promise<void> | null = null
-  private activeVoiceDictationFinalizePromise: Promise<void> | null = null
-  private activeVoiceDictationResultPromise: Promise<{
-    text: string
-    debugRecordingPath?: string
-  }> | null = null
-  private activeVoiceDictationResolve:
-    | ((value: { text: string; debugRecordingPath?: string }) => void)
-    | null = null
-  private activeVoiceDictationReject: ((error: Error) => void) | null = null
 
   // Optional TTS debug capture (persisted per utterance)
   private readonly ttsDebugStreams = new Map<string, { format: string; chunks: Buffer[] }>()
@@ -658,6 +649,7 @@ export class Session {
       )
     }
     this.voiceAgentMcpStdio = voice?.voiceAgentMcpStdio ?? null
+    this.resolveVoiceTurnDetection = toResolver(voice?.turnDetection ?? null)
     const configuredModelsDir = dictation?.localModels?.modelsDir?.trim()
     this.localSpeechModelsDir =
       configuredModelsDir && configuredModelsDir.length > 0
@@ -695,13 +687,6 @@ export class Session {
       stt: dictation?.stt ?? null,
       finalTimeoutMs: dictation?.finalTimeoutMs,
     })
-    this.voiceStreamManager = new DictationStreamManager({
-      logger: this.sessionLogger.child({ stream: 'voice-internal' }),
-      sessionId: this.sessionId,
-      emit: (msg) => this.handleDictationManagerMessage(msg),
-      stt: stt,
-      finalTimeoutMs: dictation?.finalTimeoutMs,
-    })
 
     // Initialize agent MCP client asynchronously
     void this.initializeAgentMcp()
@@ -2167,6 +2152,7 @@ export class Session {
           this.voiceModeAgentId = refreshedAgentId
         }
 
+        await this.startVoiceTurnController()
         this.isVoiceMode = true
         this.sessionLogger.info(
           {
@@ -2303,8 +2289,7 @@ export class Session {
   }
 
   private async disableVoiceModeForActiveAgent(restoreAgentConfig: boolean): Promise<void> {
-    this.clearVoiceModeInactivityTimeout()
+    await this.stopVoiceTurnController()
-    this.cancelActiveVoiceDictationStream('voice mode disabled')
 
     const agentId = this.voiceModeAgentId
     if (!agentId) {
@@ -2340,231 +2325,74 @@ export class Session {
     this.voiceModeAgentId = null
   }
 
-  private isInternalVoiceDictationId(dictationId: string): boolean {
-    return dictationId.startsWith(VOICE_INTERNAL_DICTATION_ID_PREFIX)
-  }
-
   private handleDictationManagerMessage(msg: DictationStreamOutboundMessage): void {
-    if (msg.type === 'activity_log') {
+    this.emit(msg as unknown as SessionOutboundMessage)
-      const metadata = msg.payload.metadata as { dictationId?: unknown } | undefined
-      const dictationId =
-        metadata && typeof metadata.dictationId === 'string' ? metadata.dictationId : null
-      if (dictationId && this.isInternalVoiceDictationId(dictationId)) {
-        return
-      }
-      this.emit(msg as unknown as SessionOutboundMessage)
-      return
-    }
-
-    const payloadWithDictationId = msg.payload as { dictationId?: unknown }
-    const dictationId =
-      payloadWithDictationId && typeof payloadWithDictationId.dictationId === 'string'
-        ? payloadWithDictationId.dictationId
-        : null
-
-    if (!dictationId || !this.isInternalVoiceDictationId(dictationId)) {
-      this.emit(msg as unknown as SessionOutboundMessage)
-      return
-    }
-
-    if (msg.type === 'dictation_stream_final') {
-      if (dictationId !== this.activeVoiceDictationId || !this.activeVoiceDictationResolve) {
-        return
-      }
-      this.activeVoiceDictationResolve({
-        text: msg.payload.text,
-        ...(msg.payload.debugRecordingPath
-          ? { debugRecordingPath: msg.payload.debugRecordingPath }
-          : {}),
-      })
-      return
-    }
-
-    if (msg.type === 'dictation_stream_error') {
-      if (dictationId !== this.activeVoiceDictationId || !this.activeVoiceDictationReject) {
-        return
-      }
-      this.activeVoiceDictationReject(new Error(msg.payload.error))
-      return
-    }
-
-    // Ack/partial messages for internal voice dictation are consumed server-side.
   }
 
-  private resetActiveVoiceDictationState(): void {
+  private async startVoiceTurnController(): Promise<void> {
-    this.activeVoiceDictationId = null
+    if (this.voiceTurnController) {
-    this.activeVoiceDictationFormat = null
-    this.activeVoiceDictationNextSeq = 0
-    this.activeVoiceDictationStartPromise = null
-    this.activeVoiceDictationFinalizePromise = null
-    this.activeVoiceDictationResultPromise = null
-    this.activeVoiceDictationResolve = null
-    this.activeVoiceDictationReject = null
-  }
-
-  private cancelActiveVoiceDictationStream(reason: string): void {
-    const dictationId = this.activeVoiceDictationId
-    if (!dictationId) {
       return
     }
 
-    this.sessionLogger.debug(
+    const turnDetection = this.resolveVoiceTurnDetection()
-      { dictationId, reason },
+    if (!turnDetection) {
-      'Cancelling active internal voice dictation stream'
+      throw new Error('Voice turn detection is not configured')
-    )
-    if (this.activeVoiceDictationReject) {
-      this.activeVoiceDictationReject(new Error(`Voice dictation cancelled: ${reason}`))
-    }
-    this.voiceStreamManager.handleCancel(dictationId)
-    this.resetActiveVoiceDictationState()
-  }
-
-  private async ensureActiveVoiceDictationStream(format: string): Promise<void> {
-    if (this.activeVoiceDictationId && this.activeVoiceDictationFormat === format) {
-      if (this.activeVoiceDictationStartPromise) {
-        await this.activeVoiceDictationStartPromise
-      }
-      return
     }
 
-    if (this.activeVoiceDictationId) {
+    const controller = createVoiceTurnController({
-      await this.finalizeActiveVoiceDictationStream('voice format changed')
+      logger: this.sessionLogger.child({ component: 'voice-turn-controller' }),
-    }
+      turnDetection,
-
+      utteranceSink: {
-    const dictationId = `${VOICE_INTERNAL_DICTATION_ID_PREFIX}${uuidv4()}`
+        submitUtterance: async ({ pcm16, format, sampleRate, startedAt, endedAt }) => {
-    let resolve: ((value: { text: string; debugRecordingPath?: string }) => void) | null = null
+          this.sessionLogger.debug(
-    let reject: ((error: Error) => void) | null = null
+            {
-    const resultPromise = new Promise<{ text: string; debugRecordingPath?: string }>(
+              audioBytes: pcm16.length,
-      (resolveFn, rejectFn) => {
+              sampleRate,
-        resolve = resolveFn
+              startedAt,
-        reject = rejectFn
+              endedAt,
-      }
+              durationMs: Math.max(0, endedAt - startedAt),
-    )
+            },
-    // Prevent process-level unhandled rejection warnings when cancellation races are resolved later.
+            'Submitting detected voice utterance'
-    void resultPromise.catch(() => undefined)
+          )
-
+          await this.processCompletedAudio(pcm16, format)
-    this.activeVoiceDictationId = dictationId
+        },
-    this.activeVoiceDictationFormat = format
+      },
-    this.activeVoiceDictationNextSeq = 0
+      callbacks: {
-    this.activeVoiceDictationFinalizePromise = null
+        onSpeechStarted: async () => {
-    this.activeVoiceDictationResultPromise = resultPromise
+          this.emit({
-    this.activeVoiceDictationResolve = resolve
+            type: 'voice_input_state',
-    this.activeVoiceDictationReject = reject
+            payload: {
-    this.setPhase('transcribing')
+              isSpeaking: true,
-    this.emit({
+            },
-      type: 'activity_log',
+          })
-      payload: {
+          await this.handleVoiceSpeechStart()
-        id: uuidv4(),
+        },
-        timestamp: new Date(),
+        onSpeechStopped: async () => {
-        type: 'system',
+          this.emit({
-        content: 'Transcribing audio...',
+            type: 'voice_input_state',
+            payload: {
+              isSpeaking: false,
+            },
+          })
+        },
+        onError: (error) => {
+          this.sessionLogger.error({ err: error }, 'Voice turn controller failed')
+        },
       },
     })
 
-    const startPromise = this.voiceStreamManager.handleStart(dictationId, format)
+    await controller.start()
-    this.activeVoiceDictationStartPromise = startPromise
+    this.voiceTurnController = controller
-    try {
-      await startPromise
-    } catch (error) {
-      this.resetActiveVoiceDictationState()
-      throw error
-    } finally {
-      if (this.activeVoiceDictationId === dictationId) {
-        this.activeVoiceDictationStartPromise = null
-      }
-    }
   }
 
-  private async appendToActiveVoiceDictationStream(
+  private async stopVoiceTurnController(): Promise<void> {
-    audioBase64: string,
+    if (!this.voiceTurnController) {
-    format: string
-  ): Promise<void> {
-    if (this.activeVoiceDictationFinalizePromise) {
-      await this.activeVoiceDictationFinalizePromise.catch(() => undefined)
-    }
-    await this.ensureActiveVoiceDictationStream(format)
-    const dictationId = this.activeVoiceDictationId
-    if (!dictationId) {
-      throw new Error('Voice dictation stream did not initialize')
-    }
-
-    const seq = this.activeVoiceDictationNextSeq
-    this.activeVoiceDictationNextSeq += 1
-    await this.voiceStreamManager.handleChunk({
-      dictationId,
-      seq,
-      audioBase64,
-      format,
-    })
-  }
-
-  private async finalizeActiveVoiceDictationStream(reason: string): Promise<void> {
-    const dictationId = this.activeVoiceDictationId
-    if (!dictationId) {
-      return
-    }
-    this.clearVoiceModeInactivityTimeout()
-    if (this.activeVoiceDictationStartPromise) {
-      await this.activeVoiceDictationStartPromise
-    }
-
-    if (this.activeVoiceDictationFinalizePromise) {
-      await this.activeVoiceDictationFinalizePromise
       return
     }
 
-    const finalSeq = this.activeVoiceDictationNextSeq - 1
+    const controller = this.voiceTurnController
-    const resultPromise = this.activeVoiceDictationResultPromise
+    this.voiceTurnController = null
-    if (!resultPromise) {
+    await controller.stop()
-      this.resetActiveVoiceDictationState()
-      return
-    }
-
-    this.activeVoiceDictationFinalizePromise = (async () => {
-      this.sessionLogger.debug(
-        { dictationId, finalSeq, reason },
-        'Finalizing internal voice dictation stream'
-      )
-      await this.voiceStreamManager.handleFinish(dictationId, finalSeq)
-      const result = await resultPromise
-      this.resetActiveVoiceDictationState()
-      const requestId = uuidv4()
-      const transcriptText = result.text.trim()
-      this.sessionLogger.info(
-        {
-          requestId,
-          isVoiceMode: this.isVoiceMode,
-          transcriptLength: transcriptText.length,
-          transcript: transcriptText,
-        },
-        'Transcription result'
-      )
-      await this.handleTranscriptionResultPayload({
-        text: result.text,
-        requestId,
-        ...(result.debugRecordingPath
-          ? { debugRecordingPath: result.debugRecordingPath, format: 'audio/wav' }
-          : {}),
-      })
-    })()
-
-    try {
-      await this.activeVoiceDictationFinalizePromise
-    } catch (error) {
-      this.resetActiveVoiceDictationState()
-      this.setPhase('idle')
-      this.clearSpeechInProgress('transcription error')
-      this.emit({
-        type: 'activity_log',
-        payload: {
-          id: uuidv4(),
-          timestamp: new Date(),
-          type: 'error',
-          content: `Transcription error: ${error instanceof Error ? error.message : String(error)}`,
-        },
-      })
-      throw error
-    }
   }
 
   /**
@@ -6421,21 +6249,16 @@ export class Session {
       )
     }
 
-    await this.handleVoiceSpeechStart()
-
     const chunkFormat = msg.format || 'audio/wav'
 
     if (this.isVoiceMode) {
-      await this.appendToActiveVoiceDictationStream(msg.audio, chunkFormat)
+      if (!this.voiceTurnController) {
-      if (!msg.isLast) {
+        throw new Error('Voice mode is enabled but the voice turn controller is not running')
-        this.setVoiceModeInactivityTimeout()
-        this.sessionLogger.debug('Voice mode: streaming chunk, waiting for speech end')
-        return
       }
-
+      await this.voiceTurnController.appendClientChunk({
-      this.clearVoiceModeInactivityTimeout()
+        audioBase64: msg.audio,
-      this.sessionLogger.debug('Voice mode: speech ended, finalizing streaming transcription')
+        format: chunkFormat,
-      await this.finalizeActiveVoiceDictationStream('speech ended')
+      })
       return
     }
 
@@ -6849,8 +6672,6 @@ export class Session {
       this.audioBuffer = null
     }
 
-    this.cancelActiveVoiceDictationStream('new speech turn started')
-    this.clearVoiceModeInactivityTimeout()
     this.clearBufferTimeout()
 
     this.abortController.abort()
@@ -6913,43 +6734,6 @@ export class Session {
     }, 10000) // 10 second timeout
   }
 
-  private setVoiceModeInactivityTimeout(): void {
-    if (!this.isVoiceMode) {
-      return
-    }
-
-    this.clearVoiceModeInactivityTimeout()
-    this.voiceModeInactivityTimeout = setTimeout(() => {
-      this.voiceModeInactivityTimeout = null
-      if (!this.isVoiceMode || !this.activeVoiceDictationId) {
-        return
-      }
-
-      this.sessionLogger.warn(
-        {
-          timeoutMs: VOICE_MODE_INACTIVITY_FLUSH_MS,
-          dictationId: this.activeVoiceDictationId,
-          nextSeq: this.activeVoiceDictationNextSeq,
-        },
-        'Voice mode inactivity timeout reached without isLast; finalizing active voice dictation stream'
-      )
-
-      void this.finalizeActiveVoiceDictationStream('inactivity timeout').catch((error) => {
-        this.sessionLogger.error(
-          { err: error },
-          'Failed to finalize voice dictation stream after inactivity timeout'
-        )
-      })
-    }, VOICE_MODE_INACTIVITY_FLUSH_MS)
-  }
-
-  private clearVoiceModeInactivityTimeout(): void {
-    if (this.voiceModeInactivityTimeout) {
-      clearTimeout(this.voiceModeInactivityTimeout)
-      this.voiceModeInactivityTimeout = null
-    }
-  }
-
   /**
    * Clear buffer timeout
    */
@@ -7041,18 +6825,16 @@ export class Session {
     this.abortController.abort()
 
     // Clear timeouts
-    this.clearVoiceModeInactivityTimeout()
     this.clearBufferTimeout()
 
     // Clear buffers
-    this.cancelActiveVoiceDictationStream('session cleanup')
     this.pendingAudioSegments = []
     this.audioBuffer = null
+    await this.stopVoiceTurnController()
 
     // Cleanup managers
     this.ttsManager.cleanup()
     this.sttManager.cleanup()
-    this.voiceStreamManager.cleanupAll()
     this.dictationStreamManager.cleanupAll()
 
     // Close MCP clients

packages/server/src/server/speech/providers/local/runtime.ts

@@ -3,6 +3,7 @@ import type { Logger } from "pino";
 import type { PaseoSpeechConfig } from "../../../bootstrap.js";
 import type { SpeechToTextProvider, TextToSpeechProvider } from "../../speech-provider.js";
 import type { RequestedSpeechProviders } from "../../speech-types.js";
+import type { TurnDetectionProvider } from "../../turn-detection-provider.js";
 import { PocketTtsOnnxTTS } from "./pocket/pocket-tts-onnx.js";
 import {
   ensureLocalSpeechModels,
@@ -22,6 +23,7 @@ import { SherpaParakeetRealtimeTranscriptionSession } from "./sherpa/sherpa-para
 import { SherpaRealtimeTranscriptionSession } from "./sherpa/sherpa-realtime-session.js";
 import { SherpaOnnxSTT } from "./sherpa/sherpa-stt.js";
 import { SherpaOnnxTTS } from "./sherpa/sherpa-tts.js";
+import { SherpaSileroTurnDetectionProvider } from "./sherpa/silero-vad-provider.js";
 
 type LocalSttEngine =
   | { kind: "offline"; engine: SherpaOfflineRecognizerEngine }
@@ -39,6 +41,7 @@ type LocalSpeechAvailability = {
 };
 
 export type InitializedLocalSpeech = {
+  turnDetectionService: TurnDetectionProvider | null;
   sttService: SpeechToTextProvider | null;
   ttsService: TextToSpeechProvider | null;
   dictationSttService: SpeechToTextProvider | null;
@@ -192,6 +195,7 @@ export async function initializeLocalSpeechServices(params: {
   let sttService: SpeechToTextProvider | null = null;
   let ttsService: TextToSpeechProvider | null = null;
   let dictationSttService: SpeechToTextProvider | null = null;
+  let turnDetectionService: TurnDetectionProvider | null = null;
   let localVoiceTtsProvider: TextToSpeechProvider | null = null;
 
   const requiredLocalModelIds = computeRequiredLocalModelIds({
@@ -262,6 +266,13 @@ export async function initializeLocalSpeechServices(params: {
     }
   };
 
+  if (
+    providers.voiceTurnDetection.enabled !== false &&
+    providers.voiceTurnDetection.provider === "local"
+  ) {
+    turnDetectionService = new SherpaSileroTurnDetectionProvider({}, logger);
+  }
+
   if (providers.voiceStt.enabled !== false && providers.voiceStt.provider === "local") {
     if (!localConfig) {
       logger.warn(
@@ -357,6 +368,7 @@ export async function initializeLocalSpeechServices(params: {
   };
 
   return {
+    turnDetectionService,
     sttService,
     ttsService,
     dictationSttService,

packages/server/src/server/speech/providers/local/sherpa/sherpa-onnx-node-loader.ts

@@ -12,6 +12,8 @@ export type SherpaOnnxNodeModule = {
   OfflineRecognizer: new (config: any) => any;
   OnlineRecognizer?: new (config: any) => any;
   OfflineTts?: new (config: any) => any;
+  Vad?: new (config: any, bufferSizeInSeconds: number) => any;
+  CircularBuffer?: new (capacity: number) => any;
 };
 
 let cached: SherpaOnnxNodeModule | null = null;

packages/server/src/server/speech/providers/local/sherpa/silero-vad-provider.ts

@@ -0,0 +1,40 @@
+import type { Logger } from "pino";
+
+import type {
+  TurnDetectionProvider,
+  TurnDetectionSession,
+} from "../../../turn-detection-provider.js";
+import {
+  SherpaSileroVadSession,
+  type SherpaSileroVadSessionConfig,
+} from "./silero-vad-session.js";
+
+export class SherpaSileroTurnDetectionProvider implements TurnDetectionProvider {
+  public readonly id = "local" as const;
+
+  private readonly config: SherpaSileroVadSessionConfig;
+  private readonly logger: Logger;
+
+  constructor(config: SherpaSileroVadSessionConfig, logger: Logger) {
+    this.config = config;
+    this.logger = logger.child({
+      module: "speech",
+      provider: "local",
+      component: "silero-vad",
+    });
+  }
+
+  createSession(params: { logger: Logger }): TurnDetectionSession {
+    this.logger.debug(
+      { sampleRate: this.config.sampleRate, modelPath: this.config.modelPath },
+      "Creating Silero VAD turn-detection session"
+    );
+    return new SherpaSileroVadSession({
+      logger: params.logger.child({
+        provider: "local",
+        component: "silero-vad-session",
+      }),
+      config: this.config,
+    });
+  }
+}

packages/server/src/server/speech/providers/local/sherpa/silero-vad-session.ts

@@ -0,0 +1,168 @@
+import { EventEmitter } from "node:events";
+import { fileURLToPath } from "node:url";
+
+import { pcm16leToFloat32 } from "../../../audio.js";
+import type { TurnDetectionSession } from "../../../turn-detection-provider.js";
+import { loadSherpaOnnxNode } from "./sherpa-onnx-node-loader.js";
+
+const DEFAULT_SAMPLE_RATE = 16000;
+const DEFAULT_BUFFER_SIZE_SECONDS = 60;
+const DEFAULT_SILERO_THRESHOLD = 0.5;
+const DEFAULT_MIN_SILENCE_DURATION = 1.2;
+const DEFAULT_MIN_SPEECH_DURATION = 0.12;
+const DEFAULT_WINDOW_SIZE = 512;
+
+type SherpaVadHandle = {
+  acceptWaveform(samples: Float32Array): void;
+  isDetected(): boolean;
+  isEmpty(): boolean;
+  flush(): void;
+  reset(): void;
+};
+
+type SherpaCircularBufferHandle = {
+  push(samples: Float32Array): void;
+  get(startIndex: number, n: number, enableExternalBuffer?: boolean): Float32Array;
+  pop(n: number): void;
+  size(): number;
+  head(): number;
+  reset(): void;
+};
+
+type SherpaVadModule = {
+  Vad: new (config: Record<string, unknown>, bufferSizeInSeconds: number) => SherpaVadHandle;
+  CircularBuffer: new (capacity: number) => SherpaCircularBufferHandle;
+};
+
+function resolveBundledSileroVadModelPath(): string {
+  return fileURLToPath(new URL("./assets/silero_vad.onnx", import.meta.url));
+}
+
+export interface SherpaSileroVadSessionConfig {
+  modelPath?: string;
+  sampleRate?: number;
+  threshold?: number;
+  minSilenceDuration?: number;
+  minSpeechDuration?: number;
+  windowSize?: number;
+  bufferSizeInSeconds?: number;
+}
+
+export class SherpaSileroVadSession
+  extends EventEmitter
+  implements TurnDetectionSession
+{
+  public readonly requiredSampleRate: number;
+
+  private readonly vad: SherpaVadHandle;
+  private readonly inputBuffer: SherpaCircularBufferHandle;
+  private readonly windowSize: number;
+  private connected = false;
+  private inSpeech = false;
+  private readonly logger;
+
+  constructor(params: {
+    logger: { debug: (...args: unknown[]) => void };
+    config?: SherpaSileroVadSessionConfig;
+  }) {
+    super();
+    this.logger = params.logger;
+    const config = params.config ?? {};
+    this.requiredSampleRate = config.sampleRate ?? DEFAULT_SAMPLE_RATE;
+    this.windowSize = config.windowSize ?? DEFAULT_WINDOW_SIZE;
+
+    const sherpa = loadSherpaOnnxNode() as unknown as SherpaVadModule;
+    this.vad = new sherpa.Vad(
+      {
+        sileroVad: {
+          model: config.modelPath ?? resolveBundledSileroVadModelPath(),
+          threshold: config.threshold ?? DEFAULT_SILERO_THRESHOLD,
+          minSilenceDuration:
+            config.minSilenceDuration ?? DEFAULT_MIN_SILENCE_DURATION,
+          minSpeechDuration:
+            config.minSpeechDuration ?? DEFAULT_MIN_SPEECH_DURATION,
+          windowSize: config.windowSize ?? DEFAULT_WINDOW_SIZE,
+        },
+        sampleRate: this.requiredSampleRate,
+        numThreads: 1,
+        provider: "cpu",
+        debug: 0,
+      },
+      config.bufferSizeInSeconds ?? DEFAULT_BUFFER_SIZE_SECONDS
+    );
+    this.inputBuffer = new sherpa.CircularBuffer(
+      (config.bufferSizeInSeconds ?? DEFAULT_BUFFER_SIZE_SECONDS) * this.requiredSampleRate
+    );
+  }
+
+  async connect(): Promise<void> {
+    this.connected = true;
+  }
+
+  appendPcm16(pcm16le: Buffer): void {
+    if (!this.connected) {
+      this.emit("error", new Error("Turn detection session not connected"));
+      return;
+    }
+    if (pcm16le.length === 0) {
+      return;
+    }
+
+    try {
+      const samples = pcm16leToFloat32(pcm16le, 1);
+      this.inputBuffer.push(samples);
+      while (this.inputBuffer.size() > this.windowSize) {
+        const window = this.inputBuffer.get(this.inputBuffer.head(), this.windowSize);
+        this.inputBuffer.pop(this.windowSize);
+        this.vad.acceptWaveform(window);
+        this.syncDetectionState();
+      }
+    } catch (error) {
+      this.emit("error", error instanceof Error ? error : new Error(String(error)));
+    }
+  }
+
+  flush(): void {
+    if (!this.connected) {
+      return;
+    }
+
+    try {
+      this.vad.flush();
+      this.syncDetectionState();
+      if (this.inSpeech) {
+        this.inSpeech = false;
+        this.emit("speech_stopped");
+      }
+    } catch (error) {
+      this.emit("error", error instanceof Error ? error : new Error(String(error)));
+    }
+  }
+
+  close(): void {
+    try {
+      this.vad.reset();
+      this.inputBuffer.reset();
+    } catch {
+      // ignore native cleanup failures
+    } finally {
+      this.connected = false;
+      this.inSpeech = false;
+    }
+  }
+
+  private syncDetectionState(): void {
+    const detected = this.vad.isDetected();
+    if (detected && !this.inSpeech) {
+      this.inSpeech = true;
+      this.emit("speech_started");
+      return;
+    }
+
+    if (!detected && this.inSpeech && !this.vad.isEmpty()) {
+      this.logger.debug("Silero VAD marked end of speech");
+      this.inSpeech = false;
+      this.emit("speech_stopped");
+    }
+  }
+}

packages/server/src/server/speech/providers/openai/runtime.ts

@@ -2,6 +2,7 @@ import type { Logger } from "pino";
 
 import type { SpeechToTextProvider, TextToSpeechProvider } from "../../speech-provider.js";
 import type { RequestedSpeechProviders } from "../../speech-types.js";
+import type { TurnDetectionProvider } from "../../turn-detection-provider.js";
 import {
   DEFAULT_OPENAI_REALTIME_TRANSCRIPTION_MODEL,
   DEFAULT_OPENAI_TTS_MODEL,
@@ -24,6 +25,7 @@ export type OpenAiSpeechAvailability = {
 };
 
 export type SpeechServices = {
+  turnDetectionService: TurnDetectionProvider | null;
   sttService: SpeechToTextProvider | null;
   ttsService: TextToSpeechProvider | null;
   dictationSttService: SpeechToTextProvider | null;
@@ -112,6 +114,7 @@ export function initializeOpenAiSpeechServices(params: {
   let sttService = existing.sttService;
   let ttsService = existing.ttsService;
   let dictationSttService = existing.dictationSttService;
+  const turnDetectionService = existing.turnDetectionService;
 
   const needsOpenAiStt =
     !sttService &&
@@ -181,6 +184,7 @@ export function initializeOpenAiSpeechServices(params: {
   }
 
   return {
+    turnDetectionService,
     sttService,
     ttsService,
     dictationSttService,

packages/server/src/server/speech/speech-config-resolver.test.ts

@@ -23,6 +23,11 @@ describe("resolveSpeechConfig", () => {
       explicit: false,
       enabled: true,
     });
+    expect(result.speech.providers.voiceTurnDetection).toEqual({
+      provider: "local",
+      explicit: false,
+      enabled: true,
+    });
     expect(result.speech.providers.voiceStt).toEqual({
       provider: "local",
       explicit: false,
@@ -52,6 +57,7 @@ describe("resolveSpeechConfig", () => {
     const persisted = PersistedConfigSchema.parse({
       features: {
         voiceMode: {
+          turnDetection: { provider: "local" },
           stt: { provider: "openai", model: "gpt-4o-transcribe" },
         },
       },
@@ -98,6 +104,11 @@ describe("resolveSpeechConfig", () => {
       explicit: true,
       enabled: true,
     });
+    expect(result.speech.providers.voiceTurnDetection).toEqual({
+      provider: "local",
+      explicit: true,
+      enabled: true,
+    });
     expect(result.speech.providers.voiceTts).toEqual({
       provider: "local",
       explicit: true,
@@ -150,6 +161,11 @@ describe("resolveSpeechConfig", () => {
       explicit: false,
       enabled: false,
     });
+    expect(result.speech.providers.voiceTurnDetection).toEqual({
+      provider: "local",
+      explicit: false,
+      enabled: false,
+    });
     expect(result.speech.providers.voiceStt).toEqual({
       provider: "local",
       explicit: false,

packages/server/src/server/speech/speech-config-resolver.ts

@@ -38,6 +38,7 @@ const OptionalBooleanFlagSchema = z
 
 const RequestedSpeechProvidersSchema = z.object({
   dictationStt: OptionalSpeechProviderSchema.default("local"),
+  voiceTurnDetection: OptionalSpeechProviderSchema.default("local"),
   voiceStt: OptionalSpeechProviderSchema.default("local"),
   voiceTts: OptionalSpeechProviderSchema.default("local"),
 });
@@ -62,6 +63,9 @@ function resolveRequestedSpeechProviders(params: {
   const voiceSttProviderFromConfig =
     params.env.PASEO_VOICE_STT_PROVIDER ??
     params.persisted.features?.voiceMode?.stt?.provider;
+  const voiceTurnDetectionProviderFromConfig =
+    params.env.PASEO_VOICE_TURN_DETECTION_PROVIDER ??
+    params.persisted.features?.voiceMode?.turnDetection?.provider;
   const voiceTtsProviderFromConfig =
     params.env.PASEO_VOICE_TTS_PROVIDER ??
     params.persisted.features?.voiceMode?.tts?.provider;
@@ -76,6 +80,7 @@ function resolveRequestedSpeechProviders(params: {
 
   const parsed = RequestedSpeechProvidersSchema.parse({
     dictationStt: dictationSttProviderFromConfig ?? "local",
+    voiceTurnDetection: voiceTurnDetectionProviderFromConfig ?? "local",
     voiceStt: voiceSttProviderFromConfig ?? "local",
     voiceTts: voiceTtsProviderFromConfig ?? "local",
   });
@@ -86,6 +91,11 @@ function resolveRequestedSpeechProviders(params: {
       parsed.dictationStt,
       dictationEnabled
     ),
+    voiceTurnDetection: resolveFeatureProvider(
+      voiceTurnDetectionProviderFromConfig,
+      parsed.voiceTurnDetection,
+      voiceModeEnabled
+    ),
     voiceStt: resolveFeatureProvider(
       voiceSttProviderFromConfig,
       parsed.voiceStt,

packages/server/src/server/speech/speech-runtime.test.ts

@@ -4,6 +4,7 @@ import pino from "pino";
 import type { PaseoSpeechConfig } from "../bootstrap.js";
 import type { InitializedLocalSpeech } from "./providers/local/runtime.js";
 import type { SpeechToTextProvider, TextToSpeechProvider } from "./speech-provider.js";
+import type { TurnDetectionProvider } from "./turn-detection-provider.js";
 import { initializeSpeechRuntime } from "./speech-runtime.js";
 
 const { initializeLocalSpeechServicesMock } = vi.hoisted(() => ({
@@ -20,11 +21,13 @@ vi.mock("./providers/openai/runtime.js", () => ({
   getOpenAiSpeechAvailability: () => ({ configured: false }),
   initializeOpenAiSpeechServices: (args: {
     existing: {
+      turnDetectionService: TurnDetectionProvider | null;
       sttService: SpeechToTextProvider | null;
       ttsService: TextToSpeechProvider | null;
       dictationSttService: SpeechToTextProvider | null;
     };
   }) => ({
+    turnDetectionService: args.existing.turnDetectionService,
     sttService: args.existing.sttService,
     ttsService: args.existing.ttsService,
     dictationSttService: args.existing.dictationSttService,
@@ -56,6 +59,15 @@ function createStubTts(id: string): TextToSpeechProvider {
   };
 }
 
+function createStubTurnDetection(id: string): TurnDetectionProvider {
+  return {
+    id,
+    createSession: vi.fn(() => {
+      throw new Error("not used in this test");
+    }),
+  };
+}
+
 function createSpeechConfig(
   providers: PaseoSpeechConfig["providers"]
 ): PaseoSpeechConfig {
@@ -71,6 +83,7 @@ describe("initializeSpeechRuntime readiness", () => {
     const dictationStt = createStubStt("dictation-local");
 
     initializeLocalSpeechServicesMock.mockResolvedValue({
+      turnDetectionService: null,
       sttService: null,
       ttsService: null,
       dictationSttService: dictationStt,
@@ -87,6 +100,7 @@ describe("initializeSpeechRuntime readiness", () => {
       logger: pino({ level: "silent" }),
       speechConfig: createSpeechConfig({
         dictationStt: { provider: "local", enabled: true, explicit: true },
+        voiceTurnDetection: { provider: "local", enabled: false, explicit: true },
         voiceStt: { provider: "local", enabled: false, explicit: true },
         voiceTts: { provider: "local", enabled: false, explicit: true },
       }),
@@ -104,8 +118,10 @@ describe("initializeSpeechRuntime readiness", () => {
   it("keeps voice feature available when only realtime voice is enabled and ready", async () => {
     const voiceStt = createStubStt("voice-local");
     const voiceTts = createStubTts("tts-local");
+    const turnDetection = createStubTurnDetection("turn-local");
 
     initializeLocalSpeechServicesMock.mockResolvedValue({
+      turnDetectionService: turnDetection,
       sttService: voiceStt,
       ttsService: voiceTts,
       dictationSttService: null,
@@ -122,6 +138,7 @@ describe("initializeSpeechRuntime readiness", () => {
       logger: pino({ level: "silent" }),
       speechConfig: createSpeechConfig({
         dictationStt: { provider: "local", enabled: false, explicit: true },
+        voiceTurnDetection: { provider: "local", enabled: true, explicit: true },
         voiceStt: { provider: "local", enabled: true, explicit: true },
         voiceTts: { provider: "local", enabled: true, explicit: true },
       }),

packages/server/src/server/speech/speech-runtime.ts

@@ -17,6 +17,7 @@ import {
 } from "./providers/openai/runtime.js";
 import type { SpeechToTextProvider, TextToSpeechProvider } from "./speech-provider.js";
 import type { RequestedSpeechProviders } from "./speech-types.js";
+import type { TurnDetectionProvider } from "./turn-detection-provider.js";
 
 const SPEECH_RUNTIME_MONITOR_INTERVAL_MS = 3000;
 
@@ -26,6 +27,7 @@ export type SpeechReadinessReasonCode =
   | "model_download_in_progress"
   | "models_missing"
   | "model_download_failed"
+  | "turn_detection_unavailable"
   | "stt_unavailable"
   | "tts_unavailable";
 
@@ -61,6 +63,7 @@ function resolveRequestedSpeechProviders(
 
   return {
     dictationStt: { provider: "local", explicit: false, enabled: true },
+    voiceTurnDetection: { provider: "local", explicit: false, enabled: true },
     voiceStt: { provider: "local", explicit: false, enabled: true },
     voiceTts: { provider: "local", explicit: false, enabled: true },
   };
@@ -118,12 +121,14 @@ function joinModelIds(modelIds: LocalSpeechModelId[]): string {
 
 function buildRealtimeVoiceReadiness(params: {
   providers: RequestedSpeechProviders;
+  turnDetectionService: TurnDetectionProvider | null;
   sttService: SpeechToTextProvider | null;
   ttsService: TextToSpeechProvider | null;
 }): SpeechReadinessState {
+  const voiceTurnDetectionEnabled = params.providers.voiceTurnDetection.enabled !== false;
   const voiceSttEnabled = params.providers.voiceStt.enabled !== false;
   const voiceTtsEnabled = params.providers.voiceTts.enabled !== false;
-  const enabled = voiceSttEnabled || voiceTtsEnabled;
+  const enabled = voiceTurnDetectionEnabled || voiceSttEnabled || voiceTtsEnabled;
   if (!enabled) {
     return {
       enabled: false,
@@ -134,6 +139,16 @@ function buildRealtimeVoiceReadiness(params: {
       missingModelIds: [],
     };
   }
+  if (voiceTurnDetectionEnabled && !params.turnDetectionService) {
+    return {
+      enabled: true,
+      available: false,
+      reasonCode: "turn_detection_unavailable",
+      message: "Realtime voice is unavailable: turn-detection service is not ready.",
+      retryable: false,
+      missingModelIds: [],
+    };
+  }
   if (voiceSttEnabled && !params.sttService) {
     return {
       enabled: true,
@@ -262,6 +277,7 @@ function buildVoiceFeatureReadiness(params: {
 
 function describeRequestedProviders(providers: RequestedSpeechProviders): {
   dictationStt: { provider: string; enabled: boolean; explicit: boolean };
+  voiceTurnDetection: { provider: string; enabled: boolean; explicit: boolean };
   voiceStt: { provider: string; enabled: boolean; explicit: boolean };
   voiceTts: { provider: string; enabled: boolean; explicit: boolean };
 } {
@@ -271,6 +287,11 @@ function describeRequestedProviders(providers: RequestedSpeechProviders): {
       enabled: providers.dictationStt.enabled !== false,
       explicit: providers.dictationStt.explicit,
     },
+    voiceTurnDetection: {
+      provider: providers.voiceTurnDetection.provider,
+      enabled: providers.voiceTurnDetection.enabled !== false,
+      explicit: providers.voiceTurnDetection.explicit,
+    },
     voiceStt: {
       provider: providers.voiceStt.provider,
       enabled: providers.voiceStt.enabled !== false,
@@ -285,13 +306,20 @@ function describeRequestedProviders(providers: RequestedSpeechProviders): {
 }
 
 function resolveEffectiveProviderIds(params: {
+  turnDetectionService: TurnDetectionProvider | null;
   sttService: SpeechToTextProvider | null;
   ttsService: TextToSpeechProvider | null;
   dictationSttService: SpeechToTextProvider | null;
   localVoiceTtsProvider: TextToSpeechProvider | null;
-}): { dictationStt: string; voiceStt: string; voiceTts: string } {
+}): {
+  dictationStt: string;
+  voiceTurnDetection: string;
+  voiceStt: string;
+  voiceTts: string;
+} {
   return {
     dictationStt: params.dictationSttService?.id ?? "unavailable",
+    voiceTurnDetection: params.turnDetectionService?.id ?? "unavailable",
     voiceStt: params.sttService?.id ?? "unavailable",
     voiceTts:
       !params.ttsService
@@ -303,6 +331,7 @@ function resolveEffectiveProviderIds(params: {
 }
 
 export type InitializedSpeechRuntime = {
+  resolveVoiceTurnDetection: () => TurnDetectionProvider | null;
   resolveVoiceStt: () => SpeechToTextProvider | null;
   resolveVoiceTts: () => TextToSpeechProvider | null;
   resolveDictationStt: () => SpeechToTextProvider | null;
@@ -347,6 +376,7 @@ export async function initializeSpeechRuntime(params: {
   let sttService: SpeechToTextProvider | null = null;
   let ttsService: TextToSpeechProvider | null = null;
   let dictationSttService: SpeechToTextProvider | null = null;
+  let turnDetectionService: TurnDetectionProvider | null = null;
   let localModelConfig: {
     modelsDir: string;
     defaultModelIds: LocalSpeechModelId[];
@@ -367,6 +397,7 @@ export async function initializeSpeechRuntime(params: {
   const computeReadinessSnapshot = (): SpeechReadinessSnapshot => {
     const realtimeVoice = buildRealtimeVoiceReadiness({
       providers,
+      turnDetectionService,
       sttService,
       ttsService,
     });
@@ -466,6 +497,7 @@ export async function initializeSpeechRuntime(params: {
       providers,
       openaiConfig,
       existing: {
+        turnDetectionService: nextLocalSpeech.turnDetectionService,
         sttService: nextLocalSpeech.sttService,
         ttsService: nextLocalSpeech.ttsService,
         dictationSttService: nextLocalSpeech.dictationSttService,
@@ -474,6 +506,7 @@ export async function initializeSpeechRuntime(params: {
     });
 
     const previousLocalCleanup = localCleanup;
+    turnDetectionService = nextOpenAiSpeech.turnDetectionService;
     sttService = nextOpenAiSpeech.sttService;
     ttsService = nextOpenAiSpeech.ttsService;
     dictationSttService = nextOpenAiSpeech.dictationSttService;
@@ -485,6 +518,7 @@ export async function initializeSpeechRuntime(params: {
     await refreshMissingLocalModels();
 
     const effectiveProviders = resolveEffectiveProviderIds({
+      turnDetectionService,
       sttService,
       ttsService,
       dictationSttService,
@@ -492,6 +526,9 @@ export async function initializeSpeechRuntime(params: {
     });
     const unavailableFeatures = [
       providers.dictationStt.enabled !== false && !dictationSttService ? "dictation.stt" : null,
+      providers.voiceTurnDetection.enabled !== false && !turnDetectionService
+        ? "voice.turnDetection"
+        : null,
       providers.voiceStt.enabled !== false && !sttService ? "voice.stt" : null,
       providers.voiceTts.enabled !== false && !ttsService ? "voice.tts" : null,
     ].filter((feature): feature is string => feature !== null);
@@ -642,6 +679,7 @@ export async function initializeSpeechRuntime(params: {
   };
 
   return {
+    resolveVoiceTurnDetection: () => turnDetectionService,
     resolveVoiceStt: () => sttService,
     resolveVoiceTts: () => ttsService,
     resolveDictationStt: () => dictationSttService,

packages/server/src/server/speech/speech-types.ts

@@ -12,6 +12,7 @@ export type RequestedSpeechProvider = z.infer<typeof RequestedSpeechProviderSche
 
 export type RequestedSpeechProviders = {
   dictationStt: RequestedSpeechProvider;
+  voiceTurnDetection: RequestedSpeechProvider;
   voiceStt: RequestedSpeechProvider;
   voiceTts: RequestedSpeechProvider;
 };

packages/server/src/server/speech/turn-detection-provider.ts

@@ -0,0 +1,25 @@
+import type pino from "pino";
+
+export type TurnDetectionSession = {
+  /**
+   * Required PCM16LE sample rate for `appendPcm16()`.
+   * Callers are responsible for resampling before appending.
+   */
+  requiredSampleRate: number;
+
+  connect(): Promise<void>;
+  appendPcm16(pcm16le: Buffer): void;
+  flush(): void;
+  close(): void;
+
+  on(event: "speech_started", handler: () => void): unknown;
+  on(event: "speech_stopped", handler: () => void): unknown;
+  on(event: "error", handler: (err: unknown) => void): unknown;
+};
+
+export interface TurnDetectionProvider {
+  id: "openai" | "local" | (string & {});
+  createSession(params: {
+    logger: pino.Logger;
+  }): TurnDetectionSession;
+}

packages/server/src/server/voice/fixed-duration-pcm-ring-buffer.test.ts

@@ -0,0 +1,37 @@
+import { describe, expect, it } from "vitest";
+
+import { FixedDurationPcmRingBuffer } from "./fixed-duration-pcm-ring-buffer.js";
+
+describe("FixedDurationPcmRingBuffer", () => {
+  it("retains only the configured prefix window", () => {
+    const buffer = new FixedDurationPcmRingBuffer({
+      sampleRate: 1000,
+      channels: 1,
+      bitsPerSample: 16,
+      durationMs: 100,
+    });
+
+    buffer.append(Buffer.alloc(120));
+    buffer.append(Buffer.alloc(120, 1));
+
+    const drained = buffer.drain();
+    expect(drained.length).toBe(120);
+    expect(Array.from(drained.slice(0, 4))).toEqual([1, 1, 1, 1]);
+  });
+
+  it("drains buffered chunks in arrival order", () => {
+    const buffer = new FixedDurationPcmRingBuffer({
+      sampleRate: 1000,
+      channels: 1,
+      bitsPerSample: 16,
+      durationMs: 500,
+    });
+
+    buffer.append(Buffer.from([1, 2]));
+    buffer.append(Buffer.from([3, 4]));
+    buffer.append(Buffer.from([5, 6]));
+
+    expect(Array.from(buffer.drain())).toEqual([1, 2, 3, 4, 5, 6]);
+    expect(buffer.drain().length).toBe(0);
+  });
+});

packages/server/src/server/voice/fixed-duration-pcm-ring-buffer.ts

@@ -0,0 +1,51 @@
+export class FixedDurationPcmRingBuffer {
+  private readonly maxBytes: number;
+  private chunks: Buffer[] = [];
+  private totalBytes = 0;
+
+  constructor(params: {
+    sampleRate: number;
+    channels: number;
+    bitsPerSample: number;
+    durationMs: number;
+  }) {
+    const bytesPerSecond =
+      params.sampleRate * params.channels * (params.bitsPerSample / 8);
+    this.maxBytes = Math.max(
+      1,
+      Math.round((bytesPerSecond * params.durationMs) / 1000)
+    );
+  }
+
+  append(chunk: Buffer): void {
+    if (chunk.length === 0) {
+      return;
+    }
+
+    this.chunks.push(chunk);
+    this.totalBytes += chunk.length;
+
+    while (this.totalBytes > this.maxBytes && this.chunks.length > 0) {
+      const removed = this.chunks.shift();
+      if (!removed) {
+        break;
+      }
+      this.totalBytes -= removed.length;
+    }
+  }
+
+  drain(): Buffer {
+    const combined = Buffer.concat(this.chunks);
+    this.clear();
+    return combined;
+  }
+
+  get byteLength(): number {
+    return this.totalBytes;
+  }
+
+  clear(): void {
+    this.chunks = [];
+    this.totalBytes = 0;
+  }
+}

packages/server/src/server/voice/voice-turn-controller.test.ts

@@ -0,0 +1,281 @@
+import { EventEmitter } from "node:events";
+import { describe, expect, it, vi } from "vitest";
+import pino from "pino";
+
+import type {
+  TurnDetectionProvider,
+  TurnDetectionSession,
+} from "../speech/turn-detection-provider.js";
+import {
+  createVoiceTurnController,
+  type DetectedVoiceUtterance,
+} from "./voice-turn-controller.js";
+
+class FakeTurnDetectionSession
+  extends EventEmitter
+  implements TurnDetectionSession
+{
+  public readonly requiredSampleRate = 16000;
+  public readonly appendedChunks: Buffer[] = [];
+
+  async connect(): Promise<void> {}
+
+  appendPcm16(chunk: Buffer): void {
+    this.appendedChunks.push(chunk);
+  }
+
+  flush(): void {}
+  close(): void {}
+}
+
+function createFakeTurnDetectionProvider(
+  session: FakeTurnDetectionSession
+): TurnDetectionProvider {
+  return {
+    id: "local",
+    createSession() {
+      return session;
+    },
+  };
+}
+
+async function settleSerialQueue(): Promise<void> {
+  await Promise.resolve();
+  await Promise.resolve();
+  await Promise.resolve();
+}
+
+function createControllerHarness() {
+  const detector = new FakeTurnDetectionSession();
+  const onSpeechStarted = vi.fn(async () => {});
+  const onSpeechStopped = vi.fn(async () => {});
+  const submitUtterance = vi.fn(async (_utterance: DetectedVoiceUtterance) => {});
+  const onError = vi.fn();
+
+  const controller = createVoiceTurnController({
+    logger: pino({ level: "silent" }),
+    turnDetection: createFakeTurnDetectionProvider(detector),
+    prefixDurationMs: 100,
+    utteranceSink: {
+      submitUtterance,
+    },
+    callbacks: {
+      onSpeechStarted,
+      onSpeechStopped,
+      onError,
+    },
+  });
+
+  return {
+    controller,
+    detector,
+    onSpeechStarted,
+    onSpeechStopped,
+    submitUtterance,
+    onError,
+  };
+}
+
+function createDeferredPromise<T>() {
+  let resolve!: (value: T | PromiseLike<T>) => void;
+  let reject!: (reason?: unknown) => void;
+  const promise = new Promise<T>((res, rej) => {
+    resolve = res;
+    reject = rej;
+  });
+  return { promise, resolve, reject };
+}
+
+describe("voice turn controller", () => {
+  it("buffers audio before speech start and includes the prefix in the utterance", async () => {
+    const harness = createControllerHarness();
+
+    await harness.controller.start();
+    await harness.controller.appendClientChunk({
+      audioBase64: Buffer.from([1, 2, 3, 4]).toString("base64"),
+      format: "audio/pcm;rate=16000;bits=16",
+    });
+    await harness.controller.appendClientChunk({
+      audioBase64: Buffer.from([5, 6, 7, 8]).toString("base64"),
+      format: "audio/pcm;rate=16000;bits=16",
+    });
+
+    harness.detector.emit("speech_started");
+    await settleSerialQueue();
+
+    await harness.controller.appendClientChunk({
+      audioBase64: Buffer.from([9, 10, 11, 12]).toString("base64"),
+      format: "audio/pcm;rate=16000;bits=16",
+    });
+
+    harness.detector.emit("speech_stopped");
+    await settleSerialQueue();
+
+    expect(harness.onSpeechStarted).toHaveBeenCalledTimes(1);
+    expect(harness.onSpeechStopped).toHaveBeenCalledTimes(1);
+    expect(harness.submitUtterance).toHaveBeenCalledTimes(1);
+    expect(harness.submitUtterance).toHaveBeenCalledWith(
+      expect.objectContaining({
+        pcm16: Buffer.from([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]),
+        sampleRate: 16000,
+        format: "audio/pcm;rate=16000;bits=16",
+      })
+    );
+    expect(harness.onError).not.toHaveBeenCalled();
+  });
+
+  it("finalizes one utterance when speech stops", async () => {
+    const harness = createControllerHarness();
+
+    await harness.controller.start();
+    await harness.controller.appendClientChunk({
+      audioBase64: Buffer.from([1, 1, 1, 1]).toString("base64"),
+      format: "audio/pcm;rate=16000;bits=16",
+    });
+
+    harness.detector.emit("speech_started");
+    await settleSerialQueue();
+
+    await harness.controller.appendClientChunk({
+      audioBase64: Buffer.from([2, 2, 2, 2]).toString("base64"),
+      format: "audio/pcm;rate=16000;bits=16",
+    });
+
+    harness.detector.emit("speech_stopped");
+    await settleSerialQueue();
+    harness.detector.emit("speech_stopped");
+    await settleSerialQueue();
+
+    expect(harness.submitUtterance).toHaveBeenCalledTimes(1);
+    expect(harness.onSpeechStopped).toHaveBeenCalledTimes(1);
+    expect(harness.onError).not.toHaveBeenCalled();
+  });
+
+  it("does not barge in or emit an utterance on silence-only chunks", async () => {
+    const harness = createControllerHarness();
+
+    await harness.controller.start();
+    await harness.controller.appendClientChunk({
+      audioBase64: Buffer.from([0, 0, 0, 0]).toString("base64"),
+      format: "audio/pcm;rate=16000;bits=16",
+    });
+    await harness.controller.appendClientChunk({
+      audioBase64: Buffer.from([0, 0, 0, 0]).toString("base64"),
+      format: "audio/pcm;rate=16000;bits=16",
+    });
+    await settleSerialQueue();
+
+    expect(harness.detector.appendedChunks).toEqual([
+      Buffer.from([0, 0, 0, 0]),
+      Buffer.from([0, 0, 0, 0]),
+    ]);
+    expect(harness.onSpeechStarted).not.toHaveBeenCalled();
+    expect(harness.onSpeechStopped).not.toHaveBeenCalled();
+    expect(harness.submitUtterance).not.toHaveBeenCalled();
+    expect(harness.onError).not.toHaveBeenCalled();
+  });
+
+  it("retains a rolling prefix for rapid follow-up utterances", async () => {
+    const harness = createControllerHarness();
+
+    await harness.controller.start();
+    await harness.controller.appendClientChunk({
+      audioBase64: Buffer.from([1, 2, 3, 4]).toString("base64"),
+      format: "audio/pcm;rate=16000;bits=16",
+    });
+
+    harness.detector.emit("speech_started");
+    await settleSerialQueue();
+
+    await harness.controller.appendClientChunk({
+      audioBase64: Buffer.from([5, 6, 7, 8]).toString("base64"),
+      format: "audio/pcm;rate=16000;bits=16",
+    });
+
+    harness.detector.emit("speech_stopped");
+    await settleSerialQueue();
+
+    await harness.controller.appendClientChunk({
+      audioBase64: Buffer.from([9, 10, 11, 12]).toString("base64"),
+      format: "audio/pcm;rate=16000;bits=16",
+    });
+
+    harness.detector.emit("speech_started");
+    await settleSerialQueue();
+
+    await harness.controller.appendClientChunk({
+      audioBase64: Buffer.from([13, 14, 15, 16]).toString("base64"),
+      format: "audio/pcm;rate=16000;bits=16",
+    });
+
+    harness.detector.emit("speech_stopped");
+    await settleSerialQueue();
+
+    expect(harness.submitUtterance).toHaveBeenCalledTimes(2);
+    expect(harness.onSpeechStopped).toHaveBeenCalledTimes(2);
+    expect(harness.submitUtterance.mock.calls[0]?.[0]).toEqual(
+      expect.objectContaining({
+        pcm16: Buffer.from([1, 2, 3, 4, 5, 6, 7, 8]),
+      })
+    );
+    expect(harness.submitUtterance.mock.calls[1]?.[0]).toEqual(
+      expect.objectContaining({
+        pcm16: Buffer.from([5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]),
+      })
+    );
+  });
+
+  it("continues detecting speech while a previous utterance submission is still pending", async () => {
+    const deferred = createDeferredPromise<void>();
+    const detector = new FakeTurnDetectionSession();
+    const onSpeechStarted = vi.fn(async () => {});
+    const onSpeechStopped = vi.fn(async () => {});
+    const submitUtterance = vi.fn(async () => {
+      await deferred.promise;
+    });
+    const onError = vi.fn();
+
+    const controller = createVoiceTurnController({
+      logger: pino({ level: "silent" }),
+      turnDetection: createFakeTurnDetectionProvider(detector),
+      prefixDurationMs: 100,
+      utteranceSink: {
+        submitUtterance,
+      },
+      callbacks: {
+        onSpeechStarted,
+        onSpeechStopped,
+        onError,
+      },
+    });
+
+    await controller.start();
+    await controller.appendClientChunk({
+      audioBase64: Buffer.from([1, 2, 3, 4]).toString("base64"),
+      format: "audio/pcm;rate=16000;bits=16",
+    });
+    detector.emit("speech_started");
+    await settleSerialQueue();
+    await controller.appendClientChunk({
+      audioBase64: Buffer.from([5, 6, 7, 8]).toString("base64"),
+      format: "audio/pcm;rate=16000;bits=16",
+    });
+    detector.emit("speech_stopped");
+    await settleSerialQueue();
+
+    await controller.appendClientChunk({
+      audioBase64: Buffer.from([9, 10, 11, 12]).toString("base64"),
+      format: "audio/pcm;rate=16000;bits=16",
+    });
+    detector.emit("speech_started");
+    await settleSerialQueue();
+
+    expect(onSpeechStarted).toHaveBeenCalledTimes(2);
+    expect(onSpeechStopped).toHaveBeenCalledTimes(1);
+    expect(submitUtterance).toHaveBeenCalledTimes(1);
+    expect(onError).not.toHaveBeenCalled();
+
+    deferred.resolve();
+    await settleSerialQueue();
+  });
+});

packages/server/src/server/voice/voice-turn-controller.ts

@@ -0,0 +1,226 @@
+import { Buffer } from "node:buffer";
+import type { Logger } from "pino";
+import { v4 as uuidv4 } from "uuid";
+
+import { Pcm16MonoResampler } from "../agent/pcm16-resampler.js";
+import { parsePcmRateFromFormat } from "../speech/audio.js";
+import type { TurnDetectionProvider } from "../speech/turn-detection-provider.js";
+import { FixedDurationPcmRingBuffer } from "./fixed-duration-pcm-ring-buffer.js";
+
+const PCM_CHANNELS = 1;
+const PCM_BITS_PER_SAMPLE = 16;
+const DEFAULT_PREFIX_DURATION_MS = 400;
+
+type VoiceInputState =
+  | { status: "idle" }
+  | { status: "listening"; rollingPrefixBytes: number }
+  | {
+      status: "capturing";
+      utteranceId: string;
+      startedAt: number;
+      rollingPrefixBytes: number;
+      utteranceBytes: number;
+    };
+
+export interface VoiceTurnControllerCallbacks {
+  onSpeechStarted(): Promise<void>;
+  onSpeechStopped(): Promise<void>;
+  onError(error: Error): void;
+}
+
+export interface DetectedVoiceUtterance {
+  pcm16: Buffer;
+  sampleRate: number;
+  format: string;
+  startedAt: number;
+  endedAt: number;
+}
+
+export interface VoiceUtteranceSink {
+  submitUtterance(utterance: DetectedVoiceUtterance): Promise<void>;
+}
+
+export interface VoiceTurnController {
+  start(): Promise<void>;
+  stop(): Promise<void>;
+  appendClientChunk(input: {
+    audioBase64: string;
+    format: string;
+  }): Promise<void>;
+}
+
+export function createVoiceTurnController(params: {
+  logger: Logger;
+  turnDetection: TurnDetectionProvider;
+  utteranceSink: VoiceUtteranceSink;
+  callbacks: VoiceTurnControllerCallbacks;
+  prefixDurationMs?: number;
+}): VoiceTurnController {
+  const detector = params.turnDetection.createSession({
+    logger: params.logger.child({ component: "turn-detection" }),
+  });
+  const prefixBuffer = new FixedDurationPcmRingBuffer({
+    sampleRate: detector.requiredSampleRate,
+    channels: PCM_CHANNELS,
+    bitsPerSample: PCM_BITS_PER_SAMPLE,
+    durationMs: params.prefixDurationMs ?? DEFAULT_PREFIX_DURATION_MS,
+  });
+
+  let state: VoiceInputState = { status: "idle" };
+  let resampler: Pcm16MonoResampler | null = null;
+  let inputRate = detector.requiredSampleRate;
+  let utteranceChunks: Buffer[] = [];
+  let queued = Promise.resolve();
+  let submissionQueue = Promise.resolve();
+
+  function buildVoicePcmFormat(sampleRate: number): string {
+    return `audio/pcm;rate=${sampleRate};bits=16`;
+  }
+
+  function fail(error: unknown): void {
+    params.callbacks.onError(error instanceof Error ? error : new Error(String(error)));
+  }
+
+  function runSerial(task: () => Promise<void>): Promise<void> {
+    queued = queued
+      .then(task)
+      .catch((error) => {
+        fail(error);
+      });
+    return queued;
+  }
+
+  function enqueueUtteranceSubmission(utterance: DetectedVoiceUtterance): void {
+    submissionQueue = submissionQueue
+      .then(async () => {
+        await params.utteranceSink.submitUtterance(utterance);
+      })
+      .catch((error) => {
+        fail(error);
+      });
+  }
+
+  async function handleSpeechStarted(): Promise<void> {
+    if (state.status === "capturing") {
+      return;
+    }
+
+    await params.callbacks.onSpeechStarted();
+
+    const prefix = prefixBuffer.drain();
+    const startedAt = Date.now();
+    utteranceChunks = prefix.length > 0 ? [prefix] : [];
+    state = {
+      status: "capturing",
+      utteranceId: uuidv4(),
+      startedAt,
+      rollingPrefixBytes: prefixBuffer.byteLength,
+      utteranceBytes: prefix.length,
+    };
+  }
+
+  async function handleSpeechStopped(): Promise<void> {
+    if (state.status !== "capturing") {
+      return;
+    }
+
+    const utterance = Buffer.concat(utteranceChunks);
+    const startedAt = state.startedAt;
+    const endedAt = Date.now();
+
+    utteranceChunks = [];
+    state = { status: "listening", rollingPrefixBytes: prefixBuffer.byteLength };
+
+    await params.callbacks.onSpeechStopped();
+
+    if (utterance.length === 0) {
+      return;
+    }
+
+    enqueueUtteranceSubmission({
+      pcm16: utterance,
+      sampleRate: detector.requiredSampleRate,
+      format: buildVoicePcmFormat(detector.requiredSampleRate),
+      startedAt,
+      endedAt,
+    });
+  }
+
+  detector.on("speech_started", () => {
+    void runSerial(handleSpeechStarted);
+  });
+  detector.on("speech_stopped", () => {
+    void runSerial(handleSpeechStopped);
+  });
+  detector.on("error", fail);
+
+  return {
+    async start(): Promise<void> {
+      await detector.connect();
+      state = { status: "listening", rollingPrefixBytes: prefixBuffer.byteLength };
+    },
+
+    async stop(): Promise<void> {
+      await runSerial(async () => {
+        detector.close();
+        prefixBuffer.clear();
+        utteranceChunks = [];
+        resampler?.reset();
+        resampler = null;
+        state = { status: "idle" };
+      });
+    },
+
+    async appendClientChunk(input): Promise<void> {
+      await runSerial(async () => {
+        if (state.status === "idle") {
+          return;
+        }
+
+        const pcm16 = Buffer.from(input.audioBase64, "base64");
+        if (pcm16.length === 0) {
+          return;
+        }
+
+        const parsedInputRate =
+          parsePcmRateFromFormat(input.format, detector.requiredSampleRate) ??
+          detector.requiredSampleRate;
+
+        if (parsedInputRate !== inputRate) {
+          inputRate = parsedInputRate;
+          resampler =
+            inputRate === detector.requiredSampleRate
+              ? null
+              : new Pcm16MonoResampler({
+                  inputRate,
+                  outputRate: detector.requiredSampleRate,
+                });
+        }
+
+        const normalized =
+          resampler === null ? pcm16 : resampler.processChunk(pcm16);
+        if (normalized.length === 0) {
+          return;
+        }
+
+        prefixBuffer.append(normalized);
+
+        if (state.status === "listening") {
+          state = {
+            status: "listening",
+            rollingPrefixBytes: prefixBuffer.byteLength,
+          };
+        } else if (state.status === "capturing") {
+          utteranceChunks.push(normalized);
+          state = {
+            ...state,
+            rollingPrefixBytes: prefixBuffer.byteLength,
+            utteranceBytes: state.utteranceBytes + normalized.length,
+          };
+        }
+
+        detector.appendPcm16(normalized);
+      });
+    },
+  };
+}

packages/server/src/server/websocket-server.ts

@@ -35,6 +35,7 @@ import type { AgentProviderRuntimeSettingsMap } from "./agent/provider-launch-co
 import { PushTokenStore } from "./push/token-store.js";
 import { PushService } from "./push/push-service.js";
 import type { SpeechToTextProvider, TextToSpeechProvider } from "./speech/speech-provider.js";
+import type { TurnDetectionProvider } from "./speech/turn-detection-provider.js";
 import type { Resolvable } from "./speech/provider-resolver.js";
 import type { SpeechReadinessSnapshot } from "./speech/speech-runtime.js";
 import type { LocalSpeechModelId } from "./speech/providers/local/models.js";
@@ -245,6 +246,7 @@ export class VoiceAssistantWebSocketServer {
   private readonly createAgentMcpTransport: AgentMcpTransportFactory;
   private readonly stt: Resolvable<SpeechToTextProvider | null>;
   private readonly tts: Resolvable<TextToSpeechProvider | null>;
+  private readonly turnDetection: Resolvable<TurnDetectionProvider | null>;
   private readonly terminalManager: TerminalManager | null;
   private readonly dictation: {
     finalTimeoutMs?: number;
@@ -301,6 +303,7 @@ export class VoiceAssistantWebSocketServer {
     createAgentMcpTransport: AgentMcpTransportFactory,
     wsConfig: WebSocketServerConfig,
     speech?: {
+      turnDetection: Resolvable<TurnDetectionProvider | null>;
       stt: Resolvable<SpeechToTextProvider | null>;
       tts: Resolvable<TextToSpeechProvider | null>;
     },
@@ -338,6 +341,7 @@ export class VoiceAssistantWebSocketServer {
     this.downloadTokenStore = downloadTokenStore;
     this.paseoHome = paseoHome;
     this.createAgentMcpTransport = createAgentMcpTransport;
+    this.turnDetection = speech?.turnDetection ?? null;
     this.stt = speech?.stt ?? null;
     this.tts = speech?.tts ?? null;
     this.terminalManager = terminalManager ?? null;
@@ -633,7 +637,10 @@ export class VoiceAssistantWebSocketServer {
       stt: this.stt,
       tts: this.tts,
       terminalManager: this.terminalManager,
-      voice: this.voice ?? undefined,
+      voice: {
+        ...(this.voice ?? {}),
+        turnDetection: this.turnDetection,
+      },
       voiceBridge: {
         registerVoiceSpeakHandler: (agentId, handler) => {
           this.voiceSpeakHandlers.set(agentId, handler);

packages/server/src/shared/messages.ts

@@ -1236,6 +1236,13 @@ export const TranscriptionResultMessageSchema = z.object({
   }),
 })
 
+export const VoiceInputStateMessageSchema = z.object({
+  type: z.literal('voice_input_state'),
+  payload: z.object({
+    isSpeaking: z.boolean(),
+  }),
+})
+
 export const DictationStreamAckMessageSchema = z.object({
   type: z.literal('dictation_stream_ack'),
   payload: z.object({
@@ -2159,6 +2166,7 @@ export const SessionOutboundMessageSchema = z.discriminatedUnion('type', [
   AssistantChunkMessageSchema,
   AudioOutputMessageSchema,
   TranscriptionResultMessageSchema,
+  VoiceInputStateMessageSchema,
   DictationStreamAckMessageSchema,
   DictationStreamFinishAcceptedMessageSchema,
   DictationStreamPartialMessageSchema,