jb/harness-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(dictate): support configurable ASR backends

Browse Source
This commit is contained in:
jb
2026-09-03 00:08:00 -04:00
parent 67877297d0
commit 2c3ee1f014
40 changed files with 7055 additions and 489 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/dictate/review-and-fix-plan.md
+6 -6
View File
@@ -1,10 +1,10 @@
# Dictate extension review and local Parakeet remediation plan
Status: implemented, with final host validation in progress. This document records the original review of `extensions/dictate/index.ts`, the agreed Linux implementation direction, and the acceptance criteria used for the replacement.
Status: historical local-backend review; final host validation completed. This document records the original review of `extensions/dictate/index.ts`, the agreed Linux implementation direction, and the acceptance criteria used for the local replacement. The current configurable local, Meta, and OpenAI scope supersedes its local-only decisions; see [`configurable-asr-backends-plan.md`](configurable-asr-backends-plan.md) and [`protocol.md`](protocol.md).
## Decisions
The replacement will target this machine and workflow:
The replacement will target this machine and workflow. The checked-in `profiles/nono/pi-local.json` is outside this change and remains byte-for-byte unchanged.
- **Platform:** Linux only, specifically CachyOS on x86-64 with niri and Ghostty.
- **Transcription:** local NVIDIA NeMo with `nvidia/parakeet-tdt-0.6b-v3` on the machine's RTX 4090. Deepgram and runtime cloud transcription are removed.
@@ -172,7 +172,7 @@ The service validates even-byte PCM length, converts little-endian signed 16-bit
Running capture and NeMo inside Pi would require PipeWire, model-directory, native-library, CUDA, and `/dev/nvidia*` access in the agent sandbox. Deepgram also requires cloud credentials and a network destination.
**Fix:** keep direct device and model capabilities in the host-side service. Add only `/run/user/1000/pi-dictate.sock` to `filesystem.unix_socket` in `profiles/nono/pi-local.json`. Do not add PipeWire configuration, `/run/user/1000/pipewire-0`, `/dev/snd`, `/dev/nvidia*`, model paths, Hugging Face cache paths, or `api.deepgram.com`.
**Fix:** keep direct device and model capabilities in the host-side service. Preserve the existing `/run/user/1000/pi-dictate.sock` policy entry and make no profile edits. Do not add PipeWire configuration, `/run/user/1000/pipewire-0`, `/dev/snd`, `/dev/nvidia*`, model paths, Hugging Face cache paths, or `api.deepgram.com`.
The socket is deliberately a mediated microphone and transcription capability, not a harmless health endpoint. Any process running under `pi-local` or a descendant profile can request recording and receive the resulting text. The user accepts that broad profile scope; document it prominently and keep the service single-user with socket mode `0600`.
@@ -375,10 +375,10 @@ Build a throwaway prototype outside the production extension and nono sandbox:
### Phase 4: Add the minimum nono capability
1. Add `/run/user/1000/pi-dictate.sock` to `filesystem.unix_socket` in `profiles/nono/pi-local.json`.
1. Preserve the existing `/run/user/1000/pi-dictate.sock` policy entry and make no profile edits.
2. Do not add PipeWire, GPU, model, cache, Deepgram, or extra network permissions.
3. Validate the draft and resolved policy according to the nono-profile workflow.
4. Compare the resolved policy before and after and confirm this feature adds only the exact service socket.
3. Validate the existing resolved policy separately.
4. Confirm this feature does not change the resolved Pi policy.
5. Restart Pi because an active nono sandbox cannot gain socket access in place.
6. Verify the extension can activate and use the service but still cannot connect directly to PipeWire or initialize NVML.