feat(dictate): support configurable ASR backends

This commit is contained in:
jb
2026-09-03 00:08:00 -04:00
parent 67877297d0
commit 2c3ee1f014
40 changed files with 7055 additions and 489 deletions
+11 -11
View File
@@ -1,6 +1,6 @@
# Configurable Dictate ASR backends
Status: proposed plan, 01SEP2026
Status: implemented and validated, 03SEP2026
## Decisions
@@ -43,7 +43,7 @@ Status: proposed plan, 01SEP2026
- `services/dictate/src/pi_dictate_service/model.py` supports one explicit `.nemo` archive, verifies its SHA-256, applies the coding dictionary, and runs offline on CUDA.
- `systemd/user/pi-dictate.service` denies all IP networking and limits address families to `AF_UNIX`. That must remain true in local mode but cannot be shared unchanged by cloud mode.
- `services/dictate/install.sh` always installs the full NeMo/CUDA environment and pinned model. Cloud-only installation should not require either.
- `profiles/nono/pi-local.json` already grants only the exact Dictate Unix socket and explicitly denies the installed local runtime. Add a deny for `~/.config/pi-dictate` so choosing that directory as `$WORKDIR` cannot expose provider keys; no new Pi-side network, credential, PipeWire, model, or device grant is required.
- `profiles/nono/pi-local.json` already grants the exact Dictate Unix socket and explicitly denies the installed local runtime. It is policy-controlled and must remain unchanged for this feature; no Pi-side network, credential, PipeWire, model, or device grant is required.
## External API contract baseline
@@ -92,11 +92,11 @@ Current documented shape:
- PCM is base64 encoded in bounded `input_audio_buffer.append` JSON events. The API's per-append limit is 15 MiB, but Dictate must use much smaller realtime chunks.
- Stop sends `input_audio_buffer.commit`; it starts transcription without creating an assistant response.
- Partial text arrives as `conversation.item.input_audio_transcription.delta`; final text arrives as `conversation.item.input_audio_transcription.completed` and is correlated by `item_id`. Transcription-specific failures use `conversation.item.input_audio_transcription.failed`; general protocol failures use `error`.
- `gpt-live-transcribe` is the recommended low-latency model and supports prompt, keywords, languages, and delay tuning.
- `gpt-live-transcribe` is the recommended low-latency model and supports prompt, keywords, languages, and bounded delay settings. A credentialed live spike on 03SEP2026 confirmed `delay: "low"`.
- `gpt-transcribe` can use the same Realtime session but begins transcription after commit and can return detected languages.
- `whisper-1` remains a file-transcription model for older integrations, translation, and timestamp use cases; its model page says realtime transcription is unsupported.
- OpenAI documents a 60-minute Realtime session. Dictate's own five-minute cap remains stricter.
- The current guide recommends `delay` for `gpt-live-transcribe`, while one API reference says it is supported only by `gpt-realtime-whisper`; Phase 0 must verify actual GA behavior and omit `delay` if the selected model rejects it.
- The current guide recommends `delay` for `gpt-live-transcribe`, while one API reference associates it with `gpt-realtime-whisper`. The live GA session accepted `low`; Dictate permits `minimal`, `low`, `medium`, `high`, and `xhigh`, and otherwise omits the field.
## Proposed host configuration
@@ -132,7 +132,7 @@ Illustrative schema (field names may be refined during implementation):
"openai": {
"model": "gpt-live-transcribe",
"apiKeyFile": "~/.config/pi-dictate/credentials/openai-api-key",
"delay": "medium",
"delay": "low",
"prompt": "Coding dictation",
"termsPath": "~/.local/share/pi-dictate/config/coding-terms.txt",
"languages": ["en"]
@@ -244,8 +244,8 @@ Add `backends/openai.py`:
- emit bounded `input_audio_buffer.append` events with base64 PCM, far below the provider's 15 MiB event limit;
- commit only after recorder EOF;
- correlate final completion to the committed item and ignore stale/unmatched events;
- accept configurable model, prompt, keywords, languages, and provider-supported delay values;
- default to `gpt-live-transcribe`; document and test `gpt-transcribe`'s commit-oriented behavior, and contract-test whether `delay` is accepted;
- accept configurable model, prompt, keywords, languages, and the live-verified bounded `delay` values; reject unknown values rather than silently omitting them.
- default to `gpt-live-transcribe`; document and test `gpt-transcribe`'s commit-oriented behavior. Omit `delay` when it is not configured so the provider default remains authoritative.
- map `conversation.item.input_audio_transcription.failed`, general `error` events, auth/rate limits, timeouts, malformed events, and premature closes to stable `openai_*` errors without logging request bodies or keys.
Do not build around the OpenAI SDK unless it materially simplifies Realtime event handling without hiding cancellation, queue bounds, or frame limits. A direct WebSocket adapter is likely smaller and easier to fixture-test.
@@ -298,14 +298,14 @@ install.sh check
Do not weaken local mode to make cloud mode work.
- Keep common hardening (`NoNewPrivileges`, private temp, core/swap limits, system protection, read-only home, and socket ownership) in the base service unit. Clear ambient `HTTP_PROXY`, `HTTPS_PROXY`, and `ALL_PROXY` variables unless proxy support is separately designed and reviewed.
- Keep common hardening (`NoNewPrivileges`, private temp, core/swap limits, system protection, a masked home with narrow read-only runtime/config/unit and PipeWire-socket binds, and socket ownership) in the base service unit. Bind only the selected cloud credential through its generated drop-in. Clear ambient `HTTP_PROXY`, `HTTPS_PROXY`, and `ALL_PROXY` variables unless proxy support is separately designed and reviewed.
- Generate/install an offline drop-in for local mode containing `RestrictAddressFamilies=AF_UNIX` and `IPAddressDeny=any`.
- Generate/install a cloud drop-in permitting only `AF_UNIX`, `AF_INET`, and `AF_INET6` and no inbound listener beyond the inherited Unix socket. DNS and outbound TLS must be tested under the final unit.
- Stamp the expected network mode in the generated unit environment. At startup, compare it with the selected backend and refuse to run on mismatch. This prevents a hand-edited local config from silently running under cloud-capable hardening, and a cloud backend from failing mysteriously under the offline profile.
- Restrict service reads to the installed runtime, selected model/dictionary paths, and `~/.config/pi-dictate` where practical.
- Document that cloud mode necessarily grants the host transcription process general outbound IP connectivity; systemd cannot safely express a stable hostname allowlist for changing provider addresses.
Update `profiles/nono/pi-local.json` only to deny `~/.config/pi-dictate`, mirroring the existing runtime deny. The resolved policy must add no capability: Pi still reaches only `/run/user/1000/pi-dictate.sock` and cannot read the host config or keys even if launched with that directory as `$WORKDIR`.
Preserve `profiles/nono/pi-local.json` byte-for-byte. The existing resolved policy must be reviewed separately; Pi must not read host config or keys.
## Implementation phases
@@ -359,8 +359,8 @@ Completion: OpenAI mode satisfies the same user-visible and cleanup contract as
1. Add locked local/cloud dependency profiles and backend-specific provisioning.
2. Add atomic install/configure/check operations and secure key copying.
3. Add local/cloud systemd hardening generation plus startup mismatch checks.
4. Following the repository's nono-profile workflow, draft and strictly validate the explicit deny for `~/.config/pi-dictate`, review the resolved policy, and obtain approval for the exact profile diff before copying it into `profiles/nono/pi-local.json`.
5. Verify the resolved profile adds no capability, then test fresh install, every switch direction, reinstall preservation, invalid config, missing/insecure key files, local network denial, cloud DNS/TLS, idle exit, reactivation, and uninstall.
4. Preserve `profiles/nono/pi-local.json` byte-for-byte and make no Pi policy changes.
5. Review the existing resolved profile separately, then test fresh install, every switch direction, reinstall preservation, invalid config, missing/insecure key files, local network denial, cloud DNS/TLS, idle exit, reactivation, and uninstall.
Completion: selecting a backend installs only its required runtime, applies matching hardening, and reactivates through the same Unix socket after restart.