feat(dictate): support configurable ASR backends
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# Configurable Dictate ASR backends
|
||||
|
||||
Status: proposed plan, 01SEP2026
|
||||
Status: implemented and validated, 03SEP2026
|
||||
|
||||
## Decisions
|
||||
|
||||
@@ -43,7 +43,7 @@ Status: proposed plan, 01SEP2026
|
||||
- `services/dictate/src/pi_dictate_service/model.py` supports one explicit `.nemo` archive, verifies its SHA-256, applies the coding dictionary, and runs offline on CUDA.
|
||||
- `systemd/user/pi-dictate.service` denies all IP networking and limits address families to `AF_UNIX`. That must remain true in local mode but cannot be shared unchanged by cloud mode.
|
||||
- `services/dictate/install.sh` always installs the full NeMo/CUDA environment and pinned model. Cloud-only installation should not require either.
|
||||
- `profiles/nono/pi-local.json` already grants only the exact Dictate Unix socket and explicitly denies the installed local runtime. Add a deny for `~/.config/pi-dictate` so choosing that directory as `$WORKDIR` cannot expose provider keys; no new Pi-side network, credential, PipeWire, model, or device grant is required.
|
||||
- `profiles/nono/pi-local.json` already grants the exact Dictate Unix socket and explicitly denies the installed local runtime. It is policy-controlled and must remain unchanged for this feature; no Pi-side network, credential, PipeWire, model, or device grant is required.
|
||||
|
||||
## External API contract baseline
|
||||
|
||||
@@ -92,11 +92,11 @@ Current documented shape:
|
||||
- PCM is base64 encoded in bounded `input_audio_buffer.append` JSON events. The API's per-append limit is 15 MiB, but Dictate must use much smaller realtime chunks.
|
||||
- Stop sends `input_audio_buffer.commit`; it starts transcription without creating an assistant response.
|
||||
- Partial text arrives as `conversation.item.input_audio_transcription.delta`; final text arrives as `conversation.item.input_audio_transcription.completed` and is correlated by `item_id`. Transcription-specific failures use `conversation.item.input_audio_transcription.failed`; general protocol failures use `error`.
|
||||
- `gpt-live-transcribe` is the recommended low-latency model and supports prompt, keywords, languages, and delay tuning.
|
||||
- `gpt-live-transcribe` is the recommended low-latency model and supports prompt, keywords, languages, and bounded delay settings. A credentialed live spike on 03SEP2026 confirmed `delay: "low"`.
|
||||
- `gpt-transcribe` can use the same Realtime session but begins transcription after commit and can return detected languages.
|
||||
- `whisper-1` remains a file-transcription model for older integrations, translation, and timestamp use cases; its model page says realtime transcription is unsupported.
|
||||
- OpenAI documents a 60-minute Realtime session. Dictate's own five-minute cap remains stricter.
|
||||
- The current guide recommends `delay` for `gpt-live-transcribe`, while one API reference says it is supported only by `gpt-realtime-whisper`; Phase 0 must verify actual GA behavior and omit `delay` if the selected model rejects it.
|
||||
- The current guide recommends `delay` for `gpt-live-transcribe`, while one API reference associates it with `gpt-realtime-whisper`. The live GA session accepted `low`; Dictate permits `minimal`, `low`, `medium`, `high`, and `xhigh`, and otherwise omits the field.
|
||||
|
||||
## Proposed host configuration
|
||||
|
||||
@@ -132,7 +132,7 @@ Illustrative schema (field names may be refined during implementation):
|
||||
"openai": {
|
||||
"model": "gpt-live-transcribe",
|
||||
"apiKeyFile": "~/.config/pi-dictate/credentials/openai-api-key",
|
||||
"delay": "medium",
|
||||
"delay": "low",
|
||||
"prompt": "Coding dictation",
|
||||
"termsPath": "~/.local/share/pi-dictate/config/coding-terms.txt",
|
||||
"languages": ["en"]
|
||||
@@ -244,8 +244,8 @@ Add `backends/openai.py`:
|
||||
- emit bounded `input_audio_buffer.append` events with base64 PCM, far below the provider's 15 MiB event limit;
|
||||
- commit only after recorder EOF;
|
||||
- correlate final completion to the committed item and ignore stale/unmatched events;
|
||||
- accept configurable model, prompt, keywords, languages, and provider-supported delay values;
|
||||
- default to `gpt-live-transcribe`; document and test `gpt-transcribe`'s commit-oriented behavior, and contract-test whether `delay` is accepted;
|
||||
- accept configurable model, prompt, keywords, languages, and the live-verified bounded `delay` values; reject unknown values rather than silently omitting them.
|
||||
- default to `gpt-live-transcribe`; document and test `gpt-transcribe`'s commit-oriented behavior. Omit `delay` when it is not configured so the provider default remains authoritative.
|
||||
- map `conversation.item.input_audio_transcription.failed`, general `error` events, auth/rate limits, timeouts, malformed events, and premature closes to stable `openai_*` errors without logging request bodies or keys.
|
||||
|
||||
Do not build around the OpenAI SDK unless it materially simplifies Realtime event handling without hiding cancellation, queue bounds, or frame limits. A direct WebSocket adapter is likely smaller and easier to fixture-test.
|
||||
@@ -298,14 +298,14 @@ install.sh check
|
||||
|
||||
Do not weaken local mode to make cloud mode work.
|
||||
|
||||
- Keep common hardening (`NoNewPrivileges`, private temp, core/swap limits, system protection, read-only home, and socket ownership) in the base service unit. Clear ambient `HTTP_PROXY`, `HTTPS_PROXY`, and `ALL_PROXY` variables unless proxy support is separately designed and reviewed.
|
||||
- Keep common hardening (`NoNewPrivileges`, private temp, core/swap limits, system protection, a masked home with narrow read-only runtime/config/unit and PipeWire-socket binds, and socket ownership) in the base service unit. Bind only the selected cloud credential through its generated drop-in. Clear ambient `HTTP_PROXY`, `HTTPS_PROXY`, and `ALL_PROXY` variables unless proxy support is separately designed and reviewed.
|
||||
- Generate/install an offline drop-in for local mode containing `RestrictAddressFamilies=AF_UNIX` and `IPAddressDeny=any`.
|
||||
- Generate/install a cloud drop-in permitting only `AF_UNIX`, `AF_INET`, and `AF_INET6` and no inbound listener beyond the inherited Unix socket. DNS and outbound TLS must be tested under the final unit.
|
||||
- Stamp the expected network mode in the generated unit environment. At startup, compare it with the selected backend and refuse to run on mismatch. This prevents a hand-edited local config from silently running under cloud-capable hardening, and a cloud backend from failing mysteriously under the offline profile.
|
||||
- Restrict service reads to the installed runtime, selected model/dictionary paths, and `~/.config/pi-dictate` where practical.
|
||||
- Document that cloud mode necessarily grants the host transcription process general outbound IP connectivity; systemd cannot safely express a stable hostname allowlist for changing provider addresses.
|
||||
|
||||
Update `profiles/nono/pi-local.json` only to deny `~/.config/pi-dictate`, mirroring the existing runtime deny. The resolved policy must add no capability: Pi still reaches only `/run/user/1000/pi-dictate.sock` and cannot read the host config or keys even if launched with that directory as `$WORKDIR`.
|
||||
Preserve `profiles/nono/pi-local.json` byte-for-byte. The existing resolved policy must be reviewed separately; Pi must not read host config or keys.
|
||||
|
||||
## Implementation phases
|
||||
|
||||
@@ -359,8 +359,8 @@ Completion: OpenAI mode satisfies the same user-visible and cleanup contract as
|
||||
1. Add locked local/cloud dependency profiles and backend-specific provisioning.
|
||||
2. Add atomic install/configure/check operations and secure key copying.
|
||||
3. Add local/cloud systemd hardening generation plus startup mismatch checks.
|
||||
4. Following the repository's nono-profile workflow, draft and strictly validate the explicit deny for `~/.config/pi-dictate`, review the resolved policy, and obtain approval for the exact profile diff before copying it into `profiles/nono/pi-local.json`.
|
||||
5. Verify the resolved profile adds no capability, then test fresh install, every switch direction, reinstall preservation, invalid config, missing/insecure key files, local network denial, cloud DNS/TLS, idle exit, reactivation, and uninstall.
|
||||
4. Preserve `profiles/nono/pi-local.json` byte-for-byte and make no Pi policy changes.
|
||||
5. Review the existing resolved profile separately, then test fresh install, every switch direction, reinstall preservation, invalid config, missing/insecure key files, local network denial, cloud DNS/TLS, idle exit, reactivation, and uninstall.
|
||||
|
||||
Completion: selecting a backend installs only its required runtime, applies matching hardening, and reactivates through the same Unix socket after restart.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user