feat(dictate): support configurable ASR backends

This commit is contained in:
jb
2026-09-03 00:08:00 -04:00
parent 67877297d0
commit 2c3ee1f014
40 changed files with 7055 additions and 489 deletions
+11 -11
View File
@@ -1,6 +1,6 @@
# Configurable Dictate ASR backends
Status: proposed plan, 01SEP2026
Status: implemented and validated, 03SEP2026
## Decisions
@@ -43,7 +43,7 @@ Status: proposed plan, 01SEP2026
- `services/dictate/src/pi_dictate_service/model.py` supports one explicit `.nemo` archive, verifies its SHA-256, applies the coding dictionary, and runs offline on CUDA.
- `systemd/user/pi-dictate.service` denies all IP networking and limits address families to `AF_UNIX`. That must remain true in local mode but cannot be shared unchanged by cloud mode.
- `services/dictate/install.sh` always installs the full NeMo/CUDA environment and pinned model. Cloud-only installation should not require either.
- `profiles/nono/pi-local.json` already grants only the exact Dictate Unix socket and explicitly denies the installed local runtime. Add a deny for `~/.config/pi-dictate` so choosing that directory as `$WORKDIR` cannot expose provider keys; no new Pi-side network, credential, PipeWire, model, or device grant is required.
- `profiles/nono/pi-local.json` already grants the exact Dictate Unix socket and explicitly denies the installed local runtime. It is policy-controlled and must remain unchanged for this feature; no Pi-side network, credential, PipeWire, model, or device grant is required.
## External API contract baseline
@@ -92,11 +92,11 @@ Current documented shape:
- PCM is base64 encoded in bounded `input_audio_buffer.append` JSON events. The API's per-append limit is 15 MiB, but Dictate must use much smaller realtime chunks.
- Stop sends `input_audio_buffer.commit`; it starts transcription without creating an assistant response.
- Partial text arrives as `conversation.item.input_audio_transcription.delta`; final text arrives as `conversation.item.input_audio_transcription.completed` and is correlated by `item_id`. Transcription-specific failures use `conversation.item.input_audio_transcription.failed`; general protocol failures use `error`.
- `gpt-live-transcribe` is the recommended low-latency model and supports prompt, keywords, languages, and delay tuning.
- `gpt-live-transcribe` is the recommended low-latency model and supports prompt, keywords, languages, and bounded delay settings. A credentialed live spike on 03SEP2026 confirmed `delay: "low"`.
- `gpt-transcribe` can use the same Realtime session but begins transcription after commit and can return detected languages.
- `whisper-1` remains a file-transcription model for older integrations, translation, and timestamp use cases; its model page says realtime transcription is unsupported.
- OpenAI documents a 60-minute Realtime session. Dictate's own five-minute cap remains stricter.
- The current guide recommends `delay` for `gpt-live-transcribe`, while one API reference says it is supported only by `gpt-realtime-whisper`; Phase 0 must verify actual GA behavior and omit `delay` if the selected model rejects it.
- The current guide recommends `delay` for `gpt-live-transcribe`, while one API reference associates it with `gpt-realtime-whisper`. The live GA session accepted `low`; Dictate permits `minimal`, `low`, `medium`, `high`, and `xhigh`, and otherwise omits the field.
## Proposed host configuration
@@ -132,7 +132,7 @@ Illustrative schema (field names may be refined during implementation):
"openai": {
"model": "gpt-live-transcribe",
"apiKeyFile": "~/.config/pi-dictate/credentials/openai-api-key",
"delay": "medium",
"delay": "low",
"prompt": "Coding dictation",
"termsPath": "~/.local/share/pi-dictate/config/coding-terms.txt",
"languages": ["en"]
@@ -244,8 +244,8 @@ Add `backends/openai.py`:
- emit bounded `input_audio_buffer.append` events with base64 PCM, far below the provider's 15 MiB event limit;
- commit only after recorder EOF;
- correlate final completion to the committed item and ignore stale/unmatched events;
- accept configurable model, prompt, keywords, languages, and provider-supported delay values;
- default to `gpt-live-transcribe`; document and test `gpt-transcribe`'s commit-oriented behavior, and contract-test whether `delay` is accepted;
- accept configurable model, prompt, keywords, languages, and the live-verified bounded `delay` values; reject unknown values rather than silently omitting them.
- default to `gpt-live-transcribe`; document and test `gpt-transcribe`'s commit-oriented behavior. Omit `delay` when it is not configured so the provider default remains authoritative.
- map `conversation.item.input_audio_transcription.failed`, general `error` events, auth/rate limits, timeouts, malformed events, and premature closes to stable `openai_*` errors without logging request bodies or keys.
Do not build around the OpenAI SDK unless it materially simplifies Realtime event handling without hiding cancellation, queue bounds, or frame limits. A direct WebSocket adapter is likely smaller and easier to fixture-test.
@@ -298,14 +298,14 @@ install.sh check
Do not weaken local mode to make cloud mode work.
- Keep common hardening (`NoNewPrivileges`, private temp, core/swap limits, system protection, read-only home, and socket ownership) in the base service unit. Clear ambient `HTTP_PROXY`, `HTTPS_PROXY`, and `ALL_PROXY` variables unless proxy support is separately designed and reviewed.
- Keep common hardening (`NoNewPrivileges`, private temp, core/swap limits, system protection, a masked home with narrow read-only runtime/config/unit and PipeWire-socket binds, and socket ownership) in the base service unit. Bind only the selected cloud credential through its generated drop-in. Clear ambient `HTTP_PROXY`, `HTTPS_PROXY`, and `ALL_PROXY` variables unless proxy support is separately designed and reviewed.
- Generate/install an offline drop-in for local mode containing `RestrictAddressFamilies=AF_UNIX` and `IPAddressDeny=any`.
- Generate/install a cloud drop-in permitting only `AF_UNIX`, `AF_INET`, and `AF_INET6` and no inbound listener beyond the inherited Unix socket. DNS and outbound TLS must be tested under the final unit.
- Stamp the expected network mode in the generated unit environment. At startup, compare it with the selected backend and refuse to run on mismatch. This prevents a hand-edited local config from silently running under cloud-capable hardening, and a cloud backend from failing mysteriously under the offline profile.
- Restrict service reads to the installed runtime, selected model/dictionary paths, and `~/.config/pi-dictate` where practical.
- Document that cloud mode necessarily grants the host transcription process general outbound IP connectivity; systemd cannot safely express a stable hostname allowlist for changing provider addresses.
Update `profiles/nono/pi-local.json` only to deny `~/.config/pi-dictate`, mirroring the existing runtime deny. The resolved policy must add no capability: Pi still reaches only `/run/user/1000/pi-dictate.sock` and cannot read the host config or keys even if launched with that directory as `$WORKDIR`.
Preserve `profiles/nono/pi-local.json` byte-for-byte. The existing resolved policy must be reviewed separately; Pi must not read host config or keys.
## Implementation phases
@@ -359,8 +359,8 @@ Completion: OpenAI mode satisfies the same user-visible and cleanup contract as
1. Add locked local/cloud dependency profiles and backend-specific provisioning.
2. Add atomic install/configure/check operations and secure key copying.
3. Add local/cloud systemd hardening generation plus startup mismatch checks.
4. Following the repository's nono-profile workflow, draft and strictly validate the explicit deny for `~/.config/pi-dictate`, review the resolved policy, and obtain approval for the exact profile diff before copying it into `profiles/nono/pi-local.json`.
5. Verify the resolved profile adds no capability, then test fresh install, every switch direction, reinstall preservation, invalid config, missing/insecure key files, local network denial, cloud DNS/TLS, idle exit, reactivation, and uninstall.
4. Preserve `profiles/nono/pi-local.json` byte-for-byte and make no Pi policy changes.
5. Review the existing resolved profile separately, then test fresh install, every switch direction, reinstall preservation, invalid config, missing/insecure key files, local network denial, cloud DNS/TLS, idle exit, reactivation, and uninstall.
Completion: selecting a backend installs only its required runtime, applies matching hardening, and reactivates through the same Unix socket after restart.
+26
View File
@@ -29,3 +29,29 @@ The complete transitive dependency versions and artifact hashes are recorded in
- NeMo ASR documentation: <https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/intro.html>
- CC BY 4.0: <https://creativecommons.org/licenses/by/4.0/>
- Apache License 2.0: <https://www.apache.org/licenses/LICENSE-2.0>
## Optional cloud providers
Cloud backends are opt-in and are not required for local mode. Pi Dictate does not redistribute Meta or OpenAI models and does not include either provider's SDK. The host service sends audio only after the selected provider session is authenticated and ready. Provider terms, retention, security, and billing policies apply to cloud audio and any keyword or prompt context sent with it.
### Meta Model API Muse Voice Transcribe
- Announcement: <https://research.meta.ai/blog/introducing-muse-voice-transcribe>
- Speech-to-text guide: <https://ai.developer.meta.com/docs/speech-to-text>
- Realtime Voice API: <https://ai.developer.meta.com/docs/api-reference/voice/realtime>
- Voice event schemas: <https://ai.developer.meta.com/docs/api-reference/voice/schemas>
- Authentication: <https://ai.developer.meta.com/docs/getting-started/authentication>
- Meta terms: <https://ai.meta.com/terms/>
Pi Dictate uses only the documented `api.meta.ai` realtime endpoint and `muse-voice-transcribe-1.0` contract. It does not use the undocumented `shortwave.facebook.com` Muse Code endpoint.
### OpenAI Realtime transcription
- Realtime transcription guide: <https://developers.openai.com/api/docs/guides/realtime-transcription>
- Realtime WebSocket guide: <https://developers.openai.com/api/docs/guides/realtime-websocket>
- Speech-to-text guide: <https://developers.openai.com/api/docs/guides/speech-to-text>
- `gpt-live-transcribe` model: <https://developers.openai.com/api/docs/models/gpt-live-transcribe>
- OpenAI terms: <https://openai.com/policies/terms-of-use/>
- OpenAI privacy policy: <https://openai.com/policies/privacy-policy/>
The OpenAI default is `gpt-live-transcribe`; `gpt-transcribe` is a configurable commit-oriented alternative. `whisper-1` is not treated as a Realtime model.
+15
View File
@@ -46,3 +46,18 @@ A 37-term dictionary and boost strengths from 0.25 through 2.0 were tested again
The packaging, latency, memory, offline, and normal-dictation gates pass. Alpha `1.0` produced the same expected clean transcript as baseline, the same degraded transcript on seeded noisy input, and an empty transcript for silence. No dictionary hallucination was observed.
The coding-term remediation passes the final focused live check through the installed socket-activated service. The spoken phrase “Run npm test, then git status.” produced exactly `Run npm test, then git status.` with native phrase boosting at alpha `1.0`.
The upgraded configurable service was revalidated on 03SEP2026. Immediate cancellation during preparation was delivery-neutral, a second generation started successfully, and a six-second end-to-end capture returned `Run npm test, then get status.` The wording variation does not replace the earlier accuracy fixture; it confirms the current installer, systemd sandbox, socket protocol, PipeWire capture, in-memory model path, and final-only delivery operate together. Local systemd policy exposed only the PipeWire runtime socket while retaining the masked home and IP-network denial.
## Configurable backend contract gate
Credentialed live contract checks passed on 03SEP2026 with a five-second, non-sensitive 24 kHz mono PCM microphone sample held only in memory. Dedicated mode-`0600` provider key files were read by the host-side adapters; no credential entered Pi, a command line, a URL, or captured output.
| Provider | Model | Result |
| --- | --- | --- |
| Meta | `muse-voice-transcribe-1.0` | The public `api.meta.ai` handshake accepted the session, streamed PCM, accepted `endStream`, and returned an explicit final transcript (`run npm test`). Total turn time was 30.5 seconds. |
| OpenAI | `gpt-live-transcribe` | The GA transcription session accepted 24 kHz PCM, explicit commit, and returned a correlated completion event (`Run npm test and`). Total turn time was 7.1 seconds. |
The first OpenAI spike exposed forward-compatible fields (`event_id` and `previous_item_id`) on the commit acknowledgement. The adapter and sanitized fixture now accept those bounded fields while still requiring a non-empty `item_id`. A separate credentialed session update confirmed that `gpt-live-transcribe` accepts transcription `delay: "low"`. Configuration therefore supports only the documented bounded values `minimal`, `low`, `medium`, `high`, and `xhigh`; omission keeps the provider default.
These checks validate the current wire contract and finalization path, not comparative transcription accuracy. The public contract references checked were the [Meta Realtime Voice API](https://ai.developer.meta.com/docs/api-reference/voice/realtime), [Meta event schemas](https://ai.developer.meta.com/docs/api-reference/voice/schemas), [OpenAI Realtime transcription guide](https://developers.openai.com/api/docs/guides/realtime-transcription), [OpenAI Realtime WebSocket guide](https://developers.openai.com/api/docs/guides/realtime-websocket), and [`gpt-live-transcribe` model page](https://developers.openai.com/api/docs/models/gpt-live-transcribe).
+72 -26
View File
@@ -1,28 +1,34 @@
# Pi Dictate local-service protocol
# Pi Dictate host-service protocol
## Transport and version
Protocol version `2` uses UTF-8, newline-delimited JSON over one persistent Unix stream socket. Each line contains one JSON object. The systemd socket unit owns the socket path and permissions. The service accepts the inherited listening file descriptor and does not bind or unlink the path.
Clients must send `hello` before any other request. A connection negotiates one version only. Coding terms are host-owned service configuration and never cross this protocol.
Clients must send `hello` before any other request. A connection negotiates one version only. The selected backend, provider credentials, coding dictionary, model paths, and provider wire events are host-service configuration. They never cross this protocol.
## Limits
The protocol is backend-neutral. The host service owns microphone capture and the selected transcription backend. In local mode it runs the pinned NeMo/Parakeet adapter offline. In Meta and OpenAI modes it owns the provider WebSocket, key, PCM stream, partial hypotheses, and finalization. The Pi extension sees only state, level, result, error, cancellation, and bounded diagnostics.
## Limits and validation
| Data | Limit |
| --- | ---: |
| JSON frame, excluding newline | 65,536 bytes |
| `requestId` or `generationId` | 128 UTF-8 bytes |
| `backend` or `model` diagnostic | 128 UTF-8 bytes |
| Final transcript | 32,768 UTF-8 bytes |
| Error detail | 2,048 UTF-8 bytes |
| Simultaneous client connections | 16 |
| Pending output per client | 65,536 bytes |
| Recorder stderr tail | 65,536 bytes |
| Audio-level event rate | 10 per second |
| Mono 16 kHz signed-16 PCM | 9,600,000 bytes, five minutes |
| Local retained PCM | 9,600,000 bytes, five minutes at 16 kHz |
| Cloud queued PCM | Fixed and bounded; the whole recording is never retained |
IDs are non-empty, opaque strings generated by the client. The service compares them only for request correlation, ownership, and stale-generation rejection. It does not interpret IDs as paths. Request objects reject unknown or missing fields.
Malformed JSON, a frame over the limit, or an unsupported version produces one bounded `error` frame and then closes the connection. The malformed-frame error uses an empty `requestId` when no client ID can be recovered. Schema, state, busy, and ownership errors leave the connection open.
`backend` and `model` are optional diagnostic strings on `pong`. When present, they are non-empty, bounded, and safe for one-line display. They identify only the selected backend and model. They must not contain paths, query strings, key material, credential state, provider payloads, or endpoint details. Older services that omit these fields remain protocol-compatible; clients display `unknown` rather than inferring a backend.
Malformed JSON, invalid UTF-8, a frame over the limit, or an unsupported version produces one bounded `error` frame and then closes the connection. The malformed-frame error uses an empty `requestId` when no client ID can be recovered. Schema, state, busy, and ownership errors leave the connection open.
## Client requests
@@ -36,7 +42,7 @@ All examples are one line when transmitted.
Exact fields: `version`, `type`, `requestId`.
A successful response is `pong` with the same `requestId`. `ready` reports whether the model is warm.
A successful response is `pong` with the same `requestId`. The response may include `service`, `ready`, `busy`, `backend`, and `model`. `ready` says whether the selected backend can begin immediately without preparation. The extension does not use `ready` for state transitions because a cold local model or a new cloud session may need preparation after a successful hello.
### `ping`
@@ -44,7 +50,9 @@ A successful response is `pong` with the same `requestId`. `ready` reports wheth
{"version":2,"type":"ping","requestId":"ping-1"}
```
Exact fields: `version`, `type`, `requestId`. `hello` is required first. The response is `pong` and may include `ready` and `busy`.
Exact fields: `version`, `type`, `requestId`. `hello` is required first. The response is `pong` and may include `ready`, `busy`, `backend`, and `model`.
The `/dictate-status` extension command uses a short-lived connection: it sends `hello`, waits for its `pong`, sends `ping`, reports the bounded backend/model fields from the ping response, and closes the connection. It never starts a generation or reveals configuration and credential state.
### `start`
@@ -52,7 +60,13 @@ Exact fields: `version`, `type`, `requestId`. `hello` is required first. The res
{"version":2,"type":"start","requestId":"start-1","generationId":"generation-1"}
```
Exact fields: `version`, `type`, `requestId`, `generationId`. The service accepts only one global generation. It first emits `state: loading` for a cold model or `state: listening` for a warm model. Recording starts only after model loading succeeds.
Exact fields: `version`, `type`, `requestId`, `generationId`. The service accepts only one global generation. `start` prepares the selected backend before capture:
- local: load or reuse the checksum-verified NeMo/Parakeet model;
- Meta: connect to the fixed public endpoint, authenticate with the handshake frame, and wait for the session acknowledgement;
- OpenAI: connect to the fixed Realtime transcription endpoint, wait for `session.created`, send session configuration, and wait for `session.updated`.
Recording begins only after preparation succeeds. The service declares the recorder rate internally: 16 kHz for local mode and 24 kHz for either cloud mode. A cold start emits `state: loading` while the extension displays `preparing transcription...`; it never displays `listening` before the recorder and selected backend are ready.
### `stop`
@@ -60,7 +74,7 @@ Exact fields: `version`, `type`, `requestId`, `generationId`. The service accept
{"version":2,"type":"stop","requestId":"stop-1","generationId":"generation-1"}
```
Exact fields: `version`, `type`, `requestId`, `generationId`. Only the owning connection may stop the active listening generation.
Exact fields: `version`, `type`, `requestId`, `generationId`. Only the owning connection may stop the active listening generation. The service marks the generation stopping, signals `pw-record`, drains stdout to EOF, waits for child close, and then finalizes the selected backend. Local mode converts the complete PCM buffer and runs one inference. Cloud mode sends the provider end/commit marker only after every recorder chunk has been enqueued, then waits for one explicit provider final event.
### `cancel`
@@ -68,7 +82,7 @@ Exact fields: `version`, `type`, `requestId`, `generationId`. Only the owning co
{"version":2,"type":"cancel","requestId":"cancel-1","generationId":"generation-1"}
```
Exact fields: `version`, `type`, `requestId`, `generationId`. A valid cancellation immediately emits `cancelled`. Cancellation tombstones the generation, so it can never emit a later `result` or begin recording after a cancelled load.
Exact fields: `version`, `type`, `requestId`, `generationId`. A valid cancellation immediately emits `cancelled`. Cancellation tombstones the generation, so it can never later emit a `result` or begin recording after cancelled preparation. The host closes a cloud provider turn, stops capture, joins workers with bounded timeouts, and discards retained or queued PCM as applicable.
## Service events
@@ -77,9 +91,11 @@ All service events contain `version`, `type`, and a bounded client-generated `re
### `pong`
```json
{"version":2,"type":"pong","requestId":"hello-1","service":"pi-dictate","ready":false}
{"version":2,"type":"pong","requestId":"hello-1","service":"pi-dictate","ready":false,"busy":false,"backend":"local","model":"nemo-parakeet"}
```
`service` is the fixed service identifier. `ready` and `busy` are bounded booleans. `backend` and `model` are optional bounded diagnostics. No event reports key presence, file paths, provider URLs, network state, or raw provider responses.
### `state`
```json
@@ -88,13 +104,15 @@ All service events contain `version`, `type`, and a bounded client-generated `re
`state` is one of `loading`, `listening`, `stopping`, `transcribing`, or `idle`.
The wire name `loading` is retained for compatibility. It means backend preparation and includes local model load, cloud connection/authentication, or session setup. The extension renders it as `preparing transcription...`, not `loading model...`.
### `level`
```json
{"version":2,"type":"level","requestId":"start-1","generationId":"generation-1","level":0.625}
```
While listening, the service emits at most ten `level` events per second. `level` is a finite number from `0` through `1`, derived from the recorder's current 100 ms RMS window on a logarithmic -60 dBFS to 0 dBFS scale. The event carries no PCM samples and stops before recorder shutdown begins.
While listening, the service emits at most ten `level` events per second. `level` is a finite number from `0` through `1`, derived from the recorder's current 100 ms RMS window on a logarithmic -60 dBFS to 0 dBFS scale. The event carries no PCM samples and stops before recorder shutdown begins. The recorder rate is selected by the host backend and is not exposed in the event.
### `result`
@@ -102,7 +120,7 @@ While listening, the service emits at most ten `level` events per second. `level
{"version":2,"type":"result","requestId":"stop-1","generationId":"generation-1","transcript":"example text"}
```
There is at most one result for a generation. An empty transcript is a successful result with `"transcript":""`.
There is at most one result for a generation. An empty transcript is a successful result with `"transcript":""`. Only an explicit final provider event may produce this event. Provider partials, cumulative hypotheses, endpointing, diarization, stale completions, or cleanup never produce a result.
### `cancelled`
@@ -110,7 +128,7 @@ There is at most one result for a generation. An empty transcript is a successfu
{"version":2,"type":"cancelled","requestId":"cancel-1","generationId":"generation-1"}
```
This is a terminal delivery guarantee: the generation cannot later produce a result.
This is a terminal delivery guarantee: the generation cannot later produce a result. The service may remain busy while a non-interruptible local inference or provider cleanup worker finishes.
### `error`
@@ -123,37 +141,65 @@ Stable codes include:
- framing and schema: `malformed_json`, `frame_too_large`, `unsupported_version`, `invalid_request`;
- session: `hello_required`, `busy`, `not_owner`, `generation_mismatch`, `invalid_state`;
- recorder and audio: `recorder_spawn`, `recorder_exit`, `recorder_timeout`, `duration_limit`, `invalid_audio`;
- model and output: `model_failure`, `transcript_too_large`.
- local preparation and output: `config_invalid`, `credentials_invalid`, `model_failure`, `transcript_too_large`;
- Meta provider failures: bounded `meta_auth`, `meta_rate_limit`, `meta_timeout`, `meta_protocol`, `meta_premature_close`, or another stable `meta_*` code;
- OpenAI provider failures: bounded `openai_auth`, `openai_rate_limit`, `openai_timeout`, `openai_protocol`, `openai_premature_close`, or another stable `openai_*` code.
Error details are bounded and redacted. They never include API keys, authorization headers, credential paths, request bodies, raw provider payloads, or raw audio. A provider's malformed, oversized, invalid-UTF-8, partial-only, premature-close, authentication, rate-limit, duration, and network failures are terminal for the active generation. The service discards the generation rather than returning a plausible incomplete transcript.
## Provider finalization contract
The extension does not receive provider frames, but the host service must enforce these rules before emitting a protocol result.
### Meta Model API Muse Voice Transcribe
The service uses `wss://api.meta.ai/v1/asr/realtime` with the documented first-frame bearer handshake for `muse-voice-transcribe-1.0`, `PCM_24KHZ`, `PUSH_TO_TALK`, and cumulative partials. It waits for the `sessionId` acknowledgement before capture, sends ordered binary PCM16 frames through a bounded queue, sends `{ "type": "endStream" }` after recorder EOF, and requires a `transcript` event with `final: true`. It consumes the normal `1000` close and maps close codes `1008`, `1011`, and `1013` to stable failures.
The service must use only the public endpoint. It must not use the undocumented `shortwave.facebook.com` Muse Code endpoint. See the [Meta API references](https://ai.developer.meta.com/docs/api-reference/voice/realtime), [event schemas](https://ai.developer.meta.com/docs/api-reference/voice/schemas), and [authentication guide](https://ai.developer.meta.com/docs/getting-started/authentication).
### OpenAI Realtime transcription
The service uses `wss://api.openai.com/v1/realtime?intent=transcription` with `Authorization: Bearer <key>`. It waits for `session.created`, sends the GA nested `session.update` with `type: "transcription"`, the configured model, 24 kHz mono PCM16, and `turn_detection: null`, then waits for `session.updated`. Audio uses bounded base64 `input_audio_buffer.append` events. Stop commits only after recorder EOF and correlates `conversation.item.input_audio_transcription.completed` to the committed item. Delta and unmatched events are ignored.
The default is `gpt-live-transcribe`. `gpt-transcribe` is the documented commit-oriented alternative. `whisper-1` is not silently used as a Realtime model. A credentialed live check on 03SEP2026 confirmed that `gpt-live-transcribe` accepts transcription `delay: "low"`; configuration permits `minimal`, `low`, `medium`, `high`, and `xhigh`, while omission keeps the provider default. See the [Realtime transcription guide](https://developers.openai.com/api/docs/guides/realtime-transcription), [WebSocket guide](https://developers.openai.com/api/docs/guides/realtime-websocket), and [`gpt-live-transcribe` model page](https://developers.openai.com/api/docs/models/gpt-live-transcribe).
## Lifecycle and ownership
| Current state | Request or event | Outcome |
| --- | --- | --- |
| no generation | owner `start` | `loading`, then `listening`; warm models may emit `listening` directly |
| no generation | owner `start` | `loading`, then `listening`; a prepared backend may emit `listening` directly |
| any active generation | another `start` | `busy`; connection stays open |
| `loading` | owner `cancel` | immediate `cancelled`; loading finishes in the worker but recording is suppressed |
| `loading` | owner `cancel` | immediate `cancelled`; preparation finishes in the worker but recording is suppressed |
| `listening` | recorder audio | rate-limited `level` events for the owning client |
| `listening` | owner `stop` | `stopping`; signal recorder, drain stdout to EOF, await child close, convert PCM, then `transcribing` |
| `listening` | owner `cancel` or disconnect | stop and drain recorder, discard audio, never infer |
| `listening` | owner `stop` | `stopping`; stop and drain recorder fully, then `transcribing` and one local/provider final result |
| `listening` | owner `cancel` or disconnect | stop and drain recorder, discard audio, never infer or finalize |
| `stopping` | owner `cancel` | immediate `cancelled`; completed audio is discarded |
| `transcribing` | owner `cancel` or disconnect | tombstone immediately; inference finishes but its result is discarded |
| `transcribing` | successful model result | one `result`, then `idle` |
| `transcribing` | owner `cancel` or disconnect | tombstone immediately; local inference/provider finalization finishes but its result is discarded |
| `transcribing` | successful explicit final | one `result`, then `idle` |
| any work state | terminal failure | one `error`, then `idle` |
| active generation | non-owner `stop` or `cancel` | `not_owner`; connection stays open |
| active generation | stale generation ID | `generation_mismatch`; connection stays open |
Cancelled model loading and inference are non-interruptible. The global service remains busy until the worker finishes even though `cancelled` was already emitted. Disconnecting a generation owner cancels capture or tombstones loading/inference under the same rules.
The service is globally single-generation. A cloud queue overflow, provider stall, or worker failure is terminal. PCM is never silently dropped, retried to another provider, written to disk, or exposed to Pi.
The recorder command is exactly:
The recorder command is selected by the host service and is otherwise fixed:
```text
pw-record --raw --format s16 --rate 16000 --channels 1 -
```
The service launches `pw-record` in its own process group and marks the generation stopping before SIGTERM. It continuously drains bounded stdout and stderr, waits a bounded time for pipe EOF and child close, and only then converts or transcribes. A shutdown timeout escalates to a process-group kill and fails the generation. PipeWire's exit code `1` is accepted only after this service intentionally signalled a still-running recorder and stderr remained empty; an already-exited process or stderr-bearing exit remains a failure. Raw audio and recorder logs are held only in bounded memory and are not written to disk.
The local adapter uses 16 kHz. Meta and OpenAI use 24 kHz. The service launches `pw-record` in its own process group, marks the generation stopping before SIGTERM, continuously drains bounded stdout and stderr, waits for pipe EOF and child close, and only then converts or finalizes. A shutdown timeout escalates to a process-group kill and fails the generation. The service creates no audio file.
Cancellation and disconnect are delivery tombstones. They stop capture, close a provider turn where present, release application-owned buffers, and suppress every later final event. Cleanup is best effort and cannot authorize editor delivery.
## Idle exit
A successfully parsed protocol request resets the 15-minute monotonic idle activity timer, including requests that later produce state or ownership errors. Accepting a socket, receiving malformed data, and disconnecting do not reset activity.
A successfully parsed protocol request resets the configured monotonic idle activity timer, including requests that later produce state or ownership errors. Accepting a socket, receiving malformed data, and disconnecting do not reset activity.
Idle exit is suspended while model loading, recording, recorder shutdown, PCM conversion, or inference is active. Once work becomes idle, the full timer starts again. Exiting closes accepted clients and the inherited listener but does not unlink the systemd-owned socket path.
Idle exit is suspended while backend preparation, recording, recorder shutdown, PCM conversion, local inference, cloud streaming, provider finalization, or worker cleanup is active. Once work becomes idle, the configured timer starts again. It is 15 minutes by default. Exiting closes accepted clients and the inherited listener but does not unlink the systemd-owned socket path. The service's single owned worker is daemonized so bounded core shutdown permits normal interpreter exit even if native code is stuck. Explicit systemd shutdown also has a five-second stop deadline.
## Configuration and security boundary
The service reads strict versioned host configuration at startup from `~/.config/pi-dictate/config.json` and reads only the selected backend's regular, mode-`0600`, service-owned credential file. Keys never appear in command lines, systemd environment variables, URLs, redirects, protocol frames, diagnostics, or logs. Pi has no configuration writer and no access to this directory.
Local hardening denies IP networking and keeps the pinned runtime offline. Cloud hardening permits only `AF_UNIX`, `AF_INET`, and `AF_INET6` needed for provider DNS/TLS, while the base unit denies `bind()` and `listen()` so the service cannot create an inbound TCP listener. The service rejects a selected backend when the stamped network mode does not match the installed unit. Ambient `HTTP_PROXY`, `HTTPS_PROXY`, and `ALL_PROXY` variables are cleared unless separately reviewed. The exact host install and switch operations are documented in [`extensions/dictate/README.md`](../../extensions/dictate/README.md).
+6 -6
View File
@@ -1,10 +1,10 @@
# Dictate extension review and local Parakeet remediation plan
Status: implemented, with final host validation in progress. This document records the original review of `extensions/dictate/index.ts`, the agreed Linux implementation direction, and the acceptance criteria used for the replacement.
Status: historical local-backend review; final host validation completed. This document records the original review of `extensions/dictate/index.ts`, the agreed Linux implementation direction, and the acceptance criteria used for the local replacement. The current configurable local, Meta, and OpenAI scope supersedes its local-only decisions; see [`configurable-asr-backends-plan.md`](configurable-asr-backends-plan.md) and [`protocol.md`](protocol.md).
## Decisions
The replacement will target this machine and workflow:
The replacement will target this machine and workflow. The checked-in `profiles/nono/pi-local.json` is outside this change and remains byte-for-byte unchanged.
- **Platform:** Linux only, specifically CachyOS on x86-64 with niri and Ghostty.
- **Transcription:** local NVIDIA NeMo with `nvidia/parakeet-tdt-0.6b-v3` on the machine's RTX 4090. Deepgram and runtime cloud transcription are removed.
@@ -172,7 +172,7 @@ The service validates even-byte PCM length, converts little-endian signed 16-bit
Running capture and NeMo inside Pi would require PipeWire, model-directory, native-library, CUDA, and `/dev/nvidia*` access in the agent sandbox. Deepgram also requires cloud credentials and a network destination.
**Fix:** keep direct device and model capabilities in the host-side service. Add only `/run/user/1000/pi-dictate.sock` to `filesystem.unix_socket` in `profiles/nono/pi-local.json`. Do not add PipeWire configuration, `/run/user/1000/pipewire-0`, `/dev/snd`, `/dev/nvidia*`, model paths, Hugging Face cache paths, or `api.deepgram.com`.
**Fix:** keep direct device and model capabilities in the host-side service. Preserve the existing `/run/user/1000/pi-dictate.sock` policy entry and make no profile edits. Do not add PipeWire configuration, `/run/user/1000/pipewire-0`, `/dev/snd`, `/dev/nvidia*`, model paths, Hugging Face cache paths, or `api.deepgram.com`.
The socket is deliberately a mediated microphone and transcription capability, not a harmless health endpoint. Any process running under `pi-local` or a descendant profile can request recording and receive the resulting text. The user accepts that broad profile scope; document it prominently and keep the service single-user with socket mode `0600`.
@@ -375,10 +375,10 @@ Build a throwaway prototype outside the production extension and nono sandbox:
### Phase 4: Add the minimum nono capability
1. Add `/run/user/1000/pi-dictate.sock` to `filesystem.unix_socket` in `profiles/nono/pi-local.json`.
1. Preserve the existing `/run/user/1000/pi-dictate.sock` policy entry and make no profile edits.
2. Do not add PipeWire, GPU, model, cache, Deepgram, or extra network permissions.
3. Validate the draft and resolved policy according to the nono-profile workflow.
4. Compare the resolved policy before and after and confirm this feature adds only the exact service socket.
3. Validate the existing resolved policy separately.
4. Confirm this feature does not change the resolved Pi policy.
5. Restart Pi because an active nono sandbox cannot gain socket access in place.
6. Verify the extension can activate and use the service but still cannot connect directly to PipeWire or initialize NVML.