feat(dictate): support configurable ASR backends

This commit is contained in:
jb
2026-09-03 00:08:00 -04:00
parent 67877297d0
commit 2c3ee1f014
40 changed files with 7055 additions and 489 deletions
+167 -93
View File
@@ -1,79 +1,152 @@
# Pi Dictate
Linux-only push-to-dictate for Pi. Audio capture and NVIDIA Parakeet run in a socket-activated host service. The Pi extension receives state and finalized text through a Unix socket, then inserts successful text with `pasteToEditor()`.
Pi Dictate is a Linux-only push-to-dictate extension. The extension has no microphone, model, GPU, cloud, or credential access. It is a bounded JSON client for the host-owned `pi-dictate-service` Unix socket and pastes one finalized result into Pi's main editor.
Tested on CachyOS with niri, Ghostty, PipeWire, an RTX 4090, CUDA 13.0, NeMo 3.0.0, and Parakeet-TDT 0.6B v3.
The configured backend is local by default. Cloud backends are opt-in because their audio leaves the host and may incur provider charges.
## Backends
| Backend | Capture | Runtime | Default model and endpoint |
| --- | --- | --- | --- |
| `local` | 16 kHz mono PCM16 | Offline NeMo/Parakeet on the host GPU | `nemo-parakeet` and a pinned local archive |
| `meta` | 24 kHz mono PCM16 | Meta Model API | `muse-voice-transcribe-1.0` at `api.meta.ai` |
| `openai` | 24 kHz mono PCM16 | OpenAI Realtime transcription | `gpt-live-transcribe` at `api.openai.com` |
Only one backend is selected in host configuration. There is no per-session or per-generation picker, automatic failover, or submission of one recording to multiple providers. Switching backend is a privileged host operation and takes effect after configuration validation and a service restart.
OpenAI's `gpt-transcribe` is a supported commit-oriented alternative. `whisper-1` is not silently treated as a Realtime model and is not the default.
## Requirements
- Linux with PipeWire and `pw-record`
- systemd user services
- NVIDIA GPU with a working CUDA runtime
- `uv`
- Network access during initial dependency and model provisioning
All backends require:
Runtime is offline. The systemd service restricts networking and loads an explicit local model archive.
- Linux with PipeWire and `pw-record`;
- a systemd user session;
- `uv` for installation;
- the Pi profile's exact `/run/user/1000/pi-dictate.sock` access, unless the socket path is changed for another UID.
## Install
Local mode additionally requires an NVIDIA GPU with a working CUDA runtime and several gigabytes for the locked environment and model archive. Cloud-only mode does not install NeMo, CUDA, or the Parakeet archive, but it requires a provider key and outbound DNS/TLS access from the host service.
From the repository root:
## Install and configure
Run these commands from the repository root. The installer owns the installed runtime under `~/.local/share/pi-dictate/` and host configuration under `~/.config/pi-dictate/`.
Install the default offline backend:
```bash
./services/dictate/install.sh
./services/dictate/install.sh install --backend local
```
The installer creates `~/.local/share/pi-dictate/`, installs the locked service as a non-editable wheel, downloads and verifies the pinned model snapshot, seeds the coding dictionary, installs host-owned copies of the systemd user units, and enables `pi-dictate.socket`. The installed runtime does not depend on the repository remaining at its installation path.
To reuse an already downloaded model archive:
Install a cloud backend by copying a key from a file. The key is never accepted as a command-line literal:
```bash
./services/dictate/install.sh \
--model-source /path/to/parakeet-tdt-0.6b-v3.nemo
./services/dictate/install.sh install --backend meta --api-key-file /path/to/meta-key
./services/dictate/install.sh install --backend openai --api-key-file /path/to/openai-key
```
Preview without changing the host:
Configure an installed service or inspect the effective setup:
```bash
./services/dictate/install.sh --dry-run --model-source /path/to/model.nemo
./services/dictate/install.sh configure --backend local
./services/dictate/install.sh configure --backend meta --api-key-file /path/to/meta-key
./services/dictate/install.sh configure --backend openai --api-key-file /path/to/openai-key
./services/dictate/install.sh check
```
The pinned model revision and SHA-256 are recorded in [`../../docs/dictate/parakeet-attribution.md`](../../docs/dictate/parakeet-attribution.md).
`--api-key-file` must identify a regular file. The installer copies it into a service-owned mode-`0600` file and does not put the secret in systemd environment variables, process arguments, URLs, protocol frames, diagnostics, or logs. At startup the service reads only the selected backend's key. Inactive backend keys are neither required nor read.
## Sandbox access
The installed authority is:
`profiles/nono/pi-local.json` grants exactly:
```text
~/.config/pi-dictate/
config.json
credentials/
meta-api-key
openai-api-key
```
The service validates versioned JSON, selected-backend options, paths, numeric bounds, coding terms, and credential ownership and permissions before starting. Configuration changes require the configure operation and a validated service restart. Do not edit Pi extension source or try to write this directory from Pi.
Local model reuse remains checksum-gated. A custom archive must be accompanied by its explicit SHA-256 and a supported NeMo adapter:
```bash
MODEL=/path/to/parakeet-tdt-0.6b-v3.nemo
./services/dictate/install.sh install --backend local \
--model-source "$MODEL" \
--model-sha256 "$(sha256sum "$MODEL" | cut -d' ' -f1)"
```
The installer preserves valid existing model archives, dictionaries, configuration, and credentials on reinstall. Local and cloud modes use different systemd network hardening. Local mode denies IP networking; cloud mode permits IPv4/IPv6 for provider DNS and TLS while the base unit denies `bind()` and `listen()`, preventing a TCP listener. A backend/hardening mismatch fails closed. Cloud mode does not provide a stable hostname allowlist because provider addresses can change.
## Socket and sandbox boundary
The Pi profile grants access only to the exact service socket:
```text
/run/user/1000/pi-dictate.sock
```
This is a mediated microphone and transcription capability. Any code running under `pi-local` or a descendant profile can ask the host service to record and can receive the transcript. Pi still receives no PipeWire, `/dev/snd`, CUDA device, model, cache, or dictionary access. An explicit filesystem deny protects `~/.local/share/pi-dictate` even if that directory is selected as `$WORKDIR`.
This is a mediated microphone and transcription capability, not a health-only endpoint. Any code running under `pi-local` or a descendant profile can request a recording and receive its finalized transcript. Pi still has no PipeWire, `/dev/snd`, CUDA, model, dictionary, provider-key, or direct provider-network access. The checked-in profile policy is intentionally unchanged; review any local profile policy before exposing the service socket.
The active profile is repository-linked on the tested host. Start a new Pi sandbox after updating the profile because an existing nono process cannot gain the socket capability in place.
Restart Pi after changing the nono profile or socket exposure. An existing nono process cannot gain a new socket capability in place.
## Controls
## Controls and delivery
| Control | Action |
| --- | --- |
| `Alt+M` | Start recording when idle; stop and transcribe while listening |
| `Alt+N` | Cancel the active generation |
| `/dictate-cancel` | Cancel the active generation |
| `/dictate-status` | Open a short read-only hello/ping connection and show backend/model diagnostics |
Shortcuts are handled by Pi's main editor. They do not claim to work while a selector, confirmation, or custom modal owns input. If a modal opens after recording starts, a successful result still goes only to the main editor.
Shortcuts are registered through Pi's public main-editor APIs. They work while the main editor has focus, not while a selector, confirmation, or custom modal owns input. If a modal opens after recording starts, a successful result still goes only to the main editor.
The footer shows service state. While recording, a red circle and twelve-cell scrolling waveform make active capture visible and react to the microphone level:
The footer shows backend-neutral service states:
```text
preparing transcription...
● listening ▁▂▄▇█▆▃▂▅▇▄▂
stopping...
transcribing...
```
The circle uses Pi's semantic error color and the waveform uses its accent color. The other states remain `loading model...`, `stopping...`, and `transcribing...`. Audio levels are bounded scalar protocol events; raw audio remains inside the host service.
The waveform contains only bounded scalar level events. Raw audio, provider events, partial hypotheses, and credentials never cross the socket. Preparation includes local model load or cloud connection and authentication, so recording does not begin until the selected backend is ready. `/dictate-status` reports only bounded backend and model names. It never reports paths, endpoints with query data, key state, or provider payloads.
This indicator requires protocol version 2. Run the installer again after updating the repository so the installed host service and repository-linked extension upgrade together.
Only one explicit, bounded final transcript for the active generation can change the editor. Partial hypotheses are discarded at the service boundary. Empty speech reports `No speech recognized.` without changing the editor. Cancellation, stale generations, disconnects, provider failures, and cleanup never paste text.
Only a non-empty finalized result for the current generation changes the editor. Empty speech, cancellation, stale results, disconnects, and failures do not insert text.
## Configuration and coding terms
## Coding dictionary
The service reads host-owned configuration at process startup. A representative configuration is:
```json
{
"version": 1,
"backend": "openai",
"idleSeconds": 900,
"local": {
"adapter": "nemo-parakeet",
"modelPath": "~/.local/share/pi-dictate/models/parakeet.nemo",
"modelSha256": "...",
"termsPath": "~/.local/share/pi-dictate/config/coding-terms.txt",
"boostAlpha": 1.0
},
"meta": {
"model": "muse-voice-transcribe-1.0",
"apiKeyFile": "~/.config/pi-dictate/credentials/meta-api-key",
"termsPath": "~/.local/share/pi-dictate/config/coding-terms.txt"
},
"openai": {
"model": "gpt-live-transcribe",
"apiKeyFile": "~/.config/pi-dictate/credentials/openai-api-key",
"delay": "low",
"prompt": "Coding dictation",
"termsPath": "~/.local/share/pi-dictate/config/coding-terms.txt",
"languages": ["en"]
}
}
```
The exact defaults are installer-generated. Unknown keys, malformed JSON, unsupported versions, invalid bounds, incompatible options, missing selected keys, symlinks, multi-line credentials, and insecure credential permissions are rejected. OpenAI `delay` is optional; a credentialed live check confirmed `low`, and accepted values are `minimal`, `low`, `medium`, `high`, and `xhigh`. Omitting it keeps the provider default. Provider endpoints are fixed in production and are not configurable through Pi. The service clears ambient proxy variables unless an explicitly reviewed proxy feature is added.
The installer seeds:
@@ -81,58 +154,94 @@ The installer seeds:
~/.local/share/pi-dictate/config/coding-terms.txt
```
The host service reads this file at model load and applies NeMo's native phrase boosting. Pi cannot read or modify it. Add one term or phrase per line, use `#` for comments, and restart the service after editing:
Use one term or phrase per line and `#` for comments. In local mode, terms feed NeMo phrase boosting. In cloud mode, validated terms may be sent as provider keyword or language-bias context, so do not place secrets or sensitive phrases in the dictionary if they must remain on the host. The dictionary is bounded to 64 KiB, 512 distinct terms, and 128 UTF-8 bytes per term. Restart after an edit:
```bash
systemctl --user restart pi-dictate.service
```
Reinstalling preserves an existing dictionary. The file is bounded to 64 KiB, 512 distinct terms, and 128 UTF-8 bytes per term. A conservative boost of `1.0` is configured in `pi-dictate.service`; aggressive values can create false positives.
## Privacy, network, and provider policies
## Model lifecycle and measured behavior
Local mode captures and transcribes entirely on the host after installation. The runtime uses the pinned local model with offline settings and the systemd unit denies IP networking. No audio file is created. Application-owned buffers are bounded and released after transcription, cancellation, or failure. `LimitCORE=0` and `MemorySwapMax=0` reduce additional persistence where supported. This is best-effort non-persistence, not forensic zeroization: Python, NumPy, PyTorch, CUDA, drivers, host memory management, crash handling, and swap can retain copies outside the service's direct control.
The socket remains active while the Python service exits after 15 minutes of true inactivity. A later connection activates a fresh service. Loading, recording, recorder shutdown, and inference suspend idle exit.
In Meta or OpenAI mode, the service connects and authenticates before recording. It then streams bounded 24 kHz PCM directly to the selected provider while recording. Stop drains `pw-record` completely before sending the provider end/commit marker, waits for one explicit final event, and returns only that bounded text. Audio is not written to disk and is not sent to Pi. Provider partials and wire events remain in the host service. The provider's current retention, terms, security, and billing policies apply to cloud audio and any context sent with the session. Review the links below before opting in, and monitor the provider account for usage charges.
Measured on the tested RTX 4090 using the pinned offline model:
Provider references:
- Model archive: about 2.4 GB
- Locked Python environment: about 5.6 GB in the Phase 0 prototype
- Cold base-model load: about 6.4 seconds
- Production-adapter load with phrase-boost setup: about 10 seconds, including the private Triton cache build
- Ten-second microphone inference: about 0.29 seconds, 0.029 real-time factor
- Sixty-second repeated-speech inference: about 0.43 seconds, 0.0072 real-time factor
- Five-minute repeated-speech inference: about 0.82 seconds, 0.0027 real-time factor
- Peak CUDA allocation: about 5.1 GB for short clips and 6.15 GB for five minutes
- Five-minute raw PCM cap: 9.6 MB
- Meta announcement: <https://research.meta.ai/blog/introducing-muse-voice-transcribe>
- Meta speech-to-text guide: <https://ai.developer.meta.com/docs/speech-to-text>
- Meta Realtime Voice API: <https://ai.developer.meta.com/docs/api-reference/voice/realtime>
- Meta voice event schemas: <https://ai.developer.meta.com/docs/api-reference/voice/schemas>
- Meta authentication: <https://ai.developer.meta.com/docs/getting-started/authentication>
- Meta terms: <https://ai.meta.com/terms/>
- OpenAI Realtime transcription: <https://developers.openai.com/api/docs/guides/realtime-transcription>
- OpenAI Realtime WebSocket: <https://developers.openai.com/api/docs/guides/realtime-websocket>
- OpenAI speech-to-text: <https://developers.openai.com/api/docs/guides/speech-to-text>
- OpenAI `gpt-live-transcribe`: <https://developers.openai.com/api/docs/models/gpt-live-transcribe>
- OpenAI terms: <https://openai.com/policies/terms-of-use/>
- OpenAI privacy policy: <https://openai.com/policies/privacy-policy/>
The 60-second and five-minute stress inputs repeat a real speech clip in memory. They validate bounded duration, latency, and GPU allocation, not long-form accuracy.
The service uses only the documented public endpoints. It does not reuse Pi's provider auth store, put keys in URLs, follow credential-bearing redirects, or use the undocumented `shortwave.facebook.com` Muse Code endpoint.
## Privacy
## Local model, attribution, and measured behavior
The service passes this exact recorder command through pipes:
Local mode uses NVIDIA's `parakeet-tdt-0.6b-v3` through NeMo. The model archive is about 2.4 GB, and the locked Python environment was about 5.6 GB in the Phase 0 prototype. On the tested RTX 4090, cold preparation took about 10 seconds with phrase-boost setup, warm inference was comfortably faster than real time, and peak CUDA allocation was about 5.1 GB for short clips and 6.15 GB for five minutes. The service caps a local recording at five minutes, approximately 9.6 MB of 16 kHz mono PCM16. Cloud sessions use the same five-minute Dictate cap even though provider sessions can be longer.
```text
pw-record --raw --format s16 --rate 16000 --channels 1 -
```
See [`../../docs/dictate/parakeet-attribution.md`](../../docs/dictate/parakeet-attribution.md) for the pinned revision, SHA-256, CC BY 4.0 model terms, NeMo Apache 2.0 terms, and source links. The repository does not redistribute model weights.
It intentionally creates no audio files and releases application-owned buffers after transcription, cancellation, or failure. `LimitCORE=0` and `MemorySwapMax=0` reduce additional persistence. This is best-effort non-persistence, not forensic zeroization. Python, NumPy, PyTorch, CUDA, drivers, host memory management, and crash handling can retain copies outside the service's direct control.
The service remains warm for the configured idle period, 15 minutes by default, while no recording, inference, preparation worker, or protocol activity is active. Idle exit releases local model/GPU resources. A later socket connection activates a fresh service.
The service does not log raw audio or transcripts by default. Recorder stderr and protocol output are bounded. The server accepts at most 16 clients and bounds each client's pending output to 64 KiB.
## Service operations
## Service operations and troubleshooting
```bash
systemctl --user status pi-dictate.socket pi-dictate.service
journalctl --user -u pi-dictate.service
journalctl --user -u pi-dictate.service -n 100
systemctl --user restart pi-dictate.service
systemctl --user stop pi-dictate.service
./services/dictate/install.sh check
```
Stopping only the service leaves the socket active for later reactivation. Stopping `pi-dictate.socket` removes the socket path.
Stopping the service leaves the socket unit able to reactivate it. Stopping `pi-dictate.socket` removes the socket path. Service output is bounded and does not log keys, raw audio, transcripts, provider request bodies, or partial hypotheses by default.
### Socket or activation failure
```bash
ls -l "$XDG_RUNTIME_DIR/pi-dictate.sock"
systemctl --user status pi-dictate.socket pi-dictate.service
journalctl --user -u pi-dictate.service -n 100
```
The socket should be mode `0600` and owned by the current user. Run `/dictate-status` after a restart to confirm the selected backend and model. Restart Pi after a nono policy change.
### Local PipeWire, CUDA, or model failure
```bash
wpctl status
pw-record --raw --format s16 --rate 16000 --channels 1 - >/dev/null
nvidia-smi
```
Press `Ctrl+C` after checking capture. Confirm the model archive checksum and that the local systemd unit still denies IP networking. A local preparation failure must not start the recorder.
### Cloud key, DNS, TLS, or provider failure
Run `./services/dictate/install.sh check` and verify the selected credential file is a regular mode-`0600` file owned by the service user. Confirm the cloud drop-in is active, DNS works for the fixed provider endpoint, and the host can establish outbound TLS. Do not add a key to a command line or systemd environment. `meta_*` and `openai_*` service errors are intentionally bounded and redacted; provider request bodies are not a debugging log.
For an explicit live contract check, first select the cloud backend, then provide a non-sensitive raw 24 kHz mono PCM16 fixture of at most five seconds:
```bash
~/.local/share/pi-dictate/venv/bin/python -m pi_dictate_service.live_smoke /path/to/sample.pcm
```
This command immediately sends the sample to the configured provider and may incur charges. It reads only the managed selected credential, does not accept a literal key, rejects symlinked or oversized samples, and does not write audio.
### Busy, timeout, or duration errors
The service allows one generation globally. A cancelled preparation or inference worker can remain busy until its non-interruptible operation finishes. A stalled cloud stream, queue overflow, provider timeout, recorder failure, or five-minute limit discards the generation rather than returning a plausible incomplete transcript. Wait for it to settle or restart the service if the owning client crashed and the worker is stuck.
## niri and Ghostty
No compositor-global binding is required. Confirm the live niri config does not claim `Alt+M` or `Alt+N`, then validate it outside the Pi sandbox:
No compositor-global binding is required. Confirm the live niri configuration does not claim `Alt+M` or `Alt+N`, then validate it outside the Pi sandbox:
```bash
niri validate
@@ -140,45 +249,10 @@ niri validate
The shortcuts should reach Pi only while its main editor has focus. No `wl-copy`, `wtype`, `ydotool`, X11 injection, or compositor spawn binding is used.
## Troubleshooting
### Socket or activation failure
```bash
systemctl --user status pi-dictate.socket pi-dictate.service
ls -l "$XDG_RUNTIME_DIR/pi-dictate.sock"
journalctl --user -u pi-dictate.service -n 100
```
The socket should be mode `0600` and owned by the current user. Restart Pi after changing nono policy.
### PipeWire or microphone failure
```bash
wpctl status
pw-record --raw --format s16 --rate 16000 --channels 1 - >/dev/null
```
Press `Ctrl+C` after confirming capture. The service uses the WirePlumber default input. Selectable devices are not implemented.
### CUDA or model failure
```bash
nvidia-smi
systemctl --user restart pi-dictate.service
journalctl --user -u pi-dictate.service -n 100
```
Confirm the installed model checksum matches the attribution document and that runtime network access is not needed.
### Service reports busy
Only one recording, load worker, or inference worker is allowed globally. A cancelled load or inference remains busy until its non-interruptible worker finishes. Wait for it to settle or restart the service if the owning client crashed and the worker is stuck.
## Uninstall
```bash
./services/dictate/install.sh uninstall
```
The uninstaller disables the socket, removes only installer-managed unit copies, reloads systemd, and removes the fixed `~/.local/share/pi-dictate/` directory. It ignores install-root environment overrides and refuses to replace or remove unmanaged unit files.
Uninstall disables the socket, removes installer-managed unit copies, reloads systemd, and removes the installed runtime. It preserves `~/.config/pi-dictate/` and credentials by default. Use the installer's explicit, confirmed `--purge-config` path only when those credentials and settings should be removed. Uninstall does not remove unmanaged unit files.