259 lines
16 KiB
Markdown
259 lines
16 KiB
Markdown
# Pi Dictate
|
|
|
|
Pi Dictate is a Linux-only push-to-dictate extension. The extension has no microphone, model, GPU, cloud, or credential access. It is a bounded JSON client for the host-owned `pi-dictate-service` Unix socket and pastes one finalized result into Pi's main editor.
|
|
|
|
The configured backend is local by default. Cloud backends are opt-in because their audio leaves the host and may incur provider charges.
|
|
|
|
## Backends
|
|
|
|
| Backend | Capture | Runtime | Default model and endpoint |
|
|
| --- | --- | --- | --- |
|
|
| `local` | 16 kHz mono PCM16 | Offline NeMo/Parakeet on the host GPU | `nemo-parakeet` and a pinned local archive |
|
|
| `meta` | 24 kHz mono PCM16 | Meta Model API | `muse-voice-transcribe-1.0` at `api.meta.ai` |
|
|
| `openai` | 24 kHz mono PCM16 | OpenAI Realtime transcription | `gpt-live-transcribe` at `api.openai.com` |
|
|
|
|
Only one backend is selected in host configuration. There is no per-session or per-generation picker, automatic failover, or submission of one recording to multiple providers. Switching backend is a privileged host operation and takes effect after configuration validation and a service restart.
|
|
|
|
OpenAI's `gpt-transcribe` is a supported commit-oriented alternative. `whisper-1` is not silently treated as a Realtime model and is not the default.
|
|
|
|
## Requirements
|
|
|
|
All backends require:
|
|
|
|
- Linux with PipeWire and `pw-record`;
|
|
- a systemd user session;
|
|
- `uv` for installation;
|
|
- the Pi profile's exact `/run/user/1000/pi-dictate.sock` access, unless the socket path is changed for another UID.
|
|
|
|
Local mode additionally requires an NVIDIA GPU with a working CUDA runtime and several gigabytes for the locked environment and model archive. Cloud-only mode does not install NeMo, CUDA, or the Parakeet archive, but it requires a provider key and outbound DNS/TLS access from the host service.
|
|
|
|
## Install and configure
|
|
|
|
Run these commands from the repository root. The installer owns the installed runtime under `~/.local/share/pi-dictate/` and host configuration under `~/.config/pi-dictate/`.
|
|
|
|
Install the default offline backend:
|
|
|
|
```bash
|
|
./services/dictate/install.sh install --backend local
|
|
```
|
|
|
|
Install a cloud backend by copying a key from a file. The key is never accepted as a command-line literal:
|
|
|
|
```bash
|
|
./services/dictate/install.sh install --backend meta --api-key-file /path/to/meta-key
|
|
./services/dictate/install.sh install --backend openai --api-key-file /path/to/openai-key
|
|
```
|
|
|
|
Configure an installed service or inspect the effective setup:
|
|
|
|
```bash
|
|
./services/dictate/install.sh configure --backend local
|
|
./services/dictate/install.sh configure --backend meta --api-key-file /path/to/meta-key
|
|
./services/dictate/install.sh configure --backend openai --api-key-file /path/to/openai-key
|
|
./services/dictate/install.sh check
|
|
```
|
|
|
|
`--api-key-file` must identify a regular file. The installer copies it into a service-owned mode-`0600` file and does not put the secret in systemd environment variables, process arguments, URLs, protocol frames, diagnostics, or logs. At startup the service reads only the selected backend's key. Inactive backend keys are neither required nor read.
|
|
|
|
The installed authority is:
|
|
|
|
```text
|
|
~/.config/pi-dictate/
|
|
config.json
|
|
credentials/
|
|
meta-api-key
|
|
openai-api-key
|
|
```
|
|
|
|
The service validates versioned JSON, selected-backend options, paths, numeric bounds, coding terms, and credential ownership and permissions before starting. Configuration changes require the configure operation and a validated service restart. Do not edit Pi extension source or try to write this directory from Pi.
|
|
|
|
Local model reuse remains checksum-gated. A custom archive must be accompanied by its explicit SHA-256 and a supported NeMo adapter:
|
|
|
|
```bash
|
|
MODEL=/path/to/parakeet-tdt-0.6b-v3.nemo
|
|
./services/dictate/install.sh install --backend local \
|
|
--model-source "$MODEL" \
|
|
--model-sha256 "$(sha256sum "$MODEL" | cut -d' ' -f1)"
|
|
```
|
|
|
|
The installer preserves valid existing model archives, dictionaries, configuration, and credentials on reinstall. Local and cloud modes use different systemd network hardening. Local mode denies IP networking; cloud mode permits IPv4/IPv6 for provider DNS and TLS while the base unit denies `bind()` and `listen()`, preventing a TCP listener. A backend/hardening mismatch fails closed. Cloud mode does not provide a stable hostname allowlist because provider addresses can change.
|
|
|
|
## Socket and sandbox boundary
|
|
|
|
The Pi profile grants access only to the exact service socket:
|
|
|
|
```text
|
|
/run/user/1000/pi-dictate.sock
|
|
```
|
|
|
|
This is a mediated microphone and transcription capability, not a health-only endpoint. Any code running under `pi-local` or a descendant profile can request a recording and receive its finalized transcript. Pi still has no PipeWire, `/dev/snd`, CUDA, model, dictionary, provider-key, or direct provider-network access. The checked-in profile policy is intentionally unchanged; review any local profile policy before exposing the service socket.
|
|
|
|
Restart Pi after changing the nono profile or socket exposure. An existing nono process cannot gain a new socket capability in place.
|
|
|
|
## Controls and delivery
|
|
|
|
| Control | Action |
|
|
| --- | --- |
|
|
| `Alt+M` | Start recording when idle; stop and transcribe while listening |
|
|
| `Alt+N` | Cancel the active generation |
|
|
| `/dictate-cancel` | Cancel the active generation |
|
|
| `/dictate-status` | Open a short read-only hello/ping connection and show backend/model diagnostics |
|
|
|
|
Shortcuts are registered through Pi's public main-editor APIs. They work while the main editor has focus, not while a selector, confirmation, or custom modal owns input. If a modal opens after recording starts, a successful result still goes only to the main editor.
|
|
|
|
The footer shows backend-neutral service states:
|
|
|
|
```text
|
|
preparing transcription...
|
|
● listening ▁▂▄▇█▆▃▂▅▇▄▂
|
|
stopping...
|
|
transcribing...
|
|
```
|
|
|
|
The waveform contains only bounded scalar level events. Raw audio, provider events, partial hypotheses, and credentials never cross the socket. Preparation includes local model load or cloud connection and authentication, so recording does not begin until the selected backend is ready. `/dictate-status` reports only bounded backend and model names. It never reports paths, endpoints with query data, key state, or provider payloads.
|
|
|
|
Only one explicit, bounded final transcript for the active generation can change the editor. Partial hypotheses are discarded at the service boundary. Empty speech reports `No speech recognized.` without changing the editor. Cancellation, stale generations, disconnects, provider failures, and cleanup never paste text.
|
|
|
|
## Configuration and coding terms
|
|
|
|
The service reads host-owned configuration at process startup. A representative configuration is:
|
|
|
|
```json
|
|
{
|
|
"version": 1,
|
|
"backend": "openai",
|
|
"idleSeconds": 900,
|
|
"local": {
|
|
"adapter": "nemo-parakeet",
|
|
"modelPath": "~/.local/share/pi-dictate/models/parakeet.nemo",
|
|
"modelSha256": "...",
|
|
"termsPath": "~/.local/share/pi-dictate/config/coding-terms.txt",
|
|
"boostAlpha": 1.0
|
|
},
|
|
"meta": {
|
|
"model": "muse-voice-transcribe-1.0",
|
|
"apiKeyFile": "~/.config/pi-dictate/credentials/meta-api-key",
|
|
"termsPath": "~/.local/share/pi-dictate/config/coding-terms.txt"
|
|
},
|
|
"openai": {
|
|
"model": "gpt-live-transcribe",
|
|
"apiKeyFile": "~/.config/pi-dictate/credentials/openai-api-key",
|
|
"delay": "low",
|
|
"prompt": "Coding dictation",
|
|
"termsPath": "~/.local/share/pi-dictate/config/coding-terms.txt",
|
|
"languages": ["en"]
|
|
}
|
|
}
|
|
```
|
|
|
|
The exact defaults are installer-generated. Unknown keys, malformed JSON, unsupported versions, invalid bounds, incompatible options, missing selected keys, symlinks, multi-line credentials, and insecure credential permissions are rejected. OpenAI `delay` is optional; a credentialed live check confirmed `low`, and accepted values are `minimal`, `low`, `medium`, `high`, and `xhigh`. Omitting it keeps the provider default. Provider endpoints are fixed in production and are not configurable through Pi. The service clears ambient proxy variables unless an explicitly reviewed proxy feature is added.
|
|
|
|
The installer seeds:
|
|
|
|
```text
|
|
~/.local/share/pi-dictate/config/coding-terms.txt
|
|
```
|
|
|
|
Use one term or phrase per line and `#` for comments. In local mode, terms feed NeMo phrase boosting. In cloud mode, validated terms may be sent as provider keyword or language-bias context, so do not place secrets or sensitive phrases in the dictionary if they must remain on the host. The dictionary is bounded to 64 KiB, 512 distinct terms, and 128 UTF-8 bytes per term. Restart after an edit:
|
|
|
|
```bash
|
|
systemctl --user restart pi-dictate.service
|
|
```
|
|
|
|
## Privacy, network, and provider policies
|
|
|
|
Local mode captures and transcribes entirely on the host after installation. The runtime uses the pinned local model with offline settings and the systemd unit denies IP networking. No audio file is created. Application-owned buffers are bounded and released after transcription, cancellation, or failure. `LimitCORE=0` and `MemorySwapMax=0` reduce additional persistence where supported. This is best-effort non-persistence, not forensic zeroization: Python, NumPy, PyTorch, CUDA, drivers, host memory management, crash handling, and swap can retain copies outside the service's direct control.
|
|
|
|
In Meta or OpenAI mode, the service connects and authenticates before recording. It then streams bounded 24 kHz PCM directly to the selected provider while recording. Stop drains `pw-record` completely before sending the provider end/commit marker, waits for one explicit final event, and returns only that bounded text. Audio is not written to disk and is not sent to Pi. Provider partials and wire events remain in the host service. The provider's current retention, terms, security, and billing policies apply to cloud audio and any context sent with the session. Review the links below before opting in, and monitor the provider account for usage charges.
|
|
|
|
Provider references:
|
|
|
|
- Meta announcement: <https://research.meta.ai/blog/introducing-muse-voice-transcribe>
|
|
- Meta speech-to-text guide: <https://ai.developer.meta.com/docs/speech-to-text>
|
|
- Meta Realtime Voice API: <https://ai.developer.meta.com/docs/api-reference/voice/realtime>
|
|
- Meta voice event schemas: <https://ai.developer.meta.com/docs/api-reference/voice/schemas>
|
|
- Meta authentication: <https://ai.developer.meta.com/docs/getting-started/authentication>
|
|
- Meta terms: <https://ai.meta.com/terms/>
|
|
- OpenAI Realtime transcription: <https://developers.openai.com/api/docs/guides/realtime-transcription>
|
|
- OpenAI Realtime WebSocket: <https://developers.openai.com/api/docs/guides/realtime-websocket>
|
|
- OpenAI speech-to-text: <https://developers.openai.com/api/docs/guides/speech-to-text>
|
|
- OpenAI `gpt-live-transcribe`: <https://developers.openai.com/api/docs/models/gpt-live-transcribe>
|
|
- OpenAI terms: <https://openai.com/policies/terms-of-use/>
|
|
- OpenAI privacy policy: <https://openai.com/policies/privacy-policy/>
|
|
|
|
The service uses only the documented public endpoints. It does not reuse Pi's provider auth store, put keys in URLs, follow credential-bearing redirects, or use the undocumented `shortwave.facebook.com` Muse Code endpoint.
|
|
|
|
## Local model, attribution, and measured behavior
|
|
|
|
Local mode uses NVIDIA's `parakeet-tdt-0.6b-v3` through NeMo. The model archive is about 2.4 GB, and the locked Python environment was about 5.6 GB in the Phase 0 prototype. On the tested RTX 4090, cold preparation took about 10 seconds with phrase-boost setup, warm inference was comfortably faster than real time, and peak CUDA allocation was about 5.1 GB for short clips and 6.15 GB for five minutes. The service caps a local recording at five minutes, approximately 9.6 MB of 16 kHz mono PCM16. Cloud sessions use the same five-minute Dictate cap even though provider sessions can be longer.
|
|
|
|
See [`../../docs/dictate/parakeet-attribution.md`](../../docs/dictate/parakeet-attribution.md) for the pinned revision, SHA-256, CC BY 4.0 model terms, NeMo Apache 2.0 terms, and source links. The repository does not redistribute model weights.
|
|
|
|
The service remains warm for the configured idle period, 15 minutes by default, while no recording, inference, preparation worker, or protocol activity is active. Idle exit releases local model/GPU resources. A later socket connection activates a fresh service.
|
|
|
|
## Service operations and troubleshooting
|
|
|
|
```bash
|
|
systemctl --user status pi-dictate.socket pi-dictate.service
|
|
journalctl --user -u pi-dictate.service -n 100
|
|
systemctl --user restart pi-dictate.service
|
|
systemctl --user stop pi-dictate.service
|
|
./services/dictate/install.sh check
|
|
```
|
|
|
|
Stopping the service leaves the socket unit able to reactivate it. Stopping `pi-dictate.socket` removes the socket path. Service output is bounded and does not log keys, raw audio, transcripts, provider request bodies, or partial hypotheses by default.
|
|
|
|
### Socket or activation failure
|
|
|
|
```bash
|
|
ls -l "$XDG_RUNTIME_DIR/pi-dictate.sock"
|
|
systemctl --user status pi-dictate.socket pi-dictate.service
|
|
journalctl --user -u pi-dictate.service -n 100
|
|
```
|
|
|
|
The socket should be mode `0600` and owned by the current user. Run `/dictate-status` after a restart to confirm the selected backend and model. Restart Pi after a nono policy change.
|
|
|
|
### Local PipeWire, CUDA, or model failure
|
|
|
|
```bash
|
|
wpctl status
|
|
pw-record --raw --format s16 --rate 16000 --channels 1 - >/dev/null
|
|
nvidia-smi
|
|
```
|
|
|
|
Press `Ctrl+C` after checking capture. Confirm the model archive checksum and that the local systemd unit still denies IP networking. A local preparation failure must not start the recorder.
|
|
|
|
### Cloud key, DNS, TLS, or provider failure
|
|
|
|
Run `./services/dictate/install.sh check` and verify the selected credential file is a regular mode-`0600` file owned by the service user. Confirm the cloud drop-in is active, DNS works for the fixed provider endpoint, and the host can establish outbound TLS. Do not add a key to a command line or systemd environment. `meta_*` and `openai_*` service errors are intentionally bounded and redacted; provider request bodies are not a debugging log.
|
|
|
|
For an explicit live contract check, first select the cloud backend, then provide a non-sensitive raw 24 kHz mono PCM16 fixture of at most five seconds:
|
|
|
|
```bash
|
|
~/.local/share/pi-dictate/venv/bin/python -m pi_dictate_service.live_smoke /path/to/sample.pcm
|
|
```
|
|
|
|
This command immediately sends the sample to the configured provider and may incur charges. It reads only the managed selected credential, does not accept a literal key, rejects symlinked or oversized samples, and does not write audio.
|
|
|
|
### Busy, timeout, or duration errors
|
|
|
|
The service allows one generation globally. A cancelled preparation or inference worker can remain busy until its non-interruptible operation finishes. A stalled cloud stream, queue overflow, provider timeout, recorder failure, or five-minute limit discards the generation rather than returning a plausible incomplete transcript. Wait for it to settle or restart the service if the owning client crashed and the worker is stuck.
|
|
|
|
## niri and Ghostty
|
|
|
|
No compositor-global binding is required. Confirm the live niri configuration does not claim `Alt+M` or `Alt+N`, then validate it outside the Pi sandbox:
|
|
|
|
```bash
|
|
niri validate
|
|
```
|
|
|
|
The shortcuts should reach Pi only while its main editor has focus. No `wl-copy`, `wtype`, `ydotool`, X11 injection, or compositor spawn binding is used.
|
|
|
|
## Uninstall
|
|
|
|
```bash
|
|
./services/dictate/install.sh uninstall
|
|
```
|
|
|
|
Uninstall disables the socket, removes installer-managed unit copies, reloads systemd, and removes the installed runtime. It preserves `~/.config/pi-dictate/` and credentials by default. Use the installer's explicit, confirmed `--purge-config` path only when those credentials and settings should be removed. Uninstall does not remove unmanaged unit files.
|