206 lines
16 KiB
Markdown
206 lines
16 KiB
Markdown
# Pi Dictate host-service protocol
|
|
|
|
## Transport and version
|
|
|
|
Protocol version `2` uses UTF-8, newline-delimited JSON over one persistent Unix stream socket. Each line contains one JSON object. The systemd socket unit owns the socket path and permissions. The service accepts the inherited listening file descriptor and does not bind or unlink the path.
|
|
|
|
Clients must send `hello` before any other request. A connection negotiates one version only. The selected backend, provider credentials, coding dictionary, model paths, and provider wire events are host-service configuration. They never cross this protocol.
|
|
|
|
The protocol is backend-neutral. The host service owns microphone capture and the selected transcription backend. In local mode it runs the pinned NeMo/Parakeet adapter offline. In Meta and OpenAI modes it owns the provider WebSocket, key, PCM stream, partial hypotheses, and finalization. The Pi extension sees only state, level, result, error, cancellation, and bounded diagnostics.
|
|
|
|
## Limits and validation
|
|
|
|
| Data | Limit |
|
|
| --- | ---: |
|
|
| JSON frame, excluding newline | 65,536 bytes |
|
|
| `requestId` or `generationId` | 128 UTF-8 bytes |
|
|
| `backend` or `model` diagnostic | 128 UTF-8 bytes |
|
|
| Final transcript | 32,768 UTF-8 bytes |
|
|
| Error detail | 2,048 UTF-8 bytes |
|
|
| Simultaneous client connections | 16 |
|
|
| Pending output per client | 65,536 bytes |
|
|
| Recorder stderr tail | 65,536 bytes |
|
|
| Audio-level event rate | 10 per second |
|
|
| Local retained PCM | 9,600,000 bytes, five minutes at 16 kHz |
|
|
| Cloud queued PCM | Fixed and bounded; the whole recording is never retained |
|
|
|
|
IDs are non-empty, opaque strings generated by the client. The service compares them only for request correlation, ownership, and stale-generation rejection. It does not interpret IDs as paths. Request objects reject unknown or missing fields.
|
|
|
|
`backend` and `model` are optional diagnostic strings on `pong`. When present, they are non-empty, bounded, and safe for one-line display. They identify only the selected backend and model. They must not contain paths, query strings, key material, credential state, provider payloads, or endpoint details. Older services that omit these fields remain protocol-compatible; clients display `unknown` rather than inferring a backend.
|
|
|
|
Malformed JSON, invalid UTF-8, a frame over the limit, or an unsupported version produces one bounded `error` frame and then closes the connection. The malformed-frame error uses an empty `requestId` when no client ID can be recovered. Schema, state, busy, and ownership errors leave the connection open.
|
|
|
|
## Client requests
|
|
|
|
All examples are one line when transmitted.
|
|
|
|
### `hello`
|
|
|
|
```json
|
|
{"version":2,"type":"hello","requestId":"hello-1"}
|
|
```
|
|
|
|
Exact fields: `version`, `type`, `requestId`.
|
|
|
|
A successful response is `pong` with the same `requestId`. The response may include `service`, `ready`, `busy`, `backend`, and `model`. `ready` says whether the selected backend can begin immediately without preparation. The extension does not use `ready` for state transitions because a cold local model or a new cloud session may need preparation after a successful hello.
|
|
|
|
### `ping`
|
|
|
|
```json
|
|
{"version":2,"type":"ping","requestId":"ping-1"}
|
|
```
|
|
|
|
Exact fields: `version`, `type`, `requestId`. `hello` is required first. The response is `pong` and may include `ready`, `busy`, `backend`, and `model`.
|
|
|
|
The `/dictate-status` extension command uses a short-lived connection: it sends `hello`, waits for its `pong`, sends `ping`, reports the bounded backend/model fields from the ping response, and closes the connection. It never starts a generation or reveals configuration and credential state.
|
|
|
|
### `start`
|
|
|
|
```json
|
|
{"version":2,"type":"start","requestId":"start-1","generationId":"generation-1"}
|
|
```
|
|
|
|
Exact fields: `version`, `type`, `requestId`, `generationId`. The service accepts only one global generation. `start` prepares the selected backend before capture:
|
|
|
|
- local: load or reuse the checksum-verified NeMo/Parakeet model;
|
|
- Meta: connect to the fixed public endpoint, authenticate with the handshake frame, and wait for the session acknowledgement;
|
|
- OpenAI: connect to the fixed Realtime transcription endpoint, wait for `session.created`, send session configuration, and wait for `session.updated`.
|
|
|
|
Recording begins only after preparation succeeds. The service declares the recorder rate internally: 16 kHz for local mode and 24 kHz for either cloud mode. A cold start emits `state: loading` while the extension displays `preparing transcription...`; it never displays `listening` before the recorder and selected backend are ready.
|
|
|
|
### `stop`
|
|
|
|
```json
|
|
{"version":2,"type":"stop","requestId":"stop-1","generationId":"generation-1"}
|
|
```
|
|
|
|
Exact fields: `version`, `type`, `requestId`, `generationId`. Only the owning connection may stop the active listening generation. The service marks the generation stopping, signals `pw-record`, drains stdout to EOF, waits for child close, and then finalizes the selected backend. Local mode converts the complete PCM buffer and runs one inference. Cloud mode sends the provider end/commit marker only after every recorder chunk has been enqueued, then waits for one explicit provider final event.
|
|
|
|
### `cancel`
|
|
|
|
```json
|
|
{"version":2,"type":"cancel","requestId":"cancel-1","generationId":"generation-1"}
|
|
```
|
|
|
|
Exact fields: `version`, `type`, `requestId`, `generationId`. A valid cancellation immediately emits `cancelled`. Cancellation tombstones the generation, so it can never later emit a `result` or begin recording after cancelled preparation. The host closes a cloud provider turn, stops capture, joins workers with bounded timeouts, and discards retained or queued PCM as applicable.
|
|
|
|
## Service events
|
|
|
|
All service events contain `version`, `type`, and a bounded client-generated `requestId`. Generation events also contain `generationId`.
|
|
|
|
### `pong`
|
|
|
|
```json
|
|
{"version":2,"type":"pong","requestId":"hello-1","service":"pi-dictate","ready":false,"busy":false,"backend":"local","model":"nemo-parakeet"}
|
|
```
|
|
|
|
`service` is the fixed service identifier. `ready` and `busy` are bounded booleans. `backend` and `model` are optional bounded diagnostics. No event reports key presence, file paths, provider URLs, network state, or raw provider responses.
|
|
|
|
### `state`
|
|
|
|
```json
|
|
{"version":2,"type":"state","requestId":"start-1","generationId":"generation-1","state":"loading"}
|
|
```
|
|
|
|
`state` is one of `loading`, `listening`, `stopping`, `transcribing`, or `idle`.
|
|
|
|
The wire name `loading` is retained for compatibility. It means backend preparation and includes local model load, cloud connection/authentication, or session setup. The extension renders it as `preparing transcription...`, not `loading model...`.
|
|
|
|
### `level`
|
|
|
|
```json
|
|
{"version":2,"type":"level","requestId":"start-1","generationId":"generation-1","level":0.625}
|
|
```
|
|
|
|
While listening, the service emits at most ten `level` events per second. `level` is a finite number from `0` through `1`, derived from the recorder's current 100 ms RMS window on a logarithmic -60 dBFS to 0 dBFS scale. The event carries no PCM samples and stops before recorder shutdown begins. The recorder rate is selected by the host backend and is not exposed in the event.
|
|
|
|
### `result`
|
|
|
|
```json
|
|
{"version":2,"type":"result","requestId":"stop-1","generationId":"generation-1","transcript":"example text"}
|
|
```
|
|
|
|
There is at most one result for a generation. An empty transcript is a successful result with `"transcript":""`. Only an explicit final provider event may produce this event. Provider partials, cumulative hypotheses, endpointing, diarization, stale completions, or cleanup never produce a result.
|
|
|
|
### `cancelled`
|
|
|
|
```json
|
|
{"version":2,"type":"cancelled","requestId":"cancel-1","generationId":"generation-1"}
|
|
```
|
|
|
|
This is a terminal delivery guarantee: the generation cannot later produce a result. The service may remain busy while a non-interruptible local inference or provider cleanup worker finishes.
|
|
|
|
### `error`
|
|
|
|
```json
|
|
{"version":2,"type":"error","requestId":"stop-1","generationId":"generation-1","code":"recorder_timeout","detail":"pw-record did not stop after SIGTERM and was killed"}
|
|
```
|
|
|
|
Stable codes include:
|
|
|
|
- framing and schema: `malformed_json`, `frame_too_large`, `unsupported_version`, `invalid_request`;
|
|
- session: `hello_required`, `busy`, `not_owner`, `generation_mismatch`, `invalid_state`;
|
|
- recorder and audio: `recorder_spawn`, `recorder_exit`, `recorder_timeout`, `duration_limit`, `invalid_audio`;
|
|
- local preparation and output: `config_invalid`, `credentials_invalid`, `model_failure`, `transcript_too_large`;
|
|
- Meta provider failures: bounded `meta_auth`, `meta_rate_limit`, `meta_timeout`, `meta_protocol`, `meta_premature_close`, or another stable `meta_*` code;
|
|
- OpenAI provider failures: bounded `openai_auth`, `openai_rate_limit`, `openai_timeout`, `openai_protocol`, `openai_premature_close`, or another stable `openai_*` code.
|
|
|
|
Error details are bounded and redacted. They never include API keys, authorization headers, credential paths, request bodies, raw provider payloads, or raw audio. A provider's malformed, oversized, invalid-UTF-8, partial-only, premature-close, authentication, rate-limit, duration, and network failures are terminal for the active generation. The service discards the generation rather than returning a plausible incomplete transcript.
|
|
|
|
## Provider finalization contract
|
|
|
|
The extension does not receive provider frames, but the host service must enforce these rules before emitting a protocol result.
|
|
|
|
### Meta Model API Muse Voice Transcribe
|
|
|
|
The service uses `wss://api.meta.ai/v1/asr/realtime` with the documented first-frame bearer handshake for `muse-voice-transcribe-1.0`, `PCM_24KHZ`, `PUSH_TO_TALK`, and cumulative partials. It waits for the `sessionId` acknowledgement before capture, sends ordered binary PCM16 frames through a bounded queue, sends `{ "type": "endStream" }` after recorder EOF, and requires a `transcript` event with `final: true`. It consumes the normal `1000` close and maps close codes `1008`, `1011`, and `1013` to stable failures.
|
|
|
|
The service must use only the public endpoint. It must not use the undocumented `shortwave.facebook.com` Muse Code endpoint. See the [Meta API references](https://ai.developer.meta.com/docs/api-reference/voice/realtime), [event schemas](https://ai.developer.meta.com/docs/api-reference/voice/schemas), and [authentication guide](https://ai.developer.meta.com/docs/getting-started/authentication).
|
|
|
|
### OpenAI Realtime transcription
|
|
|
|
The service uses `wss://api.openai.com/v1/realtime?intent=transcription` with `Authorization: Bearer <key>`. It waits for `session.created`, sends the GA nested `session.update` with `type: "transcription"`, the configured model, 24 kHz mono PCM16, and `turn_detection: null`, then waits for `session.updated`. Audio uses bounded base64 `input_audio_buffer.append` events. Stop commits only after recorder EOF and correlates `conversation.item.input_audio_transcription.completed` to the committed item. Delta and unmatched events are ignored.
|
|
|
|
The default is `gpt-live-transcribe`. `gpt-transcribe` is the documented commit-oriented alternative. `whisper-1` is not silently used as a Realtime model. A credentialed live check on 03SEP2026 confirmed that `gpt-live-transcribe` accepts transcription `delay: "low"`; configuration permits `minimal`, `low`, `medium`, `high`, and `xhigh`, while omission keeps the provider default. See the [Realtime transcription guide](https://developers.openai.com/api/docs/guides/realtime-transcription), [WebSocket guide](https://developers.openai.com/api/docs/guides/realtime-websocket), and [`gpt-live-transcribe` model page](https://developers.openai.com/api/docs/models/gpt-live-transcribe).
|
|
|
|
## Lifecycle and ownership
|
|
|
|
| Current state | Request or event | Outcome |
|
|
| --- | --- | --- |
|
|
| no generation | owner `start` | `loading`, then `listening`; a prepared backend may emit `listening` directly |
|
|
| any active generation | another `start` | `busy`; connection stays open |
|
|
| `loading` | owner `cancel` | immediate `cancelled`; preparation finishes in the worker but recording is suppressed |
|
|
| `listening` | recorder audio | rate-limited `level` events for the owning client |
|
|
| `listening` | owner `stop` | `stopping`; stop and drain recorder fully, then `transcribing` and one local/provider final result |
|
|
| `listening` | owner `cancel` or disconnect | stop and drain recorder, discard audio, never infer or finalize |
|
|
| `stopping` | owner `cancel` | immediate `cancelled`; completed audio is discarded |
|
|
| `transcribing` | owner `cancel` or disconnect | tombstone immediately; local inference/provider finalization finishes but its result is discarded |
|
|
| `transcribing` | successful explicit final | one `result`, then `idle` |
|
|
| any work state | terminal failure | one `error`, then `idle` |
|
|
| active generation | non-owner `stop` or `cancel` | `not_owner`; connection stays open |
|
|
| active generation | stale generation ID | `generation_mismatch`; connection stays open |
|
|
|
|
The service is globally single-generation. A cloud queue overflow, provider stall, or worker failure is terminal. PCM is never silently dropped, retried to another provider, written to disk, or exposed to Pi.
|
|
|
|
The recorder command is selected by the host service and is otherwise fixed:
|
|
|
|
```text
|
|
pw-record --raw --format s16 --rate 16000 --channels 1 -
|
|
```
|
|
|
|
The local adapter uses 16 kHz. Meta and OpenAI use 24 kHz. The service launches `pw-record` in its own process group, marks the generation stopping before SIGTERM, continuously drains bounded stdout and stderr, waits for pipe EOF and child close, and only then converts or finalizes. A shutdown timeout escalates to a process-group kill and fails the generation. The service creates no audio file.
|
|
|
|
Cancellation and disconnect are delivery tombstones. They stop capture, close a provider turn where present, release application-owned buffers, and suppress every later final event. Cleanup is best effort and cannot authorize editor delivery.
|
|
|
|
## Idle exit
|
|
|
|
A successfully parsed protocol request resets the configured monotonic idle activity timer, including requests that later produce state or ownership errors. Accepting a socket, receiving malformed data, and disconnecting do not reset activity.
|
|
|
|
Idle exit is suspended while backend preparation, recording, recorder shutdown, PCM conversion, local inference, cloud streaming, provider finalization, or worker cleanup is active. Once work becomes idle, the configured timer starts again. It is 15 minutes by default. Exiting closes accepted clients and the inherited listener but does not unlink the systemd-owned socket path. The service's single owned worker is daemonized so bounded core shutdown permits normal interpreter exit even if native code is stuck. Explicit systemd shutdown also has a five-second stop deadline.
|
|
|
|
## Configuration and security boundary
|
|
|
|
The service reads strict versioned host configuration at startup from `~/.config/pi-dictate/config.json` and reads only the selected backend's regular, mode-`0600`, service-owned credential file. Keys never appear in command lines, systemd environment variables, URLs, redirects, protocol frames, diagnostics, or logs. Pi has no configuration writer and no access to this directory.
|
|
|
|
Local hardening denies IP networking and keeps the pinned runtime offline. Cloud hardening permits only `AF_UNIX`, `AF_INET`, and `AF_INET6` needed for provider DNS/TLS, while the base unit denies `bind()` and `listen()` so the service cannot create an inbound TCP listener. The service rejects a selected backend when the stamped network mode does not match the installed unit. Ambient `HTTP_PROXY`, `HTTPS_PROXY`, and `ALL_PROXY` variables are cleared unless separately reviewed. The exact host install and switch operations are documented in [`extensions/dictate/README.md`](../../extensions/dictate/README.md).
|