feat(dictate): add voice-reactive listening waveform
This commit is contained in:
@@ -0,0 +1,27 @@
|
||||
# Stream bounded audio levels for the dictate indicator
|
||||
|
||||
Date: 26AUG2026
|
||||
Status: Accepted
|
||||
|
||||
## Context
|
||||
|
||||
Pi Dictate previously exposed only static service states. Active recording was easy to miss, and the extension had no signal from which to render a voice-reactive waveform. Raw microphone access must remain isolated in the host service.
|
||||
|
||||
## Decision
|
||||
|
||||
Protocol version 2 adds a `level` event while the service is listening. The recorder converts each 100 ms PCM window to a normalized logarithmic RMS level. The service sends the current scalar to the owning client at no more than 10 Hz.
|
||||
|
||||
The extension renders a fixed-width footer status:
|
||||
|
||||
```text
|
||||
● listening ▁▂▄▇█▆▃▂▅▇▄▂
|
||||
```
|
||||
|
||||
The circle uses Pi's semantic error color. Twelve block cells show recent levels as a scrolling history. Loading, stopping, and transcription remain static text.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The indicator reacts to speech without granting Pi microphone access or sending PCM over the socket.
|
||||
- Output remains bounded by the existing per-connection queue and frame limits.
|
||||
- The extension and installed service must upgrade together because protocol version 1 clients do not understand level events.
|
||||
- Recording causes at most ten footer renders and protocol events per second.
|
||||
+21
-11
@@ -2,7 +2,7 @@
|
||||
|
||||
## Transport and version
|
||||
|
||||
Protocol version `1` uses UTF-8, newline-delimited JSON over one persistent Unix stream socket. Each line contains one JSON object. The systemd socket unit owns the socket path and permissions. The service accepts the inherited listening file descriptor and does not bind or unlink the path.
|
||||
Protocol version `2` uses UTF-8, newline-delimited JSON over one persistent Unix stream socket. Each line contains one JSON object. The systemd socket unit owns the socket path and permissions. The service accepts the inherited listening file descriptor and does not bind or unlink the path.
|
||||
|
||||
Clients must send `hello` before any other request. A connection negotiates one version only. Coding terms are host-owned service configuration and never cross this protocol.
|
||||
|
||||
@@ -17,6 +17,7 @@ Clients must send `hello` before any other request. A connection negotiates one
|
||||
| Simultaneous client connections | 16 |
|
||||
| Pending output per client | 65,536 bytes |
|
||||
| Recorder stderr tail | 65,536 bytes |
|
||||
| Audio-level event rate | 10 per second |
|
||||
| Mono 16 kHz signed-16 PCM | 9,600,000 bytes, five minutes |
|
||||
|
||||
IDs are non-empty, opaque strings generated by the client. The service compares them only for request correlation, ownership, and stale-generation rejection. It does not interpret IDs as paths. Request objects reject unknown or missing fields.
|
||||
@@ -30,7 +31,7 @@ All examples are one line when transmitted.
|
||||
### `hello`
|
||||
|
||||
```json
|
||||
{"version":1,"type":"hello","requestId":"hello-1"}
|
||||
{"version":2,"type":"hello","requestId":"hello-1"}
|
||||
```
|
||||
|
||||
Exact fields: `version`, `type`, `requestId`.
|
||||
@@ -40,7 +41,7 @@ A successful response is `pong` with the same `requestId`. `ready` reports wheth
|
||||
### `ping`
|
||||
|
||||
```json
|
||||
{"version":1,"type":"ping","requestId":"ping-1"}
|
||||
{"version":2,"type":"ping","requestId":"ping-1"}
|
||||
```
|
||||
|
||||
Exact fields: `version`, `type`, `requestId`. `hello` is required first. The response is `pong` and may include `ready` and `busy`.
|
||||
@@ -48,7 +49,7 @@ Exact fields: `version`, `type`, `requestId`. `hello` is required first. The res
|
||||
### `start`
|
||||
|
||||
```json
|
||||
{"version":1,"type":"start","requestId":"start-1","generationId":"generation-1"}
|
||||
{"version":2,"type":"start","requestId":"start-1","generationId":"generation-1"}
|
||||
```
|
||||
|
||||
Exact fields: `version`, `type`, `requestId`, `generationId`. The service accepts only one global generation. It first emits `state: loading` for a cold model or `state: listening` for a warm model. Recording starts only after model loading succeeds.
|
||||
@@ -56,7 +57,7 @@ Exact fields: `version`, `type`, `requestId`, `generationId`. The service accept
|
||||
### `stop`
|
||||
|
||||
```json
|
||||
{"version":1,"type":"stop","requestId":"stop-1","generationId":"generation-1"}
|
||||
{"version":2,"type":"stop","requestId":"stop-1","generationId":"generation-1"}
|
||||
```
|
||||
|
||||
Exact fields: `version`, `type`, `requestId`, `generationId`. Only the owning connection may stop the active listening generation.
|
||||
@@ -64,7 +65,7 @@ Exact fields: `version`, `type`, `requestId`, `generationId`. Only the owning co
|
||||
### `cancel`
|
||||
|
||||
```json
|
||||
{"version":1,"type":"cancel","requestId":"cancel-1","generationId":"generation-1"}
|
||||
{"version":2,"type":"cancel","requestId":"cancel-1","generationId":"generation-1"}
|
||||
```
|
||||
|
||||
Exact fields: `version`, `type`, `requestId`, `generationId`. A valid cancellation immediately emits `cancelled`. Cancellation tombstones the generation, so it can never emit a later `result` or begin recording after a cancelled load.
|
||||
@@ -76,21 +77,29 @@ All service events contain `version`, `type`, and a bounded client-generated `re
|
||||
### `pong`
|
||||
|
||||
```json
|
||||
{"version":1,"type":"pong","requestId":"hello-1","service":"pi-dictate","ready":false}
|
||||
{"version":2,"type":"pong","requestId":"hello-1","service":"pi-dictate","ready":false}
|
||||
```
|
||||
|
||||
### `state`
|
||||
|
||||
```json
|
||||
{"version":1,"type":"state","requestId":"start-1","generationId":"generation-1","state":"loading"}
|
||||
{"version":2,"type":"state","requestId":"start-1","generationId":"generation-1","state":"loading"}
|
||||
```
|
||||
|
||||
`state` is one of `loading`, `listening`, `stopping`, `transcribing`, or `idle`.
|
||||
|
||||
### `level`
|
||||
|
||||
```json
|
||||
{"version":2,"type":"level","requestId":"start-1","generationId":"generation-1","level":0.625}
|
||||
```
|
||||
|
||||
While listening, the service emits at most ten `level` events per second. `level` is a finite number from `0` through `1`, derived from the recorder's current 100 ms RMS window on a logarithmic -60 dBFS to 0 dBFS scale. The event carries no PCM samples and stops before recorder shutdown begins.
|
||||
|
||||
### `result`
|
||||
|
||||
```json
|
||||
{"version":1,"type":"result","requestId":"stop-1","generationId":"generation-1","transcript":"example text"}
|
||||
{"version":2,"type":"result","requestId":"stop-1","generationId":"generation-1","transcript":"example text"}
|
||||
```
|
||||
|
||||
There is at most one result for a generation. An empty transcript is a successful result with `"transcript":""`.
|
||||
@@ -98,7 +107,7 @@ There is at most one result for a generation. An empty transcript is a successfu
|
||||
### `cancelled`
|
||||
|
||||
```json
|
||||
{"version":1,"type":"cancelled","requestId":"cancel-1","generationId":"generation-1"}
|
||||
{"version":2,"type":"cancelled","requestId":"cancel-1","generationId":"generation-1"}
|
||||
```
|
||||
|
||||
This is a terminal delivery guarantee: the generation cannot later produce a result.
|
||||
@@ -106,7 +115,7 @@ This is a terminal delivery guarantee: the generation cannot later produce a res
|
||||
### `error`
|
||||
|
||||
```json
|
||||
{"version":1,"type":"error","requestId":"stop-1","generationId":"generation-1","code":"recorder_timeout","detail":"pw-record did not stop after SIGTERM and was killed"}
|
||||
{"version":2,"type":"error","requestId":"stop-1","generationId":"generation-1","code":"recorder_timeout","detail":"pw-record did not stop after SIGTERM and was killed"}
|
||||
```
|
||||
|
||||
Stable codes include:
|
||||
@@ -123,6 +132,7 @@ Stable codes include:
|
||||
| no generation | owner `start` | `loading`, then `listening`; warm models may emit `listening` directly |
|
||||
| any active generation | another `start` | `busy`; connection stays open |
|
||||
| `loading` | owner `cancel` | immediate `cancelled`; loading finishes in the worker but recording is suppressed |
|
||||
| `listening` | recorder audio | rate-limited `level` events for the owning client |
|
||||
| `listening` | owner `stop` | `stopping`; signal recorder, drain stdout to EOF, await child close, convert PCM, then `transcribing` |
|
||||
| `listening` | owner `cancel` or disconnect | stop and drain recorder, discard audio, never infer |
|
||||
| `stopping` | owner `cancel` | immediate `cancelled`; completed audio is discarded |
|
||||
|
||||
@@ -186,6 +186,8 @@ The current meter updates every 60 ms even when its rendered value has not chang
|
||||
|
||||
**Fix:** the extension renders state changes reported by the service and does not receive raw audio levels. Use static status text for `loading model...`, `listening...`, `stopping...`, and `transcribing...`. If a meter is wanted later, add a rate-limited level event only after measuring the need.
|
||||
|
||||
**Superseded 26AUG2026:** active-listening feedback now needs a voice-reactive waveform. [The accepted follow-up decision](../adr/dictate/listening-level-events-26AUG2026.md) adds a bounded 10 Hz scalar level event without exposing raw audio to Pi.
|
||||
|
||||
### 12. Tests, installation, attribution, and user documentation are missing
|
||||
|
||||
**Severity:** medium
|
||||
|
||||
@@ -8,3 +8,20 @@ During Dictate Phase 0 on 26AUG2026, repeated overlapping throwaway benchmark pr
|
||||
2. A later production-adapter check raised `ValueError: bad marshal data (invalid reference)` while importing `torch._tensor`. An immediate standalone `import torch` succeeded, and the cached `_tensor.cpython-312.pyc` passed `marshal.loads()`.
|
||||
|
||||
The production service serializes model loading and inference and therefore does not intentionally launch concurrent NeMo processes. Before treating concurrent workers as supported, build a repeatable cold-load stress loop, monitor process and cgroup memory, disable bytecode writes, and determine whether the failures come from NeMo/OmegaConf resource pressure or concurrent access to the external environment's bytecode cache.
|
||||
|
||||
## Repair nono self-diagnosis after terminal path drift
|
||||
|
||||
During Dictate waveform testing on 26AUG2026, every Unix socket `bind()` in `services/dictate/tests/test_unix_server.py` failed with `EACCES` or `EPERM`, including when `TMPDIR` pointed inside the repository worktree. The required diagnostic also failed:
|
||||
|
||||
```text
|
||||
$ nono why --self --path /tmp --op readwrite
|
||||
nono: Configuration parse error: sandbox state path drifted at reload: serialized resolved=/dev/pts/15, actual resolved=/dev/null
|
||||
```
|
||||
|
||||
Reproduce in a non-interactive or reloaded Pi session, determine whether `/dev/pts/*` should be serialized by identity rather than resolved path, and make `nono why --self` report the active policy even after the controlling terminal changes. Also verify whether the Pi profile intentionally denies `AF_UNIX` socket creation in otherwise writable paths; the Dictate service loop tests need local throwaway sockets.
|
||||
|
||||
## Reuse the verified Dictate model during reinstall
|
||||
|
||||
The protocol-v2 reinstall on 26AUG2026 attempted to download the pinned model again even though the exact model already existed under `~/.local/share/pi-dictate/models/`. The sandbox correctly blocked unapproved access to `huggingface.co`, so installation had to be repeated with the installed model passed explicitly through `--model-source`.
|
||||
|
||||
Teach `services/dictate/install.sh` to checksum an existing model first and keep it when it matches `MODEL_SHA256`. Download only when the model is absent or invalid, while preserving the current atomic temporary-file replacement for new model sources.
|
||||
|
||||
Reference in New Issue
Block a user