|
|
|
@@ -0,0 +1,444 @@
|
|
|
|
|
# Dictate extension review and local Parakeet remediation plan
|
|
|
|
|
|
|
|
|
|
Status: implemented, with final host validation in progress. This document records the original review of `extensions/dictate/index.ts`, the agreed Linux implementation direction, and the acceptance criteria used for the replacement.
|
|
|
|
|
|
|
|
|
|
## Decisions
|
|
|
|
|
|
|
|
|
|
The replacement will target this machine and workflow:
|
|
|
|
|
|
|
|
|
|
- **Platform:** Linux only, specifically CachyOS on x86-64 with niri and Ghostty.
|
|
|
|
|
- **Transcription:** local NVIDIA NeMo with `nvidia/parakeet-tdt-0.6b-v3` on the machine's RTX 4090. Deepgram and runtime cloud transcription are removed.
|
|
|
|
|
- **Process boundary:** a Python transcription service runs outside Pi's nono sandbox. The Pi extension communicates with it only through a Unix-domain socket.
|
|
|
|
|
- **Microphone:** the local service owns PipeWire capture. Pi receives no PipeWire socket, `/dev/snd`, or GPU access.
|
|
|
|
|
- **Interaction:** push to start, push to stop, then transcribe the completed recording. Voice activity detection, partial hypotheses, and continuous dictation are out of scope.
|
|
|
|
|
- **Activation:** shortcuts work while Pi's main editor has focus. A compositor-global dictation service and shortcut handling inside Pi modal dialogs are out of scope.
|
|
|
|
|
- **Delivery:** completed text is inserted into Pi's main editor with the public `ctx.ui.pasteToEditor()` API. Do not inspect focused TUI components or synthesize input.
|
|
|
|
|
- **Service lifecycle:** a systemd user socket activates the service on first use. The service keeps the model warm for 15 minutes after the last activity, then exits and releases GPU memory.
|
|
|
|
|
- **Sandbox:** `profiles/nono/pi-local.json` gains access only to the exact transcription-service socket. It does not gain direct PipeWire, GPU, model-directory, or Deepgram access.
|
|
|
|
|
- **Mediated capability:** access to the socket is access to request microphone recording and receive transcripts. The user explicitly accepts that any code running in a profile that inherits `pi-local` can invoke this protocol.
|
|
|
|
|
- **Privacy:** the service intentionally creates no audio files and releases in-memory audio after transcription, cancellation, or failure. This is best-effort non-persistence, not a guarantee against swap, core dumps, driver copies, or forensic recovery.
|
|
|
|
|
|
|
|
|
|
These decisions isolate native audio and model execution from Pi, but the socket remains a mediated microphone capability. The agent sandbox cannot open PipeWire or CUDA devices directly; it can ask the host service to record through the documented protocol.
|
|
|
|
|
|
|
|
|
|
## Target environment
|
|
|
|
|
|
|
|
|
|
The following was observed on this machine during planning:
|
|
|
|
|
|
|
|
|
|
- CachyOS with `XDG_SESSION_TYPE=wayland` and `XDG_CURRENT_DESKTOP=niri`.
|
|
|
|
|
- niri `26.04` and Ghostty with `TERM=xterm-256color`.
|
|
|
|
|
- An NVIDIA GeForce RTX 4090 and 188 GiB of system memory.
|
|
|
|
|
- PipeWire `pw-record` is installed and linked against PipeWire `1.6.8`.
|
|
|
|
|
- `wl-copy` is installed, but clipboard delivery is excluded.
|
|
|
|
|
- SoX, `wtype`, and `ydotool` are not installed.
|
|
|
|
|
- The user is UID `1000`, with `XDG_RUNTIME_DIR=/run/user/1000`.
|
|
|
|
|
- The current Pi sandbox cannot initialize NVML or connect to PipeWire. That is acceptable because the service will run outside it.
|
|
|
|
|
- The installed niri default configuration uses `Mod+M` for maximizing a window. In a native niri session, `Mod` is Super, so this does not conflict with Pi's `Alt+M`. The user's live niri configuration must still be checked for custom `Alt+M` or `Alt+N` bindings.
|
|
|
|
|
|
|
|
|
|
## External runtime choice
|
|
|
|
|
|
|
|
|
|
NVIDIA NeMo is the canonical Python runtime for Parakeet. The selected v3 model is multilingual and published under CC-BY-4.0. NeMo is Apache-2.0. The implementation must retain model attribution, record the exact model revision, and comply with both licenses.
|
|
|
|
|
|
|
|
|
|
Parakeet v3 is not treated as a drop-in replacement for Deepgram Live. This design intentionally does not require stable partial transcripts or cache-aware streaming compatibility. The service records one bounded clip and invokes NeMo's offline transcription API after the user stops recording.
|
|
|
|
|
|
|
|
|
|
Primary references:
|
|
|
|
|
|
|
|
|
|
- NVIDIA Parakeet-TDT 0.6B v3 model card: <https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3>
|
|
|
|
|
- NVIDIA NeMo repository: <https://github.com/NVIDIA/NeMo>
|
|
|
|
|
- NeMo ASR documentation: <https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/intro.html>
|
|
|
|
|
|
|
|
|
|
The model and Python/CUDA dependency set are too large and volatile to approve from documentation alone. Phase 0 is therefore a throwaway prototype and benchmark gate. Do not alter the production extension until that gate passes.
|
|
|
|
|
|
|
|
|
|
## Review scope and verification
|
|
|
|
|
|
|
|
|
|
The review covered Pi and TUI integration, recorder and WebSocket lifecycle, transcript delivery, failure handling, Linux and Wayland portability, nono integration, test coverage, and repository typechecking.
|
|
|
|
|
|
|
|
|
|
Historical verification from the original review:
|
|
|
|
|
|
|
|
|
|
- `npm run typecheck` failed at `extensions/dictate/index.ts:386`.
|
|
|
|
|
- The extension had no automated tests.
|
|
|
|
|
|
|
|
|
|
Static revalidation before implementation confirmed:
|
|
|
|
|
|
|
|
|
|
- The original implementation defects were present in `extensions/dictate/index.ts`.
|
|
|
|
|
- Dictate had no automated test suite or README, and the root `npm test` command excluded it.
|
|
|
|
|
- `extensions/dictate/LICENSE` was already deleted in the working tree; the user approved preserving that deletion.
|
|
|
|
|
- `profiles/nono/pi-local.json` did not yet allow the planned local service socket.
|
|
|
|
|
- This historical review section describes the starting point, not the current implementation state.
|
|
|
|
|
|
|
|
|
|
## Findings
|
|
|
|
|
|
|
|
|
|
### 1. The clipboard fallback can crash Pi and does not work on Linux
|
|
|
|
|
|
|
|
|
|
**Severity:** high
|
|
|
|
|
|
|
|
|
|
`extensions/dictate/index.ts:263-268` starts `pbcopy` without an `error` listener and immediately reports success. Spawn failures arrive asynchronously, outside the surrounding `try/catch`, and an unhandled child-process error can terminate Pi. `pbcopy` is also macOS-only.
|
|
|
|
|
|
|
|
|
|
**Fix:** remove clipboard delivery completely. Insert successful transcripts through `ctx.ui.pasteToEditor()` and notify the user. This public API preserves the editor's cursor and collapsed-paste bookkeeping, unlike reconstructing editor contents through `getEditorText()` and `setEditorText()`.
|
|
|
|
|
|
|
|
|
|
### 2. Recording starts before the transcription backend is ready
|
|
|
|
|
|
|
|
|
|
**Severity:** high
|
|
|
|
|
|
|
|
|
|
The current recorder and listening UI start before Deepgram connects (`index.ts:300-387`). A quick stop or slow connection can discard speech while the UI claimed to be listening.
|
|
|
|
|
|
|
|
|
|
**Fix:** replace the cloud connection with a versioned local-service handshake. On a cold start, show `loading model...` while systemd activates the service and NeMo loads. Start `pw-record` only after the service reports ready, then show `listening...`. Stopping or cancelling during loading must prevent recorder startup even if the model finishes loading later.
|
|
|
|
|
|
|
|
|
|
### 3. The extension fails repository typechecking
|
|
|
|
|
|
|
|
|
|
**Severity:** high
|
|
|
|
|
|
|
|
|
|
`index.ts:386` passes a Node `Buffer<ArrayBufferLike>` directly to the DOM-compatible `WebSocket.send()` API, which TypeScript rejects.
|
|
|
|
|
|
|
|
|
|
**Fix:** delete the Deepgram WebSocket and PCM-send path. The extension exchanges bounded newline-delimited JSON frames with the local service and never handles audio buffers. Keep `npm run typecheck` as a mandatory gate.
|
|
|
|
|
|
|
|
|
|
### 4. Transcript targeting can misroute or lose text
|
|
|
|
|
|
|
|
|
|
**Severity:** medium
|
|
|
|
|
|
|
|
|
|
`resolveTarget()` accepts every focused component with `handleInput()` as typable (`index.ts:209-226`). `flush()` sends the transcript as one synthetic keyboard event and assumes delivery succeeded (`index.ts:247-262`). Selectors can ignore the string or interpret it as navigation.
|
|
|
|
|
|
|
|
|
|
**Fix:** delete focused-target resolution and synthetic input. Successful results always go to Pi's main editor through `pasteToEditor()`.
|
|
|
|
|
|
|
|
|
|
### 5. Dictation depends on private TUI state and input hooks
|
|
|
|
|
|
|
|
|
|
**Severity:** medium
|
|
|
|
|
|
|
|
|
|
The extension captures the concrete TUI through an invisible widget, stores it as `any`, reads private `focusedComponent`, and installs a global input listener (`index.ts:90,209-215,488-522`).
|
|
|
|
|
|
|
|
|
|
**Fix:** remove the invisible widget, TUI handle, global input listener, target adapter, and related `any` values. Register `Alt+M` and `Alt+N` with `pi.registerShortcut()` and add `/dictate-cancel`.
|
|
|
|
|
|
|
|
|
|
Installed Pi routes extension shortcuts through the main editor, not modal dialogs. The supported contract is therefore explicit:
|
|
|
|
|
|
|
|
|
|
- `Alt+M`, `Alt+N`, and `/dictate-cancel` work while the main editor has focus.
|
|
|
|
|
- They do not start or cancel dictation while a selector, confirmation, or custom modal owns input.
|
|
|
|
|
- If a modal opens after recording starts, a later successful result still goes to the main editor.
|
|
|
|
|
- The registered keys are reserved while the main editor is focused, including while dictate is idle.
|
|
|
|
|
|
|
|
|
|
### 6. Unexpected recorder exits can leave dictation active
|
|
|
|
|
|
|
|
|
|
**Severity:** medium
|
|
|
|
|
|
|
|
|
|
The current recorder exit handler cleans up only for non-null, nonzero codes (`index.ts:352-362`). Zero exits and signal exits can leave state active.
|
|
|
|
|
|
|
|
|
|
**Fix:** move recorder ownership into the local service. Every `pw-record` termination is classified as intentional or unexpected before handling its exit code and signal. The service captures a bounded stderr tail, emits one structured error, discards buffered audio, and returns to idle. A service crash or socket close makes the extension return to idle without crashing Pi.
|
|
|
|
|
|
|
|
|
|
### 7. Audio and transport buffering are unbounded
|
|
|
|
|
|
|
|
|
|
**Severity:** medium
|
|
|
|
|
|
|
|
|
|
The current extension does not inspect WebSocket backpressure (`index.ts:380-386`). A local service would introduce a different risk if it buffered microphone audio without a duration cap.
|
|
|
|
|
|
|
|
|
|
**Fix:** remove network audio transport and bound recording inside the service. Start with a five-minute maximum, equivalent to approximately 9.6 MB of mono 16 kHz signed 16-bit PCM. At the limit, stop capture, discard the clip, report a clear duration-limit error, and return to idle. Keep protocol frames and transcript size bounded independently.
|
|
|
|
|
|
|
|
|
|
### 8. Cleanup implicitly decides whether to deliver text
|
|
|
|
|
|
|
|
|
|
**Severity:** medium
|
|
|
|
|
|
|
|
|
|
Current `cleanup()` calls `flush()` before teardown, so recorder and socket failures can insert fragments without identifying them as incomplete (`index.ts:271-298,346-350,407-414`).
|
|
|
|
|
|
|
|
|
|
**Fix:** cleanup becomes idempotent and delivery-neutral. Because local Parakeet returns one offline result, terminal outcomes are simpler:
|
|
|
|
|
|
|
|
|
|
| Event | Outcome | Editor change |
|
|
|
|
|
| --- | --- | --- |
|
|
|
|
|
| Service reports a non-empty result for the active generation | finalized | Paste once into the main editor |
|
|
|
|
|
| Service reports an empty result | empty | None; notify that no speech was recognized |
|
|
|
|
|
| User cancels before inference | cancelled | None; service discards audio |
|
|
|
|
|
| User cancels during inference | cancelled | None; tombstone the generation and suppress its eventual result while the non-interruptible worker finishes |
|
|
|
|
|
| Recorder, model, protocol, service, or timeout failure | failed | None; service discards audio and reports the cause |
|
|
|
|
|
| Pi session shuts down | cancelled | None; best-effort cancel or tombstone, then close the socket |
|
|
|
|
|
| Result belongs to an old generation | stale | None; ignore it |
|
|
|
|
|
|
|
|
|
|
No path may deliver text from cleanup itself.
|
|
|
|
|
|
|
|
|
|
### 9. The recorder and setup instructions are macOS-specific
|
|
|
|
|
|
|
|
|
|
**Severity:** high for this machine
|
|
|
|
|
|
|
|
|
|
The extension spawns SoX's `rec` and recommends `brew install sox` (`index.ts:24,319-348`). SoX is absent on this CachyOS machine, while `pw-record` is installed.
|
|
|
|
|
|
|
|
|
|
**Fix:** the local service spawns exactly:
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
pw-record --raw --format s16 --rate 16000 --channels 1 -
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The service pipes stdout, bounds stderr, and lets PipeWire select the WirePlumber default input. It must set state to stopping before signaling the recorder, continue reading stdout until the stream ends, and wait for the child `close` event before decoding. A timeout escalates process termination and produces a failed outcome. This ordering prevents the final PCM chunks from being truncated.
|
|
|
|
|
|
|
|
|
|
The service validates even-byte PCM length, converts little-endian signed 16-bit samples to the float representation expected by NeMo, and never creates an audio file. A selectable microphone target is future work.
|
|
|
|
|
|
|
|
|
|
### 10. The current sandbox boundary is too broad for in-process transcription
|
|
|
|
|
|
|
|
|
|
**Severity:** high for this machine
|
|
|
|
|
|
|
|
|
|
Running capture and NeMo inside Pi would require PipeWire, model-directory, native-library, CUDA, and `/dev/nvidia*` access in the agent sandbox. Deepgram also requires cloud credentials and a network destination.
|
|
|
|
|
|
|
|
|
|
**Fix:** keep direct device and model capabilities in the host-side service. Add only `/run/user/1000/pi-dictate.sock` to `filesystem.unix_socket` in `profiles/nono/pi-local.json`. Do not add PipeWire configuration, `/run/user/1000/pipewire-0`, `/dev/snd`, `/dev/nvidia*`, model paths, Hugging Face cache paths, or `api.deepgram.com`.
|
|
|
|
|
|
|
|
|
|
The socket is deliberately a mediated microphone and transcription capability, not a harmless health endpoint. Any process running under `pi-local` or a descendant profile can request recording and receive the resulting text. The user accepts that broad profile scope; document it prominently and keep the service single-user with socket mode `0600`.
|
|
|
|
|
|
|
|
|
|
The resolved profile already contains inherited and local capabilities. Acceptance must verify that this feature adds only the one service socket relative to the previous resolved policy, not claim that the full profile grants only one capability.
|
|
|
|
|
|
|
|
|
|
### 11. Status rendering performs unnecessary continuous updates
|
|
|
|
|
|
|
|
|
|
**Severity:** low
|
|
|
|
|
|
|
|
|
|
The current meter updates every 60 ms even when its rendered value has not changed, and the finalizing spinner updates every 80 ms (`index.ts:97-105,172-207`).
|
|
|
|
|
|
|
|
|
|
**Fix:** the extension renders state changes reported by the service and does not receive raw audio levels. Use static status text for `loading model...`, `listening...`, `stopping...`, and `transcribing...`. If a meter is wanted later, add a rate-limited level event only after measuring the need.
|
|
|
|
|
|
|
|
|
|
### 12. Tests, installation, attribution, and user documentation are missing
|
|
|
|
|
|
|
|
|
|
**Severity:** medium
|
|
|
|
|
|
|
|
|
|
There is no dictate test suite or README, and the root test command excludes dictate. The repository also has no service package, model provisioning flow, systemd unit, protocol documentation, or Parakeet attribution.
|
|
|
|
|
|
|
|
|
|
**Fix:** add focused TypeScript and Python tests, a concise extension README, a service installation guide, and model/license attribution. Add dictate to the root README extension inventory because the package exposes the extensions directory.
|
|
|
|
|
|
|
|
|
|
## Local service architecture
|
|
|
|
|
|
|
|
|
|
### Repository layout
|
|
|
|
|
|
|
|
|
|
The implementation should introduce one explicit host-service boundary:
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
extensions/dictate/
|
|
|
|
|
index.ts
|
|
|
|
|
index.test.ts
|
|
|
|
|
README.md
|
|
|
|
|
services/dictate/
|
|
|
|
|
pyproject.toml
|
|
|
|
|
uv.lock
|
|
|
|
|
src/pi_dictate_service/
|
|
|
|
|
__init__.py
|
|
|
|
|
audio.py
|
|
|
|
|
model.py
|
|
|
|
|
protocol.py
|
|
|
|
|
server.py
|
|
|
|
|
tests/
|
|
|
|
|
systemd/user/
|
|
|
|
|
pi-dictate.socket
|
|
|
|
|
pi-dictate.service
|
|
|
|
|
docs/dictate/
|
|
|
|
|
protocol.md
|
|
|
|
|
parakeet-attribution.md
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Use a dedicated uv environment for the service. Pin Python, NeMo, PyTorch, CUDA-compatible wheels, and transitive dependencies in `uv.lock`. Follow the repository package-source process when adding dependencies and synchronize relevant third-party source into the OpenSrc cache.
|
|
|
|
|
|
|
|
|
|
Install the service entry point and environment outside the repository under `~/.local/share/pi-dictate/`. Store the pinned model snapshot under `~/.local/share/pi-dictate/models/parakeet-tdt-0.6b-v3/`. The runtime must load that explicit local snapshot with offline mode enabled rather than resolving a floating model name.
|
|
|
|
|
|
|
|
|
|
### Socket activation and isolation
|
|
|
|
|
|
|
|
|
|
`systemd/user/pi-dictate.socket` should:
|
|
|
|
|
|
|
|
|
|
- listen at `%t/pi-dictate.sock`, which resolves to `/run/user/1000/pi-dictate.sock` on this machine;
|
|
|
|
|
- create the socket with mode `0600`;
|
|
|
|
|
- remove the socket only when the socket unit itself stops;
|
|
|
|
|
- remain active while the service exits on idle;
|
|
|
|
|
- activate one non-template service that accepts the inherited listening file descriptor.
|
|
|
|
|
|
|
|
|
|
The service must close its inherited descriptor on exit but never unlink `%t/pi-dictate.sock`; the socket unit owns that pathname and reactivation lifecycle.
|
|
|
|
|
|
|
|
|
|
`systemd/user/pi-dictate.service` should:
|
|
|
|
|
|
|
|
|
|
- run the installed Python entry point outside nono;
|
|
|
|
|
- inherit the listening socket rather than opening TCP;
|
|
|
|
|
- load the pinned model onto CUDA and fail clearly when CUDA is unavailable;
|
|
|
|
|
- own PipeWire capture and the in-memory audio buffer;
|
|
|
|
|
- serialize model loading and inference and allow only one active recording or transcription globally;
|
|
|
|
|
- reject start requests as busy while another client owns capture or while a cancelled inference worker is still running;
|
|
|
|
|
- cancel capture if the owning client disconnects;
|
|
|
|
|
- tombstone cancellation during NeMo inference, suppress the eventual result, and keep the service alive until the worker finishes;
|
|
|
|
|
- never unload the model or perform idle exit while a loading or inference worker is active;
|
|
|
|
|
- exit after 15 minutes without recording, transcription, loading, or protocol activity;
|
|
|
|
|
- release recorder, model, CUDA, accepted-client, and inherited-listener resources on shutdown without unlinking the socket-unit pathname;
|
|
|
|
|
- disable runtime network access through supported systemd hardening after the local model-loading path is proven;
|
|
|
|
|
- set `LimitCORE=0` and, where supported for the user service, `MemorySwapMax=0`;
|
|
|
|
|
- expose no raw audio or transcript logs by default.
|
|
|
|
|
|
|
|
|
|
Model loading must not block protocol cancellation. Load NeMo in a worker while the socket loop remains responsive. Cancelling during loading must emit `cancelled` promptly and tombstone that generation. The worker may finish warming the model, but it must never start recording for the cancelled generation.
|
|
|
|
|
|
|
|
|
|
### Versioned protocol
|
|
|
|
|
|
|
|
|
|
Use bounded newline-delimited JSON over one persistent Unix stream connection. Document protocol version `1` before implementation.
|
|
|
|
|
|
|
|
|
|
Client requests:
|
|
|
|
|
|
|
|
|
|
- `hello`: negotiate protocol version and service capabilities;
|
|
|
|
|
- `start`: create a generation and request model readiness plus recording;
|
|
|
|
|
- `stop`: stop and transcribe the caller's active generation;
|
|
|
|
|
- `cancel`: discard the caller's active generation;
|
|
|
|
|
- `ping`: health and readiness check.
|
|
|
|
|
|
|
|
|
|
Service events:
|
|
|
|
|
|
|
|
|
|
- `state`: `loading | listening | stopping | transcribing | idle`;
|
|
|
|
|
- `result`: one final transcript for a generation;
|
|
|
|
|
- `error`: stable code plus bounded human-readable detail;
|
|
|
|
|
- `cancelled`: confirmation that a generation cannot later produce a result;
|
|
|
|
|
- `pong`: service and protocol version information.
|
|
|
|
|
|
|
|
|
|
Every frame includes protocol version, request or generation ID, and type. Set strict frame and transcript byte limits. Reject malformed JSON, unknown fields where unsafe, unsupported versions, invalid state transitions, and commands from a client that does not own the active generation. Never use filesystem paths supplied by the Pi client.
|
|
|
|
|
|
|
|
|
|
### Authoritative lifecycle
|
|
|
|
|
|
|
|
|
|
The extension state machine is:
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
idle -> connecting -> recording -> stopping -> transcribing -> idle
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Cancellation and failure can return any active extension state to idle. State transitions are authoritative only when received for the current connection and generation. A cancelled loading or inference worker may remain active inside the service after the extension returns to idle; it owns no future delivery right, blocks new starts as busy until it finishes, and prevents service idle exit.
|
|
|
|
|
|
|
|
|
|
The service stop sequence is mandatory:
|
|
|
|
|
|
|
|
|
|
1. Mark the generation stopping so recorder exit is intentional.
|
|
|
|
|
2. Signal `pw-record`.
|
|
|
|
|
3. Continue consuming stdout.
|
|
|
|
|
4. Wait for stdout end and the child `close` event.
|
|
|
|
|
5. Enforce a short shutdown timeout and kill on overrun.
|
|
|
|
|
6. Validate and convert the complete PCM buffer.
|
|
|
|
|
7. Mark the generation transcribing.
|
|
|
|
|
8. Run one NeMo transcription job.
|
|
|
|
|
9. If the generation was tombstoned during inference, discard the result and emit or preserve the earlier `cancelled` outcome; otherwise emit exactly one result or error.
|
|
|
|
|
10. Release application-owned audio buffers and return to idle.
|
|
|
|
|
|
|
|
|
|
A clean recorder exit alone is not transcription success. Only a `result` event for the active, non-tombstoned generation authorizes editor insertion. Buffer release is best effort because NumPy, PyTorch, CUDA, swap, and crash handling can create copies outside the service's direct control.
|
|
|
|
|
|
|
|
|
|
## niri and Wayland integration
|
|
|
|
|
|
|
|
|
|
No niri configuration change is required. niri should pass `Alt+M` and `Alt+N` to Ghostty and Pi while the main editor is focused, provided the user's config does not bind those combinations.
|
|
|
|
|
|
|
|
|
|
During implementation, inspect `~/.config/niri/config.kdl` outside the current restrictive sandbox and verify:
|
|
|
|
|
|
|
|
|
|
1. `Alt+M` and `Alt+N` are not compositor bindings.
|
|
|
|
|
2. `niri validate` passes before and after any user-performed keybinding adjustment.
|
|
|
|
|
3. Both shortcuts reach Pi's main editor in Ghostty.
|
|
|
|
|
4. The shortcuts do not claim to work while a Pi modal dialog owns input.
|
|
|
|
|
|
|
|
|
|
Do not add niri `spawn` bindings. Do not use `wl-copy`, `wtype`, `ydotool`, `xclip`, or `xdotool`.
|
|
|
|
|
|
|
|
|
|
## Recommended implementation plan
|
|
|
|
|
|
|
|
|
|
Each behavior change starts with a focused failing test in the same implementation slice. Do not postpone regression tests until after all fixes.
|
|
|
|
|
|
|
|
|
|
### Phase 0: Prove NeMo and Parakeet v3 outside Pi
|
|
|
|
|
|
|
|
|
|
Build a throwaway prototype outside the production extension and nono sandbox:
|
|
|
|
|
|
|
|
|
|
1. Create an isolated uv environment with a pinned Python version.
|
|
|
|
|
2. Install a CUDA-compatible PyTorch and NeMo stack using the package-source workflow.
|
|
|
|
|
3. Download and pin one exact `nvidia/parakeet-tdt-0.6b-v3` model revision outside runtime.
|
|
|
|
|
4. Record short, realistic dictation samples through `pw-record` only after explicit user consent.
|
|
|
|
|
5. Transcribe in-memory 16 kHz mono audio without creating an audio file.
|
|
|
|
|
6. Measure model download size, cold load time, warm transcription latency, peak GPU memory, and real-time factor for 10-second, 60-second, and five-minute clips.
|
|
|
|
|
7. Test punctuation, code terminology, silence, muted input, and at least one realistic noisy sample.
|
|
|
|
|
8. Confirm that the model works with runtime network access disabled after provisioning.
|
|
|
|
|
9. Record dependency versions, model revision, hashes where available, and license obligations.
|
|
|
|
|
|
|
|
|
|
**Gate:** proceed only if warm transcription is comfortably faster than real time, cold-start latency is acceptable for socket activation, five-minute memory use is bounded, accuracy is acceptable for normal coding dictation, and the pinned offline load is reproducible. If the gate fails, revise the runtime decision with the user rather than silently substituting sherpa-onnx, another model, or cloud transcription.
|
|
|
|
|
|
|
|
|
|
### Phase 1: Define and test the service core
|
|
|
|
|
|
|
|
|
|
1. Write `docs/dictate/protocol.md` with frame schemas, limits, ownership, errors, and the lifecycle table.
|
|
|
|
|
2. Implement Python protocol parsing, state transitions, recorder control, PCM conversion, model abstraction, and idle shutdown behind injected fakes.
|
|
|
|
|
3. Add Python tests for malformed and oversized frames, ownership, busy behavior, cancellation during model load and inference, tombstoned-result suppression, recorder spawn and exit variants, stdout drain ordering, timeout escalation, duration bounds, PCM conversion, empty result, model failure, stale generations, disconnect cleanup, worker-aware idle suppression, and 15-minute idle shutdown.
|
|
|
|
|
4. Keep NeMo imports lazy so unit tests run without loading CUDA or the model.
|
|
|
|
|
5. Implement the real NeMo adapter only after service-core tests pass.
|
|
|
|
|
|
|
|
|
|
**Completion criterion:** service unit tests run without GPU access, every event/state pair has one explicit outcome, recorder stdout always drains before transcription, and no raw audio path reaches disk or logs.
|
|
|
|
|
|
|
|
|
|
### Phase 2: Add socket activation and host installation
|
|
|
|
|
|
|
|
|
|
1. Add `pi-dictate.socket` and `pi-dictate.service` user units.
|
|
|
|
|
2. Add a repeatable installer that creates the dedicated uv environment, installs the pinned package, provisions the pinned model, links the user units, reloads systemd, and enables the socket.
|
|
|
|
|
3. Keep model download and dependency installation as explicit setup actions with network access; runtime remains offline.
|
|
|
|
|
4. Add integration tests for socket activation, protocol negotiation, cold load, warm reuse, busy clients, client disconnect, cancellation during inference, service crash, idle shutdown, socket-path ownership, fresh-connection reactivation, and GPU release.
|
|
|
|
|
5. Validate unit hardening without blocking PipeWire, CUDA, or the runtime socket.
|
|
|
|
|
|
|
|
|
|
**Completion criterion:** connecting to `/run/user/1000/pi-dictate.sock` activates the service; the first request loads the model; subsequent requests stay warm for 15 minutes; idle exit releases GPU memory; and a later connection reactivates cleanly.
|
|
|
|
|
|
|
|
|
|
### Phase 3: Replace the Pi extension internals
|
|
|
|
|
|
|
|
|
|
1. Remove SoX, Deepgram, WebSocket, clipboard, focus inspection, hidden widget, meter, spinner, and global input-listener code.
|
|
|
|
|
2. Add a small versioned Unix-socket client with strict frame limits and generation checks. After service idle exit or disconnect, discard the accepted connection; the next start must create a fresh connection and repeat `hello` negotiation so the socket unit can reactivate the service.
|
|
|
|
|
3. Register `Alt+M`, `Alt+N`, and `/dictate-cancel` through public Pi APIs.
|
|
|
|
|
4. Render only service-reported state changes.
|
|
|
|
|
5. Insert only a current-generation `result` through `ctx.ui.pasteToEditor()`.
|
|
|
|
|
6. On session shutdown, send a best-effort cancel, close the socket, clear status, and invalidate the generation.
|
|
|
|
|
7. Add TypeScript tests for registration, connection failure, version mismatch, every state event, stale events, malformed and oversized frames, empty result, `pasteToEditor()` delivery, modal-safe main-editor delivery, service disconnect, and shutdown cleanup.
|
|
|
|
|
8. Run `npm run typecheck` and the focused tests after each slice.
|
|
|
|
|
|
|
|
|
|
**Completion criterion:** the extension has no microphone, GPU, model, cloud, clipboard, WebSocket, private TUI, or synthetic-input capability; it can only exchange bounded protocol messages and paste a successful current result into Pi's main editor.
|
|
|
|
|
|
|
|
|
|
### Phase 4: Add the minimum nono capability
|
|
|
|
|
|
|
|
|
|
1. Add `/run/user/1000/pi-dictate.sock` to `filesystem.unix_socket` in `profiles/nono/pi-local.json`.
|
|
|
|
|
2. Do not add PipeWire, GPU, model, cache, Deepgram, or extra network permissions.
|
|
|
|
|
3. Validate the draft and resolved policy according to the nono-profile workflow.
|
|
|
|
|
4. Compare the resolved policy before and after and confirm this feature adds only the exact service socket.
|
|
|
|
|
5. Restart Pi because an active nono sandbox cannot gain socket access in place.
|
|
|
|
|
6. Verify the extension can activate and use the service but still cannot connect directly to PipeWire or initialize NVML.
|
|
|
|
|
|
|
|
|
|
**Completion criterion:** `pi-local` validates, the service socket works after restart, and Pi still lacks direct microphone and GPU access.
|
|
|
|
|
|
|
|
|
|
### Phase 5: Document and manually validate
|
|
|
|
|
|
|
|
|
|
Create `extensions/dictate/README.md` covering:
|
|
|
|
|
|
|
|
|
|
- Linux-only support and the tested CachyOS, niri, Ghostty, PipeWire, CUDA, and RTX 4090 environment;
|
|
|
|
|
- service installation, model provisioning, systemd socket activation, status, restart, and removal;
|
|
|
|
|
- model size, cold-start expectations, 15-minute warm period, and GPU-memory behavior measured in Phase 0;
|
|
|
|
|
- Parakeet and NeMo licenses and attribution;
|
|
|
|
|
- offline runtime and the one-time networked installation step;
|
|
|
|
|
- best-effort raw-audio non-persistence, core/swap hardening, and the limits imposed by Python, NumPy, PyTorch, CUDA, and host memory management;
|
|
|
|
|
- the accepted security consequence that any code inside `pi-local` can use the socket as a mediated microphone and transcription capability;
|
|
|
|
|
- `Alt+M`, `Alt+N`, and `/dictate-cancel`, including main-editor focus requirements;
|
|
|
|
|
- loading, listening, stopping, transcribing, failure, and cancellation states;
|
|
|
|
|
- main-editor `pasteToEditor()` delivery;
|
|
|
|
|
- niri shortcut conflict checks;
|
|
|
|
|
- PipeWire default-source, CUDA, model, socket, systemd, and bounded logging troubleshooting.
|
|
|
|
|
|
|
|
|
|
Add dictate to the root README extension inventory. Resolve the pre-existing `extensions/dictate/LICENSE` deletion with the user and add model attribution without overwriting unrelated license work.
|
|
|
|
|
|
|
|
|
|
Manually validate from a restarted, profile-confined Pi session:
|
|
|
|
|
|
|
|
|
|
1. Cold start, observing socket activation, model loading, then recorder start.
|
|
|
|
|
2. Warm start within 15 minutes.
|
|
|
|
|
3. Start and stop from the normal main editor with existing text, a moved cursor, and collapsed pasted content.
|
|
|
|
|
4. Confirm shortcuts do not claim to work while a Pi modal owns input.
|
|
|
|
|
5. Open a modal after recording starts, then confirm the result still reaches only the main editor.
|
|
|
|
|
6. Cancel during model loading, recording, stopping, and transcription.
|
|
|
|
|
7. Empty speech, muted source, missing `pw-record`, PipeWire failure, CUDA failure, model failure, service crash, malformed protocol, and duration limit.
|
|
|
|
|
8. Two Pi sessions competing for the one recorder.
|
|
|
|
|
9. Pi reload and shutdown during every active state.
|
|
|
|
|
10. Idle for 15 minutes, verify service exit and GPU-memory release, then confirm a fresh client connection reactivates the still-owned socket path.
|
|
|
|
|
11. Narrow terminal and tmux behavior.
|
|
|
|
|
12. Ghostty under niri, confirming no compositor-global shortcut or synthetic typing was introduced.
|
|
|
|
|
13. Run the full repository typecheck and test suite.
|
|
|
|
|
14. Have an agent that did not implement the feature perform the final code and security review.
|
|
|
|
|
|
|
|
|
|
**Completion criterion:** documentation matches measured behavior; every manual path returns both processes to a known state; idle shutdown releases the model; no recorder, audio buffer, socket client, timer, status, or stale generation remains; and the independent review has no unresolved blocker.
|
|
|
|
|
|
|
|
|
|
## Acceptance criteria
|
|
|
|
|
|
|
|
|
|
- Phase 0 proves the pinned Parakeet v3 and NeMo stack meets measured accuracy, latency, memory, offline, and packaging requirements before production changes begin.
|
|
|
|
|
- Runtime transcription is local and makes no network request.
|
|
|
|
|
- The service intentionally creates no audio files, bounds application-owned audio memory, applies core/swap hardening where supported, and releases its buffers after every terminal outcome; documentation does not promise forensic zeroization.
|
|
|
|
|
- The host service, not Pi, exclusively owns PipeWire, CUDA, model files, and audio buffers.
|
|
|
|
|
- The service is socket-activated, handles one recording or inference worker at a time, remains warm for 15 minutes, exits only when truly idle, leaves socket-path ownership to the socket unit, and releases GPU memory.
|
|
|
|
|
- Recorder stdout drains and the child closes before transcription starts.
|
|
|
|
|
- Five-minute audio, protocol frames, stderr, logs, and transcript output are bounded.
|
|
|
|
|
- Only a non-empty `result` for the active generation changes the editor.
|
|
|
|
|
- Text insertion uses `ctx.ui.pasteToEditor()` and preserves cursor and collapsed-paste behavior.
|
|
|
|
|
- `Alt+M`, `Alt+N`, and `/dictate-cancel` are documented as main-editor-only controls.
|
|
|
|
|
- Stop, cancellation, disconnect, failure, timeout, stale result, and session shutdown are deterministic in every state.
|
|
|
|
|
- The extension contains no SoX, Deepgram, WebSocket, clipboard, X11 injection, Wayland synthetic typing, private TUI access, hidden TUI widget, global input listener, microphone capture, or GPU code.
|
|
|
|
|
- `npm run typecheck` passes.
|
|
|
|
|
- `npm test` includes and passes dictate's TypeScript tests, and the documented service test command passes its Python tests.
|
|
|
|
|
- The systemd user units and installer work from a clean setup and can be removed cleanly.
|
|
|
|
|
- The pinned model revision, dependency lock, source references, and license attribution are documented.
|
|
|
|
|
- The `pi-local` resolved-policy diff adds only `/run/user/1000/pi-dictate.sock` for this feature, and documentation identifies that socket as a user-accepted mediated microphone and transcription capability available to every descendant profile.
|
|
|
|
|
- A restarted Pi can use the service socket but still cannot connect directly to PipeWire or initialize NVML.
|
|
|
|
|
- The user's niri configuration has no conflicting `Alt+M` or `Alt+N` binding, and no global dictate binding is added.
|
|
|
|
|
- User documentation covers setup, privacy, model behavior, controls, lifecycle, failure recovery, troubleshooting, and uninstall.
|
