64 lines
5.5 KiB
Markdown
64 lines
5.5 KiB
Markdown
# Dictate Phase 0 results
|
|
|
|
Date: 26AUG2026
|
|
|
|
## Pinned runtime
|
|
|
|
- CPython 3.12.13
|
|
- NVIDIA NeMo 3.0.0
|
|
- PyTorch 2.13.0 with CUDA 13.0
|
|
- NumPy 2.5.2
|
|
- Model revision `541d1f99c6b0c3cd0b11a95167540bb8edefd82b`
|
|
- Model SHA-256 `3cbdc85877e668ca7b82d0d56770eb1fac76691f55d6b97545e8d61ca588d10d`
|
|
- NVIDIA GeForce RTX 4090
|
|
|
|
The complete dependency graph and artifact hashes are in `services/dictate/uv.lock`. Exact direct dependency sources were synchronized into `/srv/data/opensrc`.
|
|
|
|
## Offline and in-memory validation
|
|
|
|
The model loaded from the explicit local `.nemo` archive with Hugging Face, Transformers, and NeMo offline settings enabled. Audio was passed to NeMo as an in-memory 16 kHz mono float array. The benchmark created no audio files.
|
|
|
|
| Input | Inference | Real-time factor | Peak CUDA allocation |
|
|
| --- | ---: | ---: | ---: |
|
|
| Existing 2.78-second verification speech | 0.304 s | 0.109 | 5.07 GB |
|
|
| Live 9.96-second microphone speech | 0.285 s | 0.0286 | 5.07 GB |
|
|
| Repeated 60-second speech stress input | 0.431 s | 0.00718 | 5.18 GB |
|
|
| Repeated five-minute speech stress input | 0.820 s | 0.00273 | 6.15 GB |
|
|
|
|
Cold base-model loading was consistently about 6.4 seconds. The production adapter with its native phrase-boost tree loaded in about 10 seconds and transcribed the known verification fixture exactly. The model archive occupies about 2.4 GB. The prototype environment occupies about 5.6 GB.
|
|
|
|
The 60-second and five-minute inputs repeat real speech in memory. They validate latency and bounded GPU behavior, not natural long-form accuracy. Five-minute signed-16 PCM is exactly 9,600,000 bytes; the float32 inference representation is 19,200,000 bytes.
|
|
|
|
## Accuracy observations
|
|
|
|
The verification sample transcribed exactly:
|
|
|
|
> The quick brown fox jumps over the lazy dog.
|
|
|
|
A normal live dictation sample was coherent and punctuated, and the user accepted its accuracy.
|
|
|
|
The initial code-terminology sample missed `JSON`, `npm`, and `git`. The user failed the original Phase 0 accuracy gate and approved a conservative host-owned coding dictionary using NeMo's native phrase boosting.
|
|
|
|
A 37-term dictionary and boost strengths from 0.25 through 2.0 were tested against one in-memory capture. Alpha `1.0` changed the end of the captured phrase from `run np` to `run npm` while preserving the earlier `TypeScript` and `JSON response` terms. Alpha `1.0` was selected as the conservative default. The capture ended before `git status`, so that phrase still requires a focused validation sample.
|
|
|
|
## Gate status
|
|
|
|
The packaging, latency, memory, offline, and normal-dictation gates pass. Alpha `1.0` produced the same expected clean transcript as baseline, the same degraded transcript on seeded noisy input, and an empty transcript for silence. No dictionary hallucination was observed.
|
|
|
|
The coding-term remediation passes the final focused live check through the installed socket-activated service. The spoken phrase “Run npm test, then git status.” produced exactly `Run npm test, then git status.` with native phrase boosting at alpha `1.0`.
|
|
|
|
The upgraded configurable service was revalidated on 03SEP2026. Immediate cancellation during preparation was delivery-neutral, a second generation started successfully, and a six-second end-to-end capture returned `Run npm test, then get status.` The wording variation does not replace the earlier accuracy fixture; it confirms the current installer, systemd sandbox, socket protocol, PipeWire capture, in-memory model path, and final-only delivery operate together. Local systemd policy exposed only the PipeWire runtime socket while retaining the masked home and IP-network denial.
|
|
|
|
## Configurable backend contract gate
|
|
|
|
Credentialed live contract checks passed on 03SEP2026 with a five-second, non-sensitive 24 kHz mono PCM microphone sample held only in memory. Dedicated mode-`0600` provider key files were read by the host-side adapters; no credential entered Pi, a command line, a URL, or captured output.
|
|
|
|
| Provider | Model | Result |
|
|
| --- | --- | --- |
|
|
| Meta | `muse-voice-transcribe-1.0` | The public `api.meta.ai` handshake accepted the session, streamed PCM, accepted `endStream`, and returned an explicit final transcript (`run npm test`). Total turn time was 30.5 seconds. |
|
|
| OpenAI | `gpt-live-transcribe` | The GA transcription session accepted 24 kHz PCM, explicit commit, and returned a correlated completion event (`Run npm test and`). Total turn time was 7.1 seconds. |
|
|
|
|
The first OpenAI spike exposed forward-compatible fields (`event_id` and `previous_item_id`) on the commit acknowledgement. The adapter and sanitized fixture now accept those bounded fields while still requiring a non-empty `item_id`. A separate credentialed session update confirmed that `gpt-live-transcribe` accepts transcription `delay: "low"`. Configuration therefore supports only the documented bounded values `minimal`, `low`, `medium`, `high`, and `xhigh`; omission keeps the provider default.
|
|
|
|
These checks validate the current wire contract and finalization path, not comparative transcription accuracy. The public contract references checked were the [Meta Realtime Voice API](https://ai.developer.meta.com/docs/api-reference/voice/realtime), [Meta event schemas](https://ai.developer.meta.com/docs/api-reference/voice/schemas), [OpenAI Realtime transcription guide](https://developers.openai.com/api/docs/guides/realtime-transcription), [OpenAI Realtime WebSocket guide](https://developers.openai.com/api/docs/guides/realtime-websocket), and [`gpt-live-transcribe` model page](https://developers.openai.com/api/docs/models/gpt-live-transcribe).
|