11 lines
1.1 KiB
Markdown
11 lines
1.1 KiB
Markdown
# Todos
|
|
|
|
## Investigate transient concurrent NeMo benchmark failures
|
|
|
|
During Dictate Phase 0 on 26AUG2026, repeated overlapping throwaway benchmark processes produced two one-off failures in the external `/srv/data/tools/parakeet-v3/.venv` environment:
|
|
|
|
1. `ASRModel.restore_from()` raised `MemoryError` while OmegaConf rebuilt `joint.vocabulary`. The exact clean/noisy/silence harness passed on its immediate rerun.
|
|
2. A later production-adapter check raised `ValueError: bad marshal data (invalid reference)` while importing `torch._tensor`. An immediate standalone `import torch` succeeded, and the cached `_tensor.cpython-312.pyc` passed `marshal.loads()`.
|
|
|
|
The production service serializes model loading and inference and therefore does not intentionally launch concurrent NeMo processes. Before treating concurrent workers as supported, build a repeatable cold-load stress loop, monitor process and cgroup memory, disable bytecode writes, and determine whether the failures come from NeMo/OmegaConf resource pressure or concurrent access to the external environment's bytecode cache.
|