Mohamed BoudraandGitHub
16120ebff1
* feat(plugins): expose Paseo API to local plugins Plugin surfaces borrow the selected host connection while subprocess handlers receive an exclusively owned IPC session, so plugins can use normal workspace and agent operations without inventing another transport. * feat(plugins): reload global enablement safely Treat the global plugin switch as a live daemon setting and teach plugin automation to inspect it before acting. Enabling remains permission-gated because plugin backend code can access the daemon machine and client contributions execute inside the app. * feat(plugins): expose plugin logs Capture plugin stdout and stderr separately from IPC so authors can debug initialization, RPC, cleanup, and crash behavior from the app or CLI. Keep a bounded tail across lifecycle recovery while preserving structured output in the daemon log. * fix(build): order plugin dependency builds * test(plugins): align startup and CLI expectations
48 lines
5.6 KiB
Markdown
48 lines
5.6 KiB
Markdown
# Terminal performance
|
|
|
|
How terminal output stays low-latency, what the invariants are, and how to measure before/after any change to the pipeline. Read this before touching anything under `packages/server/src/terminal/` or `packages/app/src/terminal/runtime/`.
|
|
|
|
## The pipeline
|
|
|
|
```
|
|
pty (node-pty, forked worker process)
|
|
→ headless xterm parse (worker, snapshot fidelity)
|
|
→ TerminalOutputCoalescer (worker, ≤1 IPC message per 5ms per terminal)
|
|
→ process.send IPC → daemon main process
|
|
→ TerminalOutputCoalescer (per client stream, terminal-session-controller.ts)
|
|
→ binary ws frame (2-byte header + raw bytes)
|
|
→ client decode (daemon-client.ts) → stream router → emulator runtime
|
|
→ xterm.write (back-to-back; xterm batches internally)
|
|
```
|
|
|
|
Terminal frames share the daemon main event loop with all agent traffic. The `eventLoopDelay` block in the `ws_runtime_metrics` log line (every 30s in `daemon.log`) is the ground truth for "the daemon is busy" — p99/max there directly bound worst-case terminal frame delay.
|
|
|
|
## Invariants (the easy-to-break ones)
|
|
|
|
- **Coalescers are leading+trailing throttles.** The first chunk after an idle window flushes immediately (synchronously); only sustained bursts wait for the trailing timer. Reverting to trailing-only adds a full window (~5ms) to every keystroke echo.
|
|
- **Output coalescing happens in the worker, before IPC.** One `process.send` per pty chunk was a main-loop flood under build output. Non-output messages (snapshot/snapshotReady/titleChange/exit) must flush the coalescer first so ordering is preserved.
|
|
- **Coalesced output carries the LAST chunk's revision.** Snapshot replay dedup (`replayTerminalOutputAfterSnapshot`) skips buffered output with `revision <= replayRevision`; a merged batch with a lower revision would be wrongly skipped (lost output).
|
|
- **The input-mode tracker runs once per process boundary, not per hop.** The worker owns the authoritative tracker; the daemon caches the replay preamble from `getTerminalState` responses and `snapshotReady` messages. Do not reintroduce a per-chunk `feed()` on the daemon main loop.
|
|
- **Snapshot catch-up is backpressure-gated.** A stream falls back to a full snapshot only when `outputBytesSinceSnapshot > MAX_TERMINAL_OUTPUT_FRAME_BYTES` (256KB) **and** the client transport reports `bufferedAmount > MAX_CLIENT_BUFFERED_BYTES` (4MB). A client that keeps draining streams continuously, no matter how much output is produced. Before this gate existed, every 256KB of build output dropped a frame and forced a full JSON cell-grid snapshot (~200k objects across IPC) — the historical source of spiky lag and GC hitches.
|
|
- **Plugin daemon sessions report IPC queue bytes.** Their virtual socket increments `bufferedAmount` before `process.send` and decrements it only in the send callback. Text and binary frames share that ordered queue, so the normal snapshot catch-up and physical high-water gates remain valid for server-side plugin SDK traffic.
|
|
- **Client output writes are not serialized per frame.** The emulator runtime drains contiguous plain writes straight into xterm (which buffers internally). Only barrier ops (`clear`, `snapshot`, `suppressInput` writes) wait — behind a zero-length sentinel write — so resets can't interleave with in-flight output.
|
|
- **Terminal size has one daemon-owned claimant.** Focus and direct interaction send a `claim`; later geometry changes from that connection send `update`. A claim transfers ownership even when the dimensions are unchanged, while an update from any other connection is ignored. This lets an owning pane follow splits and keyboard insets without allowing an idle phone or browser to steal the PTY size.
|
|
|
|
## Measuring
|
|
|
|
- **Node-only benchmark (fast iteration, server pipeline):** `npx tsx scripts/benchmark-terminal-latency.ts`. Boots an isolated daemon (fresh `PASEO_HOME`, random port — never 6767), measures echo latency percentiles, burst jitter, and snapshot counts under ramped mock-agent load. Writes JSON to `/tmp/paseo-terminal-bench/`. Healthy numbers (2026-06): echo p50 ~2.3ms, p95 ~3.3ms, a 2MB burst fully streamed with `snap=0`.
|
|
- **Browser perf specs (user-perceived path):** gated behind `PASEO_TERMINAL_PERF_E2E=1` —
|
|
`packages/app/e2e/browser/terminal-performance.spec.ts` and `packages/app/e2e/browser/terminal-keystroke-stress.spec.ts` (per-stage keydown→xterm-commit breakdown under mock-agent load). Healthy: keydown→commit p50 ~18ms under 600-key burst.
|
|
- **Production:** grep `daemon.log` for `ws_runtime_metrics` and read `eventLoopDelay` + `bufferedAmount`.
|
|
- **Git pressure:** the same log line includes `git.commands` (limiter occupancy, queue age,
|
|
queue wait, execution time, failures, timeouts, and top operations),
|
|
`git.workspaceService` (daemon-global Git observer ownership), and per-session workspace Git
|
|
subscription totals under `runtime`. Queue wait and execution time are separate because the Git
|
|
command timeout begins only after a command acquires a limiter slot.
|
|
|
|
## Known remaining contention (follow-up candidates)
|
|
|
|
- A single large `agent_stream` message (e.g. a 250KB diff payload) measurably delays terminal echo (~100ms-class dips) — cost is split between daemon serialization and app-side parse/render on the shared browser main thread.
|
|
- Relay-attached clients pay pure-JS tweetnacl encryption on the daemon main loop (`packages/relay/src/encrypted-channel.ts`). Negotiated binary application frames stay binary ciphertext and avoid base64 encode/decode; text and mixed-version traffic remain base64 WebSocket text frames.
|
|
- `sendToClient` re-stringifies session messages per socket; only matters for multi-socket connections.
|