Files
paseo/packages/server/src/terminal/terminal-output-coalescer.ts
T
Mohamed BoudraandGitHub 72b67f48e3 Eliminate spiky terminal lag under load (#1500)
* Eliminate spiky terminal lag under load

Terminal output stuttered during heavy output and when the daemon was
busy. Two compounding causes, both confirmed by measurement:

- Every 256KB of output, the daemon dropped pending frames and re-sent a
  full cell-grid snapshot (cloned across IPC, stringified to JSON) — a
  10MB build did this ~40 times. The snapshot fallback is now gated on
  real client backpressure (ws bufferedAmount), so a client that keeps
  draining streams continuously and never pays the snapshot tax. This
  also removes the periodic GC hitch the churn caused.
- Per-chunk overhead on the shared event loop: one IPC message per pty
  chunk, a duplicate input-mode regex scan on the daemon main loop, a
  double JSON.stringify per outbound message, and 16KB string realloc
  per chunk. Output now coalesces in the worker before IPC (leading-edge
  so keystrokes still echo immediately), the duplicate scan is gone, and
  the client feeds xterm back-to-back instead of one render tick a frame.

Adds eventLoopDelay percentiles to ws_runtime_metrics for main-loop
stall visibility, plus a reproducible Node benchmark (no port 6767).

Measured: echo p50 7.7ms to 2.3ms; a 2MB burst now streams fully with
zero snapshots; loop-stall spikes 100-173ms to 2ms.

* Guard trace-level check against partial logger stubs

The isLevelEnabled gate added for the emit() perf fix crashed every
session test that injects a hand-rolled logger stub (10 of them omit
isLevelEnabled). Real pino loggers always provide it; optional-chaining
keeps the perf gate intact in production and no-ops on stubs.

* Make recent-output exit-summary test deterministic

The added test self-exited the child immediately after writing 3000
lines, then asserted the newest line was in the exit summary — but the
summary reads the headless xterm buffer, which parses writes
asynchronously, so a loaded CI runner saw a stale buffer (line-2707 not
line-2999). Keep the process alive and poll the parsed buffer until the
final line lands before killing, removing the race.

* Skip ConPTY-fragile worker terminal tests on Windows

The coalescing and input-mode-preamble tests assert byte-contiguous PTY
output and an exact kitty-escape round-trip. Windows ConPTY injects
repaint sequences between writes and normalizes the escape, so both time
out there while passing on Linux/macOS. Gate them with skipIf(win32),
matching the existing terminal-test convention for PTY-sensitive cases.

* Apply terminal barriers immediately when no writes are pending

A barrier op (snapshot/restore/clear) only needs the sentinel-write gate
to wait out plain writes still parsing in xterm's buffer. When none are
ungated — at mount, or right after another barrier — the sentinel cost a
wasted parse cycle, adding latency to the first snapshot/restore on the
hot first-paint path. Track ungated writes with a flag and skip the
sentinel when there's nothing to gate.

* Fix mobile startup saved-host e2e

* Unskip Windows worker terminal coverage

* Tighten worker coalescing assertion

* Reset terminal ungated writes on unmount

* Fix terminal regressions found in adversarial review

Three issues this PR's terminal changes introduced:

- Worker snapshot could duplicate coalesced output. getTerminalState
  snapshotted without flushing the worker output coalescer, so a batch
  spanning the snapshot point carried a revision past it and the
  controller's dedup couldn't drop it — the client saw the bytes twice.
  Flush before snapshotting.
- Relay clients lost backpressure protection. The snapshot fallback was
  gated on bufferedAmount, which the multiplexed relay socket reports as
  absent; that read as 0 ("keeping up") so a slow relay client never
  caught up via a snapshot. Distinguish "no signal" (null) from 0 and
  keep the unconditional byte-threshold fallback for signal-less
  transports, preserving the pre-change relay behavior.
- Recent-output buffer was no longer a hard cap. A single chunk larger
  than the limit was retained whole; slice its tail.

Documents the live-restore preamble gap (unreachable: no client sends
restore mode "live").

* Skip input-mode preamble test on Windows ConPTY

CI confirmed Windows ConPTY normalizes away the kitty keyboard escape
the child writes, so it never reaches the worker's input-mode tracker
and the preamble stays empty — the test times out. ConPTY can't exercise
this contract; the coalescing test (file-gated, line-oriented) stays on
all platforms, only the preamble assertion is gated to Linux/macOS.
2026-06-13 05:38:58 +00:00

110 lines
3.0 KiB
TypeScript

export interface TerminalOutputCoalescerTimers {
setTimeout: typeof setTimeout;
clearTimeout: typeof clearTimeout;
}
export interface TerminalOutputCoalescerFlush {
payload: Buffer;
chars: number;
bytes: number;
}
export interface TerminalOutputCoalescerOptions {
timers: TerminalOutputCoalescerTimers;
flushDelayMs?: number;
now?: () => number;
onFlush: (payload: TerminalOutputCoalescerFlush) => void;
}
const DEFAULT_FLUSH_DELAY_MS = 5;
export class TerminalOutputCoalescer {
private readonly timers: TerminalOutputCoalescerTimers;
private readonly flushDelayMs: number;
private readonly now: () => number;
private readonly onFlush: (payload: TerminalOutputCoalescerFlush) => void;
private chunks: Buffer[] = [];
private bytes = 0;
private chars = 0;
private flushTimer: ReturnType<typeof setTimeout> | null = null;
private lastFlushAt: number | null = null;
constructor(options: TerminalOutputCoalescerOptions) {
this.timers = options.timers;
this.flushDelayMs = options.flushDelayMs ?? DEFAULT_FLUSH_DELAY_MS;
this.now = options.now ?? Date.now;
this.onFlush = options.onFlush;
}
handle(data: string): void {
if (data.length === 0) {
return;
}
const chunk = Buffer.from(data, "utf8");
this.chunks.push(chunk);
this.bytes += chunk.byteLength;
this.chars += data.length;
// Leading edge: if nothing is pending and no flush happened within the last
// flushDelayMs, flush immediately so interactive echo isn't delayed a full
// window. Otherwise accumulate and let the trailing timer drain the burst.
if (!this.flushTimer) {
const elapsed =
this.lastFlushAt === null ? Number.POSITIVE_INFINITY : this.now() - this.lastFlushAt;
if (elapsed >= this.flushDelayMs) {
this.flush();
return;
}
this.flushTimer = this.timers.setTimeout(() => {
this.flushTimer = null;
this.flush();
}, this.flushDelayMs);
}
}
flush(): void {
this.clearTimer();
if (this.chunks.length === 0) {
return;
}
const payload =
this.chunks.length === 1 ? this.chunks[0] : Buffer.concat(this.chunks, this.bytes);
const bytes = this.bytes;
const chars = this.chars;
this.clearPending();
this.lastFlushAt = this.now();
this.onFlush({ payload, bytes, chars });
}
// Record that a frame was emitted out-of-band on this stream (e.g. a snapshot
// frame sent directly, bypassing the coalescer). This keeps the next handled
// chunk on the trailing path instead of leading-edge flushing it back-to-back
// with that frame, preserving the small gap consumers rely on after a snapshot.
markFlushed(): void {
this.lastFlushAt = this.now();
}
dispose(): void {
this.clearTimer();
this.clearPending();
}
private clearTimer(): void {
if (!this.flushTimer) {
return;
}
this.timers.clearTimeout(this.flushTimer);
this.flushTimer = null;
}
private clearPending(): void {
this.chunks = [];
this.bytes = 0;
this.chars = 0;
}
}