Files
paseo/packages/server/src/terminal/terminal-output-coalescer.test.ts
T
Mohamed BoudraandGitHub 72b67f48e3 Eliminate spiky terminal lag under load (#1500)
* Eliminate spiky terminal lag under load

Terminal output stuttered during heavy output and when the daemon was
busy. Two compounding causes, both confirmed by measurement:

- Every 256KB of output, the daemon dropped pending frames and re-sent a
  full cell-grid snapshot (cloned across IPC, stringified to JSON) — a
  10MB build did this ~40 times. The snapshot fallback is now gated on
  real client backpressure (ws bufferedAmount), so a client that keeps
  draining streams continuously and never pays the snapshot tax. This
  also removes the periodic GC hitch the churn caused.
- Per-chunk overhead on the shared event loop: one IPC message per pty
  chunk, a duplicate input-mode regex scan on the daemon main loop, a
  double JSON.stringify per outbound message, and 16KB string realloc
  per chunk. Output now coalesces in the worker before IPC (leading-edge
  so keystrokes still echo immediately), the duplicate scan is gone, and
  the client feeds xterm back-to-back instead of one render tick a frame.

Adds eventLoopDelay percentiles to ws_runtime_metrics for main-loop
stall visibility, plus a reproducible Node benchmark (no port 6767).

Measured: echo p50 7.7ms to 2.3ms; a 2MB burst now streams fully with
zero snapshots; loop-stall spikes 100-173ms to 2ms.

* Guard trace-level check against partial logger stubs

The isLevelEnabled gate added for the emit() perf fix crashed every
session test that injects a hand-rolled logger stub (10 of them omit
isLevelEnabled). Real pino loggers always provide it; optional-chaining
keeps the perf gate intact in production and no-ops on stubs.

* Make recent-output exit-summary test deterministic

The added test self-exited the child immediately after writing 3000
lines, then asserted the newest line was in the exit summary — but the
summary reads the headless xterm buffer, which parses writes
asynchronously, so a loaded CI runner saw a stale buffer (line-2707 not
line-2999). Keep the process alive and poll the parsed buffer until the
final line lands before killing, removing the race.

* Skip ConPTY-fragile worker terminal tests on Windows

The coalescing and input-mode-preamble tests assert byte-contiguous PTY
output and an exact kitty-escape round-trip. Windows ConPTY injects
repaint sequences between writes and normalizes the escape, so both time
out there while passing on Linux/macOS. Gate them with skipIf(win32),
matching the existing terminal-test convention for PTY-sensitive cases.

* Apply terminal barriers immediately when no writes are pending

A barrier op (snapshot/restore/clear) only needs the sentinel-write gate
to wait out plain writes still parsing in xterm's buffer. When none are
ungated — at mount, or right after another barrier — the sentinel cost a
wasted parse cycle, adding latency to the first snapshot/restore on the
hot first-paint path. Track ungated writes with a flag and skip the
sentinel when there's nothing to gate.

* Fix mobile startup saved-host e2e

* Unskip Windows worker terminal coverage

* Tighten worker coalescing assertion

* Reset terminal ungated writes on unmount

* Fix terminal regressions found in adversarial review

Three issues this PR's terminal changes introduced:

- Worker snapshot could duplicate coalesced output. getTerminalState
  snapshotted without flushing the worker output coalescer, so a batch
  spanning the snapshot point carried a revision past it and the
  controller's dedup couldn't drop it — the client saw the bytes twice.
  Flush before snapshotting.
- Relay clients lost backpressure protection. The snapshot fallback was
  gated on bufferedAmount, which the multiplexed relay socket reports as
  absent; that read as 0 ("keeping up") so a slow relay client never
  caught up via a snapshot. Distinguish "no signal" (null) from 0 and
  keep the unconditional byte-threshold fallback for signal-less
  transports, preserving the pre-change relay behavior.
- Recent-output buffer was no longer a hard cap. A single chunk larger
  than the limit was retained whole; slice its tail.

Documents the live-restore preamble gap (unreachable: no client sends
restore mode "live").

* Skip input-mode preamble test on Windows ConPTY

CI confirmed Windows ConPTY normalizes away the kitty keyboard escape
the child writes, so it never reaches the worker's input-mode tracker
and the preamble stays empty — the test times out. ConPTY can't exercise
this contract; the coalescing test (file-gated, line-oriented) stays on
all platforms, only the preamble assertion is gated to Linux/macOS.
2026-06-13 05:38:58 +00:00

151 lines
4.6 KiB
TypeScript

import { describe, expect, test } from "vitest";
import { TerminalOutputCoalescer } from "./terminal-output-coalescer.js";
function createHarness() {
let nextTimer = 1;
let now = 1000;
const scheduled = new Map<number, { callback: () => void; delayMs: number }>();
const flushes: Array<{ payload: string; chars: number; bytes: number }> = [];
const coalescer = new TerminalOutputCoalescer({
timers: {
setTimeout: ((callback: () => void, delayMs?: number) => {
const id = nextTimer;
nextTimer += 1;
scheduled.set(id, { callback, delayMs: delayMs ?? 0 });
return id;
}) as typeof setTimeout,
clearTimeout: ((id: number) => {
scheduled.delete(id);
}) as typeof clearTimeout,
},
now: () => now,
onFlush: ({ payload, chars, bytes }) => {
flushes.push({ payload: payload.toString("utf8"), chars, bytes });
},
});
return {
coalescer,
flushes,
scheduled,
advance(ms: number) {
now += ms;
},
runScheduled() {
const callbacks = Array.from(scheduled.values());
scheduled.clear();
for (const { callback } of callbacks) {
callback();
}
},
};
}
describe("TerminalOutputCoalescer", () => {
test("flushes the first chunk immediately on the leading edge", () => {
const { coalescer, flushes, scheduled } = createHarness();
coalescer.handle("a");
// No timer scheduled: the leading-edge flush happened synchronously.
expect(scheduled.size).toBe(0);
expect(flushes).toEqual([{ payload: "a", chars: 1, bytes: 1 }]);
});
test("coalesces a burst that follows the leading-edge flush into one trailing flush", () => {
const { coalescer, flushes, runScheduled, scheduled } = createHarness();
// First chunk flushes immediately (leading edge).
coalescer.handle("a");
// Subsequent chunks within the window accumulate behind a trailing timer.
coalescer.handle("b");
coalescer.handle("é");
expect(scheduled.size).toBe(1);
expect(Array.from(scheduled.values())).toEqual([
{ callback: expect.any(Function), delayMs: 5 },
]);
expect(flushes).toEqual([{ payload: "a", chars: 1, bytes: 1 }]);
runScheduled();
expect(flushes).toEqual([
{ payload: "a", chars: 1, bytes: 1 },
{ payload: "bé", chars: 2, bytes: 3 },
]);
});
test("flushes immediately again once the window has elapsed", () => {
const { coalescer, flushes, advance, scheduled } = createHarness();
coalescer.handle("a");
expect(flushes).toEqual([{ payload: "a", chars: 1, bytes: 1 }]);
advance(5);
coalescer.handle("b");
expect(scheduled.size).toBe(0);
expect(flushes).toEqual([
{ payload: "a", chars: 1, bytes: 1 },
{ payload: "b", chars: 1, bytes: 1 },
]);
});
test("manual flush drains pending output and cancels the scheduled flush", () => {
const { coalescer, flushes, runScheduled, scheduled } = createHarness();
// Leading-edge flush, then accumulate a burst behind the trailing timer.
coalescer.handle("hello");
coalescer.handle(" world");
coalescer.flush();
runScheduled();
expect(scheduled.size).toBe(0);
expect(flushes).toEqual([
{ payload: "hello", chars: 5, bytes: 5 },
{ payload: " world", chars: 6, bytes: 6 },
]);
});
test("dispose drops pending output", () => {
const { coalescer, flushes, runScheduled, scheduled } = createHarness();
// First chunk flushes on the leading edge; second one stays pending.
coalescer.handle("done");
coalescer.handle("pending");
coalescer.dispose();
runScheduled();
expect(scheduled.size).toBe(0);
expect(flushes).toEqual([{ payload: "done", chars: 4, bytes: 4 }]);
});
test("markFlushed keeps the next chunk on the trailing path", () => {
const { coalescer, flushes, runScheduled, scheduled } = createHarness();
// Simulate a frame emitted out-of-band (e.g. a snapshot) right before output.
coalescer.markFlushed();
coalescer.handle("post-snapshot");
// No immediate flush: the chunk waits for the trailing timer.
expect(scheduled.size).toBe(1);
expect(flushes).toEqual([]);
runScheduled();
expect(flushes).toEqual([{ payload: "post-snapshot", chars: 13, bytes: 13 }]);
});
test("preserves ordering across leading and trailing flushes", () => {
const { coalescer, flushes, runScheduled, advance } = createHarness();
coalescer.handle("1");
coalescer.handle("2");
coalescer.handle("3");
runScheduled();
advance(5);
coalescer.handle("4");
expect(flushes.map((f) => f.payload)).toEqual(["1", "23", "4"]);
});
});