* Eliminate spiky terminal lag under load
Terminal output stuttered during heavy output and when the daemon was
busy. Two compounding causes, both confirmed by measurement:
- Every 256KB of output, the daemon dropped pending frames and re-sent a
full cell-grid snapshot (cloned across IPC, stringified to JSON) — a
10MB build did this ~40 times. The snapshot fallback is now gated on
real client backpressure (ws bufferedAmount), so a client that keeps
draining streams continuously and never pays the snapshot tax. This
also removes the periodic GC hitch the churn caused.
- Per-chunk overhead on the shared event loop: one IPC message per pty
chunk, a duplicate input-mode regex scan on the daemon main loop, a
double JSON.stringify per outbound message, and 16KB string realloc
per chunk. Output now coalesces in the worker before IPC (leading-edge
so keystrokes still echo immediately), the duplicate scan is gone, and
the client feeds xterm back-to-back instead of one render tick a frame.
Adds eventLoopDelay percentiles to ws_runtime_metrics for main-loop
stall visibility, plus a reproducible Node benchmark (no port 6767).
Measured: echo p50 7.7ms to 2.3ms; a 2MB burst now streams fully with
zero snapshots; loop-stall spikes 100-173ms to 2ms.
* Guard trace-level check against partial logger stubs
The isLevelEnabled gate added for the emit() perf fix crashed every
session test that injects a hand-rolled logger stub (10 of them omit
isLevelEnabled). Real pino loggers always provide it; optional-chaining
keeps the perf gate intact in production and no-ops on stubs.
* Make recent-output exit-summary test deterministic
The added test self-exited the child immediately after writing 3000
lines, then asserted the newest line was in the exit summary — but the
summary reads the headless xterm buffer, which parses writes
asynchronously, so a loaded CI runner saw a stale buffer (line-2707 not
line-2999). Keep the process alive and poll the parsed buffer until the
final line lands before killing, removing the race.
* Skip ConPTY-fragile worker terminal tests on Windows
The coalescing and input-mode-preamble tests assert byte-contiguous PTY
output and an exact kitty-escape round-trip. Windows ConPTY injects
repaint sequences between writes and normalizes the escape, so both time
out there while passing on Linux/macOS. Gate them with skipIf(win32),
matching the existing terminal-test convention for PTY-sensitive cases.
* Apply terminal barriers immediately when no writes are pending
A barrier op (snapshot/restore/clear) only needs the sentinel-write gate
to wait out plain writes still parsing in xterm's buffer. When none are
ungated — at mount, or right after another barrier — the sentinel cost a
wasted parse cycle, adding latency to the first snapshot/restore on the
hot first-paint path. Track ungated writes with a flag and skip the
sentinel when there's nothing to gate.
* Fix mobile startup saved-host e2e
* Unskip Windows worker terminal coverage
* Tighten worker coalescing assertion
* Reset terminal ungated writes on unmount
* Fix terminal regressions found in adversarial review
Three issues this PR's terminal changes introduced:
- Worker snapshot could duplicate coalesced output. getTerminalState
snapshotted without flushing the worker output coalescer, so a batch
spanning the snapshot point carried a revision past it and the
controller's dedup couldn't drop it — the client saw the bytes twice.
Flush before snapshotting.
- Relay clients lost backpressure protection. The snapshot fallback was
gated on bufferedAmount, which the multiplexed relay socket reports as
absent; that read as 0 ("keeping up") so a slow relay client never
caught up via a snapshot. Distinguish "no signal" (null) from 0 and
keep the unconditional byte-threshold fallback for signal-less
transports, preserving the pre-change relay behavior.
- Recent-output buffer was no longer a hard cap. A single chunk larger
than the limit was retained whole; slice its tail.
Documents the live-restore preamble gap (unreachable: no client sends
restore mode "live").
* Skip input-mode preamble test on Windows ConPTY
CI confirmed Windows ConPTY normalizes away the kitty keyboard escape
the child writes, so it never reaches the worker's input-mode tracker
and the preamble stays empty — the test times out. ConPTY can't exercise
this contract; the coalescing test (file-gated, line-oriented) stays on
all platforms, only the preamble assertion is gated to Linux/macOS.
110 lines
3.0 KiB
TypeScript
110 lines
3.0 KiB
TypeScript
export interface TerminalOutputCoalescerTimers {
|
|
setTimeout: typeof setTimeout;
|
|
clearTimeout: typeof clearTimeout;
|
|
}
|
|
|
|
export interface TerminalOutputCoalescerFlush {
|
|
payload: Buffer;
|
|
chars: number;
|
|
bytes: number;
|
|
}
|
|
|
|
export interface TerminalOutputCoalescerOptions {
|
|
timers: TerminalOutputCoalescerTimers;
|
|
flushDelayMs?: number;
|
|
now?: () => number;
|
|
onFlush: (payload: TerminalOutputCoalescerFlush) => void;
|
|
}
|
|
|
|
const DEFAULT_FLUSH_DELAY_MS = 5;
|
|
|
|
export class TerminalOutputCoalescer {
|
|
private readonly timers: TerminalOutputCoalescerTimers;
|
|
private readonly flushDelayMs: number;
|
|
private readonly now: () => number;
|
|
private readonly onFlush: (payload: TerminalOutputCoalescerFlush) => void;
|
|
private chunks: Buffer[] = [];
|
|
private bytes = 0;
|
|
private chars = 0;
|
|
private flushTimer: ReturnType<typeof setTimeout> | null = null;
|
|
private lastFlushAt: number | null = null;
|
|
|
|
constructor(options: TerminalOutputCoalescerOptions) {
|
|
this.timers = options.timers;
|
|
this.flushDelayMs = options.flushDelayMs ?? DEFAULT_FLUSH_DELAY_MS;
|
|
this.now = options.now ?? Date.now;
|
|
this.onFlush = options.onFlush;
|
|
}
|
|
|
|
handle(data: string): void {
|
|
if (data.length === 0) {
|
|
return;
|
|
}
|
|
|
|
const chunk = Buffer.from(data, "utf8");
|
|
this.chunks.push(chunk);
|
|
this.bytes += chunk.byteLength;
|
|
this.chars += data.length;
|
|
|
|
// Leading edge: if nothing is pending and no flush happened within the last
|
|
// flushDelayMs, flush immediately so interactive echo isn't delayed a full
|
|
// window. Otherwise accumulate and let the trailing timer drain the burst.
|
|
if (!this.flushTimer) {
|
|
const elapsed =
|
|
this.lastFlushAt === null ? Number.POSITIVE_INFINITY : this.now() - this.lastFlushAt;
|
|
if (elapsed >= this.flushDelayMs) {
|
|
this.flush();
|
|
return;
|
|
}
|
|
this.flushTimer = this.timers.setTimeout(() => {
|
|
this.flushTimer = null;
|
|
this.flush();
|
|
}, this.flushDelayMs);
|
|
}
|
|
}
|
|
|
|
flush(): void {
|
|
this.clearTimer();
|
|
|
|
if (this.chunks.length === 0) {
|
|
return;
|
|
}
|
|
|
|
const payload =
|
|
this.chunks.length === 1 ? this.chunks[0] : Buffer.concat(this.chunks, this.bytes);
|
|
const bytes = this.bytes;
|
|
const chars = this.chars;
|
|
this.clearPending();
|
|
this.lastFlushAt = this.now();
|
|
|
|
this.onFlush({ payload, bytes, chars });
|
|
}
|
|
|
|
// Record that a frame was emitted out-of-band on this stream (e.g. a snapshot
|
|
// frame sent directly, bypassing the coalescer). This keeps the next handled
|
|
// chunk on the trailing path instead of leading-edge flushing it back-to-back
|
|
// with that frame, preserving the small gap consumers rely on after a snapshot.
|
|
markFlushed(): void {
|
|
this.lastFlushAt = this.now();
|
|
}
|
|
|
|
dispose(): void {
|
|
this.clearTimer();
|
|
this.clearPending();
|
|
}
|
|
|
|
private clearTimer(): void {
|
|
if (!this.flushTimer) {
|
|
return;
|
|
}
|
|
this.timers.clearTimeout(this.flushTimer);
|
|
this.flushTimer = null;
|
|
}
|
|
|
|
private clearPending(): void {
|
|
this.chunks = [];
|
|
this.bytes = 0;
|
|
this.chars = 0;
|
|
}
|
|
}
|