jb/paseo
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

Eliminate spiky terminal lag under load (#1500)

Browse Source 
* Eliminate spiky terminal lag under load

Terminal output stuttered during heavy output and when the daemon was
busy. Two compounding causes, both confirmed by measurement:

- Every 256KB of output, the daemon dropped pending frames and re-sent a
  full cell-grid snapshot (cloned across IPC, stringified to JSON) — a
  10MB build did this ~40 times. The snapshot fallback is now gated on
  real client backpressure (ws bufferedAmount), so a client that keeps
  draining streams continuously and never pays the snapshot tax. This
  also removes the periodic GC hitch the churn caused.
- Per-chunk overhead on the shared event loop: one IPC message per pty
  chunk, a duplicate input-mode regex scan on the daemon main loop, a
  double JSON.stringify per outbound message, and 16KB string realloc
  per chunk. Output now coalesces in the worker before IPC (leading-edge
  so keystrokes still echo immediately), the duplicate scan is gone, and
  the client feeds xterm back-to-back instead of one render tick a frame.

Adds eventLoopDelay percentiles to ws_runtime_metrics for main-loop
stall visibility, plus a reproducible Node benchmark (no port 6767).

Measured: echo p50 7.7ms to 2.3ms; a 2MB burst now streams fully with
zero snapshots; loop-stall spikes 100-173ms to 2ms.

* Guard trace-level check against partial logger stubs

The isLevelEnabled gate added for the emit() perf fix crashed every
session test that injects a hand-rolled logger stub (10 of them omit
isLevelEnabled). Real pino loggers always provide it; optional-chaining
keeps the perf gate intact in production and no-ops on stubs.

* Make recent-output exit-summary test deterministic

The added test self-exited the child immediately after writing 3000
lines, then asserted the newest line was in the exit summary — but the
summary reads the headless xterm buffer, which parses writes
asynchronously, so a loaded CI runner saw a stale buffer (line-2707 not
line-2999). Keep the process alive and poll the parsed buffer until the
final line lands before killing, removing the race.

* Skip ConPTY-fragile worker terminal tests on Windows

The coalescing and input-mode-preamble tests assert byte-contiguous PTY
output and an exact kitty-escape round-trip. Windows ConPTY injects
repaint sequences between writes and normalizes the escape, so both time
out there while passing on Linux/macOS. Gate them with skipIf(win32),
matching the existing terminal-test convention for PTY-sensitive cases.

* Apply terminal barriers immediately when no writes are pending

A barrier op (snapshot/restore/clear) only needs the sentinel-write gate
to wait out plain writes still parsing in xterm's buffer. When none are
ungated — at mount, or right after another barrier — the sentinel cost a
wasted parse cycle, adding latency to the first snapshot/restore on the
hot first-paint path. Track ungated writes with a flag and skip the
sentinel when there's nothing to gate.

* Fix mobile startup saved-host e2e

* Unskip Windows worker terminal coverage

* Tighten worker coalescing assertion

* Reset terminal ungated writes on unmount

* Fix terminal regressions found in adversarial review

Three issues this PR's terminal changes introduced:

- Worker snapshot could duplicate coalesced output. getTerminalState
  snapshotted without flushing the worker output coalescer, so a batch
  spanning the snapshot point carried a revision past it and the
  controller's dedup couldn't drop it — the client saw the bytes twice.
  Flush before snapshotting.
- Relay clients lost backpressure protection. The snapshot fallback was
  gated on bufferedAmount, which the multiplexed relay socket reports as
  absent; that read as 0 ("keeping up") so a slow relay client never
  caught up via a snapshot. Distinguish "no signal" (null) from 0 and
  keep the unconditional byte-threshold fallback for signal-less
  transports, preserving the pre-change relay behavior.
- Recent-output buffer was no longer a hard cap. A single chunk larger
  than the limit was retained whole; slice its tail.

Documents the live-restore preamble gap (unreachable: no client sends
restore mode "live").

* Skip input-mode preamble test on Windows ConPTY

CI confirmed Windows ConPTY normalizes away the kitty keyboard escape
the child writes, so it never reaches the worker's input-mode tracker
and the preamble stays empty — the test times out. ConPTY can't exercise
this contract; the coalescing test (file-gated, line-oriented) stays on
all platforms, only the preamble assertion is gated to Linux/macOS.
This commit is contained in:
Mohamed Boudra
2026-06-13 05:38:58 +00:00
committed by GitHub
parent 442a1b9a5a
commit 72b67f48e3
19 changed files with 1693 additions and 70 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/app/src/terminal/runtime/terminal-emulator-runtime.test.ts
+169 -28
View File
@@ -104,13 +104,34 @@ function createRuntimeWithTerminal(): {
writeTexts: string[];
} {
const runtime = new TerminalEmulatorRuntime();
const terminalState = attachStubTerminal(runtime);
return {
runtime,
...terminalState,
};
}
function attachStubTerminal(runtime: TerminalEmulatorRuntime): {
terminal: StubTerminal & {
resetCalls: number;
};
writeCallbacks: Array<() => void>;
writeTexts: string[];
} {
const writeCallbacks: Array<() => void> = [];
const writeTexts: string[] = [];
let resetCalls = 0;
const terminal: StubTerminal & { resetCalls: number } = {
write: (data: string | Uint8Array, callback?: () => void) => {
writeTexts.push(decodeTerminalOutput(data));
const text = decodeTerminalOutput(data);
// The runtime submits a zero-length sentinel write to gate barrier ops behind the
// drained write run. Keep its callback (the gate resolves through it) but exclude the
// empty payload from writeTexts so assertions read like real terminal output.
if (text.length > 0) {
writeTexts.push(text);
}
if (callback) {
writeCallbacks.push(callback);
}
@@ -131,7 +152,6 @@ function createRuntimeWithTerminal(): {
(runtime as unknown as { terminal: StubTerminal }).terminal = terminal;
return {
runtime,
terminal,
writeCallbacks,
writeTexts,
@@ -164,7 +184,7 @@ describe("terminal-emulator-runtime", () => {
vi.useRealTimers();
});
it("processes write and clear operations in strict order", () => {
it("drains contiguous plain writes without waiting for each commit, gating a clear behind them", () => {
const { runtime, terminal, writeCallbacks, writeTexts } = createRuntimeWithTerminal();
const committed: string[] = [];
@@ -174,43 +194,64 @@ describe("terminal-emulator-runtime", () => {
committed.push("first");
},
});
runtime.clear({
onCommitted: () => {
committed.push("clear");
},
});
runtime.write({
data: terminalOutput("second"),
onCommitted: () => {
committed.push("second");
},
});
runtime.clear({
onCommitted: () => {
committed.push("clear");
},
});
runtime.write({
data: terminalOutput("third"),
onCommitted: () => {
committed.push("third");
},
});
expect(writeTexts).toEqual(["first"]);
// Both plain writes are submitted back-to-back; neither waited on the other's callback.
// The clear and the write after it stay queued behind the barrier gate (sentinel write).
expect(writeTexts).toEqual(["first", "second"]);
expect(terminal.resetCalls).toBe(0);
expect(committed).toEqual([]);
// writeCallbacks: [0]=first, [1]=second, [2]=barrier-gate sentinel.
writeCallbacks[0]?.();
expect(committed).toEqual(["first", "clear"]);
expect(terminal.resetCalls).toBe(1);
expect(writeTexts).toEqual(["first", "second"]);
writeCallbacks[1]?.();
expect(committed).toEqual(["first", "clear", "second"]);
expect(committed).toEqual(["first", "second"]);
expect(terminal.resetCalls).toBe(0);
// Resolving the sentinel releases the clear, which resets and then drains "third".
writeCallbacks[2]?.();
expect(committed).toEqual(["first", "second", "clear"]);
expect(terminal.resetCalls).toBe(1);
expect(writeTexts).toEqual(["first", "second", "third"]);
// "third" still commits through its own write callback.
writeCallbacks[3]?.();
expect(committed).toEqual(["first", "second", "clear", "third"]);
});
it("falls back to timeout commit when xterm write callback does not fire", () => {
it("falls back to timeout commit for a barrier op when the gate sentinel never fires", () => {
vi.useFakeTimers();
const { runtime } = createRuntimeWithTerminal();
const { runtime, terminal } = createRuntimeWithTerminal();
const onCommitted = vi.fn();
runtime.write({
// restoreOutput is a barrier; if xterm never commits the gate sentinel, the barrier still
// applies (and commits) after the 5s safety timeout instead of stalling forever.
runtime.restoreOutput({
data: terminalOutput("stuck"),
onCommitted,
});
expect(onCommitted).not.toHaveBeenCalled();
expect(terminal.resetCalls).toBe(0);
vi.advanceTimersByTime(5_000);
// The barrier now runs (write submitted) but its own commit still waits on xterm; advance
// again to fire the barrier's own write timeout.
vi.advanceTimersByTime(5_000);
expect(onCommitted).toHaveBeenCalledTimes(1);
});
@@ -239,6 +280,12 @@ describe("terminal-emulator-runtime", () => {
},
},
});
// The plain write reports kitty flags synchronously during drain; the snapshot resets
// them only after its barrier gate (the sentinel write callback) resolves.
expect(inputModeChanges).toEqual([{ kittyKeyboardFlags: 7, win32InputMode: false }]);
// The plain write carries no onCommitted, so it registers no callback; writeCallbacks[0]
// is the barrier gate sentinel.
writeCallbacks[0]?.();
expect(inputModeChanges).toEqual([
@@ -247,8 +294,8 @@ describe("terminal-emulator-runtime", () => {
]);
});
it("ignores stale duplicate write callbacks from a previous operation", () => {
const { runtime, writeCallbacks } = createRuntimeWithTerminal();
it("commits each drained plain write through its own xterm callback", () => {
const { runtime, writeTexts, writeCallbacks } = createRuntimeWithTerminal();
const committed: string[] = [];
runtime.write({
@@ -264,14 +311,89 @@ describe("terminal-emulator-runtime", () => {
},
});
writeCallbacks[0]?.();
expect(committed).toEqual(["first"]);
writeCallbacks[0]?.();
expect(committed).toEqual(["first"]);
// Both submitted without waiting; each commits independently when its callback fires.
expect(writeTexts).toEqual(["first", "second"]);
expect(committed).toEqual([]);
writeCallbacks[1]?.();
expect(committed).toEqual(["first", "second"]);
expect(committed).toEqual(["second"]);
writeCallbacks[0]?.();
expect(committed).toEqual(["second", "first"]);
});
it("applies a snapshot after the writes ahead of it, never interleaving the reset", () => {
const { runtime, terminal, writeTexts, writeCallbacks } = createRuntimeWithTerminal();
const committed: string[] = [];
runtime.write({
data: terminalOutput("before-a"),
onCommitted: () => {
committed.push("before-a");
},
});
runtime.write({
data: terminalOutput("before-b"),
onCommitted: () => {
committed.push("before-b");
},
});
runtime.restoreOutput({
data: terminalOutput("snap"),
onCommitted: () => {
committed.push("snap");
},
});
// Plain writes are submitted up front; the snapshot's reset+write has not run yet.
expect(writeTexts).toEqual(["before-a", "before-b"]);
// writeCallbacks: [0]=before-a, [1]=before-b, [2]=barrier gate sentinel.
// The snapshot is gated behind the prior writes; resolving the gate runs it after them.
writeCallbacks[2]?.();
expect(writeTexts).toEqual(["before-a", "before-b", "csnap"]);
// Snapshot commit fires only after its own write callback, strictly after the writes.
writeCallbacks[3]?.();
expect(committed).toEqual(["snap"]);
expect(terminal.resetCalls).toBe(0);
});
it("applies a barrier immediately when no writes precede it, suppressing input at once", () => {
const { runtime, writeCallbacks } = createRuntimeWithTerminal();
const readSuppressInput = () =>
(runtime as unknown as { suppressInput: boolean }).suppressInput;
expect(readSuppressInput()).toBe(false);
// No plain writes precede this barrier (mount), so there is nothing to gate: it starts
// at once with no sentinel, flipping suppressInput synchronously.
runtime.restoreOutput({ data: terminalOutput("snapshot") });
expect(readSuppressInput()).toBe(true);
// writeCallbacks[0] is the barrier's own snapshot write; committing it restores input.
writeCallbacks[0]?.();
expect(readSuppressInput()).toBe(false);
});
it("gates a barrier behind a preceding plain write before suppressing input", () => {
const { runtime, writeCallbacks } = createRuntimeWithTerminal();
const readSuppressInput = () =>
(runtime as unknown as { suppressInput: boolean }).suppressInput;
// A plain write is now ungated, so the following barrier must wait on the sentinel.
runtime.write({ data: terminalOutput("output") });
runtime.restoreOutput({ data: terminalOutput("snapshot") });
// The plain write carries no onCommitted so it registers no callback; writeCallbacks[0]
// is the sentinel gate. suppressInput only flips once the gate resolves the barrier.
expect(readSuppressInput()).toBe(false);
writeCallbacks[0]?.();
expect(readSuppressInput()).toBe(true);
// writeCallbacks[1] is the barrier's own snapshot write; committing it restores input.
writeCallbacks[1]?.();
expect(readSuppressInput()).toBe(false);
});
it("commits pending output operations during unmount to avoid deadlock", () => {
@@ -294,8 +416,21 @@ describe("terminal-emulator-runtime", () => {
expect(onCommittedB).toHaveBeenCalledTimes(1);
});
it("clears ungated writes on unmount so remount barriers apply immediately", () => {
const { runtime } = createRuntimeWithTerminal();
runtime.write({ data: terminalOutput("before unmount") });
runtime.unmount();
const remounted = attachStubTerminal(runtime);
runtime.restoreOutput({ data: terminalOutput("restored screen") });
expect(remounted.writeTexts).toEqual(["\u001bcrestored screen"]);
expect(remounted.writeCallbacks).toHaveLength(1);
});
it("replays snapshots through a single write without first painting a reset terminal", () => {
const { runtime, terminal, writeTexts } = createRuntimeWithTerminal();
const { runtime, terminal, writeTexts, writeCallbacks } = createRuntimeWithTerminal();
runtime.renderSnapshot({
state: {
@@ -313,6 +448,9 @@ describe("terminal-emulator-runtime", () => {
},
});
// The snapshot is a barrier; its write applies after the gate sentinel (writeCallbacks[0]).
writeCallbacks[0]?.();
expect(terminal.resetCalls).toBe(0);
expect(writeTexts).toHaveLength(1);
expect(writeTexts[0]?.startsWith("\u001bc")).toBe(true);
@@ -320,10 +458,13 @@ describe("terminal-emulator-runtime", () => {
});
it("restores server-rendered ANSI snapshots through the snapshot write path", () => {
const { runtime, terminal, writeTexts } = createRuntimeWithTerminal();
const { runtime, terminal, writeTexts, writeCallbacks } = createRuntimeWithTerminal();
runtime.restoreOutput({ data: terminalOutput("restored screen") });
// restoreOutput is a barrier (suppressInput); it applies after the gate sentinel.
writeCallbacks[0]?.();
expect(terminal.resetCalls).toBe(0);
expect(writeTexts).toEqual(["\u001bcrestored screen"]);
});
packages/app/src/terminal/runtime/terminal-emulator-runtime.ts
+98
View File
@@ -184,6 +184,15 @@ export class TerminalEmulatorRuntime {
private outputOperations: TerminalOutputOperation[] = [];
private inFlightOutputOperation: TerminalOutputOperation | null = null;
private inFlightOutputOperationTimeout: ReturnType<typeof setTimeout> | null = null;
// Plain writes are submitted to xterm without waiting, so their onCommitted callbacks are
// pending until xterm commits the write. unmount() disposes xterm before those fire, so we
// track them here to flush remaining callbacks and avoid stalling upstream backpressure.
private pendingWriteCommits = new Set<() => void>();
// True once a plain write has been submitted to xterm that no later barrier has gated
// yet. A barrier only needs the sentinel-write gate when this is set; at mount or right
// after another barrier it's false and the barrier applies immediately, saving a parse
// cycle of latency. Cleared when a barrier starts (it gates every write before it).
private hasUngatedWrites = false;
private readonly inputModeDecoder = new TextDecoder();
private suppressInput = false;
private readonly inputModeTracker = new TerminalInputModeTracker();
@@ -759,6 +768,10 @@ export class TerminalEmulatorRuntime {
for (const operation of pendingOperations) {
operation.onCommitted?.();
}
for (const commit of Array.from(this.pendingWriteCommits)) {
commit();
}
this.hasUngatedWrites = false;
this.cleanup?.();
this.cleanup = null;
@@ -776,6 +789,13 @@ export class TerminalEmulatorRuntime {
this.emitInputModeChange();
}
// A barrier op mutates terminal geometry or toggles suppressInput and must apply in
// isolation: clear/snapshot reset/resize the terminal, and any write carrying
// suppressInput flips input handling around the commit. Plain writes are fast-pathed.
private isBarrierOperation(operation: TerminalOutputOperation): boolean {
return operation.type !== "write" || Boolean(operation.suppressInput);
}
private processOutputQueue(): void {
if (this.inFlightOutputOperation) {
return;
@@ -786,12 +806,90 @@ export class TerminalEmulatorRuntime {
return;
}
// Fast path: drain contiguous plain writes back-to-back without waiting for each
// write's completion callback. xterm buffers internally and parses in submission
// order, so serializing one frame per parse tick only adds latency under burst.
while (this.outputOperations[0] && !this.isBarrierOperation(this.outputOperations[0])) {
const writeOperation = this.outputOperations.shift();
if (!writeOperation) {
break;
}
this.submitWrite(terminal, writeOperation);
}
const operation = this.outputOperations.shift();
if (!operation) {
return;
}
// The next op is a barrier. Before clear()/reset()/resize() touches the terminal, any
// ungated plain writes still parsing in xterm's buffer must finish — otherwise a
// synchronous reset could interleave with them. When none are ungated (mount, or right
// after another barrier) there is nothing to wait for, so apply the barrier at once.
if (!this.hasUngatedWrites) {
this.startBarrierOperation(terminal, operation);
return;
}
// Otherwise a zero-length sentinel write commits after every outstanding write, so
// waiting on its callback gates the barrier behind them. Plain writes never wait.
let started = false;
const startBarrier = () => {
// unmount() clears the in-flight op and disposes the terminal while we wait on the
// sentinel; bail if either changed so we don't reset/resize a torn-down terminal.
if (started || this.inFlightOutputOperation !== operation || this.terminal !== terminal) {
return;
}
started = true;
this.clearInFlightOutputTimeout();
this.startBarrierOperation(terminal, operation);
};
this.inFlightOutputOperation = operation;
this.inFlightOutputOperationTimeout = setTimeout(startBarrier, OUTPUT_OPERATION_TIMEOUT_MS);
try {
terminal.write(EMPTY_TERMINAL_OUTPUT, startBarrier);
} catch {
startBarrier();
}
}
private submitWrite(terminal: Terminal, operation: TerminalOutputOperation): void {
// Synchronous per-write tracking must run in frame order; doing it here in the drain
// loop preserves that ordering even though the writes are submitted without waiting.
const text = this.inputModeDecoder.decode(operation.data, { stream: true });
const result = this.inputModeTracker.feed(text);
if (result.changed) {
this.emitInputModeChange();
}
this.hasUngatedWrites = true;
const onCommitted = operation.onCommitted;
if (!onCommitted) {
try {
terminal.write(operation.data);
} catch {
// Match existing behavior: a failed write still proceeds with no commit callback.
}
return;
}
const commit = () => {
if (!this.pendingWriteCommits.delete(commit)) {
return;
}
onCommitted();
};
this.pendingWriteCommits.add(commit);
try {
terminal.write(operation.data, commit);
} catch {
commit();
}
}
private startBarrierOperation(terminal: Terminal, operation: TerminalOutputOperation): void {
this.inFlightOutputOperation = operation;
// This barrier gates every write submitted before it (the sentinel guaranteed they
// parsed, or there were none), so the next barrier can skip the sentinel until the
// next write arrives.
this.hasUngatedWrites = false;
const previousSuppressInput = this.suppressInput;
if (operation.suppressInput) {
this.suppressInput = Boolean(operation.suppressInput);