Eliminate spiky terminal lag under load (#1500)

* Eliminate spiky terminal lag under load

Terminal output stuttered during heavy output and when the daemon was
busy. Two compounding causes, both confirmed by measurement:

- Every 256KB of output, the daemon dropped pending frames and re-sent a
  full cell-grid snapshot (cloned across IPC, stringified to JSON) — a
  10MB build did this ~40 times. The snapshot fallback is now gated on
  real client backpressure (ws bufferedAmount), so a client that keeps
  draining streams continuously and never pays the snapshot tax. This
  also removes the periodic GC hitch the churn caused.
- Per-chunk overhead on the shared event loop: one IPC message per pty
  chunk, a duplicate input-mode regex scan on the daemon main loop, a
  double JSON.stringify per outbound message, and 16KB string realloc
  per chunk. Output now coalesces in the worker before IPC (leading-edge
  so keystrokes still echo immediately), the duplicate scan is gone, and
  the client feeds xterm back-to-back instead of one render tick a frame.

Adds eventLoopDelay percentiles to ws_runtime_metrics for main-loop
stall visibility, plus a reproducible Node benchmark (no port 6767).

Measured: echo p50 7.7ms to 2.3ms; a 2MB burst now streams fully with
zero snapshots; loop-stall spikes 100-173ms to 2ms.

* Guard trace-level check against partial logger stubs

The isLevelEnabled gate added for the emit() perf fix crashed every
session test that injects a hand-rolled logger stub (10 of them omit
isLevelEnabled). Real pino loggers always provide it; optional-chaining
keeps the perf gate intact in production and no-ops on stubs.

* Make recent-output exit-summary test deterministic

The added test self-exited the child immediately after writing 3000
lines, then asserted the newest line was in the exit summary — but the
summary reads the headless xterm buffer, which parses writes
asynchronously, so a loaded CI runner saw a stale buffer (line-2707 not
line-2999). Keep the process alive and poll the parsed buffer until the
final line lands before killing, removing the race.

* Skip ConPTY-fragile worker terminal tests on Windows

The coalescing and input-mode-preamble tests assert byte-contiguous PTY
output and an exact kitty-escape round-trip. Windows ConPTY injects
repaint sequences between writes and normalizes the escape, so both time
out there while passing on Linux/macOS. Gate them with skipIf(win32),
matching the existing terminal-test convention for PTY-sensitive cases.

* Apply terminal barriers immediately when no writes are pending

A barrier op (snapshot/restore/clear) only needs the sentinel-write gate
to wait out plain writes still parsing in xterm's buffer. When none are
ungated — at mount, or right after another barrier — the sentinel cost a
wasted parse cycle, adding latency to the first snapshot/restore on the
hot first-paint path. Track ungated writes with a flag and skip the
sentinel when there's nothing to gate.

* Fix mobile startup saved-host e2e

* Unskip Windows worker terminal coverage

* Tighten worker coalescing assertion

* Reset terminal ungated writes on unmount

* Fix terminal regressions found in adversarial review

Three issues this PR's terminal changes introduced:

- Worker snapshot could duplicate coalesced output. getTerminalState
  snapshotted without flushing the worker output coalescer, so a batch
  spanning the snapshot point carried a revision past it and the
  controller's dedup couldn't drop it — the client saw the bytes twice.
  Flush before snapshotting.
- Relay clients lost backpressure protection. The snapshot fallback was
  gated on bufferedAmount, which the multiplexed relay socket reports as
  absent; that read as 0 ("keeping up") so a slow relay client never
  caught up via a snapshot. Distinguish "no signal" (null) from 0 and
  keep the unconditional byte-threshold fallback for signal-less
  transports, preserving the pre-change relay behavior.
- Recent-output buffer was no longer a hard cap. A single chunk larger
  than the limit was retained whole; slice its tail.

Documents the live-restore preamble gap (unreachable: no client sends
restore mode "live").

* Skip input-mode preamble test on Windows ConPTY

CI confirmed Windows ConPTY normalizes away the kitty keyboard escape
the child writes, so it never reaches the worker's input-mode tracker
and the preamble stays empty — the test times out. ConPTY can't exercise
this contract; the coalescing test (file-gated, line-oriented) stays on
all platforms, only the preamble assertion is gated to Linux/macOS.

packages/server/src/terminal/terminal-worker-process.ts

@@ -1,14 +1,17 @@
 import { createTerminalManager } from "./terminal-manager.js";
 import { captureTerminalLines } from "./terminal-capture.js";
-import type { TerminalSession } from "./terminal.js";
+import { TerminalOutputCoalescer } from "./terminal-output-coalescer.js";
+import type { TerminalSession, TerminalStateSnapshotOptions } from "./terminal.js";
 import type {
   TerminalWorkerRequest,
+  TerminalWorkerStateResult,
   TerminalWorkerToParentMessage,
   WorkerTerminalInfo,
 } from "./terminal-worker-protocol.js";
 
 const manager = createTerminalManager();
 const unsubscribeByTerminalId = new Map<string, Array<() => void>>();
+const outputCoalescerByTerminalId = new Map<string, TerminalOutputCoalescer>();
 let ipcClosing = false;
 
 function sendToParent(message: TerminalWorkerToParentMessage): void {
@@ -26,6 +29,16 @@ function sendToParent(message: TerminalWorkerToParentMessage): void {
   }
 }
 
+function buildTerminalStateResult(
+  session: TerminalSession | undefined,
+  options?: TerminalStateSnapshotOptions,
+): TerminalWorkerStateResult {
+  if (!session) {
+    return null;
+  }
+  return { ...session.getStateSnapshot(options), replayPreamble: session.getReplayPreamble() };
+}
+
 function toTerminalInfo(session: TerminalSession): WorkerTerminalInfo {
   return {
     id: session.id,
@@ -47,12 +60,44 @@ function clearTerminalSubscriptions(terminalId: string): void {
     }
   }
   unsubscribeByTerminalId.delete(terminalId);
+  const coalescer = outputCoalescerByTerminalId.get(terminalId);
+  if (coalescer) {
+    coalescer.dispose();
+    outputCoalescerByTerminalId.delete(terminalId);
+  }
 }
 
 function watchTerminal(session: TerminalSession): void {
   clearTerminalSubscriptions(session.id);
 
+  // Coalesce pty output chunks into a single IPC message per ~5ms window so a
+  // burst of small chunks no longer costs one process.send each. The batch
+  // carries the LAST chunk's revision (the highest) so downstream snapshot
+  // replay dedup stays correct.
+  let pendingOutputRevision: number | undefined;
+  const outputCoalescer = new TerminalOutputCoalescer({
+    timers: { setTimeout, clearTimeout },
+    onFlush: ({ payload }) => {
+      const revision = pendingOutputRevision;
+      pendingOutputRevision = undefined;
+      sendToParent({
+        type: "terminalMessage",
+        terminalId: session.id,
+        message: { type: "output", data: payload.toString("utf8"), revision },
+      });
+    },
+  });
+  outputCoalescerByTerminalId.set(session.id, outputCoalescer);
+
   const unsubscribeMessage = session.subscribe((message) => {
+    if (message.type === "output") {
+      pendingOutputRevision = message.revision;
+      outputCoalescer.handle(message.data);
+      return;
+    }
+    // Non-output messages (snapshot/snapshotReady/titleChange) must not jump
+    // ahead of buffered output: flush the coalescer first, then forward.
+    outputCoalescer.flush();
     sendToParent({
       type: "terminalMessage",
       terminalId: session.id,
@@ -60,6 +105,7 @@ function watchTerminal(session: TerminalSession): void {
     });
   });
   const unsubscribeExit = session.onExit((info) => {
+    outputCoalescer.flush();
     clearTerminalSubscriptions(session.id);
     sendToParent({
       type: "terminalExit",
@@ -68,6 +114,7 @@ function watchTerminal(session: TerminalSession): void {
     });
   });
   const unsubscribeTitle = session.onTitleChange((title) => {
+    outputCoalescer.flush();
     sendToParent({
       type: "terminalTitleChange",
       terminalId: session.id,
@@ -75,6 +122,7 @@ function watchTerminal(session: TerminalSession): void {
     });
   });
   const unsubscribeCommandFinished = session.onCommandFinished((info) => {
+    outputCoalescer.flush();
     sendToParent({
       type: "terminalCommandFinished",
       terminalId: session.id,
@@ -171,12 +219,16 @@ async function handleRequest(message: TerminalWorkerRequest): Promise<void> {
     }
 
     case "getTerminalState": {
-      const session = manager.getTerminal(message.terminalId);
+      // Flush buffered output before snapshotting: the headless state already includes it,
+      // so if the coalescer emitted it afterward (in a batch carrying a revision past the
+      // snapshot's) the controller's revision dedup wouldn't drop it and the client would
+      // see the bytes twice. Flushing first sends them with a revision <= the snapshot's.
+      outputCoalescerByTerminalId.get(message.terminalId)?.flush();
       sendToParent({
         type: "response",
         requestId: message.requestId,
         ok: true,
-        result: session?.getStateSnapshot(message.options) ?? null,
+        result: buildTerminalStateResult(manager.getTerminal(message.terminalId), message.options),
       });
       return;
     }