Eliminate spiky terminal lag under load (#1500)

* Eliminate spiky terminal lag under load

Terminal output stuttered during heavy output and when the daemon was
busy. Two compounding causes, both confirmed by measurement:

- Every 256KB of output, the daemon dropped pending frames and re-sent a
  full cell-grid snapshot (cloned across IPC, stringified to JSON) — a
  10MB build did this ~40 times. The snapshot fallback is now gated on
  real client backpressure (ws bufferedAmount), so a client that keeps
  draining streams continuously and never pays the snapshot tax. This
  also removes the periodic GC hitch the churn caused.
- Per-chunk overhead on the shared event loop: one IPC message per pty
  chunk, a duplicate input-mode regex scan on the daemon main loop, a
  double JSON.stringify per outbound message, and 16KB string realloc
  per chunk. Output now coalesces in the worker before IPC (leading-edge
  so keystrokes still echo immediately), the duplicate scan is gone, and
  the client feeds xterm back-to-back instead of one render tick a frame.

Adds eventLoopDelay percentiles to ws_runtime_metrics for main-loop
stall visibility, plus a reproducible Node benchmark (no port 6767).

Measured: echo p50 7.7ms to 2.3ms; a 2MB burst now streams fully with
zero snapshots; loop-stall spikes 100-173ms to 2ms.

* Guard trace-level check against partial logger stubs

The isLevelEnabled gate added for the emit() perf fix crashed every
session test that injects a hand-rolled logger stub (10 of them omit
isLevelEnabled). Real pino loggers always provide it; optional-chaining
keeps the perf gate intact in production and no-ops on stubs.

* Make recent-output exit-summary test deterministic

The added test self-exited the child immediately after writing 3000
lines, then asserted the newest line was in the exit summary — but the
summary reads the headless xterm buffer, which parses writes
asynchronously, so a loaded CI runner saw a stale buffer (line-2707 not
line-2999). Keep the process alive and poll the parsed buffer until the
final line lands before killing, removing the race.

* Skip ConPTY-fragile worker terminal tests on Windows

The coalescing and input-mode-preamble tests assert byte-contiguous PTY
output and an exact kitty-escape round-trip. Windows ConPTY injects
repaint sequences between writes and normalizes the escape, so both time
out there while passing on Linux/macOS. Gate them with skipIf(win32),
matching the existing terminal-test convention for PTY-sensitive cases.

* Apply terminal barriers immediately when no writes are pending

A barrier op (snapshot/restore/clear) only needs the sentinel-write gate
to wait out plain writes still parsing in xterm's buffer. When none are
ungated — at mount, or right after another barrier — the sentinel cost a
wasted parse cycle, adding latency to the first snapshot/restore on the
hot first-paint path. Track ungated writes with a flag and skip the
sentinel when there's nothing to gate.

* Fix mobile startup saved-host e2e

* Unskip Windows worker terminal coverage

* Tighten worker coalescing assertion

* Reset terminal ungated writes on unmount

* Fix terminal regressions found in adversarial review

Three issues this PR's terminal changes introduced:

- Worker snapshot could duplicate coalesced output. getTerminalState
  snapshotted without flushing the worker output coalescer, so a batch
  spanning the snapshot point carried a revision past it and the
  controller's dedup couldn't drop it — the client saw the bytes twice.
  Flush before snapshotting.
- Relay clients lost backpressure protection. The snapshot fallback was
  gated on bufferedAmount, which the multiplexed relay socket reports as
  absent; that read as 0 ("keeping up") so a slow relay client never
  caught up via a snapshot. Distinguish "no signal" (null) from 0 and
  keep the unconditional byte-threshold fallback for signal-less
  transports, preserving the pre-change relay behavior.
- Recent-output buffer was no longer a hard cap. A single chunk larger
  than the limit was retained whole; slice its tail.

Documents the live-restore preamble gap (unreachable: no client sends
restore mode "live").

* Skip input-mode preamble test on Windows ConPTY

CI confirmed Windows ConPTY normalizes away the kitty keyboard escape
the child writes, so it never reaches the worker's input-mode tracker
and the preamble stays empty — the test times out. ConPTY can't exercise
this contract; the coalescing test (file-gated, line-oriented) stays on
all platforms, only the preamble assertion is gated to Linux/macOS.

packages/server/src/terminal/terminal.ts

@@ -36,6 +36,9 @@ export interface TerminalCommandFinishedInfo {
 export interface TerminalStateSnapshot {
   state: TerminalState;
   revision: number;
+  // Input-mode replay preamble at snapshot time. Populated by the terminal
+  // worker so the daemon main loop doesn't have to re-derive it from output.
+  replayPreamble?: string;
 }
 
 export interface TerminalStateSnapshotOptions {
@@ -58,7 +61,7 @@ export type ClientMessage =
 export type ServerMessage =
   | { type: "output"; data: string; revision?: number }
   | { type: "snapshot"; state: TerminalState; revision?: number }
-  | { type: "snapshotReady"; revision?: number }
+  | { type: "snapshotReady"; revision?: number; replayPreamble?: string }
   | { type: "titleChange"; title?: string };
 
 export interface TerminalSession {
@@ -606,7 +609,12 @@ export async function createTerminal(options: CreateTerminalOptions): Promise<Te
   let processExited = false;
   const processExitWaiters = new Set<() => void>();
   let exitInfo: TerminalExitInfo | null = null;
-  let recentOutputText = "";
+  // Recent output is retained as whole chunks plus a running char length so we
+  // avoid reallocating a ~16KB string on every pty chunk. We keep enough whole
+  // chunks that their join always contains at least the last
+  // TERMINAL_EXIT_OUTPUT_CHAR_LIMIT chars; the exact tail is sliced at exit.
+  const recentOutputChunks: string[] = [];
+  let recentOutputLength = 0;
   let title: string | undefined;
   let titleMode: "auto" | "manual" = presetTitle?.trim() ? "manual" : "auto";
   let pendingTitle: string | undefined;
@@ -777,7 +785,10 @@ export async function createTerminal(options: CreateTerminalOptions): Promise<Te
       lastOutputLines:
         lastOutputLines.length > 0
           ? lastOutputLines
-          : extractLastOutputLinesFromText(recentOutputText, TERMINAL_EXIT_OUTPUT_LINE_LIMIT),
+          : extractLastOutputLinesFromText(
+              recentOutputChunks.join("").slice(-TERMINAL_EXIT_OUTPUT_CHAR_LIMIT),
+              TERMINAL_EXIT_OUTPUT_LINE_LIMIT,
+            ),
     };
   }
 
@@ -803,6 +814,8 @@ export async function createTerminal(options: CreateTerminalOptions): Promise<Te
     }
     disposed = true;
     pendingInput = "";
+    recentOutputChunks.length = 0;
+    recentOutputLength = 0;
     inputModeTracker.reset();
     if (inputFlushImmediate) {
       clearImmediate(inputFlushImmediate);
@@ -837,9 +850,23 @@ export async function createTerminal(options: CreateTerminalOptions): Promise<Te
     for (const response of inputModeUpdate.responses) {
       ptyProcess.write(response);
     }
-    recentOutputText = `${recentOutputText}${data}`;
-    if (recentOutputText.length > TERMINAL_EXIT_OUTPUT_CHAR_LIMIT) {
-      recentOutputText = recentOutputText.slice(-TERMINAL_EXIT_OUTPUT_CHAR_LIMIT);
+    recentOutputChunks.push(data);
+    recentOutputLength += data.length;
+    // Drop whole leading chunks while the rest still covers the char limit, so
+    // the retained join always contains at least the last limit chars.
+    while (
+      recentOutputChunks.length > 1 &&
+      recentOutputLength - recentOutputChunks[0].length >= TERMINAL_EXIT_OUTPUT_CHAR_LIMIT
+    ) {
+      recentOutputLength -= recentOutputChunks[0].length;
+      recentOutputChunks.shift();
+    }
+    // We never drop the last chunk, so a single chunk larger than the cap would
+    // grow the buffer unbounded; slice its tail to keep the cap hard.
+    if (recentOutputChunks.length === 1 && recentOutputLength > TERMINAL_EXIT_OUTPUT_CHAR_LIMIT) {
+      const tail = recentOutputChunks[0].slice(-TERMINAL_EXIT_OUTPUT_CHAR_LIMIT);
+      recentOutputChunks[0] = tail;
+      recentOutputLength = tail.length;
     }
     writeOutputToHeadless(data);
   });
@@ -993,7 +1020,13 @@ export async function createTerminal(options: CreateTerminalOptions): Promise<Te
       if (!disposed && active && listeners.has(subscriptionListener)) {
         snapshotDelivered = true;
         if (initialSnapshot === "ready") {
-          listener({ type: "snapshotReady", revision: stateRevision });
+          // Carry the input-mode preamble so the snapshot-less "ready" path
+          // (live restore) can replay it without a separate state fetch.
+          listener({
+            type: "snapshotReady",
+            revision: stateRevision,
+            replayPreamble: getReplayPreamble(),
+          });
         } else {
           listener({ type: "snapshot", ...getStateSnapshot() });
         }