Eliminate spiky terminal lag under load (#1500)

* Eliminate spiky terminal lag under load

Terminal output stuttered during heavy output and when the daemon was
busy. Two compounding causes, both confirmed by measurement:

- Every 256KB of output, the daemon dropped pending frames and re-sent a
  full cell-grid snapshot (cloned across IPC, stringified to JSON) — a
  10MB build did this ~40 times. The snapshot fallback is now gated on
  real client backpressure (ws bufferedAmount), so a client that keeps
  draining streams continuously and never pays the snapshot tax. This
  also removes the periodic GC hitch the churn caused.
- Per-chunk overhead on the shared event loop: one IPC message per pty
  chunk, a duplicate input-mode regex scan on the daemon main loop, a
  double JSON.stringify per outbound message, and 16KB string realloc
  per chunk. Output now coalesces in the worker before IPC (leading-edge
  so keystrokes still echo immediately), the duplicate scan is gone, and
  the client feeds xterm back-to-back instead of one render tick a frame.

Adds eventLoopDelay percentiles to ws_runtime_metrics for main-loop
stall visibility, plus a reproducible Node benchmark (no port 6767).

Measured: echo p50 7.7ms to 2.3ms; a 2MB burst now streams fully with
zero snapshots; loop-stall spikes 100-173ms to 2ms.

* Guard trace-level check against partial logger stubs

The isLevelEnabled gate added for the emit() perf fix crashed every
session test that injects a hand-rolled logger stub (10 of them omit
isLevelEnabled). Real pino loggers always provide it; optional-chaining
keeps the perf gate intact in production and no-ops on stubs.

* Make recent-output exit-summary test deterministic

The added test self-exited the child immediately after writing 3000
lines, then asserted the newest line was in the exit summary — but the
summary reads the headless xterm buffer, which parses writes
asynchronously, so a loaded CI runner saw a stale buffer (line-2707 not
line-2999). Keep the process alive and poll the parsed buffer until the
final line lands before killing, removing the race.

* Skip ConPTY-fragile worker terminal tests on Windows

The coalescing and input-mode-preamble tests assert byte-contiguous PTY
output and an exact kitty-escape round-trip. Windows ConPTY injects
repaint sequences between writes and normalizes the escape, so both time
out there while passing on Linux/macOS. Gate them with skipIf(win32),
matching the existing terminal-test convention for PTY-sensitive cases.

* Apply terminal barriers immediately when no writes are pending

A barrier op (snapshot/restore/clear) only needs the sentinel-write gate
to wait out plain writes still parsing in xterm's buffer. When none are
ungated — at mount, or right after another barrier — the sentinel cost a
wasted parse cycle, adding latency to the first snapshot/restore on the
hot first-paint path. Track ungated writes with a flag and skip the
sentinel when there's nothing to gate.

* Fix mobile startup saved-host e2e

* Unskip Windows worker terminal coverage

* Tighten worker coalescing assertion

* Reset terminal ungated writes on unmount

* Fix terminal regressions found in adversarial review

Three issues this PR's terminal changes introduced:

- Worker snapshot could duplicate coalesced output. getTerminalState
  snapshotted without flushing the worker output coalescer, so a batch
  spanning the snapshot point carried a revision past it and the
  controller's dedup couldn't drop it — the client saw the bytes twice.
  Flush before snapshotting.
- Relay clients lost backpressure protection. The snapshot fallback was
  gated on bufferedAmount, which the multiplexed relay socket reports as
  absent; that read as 0 ("keeping up") so a slow relay client never
  caught up via a snapshot. Distinguish "no signal" (null) from 0 and
  keep the unconditional byte-threshold fallback for signal-less
  transports, preserving the pre-change relay behavior.
- Recent-output buffer was no longer a hard cap. A single chunk larger
  than the limit was retained whole; slice its tail.

Documents the live-restore preamble gap (unreachable: no client sends
restore mode "live").

* Skip input-mode preamble test on Windows ConPTY

CI confirmed Windows ConPTY normalizes away the kitty keyboard escape
the child writes, so it never reaches the worker's input-mode tracker
and the preamble stays empty — the test times out. ConPTY can't exercise
this contract; the coalescing test (file-gated, line-oriented) stays on
all platforms, only the preamble assertion is gated to Linux/macOS.

packages/server/src/terminal/terminal-session-controller.test.ts

@@ -168,6 +168,12 @@ async function flushMicrotasks(): Promise<void> {
   await new Promise((resolve) => setTimeout(resolve, 0));
 }
 
+// The coalescer drains on a 5ms trailing timer; wait past it (and any async
+// snapshot round-trip it kicks off) before asserting on emitted frames.
+async function waitForCoalescerFlush(): Promise<void> {
+  await new Promise((resolve) => setTimeout(resolve, 30));
+}
+
 describe("terminal-session-controller wrap-flag gating", () => {
   function setup(clientSupportsWrapReflow?: () => boolean): {
     controller: TerminalSessionController;
@@ -387,3 +393,133 @@ describe("terminal-session-controller subdirectory aggregation", () => {
     ]);
   });
 });
+
+describe("terminal-session-controller backpressure snapshot fallback", () => {
+  async function setup(getClientBufferedAmount: () => number | null): Promise<{
+    pushOutput: (data: string) => void;
+    frames: TerminalStreamFrame[];
+  }> {
+    let terminalListener: ((message: ServerMessage) => void) | null = null;
+    const terminal: TerminalSession = {
+      id: "term-1",
+      name: "Terminal",
+      cwd: "/tmp",
+      send: vi.fn(),
+      subscribe: (listener) => {
+        terminalListener = listener;
+        // Legacy stream: a snapshot arrives on subscribe (one Snapshot frame),
+        // after which output streams through the coalescer as Output frames.
+        queueMicrotask(() =>
+          listener({ type: "snapshot", state: terminalState("live"), revision: 1 }),
+        );
+        return vi.fn();
+      },
+      onExit: () => vi.fn(),
+      onCommandFinished: () => vi.fn(),
+      onTitleChange: () => vi.fn(),
+      getSize: () => ({ rows: 1, cols: 80 }),
+      getState: () => terminalState("live"),
+      getStateSnapshot: () => ({ state: terminalState("live"), revision: 1 }),
+      getReplayPreamble: () => "",
+      getTitle: () => undefined,
+      setTitle: vi.fn(),
+      getExitInfo: () => null,
+      kill: vi.fn(),
+      killAndWait: vi.fn(),
+    };
+    const terminalManager = {
+      getTerminals: vi.fn(),
+      createTerminal: vi.fn(),
+      registerCwdEnv: vi.fn(),
+      getTerminal: vi.fn(() => terminal),
+      getTerminalState: vi.fn(() =>
+        Promise.resolve<TerminalStateSnapshot>({ state: terminalState("live"), revision: 1 }),
+      ),
+      setTerminalTitle: vi.fn(),
+      killTerminal: vi.fn(),
+      killTerminalAndWait: vi.fn(),
+      captureTerminal: vi.fn(),
+      listDirectories: vi.fn(() => []),
+      killAll: vi.fn(),
+      subscribeTerminalsChanged: vi.fn(() => vi.fn()),
+    } as unknown as TerminalManager;
+
+    const frames: TerminalStreamFrame[] = [];
+    const controller = new TerminalSessionController({
+      terminalManager,
+      emit: vi.fn(),
+      emitBinary: (bytes) => {
+        const frame = decodeTerminalStreamFrame(bytes);
+        if (frame) {
+          frames.push(frame);
+        }
+      },
+      hasBinaryChannel: () => true,
+      isPathWithinRoot: () => false,
+      sessionLogger: createLogger(),
+      getClientBufferedAmount,
+    });
+
+    await controller.dispatch({
+      type: "subscribe_terminal_request",
+      terminalId: "term-1",
+      requestId: "req-1",
+    });
+    await waitForCoalescerFlush();
+    // Drop the initial subscribe snapshot frame so each test only sees frames
+    // produced by the output it pushes.
+    frames.length = 0;
+
+    return {
+      pushOutput: (data) => terminalListener?.({ type: "output", data, revision: 2 }),
+      frames,
+    };
+  }
+
+  test("streams all output without a snapshot when the client keeps up", async () => {
+    const { pushOutput, frames } = await setup(() => 0);
+
+    const chunk = "x".repeat(300 * 1024);
+    pushOutput(chunk);
+    await waitForCoalescerFlush();
+
+    expect(frames.some((frame) => frame.opcode === TerminalStreamOpcode.Snapshot)).toBe(false);
+    const outputFrames = frames.filter((frame) => frame.opcode === TerminalStreamOpcode.Output);
+    expect(outputFrames.length).toBeGreaterThan(0);
+    const receivedBytes = outputFrames.reduce(
+      (total, frame) => total + frame.payload.byteLength,
+      0,
+    );
+    expect(receivedBytes).toBe(Buffer.byteLength(chunk, "utf8"));
+  });
+
+  test("falls back to a snapshot and resets the byte counter when the client is backed up", async () => {
+    const { pushOutput, frames } = await setup(() => 8 * 1024 * 1024);
+
+    pushOutput("y".repeat(300 * 1024));
+    await waitForCoalescerFlush();
+
+    expect(frames.some((frame) => frame.opcode === TerminalStreamOpcode.Snapshot)).toBe(true);
+
+    // After the snapshot the byte counter is reset, so a small follow-up chunk
+    // streams as Output rather than tripping the fallback again.
+    frames.length = 0;
+    pushOutput("z".repeat(1024));
+    await waitForCoalescerFlush();
+
+    expect(frames.some((frame) => frame.opcode === TerminalStreamOpcode.Snapshot)).toBe(false);
+    expect(frames.some((frame) => frame.opcode === TerminalStreamOpcode.Output)).toBe(true);
+  });
+
+  test("falls back to a snapshot at the byte threshold when no backpressure signal exists", async () => {
+    // A null reading means the transport (e.g. the multiplexed relay socket) gives
+    // no signal; we can't distinguish a slow client from a fast one, so we keep the
+    // unconditional catch-up so a slow relay client can't fall unboundedly behind.
+    const { pushOutput, frames } = await setup(() => null);
+
+    pushOutput("r".repeat(300 * 1024));
+    await waitForCoalescerFlush();
+
+    expect(frames.some((frame) => frame.opcode === TerminalStreamOpcode.Snapshot)).toBe(true);
+  });
+});