Eliminate spiky terminal lag under load (#1500)
* Eliminate spiky terminal lag under load
Terminal output stuttered during heavy output and when the daemon was
busy. Two compounding causes, both confirmed by measurement:
- Every 256KB of output, the daemon dropped pending frames and re-sent a
full cell-grid snapshot (cloned across IPC, stringified to JSON) — a
10MB build did this ~40 times. The snapshot fallback is now gated on
real client backpressure (ws bufferedAmount), so a client that keeps
draining streams continuously and never pays the snapshot tax. This
also removes the periodic GC hitch the churn caused.
- Per-chunk overhead on the shared event loop: one IPC message per pty
chunk, a duplicate input-mode regex scan on the daemon main loop, a
double JSON.stringify per outbound message, and 16KB string realloc
per chunk. Output now coalesces in the worker before IPC (leading-edge
so keystrokes still echo immediately), the duplicate scan is gone, and
the client feeds xterm back-to-back instead of one render tick a frame.
Adds eventLoopDelay percentiles to ws_runtime_metrics for main-loop
stall visibility, plus a reproducible Node benchmark (no port 6767).
Measured: echo p50 7.7ms to 2.3ms; a 2MB burst now streams fully with
zero snapshots; loop-stall spikes 100-173ms to 2ms.
* Guard trace-level check against partial logger stubs
The isLevelEnabled gate added for the emit() perf fix crashed every
session test that injects a hand-rolled logger stub (10 of them omit
isLevelEnabled). Real pino loggers always provide it; optional-chaining
keeps the perf gate intact in production and no-ops on stubs.
* Make recent-output exit-summary test deterministic
The added test self-exited the child immediately after writing 3000
lines, then asserted the newest line was in the exit summary — but the
summary reads the headless xterm buffer, which parses writes
asynchronously, so a loaded CI runner saw a stale buffer (line-2707 not
line-2999). Keep the process alive and poll the parsed buffer until the
final line lands before killing, removing the race.
* Skip ConPTY-fragile worker terminal tests on Windows
The coalescing and input-mode-preamble tests assert byte-contiguous PTY
output and an exact kitty-escape round-trip. Windows ConPTY injects
repaint sequences between writes and normalizes the escape, so both time
out there while passing on Linux/macOS. Gate them with skipIf(win32),
matching the existing terminal-test convention for PTY-sensitive cases.
* Apply terminal barriers immediately when no writes are pending
A barrier op (snapshot/restore/clear) only needs the sentinel-write gate
to wait out plain writes still parsing in xterm's buffer. When none are
ungated — at mount, or right after another barrier — the sentinel cost a
wasted parse cycle, adding latency to the first snapshot/restore on the
hot first-paint path. Track ungated writes with a flag and skip the
sentinel when there's nothing to gate.
* Fix mobile startup saved-host e2e
* Unskip Windows worker terminal coverage
* Tighten worker coalescing assertion
* Reset terminal ungated writes on unmount
* Fix terminal regressions found in adversarial review
Three issues this PR's terminal changes introduced:
- Worker snapshot could duplicate coalesced output. getTerminalState
snapshotted without flushing the worker output coalescer, so a batch
spanning the snapshot point carried a revision past it and the
controller's dedup couldn't drop it — the client saw the bytes twice.
Flush before snapshotting.
- Relay clients lost backpressure protection. The snapshot fallback was
gated on bufferedAmount, which the multiplexed relay socket reports as
absent; that read as 0 ("keeping up") so a slow relay client never
caught up via a snapshot. Distinguish "no signal" (null) from 0 and
keep the unconditional byte-threshold fallback for signal-less
transports, preserving the pre-change relay behavior.
- Recent-output buffer was no longer a hard cap. A single chunk larger
than the limit was retained whole; slice its tail.
Documents the live-restore preamble gap (unreachable: no client sends
restore mode "live").
* Skip input-mode preamble test on Windows ConPTY
CI confirmed Windows ConPTY normalizes away the kitty keyboard escape
the child writes, so it never reaches the worker's input-mode tracker
and the preamble stays empty — the test times out. ConPTY can't exercise
this contract; the coalescing test (file-gated, line-oriented) stays on
all platforms, only the preamble assertion is gated to Linux/macOS.
This commit is contained in:
@@ -1,3 +1,4 @@
|
||||
import type { Route } from "@playwright/test";
|
||||
import { expect, type Page } from "../fixtures";
|
||||
import { buildCreateAgentPreferences, buildSeededHost } from "./daemon-registry";
|
||||
import { wsRoutePatternForPort } from "./daemon-port";
|
||||
@@ -6,6 +7,7 @@ const DISABLE_DEFAULT_SEED_ONCE_KEY = "@paseo:e2e-disable-default-seed-once";
|
||||
const SEED_NONCE_KEY = "@paseo:e2e-seed-nonce";
|
||||
const REGISTRY_KEY = "@paseo:daemon-registry";
|
||||
const E2E_KEY = "@paseo:e2e";
|
||||
const STORAGE_SEED_HTML = "<!doctype html><html><body>storage seed</body></html>";
|
||||
|
||||
interface SavedHostInput {
|
||||
serverId: string;
|
||||
@@ -88,9 +90,9 @@ class StartupScenario {
|
||||
return;
|
||||
}
|
||||
|
||||
// Let the shared fixture create its seed nonce, then opt out of that seed for
|
||||
// the next navigation so this scenario owns the stored host registry.
|
||||
await this.page.goto("/");
|
||||
// Create same-origin storage without booting the app. If the default host runtime
|
||||
// starts first, it can race this scenario's registry override during CI.
|
||||
await openStorageSeedDocument(this.page);
|
||||
const nowIso = new Date().toISOString();
|
||||
const registry = this.savedHosts.map((host) =>
|
||||
buildStoredHost({
|
||||
@@ -132,6 +134,21 @@ class StartupScenario {
|
||||
}
|
||||
}
|
||||
|
||||
async function openStorageSeedDocument(page: Page): Promise<void> {
|
||||
await page.route(
|
||||
"**/*",
|
||||
async (route: Route) => {
|
||||
await route.fulfill({
|
||||
status: 200,
|
||||
contentType: "text/html",
|
||||
body: STORAGE_SEED_HTML,
|
||||
});
|
||||
},
|
||||
{ times: 1 },
|
||||
);
|
||||
await page.goto("/");
|
||||
}
|
||||
|
||||
class StartupAssertions {
|
||||
private readonly page: Page;
|
||||
|
||||
@@ -140,6 +157,7 @@ class StartupAssertions {
|
||||
}
|
||||
|
||||
async expectsSavedHostShell(input: { label: string }): Promise<this> {
|
||||
await this.page.getByRole("button", { name: "Open menu", exact: true }).click();
|
||||
await expect(this.page.getByText(input.label, { exact: true })).toBeVisible({
|
||||
timeout: 15_000,
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user