docs(fast-browser): record measured performance and verification evidence
This commit is contained in:
@@ -0,0 +1,68 @@
|
||||
# Fast-browser performance
|
||||
|
||||
## Result
|
||||
|
||||
On the provider-free local Chromium fixture, 20 alternating pairs reduced median warm workflow time from **702 to 279 ms (60%)**, and p95 from **729.05 to 315.30 ms (57%)**. Both arms passed all 20 warm attempts and all three cold attempts. Counter collection and owned-process cleanup passed.
|
||||
|
||||
This is one local fixture on native Linux, not a general website or agent benchmark. The fixture includes four navigations, 0/10/60-control observations, replacing a display name, saving and previewing it, immediate and delayed actions, and popup discovery/closure. A final assertion checks the entered name independently of the action postconditions.
|
||||
|
||||
| Measurement | Baseline p50 / p95 | Optimized p50 / p95 |
|
||||
| --- | ---: | ---: |
|
||||
| Warm complete workflow (20 per arm) | 702 / 729.05 ms | 279 / 315.30 ms |
|
||||
| Observe 0 controls | 2 / 5 ms | 2 / 3.35 ms |
|
||||
| Observe 10 controls | 16 / 23.60 ms | 7 / 11.20 ms |
|
||||
| Observe 60 controls | 78 / 93.05 ms | 28.50 / 31.15 ms |
|
||||
| Cold startup + first observation (3 per arm) | 273 / 306.30 ms | 270 / 283.50 ms |
|
||||
|
||||
Warm-worker lifetime CDP calls fell **17,791 to 7,407 (58%)**; IPC helper calls fell **19,135 to 8,751 (54%)**. These are separate counters, not additive: CDP calls also use IPC. Counts include worker initialization, the unmeasured priming observation, all 20 flows, and teardown; cold-worker counts are not included in these totals.
|
||||
|
||||
Summed transport observation time per warm flow had medians of 228.23 versus 84.74 ms. Navigation medians were 339.46 versus 61.55 ms. Action medians were 112.42 versus 109.97 ms. The main gains are observation and readiness, not faster mutation execution. The three cold samples do not establish a meaningful startup improvement.
|
||||
|
||||
## Changes
|
||||
|
||||
- Batch value-free DOM metadata in chunks of 16, while retaining Chrome's AX discovery, roles, ordering and click names. Fill labels retain the existing DOM-only label extraction. Temporary object groups are released after each chunk; partial or failed batches use the legacy safe path.
|
||||
- Keep fresh per-target AX/DOM revalidation, geometry/occlusion checks, field verification and click postconditions. No node handles, visibility, geometry, document identity, or origin authorization are cached across operations.
|
||||
- Remove redundant domain-enabling calls already made by pinned browser-harness 0.1.13, collect tab and opener metadata in one target inventory, cache the browser's select-all modifier, and reuse the existing tab read for fill URL verification.
|
||||
- Compute scoped origin contexts locally, preserving lifecycle/serialization checks and enforcement on subsequent worker operations. This avoids an otherwise redundant JSONL dispatch and does not start a worker merely to narrow a policy.
|
||||
- Replace the harness's 300 ms readiness sleep with bounded 25 ms monotonic polling of the same `document.readyState == complete` predicate and timeout.
|
||||
- After one close dispatch, verify target disappearance with bounded inventory reads under the existing action-verification budget. A close acknowledgement is not proof that Chrome has removed the target. The mutation is never retried.
|
||||
|
||||
## Reproduce
|
||||
|
||||
From the repository root, export the unchanged baseline and run the same benchmark against both workers:
|
||||
|
||||
```bash
|
||||
mkdir -p /tmp/fast-browser-performance-baseline-773c628
|
||||
git show 773c628512fdfdf5a139ae1b27833bea97d74255:extensions/fast-browser/browser_worker.py \
|
||||
> /tmp/fast-browser-performance-baseline-773c628/browser_worker.py
|
||||
node --experimental-strip-types extensions/fast-browser/performance-benchmark.ts \
|
||||
--baseline-worker /tmp/fast-browser-performance-baseline-773c628/browser_worker.py \
|
||||
--optimized-worker extensions/fast-browser/browser_worker.py \
|
||||
--python .pi/fast-browser-demo/venv/bin/python \
|
||||
--browser /usr/bin/chromium \
|
||||
--samples 20 --cold-samples 3 --counts 0,10,60 \
|
||||
> /tmp/fast-browser-performance-20-pairs-final.json
|
||||
```
|
||||
|
||||
Supply an existing Python environment containing `browser-harness==0.1.13`; the runner does not install dependencies. Use a new output filename when retaining an earlier result. Missing or invalid counters, any failed attempt, or unverified cleanup makes the command fail. Temporary browser state is retained if cleanup cannot be proven.
|
||||
|
||||
Validation environment: Chromium 152.0.7977.75, Python 3.12.13, Node v26.7.0, browser-harness 0.1.13. Measured worker commit: `2dc359c48d042a2982de8c972008b76685cbe105`.
|
||||
|
||||
- Baseline worker SHA-256: `487673653dc82a029cb5533b20566527739f8f5fd2155e5cfbd89da818132035`.
|
||||
- Optimized worker SHA-256: `d33814efb085e56f0bd6f2b8cf0c119559b37fa562a31aea0af1ed20a5550d7a`.
|
||||
- Retained final report: `/tmp/fast-browser-performance-20-pairs-final.json`.
|
||||
- Report SHA-256: `2e6a965f2a7a82c624902606e80a0fba074d16c23f87a85725eaf95f6b32a4ff`.
|
||||
|
||||
## Measurement boundaries and retained failures
|
||||
|
||||
Baseline and optimized Python workers use the same runner, fixture and browser configuration. Warm pairs alternate AB/BA; browser/worker startup is measured separately. Density observations are timed separately from navigation. Reports retain every attempt, bounded operation names, elapsed times and counters, not page text, field values, credentials, raw CDP arguments or screenshots. These measurements do not establish provider latency, semantic accuracy, Internet-site performance or cross-platform behavior. The TypeScript origin-context optimization has call-count tests but is not measured by this Python-worker comparison.
|
||||
|
||||
The first stable 20-pair run, before the close/readiness correction, is retained at `/tmp/fast-browser-performance-20-pairs.json`: baseline 19/20 and optimized 18/20 warm attempts passed. Three popup closes were unverified. Faster inventory calls exposed a pre-existing one-shot close-verification race. Observation was already faster, but coarse navigation polling produced mixed workflow medians (445 versus 551 ms). A four-pair diagnostic with per-operation timing at `/tmp/fast-browser-performance-4-pairs-diagnostic-20260921.json` isolated the navigation cost. Deterministic fake-clock tests reproduced the close race before the fix. An earlier smoke failure is retained at `/tmp/performance-smoke-final.json`.
|
||||
|
||||
These earlier results are not discarded or pooled with the corrected implementation. The final 46-attempt run above passed without retries of failed mutations.
|
||||
|
||||
## Verification and inspiration
|
||||
|
||||
Focused tests cover protocol call counts, batch fallback/cleanup and privacy, accessible naming, stale targets, origin-context lifetime, benchmark accounting, and close/readiness deadlines. Existing real Chromium development validation covers navigation, popups, origin narrowing, private diagnostics and owned cleanup. Exact full-suite and review outcomes are recorded in [the workflow ledger](plans/fast-browser-performance.md).
|
||||
|
||||
[jev-ultrafast's performance report](https://github.com/browser-use/jev-ultrafast/blob/main/docs/performance.md) motivated reducing browser round trips. Its DOM/value snapshot and autonomous operation selection are not copied: fast-browser retains its narrower data and execution contracts.
|
||||
Reference in New Issue
Block a user