docs(fast-browser): record measured performance and verification evidence
This commit is contained in:
@@ -0,0 +1,68 @@
|
||||
# Fast-browser performance
|
||||
|
||||
## Result
|
||||
|
||||
On the provider-free local Chromium fixture, 20 alternating pairs reduced median warm workflow time from **702 to 279 ms (60%)**, and p95 from **729.05 to 315.30 ms (57%)**. Both arms passed all 20 warm attempts and all three cold attempts. Counter collection and owned-process cleanup passed.
|
||||
|
||||
This is one local fixture on native Linux, not a general website or agent benchmark. The fixture includes four navigations, 0/10/60-control observations, replacing a display name, saving and previewing it, immediate and delayed actions, and popup discovery/closure. A final assertion checks the entered name independently of the action postconditions.
|
||||
|
||||
| Measurement | Baseline p50 / p95 | Optimized p50 / p95 |
|
||||
| --- | ---: | ---: |
|
||||
| Warm complete workflow (20 per arm) | 702 / 729.05 ms | 279 / 315.30 ms |
|
||||
| Observe 0 controls | 2 / 5 ms | 2 / 3.35 ms |
|
||||
| Observe 10 controls | 16 / 23.60 ms | 7 / 11.20 ms |
|
||||
| Observe 60 controls | 78 / 93.05 ms | 28.50 / 31.15 ms |
|
||||
| Cold startup + first observation (3 per arm) | 273 / 306.30 ms | 270 / 283.50 ms |
|
||||
|
||||
Warm-worker lifetime CDP calls fell **17,791 to 7,407 (58%)**; IPC helper calls fell **19,135 to 8,751 (54%)**. These are separate counters, not additive: CDP calls also use IPC. Counts include worker initialization, the unmeasured priming observation, all 20 flows, and teardown; cold-worker counts are not included in these totals.
|
||||
|
||||
Summed transport observation time per warm flow had medians of 228.23 versus 84.74 ms. Navigation medians were 339.46 versus 61.55 ms. Action medians were 112.42 versus 109.97 ms. The main gains are observation and readiness, not faster mutation execution. The three cold samples do not establish a meaningful startup improvement.
|
||||
|
||||
## Changes
|
||||
|
||||
- Batch value-free DOM metadata in chunks of 16, while retaining Chrome's AX discovery, roles, ordering and click names. Fill labels retain the existing DOM-only label extraction. Temporary object groups are released after each chunk; partial or failed batches use the legacy safe path.
|
||||
- Keep fresh per-target AX/DOM revalidation, geometry/occlusion checks, field verification and click postconditions. No node handles, visibility, geometry, document identity, or origin authorization are cached across operations.
|
||||
- Remove redundant domain-enabling calls already made by pinned browser-harness 0.1.13, collect tab and opener metadata in one target inventory, cache the browser's select-all modifier, and reuse the existing tab read for fill URL verification.
|
||||
- Compute scoped origin contexts locally, preserving lifecycle/serialization checks and enforcement on subsequent worker operations. This avoids an otherwise redundant JSONL dispatch and does not start a worker merely to narrow a policy.
|
||||
- Replace the harness's 300 ms readiness sleep with bounded 25 ms monotonic polling of the same `document.readyState == complete` predicate and timeout.
|
||||
- After one close dispatch, verify target disappearance with bounded inventory reads under the existing action-verification budget. A close acknowledgement is not proof that Chrome has removed the target. The mutation is never retried.
|
||||
|
||||
## Reproduce
|
||||
|
||||
From the repository root, export the unchanged baseline and run the same benchmark against both workers:
|
||||
|
||||
```bash
|
||||
mkdir -p /tmp/fast-browser-performance-baseline-773c628
|
||||
git show 773c628512fdfdf5a139ae1b27833bea97d74255:extensions/fast-browser/browser_worker.py \
|
||||
> /tmp/fast-browser-performance-baseline-773c628/browser_worker.py
|
||||
node --experimental-strip-types extensions/fast-browser/performance-benchmark.ts \
|
||||
--baseline-worker /tmp/fast-browser-performance-baseline-773c628/browser_worker.py \
|
||||
--optimized-worker extensions/fast-browser/browser_worker.py \
|
||||
--python .pi/fast-browser-demo/venv/bin/python \
|
||||
--browser /usr/bin/chromium \
|
||||
--samples 20 --cold-samples 3 --counts 0,10,60 \
|
||||
> /tmp/fast-browser-performance-20-pairs-final.json
|
||||
```
|
||||
|
||||
Supply an existing Python environment containing `browser-harness==0.1.13`; the runner does not install dependencies. Use a new output filename when retaining an earlier result. Missing or invalid counters, any failed attempt, or unverified cleanup makes the command fail. Temporary browser state is retained if cleanup cannot be proven.
|
||||
|
||||
Validation environment: Chromium 152.0.7977.75, Python 3.12.13, Node v26.7.0, browser-harness 0.1.13. Measured worker commit: `2dc359c48d042a2982de8c972008b76685cbe105`.
|
||||
|
||||
- Baseline worker SHA-256: `487673653dc82a029cb5533b20566527739f8f5fd2155e5cfbd89da818132035`.
|
||||
- Optimized worker SHA-256: `d33814efb085e56f0bd6f2b8cf0c119559b37fa562a31aea0af1ed20a5550d7a`.
|
||||
- Retained final report: `/tmp/fast-browser-performance-20-pairs-final.json`.
|
||||
- Report SHA-256: `2e6a965f2a7a82c624902606e80a0fba074d16c23f87a85725eaf95f6b32a4ff`.
|
||||
|
||||
## Measurement boundaries and retained failures
|
||||
|
||||
Baseline and optimized Python workers use the same runner, fixture and browser configuration. Warm pairs alternate AB/BA; browser/worker startup is measured separately. Density observations are timed separately from navigation. Reports retain every attempt, bounded operation names, elapsed times and counters, not page text, field values, credentials, raw CDP arguments or screenshots. These measurements do not establish provider latency, semantic accuracy, Internet-site performance or cross-platform behavior. The TypeScript origin-context optimization has call-count tests but is not measured by this Python-worker comparison.
|
||||
|
||||
The first stable 20-pair run, before the close/readiness correction, is retained at `/tmp/fast-browser-performance-20-pairs.json`: baseline 19/20 and optimized 18/20 warm attempts passed. Three popup closes were unverified. Faster inventory calls exposed a pre-existing one-shot close-verification race. Observation was already faster, but coarse navigation polling produced mixed workflow medians (445 versus 551 ms). A four-pair diagnostic with per-operation timing at `/tmp/fast-browser-performance-4-pairs-diagnostic-20260921.json` isolated the navigation cost. Deterministic fake-clock tests reproduced the close race before the fix. An earlier smoke failure is retained at `/tmp/performance-smoke-final.json`.
|
||||
|
||||
These earlier results are not discarded or pooled with the corrected implementation. The final 46-attempt run above passed without retries of failed mutations.
|
||||
|
||||
## Verification and inspiration
|
||||
|
||||
Focused tests cover protocol call counts, batch fallback/cleanup and privacy, accessible naming, stale targets, origin-context lifetime, benchmark accounting, and close/readiness deadlines. Existing real Chromium development validation covers navigation, popups, origin narrowing, private diagnostics and owned cleanup. Exact full-suite and review outcomes are recorded in [the workflow ledger](plans/fast-browser-performance.md).
|
||||
|
||||
[jev-ultrafast's performance report](https://github.com/browser-use/jev-ultrafast/blob/main/docs/performance.md) motivated reducing browser round trips. Its DOM/value snapshot and autonomous operation selection are not copied: fast-browser retains its narrower data and execution contracts.
|
||||
@@ -3,8 +3,8 @@
|
||||
## Status
|
||||
|
||||
- Status: active
|
||||
- Current phase: correcting verified close/readiness bottlenecks after deterministic reproduction.
|
||||
- Next safe action: bounded read-only close verification and shorter readiness polling, followed by original live benchmark and full verification.
|
||||
- Current phase: final verified product ready for independent review.
|
||||
- Next safe action: pin the clean committed diff, run independent standards/specification reviews, then integrate only through the approved safe source-checkout gate.
|
||||
|
||||
## Request and scope
|
||||
|
||||
@@ -90,13 +90,13 @@ Each mutating assignment has a 15-minute deadline. B and C are separate assignme
|
||||
| Baseline typecheck | `bun run typecheck` | 0 | Passed; an accidental nonexistent chained test command then failed, unrelated to repository correctness. |
|
||||
| Baseline full tests | `bun run test` | 0 | Passed. |
|
||||
| Baseline fast-browser | `bun run test:fast-browser` | 0 | 220 TS passed, 2 skipped; 42 Python passed. |
|
||||
| Focused Python | pinned Python `-m unittest discover -s extensions/fast-browser -p 'test_*.py'` | pending | Include counting and stale/privacy regressions. |
|
||||
| Focused runtime/benchmark | `node --test --experimental-strip-types extensions/fast-browser/runtime.test.ts extensions/fast-browser/performance-benchmark.test.ts` | pending | Run after implementation. |
|
||||
| Integrated static/offline | `bun run typecheck && bun run test` | pending | Existing root suite; no new dependency. |
|
||||
| Scenario validation | scenario-cli `--check --format json` for existing local-form/login/popup examples | pending | All parse/preflight. |
|
||||
| Real runtime | `FAST_BROWSER_DEVELOPMENT_LIVE=1 node --test --experimental-strip-types extensions/fast-browser/development-live-validation.test.ts` | pending | Owned local Chromium, private artifacts, cleanup. |
|
||||
| Paired benchmark | new runner CLI, exact command recorded after runner exists | pending | At least 20 paired warm attempts, cold separately; baseline from Git, not copied production code. |
|
||||
| Hygiene/package | `git diff --check`; root package tests | pending | Ensure any new runtime resource ships. |
|
||||
| Focused Python | `.pi/fast-browser-demo/venv/bin/python -m unittest discover -s extensions/fast-browser -p 'test_*.py'` | 0 | 61 passed, including counting, batching, stale/privacy, delayed-close and readiness fake-clock regressions. |
|
||||
| Focused runtime/benchmark | individual `node --test --experimental-strip-types` runs for runtime.test.ts and performance-benchmark.test.ts | 0 | Parent verified 12 runtime and 7 benchmark tests. |
|
||||
| Integrated static/offline | `bun run typecheck && bun run test` | 0 | Full repository suite passed. Fast-browser: 230 TS passed, 2 opt-in skips; 61 Python passed. No dependency installation. |
|
||||
| Scenario validation | scenario-cli `--check --format json` for existing local-form/login/popup examples | 0 | All three validated; no browser started by preflight. |
|
||||
| Real runtime | `FAST_BROWSER_DEVELOPMENT_LIVE=1 node --test --experimental-strip-types extensions/fast-browser/development-live-validation.test.ts` | 0 | Passed with no skip on baseline, batched worker and final corrected worker. Includes private evidence and cleanup. |
|
||||
| Paired benchmark | command in `docs/fast-browser-performance.md`, 20 pairs plus 3 cold samples per arm | 0 | Final all 46 attempts passed, counters complete, cleanup passed. Warm p50 702 to 279ms; p95 729.05 to 315.30ms; CDP 17,791 to 7,407. Earlier failures retained. |
|
||||
| Hygiene/package | `git diff --check`; root package tests in full suite | 0 | Passed. Metadata script is an internal Python constant; no external runtime resource needed. |
|
||||
|
||||
## Review
|
||||
|
||||
@@ -125,4 +125,7 @@ After safe integration, make one record-only closure commit, prove reachability,
|
||||
- Timing probe committed 12a6bb6; parent reran 7 benchmark tests and full typecheck. Diagnostic `/tmp/fast-browser-performance-4-pairs-diagnostic-20260921.json` passed all attempts and confirms navigate ~335ms versus observe ~223 to ~88ms. Action times ~110ms remain similar. No provider latency is measured.
|
||||
- Parent reproduced close tests RED: 4 focused tests, 1 expected failure and 1 expected error for delayed/permanent target presence, 2 passed for inventory/cancellation handling. Test patch `/tmp/pi-present/performance-close-red.patch`.
|
||||
- Browser-hotpath correction assignment: 15 minutes, same Python worker/test paths; before HEAD 12a6bb6, only owned red tests dirty plus parent doc. Implement single-dispatch close followed by bounded inventory reads under existing ACTION_TIMEOUT_SECONDS. Replace pinned helper's 300ms readiness sleep with <=25ms bounded monotonic polling for the exact same document.readyState == complete predicate and existing timeout, retaining error propagation. Add readiness regression tests before the fix. This changes private polling cadence, not API, security posture, postconditions or budgets. Before patch `/tmp/pi-present/performance-close-red.patch`.
|
||||
- Next safe action: inspect corrected worker delivery, run final stable paired benchmark and full verification. Preserve every failed result.
|
||||
- Corrected close/readiness worker committed 2dc359c; parent inspected patch `/tmp/pi-present/performance-readiness-close-fix.patch` and reran 61 Python tests. Final report `/tmp/fast-browser-performance-20-pairs-final.json` passed all 46 attempts; hash and exact environment/commands are in performance documentation.
|
||||
- Final verification background task bg_vjl_mubjba9s_12 exited 0 after typecheck, complete repository suite, pinned Python suite, real Chromium runtime test, scenario preflight and diff check. No failures; opt-in general-profile test remains skipped in the default suite, while the development live gate was explicitly run.
|
||||
- Source fetched again: origin/main still 773c628512fdfdf5a139ae1b27833bea97d74255, no rebase needed. Source checkout still has only the unrelated untracked feasibility document; final integration requires a bounded decision or a clean checkout.
|
||||
- Next safe action: commit final documentation, pin review bytes, launch independent read-only reviewers. Preserve every failed result.
|
||||
|
||||
Reference in New Issue
Block a user