Compare commits

...
17 Commits
Author SHA1 Message Date
jb 9862ef0be9 release: prepare fast-browser 0.1.1
fast-browser scenarios / Offline scenario contract tests (push) Failing after 14s
fast-browser scenarios / Local Chromium headless scenarios (push) Failing after 25s
2026-09-21 14:15:36 -04:00
jb 5ddd98bf5f docs: close reviewed fast-browser performance workflow 2026-09-21 13:58:11 -04:00
jb 54303e0b5d docs(fast-browser): record measured performance and verification evidence 2026-09-21 13:47:35 -04:00
jb 2dc359c48d perf(fast-browser): bound readiness polling and verify asynchronous tab close 2026-09-21 13:42:57 -04:00
jb fe643e2d8d docs: record red close tests and measured readiness correction 2026-09-21 13:37:49 -04:00
jb 12a6bb6afb test(fast-browser): retain per-operation timing and failure costs 2026-09-21 13:33:51 -04:00
jb 6a725c1f47 docs: confirm asynchronous close race and test-first correction 2026-09-21 13:32:57 -04:00
jb f2e42f5faf docs: retain failed paired run and isolate verification bottlenecks 2026-09-21 13:29:08 -04:00
jb acd57ae2a7 docs: checkpoint completed performance slices and live verification 2026-09-21 13:26:44 -04:00
jb e10a9573b9 test(fast-browser): add paired local browser performance benchmark 2026-09-21 13:25:44 -04:00
jb 574e083e75 perf(fast-browser): batch value-free observation metadata 2026-09-21 13:18:18 -04:00
jb 4360f01947 docs: record runtime verification and benchmark acceptance gaps 2026-09-21 13:15:11 -04:00
jb c0c18a5191 perf(fast-browser): compute scoped origin policy without worker dispatch 2026-09-21 13:11:16 -04:00
jb 3c321b3033 docs: record browser hot-path verification and batching handoff 2026-09-21 13:05:42 -04:00
jb a79418efcb perf(fast-browser): eliminate redundant browser protocol calls 2026-09-21 13:05:17 -04:00
jb 05793fe7b0 docs: assign isolated performance implementation slices 2026-09-21 12:58:48 -04:00
jb 6fba62800a docs: approve fast-browser performance implementation plan 2026-09-21 12:58:21 -04:00
14 changed files with 1942 additions and 117 deletions
+2 -1
View File
@@ -54,10 +54,11 @@
},
"extensions/fast-browser": {
"name": "@jbcode/pi-fast-browser",
"version": "0.1.0",
"version": "0.1.1",
"peerDependencies": {
"@earendil-works/pi-ai": "catalog:pi-peer",
"@earendil-works/pi-coding-agent": "catalog:pi-peer",
"@earendil-works/pi-tui": "catalog:pi-peer",
"typebox": "catalog:pi-peer",
},
},
+68
View File
@@ -0,0 +1,68 @@
# Fast-browser performance
## Result
On the provider-free local Chromium fixture, 20 alternating pairs reduced median warm workflow time from **702 to 279 ms (60%)**, and p95 from **729.05 to 315.30 ms (57%)**. Both arms passed all 20 warm attempts and all three cold attempts. Counter collection and owned-process cleanup passed.
This is one local fixture on native Linux, not a general website or agent benchmark. The fixture includes four navigations, 0/10/60-control observations, replacing a display name, saving and previewing it, immediate and delayed actions, and popup discovery/closure. A final assertion checks the entered name independently of the action postconditions.
| Measurement | Baseline p50 / p95 | Optimized p50 / p95 |
| --- | ---: | ---: |
| Warm complete workflow (20 per arm) | 702 / 729.05 ms | 279 / 315.30 ms |
| Observe 0 controls | 2 / 5 ms | 2 / 3.35 ms |
| Observe 10 controls | 16 / 23.60 ms | 7 / 11.20 ms |
| Observe 60 controls | 78 / 93.05 ms | 28.50 / 31.15 ms |
| Cold startup + first observation (3 per arm) | 273 / 306.30 ms | 270 / 283.50 ms |
Warm-worker lifetime CDP calls fell **17,791 to 7,407 (58%)**; IPC helper calls fell **19,135 to 8,751 (54%)**. These are separate counters, not additive: CDP calls also use IPC. Counts include worker initialization, the unmeasured priming observation, all 20 flows, and teardown; cold-worker counts are not included in these totals.
Summed transport observation time per warm flow had medians of 228.23 versus 84.74 ms. Navigation medians were 339.46 versus 61.55 ms. Action medians were 112.42 versus 109.97 ms. The main gains are observation and readiness, not faster mutation execution. The three cold samples do not establish a meaningful startup improvement.
## Changes
- Batch value-free DOM metadata in chunks of 16, while retaining Chrome's AX discovery, roles, ordering and click names. Fill labels retain the existing DOM-only label extraction. Temporary object groups are released after each chunk; partial or failed batches use the legacy safe path.
- Keep fresh per-target AX/DOM revalidation, geometry/occlusion checks, field verification and click postconditions. No node handles, visibility, geometry, document identity, or origin authorization are cached across operations.
- Remove redundant domain-enabling calls already made by pinned browser-harness 0.1.13, collect tab and opener metadata in one target inventory, cache the browser's select-all modifier, and reuse the existing tab read for fill URL verification.
- Compute scoped origin contexts locally, preserving lifecycle/serialization checks and enforcement on subsequent worker operations. This avoids an otherwise redundant JSONL dispatch and does not start a worker merely to narrow a policy.
- Replace the harness's 300 ms readiness sleep with bounded 25 ms monotonic polling of the same `document.readyState == complete` predicate and timeout.
- After one close dispatch, verify target disappearance with bounded inventory reads under the existing action-verification budget. A close acknowledgement is not proof that Chrome has removed the target. The mutation is never retried.
## Reproduce
From the repository root, export the unchanged baseline and run the same benchmark against both workers:
```bash
mkdir -p /tmp/fast-browser-performance-baseline-773c628
git show 773c628512fdfdf5a139ae1b27833bea97d74255:extensions/fast-browser/browser_worker.py \
> /tmp/fast-browser-performance-baseline-773c628/browser_worker.py
node --experimental-strip-types extensions/fast-browser/performance-benchmark.ts \
--baseline-worker /tmp/fast-browser-performance-baseline-773c628/browser_worker.py \
--optimized-worker extensions/fast-browser/browser_worker.py \
--python .pi/fast-browser-demo/venv/bin/python \
--browser /usr/bin/chromium \
--samples 20 --cold-samples 3 --counts 0,10,60 \
> /tmp/fast-browser-performance-20-pairs-final.json
```
Supply an existing Python environment containing `browser-harness==0.1.13`; the runner does not install dependencies. Use a new output filename when retaining an earlier result. Missing or invalid counters, any failed attempt, or unverified cleanup makes the command fail. Temporary browser state is retained if cleanup cannot be proven.
Validation environment: Chromium 152.0.7977.75, Python 3.12.13, Node v26.7.0, browser-harness 0.1.13. Measured worker commit: `2dc359c48d042a2982de8c972008b76685cbe105`.
- Baseline worker SHA-256: `487673653dc82a029cb5533b20566527739f8f5fd2155e5cfbd89da818132035`.
- Optimized worker SHA-256: `d33814efb085e56f0bd6f2b8cf0c119559b37fa562a31aea0af1ed20a5550d7a`.
- Retained final report: `/tmp/fast-browser-performance-20-pairs-final.json`.
- Report SHA-256: `2e6a965f2a7a82c624902606e80a0fba074d16c23f87a85725eaf95f6b32a4ff`.
## Measurement boundaries and retained failures
Baseline and optimized Python workers use the same runner, fixture and browser configuration. Warm pairs alternate AB/BA; browser/worker startup is measured separately. Density observations are timed separately from navigation. Reports retain every attempt, bounded operation names, elapsed times and counters, not page text, field values, credentials, raw CDP arguments or screenshots. These measurements do not establish provider latency, semantic accuracy, Internet-site performance or cross-platform behavior. The TypeScript origin-context optimization has call-count tests but is not measured by this Python-worker comparison.
The first stable 20-pair run, before the close/readiness correction, is retained at `/tmp/fast-browser-performance-20-pairs.json`: baseline 19/20 and optimized 18/20 warm attempts passed. Three popup closes were unverified. Faster inventory calls exposed a pre-existing one-shot close-verification race. Observation was already faster, but coarse navigation polling produced mixed workflow medians (445 versus 551 ms). A four-pair diagnostic with per-operation timing at `/tmp/fast-browser-performance-4-pairs-diagnostic-20260921.json` isolated the navigation cost. Deterministic fake-clock tests reproduced the close race before the fix. An earlier smoke failure is retained at `/tmp/performance-smoke-final.json`.
These earlier results are not discarded or pooled with the corrected implementation. The final 46-attempt run above passed without retries of failed mutations.
## Verification and inspiration
Focused tests cover protocol call counts, batch fallback/cleanup and privacy, accessible naming, stale targets, origin-context lifetime, benchmark accounting, and close/readiness deadlines. Existing real Chromium development validation covers navigation, popups, origin narrowing, private diagnostics and owned cleanup. Exact full-suite and review outcomes are recorded in [the workflow ledger](plans/fast-browser-performance.md).
[jev-ultrafast's performance report](https://github.com/browser-use/jev-ultrafast/blob/main/docs/performance.md) motivated reducing browser round trips. Its DOM/value snapshot and autonomous operation selection are not copied: fast-browser retains its narrower data and execution contracts.
+1 -1
View File
@@ -49,7 +49,7 @@ Publish `@jbcode/pi-shared` before a new version of any extension that depends o
After configuring the same scope and token in the npm configuration visible to Pi, install the currently deployed release set with exact version pins:
```bash
pi install npm:@jbcode/pi-fast-browser@0.1.0
pi install npm:@jbcode/pi-fast-browser@0.1.1
pi install npm:@jbcode/pi-dictate@0.2.0
pi install npm:@jbcode/pi-effort-controls@0.1.0
pi install npm:@jbcode/pi-prompt-snippets@0.1.0
+142
View File
@@ -0,0 +1,142 @@
# Fast-browser performance workflow record
## Status
- Status: complete
- Current phase: verified, independently approved product fast-forwarded to local main.
- Next safe action: after this record-only closure commit, run final sanity/reachability checks, remove the owned clean worktree/branch, and restore the unrelated file's Git visibility. Final housekeeping evidence is written to `/tmp/pi-present/fast-browser-performance-cleanup.json`.
## Request and scope
Optimize fast-browser for local development testing and validation, borrowing proven round-trip reductions from jev-ultrafast. The parent orchestrates; subagents implement exclusive slices. Optimize browser/runtime overhead first, with measured correctness-preserving gains rather than a speculative autonomous-agent rewrite.
In scope: redundant browser calls; AX-preserving batched observation metadata; local origin-context calculation; a provider-free paired benchmark; focused regression tests and performance documentation.
Out of scope: new actions, autonomous planning, broader model context, changed model/provider, dependency changes, public API changes, uploads/downloads, personal browser profiles, production access, publishing, and speculative page-state/decision caches. No product UI change. Synthetic fixture HTML is validation material, not a new UI feature.
## Git identity
- Source branch: `main` at `773c628512fdfdf5a139ae1b27833bea97d74255`.
- Remote: `origin/main`, fetched and identical to local main.
- Feature branch: `feature/fast-browser-performance` (must be absent before creation).
- Worktree: `/home/jb/code/harness-tools/.worktrees/fast-browser-performance` (must be absent before creation).
- Omitted source changes: untracked `docs/local-decision-model-feasibility.md`, preserved untouched. No tracked source modifications observed.
- Add `/.worktrees/` to the repository-local Git exclude file before worktree creation. This is local Git housekeeping, not a product change.
- Record path after approval: `docs/plans/fast-browser-performance.md` in the new worktree.
- Reuse existing root node_modules and the pinned Python environment `/home/jb/code/harness-tools/.pi/fast-browser-demo/venv/bin/python`; no installation or lockfile modification. Any worktree-local links are ignored and explicitly owned.
## Acceptance criteria
1. Existing public tools, operation schemas, candidate ordering/limit, field-label privacy, fresh action revalidation, occlusion checks, outcome verification, budgets, session isolation, and no-blind-retry semantics remain intact.
2. Browser hot-path call counts fall deterministically: one target inventory per refresh; no redundant domain enables after the pinned harness attach; browser platform detection at most once per backend; batched observations remove per-candidate partial-AX queries and repeated evaluate/release cycles.
3. Chrome AX data remains authoritative for observation roles and click names. Fill labels retain the existing value-free DOM extraction and never fall back to AX value-bearing names. No field values, raw DOM, selectors, or arbitrary scripts enter observations, provider input, or benchmark reports.
4. Batch resources exist only for one bounded observation chunk and are released on success/failure. Fresh single-target metadata remains the pre-mutation authority. Missing/failed batch data uses the existing safe per-node path where compatible; no blind mutation retry.
5. Origin narrowing remains validated and session-lifetime/serialization aware but does not send a redundant JSONL operation. The effective context still travels with every subsequent browser operation; omitted context restores the session base, and restricted sessions cannot broaden.
6. Benchmark results cover unchanged baseline and optimized code with the same runner/fixture/browser, independent outcome assertions, all attempts retained, startup separate from warm operation timing, p50/p95 and protocol counts. Keep optimizations only when call counts improve and live results show no material regression; investigate noisy or mixed results before claiming gains.
7. Full repository tests/typecheck, focused regressions, real Chromium runtime validation, and two independent review axes pass for the final committed diff. Any environment blocker or unsupported claim remains explicit.
## Approved implementation plan and gates
Plan approval: user selected `approve` (Approve and implement) through ask_user on 2026-09-21. Approval covers the exact slices and source-file omission below, local benchmark browsers/fixtures, local commits, and eventual safe fast-forward integration. It does not authorize dependency changes, production access, API/security-posture changes, or publication.
### Slice A: provider-free measurement and regression fixture
Owner `performance-benchmark` (worker). Exclusive paths: `extensions/fast-browser/performance-benchmark.ts`, `extensions/fast-browser/performance-benchmark.test.ts`, `extensions/fast-browser/performance_probe.py`, `extensions/fast-browser/fixtures/performance.html`.
Build a bounded opt-in runner using existing browser-session/runtime/transport seams. Accept explicit worker/baseline paths and pinned Python/browser configuration rather than copying an old implementation into production. Count CDP/IPC methods and JSONL operations locally without recording arguments, URLs, values, text, or screenshots. A test-only Python probe may wrap the selected worker's helper methods for counting; no product instrumentation switch is required.
Use realistic local account/settings or search/filter form workflows with 0/10/60 eligible controls, fill replacement, immediate/delayed click outcomes, and popup discovery. Assert actual final state independently of action postconditions. Exercise warm repeated observations and complete verified flows; measure cold startup separately. Alternate baseline/optimized arms, aim for at least 20 paired warm samples, retain failures and cleanup results. Deterministic model stubs may exercise orchestration without provider cost; clearly exclude real provider latency and semantic accuracy from claims. Focused tests cover statistics/report accounting, input bounds, failure inclusion, and report privacy. No wall-clock thresholds in unit tests.
### Slice B: remove proven browser-call duplication
Owner `browser-hotpath` (worker). Exclusive paths: `extensions/fast-browser/browser_worker.py`, `extensions/fast-browser/test_browser_worker.py`.
Use a single `Target.getTargets` response for page inventory and opener metadata, preserving filtering and target identity. Remove duplicate Page/DOM/Runtime/Network enables only where pinned browser-harness 0.1.13 already guarantees attempts on attachment. Cache browser-version-derived select-all modifiers per backend lifetime. Reuse the already obtained tab result during fill URL verification. Add exact call-count and behavioral regressions. Do not lengthen polling intervals as a speculative throughput optimization or cache changing document/target state across operations.
### Slice C: AX-preserving observation batching
Owner `browser-hotpath`, serialized after B, same exclusive paths. Use an internal Python-owned script constant to avoid new runtime assets and packaging changes.
Introduce a private optional batch-metadata seam with the legacy backend/fake fallback. Keep full AX candidate discovery and ordering. Resolve eligible backend node IDs into a bounded temporary object group; evaluate existing value-free DOM metadata for a small chunk together, using full AX snapshot role/name rather than redundant partial AX reads; release the object group in finally. Never retain remote node handles or geometry across observations. Keep fresh existing single-node AX/DOM revalidation before act. Cover distinct AX/DOM labels, password/value privacy, disabled/hidden/disconnected nodes, duplicate candidates, limits, malformed/partial responses, navigation/detachment and cleanup. Measure against the baseline before retaining. If the compatible batch path cannot demonstrate safe benefit, retain B and document the blocked experiment instead of weakening contracts.
### Slice D: local origin-context calculation
Owner `runtime-hotpath` (worker). Exclusive paths: `extensions/fast-browser/runtime.ts`, `extensions/fast-browser/runtime.test.ts`.
Compute `narrowOriginPolicy` locally, retaining runtime closed/aborted behavior, serialized session semantics and current validation/error shaping. Subsequent transport/worker operations must still enforce the returned context. Preserve compatibility for direct legacy transport requests. Test no transport dispatch for narrowing, context propagation, rejected broadening, base restoration, closed/aborted sessions and concurrency behavior. Do not remove transport validation, serialized lanes, or change error/mutation semantics to save time.
### Orchestration and documentation
Parent owns `docs/plans/fast-browser-performance.md`, `docs/fast-browser-performance.md`, and the performance section of `extensions/fast-browser/README.md`. No code implementation is assigned to the parent. Record measured boundaries, exact commands, baseline identity, limitations and retained results. Add `docs/todos.md` evidence only for a newly encountered tool defect that cannot be resolved in scope.
Each mutating assignment has a 15-minute deadline. B and C are separate assignments to the same owner; A and D may run in parallel because paths are disjoint. Workers do not commit, delegate mutation, alter dependencies, modify other owners' files, or integrate branches. Parent records before/after Git evidence and inspects every returned diff. A worker hitting its deadline must stop and report its partial state; reassignment requires proven completion.
## Ownership
| Assignment | Owner identity | Deadline | Exclusive paths | Before/after Git evidence | Outcome |
| --- | --- | --- | --- | --- | --- |
| Research browser | browser-performance-research / scout | complete | none; read-only | source unchanged | delivered |
| Research runtime | runtime-performance-research / scout | complete | none; read-only | source unchanged | delivered |
| A benchmark | performance-benchmark / worker | first and correction deliveries within deadlines | four listed new files | initial owned patch `/tmp/pi-present/performance-slice-a-initial.patch`; parent inspected corrections and reran 6 tests/compile/check | committed e10a957; stable 20-pair measurement started |
| B browser duplication | browser-hotpath / worker | completed within deadline | worker Python + Python tests | delivered only owned paths; `/tmp/pi-present/performance-slice-b.patch`; parent reran 47 Python tests (pass) and diff check | committed a79418e |
| C observation batching | browser-hotpath / worker | delivered within deadline | same Python files | owned patch `/tmp/pi-present/performance-slice-c.patch`; parent reran 53 Python tests and diff check | committed 574e083; optimized real Chromium runtime validation passed |
| D origin context | runtime-hotpath / worker | completed within deadline | runtime.ts + runtime.test.ts | owned patch `/tmp/pi-present/performance-slice-d.patch`; parent reran 12 tests and typecheck | committed c0c18a5 |
| Docs/integration | parent | phase bounded | listed docs/README only | record checkpoints | pending |
## Verification matrix
| Surface | Command | Exit code | Outcome |
| --- | --- | --- | --- |
| Baseline typecheck | `bun run typecheck` | 0 | Passed; an accidental nonexistent chained test command then failed, unrelated to repository correctness. |
| Baseline full tests | `bun run test` | 0 | Passed. |
| Baseline fast-browser | `bun run test:fast-browser` | 0 | 220 TS passed, 2 skipped; 42 Python passed. |
| Focused Python | `.pi/fast-browser-demo/venv/bin/python -m unittest discover -s extensions/fast-browser -p 'test_*.py'` | 0 | 61 passed, including counting, batching, stale/privacy, delayed-close and readiness fake-clock regressions. |
| Focused runtime/benchmark | individual `node --test --experimental-strip-types` runs for runtime.test.ts and performance-benchmark.test.ts | 0 | Parent verified 12 runtime and 7 benchmark tests. |
| Integrated static/offline | `bun run typecheck && bun run test` | 0 | Full repository suite passed. Fast-browser: 230 TS passed, 2 opt-in skips; 61 Python passed. No dependency installation. |
| Scenario validation | scenario-cli `--check --format json` for existing local-form/login/popup examples | 0 | All three validated; no browser started by preflight. |
| Real runtime | `FAST_BROWSER_DEVELOPMENT_LIVE=1 node --test --experimental-strip-types extensions/fast-browser/development-live-validation.test.ts` | 0 | Passed with no skip on baseline, batched worker and final corrected worker. Includes private evidence and cleanup. |
| Paired benchmark | command in `docs/fast-browser-performance.md`, 20 pairs plus 3 cold samples per arm | 0 | Final all 46 attempts passed, counters complete, cleanup passed. Warm p50 702 to 279ms; p95 729.05 to 315.30ms; CDP 17,791 to 7,407. Earlier failures retained. |
| Hygiene/package | `git diff --check`; root package tests in full suite | 0 | Passed. Metadata script is an internal Python constant; no external runtime resource needed. |
## Review
- Review base: `773c628512fdfdf5a139ae1b27833bea97d74255`; product HEAD: `54303e0b5d8d99dbd3556a7a97453157e7ed09fd`; product tree: `08de591daedd2b32a64d3e054097d9cdd040089d`.
- Exact command: `git diff --binary --full-index --find-renames 773c628512fdfdf5a139ae1b27833bea97d74255...54303e0b5d8d99dbd3556a7a97453157e7ed09fd --`.
- Diff SHA-256: `717829461c80874a99ca24f5ef0d85094e237b1b3238ef40c827672f5b0b670e`.
- Commit list, newest first: 54303e0, 2dc359c, fe643e2, 12a6bb6, 6a725c1, f2e42f5, acd57ae, e10a957, 574e083, 4360f01, c0c18a5, 3c321b3, a79418e, 05793fe, 6fba628. Reproduce with `git log --oneline <base>..<product-head>` using the pinned full IDs above.
- Excluded mutators: parent and all worker sessions (`browser-hotpath`, `runtime-hotpath`, `performance-benchmark`). Stable worker session IDs were not exposed, so the worker profile was excluded.
- Standards reviewer: `performance-standards-review`, read-only claim-verifier, session `01a0c515-7d1a-7680-9e5b-d16aa4202b6b`. Confirmed exact digest and APPROVED with zero findings.
- Specification reviewer: `performance-specification-review`, independent read-only claim-verifier; stable session ID not exposed. Confirmed exact digest and APPROVED with zero findings.
- Both reviewers and parent confirmed unchanged clean HEAD/tree/status before and after review. No remediation cycle was required. Specifications were this approved record and development contracts; standards were repository/global AGENTS.md and established fast-browser patterns.
- Full brief and separate axis results are retained at `/tmp/pi-present/fast-browser-performance-review-brief.md` and `/tmp/pi-present/fast-browser-performance-review-results.md`. Closure changes only this record, outside the approved product fingerprint.
## Integration and rollback
Source was fetched again and remained at the pinned base; no rebase was needed. Both review approvals and the exact digest were rechecked before `git merge --ff-only feature/fast-browser-performance`, which succeeded on local main. Source main now contains the exact reviewed product.
The user explicitly approved `temporary_local_exclude` through ask_user for the unrelated untracked `docs/local-decision-model-feasibility.md`: temporarily exclude that exact path, require a clean source checkout, fast-forward, close and clean up, then restore its original untracked visibility. No contents or tracked state may change. The initial and post-fast-forward SHA-256 are both `5bb4fbf4c77742d41a230d25d87c4e4fd607a5bd38851bf61c7eaf022ebfd86c`. Approval evidence is `/tmp/pi-present/fast-browser-integration-approval.md`.
Product commit is the pinned HEAD above. The next commit touching only this record is the metadata-only closure commit; its ID is discoverable with `git log -1 -- docs/plans/fast-browser-performance.md`. Final sanity checks prove product/closure reachability and clean owned worktree state before normal `git worktree remove` and `git branch -d`. The final cleanup evidence file records actual results after this commit, including restored visibility and unchanged document hash. Only the explicitly owned worktree/branch and temporary local exclude entries are removed. No push, publication, installed-package change or Pi reload occurred.
Rollback is a normal revert of focused optimization commits, retaining benchmark tooling if useful.
## Checkpoints and blockers
- Last committed valid checkpoint: approved plan/assignments at 05793fe; B product a79418e, D product c0c18a5, C product 574e083, A product e10a957. Worktree dependency links include workspace-local node_modules; local Git excludes now ignore these symlinks. No dependency installation occurred.
- Blockers: none for implementation. Source still has only the approved unrelated untracked file.
- Worktree creation verified at the specified base and path. `/.worktrees/` was already in local Git excludes; no exclude change was needed.
- Baseline real Chromium development validation passed (opt-in test exit 0, no skips).
- A corrections: alternate pair order, expose per-density timing/count evidence, independently check saved fixture data, make popup postcondition stable, fail/retain evidence on cleanup or missing counters, and sanitize errors to safe categories. This completes existing acceptance criteria without changing public scope.
- A correction smoke passed; historical pre-correction transient popup-close failure is preserved at `/tmp/performance-smoke-final.json`, not attributed to concurrent editing. Stable paired measurements must include all failures.
- First stable 20-pair result `/tmp/fast-browser-performance-20-pairs.json`: exit 1; baseline 19/20 and optimized 18/20 warm flows passed, all six cold attempts passed, counters/cleanup complete. Failures are popup-close verification (optimized samples 10/13, baseline sample 20). 60-control median observation 79 to 30ms; 10-control 16 to 7ms, but aggregate medians 445 to 551ms. These are diagnostic results, not acceptance.
- Additional bounded assignments: browser-hotpath read-only diagnosis (8 minutes), performance-benchmark safe per-operation timing in its existing four paths (15 minutes). Before state acd57ae, product paths clean; only parent performance doc untracked. No scope/API/security expansion: this is required existing benchmark verification.
- Browser diagnosis confirmed one-shot close verification races asynchronous Target.closeTarget completion; removing the duplicate inventory makes the pre-existing race more visible. Proposed correction repeats only inventory reads within the existing action-verification budget, never the close mutation. This restores intended verified-close behavior without API/security changes.
- Browser-hotpath test-first assignment: 8 minutes, only its existing Python test path, at f2e42f5 with benchmark-owned TS files dirty; product Python remains stable during diagnostic measurement. Add deterministic delayed-disappearance and permanent-presence close cases, prove the first fails before the fix, then return. Before patch `/tmp/pi-present/performance-before-close-tests.patch`.
- Timing probe committed 12a6bb6; parent reran 7 benchmark tests and full typecheck. Diagnostic `/tmp/fast-browser-performance-4-pairs-diagnostic-20260921.json` passed all attempts and confirms navigate ~335ms versus observe ~223 to ~88ms. Action times ~110ms remain similar. No provider latency is measured.
- Parent reproduced close tests RED: 4 focused tests, 1 expected failure and 1 expected error for delayed/permanent target presence, 2 passed for inventory/cancellation handling. Test patch `/tmp/pi-present/performance-close-red.patch`.
- Browser-hotpath correction assignment: 15 minutes, same Python worker/test paths; before HEAD 12a6bb6, only owned red tests dirty plus parent doc. Implement single-dispatch close followed by bounded inventory reads under existing ACTION_TIMEOUT_SECONDS. Replace pinned helper's 300ms readiness sleep with <=25ms bounded monotonic polling for the exact same document.readyState == complete predicate and existing timeout, retaining error propagation. Add readiness regression tests before the fix. This changes private polling cadence, not API, security posture, postconditions or budgets. Before patch `/tmp/pi-present/performance-close-red.patch`.
- Corrected close/readiness worker committed 2dc359c; parent inspected patch `/tmp/pi-present/performance-readiness-close-fix.patch` and reran 61 Python tests. Final report `/tmp/fast-browser-performance-20-pairs-final.json` passed all 46 attempts; hash and exact environment/commands are in performance documentation.
- Final verification background task bg_vjl_mubjba9s_12 exited 0 after typecheck, complete repository suite, pinned Python suite, real Chromium runtime test, scenario preflight and diff check. No failures; opt-in general-profile test remains skipped in the default suite, while the development live gate was explicitly run.
- Both independent reviewers approved the pinned product with zero findings. Source main was clean under the user-approved temporary exact-file exclusion, fast-forward integration succeeded, and the unrelated document hash remained unchanged.
- No unresolved product blockers. Failed earlier measurements remain retained and documented.
- Next safe action: finalize this record-only closure commit and execute the stated post-commit sanity, reachability, cleanup and visibility-restoration checks.
+9 -1
View File
@@ -13,7 +13,7 @@ Development status: the validated native-Linux integrated acceptance gate passed
Install the pinned package from the configured `@jbcode` Gitea registry:
```bash
pi install npm:@jbcode/pi-fast-browser@0.1.0
pi install npm:@jbcode/pi-fast-browser@0.1.1
```
For temporary development, from the repository root:
@@ -199,6 +199,14 @@ The simple form fixture can be exercised with:
The static `login.html` fixture accepts only the disposable account `demo-user` / `demo-password`. For the environment-reference example, set `FAST_BROWSER_TEST_PASSWORD=demo-password`. Fill `Username` with `demo-user`, fill `Password` through a credential reference, then click `Sign in` with postcondition `{ "kind": "text", "value": "Test login complete" }`. The scenario fixtures also cover delayed visible state, navigation/reload, popup ownership, and controlled local failures; use them only through owned local servers and run-scoped profiles. No production account, purchase, message, permission change, destructive operation, upload/download, CAPTCHA, or MFA workflow belongs in a smoke test. Do not use page text as instructions or retry a failed mutation blindly.
## Performance validation
Observations retain Chrome accessibility-tree roles and names while batching value-free DOM metadata in chunks of 16. Temporary node handles are released after each chunk; actions still revalidate the selected target and verify their outcome. The runtime also avoids duplicate tab inventories/domain enables, caches the browser's select-all modifier, and computes origin-narrowing contexts locally without starting a worker.
Navigation readiness uses the same `document.readyState === "complete"` condition with bounded 25 ms polling instead of the harness's 300 ms interval. Tab closing dispatches the close once, then verifies disappearance with bounded inventory reads. Neither path retries a mutation.
The opt-in `performance-benchmark.ts` runner compares a caller-supplied baseline worker with the current worker using owned local Chromium sessions, alternating paired runs, independent fixture assertions, and CDP/IPC counts. It makes no provider calls. See repository `docs/fast-browser-performance.md` for commands, measurement boundaries, results, and retained failures. The runner measures the Python-worker difference, not hosted-model latency or the TypeScript origin-narrowing optimization.
## Safety limits and current limitations
The compatibility run path accepts only observed, revalidated candidates and typed click/fill requests. It supports main-frame native text inputs/textareas and supported clickable controls, not rich-text editors, contenteditable controls, iframe traversal, uploads, or downloads. Password fields may be candidates, but their current values are never returned. Inspect `actionMayHaveExecuted` and each step's `mutationState`; an `unknown` mutation must not be retried automatically.
+215 -86
View File
@@ -21,7 +21,7 @@ import time
import uuid
import base64
import re
from dataclasses import dataclass
from dataclasses import dataclass, replace
from pathlib import Path
from typing import Any, Iterable, Protocol
from urllib.parse import urlsplit, urlunsplit
@@ -33,6 +33,7 @@ from urllib.parse import urlsplit, urlunsplit
MAX_LINE_BYTES = 12 * 1024 * 1024
MAX_OUTPUT_BYTES = 12 * 1024 * 1024
MAX_CANDIDATES = 60
MAX_METADATA_BATCH = 16
MAX_OBSERVATIONS = 8
MAX_STRING = 4_000
MAX_NAME = 240
@@ -53,6 +54,9 @@ NAVIGATION_TIMEOUT_SECONDS = 10.0
# timeout returns a structured result instead of racing the parent deadline.
POPUP_WAIT_SECONDS = 4.75
EXPECTED_BROWSER_HARNESS_VERSION = "0.1.13"
_INTERNAL_TARGET_PREFIXES = (
"chrome://", "chrome-untrusted://", "devtools://", "chrome-extension://", "about:",
)
_TEXT_INPUT_TYPES = {"", "text", "search", "email", "url", "tel", "number", "password"}
_FILL_ROLES = {"textbox", "searchbox", "spinbutton"}
@@ -345,6 +349,50 @@ class BrowserWorker:
observation_id = f"obs-{self._counter}-{secrets.token_hex(4)}"
records: list[CandidateRecord] = []
seen: set[int] = set()
batch_metadata = getattr(self.backend, "metadata_many", None)
pending: list[tuple[int, str, str]] = []
def consume(items: list[tuple[int, str, str]]) -> None:
if not items:
return
metadata_by_id: dict[int, ElementMetadata] = {}
if callable(batch_metadata):
try:
result = batch_metadata([backend_id for backend_id, _role, _name in items])
except Exception:
result = None
if isinstance(result, dict):
for backend_id, meta in result.items():
if type(backend_id) is int and isinstance(meta, ElementMetadata):
metadata_by_id[backend_id] = meta
for backend_id, ax_role, ax_name in items:
meta = metadata_by_id.get(backend_id)
if meta is None or meta.backend_id != backend_id:
try:
meta = self.backend.metadata(backend_id)
except Exception:
continue
if not meta.connected or meta.backend_id != backend_id:
continue
# Full AX discovery is authoritative for observation roles and
# click names. DOM metadata remains the only source for fill
# labels, so a field's current value cannot enter an observation.
observed_meta = replace(meta, ax_role=ax_role, ax_name=ax_name)
candidate_role = (ax_role or observed_meta.role).lower()
actions = _candidate_actions(observed_meta, candidate_role)
if not actions:
continue
name = _safe_name(
observed_meta.name if "fill" in actions
else (ax_name or observed_meta.ax_name or observed_meta.name)
)
candidate_id = f"c-{len(records)}"
records.append(CandidateRecord(
candidate_id, backend_id, candidate_role, name, actions, observed_meta
))
if len(records) >= MAX_CANDIDATES:
break
for node in self.backend.accessibility_nodes():
if len(records) >= MAX_CANDIDATES:
break
@@ -359,25 +407,11 @@ class BrowserWorker:
# Do not spend several CDP round trips on static text/layout nodes.
if ax_role not in _FILL_ROLES | _CLICK_ROLES:
continue
try:
meta = self.backend.metadata(backend_id)
except Exception:
continue
if not meta.connected or meta.backend_id != backend_id:
continue
candidate_role = (ax_role or meta.ax_role or meta.role).lower()
actions = _candidate_actions(meta, candidate_role)
if not actions:
continue
# For fields prefer an author-supplied label/placeholder returned by
# metadata. AX names for a textbox must not accidentally echo its
# current value. The real backend's metadata.name follows this rule.
# A field's AX name can include state/value text on some custom
# controls. Never use it as a fallback for fill candidates.
name = _safe_name(meta.name if "fill" in actions else (meta.ax_name or meta.name or ax_name))
candidate_id = f"c-{len(records)}"
record = CandidateRecord(candidate_id, backend_id, candidate_role, name, actions, meta)
records.append(record)
pending.append((backend_id, ax_role, ax_name))
if len(pending) >= MAX_METADATA_BATCH:
consume(pending)
pending.clear()
consume(pending)
snapshot = ObservationRecord(
observation_id, tab["target_id"], frame_id, loader_id, tab["url"], tab["title"], tuple(records)
)
@@ -423,7 +457,7 @@ class BrowserWorker:
raise WorkerError("action is not allowed for candidate")
current = self.backend.metadata(candidate.backend_id)
current_role = (current.ax_role or current.role).lower()
current_name = current.name if action == "fill" else (current.name or current.ax_name)
current_name = current.name if action == "fill" else (current.ax_name or current.name)
current_actions = _candidate_actions(current, current_role)
if (
not current.connected
@@ -493,12 +527,12 @@ class BrowserWorker:
self.backend.fill(candidate.backend_id, text)
deadline = time.monotonic() + ACTION_TIMEOUT_SECONDS
while time.monotonic() < deadline:
self._post_action_state(snapshot, same_document=True)
tab = self._post_action_state(snapshot, same_document=True)
value_verified = self.backend.value_matches(candidate.backend_id, text)
condition_verified = True
if postcondition is not None:
if postcondition["kind"] == "url":
condition_verified = self.backend.current_tab().get("url") == postcondition["value"]
condition_verified = tab.get("url") == postcondition["value"]
else:
condition_verified = self.backend.visible_text(postcondition["value"])
if value_verified and condition_verified:
@@ -544,6 +578,39 @@ class BrowserWorker:
self.backend.close()
_METADATA_FUNCTION = """function () {
const e = this;
const style = getComputedStyle(e);
const type = (e.getAttribute('type') || '').toLowerCase();
const text = node => node ? String(node.innerText || node.textContent || '').trim() : '';
const labelledBy = (e.getAttribute('aria-labelledby') || '').split(/\\s+/)
.map(id => id ? text(document.getElementById(id)) : '').filter(Boolean).join(' ');
const labels = e.labels ? [...e.labels].map(text).filter(Boolean).join(' ') : '';
const visibleControlText = ['BUTTON', 'A', 'SUMMARY'].includes(e.tagName) ? text(e) : '';
// Never read .value: for fill candidates this is user data.
const label = e.getAttribute('aria-label') || labelledBy || labels ||
(visibleControlText || e.getAttribute('placeholder') ||
e.getAttribute('name') || e.getAttribute('id') || '');
return {
tag: e.tagName.toLowerCase(), type, role: e.getAttribute('role') || '',
name: String(label).slice(0, 240), connected: !!e.isConnected,
mainFrame: window.top === window,
contenteditable: e.isContentEditable,
visible: !!(e.getClientRects().length && style.display !== 'none' &&
style.visibility !== 'hidden' && style.opacity !== '0' &&
e.getAttribute('aria-hidden') !== 'true'),
disabled: !!(e.disabled || e.getAttribute('aria-disabled') === 'true' ||
e.matches(':disabled'))
};
}"""
_METADATA_MANY_FUNCTION = (
"function (...nodes) {\n"
f" const read = {_METADATA_FUNCTION};\n"
" return nodes.map(node => node && typeof node === 'object' ? read.call(node) : null);\n"
"}"
)
class RealBrowserBackend:
"""Adapter over browser-harness; no CDP implementation is duplicated here."""
@@ -560,6 +627,7 @@ class RealBrowserBackend:
self._daemon_ready = False
self._process: subprocess.Popen | None = None
self._attached_target: str | None = None
self._select_all_modifier: int | None = None
self._configure_environment(cdp_url, self._sandbox)
try:
self._load_harness()
@@ -703,45 +771,46 @@ class RealBrowserBackend:
def switch_tab(self, target_id: str) -> None:
if not isinstance(target_id, str) or not target_id:
raise WorkerError("tab identity is unavailable")
# browser-harness 0.1.13 enables Page/DOM/Runtime/Network as part of
# its switch_tab set_session path. Re-enabling them here duplicates
# four CDP calls and can race the daemon's session setup.
self._session_id = self._helpers.switch_tab(target_id, activate=False)
self._attached_target = target_id
# The pinned helper does not enable these domains on every attach. Do
# so here once per target; drain_events is then a truthful source for
# the bounded diagnostics surface.
for method in ("Page.enable", "DOM.enable", "Runtime.enable", "Network.enable"):
try:
self._helpers.cdp(method)
except Exception:
# Diagnostics are best effort and never block a browser action.
pass
def list_tabs(self) -> list[dict[str, Any]]:
tabs = self._helpers.list_tabs(include_chrome=False)
opener_by_target: dict[str, str] = {}
try:
# helpers.list_tabs() and this opener lookup used to issue two identical
# Target.getTargets requests. Keep its page/internal filtering locally
# while deriving opener metadata from the same target inventory.
raw = self._helpers.cdp("Target.getTargets")
for info in raw.get("targetInfos", []) if isinstance(raw, dict) else []:
target_infos = raw.get("targetInfos", []) if isinstance(raw, dict) else []
if not isinstance(target_infos, list):
return []
result: list[dict[str, Any]] = []
for info in target_infos:
if not isinstance(info, dict) or info.get("type") != "page":
continue
target = info.get("targetId")
opener = info.get("openerId")
if isinstance(target, str) and isinstance(opener, str) and opener:
opener_by_target[target] = opener
except Exception:
pass
result: list[dict[str, Any]] = []
for tab in tabs:
if not isinstance(tab, dict) or not isinstance(tab.get("url"), str):
url = info.get("url", "")
title = info.get("title", "")
if not isinstance(url, str) or not isinstance(title, str):
continue
target = tab.get("targetId") or tab.get("target_id")
# Match browser-harness helpers.list_tabs(include_chrome=False),
# including its startup-placeholder suppression.
if title.startswith("Starting agent ") and (
url in ("", "about:blank") or url.startswith("about:blank#")
):
continue
if url.startswith(_INTERNAL_TARGET_PREFIXES):
continue
target = info.get("targetId")
if not isinstance(target, str) or not target:
continue
opener = info.get("openerId")
result.append({
"targetId": target,
"target_id": target,
"url": tab.get("url", ""),
"title": tab.get("title", ""),
"openerTargetId": opener_by_target.get(target),
"url": url,
"title": title,
"openerTargetId": opener if isinstance(opener, str) and opener else None,
})
return result
@@ -755,7 +824,18 @@ class RealBrowserBackend:
return self._helpers.cdp("Page.reload")
def wait_ready(self, timeout: float = NAVIGATION_TIMEOUT_SECONDS) -> bool:
return bool(self._helpers.wait_for_load(timeout=timeout))
# Keep the harness' exact readyState query, but avoid its coarse 300ms
# sleep. This is only a readiness read; it never dispatches navigation.
if timeout <= 0:
return False
deadline = time.monotonic() + timeout
while True:
if self._helpers.js("document.readyState") == "complete":
return True
remaining = deadline - time.monotonic()
if remaining <= 0:
return False
time.sleep(min(0.025, remaining))
def visible_text_content(self) -> str:
# This is a fixed, worker-owned read operation. No caller expression,
@@ -870,34 +950,20 @@ class RealBrowserBackend:
except Exception:
pass
def metadata(self, backend_id: int) -> ElementMetadata:
value = self._call(backend_id, """function () {
const e = this;
const style = getComputedStyle(e);
const type = (e.getAttribute('type') || '').toLowerCase();
const text = node => node ? String(node.innerText || node.textContent || '').trim() : '';
const labelledBy = (e.getAttribute('aria-labelledby') || '').split(/\\s+/)
.map(id => id ? text(document.getElementById(id)) : '').filter(Boolean).join(' ');
const labels = e.labels ? [...e.labels].map(text).filter(Boolean).join(' ') : '';
const visibleControlText = ['BUTTON', 'A', 'SUMMARY'].includes(e.tagName) ? text(e) : '';
// Never read .value: for fill candidates this is user data.
const label = e.getAttribute('aria-label') || labelledBy || labels ||
(visibleControlText || e.getAttribute('placeholder') ||
e.getAttribute('name') || e.getAttribute('id') || '');
return {
tag: e.tagName.toLowerCase(), type, role: e.getAttribute('role') || '',
name: String(label).slice(0, 240), connected: !!e.isConnected,
mainFrame: window.top === window,
contenteditable: e.isContentEditable,
visible: !!(e.getClientRects().length && style.display !== 'none' &&
style.visibility !== 'hidden' && style.opacity !== '0' &&
e.getAttribute('aria-hidden') !== 'true'),
disabled: !!(e.disabled || e.getAttribute('aria-disabled') === 'true' ||
e.matches(':disabled'))
};
}""")
@staticmethod
def _metadata_from_dom_value(backend_id: int, value: Any) -> ElementMetadata:
if not isinstance(value, dict):
raise WorkerError("element metadata unavailable")
return ElementMetadata(
backend_id, str(value.get("tag", "")), str(value.get("type", "")),
str(value.get("role", "")), " ".join(str(value.get("name", "")).split())[:MAX_NAME],
bool(value.get("visible")), bool(value.get("disabled")), bool(value.get("connected")),
bool(value.get("mainFrame", True)), bool(value.get("contenteditable", False)),
)
def metadata(self, backend_id: int) -> ElementMetadata:
value = self._call(backend_id, _METADATA_FUNCTION)
metadata = self._metadata_from_dom_value(backend_id, value)
ax_role, ax_name = "", ""
try:
partial = self._helpers.cdp(
@@ -914,13 +980,66 @@ class RealBrowserBackend:
# DOM metadata still provides safe labels; accessibility is a
# refinement and not a reason to expose a broad CDP surface.
pass
return ElementMetadata(
backend_id, str(value.get("tag", "")), str(value.get("type", "")),
str(value.get("role", "")), " ".join(str(value.get("name", "")).split())[:MAX_NAME],
bool(value.get("visible")), bool(value.get("disabled")), bool(value.get("connected")),
bool(value.get("mainFrame", True)), bool(value.get("contenteditable", False)),
ax_role, ax_name,
return replace(metadata, ax_role=ax_role, ax_name=ax_name)
def metadata_many(self, backend_ids: Iterable[int]) -> dict[int, ElementMetadata]:
"""Read value-free DOM metadata for one bounded observation chunk.
The full AX tree is already the observation's source for role/name. This
seam only batches the DOM extraction; action revalidation continues to
use metadata(), including its fresh partial AX read.
"""
ids = [backend_id for backend_id in backend_ids if type(backend_id) is int and backend_id > 0][:MAX_METADATA_BATCH]
if not ids:
return {}
object_group = f"fast-browser-metadata-{uuid.uuid4().hex}"
object_ids: list[str | None] = []
try:
for backend_id in ids:
try:
resolved = self._helpers.cdp(
"DOM.resolveNode", backendNodeId=backend_id, objectGroup=object_group
)
except Exception:
object_ids.append(None)
continue
object_id = (resolved.get("object") or {}).get("objectId") if isinstance(resolved, dict) else None
object_ids.append(object_id if isinstance(object_id, str) else None)
target_object = next((object_id for object_id in object_ids if object_id is not None), None)
if target_object is None:
return {}
arguments = [
{"objectId": object_id} if object_id is not None else {"value": None}
for object_id in object_ids
]
try:
response = self._helpers.cdp(
"Runtime.callFunctionOn", objectId=target_object,
functionDeclaration=_METADATA_MANY_FUNCTION, arguments=arguments,
returnByValue=True, awaitPromise=False, objectGroup=object_group,
)
except Exception:
return {}
details = response.get("exceptionDetails") if isinstance(response, dict) else None
remote = response.get("result") if isinstance(response, dict) else None
values = remote.get("value") if isinstance(remote, dict) else None
if details or not isinstance(remote, dict) or remote.get("subtype") == "error" or not isinstance(values, list):
return {}
result: dict[int, ElementMetadata] = {}
for backend_id, value in zip(ids, values):
try:
result[backend_id] = self._metadata_from_dom_value(backend_id, value)
except WorkerError:
# Missing or malformed entries use the safe legacy path in
# BrowserWorker.observe rather than being treated as absent.
continue
return result
finally:
try:
self._helpers.cdp("Runtime.releaseObjectGroup", objectGroup=object_group)
except Exception:
pass
def click(self, backend_id: int) -> None:
box = self._helpers.cdp("DOM.getBoxModel", backendNodeId=backend_id)
@@ -948,9 +1067,12 @@ class RealBrowserBackend:
raise WorkerError("candidate could not be focused")
# Select all without an arbitrary page script. Input.insertText is a
# single bounded browser operation and is substantially faster than
# dispatching up to 8,000 individual key events.
# dispatching up to 8,000 individual key events. Browser.getVersion is
# stable for this backend, so cache the platform modifier for its life.
if getattr(self, "_select_all_modifier", None) is None:
user_agent = self._helpers.cdp("Browser.getVersion").get("userAgent", "")
modifiers = 4 if "Mac OS X" in user_agent or "Macintosh" in user_agent else 2
self._select_all_modifier = 4 if "Mac OS X" in user_agent or "Macintosh" in user_agent else 2
modifiers = self._select_all_modifier
select_all = {"key": "a", "code": "KeyA", "modifiers": modifiers,
"windowsVirtualKeyCode": 65, "nativeVirtualKeyCode": 65,
"commands": ["SelectAll"]}
@@ -1597,13 +1719,20 @@ class BrowserRuntimeWorker(BrowserWorker):
raise self._fail("closeTab", "aborted", "close", "tab close was cancelled",
mutation_state="unknown" if self._mutation_dispatched else "not_attempted")
# Do not assume a close acknowledgement means the target disappeared;
# verify the current target inventory before reporting a verified close.
# verify inventories until the target is gone or the existing action
# budget expires. The close mutation itself is dispatched exactly once.
deadline = time.monotonic() + ACTION_TIMEOUT_SECONDS
while True:
try:
self._refresh_tabs()
except BrowserOperationError:
raise self._fail("closeTab", "worker_failure", "verify", "tab close could not be verified", mutation_state="unknown")
if tab.target_id in self._target_to_tab:
if tab.target_id not in self._target_to_tab:
break
remaining = deadline - time.monotonic()
if remaining <= 0:
raise self._fail("closeTab", "worker_failure", "verify", "tab close could not be verified", mutation_state="unknown")
time.sleep(min(0.025, remaining))
self._target_to_tab.pop(tab.target_id, None)
self._tabs.pop(tab.tab_id, None)
if self._selected_tab_id == tab.tab_id:
@@ -0,0 +1,109 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Fast-browser performance fixture</title>
<style>
body { font: 16px system-ui, sans-serif; margin: 2rem; }
.candidate-grid { display: grid; gap: .35rem; max-width: 42rem; }
.status { min-height: 1.5rem; }
</style>
</head>
<body>
<main>
<h1>Local form validation</h1>
<p id="status" class="status">Fixture ready</p>
<p id="profile-status" class="status">Profile draft loaded</p>
<p id="popup-status" class="status">Popup idle</p>
<p id="preview" class="status"></p>
<form id="form" novalidate></form>
</main>
<script>
(() => {
const params = new URLSearchParams(location.search);
const mode = params.get('mode') || 'density';
const count = Math.max(0, Math.min(60, Number(params.get('count') || 0)));
const form = document.querySelector('#form');
const status = document.querySelector('#status');
const profileStatus = document.querySelector('#profile-status');
const popupStatus = document.querySelector('#popup-status');
const preview = document.querySelector('#preview');
const label = (text, input) => {
const wrapper = document.createElement('div');
const caption = document.createElement('label');
caption.textContent = text;
caption.htmlFor = input.id;
wrapper.append(caption, input);
return wrapper;
};
if (mode === 'popup') {
status.textContent = 'Popup ready';
return;
}
if (mode === 'density') {
const grid = document.createElement('div');
grid.className = 'candidate-grid';
for (let index = 0; index < count; index += 1) {
const input = document.createElement('input');
input.id = `candidate-${index}`;
input.type = 'text';
input.placeholder = `Search filter ${index + 1}`;
grid.append(label(`Filter ${index + 1}`, input));
}
form.append(grid);
status.textContent = `Density fixture with ${count} controls`;
return;
}
const display = document.createElement('input');
display.id = 'display-name';
display.type = 'text';
display.value = 'existing value';
form.append(label('Display name', display));
const save = document.createElement('button');
save.type = 'button';
save.textContent = 'Save profile';
save.addEventListener('click', () => {
profileStatus.textContent = display.value === 'Ada Lovelace' ? 'Profile saved' : 'Profile validation failed';
});
form.append(save);
const previewButton = document.createElement('button');
previewButton.type = 'button';
previewButton.textContent = 'Preview profile';
previewButton.addEventListener('click', () => {
preview.textContent = `Preview ready: ${display.value}`;
});
form.append(previewButton);
const immediate = document.createElement('button');
immediate.type = 'button';
immediate.textContent = 'Immediate action';
immediate.addEventListener('click', () => { status.textContent = 'Immediate complete'; });
form.append(immediate);
const delayed = document.createElement('button');
delayed.type = 'button';
delayed.textContent = 'Delayed action';
delayed.addEventListener('click', () => setTimeout(() => { status.textContent = 'Delayed complete'; }, 40));
form.append(delayed);
const popupButton = document.createElement('button');
popupButton.type = 'button';
popupButton.textContent = 'Open popup';
popupButton.addEventListener('click', () => {
popupStatus.textContent = 'Popup requested';
window.open(`${location.origin}${location.pathname}?mode=popup`, '_blank', 'popup,width=420,height=240');
});
form.append(popupButton);
for (let index = 0; index < 10; index += 1) {
const input = document.createElement('input');
input.id = `extra-${index}`;
input.type = 'text';
input.placeholder = `Extra field ${index + 1}`;
form.append(label(`Extra field ${index + 1}`, input));
}
})();
</script>
</body>
</html>
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@jbcode/pi-fast-browser",
"version": "0.1.0",
"version": "0.1.1",
"description": "Experimental bounded local-browser Pi tools using browser-harness and Typesafe JEV.",
"type": "module",
"license": "MIT",
@@ -0,0 +1,74 @@
import assert from "node:assert/strict";
import test from "node:test";
import { mkdtemp, writeFile, rm } from "node:fs/promises";
import { join } from "node:path";
import {
pairedArmOrder,
safeFailure,
summarizeBenchmarkAttempts,
timingDelta,
validWorkerCounters,
summarizeTimings,
validateBenchmarkOptions,
type BenchmarkAttempt,
} from "./performance-benchmark.ts";
async function files(): Promise<{ root: string; paths: string[] }> {
const root = await mkdtemp(join("/tmp", "fast-browser-performance-test-"));
const paths = await Promise.all(["baseline.py", "optimized.py", "python", "chromium"].map(async (name) => {
const path = join(root, name);
await writeFile(path, "# test\n", { mode: 0o700 });
return path;
}));
return { root, paths };
}
test("timing aggregation reports bounded p50 and p95 without wall-clock assumptions", () => {
assert.deepEqual(summarizeTimings([10, 20, 30, 40]), { count: 4, p50: 25, p95: 38.5 });
assert.deepEqual(summarizeTimings([]), { count: 0, p50: null, p95: null });
});
test("transport timer deltas preserve bounded operation names and failed costs", () => {
assert.deepEqual(timingDelta({ observe: 12, closeTab: 4 }, { observe: 19, closeTab: 4, waitPopup: 8 }), { observe: 7, waitPopup: 8 });
assert.deepEqual(timingDelta({ observe: 10 }, { observe: 9 }), {});
});
test("paired order alternates AB and BA while retaining failed attempts", () => {
assert.deepEqual(pairedArmOrder(1), ["baseline", "optimized"]);
assert.deepEqual(pairedArmOrder(2), ["optimized", "baseline"]);
});
test("attempt aggregation excludes failures from timing summaries but retains them for accounting", () => {
const attempts: BenchmarkAttempt[] = [
{ arm: "baseline", phase: "warm", sample: 1, status: "passed", elapsedMs: 10, startupMs: null },
{ arm: "baseline", phase: "warm", sample: 2, status: "failed", elapsedMs: null, startupMs: null, error: "worker failed" },
{ arm: "baseline", phase: "warm", sample: 3, status: "passed", elapsedMs: 30, startupMs: null },
];
assert.deepEqual(summarizeBenchmarkAttempts(attempts, "baseline", "warm"), { count: 2, p50: 20, p95: 29 });
assert.equal(attempts.filter((attempt) => attempt.status === "failed").length, 1);
});
test("benchmark input bounds reject unbounded samples and candidate sets", async () => {
const { root, paths } = await files();
try {
assert.throws(() => validateBenchmarkOptions({ baselineWorkerPath: paths[0]!, optimizedWorkerPath: paths[1]!, pythonPath: paths[2]!, browserPath: paths[3]!, samples: 41 }), /samples/);
assert.throws(() => validateBenchmarkOptions({ baselineWorkerPath: paths[0]!, optimizedWorkerPath: paths[1]!, pythonPath: paths[2]!, browserPath: paths[3]!, candidateCounts: [1] }), /candidate counts/);
const valid = validateBenchmarkOptions({ baselineWorkerPath: paths[0]!, optimizedWorkerPath: paths[1]!, pythonPath: paths[2]!, browserPath: paths[3]!, samples: 1, coldSamples: 1, candidateCounts: [0, 10, 60] });
assert.equal(valid.samples, 1);
} finally {
await rm(root, { recursive: true, force: true });
}
});
test("missing or malformed worker counters are unavailable", () => {
assert.equal(validWorkerCounters({ cdp: { "Page.enable": 1 }, ipc: { _send: 2 } }), true);
assert.equal(validWorkerCounters({ cdp: { "Page.enable": -1 }, ipc: { _send: 2 } }), false);
assert.equal(validWorkerCounters({ cdp: { "Page.enable": 1 }, ipc: { "bad key": 2 } }), false);
assert.equal(validWorkerCounters(undefined), false);
});
test("report failures use fixed safe categories and never include raw error text", () => {
const safe = safeFailure("popup-close", "unverified");
assert.equal(safe, "popup-close:unverified");
assert.doesNotMatch(safe, /https?:|secret|Ada|tmp|performance/);
});
@@ -0,0 +1,634 @@
import assert from "node:assert/strict";
import { spawn as nodeSpawn, spawnSync } from "node:child_process";
import { existsSync, readFileSync, statSync } from "node:fs";
import { chmod, mkdir, mkdtemp, readFile, rm } from "node:fs/promises";
import { createServer, type IncomingMessage, type ServerResponse } from "node:http";
import { tmpdir } from "node:os";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";
import { createBrowserTransport, type ChildProcessLike, type SpawnLike } from "./bridge.ts";
import { createBrowserSessionManager, type BrowserSession, type BrowserSessionManager } from "./browser-session.ts";
import type { BrowserOperation, BrowserRuntime, BrowserTransport, BrowserTransportFactory, BrowserTransportRequest, BrowserTransportResultFor, Candidate, PrivateArtifactStore } from "./contracts.ts";
import { createBrowserRuntimeFactory } from "./runtime.ts";
const ROOT = dirname(fileURLToPath(import.meta.url));
const CURRENT_WORKER = join(ROOT, "browser_worker.py");
const PROBE = join(ROOT, "performance_probe.py");
const FIXTURE = join(ROOT, "fixtures", "performance.html");
const DEFAULT_PYTHON = join(process.cwd(), ".pi", "fast-browser-demo", "venv", "bin", "python");
const DEFAULT_BROWSER = "/usr/bin/chromium";
const DEFAULT_COUNTS = [0, 10, 60] as const;
const DEFAULT_SAMPLES = 20;
const DEFAULT_COLD_SAMPLES = 1;
const MAX_SAMPLES = 40;
const MAX_COLD_SAMPLES = 5;
const MAX_COUNTS = 3;
type Arm = "baseline" | "optimized";
type Phase = "warm" | "cold-start";
export type TimedOperation = "observe" | "act" | "read" | "assert" | "navigate" | "reload" | "listTabs" | "selectTab" | "waitPopup" | "closeTab" | "captureEvidence" | "narrowOriginPolicy" | "close";
export type OperationTimings = Partial<Record<TimedOperation, number>>;
export interface PerformanceBenchmarkOptions {
baselineWorkerPath: string;
optimizedWorkerPath?: string;
pythonPath?: string;
browserPath?: string;
samples?: number;
coldSamples?: number;
candidateCounts?: readonly number[];
}
export interface TimingSummary {
count: number;
p50: number | null;
p95: number | null;
}
export interface BenchmarkAttempt {
arm: Arm;
phase: Phase;
sample: number;
order?: 1 | 2;
status: "passed" | "failed";
elapsedMs: number | null;
startupMs: number | null;
error?: string;
protocolOperations?: Record<string, number>;
densityObserveMs?: Partial<Record<"0" | "10" | "60", number>>;
transportMs?: OperationTimings;
}
export interface WorkerMethodCounts {
cdp: Record<string, number>;
ipc: Record<string, number>;
cdpCalls: number;
ipcCalls: number;
}
export interface BenchmarkReport {
ok: boolean;
measurement: { workerMethods: "selected Python worker CDP/IPC helper methods"; protocolOperations: "TypeScript JSONL operation names"; warmPriming: "one density=0 observe per arm, excluded from warm samples"; coldStartup: "browser session open plus first worker observe"; runtimeNarrowingMeasured: false };
samples: number;
coldSamples: number;
candidateCounts: readonly number[];
attempts: readonly BenchmarkAttempt[];
warm: Record<Arm, TimingSummary>;
coldStartup: Record<Arm, TimingSummary>;
workerMethods: Partial<Record<Arm, WorkerMethodCounts>>;
protocolOperations: Partial<Record<Arm, Record<string, number>>>;
densityObserve: Record<Arm, Record<"0" | "10" | "60", TimingSummary>>;
pairedOrder: readonly (readonly [Arm, Arm])[];
counterStatus: "complete" | "unavailable";
cleanup: { status: "passed" | "failed"; error?: string; retainedRoot: boolean };
}
interface FixtureServer {
origin: string;
close(): Promise<void>;
}
interface ArmHandle {
arm: Arm;
session: BrowserSession;
runtime: BrowserRuntime;
protocol: Record<string, number>;
transportMs: OperationTimings;
countFile: string;
closed: boolean;
}
const EMPTY_ARTIFACT_STORE: PrivateArtifactStore = {
async open() {
throw new Error("benchmark evidence is disabled");
},
async close() { /* no-op */ },
};
type FailureStage = "cli" | "cold-start" | "warm-setup" | "warm-flow" | "density-observe" | "popup-close" | "counter" | "cleanup";
type FailureCategory = "failed" | "unverified" | "unavailable";
export function safeFailure(stage: FailureStage, category: FailureCategory = "failed"): string {
return `${stage}:${category}`;
}
class BenchmarkFailure extends Error {
readonly stage: FailureStage;
readonly category: FailureCategory;
constructor(stage: FailureStage, category: FailureCategory) {
super(`${stage}:${category}`);
this.stage = stage;
this.category = category;
}
}
function failureFor(error: unknown, fallback: FailureStage): string {
return error instanceof BenchmarkFailure ? safeFailure(error.stage, error.category) : safeFailure(fallback);
}
function boundedInteger(value: unknown, label: string, min: number, max: number): number {
if (!Number.isSafeInteger(value) || (value as number) < min || (value as number) > max) {
throw new Error(`${label} must be an integer between ${min} and ${max}`);
}
return value as number;
}
function requiredFile(value: string, label: string): string {
if (typeof value !== "string" || value.length === 0 || value.length > 4_000 || value.includes("\0")) {
throw new Error(`${label} is invalid`);
}
if (!existsSync(value) || !statSync(value).isFile()) throw new Error(`${label} is not a file`);
return value;
}
export function validateBenchmarkOptions(options: PerformanceBenchmarkOptions): Required<PerformanceBenchmarkOptions> {
if (!options || typeof options !== "object") throw new Error("benchmark options are required");
const baselineWorkerPath = requiredFile(options.baselineWorkerPath, "baseline worker path");
const optimizedWorkerPath = requiredFile(options.optimizedWorkerPath ?? CURRENT_WORKER, "optimized worker path");
const pythonPath = requiredFile(options.pythonPath ?? process.env.FAST_BROWSER_PYTHON ?? DEFAULT_PYTHON, "Python path");
const browserPath = requiredFile(options.browserPath ?? DEFAULT_BROWSER, "browser path");
const samples = boundedInteger(options.samples ?? DEFAULT_SAMPLES, "samples", 1, MAX_SAMPLES);
const coldSamples = boundedInteger(options.coldSamples ?? DEFAULT_COLD_SAMPLES, "cold samples", 1, MAX_COLD_SAMPLES);
const candidateCounts = [...(options.candidateCounts ?? DEFAULT_COUNTS)];
if (candidateCounts.length < 1 || candidateCounts.length > MAX_COUNTS || candidateCounts.some((value) => ![0, 10, 60].includes(value))) {
throw new Error("candidate counts must be a non-empty subset of 0, 10, and 60");
}
return { baselineWorkerPath, optimizedWorkerPath, pythonPath, browserPath, samples, coldSamples, candidateCounts };
}
function percentile(values: readonly number[], fraction: number): number | null {
if (values.length === 0) return null;
const sorted = [...values].sort((a, b) => a - b);
const position = (sorted.length - 1) * fraction;
const lower = Math.floor(position);
const upper = Math.ceil(position);
if (lower === upper) return sorted[lower]!;
return sorted[lower]! + (sorted[upper]! - sorted[lower]!) * (position - lower);
}
export function summarizeTimings(values: readonly number[]): TimingSummary {
return { count: values.length, p50: percentile(values, 0.5), p95: percentile(values, 0.95) };
}
function increment(target: Record<string, number>, key: string): void {
target[key] = (target[key] ?? 0) + 1;
}
function countProtocol(child: ReturnType<typeof nodeSpawn>, target: Record<string, number>): void {
const stdin = child.stdin;
if (!stdin) throw new Error("benchmark worker stdin is unavailable");
const write = stdin.write.bind(stdin);
stdin.write = ((data: string | Uint8Array, ...args: unknown[]) => {
const text = typeof data === "string" ? data : Buffer.from(data).toString("utf8");
for (const line of text.split("\n")) {
if (!line) continue;
try {
const wire = JSON.parse(line) as { operation?: { op?: unknown } };
const op = wire.operation?.op;
if (typeof op === "string" && /^[A-Za-z][A-Za-z0-9]{0,40}$/.test(op)) increment(target, op);
} catch { /* bridge owns malformed-request handling */ }
}
return write(data, ...(args as []));
}) as typeof stdin.write;
}
function instrumentedSpawn(
workerPath: string,
countFile: string,
protocol: Record<string, number>,
): SpawnLike {
return (command, args, options) => {
const child = nodeSpawn(command, [
...args,
"--performance-worker", workerPath,
"--performance-count-file", countFile,
], options as never);
countProtocol(child, protocol);
return child as unknown as ChildProcessLike;
};
}
function timedTransport(base: BrowserTransport, timings: OperationTimings): BrowserTransport {
return {
async dispatch<Operation extends BrowserOperation>(request: BrowserTransportRequest<Operation>, signal: AbortSignal): Promise<BrowserTransportResultFor<Operation>> {
const operation = request.operation.op as TimedOperation;
const started = performance.now();
try {
return await base.dispatch(request, signal);
} finally {
timings[operation] = (timings[operation] ?? 0) + performance.now() - started;
}
},
close: () => base.close(),
};
}
function createInstrumentedRuntime(
armWorkerPath: string,
countFile: string,
protocol: Record<string, number>,
transportMs: OperationTimings,
session: BrowserSession,
origin: string,
pythonPath: string,
signal: AbortSignal,
): BrowserRuntime {
const transportFactory: BrowserTransportFactory = {
create(options, parentSignal) {
const transport = createBrowserTransport({
...options,
workerPath: PROBE,
spawn: instrumentedSpawn(armWorkerPath, countFile, protocol),
}, parentSignal);
return timedTransport(transport, transportMs);
},
};
return createBrowserRuntimeFactory(transportFactory).create({
sessionId: session.id,
cdpUrl: session.config.cdpUrl,
allowedOrigins: [origin],
python: pythonPath,
artifacts: EMPTY_ARTIFACT_STORE,
}, signal);
}
async function createFixtureServer(): Promise<FixtureServer> {
const template = readFileSync(FIXTURE, "utf8");
const server = createServer((request: IncomingMessage, response: ServerResponse) => {
const path = new URL(request.url ?? "/", "http://127.0.0.1").pathname;
if (path !== "/performance.html") {
response.writeHead(404);
response.end();
return;
}
response.writeHead(200, { "content-type": "text/html; charset=utf-8", "cache-control": "no-store" });
response.end(template);
});
await new Promise<void>((resolve, reject) => {
server.once("error", reject);
server.listen(0, "127.0.0.1", resolve);
});
const address = server.address();
if (!address || typeof address === "string") throw new Error("performance fixture did not expose a port");
return {
origin: `http://127.0.0.1:${address.port}`,
async close() {
server.closeAllConnections?.();
await new Promise<void>((resolve) => server.close(() => resolve()));
},
};
}
function launchConfig(origin: string, url: string, browserPath: string): Parameters<BrowserSessionManager["open"]>[0] {
return {
allowedOrigins: [origin],
browser: { mode: "launch", product: "chromium", executablePath: browserPath, headless: true, startUrl: url },
};
}
function candidate(observation: Awaited<ReturnType<BrowserRuntime["observe"]>>, name: string, action: "click" | "fill"): Candidate {
const matches = observation.candidates.filter((item) => item.name === name && item.actions.includes(action));
assert.equal(matches.length, 1, `expected one ${action} candidate`);
return matches[0]!;
}
async function runWarmFlow(handle: ArmHandle, origin: string, candidateCounts: readonly number[], signal: AbortSignal): Promise<Partial<Record<"0" | "10" | "60", number>>> {
const densityObserveMs: Partial<Record<"0" | "10" | "60", number>> = {};
for (const count of candidateCounts) {
const density = await handle.runtime.navigate({ url: `${origin}/performance.html?mode=density&count=${count}` }, signal);
assert.equal(density.status, "committed");
const observeStarted = Date.now();
try {
const observation = await handle.runtime.observe(signal);
densityObserveMs[String(count) as "0" | "10" | "60"] = Date.now() - observeStarted;
assert.equal(observation.candidates.length, count);
} catch {
throw new BenchmarkFailure("density-observe", "failed");
}
}
const page = await handle.runtime.navigate({ url: `${origin}/performance.html?mode=flow&count=10` }, signal);
assert.equal(page.status, "committed");
const tabs = await handle.runtime.listTabs(signal);
const main = tabs.find((tab) => tab.selected);
assert(main, "main benchmark tab is unavailable");
const mainTab = main;
let observation = await handle.runtime.observe(signal);
const field = candidate(observation, "Display name", "fill");
const filled = await handle.runtime.act({ observationId: observation.observationId, candidateId: field.id, action: "fill", text: "Ada Lovelace" }, signal);
assert.equal(filled.verified, true);
observation = await handle.runtime.observe(signal);
const save = candidate(observation, "Save profile", "click");
const saved = await handle.runtime.act({ observationId: observation.observationId, candidateId: save.id, action: "click", postcondition: { kind: "text", value: "Profile saved" } }, signal);
assert.equal(saved.verified, true);
const saveAssert = await handle.runtime.assert({ assertion: { kind: "visibleText", match: "contains", value: "Profile saved" }, wait: "bounded" }, signal);
assert.equal(saveAssert.status, "satisfied");
observation = await handle.runtime.observe(signal);
const previewButton = candidate(observation, "Preview profile", "click");
const previewRequested = await handle.runtime.act({ observationId: observation.observationId, candidateId: previewButton.id, action: "click", postcondition: { kind: "text", value: "Preview ready" } }, signal);
assert.equal(previewRequested.verified, true);
observation = await handle.runtime.observe(signal);
const immediate = candidate(observation, "Immediate action", "click");
const clicked = await handle.runtime.act({ observationId: observation.observationId, candidateId: immediate.id, action: "click", postcondition: { kind: "text", value: "Immediate complete" } }, signal);
assert.equal(clicked.verified, true);
const immediateAssert = await handle.runtime.assert({ assertion: { kind: "visibleText", match: "contains", value: "Immediate complete" }, wait: "none" }, signal);
assert.equal(immediateAssert.status, "satisfied");
observation = await handle.runtime.observe(signal);
const delayed = candidate(observation, "Delayed action", "click");
const delayedResult = await handle.runtime.act({ observationId: observation.observationId, candidateId: delayed.id, action: "click", postcondition: { kind: "text", value: "Delayed complete" } }, signal);
assert.equal(delayedResult.verified, true);
const delayedAssert = await handle.runtime.assert({ assertion: { kind: "visibleText", match: "contains", value: "Delayed complete" }, wait: "bounded" }, signal);
assert.equal(delayedAssert.status, "satisfied");
observation = await handle.runtime.observe(signal);
const popupButton = candidate(observation, "Open popup", "click");
const popupRequest = await handle.runtime.act({ observationId: observation.observationId, candidateId: popupButton.id, action: "click", postcondition: { kind: "text", value: "Popup requested" } }, signal);
assert.equal(popupRequest.verified, true);
const popup = await handle.runtime.waitPopup({ openerTabId: mainTab.tabId }, signal);
assert.equal(popup.status, "found");
assert(popup.tab);
const popupTab = popup.tab;
const popupRead = await handle.runtime.read({ tabId: popupTab.tabId }, signal);
assert.match(popupRead.visibleText, /Popup ready/);
const popupReady = await handle.runtime.assert({ tabId: popupTab.tabId, assertion: { kind: "visibleText", match: "contains", value: "Popup ready" }, wait: "bounded" }, signal);
assert.equal(popupReady.status, "satisfied");
try {
await handle.runtime.closeTab({ tabId: popupTab.tabId }, signal);
} catch {
throw new BenchmarkFailure("popup-close", "unverified");
}
const preview = await handle.runtime.assert({ assertion: { kind: "visibleText", match: "contains", value: "Preview ready: Ada Lovelace" }, wait: "bounded" }, signal);
assert.equal(preview.status, "satisfied");
return densityObserveMs;
}
function validCounterMap(value: unknown): value is Record<string, number> {
return !!value && typeof value === "object" && !Array.isArray(value)
&& Object.entries(value).every(([key, count]) => /^[A-Za-z0-9_.-]{1,120}$/.test(key) && Number.isSafeInteger(count) && count >= 0);
}
export function validWorkerCounters(value: unknown): value is { cdp: Record<string, number>; ipc: Record<string, number> } {
if (!value || typeof value !== "object") return false;
const counters = value as { cdp?: unknown; ipc?: unknown };
return validCounterMap(counters.cdp) && validCounterMap(counters.ipc);
}
async function readWorkerCounts(path: string): Promise<WorkerMethodCounts | undefined> {
try {
const value = JSON.parse(await readFile(path, "utf8")) as { cdp?: unknown; ipc?: unknown };
if (!validWorkerCounters(value)) return undefined;
const cdp = value.cdp;
const ipc = value.ipc;
return {
cdp,
ipc,
cdpCalls: Object.values(cdp).reduce((sum, count) => sum + count, 0),
ipcCalls: Object.values(ipc).reduce((sum, count) => sum + count, 0),
};
} catch {
return undefined;
}
}
function ownedProcessLines(root: string): string[] | undefined {
const result = spawnSync("ps", ["-eo", "pid=,args="], { encoding: "utf8", timeout: 5_000 });
if (result.status !== 0) return undefined;
return String(result.stdout).split(/\r?\n/).filter((line) => line.includes(root));
}
function snapshotCounts(value: Record<string, number>): Record<string, number> {
return { ...value };
}
export function timingDelta(before: OperationTimings, after: OperationTimings): OperationTimings {
const result: OperationTimings = {};
for (const key of new Set([...Object.keys(before), ...Object.keys(after)] as TimedOperation[])) {
const difference = (after[key] ?? 0) - (before[key] ?? 0);
if (difference > 0) result[key] = difference;
}
return result;
}
function countDelta(before: Record<string, number>, after: Record<string, number>): Record<string, number> {
const result: Record<string, number> = {};
for (const key of new Set([...Object.keys(before), ...Object.keys(after)])) {
const difference = (after[key] ?? 0) - (before[key] ?? 0);
if (difference > 0) result[key] = difference;
}
return result;
}
async function closeHandle(handle: ArmHandle | undefined): Promise<string | undefined> {
if (!handle || handle.closed) return undefined;
handle.closed = true;
try {
await handle.runtime.close();
return undefined;
} catch {
return safeFailure("cleanup");
}
}
export function pairedArmOrder(sample: number): readonly [Arm, Arm] {
return sample % 2 === 1 ? ["baseline", "optimized"] : ["optimized", "baseline"];
}
function attemptsFor(attempts: readonly BenchmarkAttempt[], arm: Arm, phase: Phase): number[] {
return attempts.filter((item) => item.arm === arm && item.phase === phase && item.status === "passed" && item.elapsedMs !== null).map((item) => item.elapsedMs!);
}
export function summarizeBenchmarkAttempts(attempts: readonly BenchmarkAttempt[], arm: Arm, phase: Phase): TimingSummary {
return summarizeTimings(attemptsFor(attempts, arm, phase));
}
function densitySummaries(attempts: readonly BenchmarkAttempt[]): Record<Arm, Record<"0" | "10" | "60", TimingSummary>> {
const result = {} as Record<Arm, Record<"0" | "10" | "60", TimingSummary>>;
for (const arm of ["baseline", "optimized"] as const) {
result[arm] = {} as Record<"0" | "10" | "60", TimingSummary>;
for (const count of ["0", "10", "60"] as const) {
result[arm][count] = summarizeTimings(attempts
.filter((attempt) => attempt.arm === arm && attempt.phase === "warm" && attempt.status === "passed" && attempt.densityObserveMs?.[count] !== undefined)
.map((attempt) => attempt.densityObserveMs![count]!));
}
}
return result;
}
export async function runPerformanceBenchmark(options: PerformanceBenchmarkOptions): Promise<BenchmarkReport> {
const config = validateBenchmarkOptions(options);
const fixture = await createFixtureServer();
const root = await mkdtemp(join(tmpdir(), "fast-browser-performance-"));
await mkdir(join(root, "leases"), { recursive: true, mode: 0o700 });
await chmod(join(root, "leases"), 0o700);
const manager = createBrowserSessionManager({ profileDirectory: join(root, "profiles"), leaseDirectory: join(root, "leases"), startupTimeoutMs: 20_000, closeTimeoutMs: 4_000 });
const controller = new AbortController();
const attempts: BenchmarkAttempt[] = [];
const workerMethods: Partial<Record<Arm, WorkerMethodCounts>> = {};
const protocolOperations: Partial<Record<Arm, Record<string, number>>> = {};
const handles: Partial<Record<Arm, ArmHandle>> = {};
const setupErrors: Partial<Record<Arm, string>> = {};
const pairedOrder: Array<readonly [Arm, Arm]> = [];
let cleanupError: string | undefined;
let retainedRoot = false;
let counterUnavailable = false;
try {
const armPaths: Record<Arm, string> = { baseline: config.baselineWorkerPath, optimized: config.optimizedWorkerPath };
for (const arm of ["baseline", "optimized"] as const) {
for (let sample = 1; sample <= config.coldSamples; sample += 1) {
const countFile = join(root, `${arm}-cold-${sample}.json`);
const protocol: Record<string, number> = {};
const transportMs: OperationTimings = {};
let session: BrowserSession | undefined;
let runtime: BrowserRuntime | undefined;
const started = Date.now();
let startupMs: number | null = null;
try {
const opened = Date.now();
session = await manager.open(launchConfig(fixture.origin, `${fixture.origin}/performance.html?mode=density&count=10`, config.browserPath), controller.signal);
const first = Date.now();
runtime = createInstrumentedRuntime(armPaths[arm], countFile, protocol, transportMs, session, fixture.origin, config.pythonPath, controller.signal);
const observation = await runtime.observe(controller.signal);
assert.equal(observation.candidates.length, 10);
startupMs = first - opened + (Date.now() - first);
attempts.push({ arm, phase: "cold-start", sample, status: "passed", elapsedMs: Date.now() - started, startupMs, protocolOperations: { ...protocol }, transportMs: { ...transportMs } });
} catch (error) {
attempts.push({ arm, phase: "cold-start", sample, status: "failed", elapsedMs: Date.now() - started, startupMs, error: safeFailure("cold-start"), protocolOperations: { ...protocol }, transportMs: { ...transportMs } });
} finally {
try { await runtime?.close(); } catch { cleanupError ??= safeFailure("cleanup"); }
if (session) await manager.close(session.id).catch(() => { cleanupError ??= safeFailure("cleanup"); });
const counts = await readWorkerCounts(countFile);
if (counts) workerMethods[arm] = counts;
else counterUnavailable = true;
}
}
try {
const countFile = join(root, `${arm}-warm.json`);
const protocol: Record<string, number> = {};
const transportMs: OperationTimings = {};
const session = await manager.open(launchConfig(fixture.origin, `${fixture.origin}/performance.html?mode=density&count=0`, config.browserPath), controller.signal);
const runtime = createInstrumentedRuntime(armPaths[arm], countFile, protocol, transportMs, session, fixture.origin, config.pythonPath, controller.signal);
const handle: ArmHandle = { arm, session, runtime, protocol, transportMs, countFile, closed: false };
handles[arm] = handle;
await runtime.observe(controller.signal); // prime the persistent worker; excluded from warm samples
} catch (error) {
const failedHandle = handles[arm];
if (failedHandle) {
await closeHandle(failedHandle);
await manager.close(failedHandle.session.id).catch(() => {});
}
delete handles[arm];
setupErrors[arm] = safeFailure("warm-setup");
}
}
for (let sample = 1; sample <= config.samples; sample += 1) {
const order = pairedArmOrder(sample);
pairedOrder.push(order);
for (let position = 0; position < order.length; position += 1) {
const arm = order[position]!;
const handle = handles[arm];
const before = handle ? snapshotCounts(handle.protocol) : {};
const beforeTransport = handle ? { ...handle.transportMs } : {};
const started = Date.now();
try {
if (!handle) throw new Error("arm setup failed");
const densityObserveMs = await runWarmFlow(handle, fixture.origin, config.candidateCounts, controller.signal);
attempts.push({ arm, phase: "warm", sample, order: (position + 1) as 1 | 2, status: "passed", elapsedMs: Date.now() - started, startupMs: null, protocolOperations: countDelta(before, handle.protocol), transportMs: timingDelta(beforeTransport, handle.transportMs), densityObserveMs });
} catch (error) {
attempts.push({ arm, phase: "warm", sample, order: (position + 1) as 1 | 2, status: "failed", elapsedMs: Date.now() - started, startupMs: null, error: handle ? failureFor(error, "warm-flow") : (setupErrors[arm] ?? safeFailure("warm-setup")), protocolOperations: handle ? countDelta(before, handle.protocol) : {}, transportMs: handle ? timingDelta(beforeTransport, handle.transportMs) : {} });
}
}
}
} finally {
for (const arm of ["baseline", "optimized"] as const) {
const handle = handles[arm];
const error = await closeHandle(handle);
if (error) cleanupError ??= error;
if (handle) {
const counts = await readWorkerCounts(handle.countFile);
if (counts) workerMethods[arm] = counts;
else counterUnavailable = true;
protocolOperations[arm] = { ...handle.protocol };
}
}
await manager.closeAll().catch(() => { cleanupError ??= safeFailure("cleanup"); });
await fixture.close().catch(() => { cleanupError ??= safeFailure("cleanup"); });
const leftovers = ownedProcessLines(root);
if (leftovers === undefined || leftovers.length > 0) {
cleanupError ??= safeFailure("cleanup", "unavailable");
retainedRoot = true;
}
if (!cleanupError) {
await rm(root, { recursive: true, force: true }).catch(() => {
cleanupError ??= safeFailure("cleanup");
retainedRoot = true;
});
} else {
retainedRoot = true;
}
}
const warm: Record<Arm, TimingSummary> = {
baseline: summarizeBenchmarkAttempts(attempts, "baseline", "warm"),
optimized: summarizeBenchmarkAttempts(attempts, "optimized", "warm"),
};
const coldStartup: Record<Arm, TimingSummary> = {
baseline: summarizeTimings(attempts.filter((item) => item.arm === "baseline" && item.phase === "cold-start" && item.status === "passed" && item.startupMs !== null).map((item) => item.startupMs!)),
optimized: summarizeTimings(attempts.filter((item) => item.arm === "optimized" && item.phase === "cold-start" && item.status === "passed" && item.startupMs !== null).map((item) => item.startupMs!)),
};
return {
ok: !cleanupError && !counterUnavailable && workerMethods.baseline !== undefined && workerMethods.optimized !== undefined && attempts.length === config.samples * 2 + config.coldSamples * 2 && attempts.every((item) => item.status === "passed"),
measurement: { workerMethods: "selected Python worker CDP/IPC helper methods", protocolOperations: "TypeScript JSONL operation names", warmPriming: "one density=0 observe per arm, excluded from warm samples", coldStartup: "browser session open plus first worker observe", runtimeNarrowingMeasured: false },
samples: config.samples,
coldSamples: config.coldSamples,
candidateCounts: config.candidateCounts,
attempts,
warm,
coldStartup,
workerMethods,
protocolOperations,
densityObserve: densitySummaries(attempts),
pairedOrder,
counterStatus: !counterUnavailable && workerMethods.baseline && workerMethods.optimized ? "complete" : "unavailable",
cleanup: cleanupError ? { status: "failed", error: cleanupError, retainedRoot } : { status: "passed", retainedRoot: false },
};
}
function parseArgs(argv: readonly string[]): PerformanceBenchmarkOptions & { help?: boolean } {
const result: Record<string, unknown> = {};
for (let index = 0; index < argv.length; index += 1) {
const arg = argv[index]!;
if (arg === "--help" || arg === "-h") return { baselineWorkerPath: "", help: true };
const values: Record<string, string> = { "--baseline-worker": "baselineWorkerPath", "--optimized-worker": "optimizedWorkerPath", "--python": "pythonPath", "--browser": "browserPath", "--samples": "samples", "--cold-samples": "coldSamples", "--counts": "candidateCounts" };
const key = values[arg];
if (!key || index + 1 >= argv.length) throw new Error(`unknown or incomplete option: ${arg}`);
const value = argv[++index]!;
result[key] = key === "samples" || key === "coldSamples" ? Number(value) : key === "candidateCounts" ? value.split(",").map(Number) : value;
}
if (typeof result.baselineWorkerPath !== "string") throw new Error("--baseline-worker is required");
return result as unknown as PerformanceBenchmarkOptions;
}
function printReport(report: BenchmarkReport): void {
console.log(JSON.stringify(report, null, 2));
}
if (import.meta.main) {
try {
const parsed = parseArgs(process.argv.slice(2));
if (parsed.help) {
console.log("usage: node --experimental-strip-types performance-benchmark.ts --baseline-worker PATH [--optimized-worker PATH] [--samples N] [--cold-samples N] [--counts 0,10,60]");
} else {
const report = await runPerformanceBenchmark(parsed);
printReport(report);
if (!report.ok) process.exitCode = 1;
}
} catch (error) {
console.error(safeFailure("cli"));
process.exitCode = 1;
}
}
@@ -0,0 +1,119 @@
"""Private benchmark wrapper for counting browser-harness calls.
The wrapper is deliberately a transport-only probe. It records method names and
counts, never CDP arguments, URLs, DOM values, page text, or screenshots. The
selected worker is executed unchanged after the helper module is wrapped.
"""
from __future__ import annotations
import importlib
import json
import os
from pathlib import Path
import runpy
import sys
import tempfile
from collections import Counter
from typing import Any
def _take_flag(argv: list[str], flag: str) -> tuple[str | None, list[str]]:
remaining: list[str] = []
value: str | None = None
index = 0
while index < len(argv):
item = argv[index]
if item == flag:
if value is not None or index + 1 >= len(argv):
raise ValueError(f"{flag} must be supplied once")
value = argv[index + 1]
index += 2
continue
remaining.append(item)
index += 1
return value, remaining
def _method_name(value: Any) -> str:
if isinstance(value, str) and value and len(value) <= 120 and all(
character.isalnum() or character in "._-" for character in value
):
return value
return "unknown"
def _wrap_helpers(counts: dict[str, Counter[str]] | None = None) -> dict[str, Counter[str]]:
helpers = importlib.import_module("browser_harness.helpers")
counts = counts or {"cdp": Counter(), "ipc": Counter()}
original_cdp = getattr(helpers, "cdp", None)
if callable(original_cdp):
def counted_cdp(*args: Any, **kwargs: Any) -> Any:
method = _method_name(args[0] if args else None)
counts["cdp"][method] += 1
return original_cdp(*args, **kwargs)
helpers.cdp = counted_cdp
original_send = getattr(helpers, "_send", None)
if callable(original_send):
def counted_send(*args: Any, **kwargs: Any) -> Any:
# Do not inspect the request. The IPC method is intentionally the
# fixed helper method name rather than an untrusted operation value.
counts["ipc"]["_send"] += 1
return original_send(*args, **kwargs)
helpers._send = counted_send
return counts
def _write_counts(path: str, counts: dict[str, Counter[str]]) -> None:
payload = {
"cdp": dict(sorted(counts["cdp"].items())),
"ipc": dict(sorted(counts["ipc"].items())),
}
destination = Path(path)
destination.parent.mkdir(parents=True, exist_ok=True)
fd, temporary = tempfile.mkstemp(prefix="performance-count-", dir=str(destination.parent))
try:
with os.fdopen(fd, "w", encoding="utf-8") as stream:
json.dump(payload, stream, separators=(",", ":"), sort_keys=True)
stream.write("\n")
os.chmod(temporary, 0o600)
os.replace(temporary, destination)
os.chmod(destination, 0o600)
finally:
try:
os.unlink(temporary)
except FileNotFoundError:
pass
def main(argv: list[str] | None = None) -> int:
raw = list(sys.argv[1:] if argv is None else argv)
worker_path, raw = _take_flag(raw, "--performance-worker")
count_path, worker_argv = _take_flag(raw, "--performance-count-file")
if not worker_path or not count_path:
raise SystemExit("performance_probe requires --performance-worker and --performance-count-file")
if not Path(worker_path).is_file():
raise SystemExit("performance worker path is not a file")
counts: dict[str, Counter[str]] = {"cdp": Counter(), "ipc": Counter()}
try:
sys.argv = [worker_path, *worker_argv]
namespace = runpy.run_path(worker_path, run_name="performance_worker")
backend = namespace.get("RealBrowserBackend")
main_function = namespace.get("main")
if backend is None or not callable(main_function):
raise SystemExit("performance worker does not expose its expected entry points")
original_load = backend._load_harness
def load_and_probe(instance: Any) -> Any:
result = original_load(instance)
_wrap_helpers(counts)
return result
backend._load_harness = load_and_probe
main_function(worker_argv)
finally:
_write_counts(count_path, counts)
return 0
if __name__ == "__main__":
raise SystemExit(main())
+99 -12
View File
@@ -1,6 +1,7 @@
import assert from "node:assert/strict";
import test from "node:test";
import { createBrowserRuntimeFactory, withRuntimeOriginPolicy } from "./runtime.ts";
import { createBrowserRuntimeFactory } from "./runtime.ts";
import { BrowserOperationError } from "./contracts.ts";
import type {
BrowserOperation,
BrowserRuntimeOptions,
@@ -9,7 +10,6 @@ import type {
BrowserTransportResultFor,
PrivateArtifactMetadata,
PrivateArtifactStore,
RuntimeOperationContext,
} from "./contracts.ts";
const tab = (id: string, url = "https://example.test/form") => ({
@@ -153,18 +153,105 @@ test("factory forwards pinned Python selection to the transport", async () => {
await runtime.close();
});
test("wildcard narrowing is scoped while restricted bases cannot broaden", async () => {
const first = artifacts();
test("narrowing is local, lazy, and propagates only its effective context", async () => {
const store = artifacts();
const fixture = transportFixture();
const runtime = createBrowserRuntimeFactory(options(first.store, fixture.transport).transportFactory).create(options(first.store, fixture.transport));
await withRuntimeOriginPolicy(runtime, ["https://example.test"], (context) => runtime.read({}, new AbortController().signal, context));
const reads = fixture.calls.filter((call) => call.operation.op === "read");
assert.deepEqual((reads[0]?.context as RuntimeOperationContext).allowedOrigins, ["https://example.test"]);
let created = 0;
const runtime = createBrowserRuntimeFactory({ create() { created++; return fixture.transport; } }).create(options(store.store, fixture.transport));
const context = await runtime.narrowOriginPolicy({ allowedOrigins: ["https://example.test"] }, new AbortController().signal);
assert.deepEqual(context, { allowedOrigins: ["https://example.test"] });
assert.equal(created, 0);
assert.equal(fixture.calls.length, 0);
await runtime.read({}, new AbortController().signal, context);
await runtime.read({}, new AbortController().signal);
assert.equal(fixture.calls.at(-1)?.context, undefined);
const restricted = createBrowserRuntimeFactory({ create() { return fixture.transport; } }).create({ ...options(first.store, fixture.transport), allowedOrigins: ["https://example.test"] });
await assert.rejects(restricted.narrowOriginPolicy({ allowedOrigins: "*" }, new AbortController().signal), /broadened/);
await runtime.close(); await restricted.close();
assert.equal(created, 1);
assert.deepEqual(fixture.calls[0]?.context, { allowedOrigins: ["https://example.test"] });
assert.equal(fixture.calls[1]?.context, undefined);
assert.equal(fixture.calls.some((call) => call.operation.op === "narrowOriginPolicy"), false);
await runtime.close();
});
test("narrowing validates invalid and broadened policies without dispatch", async () => {
const store = artifacts();
const fixture = transportFixture();
let created = 0;
const runtime = createBrowserRuntimeFactory({ create() { created++; return fixture.transport; } }).create({
...options(store.store, fixture.transport), allowedOrigins: ["https://example.test"],
});
await assert.rejects(runtime.narrowOriginPolicy({ allowedOrigins: [] }, new AbortController().signal), /must not be empty/);
await assert.rejects(runtime.narrowOriginPolicy({ allowedOrigins: "*" }, new AbortController().signal), (error: unknown) => {
assert.ok(error instanceof BrowserOperationError);
assert.equal(error.details.code, "policy_denied");
assert.equal(error.details.mutationState, "not_attempted");
return true;
});
assert.equal(created, 0);
assert.equal(fixture.calls.length, 0);
await runtime.close();
});
test("narrowing preserves closed and aborted lifecycle errors", async () => {
const store = artifacts();
const fixture = transportFixture();
const runtime = createBrowserRuntimeFactory({ create() { return fixture.transport; } }).create(options(store.store, fixture.transport));
await runtime.close();
await assert.rejects(runtime.narrowOriginPolicy({ allowedOrigins: ["https://example.test"] }, new AbortController().signal), (error: unknown) => {
assert.ok(error instanceof BrowserOperationError);
assert.equal(error.details.code, "browser_closed");
return true;
});
const parent = new AbortController();
parent.abort(new Error("browser session closed"));
const abortedRuntime = createBrowserRuntimeFactory({ create() { return fixture.transport; } }).create(options(store.store, fixture.transport), parent.signal);
await assert.rejects(abortedRuntime.narrowOriginPolicy({ allowedOrigins: ["https://example.test"] }, new AbortController().signal), (error: unknown) => {
assert.ok(error instanceof BrowserOperationError);
assert.equal(error.details.code, "browser_closed");
return true;
});
const operation = new AbortController();
operation.abort();
const liveRuntime = createBrowserRuntimeFactory({ create() { return fixture.transport; } }).create(options(store.store, fixture.transport));
await assert.rejects(liveRuntime.narrowOriginPolicy({ allowedOrigins: ["https://example.test"] }, operation.signal), (error: unknown) => {
assert.ok(error instanceof Error);
assert.equal(error.name, "AbortError");
return true;
});
await abortedRuntime.close();
await liveRuntime.close();
});
test("narrowing remains serialized behind an in-flight transport operation", async () => {
const store = artifacts();
const fixture = transportFixture();
let releaseRead!: () => void;
let readStarted!: () => void;
const readGate = new Promise<void>((resolve) => { releaseRead = resolve; });
const started = new Promise<void>((resolve) => { readStarted = resolve; });
const transport: BrowserTransport = {
async dispatch<Operation extends BrowserOperation>(request: BrowserTransportRequest<Operation>, signal: AbortSignal): Promise<BrowserTransportResultFor<Operation>> {
if (request.operation.op === "read") {
readStarted();
await readGate;
}
return fixture.transport.dispatch(request, signal);
},
async close() { return fixture.transport.close(); },
};
const runtime = createBrowserRuntimeFactory({ create() { return transport; } }).create(options(store.store, transport));
const read = runtime.read({}, new AbortController().signal);
await started;
let narrowed = false;
const narrow = runtime.narrowOriginPolicy({ allowedOrigins: ["https://example.test"] }, new AbortController().signal).then(() => { narrowed = true; });
await Promise.resolve();
assert.equal(narrowed, false);
releaseRead();
await Promise.all([read, narrow]);
assert.equal(narrowed, true);
await runtime.close();
});
test("policy is enforced before navigation and tabs are origin filtered", async () => {
+29 -5
View File
@@ -449,6 +449,30 @@ export function createBrowserRuntime(
return run;
};
// Policy narrowing is a runtime concern, not a worker operation. Keep it
// on the same lane so it observes the same lifecycle and serialization
// boundaries without starting a transport or emitting JSONL.
const invokeLocal = <T>(
operation: BrowserRuntimeOperationName,
signal: AbortSignal,
action: () => T,
): Promise<T> => {
const run = lane.then(() => {
if (closed && operation !== "close") throw operationError(operation, "lifecycle", "browser_closed", "browser session is closed", "not_attempted");
if (parentSignal.aborted && operation !== "close") throw operationError(operation, "lifecycle", "browser_closed", "browser session is unavailable", "not_attempted");
// Match invoke(): a pre-aborted operation signal is rejected before
// dispatch/error shaping, while lifecycle failures remain typed.
throwIfAborted(signal);
try {
return action();
} catch (error) {
throw asOperationError(error, operation, "dispatch", false);
}
});
lane = run.catch(() => undefined);
return run;
};
const captureAfterFailure = async (operation: BrowserRuntimeOperationName, context?: RuntimeOperationContext): Promise<FailureCaptureSummary> => {
if (parentSignal.aborted) return { status: "unavailable", reason: "browser session is unavailable" };
if (operation === "captureEvidence" || operation === "close" || operation === "narrowOriginPolicy") {
@@ -500,6 +524,10 @@ export function createBrowserRuntime(
const controller = new AbortController();
const timed = timedSignal(controller.signal, OPERATION_BUDGETS.close);
try {
// Wait for local-only operations too. A narrow policy must not race
// session teardown, and waiting here still keeps an unused transport
// unstarted.
await lane;
if (transport) {
await invoke("close", { op: "close" }, timed.signal, undefined, () => undefined).catch(() => undefined);
await transport.close().catch(() => {});
@@ -612,11 +640,7 @@ export function createBrowserRuntime(
const allowedOrigins = normalizePolicy(request.allowedOrigins);
if (basePolicy !== "*" && (allowedOrigins === "*" || allowedOrigins.some((origin) => !basePolicy.includes(origin)))) throw operationError("narrowOriginPolicy", "policy", "policy_denied", "origin policy cannot be broadened", "not_attempted");
const context = { allowedOrigins };
const result = await invoke("narrowOriginPolicy", { op: "narrowOriginPolicy", request: { allowedOrigins } }, signal, { allowedOrigins: basePolicy }, (raw) => {
if (!raw || typeof raw !== "object" || !("allowedOrigins" in raw)) throw new Error("worker returned invalid origin policy");
return context;
});
return result;
return invokeLocal("narrowOriginPolicy", signal, () => context);
},
close,
};
+431 -1
View File
@@ -8,7 +8,7 @@ from unittest.mock import Mock, patch
from browser_worker import (
BrowserWorker, BrowserRuntimeWorker, ElementMetadata, WorkerError, BrowserOperationError,
RealBrowserBackend, MAX_LINE_BYTES, MAX_READ_TEXT, process_request, run_stream,
RealBrowserBackend, ACTION_TIMEOUT_SECONDS, MAX_LINE_BYTES, MAX_READ_TEXT, process_request, run_stream,
validate_cdp_url, main,
)
@@ -103,6 +103,25 @@ class FakeBackend:
pass
class BatchFakeBackend(FakeBackend):
def __init__(self):
super().__init__()
self.metadata_many_calls = []
self.metadata_calls = []
self.batch_result = None
def metadata(self, backend_id):
self.metadata_calls.append(backend_id)
return super().metadata(backend_id)
def metadata_many(self, backend_ids):
ids = list(backend_ids)
self.metadata_many_calls.append(ids)
if self.batch_result is not None:
return self.batch_result
return {backend_id: self.elements[backend_id] for backend_id in ids if backend_id in self.elements}
class PopupFakeBackend(FakeBackend):
def __init__(self):
super().__init__()
@@ -171,6 +190,46 @@ class RuntimeFakeBackend(FakeBackend):
return b"\\x89PNG" + (b"x" * 32)
class FakeClock:
def __init__(self):
self.now = 0.0
self.sleeps = []
def monotonic(self):
return self.now
def sleep(self, duration):
self.sleeps.append(duration)
self.now += duration
class DelayedCloseBackend(RuntimeFakeBackend):
def __init__(self, disappear_after=None):
super().__init__()
self.tabs = [self.tabs[0]]
self.disappear_after = disappear_after
self.inventory_reads = 0
self.post_close_reads = 0
self.pending_close = None
self.close_dispatches = []
def list_tabs(self):
self.inventory_reads += 1
rows = [dict(tab) for tab in self.tabs]
if self.pending_close is not None:
self.post_close_reads += 1
if self.disappear_after is not None and self.post_close_reads >= self.disappear_after:
target_id = self.pending_close
self.tabs[:] = [tab for tab in self.tabs if tab["targetId"] != target_id]
self.pending_close = None
rows = [dict(tab) for tab in self.tabs]
return rows
def close_tab(self, target_id):
self.close_dispatches.append(target_id)
self.pending_close = target_id
class RuntimeTests(unittest.TestCase):
def make_runtime(self, backend=None, origins="*"):
backend = backend or RuntimeFakeBackend()
@@ -385,6 +444,102 @@ class RuntimeTests(unittest.TestCase):
with self.assertRaisesRegex(BrowserOperationError, "stale|unavailable"):
runtime.select_tab({"tabId": tab_id})
def test_close_waits_for_delayed_target_disappearance_without_redispatching(self):
backend = DelayedCloseBackend(disappear_after=2)
runtime = self.make_runtime(backend)
opener = runtime.list_tabs()[0]["tabId"]
backend.tabs.append({"targetId": "popup", "url": "http://fixture.test/popup",
"title": "Popup", "openerTargetId": "tab-a"})
popup = runtime.wait_popup({"openerTabId": opener})["tab"]
self.assertIsNotNone(popup)
popup_id = popup["tabId"]
clock = FakeClock()
with patch("browser_worker.time.monotonic", side_effect=clock.monotonic), \
patch("browser_worker.time.sleep", side_effect=clock.sleep):
result = runtime.close_tab({"tabId": popup_id})
self.assertEqual(result["mutationState"], "verified")
self.assertEqual(backend.close_dispatches, ["popup"])
self.assertGreaterEqual(backend.post_close_reads, 2)
self.assertGreater(clock.now, 0)
tabs = runtime.list_tabs()
self.assertEqual([tab["tabId"] for tab in tabs], [opener])
self.assertTrue(tabs[0]["selected"])
with self.assertRaisesRegex(BrowserOperationError, "stale|unavailable"):
runtime.select_tab({"tabId": popup_id})
def test_close_deadline_reports_unknown_without_retry_when_target_persists(self):
backend = DelayedCloseBackend(disappear_after=None)
runtime = self.make_runtime(backend)
opener = runtime.list_tabs()[0]["tabId"]
backend.tabs.append({"targetId": "popup", "url": "http://fixture.test/popup",
"title": "Popup", "openerTargetId": "tab-a"})
popup = runtime.wait_popup({"openerTabId": opener})["tab"]
self.assertIsNotNone(popup)
clock = FakeClock()
with patch("browser_worker.time.monotonic", side_effect=clock.monotonic), \
patch("browser_worker.time.sleep", side_effect=clock.sleep), \
self.assertRaises(BrowserOperationError) as caught:
runtime.close_tab({"tabId": popup["tabId"]})
details = caught.exception.details
self.assertEqual(details["code"], "worker_failure")
self.assertEqual(details["mutationState"], "unknown")
self.assertEqual(details["retry"], "never")
self.assertEqual(backend.close_dispatches, ["popup"])
self.assertGreaterEqual(backend.post_close_reads, 2)
self.assertGreaterEqual(clock.now, ACTION_TIMEOUT_SECONDS)
def test_close_inventory_failure_preserves_unknown_state_without_retry(self):
backend = DelayedCloseBackend(disappear_after=None)
runtime = self.make_runtime(backend)
opener = runtime.list_tabs()[0]["tabId"]
backend.tabs.append({"targetId": "popup", "url": "http://fixture.test/popup",
"title": "Popup", "openerTargetId": "tab-a"})
popup = runtime.wait_popup({"openerTabId": opener})["tab"]
self.assertIsNotNone(popup)
original_list_tabs = backend.list_tabs
def fail_after_close():
if backend.pending_close is not None:
raise RuntimeError("inventory unavailable")
return original_list_tabs()
backend.list_tabs = fail_after_close
with self.assertRaises(BrowserOperationError) as caught:
runtime.close_tab({"tabId": popup["tabId"]})
details = caught.exception.details
self.assertEqual(details["code"], "worker_failure")
self.assertEqual(details["mutationState"], "unknown")
self.assertEqual(details["retry"], "never")
self.assertEqual(backend.close_dispatches, ["popup"])
def test_close_cancellation_preserves_unknown_state_without_retry(self):
backend = DelayedCloseBackend(disappear_after=2)
runtime = self.make_runtime(backend)
opener = runtime.list_tabs()[0]["tabId"]
backend.tabs.append({"targetId": "popup", "url": "http://fixture.test/popup",
"title": "Popup", "openerTargetId": "tab-a"})
popup = runtime.wait_popup({"openerTabId": opener})["tab"]
self.assertIsNotNone(popup)
class Cancelled(BaseException):
pass
def cancel(_target_id):
backend.close_dispatches.append(_target_id)
raise Cancelled()
backend.close_tab = cancel
with self.assertRaises(BrowserOperationError) as caught:
runtime.close_tab({"tabId": popup["tabId"]})
details = caught.exception.details
self.assertEqual(details["code"], "aborted")
self.assertEqual(details["mutationState"], "unknown")
self.assertEqual(details["retry"], "never")
self.assertEqual(backend.close_dispatches, ["popup"])
def test_transport_wrapper_context_restores_base_and_uses_exact_result_shape(self):
backend = RuntimeFakeBackend()
runtime = self.make_runtime(backend)
@@ -550,6 +705,82 @@ class WorkerTests(unittest.TestCase):
self.assertNotIn("values", json.dumps(observation))
self.assertNotIn("Disabled", json.dumps(observation))
def test_observe_batches_dom_metadata_but_keeps_full_ax_roles_and_names(self):
backend = BatchFakeBackend()
backend.elements[1] = replace(backend.elements[1], name="DOM Email", ax_role="textbox", ax_name="AX Email")
backend.elements[3] = replace(backend.elements[3], name="DOM Submit", ax_role="button", ax_name="AX Submit")
backend.nodes[0]["name"] = {"value": "AX Email old-test-secret"}
backend.nodes[1]["name"] = {"value": "AX Submit"}
worker = BrowserWorker(backend, ["http://fixture.test"])
observation = worker.observe()
self.assertEqual(backend.metadata_many_calls, [[1, 3, 4, 2]])
self.assertEqual(backend.metadata_calls, [])
self.assertEqual([candidate["name"] for candidate in observation["candidates"]],
["DOM Email", "AX Submit", "Password"])
self.assertEqual([candidate["role"] for candidate in observation["candidates"]],
["textbox", "button", "textbox"])
self.assertNotIn("old-test-secret", json.dumps(observation))
result = worker.act({
"observationId": observation["observationId"], "candidateId": "c-1", "action": "click",
"postcondition": {"kind": "url", "value": "http://fixture.test/done"},
})
self.assertTrue(result["verified"])
self.assertEqual(backend.metadata_calls, [3])
def test_observe_falls_back_for_missing_batch_metadata(self):
backend = BatchFakeBackend()
backend.batch_result = {1: backend.elements[1]}
worker = BrowserWorker(backend, ["http://fixture.test"])
observation = worker.observe()
self.assertEqual([candidate["name"] for candidate in observation["candidates"]],
["Email", "Submit", "Password"])
self.assertEqual(backend.metadata_calls, [3, 4, 2])
def test_observe_batches_in_bounded_chunks_and_preserves_order_and_deduplication(self):
backend = BatchFakeBackend()
backend.elements = {
index: ElementMetadata(index, "button", "", "button", f"Button {index}", True, False,
ax_role="button", ax_name=f"Button {index}")
for index in range(1, 34)
}
backend.nodes = [
{"backendDOMNodeId": index, "role": {"value": "button"}, "name": {"value": f"AX {index}"}}
for index in range(1, 34)
]
backend.nodes.insert(10, dict(backend.nodes[9]))
worker = BrowserWorker(backend, ["http://fixture.test"])
observation = worker.observe()
self.assertEqual([len(chunk) for chunk in backend.metadata_many_calls], [16, 16, 1])
self.assertEqual([candidate["name"] for candidate in observation["candidates"]],
[f"AX {index}" for index in range(1, 34)])
self.assertEqual(len(observation["candidates"]), 33)
def test_observe_skips_detached_nodes_and_keeps_candidate_limit(self):
backend = BatchFakeBackend()
backend.elements = {
index: ElementMetadata(index, "button", "", "button", f"Button {index}", True, False,
connected=index != 10, ax_role="button", ax_name=f"Button {index}")
for index in range(1, 63)
}
backend.nodes = [
{"backendDOMNodeId": index, "role": {"value": "button"}, "name": {"value": f"AX {index}"}}
for index in range(1, 63)
]
worker = BrowserWorker(backend, ["http://fixture.test"])
observation = worker.observe()
self.assertEqual(len(observation["candidates"]), 60)
self.assertNotIn("AX 10", json.dumps(observation))
self.assertTrue(all(len(chunk) <= 16 for chunk in backend.metadata_many_calls))
def test_password_fill_uses_literal_without_exposing_value_in_metadata_or_result(self):
backend = FakeBackend()
backend.values[2] = "old-test-secret"
@@ -721,6 +952,26 @@ class BridgeRegressionTests(unittest.TestCase):
with self.assertRaisesRegex(WorkerError, "outside"):
worker.act({"observationId": obs["observationId"], "candidateId": "c-0", "action": "fill", "text": "Ada"})
def test_fill_url_postcondition_reuses_post_action_tab_state(self):
backend = FakeBackend()
original_fill = backend.fill
def fill_and_navigate(node, text):
original_fill(node, text)
backend.url = "http://fixture.test/done"
backend.fill = fill_and_navigate
current_tab = backend.current_tab
backend.current_tab = Mock(side_effect=current_tab)
worker = BrowserWorker(backend, ["http://fixture.test"])
observation = worker.observe()
result = worker.act({
"observationId": observation["observationId"], "candidateId": "c-0",
"action": "fill", "text": "Ada",
"postcondition": {"kind": "url", "value": "http://fixture.test/done"},
})
self.assertTrue(result["verified"])
# observe, fresh pre-mutation revalidation, and one post-action read.
self.assertEqual(backend.current_tab.call_count, 3)
def test_oversized_input_and_sensitive_error_are_not_echoed(self):
worker = BrowserWorker(FakeBackend(), ["http://fixture.test"])
out = io.BytesIO()
@@ -738,6 +989,136 @@ class BridgeRegressionTests(unittest.TestCase):
backend._load_harness()
spawn.assert_not_called()
def test_real_backend_list_tabs_uses_one_inventory_with_helper_filtering_and_openers(self):
helper = Mock()
helper.cdp.return_value = {"targetInfos": [
{"type": "page", "targetId": "page", "url": "http://fixture.test/form", "title": "Fixture",
"openerId": "opener"},
{"type": "page", "targetId": "internal", "url": "chrome://settings", "title": "Settings"},
{"type": "page", "targetId": "devtools", "url": "devtools://devtools", "title": "DevTools"},
{"type": "page", "targetId": "placeholder", "url": "about:blank#startup", "title": "Starting agent 1"},
{"type": "service_worker", "targetId": "worker", "url": "http://fixture.test/sw", "title": "Worker"},
{"type": "page", "targetId": "plain", "url": "https://other.test/", "title": "Other", "openerId": ""},
{"type": "page", "targetId": "malformed-opener", "url": "http://fixture.test/other", "title": "Other",
"openerId": 42},
]}
backend = RealBrowserBackend.__new__(RealBrowserBackend)
backend._helpers = helper
tabs = backend.list_tabs()
self.assertEqual([tab["targetId"] for tab in tabs], ["page", "plain", "malformed-opener"])
self.assertEqual(tabs[0]["openerTargetId"], "opener")
self.assertIsNone(tabs[1]["openerTargetId"])
self.assertIsNone(tabs[2]["openerTargetId"])
helper.cdp.assert_called_once_with("Target.getTargets")
helper.list_tabs.assert_not_called()
def test_real_backend_list_tabs_propagates_inventory_errors_without_fallback(self):
helper = Mock()
helper.cdp.side_effect = RuntimeError("inventory unavailable")
backend = RealBrowserBackend.__new__(RealBrowserBackend)
backend._helpers = helper
with self.assertRaisesRegex(RuntimeError, "inventory unavailable"):
backend.list_tabs()
helper.cdp.assert_called_once_with("Target.getTargets")
helper.list_tabs.assert_not_called()
def test_real_backend_switch_tab_does_not_reenable_harness_domains(self):
helper = Mock()
helper.switch_tab.return_value = "session-1"
backend = RealBrowserBackend.__new__(RealBrowserBackend)
backend._helpers = helper
backend.switch_tab("target-1")
self.assertEqual(backend._session_id, "session-1")
self.assertEqual(backend._attached_target, "target-1")
helper.switch_tab.assert_called_once_with("target-1", activate=False)
helper.cdp.assert_not_called()
def test_real_backend_caches_select_all_modifier_for_backend_lifetime(self):
helper = Mock()
calls = []
def cdp(method, **_kwargs):
calls.append(method)
return {"userAgent": "Macintosh"} if method == "Browser.getVersion" else {}
helper.cdp.side_effect = cdp
backend = RealBrowserBackend.__new__(RealBrowserBackend)
backend._helpers = helper
backend._call = Mock(return_value=True)
backend.fill(1, "first")
backend.fill(1, "second")
self.assertEqual(calls.count("Browser.getVersion"), 1)
keydowns = [call for call in helper.cdp.call_args_list
if call.args[0] == "Input.dispatchKeyEvent" and call.kwargs["type"] == "rawKeyDown"]
self.assertEqual([call.kwargs["modifiers"] for call in keydowns], [4, 4])
def test_wait_ready_returns_immediately_for_complete_document(self):
backend = RealBrowserBackend.__new__(RealBrowserBackend)
helper = Mock()
helper.js.return_value = "complete"
backend._helpers = helper
clock = FakeClock()
with patch("browser_worker.time.monotonic", side_effect=clock.monotonic), \
patch("browser_worker.time.sleep", side_effect=clock.sleep):
self.assertTrue(backend.wait_ready(timeout=3.0))
helper.js.assert_called_once_with("document.readyState")
helper.wait_for_load.assert_not_called()
self.assertEqual(clock.sleeps, [])
def test_wait_ready_polls_loading_document_at_short_bounded_intervals(self):
backend = RealBrowserBackend.__new__(RealBrowserBackend)
helper = Mock()
helper.js.side_effect = ["loading", "complete"]
backend._helpers = helper
clock = FakeClock()
with patch("browser_worker.time.monotonic", side_effect=clock.monotonic), \
patch("browser_worker.time.sleep", side_effect=clock.sleep):
self.assertTrue(backend.wait_ready(timeout=3.0))
self.assertEqual(helper.js.call_count, 2)
self.assertEqual(len(clock.sleeps), 1)
self.assertGreater(clock.sleeps[0], 0)
self.assertLessEqual(clock.sleeps[0], 0.025)
helper.wait_for_load.assert_not_called()
def test_wait_ready_times_out_without_sleeping_past_deadline(self):
backend = RealBrowserBackend.__new__(RealBrowserBackend)
helper = Mock()
helper.js.return_value = "loading"
backend._helpers = helper
clock = FakeClock()
with patch("browser_worker.time.monotonic", side_effect=clock.monotonic), \
patch("browser_worker.time.sleep", side_effect=clock.sleep):
self.assertFalse(backend.wait_ready(timeout=0.075))
self.assertEqual(clock.now, 0.075)
self.assertTrue(all(duration <= 0.025 for duration in clock.sleeps))
helper.wait_for_load.assert_not_called()
def test_wait_ready_preserves_failed_ready_state_query(self):
backend = RealBrowserBackend.__new__(RealBrowserBackend)
helper = Mock()
helper.js.side_effect = RuntimeError("ready state query failed")
backend._helpers = helper
clock = FakeClock()
with patch("browser_worker.time.monotonic", side_effect=clock.monotonic), \
patch("browser_worker.time.sleep", side_effect=clock.sleep):
with self.assertRaisesRegex(RuntimeError, "ready state query failed"):
backend.wait_ready(timeout=3.0)
self.assertEqual(clock.sleeps, [])
helper.wait_for_load.assert_not_called()
def test_real_backend_unrestricted_policy_still_requires_one_web_tab(self):
helper = Mock()
helper.list_tabs.return_value = [
@@ -819,6 +1200,55 @@ class BridgeRegressionTests(unittest.TestCase):
self.assertEqual(keydowns[0].kwargs["modifiers"], 4)
backend._helpers.press_key.assert_called_once_with("Backspace")
def test_metadata_many_uses_one_call_and_releases_temporary_object_group(self):
backend = RealBrowserBackend.__new__(RealBrowserBackend)
helper = Mock()
backend._helpers = helper
def cdp(method, **kwargs):
if method == "DOM.resolveNode":
return {"object": {"objectId": f"object-{kwargs['backendNodeId']}"}}
if method == "Runtime.callFunctionOn":
return {"result": {"value": [
{"tag": "input", "type": "text", "role": "", "name": "Email",
"connected": True, "mainFrame": True, "contenteditable": False,
"visible": True, "disabled": False},
{"tag": "button", "type": "", "role": "button", "name": "Submit",
"connected": True, "mainFrame": True, "contenteditable": False,
"visible": True, "disabled": False},
]}}
return {}
helper.cdp.side_effect = cdp
result = backend.metadata_many([1, 2])
self.assertEqual([metadata.name for metadata in result.values()], ["Email", "Submit"])
calls = [call.args[0] for call in helper.cdp.call_args_list]
self.assertEqual(calls.count("DOM.resolveNode"), 2)
self.assertEqual(calls.count("Runtime.callFunctionOn"), 1)
self.assertEqual(calls.count("Runtime.releaseObjectGroup"), 1)
call = next(call for call in helper.cdp.call_args_list if call.args[0] == "Runtime.callFunctionOn")
self.assertEqual([argument.get("objectId") for argument in call.kwargs["arguments"]],
["object-1", "object-2"])
def test_metadata_many_releases_object_group_when_batch_call_fails(self):
backend = RealBrowserBackend.__new__(RealBrowserBackend)
helper = Mock()
backend._helpers = helper
def cdp(method, **kwargs):
if method == "DOM.resolveNode":
return {"object": {"objectId": f"object-{kwargs['backendNodeId']}"}}
if method == "Runtime.callFunctionOn":
raise RuntimeError("detached")
return {}
helper.cdp.side_effect = cdp
self.assertEqual(backend.metadata_many([1, 2]), {})
self.assertEqual(
[call.args[0] for call in helper.cdp.call_args_list].count("Runtime.releaseObjectGroup"), 1
)
def test_call_releases_remote_object_on_browser_exception(self):
backend = RealBrowserBackend.__new__(RealBrowserBackend)
backend._helpers = Mock()