Compare commits
21
Commits
89474017e1
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b667b9ffc4 | ||
|
|
0cc0e82bbc | ||
|
|
284fdd638d | ||
|
|
9862ef0be9 | ||
|
|
5ddd98bf5f | ||
|
|
54303e0b5d | ||
|
|
2dc359c48d | ||
|
|
fe643e2d8d | ||
|
|
12a6bb6afb | ||
|
|
6a725c1f47 | ||
|
|
f2e42f5faf | ||
|
|
acd57ae2a7 | ||
|
|
e10a9573b9 | ||
|
|
574e083e75 | ||
|
|
4360f01947 | ||
|
|
c0c18a5191 | ||
|
|
3c321b3033 | ||
|
|
a79418efcb | ||
|
|
05793fe7b0 | ||
|
|
6fba62800a | ||
|
|
773c628512 |
@@ -54,10 +54,11 @@
|
||||
},
|
||||
"extensions/fast-browser": {
|
||||
"name": "@jbcode/pi-fast-browser",
|
||||
"version": "0.1.0",
|
||||
"version": "0.1.1",
|
||||
"peerDependencies": {
|
||||
"@earendil-works/pi-ai": "catalog:pi-peer",
|
||||
"@earendil-works/pi-coding-agent": "catalog:pi-peer",
|
||||
"@earendil-works/pi-tui": "catalog:pi-peer",
|
||||
"typebox": "catalog:pi-peer",
|
||||
},
|
||||
},
|
||||
|
||||
@@ -8,7 +8,7 @@ supersedes:
|
||||
|
||||
# Replace the custom workflow engine with a skill-led workflow
|
||||
|
||||
> The legacy-runtime retention and pilot-before-retirement prerequisite below are superseded by [ADR 0007](0007-archive-legacy-workflows-05SEP2026.md). The skill-led workflow and its outstanding validation requirements remain in force.
|
||||
> The legacy-runtime retention and pilot-before-retirement prerequisite below are superseded by [ADR 0007](0007-archive-legacy-workflows-05SEP2026.md). The approval, disposable-data, review, verification, and record-commit policies are superseded by [ADR 0008](0008-risk-based-development-workflow-07OCT2026.md). The skill-led architecture and outstanding validation obligations remain in force; interpret scenario expectations under the current policy.
|
||||
|
||||
## Context
|
||||
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
# Use risk-based development workflow gates
|
||||
|
||||
On 07OCT2026, JB approved replacing routine plan-approval pauses and repeated full verification with risk-based automation. Clear work proceeds from a short recorded plan; unresolved product/design choices and meaningful risk retain human gates. This supersedes the categorical plan approval, migration approval, commit-per-gate, two-reviewer, and repeat-full-suite policies in [ADR 0006](0006-skill-led-development-workflow-04SEP2026.md) and its [implementation plan](../workflow/skill-led-development-workflow-plan.md).
|
||||
|
||||
A standing repository policy may authorize schema changes and reset/reseed for named, verified disposable environments. Pre-production alone grants no authority; valuable/shared data, production access, external compatibility, and uncertain targets remain protected. Prefer direct changes over speculative compatibility or rollback machinery when the actual preservation contract permits rebuilding.
|
||||
|
||||
Use focused checks while developing, one independent read-only reviewer covering standards and specification, targeted follow-up on bounded fixes, and one full required suite after remediation. Reuse evidence only while its inputs remain applicable; failures and broader changes still require appropriate reruns. Keep tests complete for meaningful behavior and credible failures rather than exhaustive permutations at every layer. Stricter repository/CI/hook requirements remain binding until explicitly revised.
|
||||
|
||||
Preserve worktrees, scoped authorization, non-trivial UI direction approval, real interface validation, mutator exclusion, pinned review evidence, recoverable records, and safe local integration. Batch record updates into meaningful commits rather than requiring initial, per-gate, and closure commits. Keep the existing checkpoint heading contract and archived engines; add no replacement extension or hook runtime changes.
|
||||
|
||||
The prior twelve-scenario and three-pilot validation obligation remains open in [todos](../todos.md#replace-workflow-with-a-skill-based-orchestrator), with expected decisions interpreted under this superseding policy. Focused policy walkthroughs for this revision are not claims that those execution scenarios or pilots passed.
|
||||
@@ -0,0 +1,68 @@
|
||||
# Fast-browser performance
|
||||
|
||||
## Result
|
||||
|
||||
On the provider-free local Chromium fixture, 20 alternating pairs reduced median warm workflow time from **702 to 279 ms (60%)**, and p95 from **729.05 to 315.30 ms (57%)**. Both arms passed all 20 warm attempts and all three cold attempts. Counter collection and owned-process cleanup passed.
|
||||
|
||||
This is one local fixture on native Linux, not a general website or agent benchmark. The fixture includes four navigations, 0/10/60-control observations, replacing a display name, saving and previewing it, immediate and delayed actions, and popup discovery/closure. A final assertion checks the entered name independently of the action postconditions.
|
||||
|
||||
| Measurement | Baseline p50 / p95 | Optimized p50 / p95 |
|
||||
| --- | ---: | ---: |
|
||||
| Warm complete workflow (20 per arm) | 702 / 729.05 ms | 279 / 315.30 ms |
|
||||
| Observe 0 controls | 2 / 5 ms | 2 / 3.35 ms |
|
||||
| Observe 10 controls | 16 / 23.60 ms | 7 / 11.20 ms |
|
||||
| Observe 60 controls | 78 / 93.05 ms | 28.50 / 31.15 ms |
|
||||
| Cold startup + first observation (3 per arm) | 273 / 306.30 ms | 270 / 283.50 ms |
|
||||
|
||||
Warm-worker lifetime CDP calls fell **17,791 to 7,407 (58%)**; IPC helper calls fell **19,135 to 8,751 (54%)**. These are separate counters, not additive: CDP calls also use IPC. Counts include worker initialization, the unmeasured priming observation, all 20 flows, and teardown; cold-worker counts are not included in these totals.
|
||||
|
||||
Summed transport observation time per warm flow had medians of 228.23 versus 84.74 ms. Navigation medians were 339.46 versus 61.55 ms. Action medians were 112.42 versus 109.97 ms. The main gains are observation and readiness, not faster mutation execution. The three cold samples do not establish a meaningful startup improvement.
|
||||
|
||||
## Changes
|
||||
|
||||
- Batch value-free DOM metadata in chunks of 16, while retaining Chrome's AX discovery, roles, ordering and click names. Fill labels retain the existing DOM-only label extraction. Temporary object groups are released after each chunk; partial or failed batches use the legacy safe path.
|
||||
- Keep fresh per-target AX/DOM revalidation, geometry/occlusion checks, field verification and click postconditions. No node handles, visibility, geometry, document identity, or origin authorization are cached across operations.
|
||||
- Remove redundant domain-enabling calls already made by pinned browser-harness 0.1.13, collect tab and opener metadata in one target inventory, cache the browser's select-all modifier, and reuse the existing tab read for fill URL verification.
|
||||
- Compute scoped origin contexts locally, preserving lifecycle/serialization checks and enforcement on subsequent worker operations. This avoids an otherwise redundant JSONL dispatch and does not start a worker merely to narrow a policy.
|
||||
- Replace the harness's 300 ms readiness sleep with bounded 25 ms monotonic polling of the same `document.readyState == complete` predicate and timeout.
|
||||
- After one close dispatch, verify target disappearance with bounded inventory reads under the existing action-verification budget. A close acknowledgement is not proof that Chrome has removed the target. The mutation is never retried.
|
||||
|
||||
## Reproduce
|
||||
|
||||
From the repository root, export the unchanged baseline and run the same benchmark against both workers:
|
||||
|
||||
```bash
|
||||
mkdir -p /tmp/fast-browser-performance-baseline-773c628
|
||||
git show 773c628512fdfdf5a139ae1b27833bea97d74255:extensions/fast-browser/browser_worker.py \
|
||||
> /tmp/fast-browser-performance-baseline-773c628/browser_worker.py
|
||||
node --experimental-strip-types extensions/fast-browser/performance-benchmark.ts \
|
||||
--baseline-worker /tmp/fast-browser-performance-baseline-773c628/browser_worker.py \
|
||||
--optimized-worker extensions/fast-browser/browser_worker.py \
|
||||
--python .pi/fast-browser-demo/venv/bin/python \
|
||||
--browser /usr/bin/chromium \
|
||||
--samples 20 --cold-samples 3 --counts 0,10,60 \
|
||||
> /tmp/fast-browser-performance-20-pairs-final.json
|
||||
```
|
||||
|
||||
Supply an existing Python environment containing `browser-harness==0.1.13`; the runner does not install dependencies. Use a new output filename when retaining an earlier result. Missing or invalid counters, any failed attempt, or unverified cleanup makes the command fail. Temporary browser state is retained if cleanup cannot be proven.
|
||||
|
||||
Validation environment: Chromium 152.0.7977.75, Python 3.12.13, Node v26.7.0, browser-harness 0.1.13. Measured worker commit: `2dc359c48d042a2982de8c972008b76685cbe105`.
|
||||
|
||||
- Baseline worker SHA-256: `487673653dc82a029cb5533b20566527739f8f5fd2155e5cfbd89da818132035`.
|
||||
- Optimized worker SHA-256: `d33814efb085e56f0bd6f2b8cf0c119559b37fa562a31aea0af1ed20a5550d7a`.
|
||||
- Retained final report: `/tmp/fast-browser-performance-20-pairs-final.json`.
|
||||
- Report SHA-256: `2e6a965f2a7a82c624902606e80a0fba074d16c23f87a85725eaf95f6b32a4ff`.
|
||||
|
||||
## Measurement boundaries and retained failures
|
||||
|
||||
Baseline and optimized Python workers use the same runner, fixture and browser configuration. Warm pairs alternate AB/BA; browser/worker startup is measured separately. Density observations are timed separately from navigation. Reports retain every attempt, bounded operation names, elapsed times and counters, not page text, field values, credentials, raw CDP arguments or screenshots. These measurements do not establish provider latency, semantic accuracy, Internet-site performance or cross-platform behavior. The TypeScript origin-context optimization has call-count tests but is not measured by this Python-worker comparison.
|
||||
|
||||
The first stable 20-pair run, before the close/readiness correction, is retained at `/tmp/fast-browser-performance-20-pairs.json`: baseline 19/20 and optimized 18/20 warm attempts passed. Three popup closes were unverified. Faster inventory calls exposed a pre-existing one-shot close-verification race. Observation was already faster, but coarse navigation polling produced mixed workflow medians (445 versus 551 ms). A four-pair diagnostic with per-operation timing at `/tmp/fast-browser-performance-4-pairs-diagnostic-20260921.json` isolated the navigation cost. Deterministic fake-clock tests reproduced the close race before the fix. An earlier smoke failure is retained at `/tmp/performance-smoke-final.json`.
|
||||
|
||||
These earlier results are not discarded or pooled with the corrected implementation. The final 46-attempt run above passed without retries of failed mutations.
|
||||
|
||||
## Verification and inspiration
|
||||
|
||||
Focused tests cover protocol call counts, batch fallback/cleanup and privacy, accessible naming, stale targets, origin-context lifetime, benchmark accounting, and close/readiness deadlines. Existing real Chromium development validation covers navigation, popups, origin narrowing, private diagnostics and owned cleanup. Exact full-suite and review outcomes are recorded in [the workflow ledger](plans/fast-browser-performance.md).
|
||||
|
||||
[jev-ultrafast's performance report](https://github.com/browser-use/jev-ultrafast/blob/main/docs/performance.md) motivated reducing browser round trips. Its DOM/value snapshot and autonomous operation selection are not copied: fast-browser retains its narrower data and execution contracts.
|
||||
@@ -0,0 +1,134 @@
|
||||
# Local alternatives to JEV: SemIf, decider, and djev
|
||||
|
||||
Reviewed: 2026-09-21. Research only; no deployment or integration decision approved.
|
||||
|
||||
## Recommendation
|
||||
|
||||
**Trial Mapika decider first on the existing RTX 4090, then f302. Keep SemIf as a frozen general-model scoring baseline; defer djev. Do not rent a GPU just to reduce JEV costs.**
|
||||
|
||||
Only **decider** releases purpose-trained decision weights. SemIf and djev implement decision interfaces around existing open-weight models. All three can avoid autoregressive answer generation, but that alone does not establish JEV-equivalent accuracy, calibrated confidence, or faster end-to-end decisions.
|
||||
|
||||
Your hardware is sufficient to investigate the most promising small models without renting anything. Privacy and offline availability are the strongest reasons to proceed. Cost savings and latency improvements remain workload-dependent.
|
||||
|
||||
## Comparison
|
||||
|
||||
| Candidate | What it actually is | Fit for fast-browser | Main limitation | Priority |
|
||||
|---|---|---|---|---|
|
||||
| **decider** | Fine-tuned Qwen3.5 decision models plus Python inference and a JEV-shaped server | Choice supports 2–255 options; includes routing and predicate APIs; some browser-selection evidence | No validated 61-way fast-browser performance; API and confidence differences need handling | **First trial** |
|
||||
| **SemIf** | MIT scoring framework using existing frozen models, strongest tested baseline Qwen3.5-4B | Useful direct-logit baseline for small candidate sets | Python supports only 2–16 options; browser demo 2–20; scores explicitly uncalibrated | Comparison baseline |
|
||||
| **djev** | Apache-2.0 typed-decision framework over DiffusionGemma-26B-A4B-it | Nominally supports 61-option Choice, with exact-token validation | Published BF16 runtime needs roughly 52 GB just for weights; no held-out browser-quality evidence | Defer |
|
||||
|
||||
Fast-browser allows **60 controls plus `none_of_the_above`**, not just a binary decision. Its controller already accepts a small `Decision` interface: candidate ID or null, confidence, and model identity. Keep that boundary and existing execution safeguards. [1–4]
|
||||
|
||||
### decider: best practical starting point
|
||||
|
||||
Start with **decider-2b v10 BF16 on the 4090**. It has approximately 1.9B parameters and 3.5 GB of weights. On f302, evaluate **decider-0.8b BF16** first for a low-memory baseline, then 2B with bounded short inputs if memory permits. Neither GPU has project-published measurements. [2]
|
||||
|
||||
The author reports 2B v10 MiniWoB++ success of **90.9% greedy**, or 93.2% sampled, across 22 small synthetic click tasks and eight seeds. Greedy is the more relevant comparison because fast-browser selects the highest-probability option. The separate 82.7% Mind2Web result is offline candidate selection, not completed browser workflows. Neither proves performance on real 60-control pages, filling forms, missing targets, or malicious labels. [2]
|
||||
|
||||
For routing, it has full-label classification evaluations and runtime-defined descriptions. For semantic checks, Noul returns P(yes); use a three-way Choice when “unknown” needs an explicit outcome. Its temperature scaling improves calibration on its own evaluation distribution, not necessarily ours. The release also documents weak long-array index lookup, relevant because our current criteria refer to `controls[index]`. Compare the existing request with a separately evaluated variant that places role/name descriptions directly in each criterion. [1,2]
|
||||
|
||||
**Integration is not a URL substitution:**
|
||||
|
||||
- `systemone.format_answer` rounds probabilities to four decimals. A dependency-free probe with three equal probabilities returned a sum of **0.9999**, which fails our parser's `1e-6` normalization tolerance. Use backend-specific, bounded numeric validation or preserve greater upstream precision; do not weaken all validation indiscriminately.
|
||||
- The server defaults to a local training-model path and enables compile/FP8. Explicitly select released weights and begin with **BF16/eager, compile and FP8 disabled** on Ada hardware. Published optimized timings use GH200/B300.
|
||||
- State truncation is possible. Reject oversized evidence before inference rather than silently losing candidate information.
|
||||
- Its example server binds all interfaces without a documented authentication layer. Start loopback-only; remote serving requires authenticated, restricted access. [2]
|
||||
|
||||
The 35B model is not a sensible initial target: BF16 requires an 80 GB-class GPU. Its 19.6 GB NVFP4 checkpoint targets Blackwell runtimes; fitting the weight-file size inside 24 GB does not establish RTX 4090 compatibility. [2]
|
||||
|
||||
### SemIf: useful control experiment, not a replacement yet
|
||||
|
||||
SemIf reads option-token logits from frozen language models. Qwen3.5-4B is its strongest tested baseline. On a selected 102-row TypeSafe public subset, it reports **84.5% reference agreement versus 88.3% for published JEV values**. This was not a live head-to-head, is not browser accuracy, and does not establish confidence parity. [3]
|
||||
|
||||
Its native Qwen3.5 run reports approximately **8.89 GB peak CUDA allocation**, rising to 11.61 GB with shared-state parallelism. That is suitable for the 4090, not safely for f302's 8 GB GPU. Quantized browser artifacts exist, but their quality was not measured by the BF16 quality table. [3]
|
||||
|
||||
The advertised roughly 20 decisions/second comes from sharing state across many criteria. It is **not a 50 ms measurement of one fresh browser decision**. The WebGPU Qwen3.5 smoke test took about 3.27 seconds for a direct decision on an RTX 3090. Prefer a native baseline, not an assumption that the browser demo is the fastest deployment. [3]
|
||||
|
||||
Testing the full fast-browser candidate range would require extending and validating its label/readout scheme and prompt budget. Do not silently shortlist away controls just to satisfy its limit.
|
||||
|
||||
### djev: interesting architecture, wrong first deployment
|
||||
|
||||
Djev uses one structured read of DiffusionGemma rather than ordinary multi-step diffusion generation. It releases no new trained weights. Its published Linux/CUDA/vLLM composite uses unquantized BF16 and a B200 reference host. **Neither your 24 GB 4090 nor 8 GB 4060 can run that published configuration.** The desktop's system RAM does not solve the documented single-GPU VRAM requirement. [4]
|
||||
|
||||
Quantized DiffusionGemma paths exist upstream, but compatibility with djev's pinned patches, exact-token reads, and consumer GPUs is a separate porting experiment. Do not treat it as an available supported local setup.
|
||||
|
||||
Its own original BF16 development run reported model-call p50 **59 ms**, but full HTTP p50 **376 ms**, p95 **1,099 ms**, across only 12 requests. A faster historical quantized run is not the published BF16 release. These are development fixtures, not held-out quality or matched latency evidence. [4]
|
||||
|
||||
**Its Choice confidence is normalized entropy concentration, not the winner's probability.** Reusing fast-browser's 0.85 threshold would have a different meaning. Calibration is explicitly unvalidated. [4]
|
||||
|
||||
## Hardware assessment
|
||||
|
||||
Read-only system queries observed:
|
||||
|
||||
| Target | Hardware | Sensible investigation |
|
||||
|---|---|---|
|
||||
| **This host** | RTX 4090, 24 GB VRAM; 186 GiB RAM; Ryzen 9950X3D identification with eight exposed CPUs | decider-2b BF16; SemIf Qwen3.5-4B BF16 comparison |
|
||||
| **f302** | RTX 4060 Laptop, 8 GB VRAM; 14 GiB RAM, about 8.3 GiB available; Core Ultra 7 155H | decider-0.8b BF16; test 2B memory at realistic prompt sizes |
|
||||
| **CPU-only** | Existing CPU/RAM | Decider's plain Transformers path is a possible compatibility experiment; SemIf's inspected native path and djev's published path require GPUs. Competitive CPU latency is unestablished. |
|
||||
| **Rented 24 GB** | Similar capacity to the already-owned desktop | Only useful for separate always-on availability or contention, not additional model capability |
|
||||
| **Rented 80 GB+** | Potential BF16 capacity for larger models | Later quality experiment if small models fail; capacity alone does not validate djev's runtime on a different GPU |
|
||||
|
||||
Hardware fit above is an inference from requirements, **not a successful load test**. The desktop had about 3.2 GB VRAM occupied at inspection. Include existing workloads, context/workspace allocation, power state, and laptop thermal limits in measurements.
|
||||
|
||||
“Local” inference on a rented host still sends inputs off your premises and depends on its network/operator. Serving from the desktop to f302 keeps inference on your hardware but is not offline when f302 cannot reach it. Fully offline operation on f302 requires its own model and cached runtime artifacts.
|
||||
|
||||
## Economics
|
||||
|
||||
Current published JEV price is **$0.042 per million input tokens**, with free output. At 2,000 input tokens per request: [5]
|
||||
|
||||
| Requests | JEV input cost |
|
||||
|---|---:|
|
||||
| 1,000 | $0.084 |
|
||||
| 100,000 | $8.40 |
|
||||
| 1,000,000 | $84 |
|
||||
|
||||
These are **semantic requests actually sent to JEV**, not all browser actions. A unique exact accessible-name match already bypasses JEV. The current adapter sends one target question per request. TypeSafe supports sharing state across independent questions, but that is a provider capability, not batching currently implemented here. Any planner/text or recovery calls have separate costs. [1,5]
|
||||
|
||||
Rental snapshots checked on 2026-09-21, excluding storage, bandwidth, taxes, and other extras:
|
||||
|
||||
| Example | Hourly | Always-on, 730 h/month | Break-even at 2k tokens/request |
|
||||
|---|---:|---:|---:|
|
||||
| Vast RTX 4090 marketplace listing | $0.14 | $102 | 1,667 requests/hour |
|
||||
| RunPod RTX A5000 24 GB | $0.27 | $197 | 3,214/hour |
|
||||
| RunPod RTX 4090 | $0.74 | $540 | 8,810/hour |
|
||||
|
||||
Marketplace prices and capacity are volatile; these are not availability guarantees. [6]
|
||||
|
||||
`JEV cost/request = 0.042 × input_tokens / 1,000,000`
|
||||
|
||||
`Rental break-even requests/hour = hourly_price / JEV_cost_per_request`
|
||||
|
||||
For owned hardware, count **incremental measured wall power** and operational overhead. Do not charge the entire PC's existing idle consumption to one intermittent model, or mistake GPU TDP for measured inference consumption. Extra model fallbacks can outweigh savings on cheap JEV calls.
|
||||
|
||||
Scale-to-zero hosting avoids some idle rent but introduces model-loading/cold-start delays. Keeping a worker warm trades that latency back for idle cost. For this use case, benchmark owned hardware before considering rental. [6]
|
||||
|
||||
## What would establish efficacy
|
||||
|
||||
Proposed evaluation, not yet approved or executed:
|
||||
|
||||
1. **Reproducible load gate.** Pin source, weights, tokenizer, runtime, precision, and settings. Verify BF16/eager loading, memory, 61 labels, explicit abstention, full probability output, cancellation, and token-budget rejection. Test decider-2b on the 4090, then 0.8b/2b on f302. Start with one persistent worker, not a cold model per decision.
|
||||
2. **Recorded-fixture comparison, without browser actions.** Replay the same labeled observations through JEV and local adapters. JEV calls remain remote: use synthetic/redacted fixtures, or explicitly authorize any real data sent to TypeSafe. Task text and control labels can contain sensitive information even when credentials are excluded. Preserve representative AX/control shapes. Separate calibration and held-out sets by page/task family. Cover candidate counts through 60 plus abstain, absent/duplicate targets, paraphrases, distractors, option-order changes, long labels, untrusted instructions, routing labels, and yes/no/unknown predicates. SemIf comparisons must explicitly use its supported cardinality unless extended.
|
||||
3. **Measure useful accuracy.** Report correct-target inclusion, top-choice accuracy, wrong accepted choices, abstention precision/recall, coverage, calibration, and fallback cost. Calibrate thresholds per backend and workload; neither aggregate ECE nor a high confidence is an authorization guarantee. Candidate freshness and permissions remain code checks.
|
||||
4. **Measure equivalent latency.** Record cold load separately from warm request p50/p95/p99, at concurrency one first. Include serialization, local HTTP, queueing, GPU work, memory, and incremental power. Existing JEV demo calls were 207 and 374 ms; rerun a matched baseline rather than generalizing from those two historical samples. [7]
|
||||
5. **Browser gate.** Only after offline results, run bounded browser fixtures with deterministic postconditions. Preserve candidate identity, origin policy, verification, and no-blind-retry behavior. Report false actions and uncertainty, not just successful clicks. Agree acceptance thresholds before selecting a winner; no observed failures in a small sample does not prove safety.
|
||||
|
||||
Suggested first experiment: **decider-2b on the 4090 versus remote JEV using synthetic/redacted recorded fixtures**, with no production browser execution. A fully offline local-only run can use those same labeled fixtures without contacting JEV. If quality holds and warm latency improves, test f302 and consider a reusable local decision service for routing and predicates. Keep local-only mode distinct from any explicitly permitted remote fallback. Browser text composition and Pi planning still use their configured models; replacing JEV alone does not make the entire workflow private or offline.
|
||||
|
||||
## Evidence and limitations
|
||||
|
||||
No model weights were downloaded, no inference was performed, and no production code or configuration was changed. Source was fetched with OpenSrc; hardware was inspected on both hosts. The only executed compatibility probe exercised decider's pure-Python response formatter. Djev's researcher attempted CPU tests but could not run them because `httpx` was absent; no test pass is claimed.
|
||||
|
||||
All candidate benchmark numbers are author-reported. Source snapshots live under `/srv/data/opensrc/repos/github.com/`. OpenSrc omits Git metadata; the cited upstream commits were identified during research, but byte-for-byte snapshot identity was not independently verified for decider/djev. Pin and verify exact artifacts before a trial.
|
||||
|
||||
Licenses inspected: SemIf code MIT; decider/djev code Apache-2.0; the cited Qwen, MiniCPM5, and DiffusionGemma weight cards declare Apache-2.0. Preserve notices and separately check any chosen third-party quantization/runtime artifacts. An open weight license does not establish training-data provenance or decision quality. [2–4]
|
||||
|
||||
## Sources
|
||||
|
||||
1. Local integration: [`jev.ts`](../extensions/fast-browser/jev.ts), [`contracts.ts`](../extensions/fast-browser/contracts.ts), [`controller.ts`](../extensions/fast-browser/controller.ts), [`scenario-cli.ts`](../extensions/fast-browser/scenario-cli.ts).
|
||||
2. Decider source at [`c4daaac`](https://github.com/Mapika/decider/tree/c4daaac28af9fea95d627015cffa2dd5a5926ee6): [README/results](https://github.com/Mapika/decider/blob/c4daaac28af9fea95d627015cffa2dd5a5926ee6/README.md), [response formatter](https://github.com/Mapika/decider/blob/c4daaac28af9fea95d627015cffa2dd5a5926ee6/decider/systemone.py#L119-L130), [server defaults](https://github.com/Mapika/decider/blob/c4daaac28af9fea95d627015cffa2dd5a5926ee6/decider/serve.py#L18-L25); pinned [2B card](https://huggingface.co/Mapika/decider-2b/blob/b37f7e1ba3fbc9238004cf531fabbee2619973fd/README.md), [0.8B card](https://huggingface.co/Mapika/decider-0.8b/blob/1ea54127d3bd52f6d753d9257b32a6380b873907/README.md), [35B NVFP4 card](https://huggingface.co/Mapika/decider-35b-a3b-nvfp4/blob/080221dad5a9184dc5edbba26909089abea42d73/README.md).
|
||||
3. SemIf at [`ca3ba65`](https://github.com/TheoLeeCJ/SemIf/tree/ca3ba65f142967030ecb453346e94d6f476a69df): [results and limitations](https://github.com/TheoLeeCJ/SemIf/blob/ca3ba65f142967030ecb453346e94d6f476a69df/docs/RESULTS.md), [Python limits](https://github.com/TheoLeeCJ/SemIf/blob/ca3ba65f142967030ecb453346e94d6f476a69df/src/semif_phase1/core.py), [browser worker](https://github.com/TheoLeeCJ/SemIf/blob/ca3ba65f142967030ecb453346e94d6f476a69df/webgpu-demo/worker.js), [memory measurements](https://github.com/TheoLeeCJ/SemIf/blob/ca3ba65f142967030ecb453346e94d6f476a69df/results/phase1-summary.json), [artifact/license inventory](https://github.com/TheoLeeCJ/SemIf/blob/ca3ba65f142967030ecb453346e94d6f476a69df/THIRD_PARTY.md).
|
||||
4. Djev at [`3ce907e`](https://github.com/Davipar/djev-dev/tree/3ce907e6835212f27ee82b4cee9039198c4abe35): [runtime](https://github.com/Davipar/djev-dev/blob/3ce907e6835212f27ee82b4cee9039198c4abe35/docs/runtime.md), [performance caveats](https://github.com/Davipar/djev-dev/blob/3ce907e6835212f27ee82b4cee9039198c4abe35/docs/performance.md), [confidence contract](https://github.com/Davipar/djev-dev/blob/3ce907e6835212f27ee82b4cee9039198c4abe35/djev/contracts.py#L225-L271); [pinned DiffusionGemma card](https://huggingface.co/google/diffusiongemma-26B-A4B-it/blob/f7f5b7f5fa82ffc52addd066915886d497f5517b/README.md).
|
||||
5. TypeSafe: [current pricing/limits](https://docs.typesafe.ai/models), [latency claims](https://typesafe.ai/blog/introducing-system-one-models-and-jev), [parallel questions](https://docs.typesafe.ai/cookbooks/parallel_questions), [privacy policy](https://typesafe.ai/legal/privacy-policy). No-training claims do not imply zero retention.
|
||||
6. Hosting: [Vast 4090 pricing](https://vast.ai/pricing/gpu/RTX-4090), [Vast billing](https://docs.vast.ai/guides/reference/billing), [RunPod pricing](https://www.runpod.io/pricing), [serverless operation](https://docs.runpod.io/serverless/overview).
|
||||
7. Historical local evidence: [fast-browser demo](fast-browser-live-demo.md), [prototype validation](fast-browser-prototype-validation.md), [original JEV feasibility](browser-use-jev-feasibility.md).
|
||||
@@ -49,7 +49,7 @@ Publish `@jbcode/pi-shared` before a new version of any extension that depends o
|
||||
After configuring the same scope and token in the npm configuration visible to Pi, install the currently deployed release set with exact version pins:
|
||||
|
||||
```bash
|
||||
pi install npm:@jbcode/pi-fast-browser@0.1.0
|
||||
pi install npm:@jbcode/pi-fast-browser@0.1.1
|
||||
pi install npm:@jbcode/pi-dictate@0.2.0
|
||||
pi install npm:@jbcode/pi-effort-controls@0.1.0
|
||||
pi install npm:@jbcode/pi-prompt-snippets@0.1.0
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
# Explicit repository relocation
|
||||
|
||||
The link manager normally rejects symlinks outside the current checkout. After a repository move, explicitly authorize the previous root rather than weakening that default:
|
||||
|
||||
```bash
|
||||
node scripts/pi-links.mjs sync --dry-run --migrate-from /absolute/old/root
|
||||
node scripts/pi-links.mjs sync --migrate-from /absolute/old/root
|
||||
node scripts/pi-links.mjs check
|
||||
```
|
||||
|
||||
Inspect the dry run before applying. Migration accepts only a separate absolute old root and only repairs manifest-declared destinations whose resolved link target exactly equals the old root plus the current source's repository-relative path. Dangling old targets are supported. Regular files, directories, unrelated targets, and mismatched relative paths remain conflicts. Replacement rechecks the target immediately before unlinking; this is not an atomic filesystem compare-and-swap.
|
||||
|
||||
The option does not claim stale/unlisted links or old local package registrations. It is rejected for `prune`; normal pruning still requires its own preview and approval. Linked-worktree ownership is a separate issue, not solved by this relocation option.
|
||||
|
||||
## Validated host migration
|
||||
|
||||
Moved exposure from `/srv/data/code/jb-pi/harness-tools` to `/home/jb/code/harness-tools`: 38 global links repaired (35 skills and three instruction files), two project-local skill links created. Normal `check` passed with 40 links and zero local package entries. A bounded scan of harness configuration directories found no remaining symlinks with that old absolute target prefix. All 12 link-manager tests passed, including relocation, dry-run, collision preservation, invalid-input, and no-prune coverage.
|
||||
|
||||
Browser routing was also updated in `skills/tools/browser-harness/SKILL.md`: fast-browser is the default for supported local tests, while browser-harness requires explicit opt-in and must never bypass failed actions or share a fast-browser-owned session. Run `/reload` to refresh skill exposure.
|
||||
@@ -0,0 +1,142 @@
|
||||
# Fast-browser performance workflow record
|
||||
|
||||
## Status
|
||||
|
||||
- Status: complete
|
||||
- Current phase: verified, independently approved product fast-forwarded to local main.
|
||||
- Next safe action: after this record-only closure commit, run final sanity/reachability checks, remove the owned clean worktree/branch, and restore the unrelated file's Git visibility. Final housekeeping evidence is written to `/tmp/pi-present/fast-browser-performance-cleanup.json`.
|
||||
|
||||
## Request and scope
|
||||
|
||||
Optimize fast-browser for local development testing and validation, borrowing proven round-trip reductions from jev-ultrafast. The parent orchestrates; subagents implement exclusive slices. Optimize browser/runtime overhead first, with measured correctness-preserving gains rather than a speculative autonomous-agent rewrite.
|
||||
|
||||
In scope: redundant browser calls; AX-preserving batched observation metadata; local origin-context calculation; a provider-free paired benchmark; focused regression tests and performance documentation.
|
||||
|
||||
Out of scope: new actions, autonomous planning, broader model context, changed model/provider, dependency changes, public API changes, uploads/downloads, personal browser profiles, production access, publishing, and speculative page-state/decision caches. No product UI change. Synthetic fixture HTML is validation material, not a new UI feature.
|
||||
|
||||
## Git identity
|
||||
|
||||
- Source branch: `main` at `773c628512fdfdf5a139ae1b27833bea97d74255`.
|
||||
- Remote: `origin/main`, fetched and identical to local main.
|
||||
- Feature branch: `feature/fast-browser-performance` (must be absent before creation).
|
||||
- Worktree: `/home/jb/code/harness-tools/.worktrees/fast-browser-performance` (must be absent before creation).
|
||||
- Omitted source changes: untracked `docs/local-decision-model-feasibility.md`, preserved untouched. No tracked source modifications observed.
|
||||
- Add `/.worktrees/` to the repository-local Git exclude file before worktree creation. This is local Git housekeeping, not a product change.
|
||||
- Record path after approval: `docs/plans/fast-browser-performance.md` in the new worktree.
|
||||
- Reuse existing root node_modules and the pinned Python environment `/home/jb/code/harness-tools/.pi/fast-browser-demo/venv/bin/python`; no installation or lockfile modification. Any worktree-local links are ignored and explicitly owned.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
1. Existing public tools, operation schemas, candidate ordering/limit, field-label privacy, fresh action revalidation, occlusion checks, outcome verification, budgets, session isolation, and no-blind-retry semantics remain intact.
|
||||
2. Browser hot-path call counts fall deterministically: one target inventory per refresh; no redundant domain enables after the pinned harness attach; browser platform detection at most once per backend; batched observations remove per-candidate partial-AX queries and repeated evaluate/release cycles.
|
||||
3. Chrome AX data remains authoritative for observation roles and click names. Fill labels retain the existing value-free DOM extraction and never fall back to AX value-bearing names. No field values, raw DOM, selectors, or arbitrary scripts enter observations, provider input, or benchmark reports.
|
||||
4. Batch resources exist only for one bounded observation chunk and are released on success/failure. Fresh single-target metadata remains the pre-mutation authority. Missing/failed batch data uses the existing safe per-node path where compatible; no blind mutation retry.
|
||||
5. Origin narrowing remains validated and session-lifetime/serialization aware but does not send a redundant JSONL operation. The effective context still travels with every subsequent browser operation; omitted context restores the session base, and restricted sessions cannot broaden.
|
||||
6. Benchmark results cover unchanged baseline and optimized code with the same runner/fixture/browser, independent outcome assertions, all attempts retained, startup separate from warm operation timing, p50/p95 and protocol counts. Keep optimizations only when call counts improve and live results show no material regression; investigate noisy or mixed results before claiming gains.
|
||||
7. Full repository tests/typecheck, focused regressions, real Chromium runtime validation, and two independent review axes pass for the final committed diff. Any environment blocker or unsupported claim remains explicit.
|
||||
|
||||
## Approved implementation plan and gates
|
||||
|
||||
Plan approval: user selected `approve` (Approve and implement) through ask_user on 2026-09-21. Approval covers the exact slices and source-file omission below, local benchmark browsers/fixtures, local commits, and eventual safe fast-forward integration. It does not authorize dependency changes, production access, API/security-posture changes, or publication.
|
||||
|
||||
### Slice A: provider-free measurement and regression fixture
|
||||
|
||||
Owner `performance-benchmark` (worker). Exclusive paths: `extensions/fast-browser/performance-benchmark.ts`, `extensions/fast-browser/performance-benchmark.test.ts`, `extensions/fast-browser/performance_probe.py`, `extensions/fast-browser/fixtures/performance.html`.
|
||||
|
||||
Build a bounded opt-in runner using existing browser-session/runtime/transport seams. Accept explicit worker/baseline paths and pinned Python/browser configuration rather than copying an old implementation into production. Count CDP/IPC methods and JSONL operations locally without recording arguments, URLs, values, text, or screenshots. A test-only Python probe may wrap the selected worker's helper methods for counting; no product instrumentation switch is required.
|
||||
|
||||
Use realistic local account/settings or search/filter form workflows with 0/10/60 eligible controls, fill replacement, immediate/delayed click outcomes, and popup discovery. Assert actual final state independently of action postconditions. Exercise warm repeated observations and complete verified flows; measure cold startup separately. Alternate baseline/optimized arms, aim for at least 20 paired warm samples, retain failures and cleanup results. Deterministic model stubs may exercise orchestration without provider cost; clearly exclude real provider latency and semantic accuracy from claims. Focused tests cover statistics/report accounting, input bounds, failure inclusion, and report privacy. No wall-clock thresholds in unit tests.
|
||||
|
||||
### Slice B: remove proven browser-call duplication
|
||||
|
||||
Owner `browser-hotpath` (worker). Exclusive paths: `extensions/fast-browser/browser_worker.py`, `extensions/fast-browser/test_browser_worker.py`.
|
||||
|
||||
Use a single `Target.getTargets` response for page inventory and opener metadata, preserving filtering and target identity. Remove duplicate Page/DOM/Runtime/Network enables only where pinned browser-harness 0.1.13 already guarantees attempts on attachment. Cache browser-version-derived select-all modifiers per backend lifetime. Reuse the already obtained tab result during fill URL verification. Add exact call-count and behavioral regressions. Do not lengthen polling intervals as a speculative throughput optimization or cache changing document/target state across operations.
|
||||
|
||||
### Slice C: AX-preserving observation batching
|
||||
|
||||
Owner `browser-hotpath`, serialized after B, same exclusive paths. Use an internal Python-owned script constant to avoid new runtime assets and packaging changes.
|
||||
|
||||
Introduce a private optional batch-metadata seam with the legacy backend/fake fallback. Keep full AX candidate discovery and ordering. Resolve eligible backend node IDs into a bounded temporary object group; evaluate existing value-free DOM metadata for a small chunk together, using full AX snapshot role/name rather than redundant partial AX reads; release the object group in finally. Never retain remote node handles or geometry across observations. Keep fresh existing single-node AX/DOM revalidation before act. Cover distinct AX/DOM labels, password/value privacy, disabled/hidden/disconnected nodes, duplicate candidates, limits, malformed/partial responses, navigation/detachment and cleanup. Measure against the baseline before retaining. If the compatible batch path cannot demonstrate safe benefit, retain B and document the blocked experiment instead of weakening contracts.
|
||||
|
||||
### Slice D: local origin-context calculation
|
||||
|
||||
Owner `runtime-hotpath` (worker). Exclusive paths: `extensions/fast-browser/runtime.ts`, `extensions/fast-browser/runtime.test.ts`.
|
||||
|
||||
Compute `narrowOriginPolicy` locally, retaining runtime closed/aborted behavior, serialized session semantics and current validation/error shaping. Subsequent transport/worker operations must still enforce the returned context. Preserve compatibility for direct legacy transport requests. Test no transport dispatch for narrowing, context propagation, rejected broadening, base restoration, closed/aborted sessions and concurrency behavior. Do not remove transport validation, serialized lanes, or change error/mutation semantics to save time.
|
||||
|
||||
### Orchestration and documentation
|
||||
|
||||
Parent owns `docs/plans/fast-browser-performance.md`, `docs/fast-browser-performance.md`, and the performance section of `extensions/fast-browser/README.md`. No code implementation is assigned to the parent. Record measured boundaries, exact commands, baseline identity, limitations and retained results. Add `docs/todos.md` evidence only for a newly encountered tool defect that cannot be resolved in scope.
|
||||
|
||||
Each mutating assignment has a 15-minute deadline. B and C are separate assignments to the same owner; A and D may run in parallel because paths are disjoint. Workers do not commit, delegate mutation, alter dependencies, modify other owners' files, or integrate branches. Parent records before/after Git evidence and inspects every returned diff. A worker hitting its deadline must stop and report its partial state; reassignment requires proven completion.
|
||||
|
||||
## Ownership
|
||||
|
||||
| Assignment | Owner identity | Deadline | Exclusive paths | Before/after Git evidence | Outcome |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| Research browser | browser-performance-research / scout | complete | none; read-only | source unchanged | delivered |
|
||||
| Research runtime | runtime-performance-research / scout | complete | none; read-only | source unchanged | delivered |
|
||||
| A benchmark | performance-benchmark / worker | first and correction deliveries within deadlines | four listed new files | initial owned patch `/tmp/pi-present/performance-slice-a-initial.patch`; parent inspected corrections and reran 6 tests/compile/check | committed e10a957; stable 20-pair measurement started |
|
||||
| B browser duplication | browser-hotpath / worker | completed within deadline | worker Python + Python tests | delivered only owned paths; `/tmp/pi-present/performance-slice-b.patch`; parent reran 47 Python tests (pass) and diff check | committed a79418e |
|
||||
| C observation batching | browser-hotpath / worker | delivered within deadline | same Python files | owned patch `/tmp/pi-present/performance-slice-c.patch`; parent reran 53 Python tests and diff check | committed 574e083; optimized real Chromium runtime validation passed |
|
||||
| D origin context | runtime-hotpath / worker | completed within deadline | runtime.ts + runtime.test.ts | owned patch `/tmp/pi-present/performance-slice-d.patch`; parent reran 12 tests and typecheck | committed c0c18a5 |
|
||||
| Docs/integration | parent | phase bounded | listed docs/README only | record checkpoints | pending |
|
||||
|
||||
## Verification matrix
|
||||
|
||||
| Surface | Command | Exit code | Outcome |
|
||||
| --- | --- | --- | --- |
|
||||
| Baseline typecheck | `bun run typecheck` | 0 | Passed; an accidental nonexistent chained test command then failed, unrelated to repository correctness. |
|
||||
| Baseline full tests | `bun run test` | 0 | Passed. |
|
||||
| Baseline fast-browser | `bun run test:fast-browser` | 0 | 220 TS passed, 2 skipped; 42 Python passed. |
|
||||
| Focused Python | `.pi/fast-browser-demo/venv/bin/python -m unittest discover -s extensions/fast-browser -p 'test_*.py'` | 0 | 61 passed, including counting, batching, stale/privacy, delayed-close and readiness fake-clock regressions. |
|
||||
| Focused runtime/benchmark | individual `node --test --experimental-strip-types` runs for runtime.test.ts and performance-benchmark.test.ts | 0 | Parent verified 12 runtime and 7 benchmark tests. |
|
||||
| Integrated static/offline | `bun run typecheck && bun run test` | 0 | Full repository suite passed. Fast-browser: 230 TS passed, 2 opt-in skips; 61 Python passed. No dependency installation. |
|
||||
| Scenario validation | scenario-cli `--check --format json` for existing local-form/login/popup examples | 0 | All three validated; no browser started by preflight. |
|
||||
| Real runtime | `FAST_BROWSER_DEVELOPMENT_LIVE=1 node --test --experimental-strip-types extensions/fast-browser/development-live-validation.test.ts` | 0 | Passed with no skip on baseline, batched worker and final corrected worker. Includes private evidence and cleanup. |
|
||||
| Paired benchmark | command in `docs/fast-browser-performance.md`, 20 pairs plus 3 cold samples per arm | 0 | Final all 46 attempts passed, counters complete, cleanup passed. Warm p50 702 to 279ms; p95 729.05 to 315.30ms; CDP 17,791 to 7,407. Earlier failures retained. |
|
||||
| Hygiene/package | `git diff --check`; root package tests in full suite | 0 | Passed. Metadata script is an internal Python constant; no external runtime resource needed. |
|
||||
|
||||
## Review
|
||||
|
||||
- Review base: `773c628512fdfdf5a139ae1b27833bea97d74255`; product HEAD: `54303e0b5d8d99dbd3556a7a97453157e7ed09fd`; product tree: `08de591daedd2b32a64d3e054097d9cdd040089d`.
|
||||
- Exact command: `git diff --binary --full-index --find-renames 773c628512fdfdf5a139ae1b27833bea97d74255...54303e0b5d8d99dbd3556a7a97453157e7ed09fd --`.
|
||||
- Diff SHA-256: `717829461c80874a99ca24f5ef0d85094e237b1b3238ef40c827672f5b0b670e`.
|
||||
- Commit list, newest first: 54303e0, 2dc359c, fe643e2, 12a6bb6, 6a725c1, f2e42f5, acd57ae, e10a957, 574e083, 4360f01, c0c18a5, 3c321b3, a79418e, 05793fe, 6fba628. Reproduce with `git log --oneline <base>..<product-head>` using the pinned full IDs above.
|
||||
- Excluded mutators: parent and all worker sessions (`browser-hotpath`, `runtime-hotpath`, `performance-benchmark`). Stable worker session IDs were not exposed, so the worker profile was excluded.
|
||||
- Standards reviewer: `performance-standards-review`, read-only claim-verifier, session `01a0c515-7d1a-7680-9e5b-d16aa4202b6b`. Confirmed exact digest and APPROVED with zero findings.
|
||||
- Specification reviewer: `performance-specification-review`, independent read-only claim-verifier; stable session ID not exposed. Confirmed exact digest and APPROVED with zero findings.
|
||||
- Both reviewers and parent confirmed unchanged clean HEAD/tree/status before and after review. No remediation cycle was required. Specifications were this approved record and development contracts; standards were repository/global AGENTS.md and established fast-browser patterns.
|
||||
- Full brief and separate axis results are retained at `/tmp/pi-present/fast-browser-performance-review-brief.md` and `/tmp/pi-present/fast-browser-performance-review-results.md`. Closure changes only this record, outside the approved product fingerprint.
|
||||
|
||||
## Integration and rollback
|
||||
|
||||
Source was fetched again and remained at the pinned base; no rebase was needed. Both review approvals and the exact digest were rechecked before `git merge --ff-only feature/fast-browser-performance`, which succeeded on local main. Source main now contains the exact reviewed product.
|
||||
|
||||
The user explicitly approved `temporary_local_exclude` through ask_user for the unrelated untracked `docs/local-decision-model-feasibility.md`: temporarily exclude that exact path, require a clean source checkout, fast-forward, close and clean up, then restore its original untracked visibility. No contents or tracked state may change. The initial and post-fast-forward SHA-256 are both `5bb4fbf4c77742d41a230d25d87c4e4fd607a5bd38851bf61c7eaf022ebfd86c`. Approval evidence is `/tmp/pi-present/fast-browser-integration-approval.md`.
|
||||
|
||||
Product commit is the pinned HEAD above. The next commit touching only this record is the metadata-only closure commit; its ID is discoverable with `git log -1 -- docs/plans/fast-browser-performance.md`. Final sanity checks prove product/closure reachability and clean owned worktree state before normal `git worktree remove` and `git branch -d`. The final cleanup evidence file records actual results after this commit, including restored visibility and unchanged document hash. Only the explicitly owned worktree/branch and temporary local exclude entries are removed. No push, publication, installed-package change or Pi reload occurred.
|
||||
|
||||
Rollback is a normal revert of focused optimization commits, retaining benchmark tooling if useful.
|
||||
|
||||
## Checkpoints and blockers
|
||||
|
||||
- Last committed valid checkpoint: approved plan/assignments at 05793fe; B product a79418e, D product c0c18a5, C product 574e083, A product e10a957. Worktree dependency links include workspace-local node_modules; local Git excludes now ignore these symlinks. No dependency installation occurred.
|
||||
- Blockers: none for implementation. Source still has only the approved unrelated untracked file.
|
||||
- Worktree creation verified at the specified base and path. `/.worktrees/` was already in local Git excludes; no exclude change was needed.
|
||||
- Baseline real Chromium development validation passed (opt-in test exit 0, no skips).
|
||||
- A corrections: alternate pair order, expose per-density timing/count evidence, independently check saved fixture data, make popup postcondition stable, fail/retain evidence on cleanup or missing counters, and sanitize errors to safe categories. This completes existing acceptance criteria without changing public scope.
|
||||
- A correction smoke passed; historical pre-correction transient popup-close failure is preserved at `/tmp/performance-smoke-final.json`, not attributed to concurrent editing. Stable paired measurements must include all failures.
|
||||
- First stable 20-pair result `/tmp/fast-browser-performance-20-pairs.json`: exit 1; baseline 19/20 and optimized 18/20 warm flows passed, all six cold attempts passed, counters/cleanup complete. Failures are popup-close verification (optimized samples 10/13, baseline sample 20). 60-control median observation 79 to 30ms; 10-control 16 to 7ms, but aggregate medians 445 to 551ms. These are diagnostic results, not acceptance.
|
||||
- Additional bounded assignments: browser-hotpath read-only diagnosis (8 minutes), performance-benchmark safe per-operation timing in its existing four paths (15 minutes). Before state acd57ae, product paths clean; only parent performance doc untracked. No scope/API/security expansion: this is required existing benchmark verification.
|
||||
- Browser diagnosis confirmed one-shot close verification races asynchronous Target.closeTarget completion; removing the duplicate inventory makes the pre-existing race more visible. Proposed correction repeats only inventory reads within the existing action-verification budget, never the close mutation. This restores intended verified-close behavior without API/security changes.
|
||||
- Browser-hotpath test-first assignment: 8 minutes, only its existing Python test path, at f2e42f5 with benchmark-owned TS files dirty; product Python remains stable during diagnostic measurement. Add deterministic delayed-disappearance and permanent-presence close cases, prove the first fails before the fix, then return. Before patch `/tmp/pi-present/performance-before-close-tests.patch`.
|
||||
- Timing probe committed 12a6bb6; parent reran 7 benchmark tests and full typecheck. Diagnostic `/tmp/fast-browser-performance-4-pairs-diagnostic-20260921.json` passed all attempts and confirms navigate ~335ms versus observe ~223 to ~88ms. Action times ~110ms remain similar. No provider latency is measured.
|
||||
- Parent reproduced close tests RED: 4 focused tests, 1 expected failure and 1 expected error for delayed/permanent target presence, 2 passed for inventory/cancellation handling. Test patch `/tmp/pi-present/performance-close-red.patch`.
|
||||
- Browser-hotpath correction assignment: 15 minutes, same Python worker/test paths; before HEAD 12a6bb6, only owned red tests dirty plus parent doc. Implement single-dispatch close followed by bounded inventory reads under existing ACTION_TIMEOUT_SECONDS. Replace pinned helper's 300ms readiness sleep with <=25ms bounded monotonic polling for the exact same document.readyState == complete predicate and existing timeout, retaining error propagation. Add readiness regression tests before the fix. This changes private polling cadence, not API, security posture, postconditions or budgets. Before patch `/tmp/pi-present/performance-close-red.patch`.
|
||||
- Corrected close/readiness worker committed 2dc359c; parent inspected patch `/tmp/pi-present/performance-readiness-close-fix.patch` and reran 61 Python tests. Final report `/tmp/fast-browser-performance-20-pairs-final.json` passed all 46 attempts; hash and exact environment/commands are in performance documentation.
|
||||
- Final verification background task bg_vjl_mubjba9s_12 exited 0 after typecheck, complete repository suite, pinned Python suite, real Chromium runtime test, scenario preflight and diff check. No failures; opt-in general-profile test remains skipped in the default suite, while the development live gate was explicitly run.
|
||||
- Both independent reviewers approved the pinned product with zero findings. Source main was clean under the user-approved temporary exact-file exclusion, fast-forward integration succeeded, and the unrelated document hash remained unchanged.
|
||||
- No unresolved product blockers. Failed earlier measurements remain retained and documented.
|
||||
- Next safe action: finalize this record-only closure commit and execute the stated post-commit sanity, reachability, cleanup and visibility-restoration checks.
|
||||
+1
-1
@@ -62,7 +62,7 @@ Slice 1 found two Nono 0.74 limitations while building behavioral policy proofs.
|
||||
|
||||
## Replace `workflow` with a skill-based orchestrator
|
||||
|
||||
The strict `workflow` extension repeatedly failed before implementation because planner output and persisted state-machine constraints were too brittle for adaptive subagent orchestration. Implement the repository-managed `develop-workflow` skill under [ADR 0006](adr/0006-skill-led-development-workflow-04SEP2026.md), where the parent owns the plan, task tree, isolated worktree, explicit approval gates, verification, and independent review while launching bounded subagents directly. Preserve file ownership, reviewer independence, and resumable repository evidence while preferring native Pi primitives. Keep this TODO open until all twelve fresh-session scenarios and three real-change pilots pass. The legacy implementations were archived separately with explicit approval to waive the retirement prerequisite under [ADR 0007](adr/0007-archive-legacy-workflows-05SEP2026.md); archival does not establish validation success.
|
||||
The repository-managed `develop-workflow` skill replaces the brittle legacy engine under [ADR 0006](adr/0006-skill-led-development-workflow-04SEP2026.md), with risk-based automation now defined by [ADR 0008](adr/0008-risk-based-development-workflow-07OCT2026.md). Validate the parent-owned plan, isolated worktree, scoped approval boundaries, proportionate verification, independent review, ownership, and recoverable evidence through native Pi primitives. Keep this TODO open until all twelve fresh-session scenarios and three real-change pilots pass, interpreting old scenario expectations under the superseding policy: clear routine work is autonomous, declared disposable targets use standing authority, one reviewer covers both axes, and valid verification evidence is reused. Policy walkthroughs are not execution-pilot evidence. The legacy implementations were archived separately with explicit approval to waive the retirement prerequisite under [ADR 0007](adr/0007-archive-legacy-workflows-05SEP2026.md); archival does not establish validation success.
|
||||
|
||||
## Support finished-session resume for Claude CLI subagents
|
||||
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
# Skill-led development workflow plan
|
||||
|
||||
**Status:** Accepted
|
||||
**Status:** Historical accepted plan; policy partially superseded
|
||||
|
||||
> [ADR 0008](../adr/0008-risk-based-development-workflow-07OCT2026.md) supersedes categorical plan/migration gates, mandatory initial/per-gate/closure commits, two-reviewer defaults, and repeated full verification below. Use the current [develop-workflow skill](../../skills/develop/develop-workflow/SKILL.md) for execution. [ADR 0007](../adr/0007-archive-legacy-workflows-05SEP2026.md) already superseded the pilot-before-archival prerequisite. The twelve scenarios and three real-change pilots remain outstanding, evaluated against current policy rather than obsolete gate expectations.
|
||||
|
||||
**Date:** 03SEP2026
|
||||
**Working skill name:** `develop-workflow`
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
# Risk-based workflow automation record
|
||||
|
||||
## Status
|
||||
|
||||
- Status: complete
|
||||
- Current phase: integrated locally
|
||||
- Next safe action: none for this implementation; retain the separate execution-pilot TODO.
|
||||
|
||||
## Request and scope
|
||||
|
||||
- Request: reduce routine human interaction and redundant testing, especially for replaceable pre-production data.
|
||||
- In scope: develop-workflow skill and references/template; code-review skill; ADR and historical pointers; validation evidence and existing TODO alignment.
|
||||
- Out of scope: extension runtime/config changes, global instructions, database access, archived engines, checkpoint format changes, replacement of outstanding twelve-scenario/three-pilot validation.
|
||||
|
||||
## Git identity
|
||||
|
||||
- Source branch: main
|
||||
- Base commit: 284fdd638d1f2c790290491e0cfe2802880cec25
|
||||
- Feature branch: feature/workflow-automation
|
||||
- Worktree: /home/jb/code/harness-tools/.worktrees/workflow-automation
|
||||
- Omitted source changes: none; source checkout clean.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [x] Routine work proceeds from a recorded plan without an approval pause; substantive choices and protected actions remain gated.
|
||||
- [x] Disposable-data policy requires named authorized targets, identity checks, scope and reproducible reset/reseed; pre-production alone grants nothing.
|
||||
- [x] Focused tests and review precede one final full verification boundary; valid evidence can be reused and invalidated evidence is rerun.
|
||||
- [x] One non-author reviewer covers standards and specification; targeted fixes do not force fresh full reviews.
|
||||
- [x] Records remain recoverable with existing heading contract; no initial/per-gate/closure commit ceremony is mandatory.
|
||||
- [x] UI direction, real interface validation, worktree isolation, honest results and safe integration remain intact.
|
||||
- [x] No new extension, dependency, runtime behavior or user-level policy change.
|
||||
|
||||
## Approved implementation plan and gates
|
||||
|
||||
- Plan/specification source: user-approved /tmp/pi-present/develop-workflow-automation-review.md and the acceptance criteria above.
|
||||
- Exact plan: revise workflow and supporting references, consolidate review policy in code-review, retain checkpoint heading schema, add ADR 0008 and supersession pointers, run focused checks and scenario walkthroughs, obtain one independent review, run final full suite/static checks, commit and integrate locally.
|
||||
- Plan approval: user said "approved, let's proceed and implement this plan." on 07OCT2026. This explicitly authorizes the risk-based policy and its use for this revision; no redundant re-approval requested.
|
||||
- UI classification and direction approval: documentation-only skill behavior; no product UI change.
|
||||
- Risk decisions and scope: no database/environment declared disposable by this change; no dependency, security-posture, production, publishing or deployment changes. Existing external native-hooks default config directory is absent; no installed policy inferred or modified.
|
||||
|
||||
## Ownership
|
||||
|
||||
- Parent agent owns all edits under skills/develop/develop-workflow/, skills/develop/code-review/SKILL.md, docs/adr/0008-risk-based-development-workflow-07OCT2026.md, supersession pointers in ADR 0006 and the accepted workflow plan, docs/todos.md, and this record.
|
||||
- Independent reviewer owns no files and cannot remediate. No mutating child assignments.
|
||||
|
||||
## Verification matrix
|
||||
|
||||
| Surface | Command or method | Exit code | Outcome |
|
||||
| --- | --- | --- | --- |
|
||||
| Checkpoint compatibility | node --test skills/develop/develop-workflow/tests/checkpoint-record.test.mjs | 0 | 13/13 passed; existing schema retained |
|
||||
| Policy consistency | Node validateRecord on template/record, Python local-link validation, git diff --check | 0 | Both heading contracts valid; 18 links across 9 policy files resolve; whitespace clean. No prose snapshot tests added. |
|
||||
| Scenarios | independent policy walkthrough: routine feature, disposable/protected data, targeted review fix/shared-code change, non-trivial UI | complete | Reviewer confirmed all four intended routes. These are policy walkthroughs, not execution pilots; existing validation TODO remains open. |
|
||||
| Final repository suite | bun run test | 0 | Entire repository script completed after review, 25 seconds, tested product 0cc0e82 from the isolated worktree. One full-suite run. |
|
||||
| Final static check | bun run typecheck | 0 | Passed against product 0cc0e82 after frozen dependency restore. Initial missing-dependency failure retained below. |
|
||||
|
||||
## Review
|
||||
|
||||
- Base and product HEAD: 284fdd638d1f2c790290491e0cfe2802880cec25 / 0cc0e82bbc380a3e5f2d8c689098ae9166d00302
|
||||
- Clean status and tree hash: clean; 2798112031a90797ce99cb7bd92bf75dd84f3e6e
|
||||
- Commit list: 0cc0e82 Make development workflow risk-based and verification proportionate
|
||||
- Exact diff command: git diff --binary --full-index --find-renames 284fdd638d1f2c790290491e0cfe2802880cec25...0cc0e82bbc380a3e5f2d8c689098ae9166d00302 --
|
||||
- SHA-256: 24ebf984eda2c74298d9865dfe61248502d40c2060e082f0234574c2279b804e
|
||||
- Mutator identities: parent agent only
|
||||
- Reviewer identity: parent-launched workflow-policy-reviewer using client-import-reviewer-01a0e83c read-only profile; child/session IDs not exposed. No overlap with sole mutator.
|
||||
- Verdict: regenerated digest confirmed; Standards approved; Specification approved; all four policy walkthroughs matched requirements.
|
||||
- Findings and remediation: no material findings, no remediation cycles.
|
||||
- Before/after reviewer Git evidence: clean status, HEAD, tree, and pinned diff digest unchanged, independently rechecked by parent.
|
||||
|
||||
## Integration and rollback
|
||||
|
||||
- Synchronization and re-verification: git fetch origin completed with exit 0; origin/main and source main both remain 284fdd638d1f2c790290491e0cfe2802880cec25. No rebase delta.
|
||||
- Fast-forward result: main advanced cleanly from 284fdd6 to reviewed product 0cc0e82 with --ff-only.
|
||||
- Product and evidence commits: 0cc0e82; subsequent record-only evidence commit contains this completed report. No product inputs changed after final verification.
|
||||
- Reachability and cleanup: parent proved product reachable from main and both checkouts clean, then removed only .worktrees/workflow-automation and feature/workflow-automation. The unrelated feature-jev-tools worktree was preserved. Final evidence retained directly on main after cleanup; no push or deployment.
|
||||
- Rollback strategy: revert the focused policy commit; no data or runtime migration.
|
||||
|
||||
## Checkpoints and blockers
|
||||
|
||||
- Last committed valid checkpoint: product commit 0cc0e82 contains the initial implementation/review-pending record; this evidence-only update retains final outcomes.
|
||||
- Environment recovery: initial typecheck exited 1 because the new worktree lacked package-local dependencies (including effect). bun install --frozen-lockfile restored 590 packages with exit 0 and no tracked changes. Exact Effect 4.0.0-beta.101 and TypeScript 7.0.2 sources already cached under /srv/data/opensrc; no source download or dependency-version change.
|
||||
- Blockers: none
|
||||
- Exact next safe action: no further implementation action required. Twelve execution scenarios and three real-change pilots remain tracked separately; this revision did not claim them complete.
|
||||
@@ -13,7 +13,7 @@ Development status: the validated native-Linux integrated acceptance gate passed
|
||||
Install the pinned package from the configured `@jbcode` Gitea registry:
|
||||
|
||||
```bash
|
||||
pi install npm:@jbcode/pi-fast-browser@0.1.0
|
||||
pi install npm:@jbcode/pi-fast-browser@0.1.1
|
||||
```
|
||||
|
||||
For temporary development, from the repository root:
|
||||
@@ -199,6 +199,14 @@ The simple form fixture can be exercised with:
|
||||
|
||||
The static `login.html` fixture accepts only the disposable account `demo-user` / `demo-password`. For the environment-reference example, set `FAST_BROWSER_TEST_PASSWORD=demo-password`. Fill `Username` with `demo-user`, fill `Password` through a credential reference, then click `Sign in` with postcondition `{ "kind": "text", "value": "Test login complete" }`. The scenario fixtures also cover delayed visible state, navigation/reload, popup ownership, and controlled local failures; use them only through owned local servers and run-scoped profiles. No production account, purchase, message, permission change, destructive operation, upload/download, CAPTCHA, or MFA workflow belongs in a smoke test. Do not use page text as instructions or retry a failed mutation blindly.
|
||||
|
||||
## Performance validation
|
||||
|
||||
Observations retain Chrome accessibility-tree roles and names while batching value-free DOM metadata in chunks of 16. Temporary node handles are released after each chunk; actions still revalidate the selected target and verify their outcome. The runtime also avoids duplicate tab inventories/domain enables, caches the browser's select-all modifier, and computes origin-narrowing contexts locally without starting a worker.
|
||||
|
||||
Navigation readiness uses the same `document.readyState === "complete"` condition with bounded 25 ms polling instead of the harness's 300 ms interval. Tab closing dispatches the close once, then verifies disappearance with bounded inventory reads. Neither path retries a mutation.
|
||||
|
||||
The opt-in `performance-benchmark.ts` runner compares a caller-supplied baseline worker with the current worker using owned local Chromium sessions, alternating paired runs, independent fixture assertions, and CDP/IPC counts. It makes no provider calls. See repository `docs/fast-browser-performance.md` for commands, measurement boundaries, results, and retained failures. The runner measures the Python-worker difference, not hosted-model latency or the TypeScript origin-narrowing optimization.
|
||||
|
||||
## Safety limits and current limitations
|
||||
|
||||
The compatibility run path accepts only observed, revalidated candidates and typed click/fill requests. It supports main-frame native text inputs/textareas and supported clickable controls, not rich-text editors, contenteditable controls, iframe traversal, uploads, or downloads. Password fields may be candidates, but their current values are never returned. Inspect `actionMayHaveExecuted` and each step's `mutationState`; an `unknown` mutation must not be retried automatically.
|
||||
|
||||
@@ -21,7 +21,7 @@ import time
|
||||
import uuid
|
||||
import base64
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from dataclasses import dataclass, replace
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable, Protocol
|
||||
from urllib.parse import urlsplit, urlunsplit
|
||||
@@ -33,6 +33,7 @@ from urllib.parse import urlsplit, urlunsplit
|
||||
MAX_LINE_BYTES = 12 * 1024 * 1024
|
||||
MAX_OUTPUT_BYTES = 12 * 1024 * 1024
|
||||
MAX_CANDIDATES = 60
|
||||
MAX_METADATA_BATCH = 16
|
||||
MAX_OBSERVATIONS = 8
|
||||
MAX_STRING = 4_000
|
||||
MAX_NAME = 240
|
||||
@@ -53,6 +54,9 @@ NAVIGATION_TIMEOUT_SECONDS = 10.0
|
||||
# timeout returns a structured result instead of racing the parent deadline.
|
||||
POPUP_WAIT_SECONDS = 4.75
|
||||
EXPECTED_BROWSER_HARNESS_VERSION = "0.1.13"
|
||||
_INTERNAL_TARGET_PREFIXES = (
|
||||
"chrome://", "chrome-untrusted://", "devtools://", "chrome-extension://", "about:",
|
||||
)
|
||||
_TEXT_INPUT_TYPES = {"", "text", "search", "email", "url", "tel", "number", "password"}
|
||||
|
||||
_FILL_ROLES = {"textbox", "searchbox", "spinbutton"}
|
||||
@@ -345,6 +349,50 @@ class BrowserWorker:
|
||||
observation_id = f"obs-{self._counter}-{secrets.token_hex(4)}"
|
||||
records: list[CandidateRecord] = []
|
||||
seen: set[int] = set()
|
||||
batch_metadata = getattr(self.backend, "metadata_many", None)
|
||||
pending: list[tuple[int, str, str]] = []
|
||||
|
||||
def consume(items: list[tuple[int, str, str]]) -> None:
|
||||
if not items:
|
||||
return
|
||||
metadata_by_id: dict[int, ElementMetadata] = {}
|
||||
if callable(batch_metadata):
|
||||
try:
|
||||
result = batch_metadata([backend_id for backend_id, _role, _name in items])
|
||||
except Exception:
|
||||
result = None
|
||||
if isinstance(result, dict):
|
||||
for backend_id, meta in result.items():
|
||||
if type(backend_id) is int and isinstance(meta, ElementMetadata):
|
||||
metadata_by_id[backend_id] = meta
|
||||
for backend_id, ax_role, ax_name in items:
|
||||
meta = metadata_by_id.get(backend_id)
|
||||
if meta is None or meta.backend_id != backend_id:
|
||||
try:
|
||||
meta = self.backend.metadata(backend_id)
|
||||
except Exception:
|
||||
continue
|
||||
if not meta.connected or meta.backend_id != backend_id:
|
||||
continue
|
||||
# Full AX discovery is authoritative for observation roles and
|
||||
# click names. DOM metadata remains the only source for fill
|
||||
# labels, so a field's current value cannot enter an observation.
|
||||
observed_meta = replace(meta, ax_role=ax_role, ax_name=ax_name)
|
||||
candidate_role = (ax_role or observed_meta.role).lower()
|
||||
actions = _candidate_actions(observed_meta, candidate_role)
|
||||
if not actions:
|
||||
continue
|
||||
name = _safe_name(
|
||||
observed_meta.name if "fill" in actions
|
||||
else (ax_name or observed_meta.ax_name or observed_meta.name)
|
||||
)
|
||||
candidate_id = f"c-{len(records)}"
|
||||
records.append(CandidateRecord(
|
||||
candidate_id, backend_id, candidate_role, name, actions, observed_meta
|
||||
))
|
||||
if len(records) >= MAX_CANDIDATES:
|
||||
break
|
||||
|
||||
for node in self.backend.accessibility_nodes():
|
||||
if len(records) >= MAX_CANDIDATES:
|
||||
break
|
||||
@@ -359,25 +407,11 @@ class BrowserWorker:
|
||||
# Do not spend several CDP round trips on static text/layout nodes.
|
||||
if ax_role not in _FILL_ROLES | _CLICK_ROLES:
|
||||
continue
|
||||
try:
|
||||
meta = self.backend.metadata(backend_id)
|
||||
except Exception:
|
||||
continue
|
||||
if not meta.connected or meta.backend_id != backend_id:
|
||||
continue
|
||||
candidate_role = (ax_role or meta.ax_role or meta.role).lower()
|
||||
actions = _candidate_actions(meta, candidate_role)
|
||||
if not actions:
|
||||
continue
|
||||
# For fields prefer an author-supplied label/placeholder returned by
|
||||
# metadata. AX names for a textbox must not accidentally echo its
|
||||
# current value. The real backend's metadata.name follows this rule.
|
||||
# A field's AX name can include state/value text on some custom
|
||||
# controls. Never use it as a fallback for fill candidates.
|
||||
name = _safe_name(meta.name if "fill" in actions else (meta.ax_name or meta.name or ax_name))
|
||||
candidate_id = f"c-{len(records)}"
|
||||
record = CandidateRecord(candidate_id, backend_id, candidate_role, name, actions, meta)
|
||||
records.append(record)
|
||||
pending.append((backend_id, ax_role, ax_name))
|
||||
if len(pending) >= MAX_METADATA_BATCH:
|
||||
consume(pending)
|
||||
pending.clear()
|
||||
consume(pending)
|
||||
snapshot = ObservationRecord(
|
||||
observation_id, tab["target_id"], frame_id, loader_id, tab["url"], tab["title"], tuple(records)
|
||||
)
|
||||
@@ -423,7 +457,7 @@ class BrowserWorker:
|
||||
raise WorkerError("action is not allowed for candidate")
|
||||
current = self.backend.metadata(candidate.backend_id)
|
||||
current_role = (current.ax_role or current.role).lower()
|
||||
current_name = current.name if action == "fill" else (current.name or current.ax_name)
|
||||
current_name = current.name if action == "fill" else (current.ax_name or current.name)
|
||||
current_actions = _candidate_actions(current, current_role)
|
||||
if (
|
||||
not current.connected
|
||||
@@ -493,12 +527,12 @@ class BrowserWorker:
|
||||
self.backend.fill(candidate.backend_id, text)
|
||||
deadline = time.monotonic() + ACTION_TIMEOUT_SECONDS
|
||||
while time.monotonic() < deadline:
|
||||
self._post_action_state(snapshot, same_document=True)
|
||||
tab = self._post_action_state(snapshot, same_document=True)
|
||||
value_verified = self.backend.value_matches(candidate.backend_id, text)
|
||||
condition_verified = True
|
||||
if postcondition is not None:
|
||||
if postcondition["kind"] == "url":
|
||||
condition_verified = self.backend.current_tab().get("url") == postcondition["value"]
|
||||
condition_verified = tab.get("url") == postcondition["value"]
|
||||
else:
|
||||
condition_verified = self.backend.visible_text(postcondition["value"])
|
||||
if value_verified and condition_verified:
|
||||
@@ -544,6 +578,39 @@ class BrowserWorker:
|
||||
self.backend.close()
|
||||
|
||||
|
||||
_METADATA_FUNCTION = """function () {
|
||||
const e = this;
|
||||
const style = getComputedStyle(e);
|
||||
const type = (e.getAttribute('type') || '').toLowerCase();
|
||||
const text = node => node ? String(node.innerText || node.textContent || '').trim() : '';
|
||||
const labelledBy = (e.getAttribute('aria-labelledby') || '').split(/\\s+/)
|
||||
.map(id => id ? text(document.getElementById(id)) : '').filter(Boolean).join(' ');
|
||||
const labels = e.labels ? [...e.labels].map(text).filter(Boolean).join(' ') : '';
|
||||
const visibleControlText = ['BUTTON', 'A', 'SUMMARY'].includes(e.tagName) ? text(e) : '';
|
||||
// Never read .value: for fill candidates this is user data.
|
||||
const label = e.getAttribute('aria-label') || labelledBy || labels ||
|
||||
(visibleControlText || e.getAttribute('placeholder') ||
|
||||
e.getAttribute('name') || e.getAttribute('id') || '');
|
||||
return {
|
||||
tag: e.tagName.toLowerCase(), type, role: e.getAttribute('role') || '',
|
||||
name: String(label).slice(0, 240), connected: !!e.isConnected,
|
||||
mainFrame: window.top === window,
|
||||
contenteditable: e.isContentEditable,
|
||||
visible: !!(e.getClientRects().length && style.display !== 'none' &&
|
||||
style.visibility !== 'hidden' && style.opacity !== '0' &&
|
||||
e.getAttribute('aria-hidden') !== 'true'),
|
||||
disabled: !!(e.disabled || e.getAttribute('aria-disabled') === 'true' ||
|
||||
e.matches(':disabled'))
|
||||
};
|
||||
}"""
|
||||
_METADATA_MANY_FUNCTION = (
|
||||
"function (...nodes) {\n"
|
||||
f" const read = {_METADATA_FUNCTION};\n"
|
||||
" return nodes.map(node => node && typeof node === 'object' ? read.call(node) : null);\n"
|
||||
"}"
|
||||
)
|
||||
|
||||
|
||||
class RealBrowserBackend:
|
||||
"""Adapter over browser-harness; no CDP implementation is duplicated here."""
|
||||
|
||||
@@ -560,6 +627,7 @@ class RealBrowserBackend:
|
||||
self._daemon_ready = False
|
||||
self._process: subprocess.Popen | None = None
|
||||
self._attached_target: str | None = None
|
||||
self._select_all_modifier: int | None = None
|
||||
self._configure_environment(cdp_url, self._sandbox)
|
||||
try:
|
||||
self._load_harness()
|
||||
@@ -703,45 +771,46 @@ class RealBrowserBackend:
|
||||
def switch_tab(self, target_id: str) -> None:
|
||||
if not isinstance(target_id, str) or not target_id:
|
||||
raise WorkerError("tab identity is unavailable")
|
||||
# browser-harness 0.1.13 enables Page/DOM/Runtime/Network as part of
|
||||
# its switch_tab set_session path. Re-enabling them here duplicates
|
||||
# four CDP calls and can race the daemon's session setup.
|
||||
self._session_id = self._helpers.switch_tab(target_id, activate=False)
|
||||
self._attached_target = target_id
|
||||
# The pinned helper does not enable these domains on every attach. Do
|
||||
# so here once per target; drain_events is then a truthful source for
|
||||
# the bounded diagnostics surface.
|
||||
for method in ("Page.enable", "DOM.enable", "Runtime.enable", "Network.enable"):
|
||||
try:
|
||||
self._helpers.cdp(method)
|
||||
except Exception:
|
||||
# Diagnostics are best effort and never block a browser action.
|
||||
pass
|
||||
|
||||
def list_tabs(self) -> list[dict[str, Any]]:
|
||||
tabs = self._helpers.list_tabs(include_chrome=False)
|
||||
opener_by_target: dict[str, str] = {}
|
||||
try:
|
||||
raw = self._helpers.cdp("Target.getTargets")
|
||||
for info in raw.get("targetInfos", []) if isinstance(raw, dict) else []:
|
||||
if not isinstance(info, dict) or info.get("type") != "page":
|
||||
continue
|
||||
target = info.get("targetId")
|
||||
opener = info.get("openerId")
|
||||
if isinstance(target, str) and isinstance(opener, str) and opener:
|
||||
opener_by_target[target] = opener
|
||||
except Exception:
|
||||
pass
|
||||
# helpers.list_tabs() and this opener lookup used to issue two identical
|
||||
# Target.getTargets requests. Keep its page/internal filtering locally
|
||||
# while deriving opener metadata from the same target inventory.
|
||||
raw = self._helpers.cdp("Target.getTargets")
|
||||
target_infos = raw.get("targetInfos", []) if isinstance(raw, dict) else []
|
||||
if not isinstance(target_infos, list):
|
||||
return []
|
||||
result: list[dict[str, Any]] = []
|
||||
for tab in tabs:
|
||||
if not isinstance(tab, dict) or not isinstance(tab.get("url"), str):
|
||||
for info in target_infos:
|
||||
if not isinstance(info, dict) or info.get("type") != "page":
|
||||
continue
|
||||
target = tab.get("targetId") or tab.get("target_id")
|
||||
url = info.get("url", "")
|
||||
title = info.get("title", "")
|
||||
if not isinstance(url, str) or not isinstance(title, str):
|
||||
continue
|
||||
# Match browser-harness helpers.list_tabs(include_chrome=False),
|
||||
# including its startup-placeholder suppression.
|
||||
if title.startswith("Starting agent ") and (
|
||||
url in ("", "about:blank") or url.startswith("about:blank#")
|
||||
):
|
||||
continue
|
||||
if url.startswith(_INTERNAL_TARGET_PREFIXES):
|
||||
continue
|
||||
target = info.get("targetId")
|
||||
if not isinstance(target, str) or not target:
|
||||
continue
|
||||
opener = info.get("openerId")
|
||||
result.append({
|
||||
"targetId": target,
|
||||
"target_id": target,
|
||||
"url": tab.get("url", ""),
|
||||
"title": tab.get("title", ""),
|
||||
"openerTargetId": opener_by_target.get(target),
|
||||
"url": url,
|
||||
"title": title,
|
||||
"openerTargetId": opener if isinstance(opener, str) and opener else None,
|
||||
})
|
||||
return result
|
||||
|
||||
@@ -755,7 +824,18 @@ class RealBrowserBackend:
|
||||
return self._helpers.cdp("Page.reload")
|
||||
|
||||
def wait_ready(self, timeout: float = NAVIGATION_TIMEOUT_SECONDS) -> bool:
|
||||
return bool(self._helpers.wait_for_load(timeout=timeout))
|
||||
# Keep the harness' exact readyState query, but avoid its coarse 300ms
|
||||
# sleep. This is only a readiness read; it never dispatches navigation.
|
||||
if timeout <= 0:
|
||||
return False
|
||||
deadline = time.monotonic() + timeout
|
||||
while True:
|
||||
if self._helpers.js("document.readyState") == "complete":
|
||||
return True
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
return False
|
||||
time.sleep(min(0.025, remaining))
|
||||
|
||||
def visible_text_content(self) -> str:
|
||||
# This is a fixed, worker-owned read operation. No caller expression,
|
||||
@@ -870,34 +950,20 @@ class RealBrowserBackend:
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def metadata(self, backend_id: int) -> ElementMetadata:
|
||||
value = self._call(backend_id, """function () {
|
||||
const e = this;
|
||||
const style = getComputedStyle(e);
|
||||
const type = (e.getAttribute('type') || '').toLowerCase();
|
||||
const text = node => node ? String(node.innerText || node.textContent || '').trim() : '';
|
||||
const labelledBy = (e.getAttribute('aria-labelledby') || '').split(/\\s+/)
|
||||
.map(id => id ? text(document.getElementById(id)) : '').filter(Boolean).join(' ');
|
||||
const labels = e.labels ? [...e.labels].map(text).filter(Boolean).join(' ') : '';
|
||||
const visibleControlText = ['BUTTON', 'A', 'SUMMARY'].includes(e.tagName) ? text(e) : '';
|
||||
// Never read .value: for fill candidates this is user data.
|
||||
const label = e.getAttribute('aria-label') || labelledBy || labels ||
|
||||
(visibleControlText || e.getAttribute('placeholder') ||
|
||||
e.getAttribute('name') || e.getAttribute('id') || '');
|
||||
return {
|
||||
tag: e.tagName.toLowerCase(), type, role: e.getAttribute('role') || '',
|
||||
name: String(label).slice(0, 240), connected: !!e.isConnected,
|
||||
mainFrame: window.top === window,
|
||||
contenteditable: e.isContentEditable,
|
||||
visible: !!(e.getClientRects().length && style.display !== 'none' &&
|
||||
style.visibility !== 'hidden' && style.opacity !== '0' &&
|
||||
e.getAttribute('aria-hidden') !== 'true'),
|
||||
disabled: !!(e.disabled || e.getAttribute('aria-disabled') === 'true' ||
|
||||
e.matches(':disabled'))
|
||||
};
|
||||
}""")
|
||||
@staticmethod
|
||||
def _metadata_from_dom_value(backend_id: int, value: Any) -> ElementMetadata:
|
||||
if not isinstance(value, dict):
|
||||
raise WorkerError("element metadata unavailable")
|
||||
return ElementMetadata(
|
||||
backend_id, str(value.get("tag", "")), str(value.get("type", "")),
|
||||
str(value.get("role", "")), " ".join(str(value.get("name", "")).split())[:MAX_NAME],
|
||||
bool(value.get("visible")), bool(value.get("disabled")), bool(value.get("connected")),
|
||||
bool(value.get("mainFrame", True)), bool(value.get("contenteditable", False)),
|
||||
)
|
||||
|
||||
def metadata(self, backend_id: int) -> ElementMetadata:
|
||||
value = self._call(backend_id, _METADATA_FUNCTION)
|
||||
metadata = self._metadata_from_dom_value(backend_id, value)
|
||||
ax_role, ax_name = "", ""
|
||||
try:
|
||||
partial = self._helpers.cdp(
|
||||
@@ -914,13 +980,66 @@ class RealBrowserBackend:
|
||||
# DOM metadata still provides safe labels; accessibility is a
|
||||
# refinement and not a reason to expose a broad CDP surface.
|
||||
pass
|
||||
return ElementMetadata(
|
||||
backend_id, str(value.get("tag", "")), str(value.get("type", "")),
|
||||
str(value.get("role", "")), " ".join(str(value.get("name", "")).split())[:MAX_NAME],
|
||||
bool(value.get("visible")), bool(value.get("disabled")), bool(value.get("connected")),
|
||||
bool(value.get("mainFrame", True)), bool(value.get("contenteditable", False)),
|
||||
ax_role, ax_name,
|
||||
)
|
||||
return replace(metadata, ax_role=ax_role, ax_name=ax_name)
|
||||
|
||||
def metadata_many(self, backend_ids: Iterable[int]) -> dict[int, ElementMetadata]:
|
||||
"""Read value-free DOM metadata for one bounded observation chunk.
|
||||
|
||||
The full AX tree is already the observation's source for role/name. This
|
||||
seam only batches the DOM extraction; action revalidation continues to
|
||||
use metadata(), including its fresh partial AX read.
|
||||
"""
|
||||
ids = [backend_id for backend_id in backend_ids if type(backend_id) is int and backend_id > 0][:MAX_METADATA_BATCH]
|
||||
if not ids:
|
||||
return {}
|
||||
object_group = f"fast-browser-metadata-{uuid.uuid4().hex}"
|
||||
object_ids: list[str | None] = []
|
||||
try:
|
||||
for backend_id in ids:
|
||||
try:
|
||||
resolved = self._helpers.cdp(
|
||||
"DOM.resolveNode", backendNodeId=backend_id, objectGroup=object_group
|
||||
)
|
||||
except Exception:
|
||||
object_ids.append(None)
|
||||
continue
|
||||
object_id = (resolved.get("object") or {}).get("objectId") if isinstance(resolved, dict) else None
|
||||
object_ids.append(object_id if isinstance(object_id, str) else None)
|
||||
|
||||
target_object = next((object_id for object_id in object_ids if object_id is not None), None)
|
||||
if target_object is None:
|
||||
return {}
|
||||
arguments = [
|
||||
{"objectId": object_id} if object_id is not None else {"value": None}
|
||||
for object_id in object_ids
|
||||
]
|
||||
try:
|
||||
response = self._helpers.cdp(
|
||||
"Runtime.callFunctionOn", objectId=target_object,
|
||||
functionDeclaration=_METADATA_MANY_FUNCTION, arguments=arguments,
|
||||
returnByValue=True, awaitPromise=False, objectGroup=object_group,
|
||||
)
|
||||
except Exception:
|
||||
return {}
|
||||
details = response.get("exceptionDetails") if isinstance(response, dict) else None
|
||||
remote = response.get("result") if isinstance(response, dict) else None
|
||||
values = remote.get("value") if isinstance(remote, dict) else None
|
||||
if details or not isinstance(remote, dict) or remote.get("subtype") == "error" or not isinstance(values, list):
|
||||
return {}
|
||||
result: dict[int, ElementMetadata] = {}
|
||||
for backend_id, value in zip(ids, values):
|
||||
try:
|
||||
result[backend_id] = self._metadata_from_dom_value(backend_id, value)
|
||||
except WorkerError:
|
||||
# Missing or malformed entries use the safe legacy path in
|
||||
# BrowserWorker.observe rather than being treated as absent.
|
||||
continue
|
||||
return result
|
||||
finally:
|
||||
try:
|
||||
self._helpers.cdp("Runtime.releaseObjectGroup", objectGroup=object_group)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def click(self, backend_id: int) -> None:
|
||||
box = self._helpers.cdp("DOM.getBoxModel", backendNodeId=backend_id)
|
||||
@@ -948,9 +1067,12 @@ class RealBrowserBackend:
|
||||
raise WorkerError("candidate could not be focused")
|
||||
# Select all without an arbitrary page script. Input.insertText is a
|
||||
# single bounded browser operation and is substantially faster than
|
||||
# dispatching up to 8,000 individual key events.
|
||||
user_agent = self._helpers.cdp("Browser.getVersion").get("userAgent", "")
|
||||
modifiers = 4 if "Mac OS X" in user_agent or "Macintosh" in user_agent else 2
|
||||
# dispatching up to 8,000 individual key events. Browser.getVersion is
|
||||
# stable for this backend, so cache the platform modifier for its life.
|
||||
if getattr(self, "_select_all_modifier", None) is None:
|
||||
user_agent = self._helpers.cdp("Browser.getVersion").get("userAgent", "")
|
||||
self._select_all_modifier = 4 if "Mac OS X" in user_agent or "Macintosh" in user_agent else 2
|
||||
modifiers = self._select_all_modifier
|
||||
select_all = {"key": "a", "code": "KeyA", "modifiers": modifiers,
|
||||
"windowsVirtualKeyCode": 65, "nativeVirtualKeyCode": 65,
|
||||
"commands": ["SelectAll"]}
|
||||
@@ -1597,13 +1719,20 @@ class BrowserRuntimeWorker(BrowserWorker):
|
||||
raise self._fail("closeTab", "aborted", "close", "tab close was cancelled",
|
||||
mutation_state="unknown" if self._mutation_dispatched else "not_attempted")
|
||||
# Do not assume a close acknowledgement means the target disappeared;
|
||||
# verify the current target inventory before reporting a verified close.
|
||||
try:
|
||||
self._refresh_tabs()
|
||||
except BrowserOperationError:
|
||||
raise self._fail("closeTab", "worker_failure", "verify", "tab close could not be verified", mutation_state="unknown")
|
||||
if tab.target_id in self._target_to_tab:
|
||||
raise self._fail("closeTab", "worker_failure", "verify", "tab close could not be verified", mutation_state="unknown")
|
||||
# verify inventories until the target is gone or the existing action
|
||||
# budget expires. The close mutation itself is dispatched exactly once.
|
||||
deadline = time.monotonic() + ACTION_TIMEOUT_SECONDS
|
||||
while True:
|
||||
try:
|
||||
self._refresh_tabs()
|
||||
except BrowserOperationError:
|
||||
raise self._fail("closeTab", "worker_failure", "verify", "tab close could not be verified", mutation_state="unknown")
|
||||
if tab.target_id not in self._target_to_tab:
|
||||
break
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
raise self._fail("closeTab", "worker_failure", "verify", "tab close could not be verified", mutation_state="unknown")
|
||||
time.sleep(min(0.025, remaining))
|
||||
self._target_to_tab.pop(tab.target_id, None)
|
||||
self._tabs.pop(tab.tab_id, None)
|
||||
if self._selected_tab_id == tab.tab_id:
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<title>Fast-browser performance fixture</title>
|
||||
<style>
|
||||
body { font: 16px system-ui, sans-serif; margin: 2rem; }
|
||||
.candidate-grid { display: grid; gap: .35rem; max-width: 42rem; }
|
||||
.status { min-height: 1.5rem; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<main>
|
||||
<h1>Local form validation</h1>
|
||||
<p id="status" class="status">Fixture ready</p>
|
||||
<p id="profile-status" class="status">Profile draft loaded</p>
|
||||
<p id="popup-status" class="status">Popup idle</p>
|
||||
<p id="preview" class="status"></p>
|
||||
<form id="form" novalidate></form>
|
||||
</main>
|
||||
<script>
|
||||
(() => {
|
||||
const params = new URLSearchParams(location.search);
|
||||
const mode = params.get('mode') || 'density';
|
||||
const count = Math.max(0, Math.min(60, Number(params.get('count') || 0)));
|
||||
const form = document.querySelector('#form');
|
||||
const status = document.querySelector('#status');
|
||||
const profileStatus = document.querySelector('#profile-status');
|
||||
const popupStatus = document.querySelector('#popup-status');
|
||||
const preview = document.querySelector('#preview');
|
||||
const label = (text, input) => {
|
||||
const wrapper = document.createElement('div');
|
||||
const caption = document.createElement('label');
|
||||
caption.textContent = text;
|
||||
caption.htmlFor = input.id;
|
||||
wrapper.append(caption, input);
|
||||
return wrapper;
|
||||
};
|
||||
if (mode === 'popup') {
|
||||
status.textContent = 'Popup ready';
|
||||
return;
|
||||
}
|
||||
if (mode === 'density') {
|
||||
const grid = document.createElement('div');
|
||||
grid.className = 'candidate-grid';
|
||||
for (let index = 0; index < count; index += 1) {
|
||||
const input = document.createElement('input');
|
||||
input.id = `candidate-${index}`;
|
||||
input.type = 'text';
|
||||
input.placeholder = `Search filter ${index + 1}`;
|
||||
grid.append(label(`Filter ${index + 1}`, input));
|
||||
}
|
||||
form.append(grid);
|
||||
status.textContent = `Density fixture with ${count} controls`;
|
||||
return;
|
||||
}
|
||||
const display = document.createElement('input');
|
||||
display.id = 'display-name';
|
||||
display.type = 'text';
|
||||
display.value = 'existing value';
|
||||
form.append(label('Display name', display));
|
||||
|
||||
const save = document.createElement('button');
|
||||
save.type = 'button';
|
||||
save.textContent = 'Save profile';
|
||||
save.addEventListener('click', () => {
|
||||
profileStatus.textContent = display.value === 'Ada Lovelace' ? 'Profile saved' : 'Profile validation failed';
|
||||
});
|
||||
form.append(save);
|
||||
|
||||
const previewButton = document.createElement('button');
|
||||
previewButton.type = 'button';
|
||||
previewButton.textContent = 'Preview profile';
|
||||
previewButton.addEventListener('click', () => {
|
||||
preview.textContent = `Preview ready: ${display.value}`;
|
||||
});
|
||||
form.append(previewButton);
|
||||
|
||||
const immediate = document.createElement('button');
|
||||
immediate.type = 'button';
|
||||
immediate.textContent = 'Immediate action';
|
||||
immediate.addEventListener('click', () => { status.textContent = 'Immediate complete'; });
|
||||
form.append(immediate);
|
||||
|
||||
const delayed = document.createElement('button');
|
||||
delayed.type = 'button';
|
||||
delayed.textContent = 'Delayed action';
|
||||
delayed.addEventListener('click', () => setTimeout(() => { status.textContent = 'Delayed complete'; }, 40));
|
||||
form.append(delayed);
|
||||
|
||||
const popupButton = document.createElement('button');
|
||||
popupButton.type = 'button';
|
||||
popupButton.textContent = 'Open popup';
|
||||
popupButton.addEventListener('click', () => {
|
||||
popupStatus.textContent = 'Popup requested';
|
||||
window.open(`${location.origin}${location.pathname}?mode=popup`, '_blank', 'popup,width=420,height=240');
|
||||
});
|
||||
form.append(popupButton);
|
||||
for (let index = 0; index < 10; index += 1) {
|
||||
const input = document.createElement('input');
|
||||
input.id = `extra-${index}`;
|
||||
input.type = 'text';
|
||||
input.placeholder = `Extra field ${index + 1}`;
|
||||
form.append(label(`Extra field ${index + 1}`, input));
|
||||
}
|
||||
})();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@jbcode/pi-fast-browser",
|
||||
"version": "0.1.0",
|
||||
"version": "0.1.1",
|
||||
"description": "Experimental bounded local-browser Pi tools using browser-harness and Typesafe JEV.",
|
||||
"type": "module",
|
||||
"license": "MIT",
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
import assert from "node:assert/strict";
|
||||
import test from "node:test";
|
||||
import { mkdtemp, writeFile, rm } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import {
|
||||
pairedArmOrder,
|
||||
safeFailure,
|
||||
summarizeBenchmarkAttempts,
|
||||
timingDelta,
|
||||
validWorkerCounters,
|
||||
summarizeTimings,
|
||||
validateBenchmarkOptions,
|
||||
type BenchmarkAttempt,
|
||||
} from "./performance-benchmark.ts";
|
||||
|
||||
async function files(): Promise<{ root: string; paths: string[] }> {
|
||||
const root = await mkdtemp(join("/tmp", "fast-browser-performance-test-"));
|
||||
const paths = await Promise.all(["baseline.py", "optimized.py", "python", "chromium"].map(async (name) => {
|
||||
const path = join(root, name);
|
||||
await writeFile(path, "# test\n", { mode: 0o700 });
|
||||
return path;
|
||||
}));
|
||||
return { root, paths };
|
||||
}
|
||||
|
||||
test("timing aggregation reports bounded p50 and p95 without wall-clock assumptions", () => {
|
||||
assert.deepEqual(summarizeTimings([10, 20, 30, 40]), { count: 4, p50: 25, p95: 38.5 });
|
||||
assert.deepEqual(summarizeTimings([]), { count: 0, p50: null, p95: null });
|
||||
});
|
||||
|
||||
test("transport timer deltas preserve bounded operation names and failed costs", () => {
|
||||
assert.deepEqual(timingDelta({ observe: 12, closeTab: 4 }, { observe: 19, closeTab: 4, waitPopup: 8 }), { observe: 7, waitPopup: 8 });
|
||||
assert.deepEqual(timingDelta({ observe: 10 }, { observe: 9 }), {});
|
||||
});
|
||||
|
||||
test("paired order alternates AB and BA while retaining failed attempts", () => {
|
||||
assert.deepEqual(pairedArmOrder(1), ["baseline", "optimized"]);
|
||||
assert.deepEqual(pairedArmOrder(2), ["optimized", "baseline"]);
|
||||
});
|
||||
|
||||
test("attempt aggregation excludes failures from timing summaries but retains them for accounting", () => {
|
||||
const attempts: BenchmarkAttempt[] = [
|
||||
{ arm: "baseline", phase: "warm", sample: 1, status: "passed", elapsedMs: 10, startupMs: null },
|
||||
{ arm: "baseline", phase: "warm", sample: 2, status: "failed", elapsedMs: null, startupMs: null, error: "worker failed" },
|
||||
{ arm: "baseline", phase: "warm", sample: 3, status: "passed", elapsedMs: 30, startupMs: null },
|
||||
];
|
||||
assert.deepEqual(summarizeBenchmarkAttempts(attempts, "baseline", "warm"), { count: 2, p50: 20, p95: 29 });
|
||||
assert.equal(attempts.filter((attempt) => attempt.status === "failed").length, 1);
|
||||
});
|
||||
|
||||
test("benchmark input bounds reject unbounded samples and candidate sets", async () => {
|
||||
const { root, paths } = await files();
|
||||
try {
|
||||
assert.throws(() => validateBenchmarkOptions({ baselineWorkerPath: paths[0]!, optimizedWorkerPath: paths[1]!, pythonPath: paths[2]!, browserPath: paths[3]!, samples: 41 }), /samples/);
|
||||
assert.throws(() => validateBenchmarkOptions({ baselineWorkerPath: paths[0]!, optimizedWorkerPath: paths[1]!, pythonPath: paths[2]!, browserPath: paths[3]!, candidateCounts: [1] }), /candidate counts/);
|
||||
const valid = validateBenchmarkOptions({ baselineWorkerPath: paths[0]!, optimizedWorkerPath: paths[1]!, pythonPath: paths[2]!, browserPath: paths[3]!, samples: 1, coldSamples: 1, candidateCounts: [0, 10, 60] });
|
||||
assert.equal(valid.samples, 1);
|
||||
} finally {
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test("missing or malformed worker counters are unavailable", () => {
|
||||
assert.equal(validWorkerCounters({ cdp: { "Page.enable": 1 }, ipc: { _send: 2 } }), true);
|
||||
assert.equal(validWorkerCounters({ cdp: { "Page.enable": -1 }, ipc: { _send: 2 } }), false);
|
||||
assert.equal(validWorkerCounters({ cdp: { "Page.enable": 1 }, ipc: { "bad key": 2 } }), false);
|
||||
assert.equal(validWorkerCounters(undefined), false);
|
||||
});
|
||||
|
||||
test("report failures use fixed safe categories and never include raw error text", () => {
|
||||
const safe = safeFailure("popup-close", "unverified");
|
||||
assert.equal(safe, "popup-close:unverified");
|
||||
assert.doesNotMatch(safe, /https?:|secret|Ada|tmp|performance/);
|
||||
});
|
||||
@@ -0,0 +1,634 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { spawn as nodeSpawn, spawnSync } from "node:child_process";
|
||||
import { existsSync, readFileSync, statSync } from "node:fs";
|
||||
import { chmod, mkdir, mkdtemp, readFile, rm } from "node:fs/promises";
|
||||
import { createServer, type IncomingMessage, type ServerResponse } from "node:http";
|
||||
import { tmpdir } from "node:os";
|
||||
import { dirname, join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { createBrowserTransport, type ChildProcessLike, type SpawnLike } from "./bridge.ts";
|
||||
import { createBrowserSessionManager, type BrowserSession, type BrowserSessionManager } from "./browser-session.ts";
|
||||
import type { BrowserOperation, BrowserRuntime, BrowserTransport, BrowserTransportFactory, BrowserTransportRequest, BrowserTransportResultFor, Candidate, PrivateArtifactStore } from "./contracts.ts";
|
||||
import { createBrowserRuntimeFactory } from "./runtime.ts";
|
||||
|
||||
const ROOT = dirname(fileURLToPath(import.meta.url));
|
||||
const CURRENT_WORKER = join(ROOT, "browser_worker.py");
|
||||
const PROBE = join(ROOT, "performance_probe.py");
|
||||
const FIXTURE = join(ROOT, "fixtures", "performance.html");
|
||||
const DEFAULT_PYTHON = join(process.cwd(), ".pi", "fast-browser-demo", "venv", "bin", "python");
|
||||
const DEFAULT_BROWSER = "/usr/bin/chromium";
|
||||
const DEFAULT_COUNTS = [0, 10, 60] as const;
|
||||
const DEFAULT_SAMPLES = 20;
|
||||
const DEFAULT_COLD_SAMPLES = 1;
|
||||
const MAX_SAMPLES = 40;
|
||||
const MAX_COLD_SAMPLES = 5;
|
||||
const MAX_COUNTS = 3;
|
||||
|
||||
type Arm = "baseline" | "optimized";
|
||||
type Phase = "warm" | "cold-start";
|
||||
export type TimedOperation = "observe" | "act" | "read" | "assert" | "navigate" | "reload" | "listTabs" | "selectTab" | "waitPopup" | "closeTab" | "captureEvidence" | "narrowOriginPolicy" | "close";
|
||||
export type OperationTimings = Partial<Record<TimedOperation, number>>;
|
||||
|
||||
export interface PerformanceBenchmarkOptions {
|
||||
baselineWorkerPath: string;
|
||||
optimizedWorkerPath?: string;
|
||||
pythonPath?: string;
|
||||
browserPath?: string;
|
||||
samples?: number;
|
||||
coldSamples?: number;
|
||||
candidateCounts?: readonly number[];
|
||||
}
|
||||
|
||||
export interface TimingSummary {
|
||||
count: number;
|
||||
p50: number | null;
|
||||
p95: number | null;
|
||||
}
|
||||
|
||||
export interface BenchmarkAttempt {
|
||||
arm: Arm;
|
||||
phase: Phase;
|
||||
sample: number;
|
||||
order?: 1 | 2;
|
||||
status: "passed" | "failed";
|
||||
elapsedMs: number | null;
|
||||
startupMs: number | null;
|
||||
error?: string;
|
||||
protocolOperations?: Record<string, number>;
|
||||
densityObserveMs?: Partial<Record<"0" | "10" | "60", number>>;
|
||||
transportMs?: OperationTimings;
|
||||
}
|
||||
|
||||
export interface WorkerMethodCounts {
|
||||
cdp: Record<string, number>;
|
||||
ipc: Record<string, number>;
|
||||
cdpCalls: number;
|
||||
ipcCalls: number;
|
||||
}
|
||||
|
||||
export interface BenchmarkReport {
|
||||
ok: boolean;
|
||||
measurement: { workerMethods: "selected Python worker CDP/IPC helper methods"; protocolOperations: "TypeScript JSONL operation names"; warmPriming: "one density=0 observe per arm, excluded from warm samples"; coldStartup: "browser session open plus first worker observe"; runtimeNarrowingMeasured: false };
|
||||
samples: number;
|
||||
coldSamples: number;
|
||||
candidateCounts: readonly number[];
|
||||
attempts: readonly BenchmarkAttempt[];
|
||||
warm: Record<Arm, TimingSummary>;
|
||||
coldStartup: Record<Arm, TimingSummary>;
|
||||
workerMethods: Partial<Record<Arm, WorkerMethodCounts>>;
|
||||
protocolOperations: Partial<Record<Arm, Record<string, number>>>;
|
||||
densityObserve: Record<Arm, Record<"0" | "10" | "60", TimingSummary>>;
|
||||
pairedOrder: readonly (readonly [Arm, Arm])[];
|
||||
counterStatus: "complete" | "unavailable";
|
||||
cleanup: { status: "passed" | "failed"; error?: string; retainedRoot: boolean };
|
||||
}
|
||||
|
||||
interface FixtureServer {
|
||||
origin: string;
|
||||
close(): Promise<void>;
|
||||
}
|
||||
|
||||
interface ArmHandle {
|
||||
arm: Arm;
|
||||
session: BrowserSession;
|
||||
runtime: BrowserRuntime;
|
||||
protocol: Record<string, number>;
|
||||
transportMs: OperationTimings;
|
||||
countFile: string;
|
||||
closed: boolean;
|
||||
}
|
||||
|
||||
const EMPTY_ARTIFACT_STORE: PrivateArtifactStore = {
|
||||
async open() {
|
||||
throw new Error("benchmark evidence is disabled");
|
||||
},
|
||||
async close() { /* no-op */ },
|
||||
};
|
||||
|
||||
type FailureStage = "cli" | "cold-start" | "warm-setup" | "warm-flow" | "density-observe" | "popup-close" | "counter" | "cleanup";
|
||||
type FailureCategory = "failed" | "unverified" | "unavailable";
|
||||
|
||||
export function safeFailure(stage: FailureStage, category: FailureCategory = "failed"): string {
|
||||
return `${stage}:${category}`;
|
||||
}
|
||||
|
||||
class BenchmarkFailure extends Error {
|
||||
readonly stage: FailureStage;
|
||||
readonly category: FailureCategory;
|
||||
|
||||
constructor(stage: FailureStage, category: FailureCategory) {
|
||||
super(`${stage}:${category}`);
|
||||
this.stage = stage;
|
||||
this.category = category;
|
||||
}
|
||||
}
|
||||
|
||||
function failureFor(error: unknown, fallback: FailureStage): string {
|
||||
return error instanceof BenchmarkFailure ? safeFailure(error.stage, error.category) : safeFailure(fallback);
|
||||
}
|
||||
|
||||
function boundedInteger(value: unknown, label: string, min: number, max: number): number {
|
||||
if (!Number.isSafeInteger(value) || (value as number) < min || (value as number) > max) {
|
||||
throw new Error(`${label} must be an integer between ${min} and ${max}`);
|
||||
}
|
||||
return value as number;
|
||||
}
|
||||
|
||||
function requiredFile(value: string, label: string): string {
|
||||
if (typeof value !== "string" || value.length === 0 || value.length > 4_000 || value.includes("\0")) {
|
||||
throw new Error(`${label} is invalid`);
|
||||
}
|
||||
if (!existsSync(value) || !statSync(value).isFile()) throw new Error(`${label} is not a file`);
|
||||
return value;
|
||||
}
|
||||
|
||||
export function validateBenchmarkOptions(options: PerformanceBenchmarkOptions): Required<PerformanceBenchmarkOptions> {
|
||||
if (!options || typeof options !== "object") throw new Error("benchmark options are required");
|
||||
const baselineWorkerPath = requiredFile(options.baselineWorkerPath, "baseline worker path");
|
||||
const optimizedWorkerPath = requiredFile(options.optimizedWorkerPath ?? CURRENT_WORKER, "optimized worker path");
|
||||
const pythonPath = requiredFile(options.pythonPath ?? process.env.FAST_BROWSER_PYTHON ?? DEFAULT_PYTHON, "Python path");
|
||||
const browserPath = requiredFile(options.browserPath ?? DEFAULT_BROWSER, "browser path");
|
||||
const samples = boundedInteger(options.samples ?? DEFAULT_SAMPLES, "samples", 1, MAX_SAMPLES);
|
||||
const coldSamples = boundedInteger(options.coldSamples ?? DEFAULT_COLD_SAMPLES, "cold samples", 1, MAX_COLD_SAMPLES);
|
||||
const candidateCounts = [...(options.candidateCounts ?? DEFAULT_COUNTS)];
|
||||
if (candidateCounts.length < 1 || candidateCounts.length > MAX_COUNTS || candidateCounts.some((value) => ![0, 10, 60].includes(value))) {
|
||||
throw new Error("candidate counts must be a non-empty subset of 0, 10, and 60");
|
||||
}
|
||||
return { baselineWorkerPath, optimizedWorkerPath, pythonPath, browserPath, samples, coldSamples, candidateCounts };
|
||||
}
|
||||
|
||||
function percentile(values: readonly number[], fraction: number): number | null {
|
||||
if (values.length === 0) return null;
|
||||
const sorted = [...values].sort((a, b) => a - b);
|
||||
const position = (sorted.length - 1) * fraction;
|
||||
const lower = Math.floor(position);
|
||||
const upper = Math.ceil(position);
|
||||
if (lower === upper) return sorted[lower]!;
|
||||
return sorted[lower]! + (sorted[upper]! - sorted[lower]!) * (position - lower);
|
||||
}
|
||||
|
||||
export function summarizeTimings(values: readonly number[]): TimingSummary {
|
||||
return { count: values.length, p50: percentile(values, 0.5), p95: percentile(values, 0.95) };
|
||||
}
|
||||
|
||||
function increment(target: Record<string, number>, key: string): void {
|
||||
target[key] = (target[key] ?? 0) + 1;
|
||||
}
|
||||
|
||||
function countProtocol(child: ReturnType<typeof nodeSpawn>, target: Record<string, number>): void {
|
||||
const stdin = child.stdin;
|
||||
if (!stdin) throw new Error("benchmark worker stdin is unavailable");
|
||||
const write = stdin.write.bind(stdin);
|
||||
stdin.write = ((data: string | Uint8Array, ...args: unknown[]) => {
|
||||
const text = typeof data === "string" ? data : Buffer.from(data).toString("utf8");
|
||||
for (const line of text.split("\n")) {
|
||||
if (!line) continue;
|
||||
try {
|
||||
const wire = JSON.parse(line) as { operation?: { op?: unknown } };
|
||||
const op = wire.operation?.op;
|
||||
if (typeof op === "string" && /^[A-Za-z][A-Za-z0-9]{0,40}$/.test(op)) increment(target, op);
|
||||
} catch { /* bridge owns malformed-request handling */ }
|
||||
}
|
||||
return write(data, ...(args as []));
|
||||
}) as typeof stdin.write;
|
||||
}
|
||||
|
||||
function instrumentedSpawn(
|
||||
workerPath: string,
|
||||
countFile: string,
|
||||
protocol: Record<string, number>,
|
||||
): SpawnLike {
|
||||
return (command, args, options) => {
|
||||
const child = nodeSpawn(command, [
|
||||
...args,
|
||||
"--performance-worker", workerPath,
|
||||
"--performance-count-file", countFile,
|
||||
], options as never);
|
||||
countProtocol(child, protocol);
|
||||
return child as unknown as ChildProcessLike;
|
||||
};
|
||||
}
|
||||
|
||||
function timedTransport(base: BrowserTransport, timings: OperationTimings): BrowserTransport {
|
||||
return {
|
||||
async dispatch<Operation extends BrowserOperation>(request: BrowserTransportRequest<Operation>, signal: AbortSignal): Promise<BrowserTransportResultFor<Operation>> {
|
||||
const operation = request.operation.op as TimedOperation;
|
||||
const started = performance.now();
|
||||
try {
|
||||
return await base.dispatch(request, signal);
|
||||
} finally {
|
||||
timings[operation] = (timings[operation] ?? 0) + performance.now() - started;
|
||||
}
|
||||
},
|
||||
close: () => base.close(),
|
||||
};
|
||||
}
|
||||
|
||||
function createInstrumentedRuntime(
|
||||
armWorkerPath: string,
|
||||
countFile: string,
|
||||
protocol: Record<string, number>,
|
||||
transportMs: OperationTimings,
|
||||
session: BrowserSession,
|
||||
origin: string,
|
||||
pythonPath: string,
|
||||
signal: AbortSignal,
|
||||
): BrowserRuntime {
|
||||
const transportFactory: BrowserTransportFactory = {
|
||||
create(options, parentSignal) {
|
||||
const transport = createBrowserTransport({
|
||||
...options,
|
||||
workerPath: PROBE,
|
||||
spawn: instrumentedSpawn(armWorkerPath, countFile, protocol),
|
||||
}, parentSignal);
|
||||
return timedTransport(transport, transportMs);
|
||||
},
|
||||
};
|
||||
return createBrowserRuntimeFactory(transportFactory).create({
|
||||
sessionId: session.id,
|
||||
cdpUrl: session.config.cdpUrl,
|
||||
allowedOrigins: [origin],
|
||||
python: pythonPath,
|
||||
artifacts: EMPTY_ARTIFACT_STORE,
|
||||
}, signal);
|
||||
}
|
||||
|
||||
async function createFixtureServer(): Promise<FixtureServer> {
|
||||
const template = readFileSync(FIXTURE, "utf8");
|
||||
const server = createServer((request: IncomingMessage, response: ServerResponse) => {
|
||||
const path = new URL(request.url ?? "/", "http://127.0.0.1").pathname;
|
||||
if (path !== "/performance.html") {
|
||||
response.writeHead(404);
|
||||
response.end();
|
||||
return;
|
||||
}
|
||||
response.writeHead(200, { "content-type": "text/html; charset=utf-8", "cache-control": "no-store" });
|
||||
response.end(template);
|
||||
});
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
server.once("error", reject);
|
||||
server.listen(0, "127.0.0.1", resolve);
|
||||
});
|
||||
const address = server.address();
|
||||
if (!address || typeof address === "string") throw new Error("performance fixture did not expose a port");
|
||||
return {
|
||||
origin: `http://127.0.0.1:${address.port}`,
|
||||
async close() {
|
||||
server.closeAllConnections?.();
|
||||
await new Promise<void>((resolve) => server.close(() => resolve()));
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function launchConfig(origin: string, url: string, browserPath: string): Parameters<BrowserSessionManager["open"]>[0] {
|
||||
return {
|
||||
allowedOrigins: [origin],
|
||||
browser: { mode: "launch", product: "chromium", executablePath: browserPath, headless: true, startUrl: url },
|
||||
};
|
||||
}
|
||||
|
||||
function candidate(observation: Awaited<ReturnType<BrowserRuntime["observe"]>>, name: string, action: "click" | "fill"): Candidate {
|
||||
const matches = observation.candidates.filter((item) => item.name === name && item.actions.includes(action));
|
||||
assert.equal(matches.length, 1, `expected one ${action} candidate`);
|
||||
return matches[0]!;
|
||||
}
|
||||
|
||||
async function runWarmFlow(handle: ArmHandle, origin: string, candidateCounts: readonly number[], signal: AbortSignal): Promise<Partial<Record<"0" | "10" | "60", number>>> {
|
||||
const densityObserveMs: Partial<Record<"0" | "10" | "60", number>> = {};
|
||||
for (const count of candidateCounts) {
|
||||
const density = await handle.runtime.navigate({ url: `${origin}/performance.html?mode=density&count=${count}` }, signal);
|
||||
assert.equal(density.status, "committed");
|
||||
const observeStarted = Date.now();
|
||||
try {
|
||||
const observation = await handle.runtime.observe(signal);
|
||||
densityObserveMs[String(count) as "0" | "10" | "60"] = Date.now() - observeStarted;
|
||||
assert.equal(observation.candidates.length, count);
|
||||
} catch {
|
||||
throw new BenchmarkFailure("density-observe", "failed");
|
||||
}
|
||||
}
|
||||
const page = await handle.runtime.navigate({ url: `${origin}/performance.html?mode=flow&count=10` }, signal);
|
||||
assert.equal(page.status, "committed");
|
||||
const tabs = await handle.runtime.listTabs(signal);
|
||||
const main = tabs.find((tab) => tab.selected);
|
||||
assert(main, "main benchmark tab is unavailable");
|
||||
const mainTab = main;
|
||||
|
||||
let observation = await handle.runtime.observe(signal);
|
||||
const field = candidate(observation, "Display name", "fill");
|
||||
const filled = await handle.runtime.act({ observationId: observation.observationId, candidateId: field.id, action: "fill", text: "Ada Lovelace" }, signal);
|
||||
assert.equal(filled.verified, true);
|
||||
|
||||
observation = await handle.runtime.observe(signal);
|
||||
const save = candidate(observation, "Save profile", "click");
|
||||
const saved = await handle.runtime.act({ observationId: observation.observationId, candidateId: save.id, action: "click", postcondition: { kind: "text", value: "Profile saved" } }, signal);
|
||||
assert.equal(saved.verified, true);
|
||||
const saveAssert = await handle.runtime.assert({ assertion: { kind: "visibleText", match: "contains", value: "Profile saved" }, wait: "bounded" }, signal);
|
||||
assert.equal(saveAssert.status, "satisfied");
|
||||
|
||||
observation = await handle.runtime.observe(signal);
|
||||
const previewButton = candidate(observation, "Preview profile", "click");
|
||||
const previewRequested = await handle.runtime.act({ observationId: observation.observationId, candidateId: previewButton.id, action: "click", postcondition: { kind: "text", value: "Preview ready" } }, signal);
|
||||
assert.equal(previewRequested.verified, true);
|
||||
|
||||
observation = await handle.runtime.observe(signal);
|
||||
const immediate = candidate(observation, "Immediate action", "click");
|
||||
const clicked = await handle.runtime.act({ observationId: observation.observationId, candidateId: immediate.id, action: "click", postcondition: { kind: "text", value: "Immediate complete" } }, signal);
|
||||
assert.equal(clicked.verified, true);
|
||||
const immediateAssert = await handle.runtime.assert({ assertion: { kind: "visibleText", match: "contains", value: "Immediate complete" }, wait: "none" }, signal);
|
||||
assert.equal(immediateAssert.status, "satisfied");
|
||||
|
||||
observation = await handle.runtime.observe(signal);
|
||||
const delayed = candidate(observation, "Delayed action", "click");
|
||||
const delayedResult = await handle.runtime.act({ observationId: observation.observationId, candidateId: delayed.id, action: "click", postcondition: { kind: "text", value: "Delayed complete" } }, signal);
|
||||
assert.equal(delayedResult.verified, true);
|
||||
const delayedAssert = await handle.runtime.assert({ assertion: { kind: "visibleText", match: "contains", value: "Delayed complete" }, wait: "bounded" }, signal);
|
||||
assert.equal(delayedAssert.status, "satisfied");
|
||||
|
||||
observation = await handle.runtime.observe(signal);
|
||||
const popupButton = candidate(observation, "Open popup", "click");
|
||||
const popupRequest = await handle.runtime.act({ observationId: observation.observationId, candidateId: popupButton.id, action: "click", postcondition: { kind: "text", value: "Popup requested" } }, signal);
|
||||
assert.equal(popupRequest.verified, true);
|
||||
const popup = await handle.runtime.waitPopup({ openerTabId: mainTab.tabId }, signal);
|
||||
assert.equal(popup.status, "found");
|
||||
assert(popup.tab);
|
||||
const popupTab = popup.tab;
|
||||
const popupRead = await handle.runtime.read({ tabId: popupTab.tabId }, signal);
|
||||
assert.match(popupRead.visibleText, /Popup ready/);
|
||||
const popupReady = await handle.runtime.assert({ tabId: popupTab.tabId, assertion: { kind: "visibleText", match: "contains", value: "Popup ready" }, wait: "bounded" }, signal);
|
||||
assert.equal(popupReady.status, "satisfied");
|
||||
try {
|
||||
await handle.runtime.closeTab({ tabId: popupTab.tabId }, signal);
|
||||
} catch {
|
||||
throw new BenchmarkFailure("popup-close", "unverified");
|
||||
}
|
||||
const preview = await handle.runtime.assert({ assertion: { kind: "visibleText", match: "contains", value: "Preview ready: Ada Lovelace" }, wait: "bounded" }, signal);
|
||||
assert.equal(preview.status, "satisfied");
|
||||
return densityObserveMs;
|
||||
}
|
||||
|
||||
function validCounterMap(value: unknown): value is Record<string, number> {
|
||||
return !!value && typeof value === "object" && !Array.isArray(value)
|
||||
&& Object.entries(value).every(([key, count]) => /^[A-Za-z0-9_.-]{1,120}$/.test(key) && Number.isSafeInteger(count) && count >= 0);
|
||||
}
|
||||
|
||||
export function validWorkerCounters(value: unknown): value is { cdp: Record<string, number>; ipc: Record<string, number> } {
|
||||
if (!value || typeof value !== "object") return false;
|
||||
const counters = value as { cdp?: unknown; ipc?: unknown };
|
||||
return validCounterMap(counters.cdp) && validCounterMap(counters.ipc);
|
||||
}
|
||||
|
||||
async function readWorkerCounts(path: string): Promise<WorkerMethodCounts | undefined> {
|
||||
try {
|
||||
const value = JSON.parse(await readFile(path, "utf8")) as { cdp?: unknown; ipc?: unknown };
|
||||
if (!validWorkerCounters(value)) return undefined;
|
||||
const cdp = value.cdp;
|
||||
const ipc = value.ipc;
|
||||
return {
|
||||
cdp,
|
||||
ipc,
|
||||
cdpCalls: Object.values(cdp).reduce((sum, count) => sum + count, 0),
|
||||
ipcCalls: Object.values(ipc).reduce((sum, count) => sum + count, 0),
|
||||
};
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
function ownedProcessLines(root: string): string[] | undefined {
|
||||
const result = spawnSync("ps", ["-eo", "pid=,args="], { encoding: "utf8", timeout: 5_000 });
|
||||
if (result.status !== 0) return undefined;
|
||||
return String(result.stdout).split(/\r?\n/).filter((line) => line.includes(root));
|
||||
}
|
||||
|
||||
function snapshotCounts(value: Record<string, number>): Record<string, number> {
|
||||
return { ...value };
|
||||
}
|
||||
|
||||
export function timingDelta(before: OperationTimings, after: OperationTimings): OperationTimings {
|
||||
const result: OperationTimings = {};
|
||||
for (const key of new Set([...Object.keys(before), ...Object.keys(after)] as TimedOperation[])) {
|
||||
const difference = (after[key] ?? 0) - (before[key] ?? 0);
|
||||
if (difference > 0) result[key] = difference;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
function countDelta(before: Record<string, number>, after: Record<string, number>): Record<string, number> {
|
||||
const result: Record<string, number> = {};
|
||||
for (const key of new Set([...Object.keys(before), ...Object.keys(after)])) {
|
||||
const difference = (after[key] ?? 0) - (before[key] ?? 0);
|
||||
if (difference > 0) result[key] = difference;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
async function closeHandle(handle: ArmHandle | undefined): Promise<string | undefined> {
|
||||
if (!handle || handle.closed) return undefined;
|
||||
handle.closed = true;
|
||||
try {
|
||||
await handle.runtime.close();
|
||||
return undefined;
|
||||
} catch {
|
||||
return safeFailure("cleanup");
|
||||
}
|
||||
}
|
||||
|
||||
export function pairedArmOrder(sample: number): readonly [Arm, Arm] {
|
||||
return sample % 2 === 1 ? ["baseline", "optimized"] : ["optimized", "baseline"];
|
||||
}
|
||||
|
||||
function attemptsFor(attempts: readonly BenchmarkAttempt[], arm: Arm, phase: Phase): number[] {
|
||||
return attempts.filter((item) => item.arm === arm && item.phase === phase && item.status === "passed" && item.elapsedMs !== null).map((item) => item.elapsedMs!);
|
||||
}
|
||||
|
||||
export function summarizeBenchmarkAttempts(attempts: readonly BenchmarkAttempt[], arm: Arm, phase: Phase): TimingSummary {
|
||||
return summarizeTimings(attemptsFor(attempts, arm, phase));
|
||||
}
|
||||
|
||||
function densitySummaries(attempts: readonly BenchmarkAttempt[]): Record<Arm, Record<"0" | "10" | "60", TimingSummary>> {
|
||||
const result = {} as Record<Arm, Record<"0" | "10" | "60", TimingSummary>>;
|
||||
for (const arm of ["baseline", "optimized"] as const) {
|
||||
result[arm] = {} as Record<"0" | "10" | "60", TimingSummary>;
|
||||
for (const count of ["0", "10", "60"] as const) {
|
||||
result[arm][count] = summarizeTimings(attempts
|
||||
.filter((attempt) => attempt.arm === arm && attempt.phase === "warm" && attempt.status === "passed" && attempt.densityObserveMs?.[count] !== undefined)
|
||||
.map((attempt) => attempt.densityObserveMs![count]!));
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
export async function runPerformanceBenchmark(options: PerformanceBenchmarkOptions): Promise<BenchmarkReport> {
|
||||
const config = validateBenchmarkOptions(options);
|
||||
const fixture = await createFixtureServer();
|
||||
const root = await mkdtemp(join(tmpdir(), "fast-browser-performance-"));
|
||||
await mkdir(join(root, "leases"), { recursive: true, mode: 0o700 });
|
||||
await chmod(join(root, "leases"), 0o700);
|
||||
const manager = createBrowserSessionManager({ profileDirectory: join(root, "profiles"), leaseDirectory: join(root, "leases"), startupTimeoutMs: 20_000, closeTimeoutMs: 4_000 });
|
||||
const controller = new AbortController();
|
||||
const attempts: BenchmarkAttempt[] = [];
|
||||
const workerMethods: Partial<Record<Arm, WorkerMethodCounts>> = {};
|
||||
const protocolOperations: Partial<Record<Arm, Record<string, number>>> = {};
|
||||
const handles: Partial<Record<Arm, ArmHandle>> = {};
|
||||
const setupErrors: Partial<Record<Arm, string>> = {};
|
||||
const pairedOrder: Array<readonly [Arm, Arm]> = [];
|
||||
let cleanupError: string | undefined;
|
||||
let retainedRoot = false;
|
||||
let counterUnavailable = false;
|
||||
try {
|
||||
const armPaths: Record<Arm, string> = { baseline: config.baselineWorkerPath, optimized: config.optimizedWorkerPath };
|
||||
for (const arm of ["baseline", "optimized"] as const) {
|
||||
for (let sample = 1; sample <= config.coldSamples; sample += 1) {
|
||||
const countFile = join(root, `${arm}-cold-${sample}.json`);
|
||||
const protocol: Record<string, number> = {};
|
||||
const transportMs: OperationTimings = {};
|
||||
let session: BrowserSession | undefined;
|
||||
let runtime: BrowserRuntime | undefined;
|
||||
const started = Date.now();
|
||||
let startupMs: number | null = null;
|
||||
try {
|
||||
const opened = Date.now();
|
||||
session = await manager.open(launchConfig(fixture.origin, `${fixture.origin}/performance.html?mode=density&count=10`, config.browserPath), controller.signal);
|
||||
const first = Date.now();
|
||||
runtime = createInstrumentedRuntime(armPaths[arm], countFile, protocol, transportMs, session, fixture.origin, config.pythonPath, controller.signal);
|
||||
const observation = await runtime.observe(controller.signal);
|
||||
assert.equal(observation.candidates.length, 10);
|
||||
startupMs = first - opened + (Date.now() - first);
|
||||
attempts.push({ arm, phase: "cold-start", sample, status: "passed", elapsedMs: Date.now() - started, startupMs, protocolOperations: { ...protocol }, transportMs: { ...transportMs } });
|
||||
} catch (error) {
|
||||
attempts.push({ arm, phase: "cold-start", sample, status: "failed", elapsedMs: Date.now() - started, startupMs, error: safeFailure("cold-start"), protocolOperations: { ...protocol }, transportMs: { ...transportMs } });
|
||||
} finally {
|
||||
try { await runtime?.close(); } catch { cleanupError ??= safeFailure("cleanup"); }
|
||||
if (session) await manager.close(session.id).catch(() => { cleanupError ??= safeFailure("cleanup"); });
|
||||
const counts = await readWorkerCounts(countFile);
|
||||
if (counts) workerMethods[arm] = counts;
|
||||
else counterUnavailable = true;
|
||||
}
|
||||
}
|
||||
try {
|
||||
const countFile = join(root, `${arm}-warm.json`);
|
||||
const protocol: Record<string, number> = {};
|
||||
const transportMs: OperationTimings = {};
|
||||
const session = await manager.open(launchConfig(fixture.origin, `${fixture.origin}/performance.html?mode=density&count=0`, config.browserPath), controller.signal);
|
||||
const runtime = createInstrumentedRuntime(armPaths[arm], countFile, protocol, transportMs, session, fixture.origin, config.pythonPath, controller.signal);
|
||||
const handle: ArmHandle = { arm, session, runtime, protocol, transportMs, countFile, closed: false };
|
||||
handles[arm] = handle;
|
||||
await runtime.observe(controller.signal); // prime the persistent worker; excluded from warm samples
|
||||
} catch (error) {
|
||||
const failedHandle = handles[arm];
|
||||
if (failedHandle) {
|
||||
await closeHandle(failedHandle);
|
||||
await manager.close(failedHandle.session.id).catch(() => {});
|
||||
}
|
||||
delete handles[arm];
|
||||
setupErrors[arm] = safeFailure("warm-setup");
|
||||
}
|
||||
}
|
||||
|
||||
for (let sample = 1; sample <= config.samples; sample += 1) {
|
||||
const order = pairedArmOrder(sample);
|
||||
pairedOrder.push(order);
|
||||
for (let position = 0; position < order.length; position += 1) {
|
||||
const arm = order[position]!;
|
||||
const handle = handles[arm];
|
||||
const before = handle ? snapshotCounts(handle.protocol) : {};
|
||||
const beforeTransport = handle ? { ...handle.transportMs } : {};
|
||||
const started = Date.now();
|
||||
try {
|
||||
if (!handle) throw new Error("arm setup failed");
|
||||
const densityObserveMs = await runWarmFlow(handle, fixture.origin, config.candidateCounts, controller.signal);
|
||||
attempts.push({ arm, phase: "warm", sample, order: (position + 1) as 1 | 2, status: "passed", elapsedMs: Date.now() - started, startupMs: null, protocolOperations: countDelta(before, handle.protocol), transportMs: timingDelta(beforeTransport, handle.transportMs), densityObserveMs });
|
||||
} catch (error) {
|
||||
attempts.push({ arm, phase: "warm", sample, order: (position + 1) as 1 | 2, status: "failed", elapsedMs: Date.now() - started, startupMs: null, error: handle ? failureFor(error, "warm-flow") : (setupErrors[arm] ?? safeFailure("warm-setup")), protocolOperations: handle ? countDelta(before, handle.protocol) : {}, transportMs: handle ? timingDelta(beforeTransport, handle.transportMs) : {} });
|
||||
}
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
for (const arm of ["baseline", "optimized"] as const) {
|
||||
const handle = handles[arm];
|
||||
const error = await closeHandle(handle);
|
||||
if (error) cleanupError ??= error;
|
||||
if (handle) {
|
||||
const counts = await readWorkerCounts(handle.countFile);
|
||||
if (counts) workerMethods[arm] = counts;
|
||||
else counterUnavailable = true;
|
||||
protocolOperations[arm] = { ...handle.protocol };
|
||||
}
|
||||
}
|
||||
await manager.closeAll().catch(() => { cleanupError ??= safeFailure("cleanup"); });
|
||||
await fixture.close().catch(() => { cleanupError ??= safeFailure("cleanup"); });
|
||||
const leftovers = ownedProcessLines(root);
|
||||
if (leftovers === undefined || leftovers.length > 0) {
|
||||
cleanupError ??= safeFailure("cleanup", "unavailable");
|
||||
retainedRoot = true;
|
||||
}
|
||||
if (!cleanupError) {
|
||||
await rm(root, { recursive: true, force: true }).catch(() => {
|
||||
cleanupError ??= safeFailure("cleanup");
|
||||
retainedRoot = true;
|
||||
});
|
||||
} else {
|
||||
retainedRoot = true;
|
||||
}
|
||||
}
|
||||
const warm: Record<Arm, TimingSummary> = {
|
||||
baseline: summarizeBenchmarkAttempts(attempts, "baseline", "warm"),
|
||||
optimized: summarizeBenchmarkAttempts(attempts, "optimized", "warm"),
|
||||
};
|
||||
const coldStartup: Record<Arm, TimingSummary> = {
|
||||
baseline: summarizeTimings(attempts.filter((item) => item.arm === "baseline" && item.phase === "cold-start" && item.status === "passed" && item.startupMs !== null).map((item) => item.startupMs!)),
|
||||
optimized: summarizeTimings(attempts.filter((item) => item.arm === "optimized" && item.phase === "cold-start" && item.status === "passed" && item.startupMs !== null).map((item) => item.startupMs!)),
|
||||
};
|
||||
return {
|
||||
ok: !cleanupError && !counterUnavailable && workerMethods.baseline !== undefined && workerMethods.optimized !== undefined && attempts.length === config.samples * 2 + config.coldSamples * 2 && attempts.every((item) => item.status === "passed"),
|
||||
measurement: { workerMethods: "selected Python worker CDP/IPC helper methods", protocolOperations: "TypeScript JSONL operation names", warmPriming: "one density=0 observe per arm, excluded from warm samples", coldStartup: "browser session open plus first worker observe", runtimeNarrowingMeasured: false },
|
||||
samples: config.samples,
|
||||
coldSamples: config.coldSamples,
|
||||
candidateCounts: config.candidateCounts,
|
||||
attempts,
|
||||
warm,
|
||||
coldStartup,
|
||||
workerMethods,
|
||||
protocolOperations,
|
||||
densityObserve: densitySummaries(attempts),
|
||||
pairedOrder,
|
||||
counterStatus: !counterUnavailable && workerMethods.baseline && workerMethods.optimized ? "complete" : "unavailable",
|
||||
cleanup: cleanupError ? { status: "failed", error: cleanupError, retainedRoot } : { status: "passed", retainedRoot: false },
|
||||
};
|
||||
}
|
||||
|
||||
function parseArgs(argv: readonly string[]): PerformanceBenchmarkOptions & { help?: boolean } {
|
||||
const result: Record<string, unknown> = {};
|
||||
for (let index = 0; index < argv.length; index += 1) {
|
||||
const arg = argv[index]!;
|
||||
if (arg === "--help" || arg === "-h") return { baselineWorkerPath: "", help: true };
|
||||
const values: Record<string, string> = { "--baseline-worker": "baselineWorkerPath", "--optimized-worker": "optimizedWorkerPath", "--python": "pythonPath", "--browser": "browserPath", "--samples": "samples", "--cold-samples": "coldSamples", "--counts": "candidateCounts" };
|
||||
const key = values[arg];
|
||||
if (!key || index + 1 >= argv.length) throw new Error(`unknown or incomplete option: ${arg}`);
|
||||
const value = argv[++index]!;
|
||||
result[key] = key === "samples" || key === "coldSamples" ? Number(value) : key === "candidateCounts" ? value.split(",").map(Number) : value;
|
||||
}
|
||||
if (typeof result.baselineWorkerPath !== "string") throw new Error("--baseline-worker is required");
|
||||
return result as unknown as PerformanceBenchmarkOptions;
|
||||
}
|
||||
|
||||
function printReport(report: BenchmarkReport): void {
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
}
|
||||
|
||||
if (import.meta.main) {
|
||||
try {
|
||||
const parsed = parseArgs(process.argv.slice(2));
|
||||
if (parsed.help) {
|
||||
console.log("usage: node --experimental-strip-types performance-benchmark.ts --baseline-worker PATH [--optimized-worker PATH] [--samples N] [--cold-samples N] [--counts 0,10,60]");
|
||||
} else {
|
||||
const report = await runPerformanceBenchmark(parsed);
|
||||
printReport(report);
|
||||
if (!report.ok) process.exitCode = 1;
|
||||
}
|
||||
} catch (error) {
|
||||
console.error(safeFailure("cli"));
|
||||
process.exitCode = 1;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,119 @@
|
||||
"""Private benchmark wrapper for counting browser-harness calls.
|
||||
|
||||
The wrapper is deliberately a transport-only probe. It records method names and
|
||||
counts, never CDP arguments, URLs, DOM values, page text, or screenshots. The
|
||||
selected worker is executed unchanged after the helper module is wrapped.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import runpy
|
||||
import sys
|
||||
import tempfile
|
||||
from collections import Counter
|
||||
from typing import Any
|
||||
|
||||
|
||||
def _take_flag(argv: list[str], flag: str) -> tuple[str | None, list[str]]:
|
||||
remaining: list[str] = []
|
||||
value: str | None = None
|
||||
index = 0
|
||||
while index < len(argv):
|
||||
item = argv[index]
|
||||
if item == flag:
|
||||
if value is not None or index + 1 >= len(argv):
|
||||
raise ValueError(f"{flag} must be supplied once")
|
||||
value = argv[index + 1]
|
||||
index += 2
|
||||
continue
|
||||
remaining.append(item)
|
||||
index += 1
|
||||
return value, remaining
|
||||
|
||||
|
||||
def _method_name(value: Any) -> str:
|
||||
if isinstance(value, str) and value and len(value) <= 120 and all(
|
||||
character.isalnum() or character in "._-" for character in value
|
||||
):
|
||||
return value
|
||||
return "unknown"
|
||||
|
||||
|
||||
def _wrap_helpers(counts: dict[str, Counter[str]] | None = None) -> dict[str, Counter[str]]:
|
||||
helpers = importlib.import_module("browser_harness.helpers")
|
||||
counts = counts or {"cdp": Counter(), "ipc": Counter()}
|
||||
|
||||
original_cdp = getattr(helpers, "cdp", None)
|
||||
if callable(original_cdp):
|
||||
def counted_cdp(*args: Any, **kwargs: Any) -> Any:
|
||||
method = _method_name(args[0] if args else None)
|
||||
counts["cdp"][method] += 1
|
||||
return original_cdp(*args, **kwargs)
|
||||
helpers.cdp = counted_cdp
|
||||
|
||||
original_send = getattr(helpers, "_send", None)
|
||||
if callable(original_send):
|
||||
def counted_send(*args: Any, **kwargs: Any) -> Any:
|
||||
# Do not inspect the request. The IPC method is intentionally the
|
||||
# fixed helper method name rather than an untrusted operation value.
|
||||
counts["ipc"]["_send"] += 1
|
||||
return original_send(*args, **kwargs)
|
||||
helpers._send = counted_send
|
||||
return counts
|
||||
|
||||
|
||||
def _write_counts(path: str, counts: dict[str, Counter[str]]) -> None:
|
||||
payload = {
|
||||
"cdp": dict(sorted(counts["cdp"].items())),
|
||||
"ipc": dict(sorted(counts["ipc"].items())),
|
||||
}
|
||||
destination = Path(path)
|
||||
destination.parent.mkdir(parents=True, exist_ok=True)
|
||||
fd, temporary = tempfile.mkstemp(prefix="performance-count-", dir=str(destination.parent))
|
||||
try:
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as stream:
|
||||
json.dump(payload, stream, separators=(",", ":"), sort_keys=True)
|
||||
stream.write("\n")
|
||||
os.chmod(temporary, 0o600)
|
||||
os.replace(temporary, destination)
|
||||
os.chmod(destination, 0o600)
|
||||
finally:
|
||||
try:
|
||||
os.unlink(temporary)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
raw = list(sys.argv[1:] if argv is None else argv)
|
||||
worker_path, raw = _take_flag(raw, "--performance-worker")
|
||||
count_path, worker_argv = _take_flag(raw, "--performance-count-file")
|
||||
if not worker_path or not count_path:
|
||||
raise SystemExit("performance_probe requires --performance-worker and --performance-count-file")
|
||||
if not Path(worker_path).is_file():
|
||||
raise SystemExit("performance worker path is not a file")
|
||||
counts: dict[str, Counter[str]] = {"cdp": Counter(), "ipc": Counter()}
|
||||
try:
|
||||
sys.argv = [worker_path, *worker_argv]
|
||||
namespace = runpy.run_path(worker_path, run_name="performance_worker")
|
||||
backend = namespace.get("RealBrowserBackend")
|
||||
main_function = namespace.get("main")
|
||||
if backend is None or not callable(main_function):
|
||||
raise SystemExit("performance worker does not expose its expected entry points")
|
||||
original_load = backend._load_harness
|
||||
def load_and_probe(instance: Any) -> Any:
|
||||
result = original_load(instance)
|
||||
_wrap_helpers(counts)
|
||||
return result
|
||||
backend._load_harness = load_and_probe
|
||||
main_function(worker_argv)
|
||||
finally:
|
||||
_write_counts(count_path, counts)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,6 +1,7 @@
|
||||
import assert from "node:assert/strict";
|
||||
import test from "node:test";
|
||||
import { createBrowserRuntimeFactory, withRuntimeOriginPolicy } from "./runtime.ts";
|
||||
import { createBrowserRuntimeFactory } from "./runtime.ts";
|
||||
import { BrowserOperationError } from "./contracts.ts";
|
||||
import type {
|
||||
BrowserOperation,
|
||||
BrowserRuntimeOptions,
|
||||
@@ -9,7 +10,6 @@ import type {
|
||||
BrowserTransportResultFor,
|
||||
PrivateArtifactMetadata,
|
||||
PrivateArtifactStore,
|
||||
RuntimeOperationContext,
|
||||
} from "./contracts.ts";
|
||||
|
||||
const tab = (id: string, url = "https://example.test/form") => ({
|
||||
@@ -153,18 +153,105 @@ test("factory forwards pinned Python selection to the transport", async () => {
|
||||
await runtime.close();
|
||||
});
|
||||
|
||||
test("wildcard narrowing is scoped while restricted bases cannot broaden", async () => {
|
||||
const first = artifacts();
|
||||
test("narrowing is local, lazy, and propagates only its effective context", async () => {
|
||||
const store = artifacts();
|
||||
const fixture = transportFixture();
|
||||
const runtime = createBrowserRuntimeFactory(options(first.store, fixture.transport).transportFactory).create(options(first.store, fixture.transport));
|
||||
await withRuntimeOriginPolicy(runtime, ["https://example.test"], (context) => runtime.read({}, new AbortController().signal, context));
|
||||
const reads = fixture.calls.filter((call) => call.operation.op === "read");
|
||||
assert.deepEqual((reads[0]?.context as RuntimeOperationContext).allowedOrigins, ["https://example.test"]);
|
||||
let created = 0;
|
||||
const runtime = createBrowserRuntimeFactory({ create() { created++; return fixture.transport; } }).create(options(store.store, fixture.transport));
|
||||
|
||||
const context = await runtime.narrowOriginPolicy({ allowedOrigins: ["https://example.test"] }, new AbortController().signal);
|
||||
assert.deepEqual(context, { allowedOrigins: ["https://example.test"] });
|
||||
assert.equal(created, 0);
|
||||
assert.equal(fixture.calls.length, 0);
|
||||
|
||||
await runtime.read({}, new AbortController().signal, context);
|
||||
await runtime.read({}, new AbortController().signal);
|
||||
assert.equal(fixture.calls.at(-1)?.context, undefined);
|
||||
const restricted = createBrowserRuntimeFactory({ create() { return fixture.transport; } }).create({ ...options(first.store, fixture.transport), allowedOrigins: ["https://example.test"] });
|
||||
await assert.rejects(restricted.narrowOriginPolicy({ allowedOrigins: "*" }, new AbortController().signal), /broadened/);
|
||||
await runtime.close(); await restricted.close();
|
||||
assert.equal(created, 1);
|
||||
assert.deepEqual(fixture.calls[0]?.context, { allowedOrigins: ["https://example.test"] });
|
||||
assert.equal(fixture.calls[1]?.context, undefined);
|
||||
assert.equal(fixture.calls.some((call) => call.operation.op === "narrowOriginPolicy"), false);
|
||||
await runtime.close();
|
||||
});
|
||||
|
||||
test("narrowing validates invalid and broadened policies without dispatch", async () => {
|
||||
const store = artifacts();
|
||||
const fixture = transportFixture();
|
||||
let created = 0;
|
||||
const runtime = createBrowserRuntimeFactory({ create() { created++; return fixture.transport; } }).create({
|
||||
...options(store.store, fixture.transport), allowedOrigins: ["https://example.test"],
|
||||
});
|
||||
|
||||
await assert.rejects(runtime.narrowOriginPolicy({ allowedOrigins: [] }, new AbortController().signal), /must not be empty/);
|
||||
await assert.rejects(runtime.narrowOriginPolicy({ allowedOrigins: "*" }, new AbortController().signal), (error: unknown) => {
|
||||
assert.ok(error instanceof BrowserOperationError);
|
||||
assert.equal(error.details.code, "policy_denied");
|
||||
assert.equal(error.details.mutationState, "not_attempted");
|
||||
return true;
|
||||
});
|
||||
assert.equal(created, 0);
|
||||
assert.equal(fixture.calls.length, 0);
|
||||
await runtime.close();
|
||||
});
|
||||
|
||||
test("narrowing preserves closed and aborted lifecycle errors", async () => {
|
||||
const store = artifacts();
|
||||
const fixture = transportFixture();
|
||||
const runtime = createBrowserRuntimeFactory({ create() { return fixture.transport; } }).create(options(store.store, fixture.transport));
|
||||
await runtime.close();
|
||||
await assert.rejects(runtime.narrowOriginPolicy({ allowedOrigins: ["https://example.test"] }, new AbortController().signal), (error: unknown) => {
|
||||
assert.ok(error instanceof BrowserOperationError);
|
||||
assert.equal(error.details.code, "browser_closed");
|
||||
return true;
|
||||
});
|
||||
|
||||
const parent = new AbortController();
|
||||
parent.abort(new Error("browser session closed"));
|
||||
const abortedRuntime = createBrowserRuntimeFactory({ create() { return fixture.transport; } }).create(options(store.store, fixture.transport), parent.signal);
|
||||
await assert.rejects(abortedRuntime.narrowOriginPolicy({ allowedOrigins: ["https://example.test"] }, new AbortController().signal), (error: unknown) => {
|
||||
assert.ok(error instanceof BrowserOperationError);
|
||||
assert.equal(error.details.code, "browser_closed");
|
||||
return true;
|
||||
});
|
||||
const operation = new AbortController();
|
||||
operation.abort();
|
||||
const liveRuntime = createBrowserRuntimeFactory({ create() { return fixture.transport; } }).create(options(store.store, fixture.transport));
|
||||
await assert.rejects(liveRuntime.narrowOriginPolicy({ allowedOrigins: ["https://example.test"] }, operation.signal), (error: unknown) => {
|
||||
assert.ok(error instanceof Error);
|
||||
assert.equal(error.name, "AbortError");
|
||||
return true;
|
||||
});
|
||||
await abortedRuntime.close();
|
||||
await liveRuntime.close();
|
||||
});
|
||||
|
||||
test("narrowing remains serialized behind an in-flight transport operation", async () => {
|
||||
const store = artifacts();
|
||||
const fixture = transportFixture();
|
||||
let releaseRead!: () => void;
|
||||
let readStarted!: () => void;
|
||||
const readGate = new Promise<void>((resolve) => { releaseRead = resolve; });
|
||||
const started = new Promise<void>((resolve) => { readStarted = resolve; });
|
||||
const transport: BrowserTransport = {
|
||||
async dispatch<Operation extends BrowserOperation>(request: BrowserTransportRequest<Operation>, signal: AbortSignal): Promise<BrowserTransportResultFor<Operation>> {
|
||||
if (request.operation.op === "read") {
|
||||
readStarted();
|
||||
await readGate;
|
||||
}
|
||||
return fixture.transport.dispatch(request, signal);
|
||||
},
|
||||
async close() { return fixture.transport.close(); },
|
||||
};
|
||||
const runtime = createBrowserRuntimeFactory({ create() { return transport; } }).create(options(store.store, transport));
|
||||
const read = runtime.read({}, new AbortController().signal);
|
||||
await started;
|
||||
let narrowed = false;
|
||||
const narrow = runtime.narrowOriginPolicy({ allowedOrigins: ["https://example.test"] }, new AbortController().signal).then(() => { narrowed = true; });
|
||||
await Promise.resolve();
|
||||
assert.equal(narrowed, false);
|
||||
releaseRead();
|
||||
await Promise.all([read, narrow]);
|
||||
assert.equal(narrowed, true);
|
||||
await runtime.close();
|
||||
});
|
||||
|
||||
test("policy is enforced before navigation and tabs are origin filtered", async () => {
|
||||
|
||||
@@ -449,6 +449,30 @@ export function createBrowserRuntime(
|
||||
return run;
|
||||
};
|
||||
|
||||
// Policy narrowing is a runtime concern, not a worker operation. Keep it
|
||||
// on the same lane so it observes the same lifecycle and serialization
|
||||
// boundaries without starting a transport or emitting JSONL.
|
||||
const invokeLocal = <T>(
|
||||
operation: BrowserRuntimeOperationName,
|
||||
signal: AbortSignal,
|
||||
action: () => T,
|
||||
): Promise<T> => {
|
||||
const run = lane.then(() => {
|
||||
if (closed && operation !== "close") throw operationError(operation, "lifecycle", "browser_closed", "browser session is closed", "not_attempted");
|
||||
if (parentSignal.aborted && operation !== "close") throw operationError(operation, "lifecycle", "browser_closed", "browser session is unavailable", "not_attempted");
|
||||
// Match invoke(): a pre-aborted operation signal is rejected before
|
||||
// dispatch/error shaping, while lifecycle failures remain typed.
|
||||
throwIfAborted(signal);
|
||||
try {
|
||||
return action();
|
||||
} catch (error) {
|
||||
throw asOperationError(error, operation, "dispatch", false);
|
||||
}
|
||||
});
|
||||
lane = run.catch(() => undefined);
|
||||
return run;
|
||||
};
|
||||
|
||||
const captureAfterFailure = async (operation: BrowserRuntimeOperationName, context?: RuntimeOperationContext): Promise<FailureCaptureSummary> => {
|
||||
if (parentSignal.aborted) return { status: "unavailable", reason: "browser session is unavailable" };
|
||||
if (operation === "captureEvidence" || operation === "close" || operation === "narrowOriginPolicy") {
|
||||
@@ -500,6 +524,10 @@ export function createBrowserRuntime(
|
||||
const controller = new AbortController();
|
||||
const timed = timedSignal(controller.signal, OPERATION_BUDGETS.close);
|
||||
try {
|
||||
// Wait for local-only operations too. A narrow policy must not race
|
||||
// session teardown, and waiting here still keeps an unused transport
|
||||
// unstarted.
|
||||
await lane;
|
||||
if (transport) {
|
||||
await invoke("close", { op: "close" }, timed.signal, undefined, () => undefined).catch(() => undefined);
|
||||
await transport.close().catch(() => {});
|
||||
@@ -612,11 +640,7 @@ export function createBrowserRuntime(
|
||||
const allowedOrigins = normalizePolicy(request.allowedOrigins);
|
||||
if (basePolicy !== "*" && (allowedOrigins === "*" || allowedOrigins.some((origin) => !basePolicy.includes(origin)))) throw operationError("narrowOriginPolicy", "policy", "policy_denied", "origin policy cannot be broadened", "not_attempted");
|
||||
const context = { allowedOrigins };
|
||||
const result = await invoke("narrowOriginPolicy", { op: "narrowOriginPolicy", request: { allowedOrigins } }, signal, { allowedOrigins: basePolicy }, (raw) => {
|
||||
if (!raw || typeof raw !== "object" || !("allowedOrigins" in raw)) throw new Error("worker returned invalid origin policy");
|
||||
return context;
|
||||
});
|
||||
return result;
|
||||
return invokeLocal("narrowOriginPolicy", signal, () => context);
|
||||
},
|
||||
close,
|
||||
};
|
||||
|
||||
@@ -8,7 +8,7 @@ from unittest.mock import Mock, patch
|
||||
|
||||
from browser_worker import (
|
||||
BrowserWorker, BrowserRuntimeWorker, ElementMetadata, WorkerError, BrowserOperationError,
|
||||
RealBrowserBackend, MAX_LINE_BYTES, MAX_READ_TEXT, process_request, run_stream,
|
||||
RealBrowserBackend, ACTION_TIMEOUT_SECONDS, MAX_LINE_BYTES, MAX_READ_TEXT, process_request, run_stream,
|
||||
validate_cdp_url, main,
|
||||
)
|
||||
|
||||
@@ -103,6 +103,25 @@ class FakeBackend:
|
||||
pass
|
||||
|
||||
|
||||
class BatchFakeBackend(FakeBackend):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.metadata_many_calls = []
|
||||
self.metadata_calls = []
|
||||
self.batch_result = None
|
||||
|
||||
def metadata(self, backend_id):
|
||||
self.metadata_calls.append(backend_id)
|
||||
return super().metadata(backend_id)
|
||||
|
||||
def metadata_many(self, backend_ids):
|
||||
ids = list(backend_ids)
|
||||
self.metadata_many_calls.append(ids)
|
||||
if self.batch_result is not None:
|
||||
return self.batch_result
|
||||
return {backend_id: self.elements[backend_id] for backend_id in ids if backend_id in self.elements}
|
||||
|
||||
|
||||
class PopupFakeBackend(FakeBackend):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
@@ -171,6 +190,46 @@ class RuntimeFakeBackend(FakeBackend):
|
||||
return b"\\x89PNG" + (b"x" * 32)
|
||||
|
||||
|
||||
class FakeClock:
|
||||
def __init__(self):
|
||||
self.now = 0.0
|
||||
self.sleeps = []
|
||||
|
||||
def monotonic(self):
|
||||
return self.now
|
||||
|
||||
def sleep(self, duration):
|
||||
self.sleeps.append(duration)
|
||||
self.now += duration
|
||||
|
||||
|
||||
class DelayedCloseBackend(RuntimeFakeBackend):
|
||||
def __init__(self, disappear_after=None):
|
||||
super().__init__()
|
||||
self.tabs = [self.tabs[0]]
|
||||
self.disappear_after = disappear_after
|
||||
self.inventory_reads = 0
|
||||
self.post_close_reads = 0
|
||||
self.pending_close = None
|
||||
self.close_dispatches = []
|
||||
|
||||
def list_tabs(self):
|
||||
self.inventory_reads += 1
|
||||
rows = [dict(tab) for tab in self.tabs]
|
||||
if self.pending_close is not None:
|
||||
self.post_close_reads += 1
|
||||
if self.disappear_after is not None and self.post_close_reads >= self.disappear_after:
|
||||
target_id = self.pending_close
|
||||
self.tabs[:] = [tab for tab in self.tabs if tab["targetId"] != target_id]
|
||||
self.pending_close = None
|
||||
rows = [dict(tab) for tab in self.tabs]
|
||||
return rows
|
||||
|
||||
def close_tab(self, target_id):
|
||||
self.close_dispatches.append(target_id)
|
||||
self.pending_close = target_id
|
||||
|
||||
|
||||
class RuntimeTests(unittest.TestCase):
|
||||
def make_runtime(self, backend=None, origins="*"):
|
||||
backend = backend or RuntimeFakeBackend()
|
||||
@@ -385,6 +444,102 @@ class RuntimeTests(unittest.TestCase):
|
||||
with self.assertRaisesRegex(BrowserOperationError, "stale|unavailable"):
|
||||
runtime.select_tab({"tabId": tab_id})
|
||||
|
||||
def test_close_waits_for_delayed_target_disappearance_without_redispatching(self):
|
||||
backend = DelayedCloseBackend(disappear_after=2)
|
||||
runtime = self.make_runtime(backend)
|
||||
opener = runtime.list_tabs()[0]["tabId"]
|
||||
backend.tabs.append({"targetId": "popup", "url": "http://fixture.test/popup",
|
||||
"title": "Popup", "openerTargetId": "tab-a"})
|
||||
popup = runtime.wait_popup({"openerTabId": opener})["tab"]
|
||||
self.assertIsNotNone(popup)
|
||||
popup_id = popup["tabId"]
|
||||
clock = FakeClock()
|
||||
|
||||
with patch("browser_worker.time.monotonic", side_effect=clock.monotonic), \
|
||||
patch("browser_worker.time.sleep", side_effect=clock.sleep):
|
||||
result = runtime.close_tab({"tabId": popup_id})
|
||||
|
||||
self.assertEqual(result["mutationState"], "verified")
|
||||
self.assertEqual(backend.close_dispatches, ["popup"])
|
||||
self.assertGreaterEqual(backend.post_close_reads, 2)
|
||||
self.assertGreater(clock.now, 0)
|
||||
tabs = runtime.list_tabs()
|
||||
self.assertEqual([tab["tabId"] for tab in tabs], [opener])
|
||||
self.assertTrue(tabs[0]["selected"])
|
||||
with self.assertRaisesRegex(BrowserOperationError, "stale|unavailable"):
|
||||
runtime.select_tab({"tabId": popup_id})
|
||||
|
||||
def test_close_deadline_reports_unknown_without_retry_when_target_persists(self):
|
||||
backend = DelayedCloseBackend(disappear_after=None)
|
||||
runtime = self.make_runtime(backend)
|
||||
opener = runtime.list_tabs()[0]["tabId"]
|
||||
backend.tabs.append({"targetId": "popup", "url": "http://fixture.test/popup",
|
||||
"title": "Popup", "openerTargetId": "tab-a"})
|
||||
popup = runtime.wait_popup({"openerTabId": opener})["tab"]
|
||||
self.assertIsNotNone(popup)
|
||||
clock = FakeClock()
|
||||
|
||||
with patch("browser_worker.time.monotonic", side_effect=clock.monotonic), \
|
||||
patch("browser_worker.time.sleep", side_effect=clock.sleep), \
|
||||
self.assertRaises(BrowserOperationError) as caught:
|
||||
runtime.close_tab({"tabId": popup["tabId"]})
|
||||
|
||||
details = caught.exception.details
|
||||
self.assertEqual(details["code"], "worker_failure")
|
||||
self.assertEqual(details["mutationState"], "unknown")
|
||||
self.assertEqual(details["retry"], "never")
|
||||
self.assertEqual(backend.close_dispatches, ["popup"])
|
||||
self.assertGreaterEqual(backend.post_close_reads, 2)
|
||||
self.assertGreaterEqual(clock.now, ACTION_TIMEOUT_SECONDS)
|
||||
|
||||
def test_close_inventory_failure_preserves_unknown_state_without_retry(self):
|
||||
backend = DelayedCloseBackend(disappear_after=None)
|
||||
runtime = self.make_runtime(backend)
|
||||
opener = runtime.list_tabs()[0]["tabId"]
|
||||
backend.tabs.append({"targetId": "popup", "url": "http://fixture.test/popup",
|
||||
"title": "Popup", "openerTargetId": "tab-a"})
|
||||
popup = runtime.wait_popup({"openerTabId": opener})["tab"]
|
||||
self.assertIsNotNone(popup)
|
||||
original_list_tabs = backend.list_tabs
|
||||
def fail_after_close():
|
||||
if backend.pending_close is not None:
|
||||
raise RuntimeError("inventory unavailable")
|
||||
return original_list_tabs()
|
||||
backend.list_tabs = fail_after_close
|
||||
|
||||
with self.assertRaises(BrowserOperationError) as caught:
|
||||
runtime.close_tab({"tabId": popup["tabId"]})
|
||||
|
||||
details = caught.exception.details
|
||||
self.assertEqual(details["code"], "worker_failure")
|
||||
self.assertEqual(details["mutationState"], "unknown")
|
||||
self.assertEqual(details["retry"], "never")
|
||||
self.assertEqual(backend.close_dispatches, ["popup"])
|
||||
|
||||
def test_close_cancellation_preserves_unknown_state_without_retry(self):
|
||||
backend = DelayedCloseBackend(disappear_after=2)
|
||||
runtime = self.make_runtime(backend)
|
||||
opener = runtime.list_tabs()[0]["tabId"]
|
||||
backend.tabs.append({"targetId": "popup", "url": "http://fixture.test/popup",
|
||||
"title": "Popup", "openerTargetId": "tab-a"})
|
||||
popup = runtime.wait_popup({"openerTabId": opener})["tab"]
|
||||
self.assertIsNotNone(popup)
|
||||
class Cancelled(BaseException):
|
||||
pass
|
||||
def cancel(_target_id):
|
||||
backend.close_dispatches.append(_target_id)
|
||||
raise Cancelled()
|
||||
backend.close_tab = cancel
|
||||
|
||||
with self.assertRaises(BrowserOperationError) as caught:
|
||||
runtime.close_tab({"tabId": popup["tabId"]})
|
||||
|
||||
details = caught.exception.details
|
||||
self.assertEqual(details["code"], "aborted")
|
||||
self.assertEqual(details["mutationState"], "unknown")
|
||||
self.assertEqual(details["retry"], "never")
|
||||
self.assertEqual(backend.close_dispatches, ["popup"])
|
||||
|
||||
def test_transport_wrapper_context_restores_base_and_uses_exact_result_shape(self):
|
||||
backend = RuntimeFakeBackend()
|
||||
runtime = self.make_runtime(backend)
|
||||
@@ -550,6 +705,82 @@ class WorkerTests(unittest.TestCase):
|
||||
self.assertNotIn("values", json.dumps(observation))
|
||||
self.assertNotIn("Disabled", json.dumps(observation))
|
||||
|
||||
def test_observe_batches_dom_metadata_but_keeps_full_ax_roles_and_names(self):
|
||||
backend = BatchFakeBackend()
|
||||
backend.elements[1] = replace(backend.elements[1], name="DOM Email", ax_role="textbox", ax_name="AX Email")
|
||||
backend.elements[3] = replace(backend.elements[3], name="DOM Submit", ax_role="button", ax_name="AX Submit")
|
||||
backend.nodes[0]["name"] = {"value": "AX Email old-test-secret"}
|
||||
backend.nodes[1]["name"] = {"value": "AX Submit"}
|
||||
worker = BrowserWorker(backend, ["http://fixture.test"])
|
||||
|
||||
observation = worker.observe()
|
||||
|
||||
self.assertEqual(backend.metadata_many_calls, [[1, 3, 4, 2]])
|
||||
self.assertEqual(backend.metadata_calls, [])
|
||||
self.assertEqual([candidate["name"] for candidate in observation["candidates"]],
|
||||
["DOM Email", "AX Submit", "Password"])
|
||||
self.assertEqual([candidate["role"] for candidate in observation["candidates"]],
|
||||
["textbox", "button", "textbox"])
|
||||
self.assertNotIn("old-test-secret", json.dumps(observation))
|
||||
|
||||
result = worker.act({
|
||||
"observationId": observation["observationId"], "candidateId": "c-1", "action": "click",
|
||||
"postcondition": {"kind": "url", "value": "http://fixture.test/done"},
|
||||
})
|
||||
self.assertTrue(result["verified"])
|
||||
self.assertEqual(backend.metadata_calls, [3])
|
||||
|
||||
def test_observe_falls_back_for_missing_batch_metadata(self):
|
||||
backend = BatchFakeBackend()
|
||||
backend.batch_result = {1: backend.elements[1]}
|
||||
worker = BrowserWorker(backend, ["http://fixture.test"])
|
||||
|
||||
observation = worker.observe()
|
||||
|
||||
self.assertEqual([candidate["name"] for candidate in observation["candidates"]],
|
||||
["Email", "Submit", "Password"])
|
||||
self.assertEqual(backend.metadata_calls, [3, 4, 2])
|
||||
|
||||
def test_observe_batches_in_bounded_chunks_and_preserves_order_and_deduplication(self):
|
||||
backend = BatchFakeBackend()
|
||||
backend.elements = {
|
||||
index: ElementMetadata(index, "button", "", "button", f"Button {index}", True, False,
|
||||
ax_role="button", ax_name=f"Button {index}")
|
||||
for index in range(1, 34)
|
||||
}
|
||||
backend.nodes = [
|
||||
{"backendDOMNodeId": index, "role": {"value": "button"}, "name": {"value": f"AX {index}"}}
|
||||
for index in range(1, 34)
|
||||
]
|
||||
backend.nodes.insert(10, dict(backend.nodes[9]))
|
||||
worker = BrowserWorker(backend, ["http://fixture.test"])
|
||||
|
||||
observation = worker.observe()
|
||||
|
||||
self.assertEqual([len(chunk) for chunk in backend.metadata_many_calls], [16, 16, 1])
|
||||
self.assertEqual([candidate["name"] for candidate in observation["candidates"]],
|
||||
[f"AX {index}" for index in range(1, 34)])
|
||||
self.assertEqual(len(observation["candidates"]), 33)
|
||||
|
||||
def test_observe_skips_detached_nodes_and_keeps_candidate_limit(self):
|
||||
backend = BatchFakeBackend()
|
||||
backend.elements = {
|
||||
index: ElementMetadata(index, "button", "", "button", f"Button {index}", True, False,
|
||||
connected=index != 10, ax_role="button", ax_name=f"Button {index}")
|
||||
for index in range(1, 63)
|
||||
}
|
||||
backend.nodes = [
|
||||
{"backendDOMNodeId": index, "role": {"value": "button"}, "name": {"value": f"AX {index}"}}
|
||||
for index in range(1, 63)
|
||||
]
|
||||
worker = BrowserWorker(backend, ["http://fixture.test"])
|
||||
|
||||
observation = worker.observe()
|
||||
|
||||
self.assertEqual(len(observation["candidates"]), 60)
|
||||
self.assertNotIn("AX 10", json.dumps(observation))
|
||||
self.assertTrue(all(len(chunk) <= 16 for chunk in backend.metadata_many_calls))
|
||||
|
||||
def test_password_fill_uses_literal_without_exposing_value_in_metadata_or_result(self):
|
||||
backend = FakeBackend()
|
||||
backend.values[2] = "old-test-secret"
|
||||
@@ -721,6 +952,26 @@ class BridgeRegressionTests(unittest.TestCase):
|
||||
with self.assertRaisesRegex(WorkerError, "outside"):
|
||||
worker.act({"observationId": obs["observationId"], "candidateId": "c-0", "action": "fill", "text": "Ada"})
|
||||
|
||||
def test_fill_url_postcondition_reuses_post_action_tab_state(self):
|
||||
backend = FakeBackend()
|
||||
original_fill = backend.fill
|
||||
def fill_and_navigate(node, text):
|
||||
original_fill(node, text)
|
||||
backend.url = "http://fixture.test/done"
|
||||
backend.fill = fill_and_navigate
|
||||
current_tab = backend.current_tab
|
||||
backend.current_tab = Mock(side_effect=current_tab)
|
||||
worker = BrowserWorker(backend, ["http://fixture.test"])
|
||||
observation = worker.observe()
|
||||
result = worker.act({
|
||||
"observationId": observation["observationId"], "candidateId": "c-0",
|
||||
"action": "fill", "text": "Ada",
|
||||
"postcondition": {"kind": "url", "value": "http://fixture.test/done"},
|
||||
})
|
||||
self.assertTrue(result["verified"])
|
||||
# observe, fresh pre-mutation revalidation, and one post-action read.
|
||||
self.assertEqual(backend.current_tab.call_count, 3)
|
||||
|
||||
def test_oversized_input_and_sensitive_error_are_not_echoed(self):
|
||||
worker = BrowserWorker(FakeBackend(), ["http://fixture.test"])
|
||||
out = io.BytesIO()
|
||||
@@ -738,6 +989,136 @@ class BridgeRegressionTests(unittest.TestCase):
|
||||
backend._load_harness()
|
||||
spawn.assert_not_called()
|
||||
|
||||
def test_real_backend_list_tabs_uses_one_inventory_with_helper_filtering_and_openers(self):
|
||||
helper = Mock()
|
||||
helper.cdp.return_value = {"targetInfos": [
|
||||
{"type": "page", "targetId": "page", "url": "http://fixture.test/form", "title": "Fixture",
|
||||
"openerId": "opener"},
|
||||
{"type": "page", "targetId": "internal", "url": "chrome://settings", "title": "Settings"},
|
||||
{"type": "page", "targetId": "devtools", "url": "devtools://devtools", "title": "DevTools"},
|
||||
{"type": "page", "targetId": "placeholder", "url": "about:blank#startup", "title": "Starting agent 1"},
|
||||
{"type": "service_worker", "targetId": "worker", "url": "http://fixture.test/sw", "title": "Worker"},
|
||||
{"type": "page", "targetId": "plain", "url": "https://other.test/", "title": "Other", "openerId": ""},
|
||||
{"type": "page", "targetId": "malformed-opener", "url": "http://fixture.test/other", "title": "Other",
|
||||
"openerId": 42},
|
||||
]}
|
||||
backend = RealBrowserBackend.__new__(RealBrowserBackend)
|
||||
backend._helpers = helper
|
||||
|
||||
tabs = backend.list_tabs()
|
||||
|
||||
self.assertEqual([tab["targetId"] for tab in tabs], ["page", "plain", "malformed-opener"])
|
||||
self.assertEqual(tabs[0]["openerTargetId"], "opener")
|
||||
self.assertIsNone(tabs[1]["openerTargetId"])
|
||||
self.assertIsNone(tabs[2]["openerTargetId"])
|
||||
helper.cdp.assert_called_once_with("Target.getTargets")
|
||||
helper.list_tabs.assert_not_called()
|
||||
|
||||
def test_real_backend_list_tabs_propagates_inventory_errors_without_fallback(self):
|
||||
helper = Mock()
|
||||
helper.cdp.side_effect = RuntimeError("inventory unavailable")
|
||||
backend = RealBrowserBackend.__new__(RealBrowserBackend)
|
||||
backend._helpers = helper
|
||||
|
||||
with self.assertRaisesRegex(RuntimeError, "inventory unavailable"):
|
||||
backend.list_tabs()
|
||||
helper.cdp.assert_called_once_with("Target.getTargets")
|
||||
helper.list_tabs.assert_not_called()
|
||||
|
||||
def test_real_backend_switch_tab_does_not_reenable_harness_domains(self):
|
||||
helper = Mock()
|
||||
helper.switch_tab.return_value = "session-1"
|
||||
backend = RealBrowserBackend.__new__(RealBrowserBackend)
|
||||
backend._helpers = helper
|
||||
|
||||
backend.switch_tab("target-1")
|
||||
|
||||
self.assertEqual(backend._session_id, "session-1")
|
||||
self.assertEqual(backend._attached_target, "target-1")
|
||||
helper.switch_tab.assert_called_once_with("target-1", activate=False)
|
||||
helper.cdp.assert_not_called()
|
||||
|
||||
def test_real_backend_caches_select_all_modifier_for_backend_lifetime(self):
|
||||
helper = Mock()
|
||||
calls = []
|
||||
def cdp(method, **_kwargs):
|
||||
calls.append(method)
|
||||
return {"userAgent": "Macintosh"} if method == "Browser.getVersion" else {}
|
||||
helper.cdp.side_effect = cdp
|
||||
backend = RealBrowserBackend.__new__(RealBrowserBackend)
|
||||
backend._helpers = helper
|
||||
backend._call = Mock(return_value=True)
|
||||
|
||||
backend.fill(1, "first")
|
||||
backend.fill(1, "second")
|
||||
|
||||
self.assertEqual(calls.count("Browser.getVersion"), 1)
|
||||
keydowns = [call for call in helper.cdp.call_args_list
|
||||
if call.args[0] == "Input.dispatchKeyEvent" and call.kwargs["type"] == "rawKeyDown"]
|
||||
self.assertEqual([call.kwargs["modifiers"] for call in keydowns], [4, 4])
|
||||
|
||||
def test_wait_ready_returns_immediately_for_complete_document(self):
|
||||
backend = RealBrowserBackend.__new__(RealBrowserBackend)
|
||||
helper = Mock()
|
||||
helper.js.return_value = "complete"
|
||||
backend._helpers = helper
|
||||
clock = FakeClock()
|
||||
|
||||
with patch("browser_worker.time.monotonic", side_effect=clock.monotonic), \
|
||||
patch("browser_worker.time.sleep", side_effect=clock.sleep):
|
||||
self.assertTrue(backend.wait_ready(timeout=3.0))
|
||||
|
||||
helper.js.assert_called_once_with("document.readyState")
|
||||
helper.wait_for_load.assert_not_called()
|
||||
self.assertEqual(clock.sleeps, [])
|
||||
|
||||
def test_wait_ready_polls_loading_document_at_short_bounded_intervals(self):
|
||||
backend = RealBrowserBackend.__new__(RealBrowserBackend)
|
||||
helper = Mock()
|
||||
helper.js.side_effect = ["loading", "complete"]
|
||||
backend._helpers = helper
|
||||
clock = FakeClock()
|
||||
|
||||
with patch("browser_worker.time.monotonic", side_effect=clock.monotonic), \
|
||||
patch("browser_worker.time.sleep", side_effect=clock.sleep):
|
||||
self.assertTrue(backend.wait_ready(timeout=3.0))
|
||||
|
||||
self.assertEqual(helper.js.call_count, 2)
|
||||
self.assertEqual(len(clock.sleeps), 1)
|
||||
self.assertGreater(clock.sleeps[0], 0)
|
||||
self.assertLessEqual(clock.sleeps[0], 0.025)
|
||||
helper.wait_for_load.assert_not_called()
|
||||
|
||||
def test_wait_ready_times_out_without_sleeping_past_deadline(self):
|
||||
backend = RealBrowserBackend.__new__(RealBrowserBackend)
|
||||
helper = Mock()
|
||||
helper.js.return_value = "loading"
|
||||
backend._helpers = helper
|
||||
clock = FakeClock()
|
||||
|
||||
with patch("browser_worker.time.monotonic", side_effect=clock.monotonic), \
|
||||
patch("browser_worker.time.sleep", side_effect=clock.sleep):
|
||||
self.assertFalse(backend.wait_ready(timeout=0.075))
|
||||
|
||||
self.assertEqual(clock.now, 0.075)
|
||||
self.assertTrue(all(duration <= 0.025 for duration in clock.sleeps))
|
||||
helper.wait_for_load.assert_not_called()
|
||||
|
||||
def test_wait_ready_preserves_failed_ready_state_query(self):
|
||||
backend = RealBrowserBackend.__new__(RealBrowserBackend)
|
||||
helper = Mock()
|
||||
helper.js.side_effect = RuntimeError("ready state query failed")
|
||||
backend._helpers = helper
|
||||
clock = FakeClock()
|
||||
|
||||
with patch("browser_worker.time.monotonic", side_effect=clock.monotonic), \
|
||||
patch("browser_worker.time.sleep", side_effect=clock.sleep):
|
||||
with self.assertRaisesRegex(RuntimeError, "ready state query failed"):
|
||||
backend.wait_ready(timeout=3.0)
|
||||
|
||||
self.assertEqual(clock.sleeps, [])
|
||||
helper.wait_for_load.assert_not_called()
|
||||
|
||||
def test_real_backend_unrestricted_policy_still_requires_one_web_tab(self):
|
||||
helper = Mock()
|
||||
helper.list_tabs.return_value = [
|
||||
@@ -819,6 +1200,55 @@ class BridgeRegressionTests(unittest.TestCase):
|
||||
self.assertEqual(keydowns[0].kwargs["modifiers"], 4)
|
||||
backend._helpers.press_key.assert_called_once_with("Backspace")
|
||||
|
||||
def test_metadata_many_uses_one_call_and_releases_temporary_object_group(self):
|
||||
backend = RealBrowserBackend.__new__(RealBrowserBackend)
|
||||
helper = Mock()
|
||||
backend._helpers = helper
|
||||
|
||||
def cdp(method, **kwargs):
|
||||
if method == "DOM.resolveNode":
|
||||
return {"object": {"objectId": f"object-{kwargs['backendNodeId']}"}}
|
||||
if method == "Runtime.callFunctionOn":
|
||||
return {"result": {"value": [
|
||||
{"tag": "input", "type": "text", "role": "", "name": "Email",
|
||||
"connected": True, "mainFrame": True, "contenteditable": False,
|
||||
"visible": True, "disabled": False},
|
||||
{"tag": "button", "type": "", "role": "button", "name": "Submit",
|
||||
"connected": True, "mainFrame": True, "contenteditable": False,
|
||||
"visible": True, "disabled": False},
|
||||
]}}
|
||||
return {}
|
||||
helper.cdp.side_effect = cdp
|
||||
|
||||
result = backend.metadata_many([1, 2])
|
||||
|
||||
self.assertEqual([metadata.name for metadata in result.values()], ["Email", "Submit"])
|
||||
calls = [call.args[0] for call in helper.cdp.call_args_list]
|
||||
self.assertEqual(calls.count("DOM.resolveNode"), 2)
|
||||
self.assertEqual(calls.count("Runtime.callFunctionOn"), 1)
|
||||
self.assertEqual(calls.count("Runtime.releaseObjectGroup"), 1)
|
||||
call = next(call for call in helper.cdp.call_args_list if call.args[0] == "Runtime.callFunctionOn")
|
||||
self.assertEqual([argument.get("objectId") for argument in call.kwargs["arguments"]],
|
||||
["object-1", "object-2"])
|
||||
|
||||
def test_metadata_many_releases_object_group_when_batch_call_fails(self):
|
||||
backend = RealBrowserBackend.__new__(RealBrowserBackend)
|
||||
helper = Mock()
|
||||
backend._helpers = helper
|
||||
|
||||
def cdp(method, **kwargs):
|
||||
if method == "DOM.resolveNode":
|
||||
return {"object": {"objectId": f"object-{kwargs['backendNodeId']}"}}
|
||||
if method == "Runtime.callFunctionOn":
|
||||
raise RuntimeError("detached")
|
||||
return {}
|
||||
helper.cdp.side_effect = cdp
|
||||
|
||||
self.assertEqual(backend.metadata_many([1, 2]), {})
|
||||
self.assertEqual(
|
||||
[call.args[0] for call in helper.cdp.call_args_list].count("Runtime.releaseObjectGroup"), 1
|
||||
)
|
||||
|
||||
def test_call_releases_remote_object_on_browser_exception(self):
|
||||
backend = RealBrowserBackend.__new__(RealBrowserBackend)
|
||||
backend._helpers = Mock()
|
||||
|
||||
+41
-13
@@ -26,11 +26,12 @@ const COMMANDS = new Set(["check", "sync", "prune"]);
|
||||
|
||||
function usage() {
|
||||
return (
|
||||
`Usage: node scripts/pi-links.mjs <check|sync|prune> [--dry-run]\n\n` +
|
||||
`Usage: node scripts/pi-links.mjs <check|sync|prune> [--dry-run] [--migrate-from /old/repo]\n\n` +
|
||||
` check Report missing, stale, or conflicting exposure without writing\n` +
|
||||
` sync Create or repair manifest-declared links and local package entries\n` +
|
||||
` prune Remove repository-owned links and local package entries not in the manifest\n` +
|
||||
` --dry-run Print sync/prune changes without applying them`
|
||||
` --dry-run Print sync/prune changes without applying them\n` +
|
||||
` --migrate-from Explicit old root for exact manifest-link repairs (check/sync only)`
|
||||
);
|
||||
}
|
||||
|
||||
@@ -384,7 +385,7 @@ async function readLocalPackages(settingsFile) {
|
||||
.filter((entry) => entry.resolved);
|
||||
}
|
||||
|
||||
async function inspectDesiredLink(link, repoRoot) {
|
||||
async function inspectDesiredLink(link, repoRoot, migrateFrom) {
|
||||
const stat = await statOrUndefined(link.destination);
|
||||
if (!stat) return { type: "create-link", link };
|
||||
if (!stat.isSymbolicLink()) {
|
||||
@@ -402,7 +403,10 @@ async function inspectDesiredLink(link, repoRoot) {
|
||||
);
|
||||
if ((await canonical(resolvedTarget)) === (await canonical(link.source)))
|
||||
return undefined;
|
||||
if (isInside(repoRoot, resolvedTarget))
|
||||
const expectedOldTarget = migrateFrom
|
||||
? path.join(migrateFrom, path.relative(repoRoot, link.source))
|
||||
: undefined;
|
||||
if (isInside(repoRoot, resolvedTarget) || resolvedTarget === expectedOldTarget)
|
||||
return { type: "replace-link", link, previousTarget: resolvedTarget };
|
||||
return {
|
||||
type: "conflict",
|
||||
@@ -451,6 +455,14 @@ async function discoverStaleLinks(state, desiredDestinations) {
|
||||
|
||||
export async function buildPlan(options = {}) {
|
||||
const state = await resolveDesiredState(options);
|
||||
let migrateFrom;
|
||||
if (options.migrateFrom !== undefined) {
|
||||
if (typeof options.migrateFrom !== "string" || !path.isAbsolute(options.migrateFrom))
|
||||
throw new Error("migrate-from must be an absolute old repository root");
|
||||
migrateFrom = path.resolve(options.migrateFrom);
|
||||
if (isInside(migrateFrom, state.repoRoot) || isInside(state.repoRoot, migrateFrom))
|
||||
throw new Error("migrate-from must be separate from the current repository root");
|
||||
}
|
||||
const desiredDestinations = new Set(
|
||||
state.links.map((link) => link.destination),
|
||||
);
|
||||
@@ -461,7 +473,7 @@ export async function buildPlan(options = {}) {
|
||||
const conflicts = [];
|
||||
|
||||
for (const link of state.links) {
|
||||
const result = await inspectDesiredLink(link, state.repoRoot);
|
||||
const result = await inspectDesiredLink(link, state.repoRoot, migrateFrom);
|
||||
if (result?.type === "conflict") conflicts.push(result);
|
||||
else if (result) operations.push(result);
|
||||
}
|
||||
@@ -545,7 +557,13 @@ function selectOperations(command, operations) {
|
||||
async function createOrReplaceLink(operation) {
|
||||
const { link } = operation;
|
||||
await mkdir(path.dirname(link.destination), { recursive: true });
|
||||
if (operation.type === "replace-link") await unlink(link.destination);
|
||||
if (operation.type === "replace-link") {
|
||||
const current = await statOrUndefined(link.destination);
|
||||
if (!current?.isSymbolicLink() ||
|
||||
path.resolve(path.dirname(link.destination), await readlink(link.destination)) !== operation.previousTarget)
|
||||
throw new Error(`${link.destination}: destination changed after planning; refusing replacement`);
|
||||
await unlink(link.destination);
|
||||
}
|
||||
const target = link.relative
|
||||
? path.relative(path.dirname(link.destination), link.source)
|
||||
: link.source;
|
||||
@@ -576,6 +594,9 @@ async function applyOperations(operations) {
|
||||
}
|
||||
|
||||
export async function run(command, options = {}) {
|
||||
if (!COMMANDS.has(command)) throw new Error("unknown link-manager command");
|
||||
if (command === "prune" && options.migrateFrom !== undefined)
|
||||
throw new Error("migrate-from is only supported for check/sync; it never authorizes pruning");
|
||||
const plan = await buildPlan(options);
|
||||
for (const conflict of plan.conflicts)
|
||||
console.error(`CONFLICT ${conflict.path}: ${conflict.reason}`);
|
||||
@@ -606,16 +627,23 @@ export async function run(command, options = {}) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
async function main(argv = process.argv.slice(2)) {
|
||||
const command = argv.find((arg) => !arg.startsWith("-"));
|
||||
const unknown = argv.filter(
|
||||
(arg) => arg.startsWith("-") && arg !== "--dry-run",
|
||||
);
|
||||
if (!COMMANDS.has(command) || unknown.length > 0) {
|
||||
export async function main(argv = process.argv.slice(2)) {
|
||||
const [command, ...args] = argv;
|
||||
const options = {};
|
||||
let invalid = !COMMANDS.has(command);
|
||||
for (let index = 0; index < args.length; index++) {
|
||||
const arg = args[index];
|
||||
if (arg === "--dry-run" && options.dryRun === undefined) options.dryRun = true;
|
||||
else if (arg === "--migrate-from" && options.migrateFrom === undefined &&
|
||||
args[index + 1] && !args[index + 1].startsWith("--"))
|
||||
options.migrateFrom = args[++index];
|
||||
else invalid = true;
|
||||
}
|
||||
if (invalid) {
|
||||
console.error(usage());
|
||||
return 2;
|
||||
}
|
||||
return run(command, { dryRun: argv.includes("--dry-run") });
|
||||
return run(command, options);
|
||||
}
|
||||
|
||||
if (
|
||||
|
||||
@@ -11,7 +11,7 @@ import os from "node:os";
|
||||
import path from "node:path";
|
||||
import test from "node:test";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { buildPlan, run } from "./pi-links.mjs";
|
||||
import { buildPlan, main, run } from "./pi-links.mjs";
|
||||
|
||||
const repositoryRoot = path.resolve(
|
||||
path.dirname(fileURLToPath(import.meta.url)),
|
||||
@@ -295,6 +295,68 @@ test("normal exposure and package evidence retain the Slice 0 security boundary"
|
||||
);
|
||||
});
|
||||
|
||||
test("explicit relocation repairs only exact manifest links, including dangling targets", async () => {
|
||||
const options = await fixture();
|
||||
const oldRoot = path.join(options.root, "old-repo");
|
||||
const initial = await buildPlan(options);
|
||||
for (const link of initial.state.links) {
|
||||
await mkdir(path.dirname(link.destination), { recursive: true });
|
||||
const oldTarget = path.join(oldRoot, path.relative(options.repoRoot, link.source));
|
||||
await symlink(link.relative ? path.relative(path.dirname(link.destination), oldTarget) : oldTarget, link.destination);
|
||||
}
|
||||
assert.equal((await buildPlan(options)).conflicts.length, initial.state.links.length);
|
||||
const migration = { ...options, migrateFrom: oldRoot };
|
||||
const plan = await buildPlan(migration);
|
||||
assert.equal(plan.conflicts.length, 0);
|
||||
assert.equal(plan.operations.length, initial.state.links.length);
|
||||
assert.ok(plan.operations.every(op => op.type === "replace-link"));
|
||||
assert.equal(await run("sync", { ...migration, dryRun: true }), 0);
|
||||
assert.equal((await buildPlan(options)).conflicts.length, initial.state.links.length);
|
||||
assert.equal(await run("sync", migration), 0);
|
||||
assert.equal(await run("check", options), 0);
|
||||
});
|
||||
|
||||
test("relocation preserves wrong-path, unrelated, prefix-lookalike and regular-file collisions", async () => {
|
||||
for (const kind of ["wrong-path", "unrelated", "prefix", "file"]) {
|
||||
const options = await fixture();
|
||||
const oldRoot = path.join(options.root, "old-repo");
|
||||
const destination = path.join(options.homeDir, ".agents", "skills", "global-skill");
|
||||
await mkdir(path.dirname(destination), { recursive: true });
|
||||
const target = kind === "wrong-path" ? path.join(oldRoot, "skills/wrong")
|
||||
: kind === "prefix" ? `${oldRoot}-other/skills/group/global-skill`
|
||||
: path.join(options.root, "unrelated");
|
||||
if (kind === "file") await writeFile(destination, "keep");
|
||||
else await symlink(target, destination);
|
||||
const migration = { ...options, migrateFrom: oldRoot };
|
||||
assert.equal((await buildPlan(migration)).conflicts.length, 1);
|
||||
assert.equal(await run("sync", migration), 2);
|
||||
assert.equal(kind === "file" ? await readFile(destination, "utf8") : await readlink(destination), kind === "file" ? "keep" : target);
|
||||
}
|
||||
});
|
||||
|
||||
test("relocation never claims stale old-root links or local packages for pruning", async () => {
|
||||
const options = await fixture();
|
||||
const oldRoot = path.join(options.root, "old-repo");
|
||||
await run("sync", options);
|
||||
const stale = path.join(options.homeDir, ".agents", "skills", "old-excluded");
|
||||
await symlink(path.join(oldRoot, "skills/old-excluded"), stale);
|
||||
const settings = JSON.parse(await readFile(options.settingsFile, "utf8"));
|
||||
settings.packages.push(path.join(oldRoot, "extensions/old/index.ts"));
|
||||
await writeJson(options.settingsFile, settings);
|
||||
const plan = await buildPlan({ ...options, migrateFrom: oldRoot });
|
||||
assert.deepEqual(plan.operations, []);
|
||||
await assert.rejects(() => run("prune", { ...options, migrateFrom: oldRoot }), /never authorizes pruning/);
|
||||
assert.equal(await readlink(stale), path.join(oldRoot, "skills/old-excluded"));
|
||||
});
|
||||
|
||||
test("migration rejects invalid roots and malformed CLI options", async () => {
|
||||
const options = await fixture();
|
||||
for (const migrateFrom of ["relative", "", options.repoRoot, options.root, path.join(options.repoRoot, "nested")])
|
||||
await assert.rejects(() => buildPlan({ ...options, migrateFrom }), /migrate-from/);
|
||||
for (const args of [["sync", "--migrate-from"], ["sync", "unexpected"], ["sync", "--migrate-from", "/old", "--migrate-from", "/other"]])
|
||||
assert.equal(await main(args), 2);
|
||||
});
|
||||
|
||||
test("duplicate declared skill names fail before planning changes", async () => {
|
||||
const options = await fixture();
|
||||
await writeSkill(
|
||||
|
||||
@@ -1,83 +1,64 @@
|
||||
---
|
||||
name: code-review
|
||||
description: Review changes since a fixed commit, branch, tag, or merge-base along separate repository-standards and specification axes. Use for branch, PR, work-in-progress, or workflow-record review after implementation and verification.
|
||||
description: Independently review changes against repository standards and the requested behavior using a pinned Git diff. Use for branch, PR, work-in-progress, or workflow-record review, with targeted follow-up after fixes.
|
||||
---
|
||||
|
||||
# Code review
|
||||
|
||||
Run two independent, read-only reviews over one pinned diff:
|
||||
Use one independent, read-only reviewer by default. Assess two axes and report them separately:
|
||||
|
||||
- **Standards:** repository instructions and documented conventions.
|
||||
- **Specification:** the originating issue, spec, or approved workflow record.
|
||||
- **Specification:** the originating request, issue, spec, or workflow record.
|
||||
|
||||
Keep the axes separate so conformity cannot hide incorrect scope and correct scope cannot hide standards violations.
|
||||
Add specialist reviewers when distinct risk or repository policy warrants them, not merely because there are two axes. The agent that authored or mutated the diff cannot review it or its own remediation.
|
||||
|
||||
## 1. Pin the review bytes
|
||||
|
||||
Require a fixed point; ask when none was supplied. Resolve and record:
|
||||
Resolve a fixed point from the caller's brief or workflow record; ask only when the intended scope cannot be established. Record:
|
||||
|
||||
- review base commit;
|
||||
- immutable product HEAD commit, never mutable `HEAD` in the reviewer brief;
|
||||
- clean status and product tree hash;
|
||||
- `git log <base>..<product-head> --oneline`;
|
||||
- exact command `git diff --binary --full-index --find-renames <base>...<product-head> --`.
|
||||
- review base and immutable product HEAD commits, never mutable `HEAD` in the brief;
|
||||
- clean status, product tree hash, and `git log <base>..<product-head> --oneline`;
|
||||
- exact command `git diff --binary --full-index --find-renames <base>...<product-head> --`;
|
||||
- SHA-256 of those exact bytes, generated by the parent.
|
||||
|
||||
Run that command once in the parent, hash its exact bytes with SHA-256, and record the digest. Stop on a bad ref, dirty product state, or empty diff unless an empty comparison was explicitly expected.
|
||||
Stop on a bad ref, dirty product state, or empty diff unless an empty comparison was expected. During a development workflow, review after focused checks and before the final full suite so fixes do not cause scheduled duplicate suite runs. Independent review does not establish test success.
|
||||
|
||||
## 2. Resolve specification and standards
|
||||
|
||||
Accept the first available specification source:
|
||||
Use the caller's explicit request, supplied spec, or retained workflow record as the primary specification. Otherwise inspect linked issues (using repository issue-tracker instructions when configured) or matching repository docs. Ask only when the intended behavior remains unclear. A workflow record needs no issue-tracker configuration or ceremonial plan approval to be a valid specification.
|
||||
|
||||
1. a caller-supplied approved workflow record or spec path;
|
||||
2. issue references in commits, using `docs/agents/issue-tracker.md` when configured;
|
||||
3. a matching file under `docs/`, `specs/`, or `.scratch/`;
|
||||
4. an explicit user statement that no specification exists.
|
||||
If the user explicitly requests standards-only review without a specification, report “specification not assessed”; do not claim both axes passed.
|
||||
|
||||
A workflow record is a complete specification source and never requires issue-tracker configuration. Ask only when no source or explicit no-spec decision exists. Skip the specification reviewer and report “no specification available” only after that explicit decision.
|
||||
Collect applicable `AGENTS.md`, `CLAUDE.md`, contributing, coding-standard, and ADR sources. Where repository standards do not override it, consider this judgment-based baseline and skip tooling-enforced issues:
|
||||
|
||||
Collect applicable `AGENTS.md`, `CLAUDE.md`, contributing, coding-standard, and ADR sources. Add this smell baseline where repository standards do not override it; treat each as a judgment call and skip tooling-enforced issues:
|
||||
- Names hide purpose; duplicated logic obscures a shared contract.
|
||||
- Feature envy, data clumps, primitive obsession, or repeated switches complicate change.
|
||||
- Shotgun surgery or divergent responsibilities scatter one change across unrelated areas.
|
||||
- Speculative generality, middlemen, message chains, or inappropriate inheritance add indirection without a current need.
|
||||
|
||||
- **Mysterious Name:** a name hides purpose.
|
||||
- **Duplicated Code:** the same logic shape recurs.
|
||||
- **Feature Envy:** behavior reaches into another object's data more than its own.
|
||||
- **Data Clumps:** fields or parameters repeatedly travel together.
|
||||
- **Primitive Obsession:** primitives stand in for a domain concept.
|
||||
- **Repeated Switches:** the same type cascade recurs.
|
||||
- **Shotgun Surgery:** one change requires scattered edits.
|
||||
- **Divergent Change:** one module changes for unrelated reasons.
|
||||
- **Speculative Generality:** abstraction exists without a current requirement.
|
||||
- **Message Chains:** callers navigate long object chains.
|
||||
- **Middle Man:** code delegates without adding value.
|
||||
- **Refused Bequest:** an implementation ignores most inherited behavior.
|
||||
Distinguish consequential defects and documented violations from optional style preferences. Do not require speculative abstractions, compatibility paths, or exhaustive tests without a concrete contract or risk. Judge coverage against changed behavior and credible failures.
|
||||
|
||||
## 3. Enforce reviewer independence
|
||||
## 3. Enforce independence and read-only review
|
||||
|
||||
List every display name, profile, child/session ID, and ambiguous run that authored or mutated the diff. Select two different reviewer identities outside that set. Require repository read access plus Git-capable `bash`, with no declared `write`, `edit`, or nested-agent capability.
|
||||
List every author/mutator by display name and stable child/session identity when exposed. Select a reviewer outside that set with repository read access and Git-capable `bash`, but no declared `write`, `edit`, or nested-agent tools. When identity cannot be distinguished, exclude the ambiguous profile/run rather than assuming independence.
|
||||
|
||||
Capture branch, HEAD, status, tree hash, and changed paths before review. Launch the standards and specification reviewers as separate direct subagent calls rather than a wrapped parallel call. Give both the pinned commits, exact diff command, expected digest, commit list, mutator exclusions, and identity-reporting requirement.
|
||||
Capture branch, HEAD, status, tree hash, and changed paths before review. Launch the reviewer directly with the pinned commits, exact diff command and digest, specification, standards, mutator exclusions, and identity-reporting requirement. Require the reviewer to:
|
||||
|
||||
Require each reviewer to:
|
||||
1. Regenerate the exact diff and confirm its SHA-256 before analysis.
|
||||
2. Remain read-only, run no mutating commands, and perform no remediation.
|
||||
3. Assess both axes, grounding findings in file/hunk evidence and severity; quote the requirement for specification findings.
|
||||
4. Report identity when exposed and stay under 400 words unless complexity warrants more.
|
||||
|
||||
1. regenerate the exact diff and confirm its SHA-256 before analysis;
|
||||
2. remain read-only and perform no remediation;
|
||||
3. bind every finding to file/hunk evidence and severity;
|
||||
4. report display/profile/child/session identity when exposed;
|
||||
5. stay under 400 words.
|
||||
After delivery, verify Git state is unchanged. Missing output, digest mismatch, mutator overlap, or reviewer mutation means no valid verdict. Mutation also makes that reviewer ineligible for subsequent review of the diff.
|
||||
|
||||
For standards, include every standards source and the full baseline above. Ask for documented violations separately from baseline judgments. For specification, include the approved source and ask for missing/partial requirements, scope creep, and apparently implemented but incorrect behavior with quoted requirements.
|
||||
## 4. Report findings and remediate
|
||||
|
||||
Treat missing delivery, digest mismatch, reviewer/mutator identity overlap, or before/after repository mutation as no verdict. Any reviewer mutation also makes that identity a mutator.
|
||||
Present `## Standards` and `## Specification`, separating documented violations from judgment-based concerns. Preserve each axis's findings and severity; report approval explicitly when no material findings remain. Keep optional suggestions non-blocking unless tied to a requirement or concrete risk.
|
||||
|
||||
## 4. Aggregate without reranking
|
||||
Validate findings against repository evidence. An original implementer or another non-reviewer makes accepted fixes, then runs affected checks. Use [verification policy](../develop-workflow/references/verification.md) for evidence reuse and final-suite timing; obey stricter repository requirements.
|
||||
|
||||
Present the reports under `## Standards` and `## Specification`. Lightly clean formatting only; keep findings and axis severity separate. End with finding counts and the worst issue within each axis, not one combined winner.
|
||||
For bounded corrections, resume the same independent reviewer with the previous verdict, old and new pinned commits/digests, fix delta, and check results. Require it to inspect fixes and affected interactions, confirm the new full-diff fingerprint, and state which prior conclusions remain applicable. A prior verdict alone never approves later product changes.
|
||||
|
||||
Bind approval to the base, product HEAD, clean status, tree hash, and diff digest.
|
||||
Request a broader fresh review when scope, architecture, shared behavior, or uncertain interactions invalidate previous conclusions; use a new specialist identity only when useful or required. Ask the user only for changed requirements, approval-boundary risk, or an unresolved decision, not because an automatic remediation cycle count expired. Stop repetitive ineffective retries, reassess the cause, and surface blockers honestly.
|
||||
|
||||
## 5. Remediate through non-reviewers
|
||||
|
||||
Validate findings against repository evidence. Assign accepted fixes to an original implementer or another non-reviewer. Rerun affected checks and the full required suite, create a new pinned fingerprint, and launch two fresh independent reviewers.
|
||||
|
||||
Escalate after two unsuccessful remediation cycles or when a finding materially changes scope, acceptance criteria, ownership, verification, or architecture. Return material changes to plan approval before production mutation.
|
||||
|
||||
**Complete when:** both axes approve the exact verified fingerprint and before/after evidence proves reviewers stayed read-only.
|
||||
**Complete when:** independent review covers the final pinned product diff, applicable axes have no unresolved material findings, and before/after evidence proves the reviewer stayed read-only. Final test completion remains a separate integration requirement.
|
||||
|
||||
@@ -1,97 +1,88 @@
|
||||
---
|
||||
name: develop-workflow
|
||||
description: Runs feature work and other multi-step code changes through repository research, an approved implementation plan, isolated Git worktrees, explicit UI and risk gates, verification, independent review, and local integration. Use by default for features, non-trivial bug fixes, refactors, migrations, dependency changes, security-sensitive work, and UI changes. Excludes questions, research-only requests, and obvious low-risk edits requiring at most two steps.
|
||||
description: Runs feature work, bug fixes, refactors, migrations, dependency changes, security-sensitive work, and UI changes through risk-based planning, isolated worktrees, proportionate verification, independent review, and local integration. Use by default for implementation work; keep questions read-only and straightforward low-risk edits lightweight.
|
||||
---
|
||||
|
||||
# Develop workflow
|
||||
|
||||
Own orchestration in the parent agent. Treat Git and the retained workflow record as recovery evidence; use task tools only as the live progress view. Give subagents bounded assignments, then verify their claims against repository state.
|
||||
Own orchestration in the parent agent. Use Git and one retained record for recovery, task tools for live progress, and subagents only when their benefit exceeds coordination cost. Proceed within clear requirements and established patterns; reserve human interaction for decisions and meaningful risk.
|
||||
|
||||
## 0. Classify the request
|
||||
## 0. Classify by impact
|
||||
|
||||
Choose the smallest sufficient path before mutation:
|
||||
|
||||
- **Answer:** explain or research read-only. Make no changes.
|
||||
- **Light:** make one obvious low-risk edit requiring at most two steps. Inspect, edit, run focused checks, and summarize. Record why this is safe when near the boundary.
|
||||
- **Full:** use the phases below for three or more steps, multiple files or modules, ambiguity, architecture, dependencies, migrations, security implications, non-trivial UI, or meaningful regression risk.
|
||||
- **Answer:** explain or research read-only.
|
||||
- **Light:** a straightforward, bounded edit with understood behavior and low regression risk. Inspect, edit, run relevant checks, and summarize. Respect repository isolation and review requirements.
|
||||
- **Standard:** feature work or changes needing coordinated implementation, verification, or recovery. Follow the phases below. Multiple files or steps alone do not imply high risk or require approval.
|
||||
|
||||
Promote light work to full before editing when discovery reveals meaningful scope or risk.
|
||||
For either implementation path, assess affected consumers, data value, security, reversibility, and uncertainty. Record the reason when classification is near the boundary. Escalate the affected decision when new evidence reveals risk; do not restart the whole workflow for routine execution adjustments.
|
||||
|
||||
**Complete when:** observed scope and risk justify the selected path.
|
||||
**Complete when:** scope and impact justify the path and any pending decisions are identified.
|
||||
|
||||
## Risk gates for every path
|
||||
## Approval boundaries for every path
|
||||
|
||||
Ask just in time before adding, removing, or materially upgrading dependencies; schema/data migration; production or live-database access; public API or compatibility breaks; destructive Git/filesystem actions outside the narrowly proven final cleanup; force push or shared-history rewrite; non-fast-forward integration; conflict resolution that changes approved behavior; material scope or acceptance expansion; security-posture changes; publishing; or deployment.
|
||||
Honor repository and global instructions. Ask for unresolved product or preference-sensitive design choices, material scope expansion, security-posture changes, externally breaking contracts, production/live-database access, protected data, publishing/deployment, unsafe cleanup, force operations, shared-history rewrite, non-fast-forward integration, or conflict resolution that changes agreed behavior. Dependency additions, removals, and material upgrades still need scoped authorization unless already explicitly approved.
|
||||
|
||||
Present concrete evidence, scope the authorization, and mark one option recommended with its reason. Checkpoint the decision before acting. Treat approval as specific to that action; ask again when later evidence broadens it. Promote light work to full before editing when meaningful risk appears.
|
||||
For schema/data work, apply [data policy](references/data-policy.md): a standing authorization may cover named disposable targets, but pre-production alone grants no permission. Verify the actual target before any authorized operation.
|
||||
|
||||
Reuse an explicit authorization within its scope instead of asking again at execution. Ask again only when evidence changes that scope or a rule explicitly requires a fresh decision. Present concrete evidence, a recommended option with its reason, and record the answer before acting. Routine worker reassignment after proven termination, equivalent checks, or internal implementation adjustments under established patterns are not new product decisions.
|
||||
|
||||
## 1. Preflight and isolate
|
||||
|
||||
For the full path, require Git worktrees, a writable repository record, `ask_user`, Git-capable independent reviewers, and target-interface validation when UI applies. Stop before product mutation when a mandatory capability is unavailable. Let task tools degrade to the retained record and let `artifact-presenter` degrade to reporting the exact artifact path; neither optional convenience may bypass a gate. In print or JSON mode, emit the pending gate or validation instructions and stop.
|
||||
Read repository instructions, relevant docs/ADRs/tests, and Git state. For standard work, require an isolated Git worktree, a writable record, and a Git-capable independent reviewer. Require target-interface validation for UI changes. Follow [Git and integration](references/git-and-integration.md) for isolation and dirty-source handling.
|
||||
|
||||
Read repository instructions, relevant docs/ADRs/tests, and Git state. Follow [Git and integration](references/git-and-integration.md) to capture source evidence, resolve dirty-source approval, and establish the isolated branch/worktree. Prepare a unique record from [`assets/workflow-record.md`](assets/workflow-record.md) and publish it with the helper in [evidence and recovery](references/evidence-and-recovery.md). Create the available live progress view, run applicable documentation checks, and commit the initial record before product mutation.
|
||||
Create a unique record from [`assets/workflow-record.md`](assets/workflow-record.md) using [evidence and recovery](references/evidence-and-recovery.md). Create the available task view. Save the record before product mutation; a separate initial commit is unnecessary.
|
||||
|
||||
**Complete when:** worktree identity is proven and the initial record commit names the next safe action.
|
||||
Stop only at an unavailable capability needed for the next action. Noninteractive runs may proceed within clear authorization; if a human decision is pending and `ask_user` is unavailable, report the decision and stop before dependent mutation. Task tools may degrade to the record, and artifact presentation to an exact path.
|
||||
|
||||
## 2. Discover and decide requirements
|
||||
**Complete when:** isolation and recovery evidence exist and the next action is authorized.
|
||||
|
||||
Launch independent read-only research lanes directly when concerns can be investigated separately; state that no files are owned. Give each child a self-contained prompt. Keep the parent active on other work and verify that every required result was delivered. Treat missing delivery as missing evidence.
|
||||
## 2. Discover and plan
|
||||
|
||||
Reconcile repository evidence into scope, out-of-scope items, acceptance criteria, affected surfaces, constraints, and risks. Use `ask_user` only for decisions repository evidence cannot answer. Checkpoint decisions.
|
||||
Resolve scope, acceptance criteria, affected surfaces, existing seams, and risks from repository evidence. Launch read-only research lanes only for useful independent questions; state that they own no files. Verify required results were delivered.
|
||||
|
||||
**Complete when:** requirements and unresolved questions are explicit.
|
||||
Record a concise plan with implementation slices, path ownership, focused checks, final required verification, UI/data classification, outstanding decisions, and local integration/rollback. Use [proportionate verification](references/verification.md) to choose tests. Give mutating children bounded assignments, normally 15 minutes; the parent may adjust the bound to the task without a user gate.
|
||||
|
||||
## 3. Design and obtain plan approval
|
||||
For clear routine work, state the plan and proceed. Present a plan through `artifact-presenter` when human review is needed, then use `ask_user` for the unresolved choices. An explicit user approval already supplied in conversation counts. Revisit only decisions affected by changed evidence; changing an owner or equivalent verification command does not itself require reapproval.
|
||||
|
||||
Record an exact implementation plan containing:
|
||||
**Complete when:** the record states the intended outcome, checks, ownership, and authority to proceed.
|
||||
|
||||
- acceptance criteria mapped to slices;
|
||||
- one named owner and concrete path ownership per mutating assignment;
|
||||
- existing seams and patterns;
|
||||
- focused and full verification commands with realistic data needs;
|
||||
- UI classification, dependencies, migrations, production effects, and risk gates;
|
||||
- review fixed point and specification source;
|
||||
- integration and rollback strategy;
|
||||
- a 15-minute default deadline per mutating assignment, or an explicitly approved alternative.
|
||||
## 3. Resolve non-trivial UI direction
|
||||
|
||||
Present the complete record through `artifact-presenter`, then obtain explicit approval with `ask_user`. Checkpoint and commit the decision before production edits. Return to this gate after any material scope, acceptance, ownership, verification, or architecture change.
|
||||
For non-trivial UI, invoke `prototype` and create three structurally different throwaway candidates before production-component edits. Ask approval for a smaller count when three meaningful alternatives are impossible. Validate candidates in the browser or real PTY/application, including representative responsive sizes. Prefer fast-browser for supported local web validation; use browser-harness when explicitly requested or its additional capabilities are needed.
|
||||
|
||||
**Complete when:** the user explicitly approved the exact committed plan.
|
||||
Retain candidates on a dedicated throwaway branch and record its commit. Present them, obtain direction approval through `ask_user`, and checkpoint the choice. Rewrite the chosen direction as production code. Record why minor or non-visual UI skips prototyping; changed UI still needs actual interface validation.
|
||||
|
||||
## 4. Resolve non-trivial UI direction
|
||||
**Complete when:** the selected direction and target-interface evidence are recorded before production UI mutation.
|
||||
|
||||
For non-trivial UI, invoke `prototype` and create three structurally different throwaway candidates before production-component edits. Ask approval for a smaller explicit count when three meaningful alternatives are impossible. Validate candidates through `browser-harness` for web UI or the real PTY/application for terminal/native UI, including representative responsive sizes.
|
||||
## 4. Implement and run focused checks
|
||||
|
||||
Retain candidates on a dedicated throwaway branch and record its commit. Present them and obtain direction approval through `ask_user`. Checkpoint the selected direction before production mutation. Rewrite the chosen direction as production code rather than promoting prototype code. Record why minor or non-visual UI skips this phase.
|
||||
State one owner and exclusive paths for each mutating assignment. Serialize by default; parallelize disjoint low-integration paths. Give workers the plan, decisions, acceptance criteria, test seam, commands, and deadline. Leave commits, integration, user interaction, and mutation delegation to the parent.
|
||||
|
||||
**Complete when:** target-interface evidence, retained prototype commit, selected direction, and reflected plan are recorded.
|
||||
Follow [assignment evidence](references/evidence-and-recovery.md#assignment-evidence). Use TDD when appropriate under its agreed-seam rules, rather than invoking it for every edit. Run focused checks after meaningful behavioral slices; reuse delivered evidence when its inputs remain valid. Inspect returned diffs and repository state before accepting work.
|
||||
|
||||
## 5. Implement owned slices
|
||||
**Complete when:** acceptance criteria have intentional implementation and proportionate focused evidence.
|
||||
|
||||
State one owner and exclusive paths for every mutating assignment. Serialize by default; parallelize only disjoint low-integration paths. Give each implementer the approved plan, decisions, acceptance criteria, test seam, allowed commands, and deadline. Tell workers to stop at the deadline and leave commits, integration, user interaction, and mutation delegation to the parent.
|
||||
## 5. Synchronize and independently review
|
||||
|
||||
Follow [assignment evidence and recovery](references/evidence-and-recovery.md#assignment-evidence) before accepting, retrying, or reassigning mutation. Use TDD at agreed seams where practical. Run focused tests after behavioral slices. Inspect each returned diff and repository state before marking it complete.
|
||||
Inspect the complete diff, synchronize affected documentation, and follow [Git and integration](references/git-and-integration.md) to fetch/rebase before final verification where practical. Run checks affected by synchronization, then create focused commits after their applicable checks pass.
|
||||
|
||||
**Complete when:** every acceptance criterion has intentional owned code and test evidence.
|
||||
Follow [review and remediation](references/review-and-remediation.md). One independent, read-only reviewer covers standards and specification. Resolve material findings through a non-reviewer, run affected checks, and obtain targeted follow-up. Routine fixes do not require two new reviewers or a full-suite rerun before each review.
|
||||
|
||||
## 6. Verify the integrated worktree
|
||||
**Complete when:** the final product diff has independent approval covering both axes and all material fixes.
|
||||
|
||||
Inspect the complete diff and status. Run focused behavioral tests, then repository-required typecheck, lint, build, and full tests. Exercise realistic E2E data across process/UI boundaries. Validate changed UI in its target interface and representative sizes. Synchronize affected documentation.
|
||||
## 6. Run final verification
|
||||
|
||||
Record exact commands, exit codes, and concise outcomes. Keep flaky or environmental failures visible until resolved or escalated. Never infer success from a child transcript.
|
||||
After review fixes, run the full repository-required test suite once against the integrated worktree, plus applicable typecheck, lint, and build checks. Use [verification evidence](references/verification.md) to reuse valid results and rerun invalidated checks. A failure is not permission to skip completion; fix it, obtain review for changed behavior, and rerun the necessary checks.
|
||||
|
||||
**Complete when:** the approved verification matrix passes against the integrated worktree state.
|
||||
Validate changed UI in its target interface and important cross-boundary behavior with realistic data. Record commands, exit codes, and concise outcomes. Keep flaky/environmental failures visible until resolved or explicitly escalated. Distinguish a policy-authorized waiver from a pass.
|
||||
|
||||
## 7. Commit, synchronize, and review
|
||||
**Complete when:** required final checks passed or exceptions were explicitly authorized under repository policy, and review still covers the final product state.
|
||||
|
||||
Create focused commits after their checks pass. Follow [Git and integration](references/git-and-integration.md) to fetch/rebase, reverify changed trees, and produce the pinned review brief and exact diff fingerprint. Follow [review and remediation](references/review-and-remediation.md) for two independent review axes, mutator exclusion, findings, and fresh-review rules.
|
||||
## 7. Integrate and close
|
||||
|
||||
**Complete when:** standards and specification reviewers approve the exact verified product diff.
|
||||
Follow [Git and integration](references/git-and-integration.md) for clean fast-forward integration, evidence updates, and proven-safe cleanup. Batch record changes with meaningful commits; a separate metadata closure commit is optional, not a gate. A later evidence-only record update cannot silently change requirements or executable instructions.
|
||||
|
||||
## 8. Integrate and close
|
||||
Mark tasks complete and report commits, verification outcomes, review result, and record path concisely. Preserve detailed identity and fingerprint evidence in the record rather than repeating it in chat. Push, publish, deployment, and other protected actions remain outside local integration unless explicitly authorized.
|
||||
|
||||
Follow [Git and integration](references/git-and-integration.md). Recheck the source branch and checkout, fast-forward only when evidence remains valid, then create one metadata-only closure commit touching only the record. Run a cheap final sanity check. Remove the workflow-owned worktree and branch only after reachability and clean-state proofs.
|
||||
|
||||
Mark tasks complete and report product/closure commits, review command and digest, reviewer identities, record path, and verification evidence. Keep push, publish, deployment, production access, unsafe cleanup, conflict-driven behavior changes, force operations, and non-fast-forward integration behind explicit just-in-time approval.
|
||||
|
||||
**Complete when:** reviewed work and closure are reachable from the intended local source branch and temporary state is safely removed.
|
||||
**Complete when:** reviewed work and retained evidence are reachable from the intended local branch and workflow-owned temporary state is safely removed.
|
||||
|
||||
@@ -28,19 +28,20 @@
|
||||
|
||||
- Plan/specification source:
|
||||
- Exact plan:
|
||||
- Plan approval:
|
||||
- Authority to proceed: clear request | standing policy | explicit approval
|
||||
- Pending decisions (none when proceeding):
|
||||
- UI classification and direction approval:
|
||||
- Risk decisions and scope:
|
||||
|
||||
## Ownership
|
||||
|
||||
| Assignment | Owner identity | Deadline | Exclusive paths | Before/after Git evidence | Outcome |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
- Parent-owned paths:
|
||||
- Child assignments, when useful: owner/identity, paths, deadline, before/after Git evidence, outcome.
|
||||
|
||||
## Verification matrix
|
||||
|
||||
| Surface | Command | Exit code | Outcome |
|
||||
| --- | --- | --- | --- |
|
||||
| Surface | Command and cwd | Tested state/environment | Exit code | Outcome or reuse rationale |
|
||||
| --- | --- | --- | --- | --- |
|
||||
|
||||
## Review
|
||||
|
||||
@@ -50,21 +51,22 @@
|
||||
- Exact diff command:
|
||||
- SHA-256:
|
||||
- Mutator identities:
|
||||
- Standards reviewer identity, digest confirmation, and verdict:
|
||||
- Specification reviewer identity, digest confirmation, and verdict:
|
||||
- Findings and remediation cycles:
|
||||
- Reviewer identity and digest confirmation:
|
||||
- Standards verdict:
|
||||
- Specification verdict:
|
||||
- Findings and targeted follow-up:
|
||||
- Before/after reviewer Git evidence:
|
||||
|
||||
## Integration and rollback
|
||||
|
||||
- Synchronization and re-verification:
|
||||
- Fast-forward result:
|
||||
- Product and closure commits:
|
||||
- Product and evidence commits:
|
||||
- Reachability and cleanup:
|
||||
- Rollback strategy:
|
||||
|
||||
## Checkpoints and blockers
|
||||
|
||||
- Last committed valid checkpoint:
|
||||
- Latest saved checkpoint and last committed checkpoint (distinguish):
|
||||
- Blockers:
|
||||
- Exact next safe action:
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
# Disposable and protected data
|
||||
|
||||
Pre-production is a lifecycle stage, not permission to discard data. Disposable means the owner authorizes replacement of specified data and it can be reproduced. Idempotent describes an operation's repeated effect; it does not establish disposability.
|
||||
|
||||
## Establish authority
|
||||
|
||||
Read the repository's existing instructions for a standing data policy. It must identify:
|
||||
|
||||
- named disposable environments and how to verify the actual connection target;
|
||||
- the data/resources that may be deleted or rebuilt;
|
||||
- the authorized reset/reseed procedure and source of required fixtures;
|
||||
- protected environments, valuable/shared data, and external compatibility obligations.
|
||||
|
||||
Treat this as scoped authorization, not a reason to reset unnecessarily. Before each operation, verify the actual target matches the declaration, the operation stays within the allowed resources, and required data can be recreated. Stop and ask when identity, scope, or reproducibility is uncertain. Do not infer permission from a branch name, `NODE_ENV`, a database name containing “dev,” or the absence of production users.
|
||||
|
||||
When no policy exists, ask for the intended preservation contract and explicit authority before schema/data changes. Record any approved standing policy in the repository's established instructions; never label an environment disposable on the agent's own authority. Production/live-database access remains explicitly gated even if another target is disposable.
|
||||
|
||||
## Implement the smallest sufficient change
|
||||
|
||||
For a verified, authorized disposable target, prefer a direct schema change plus reset/reseed over compatibility adapters, phased backfills, dual writes, or bespoke rollback machinery. Keep migration files when the framework needs them to create a fresh database. Verify initialization/rebuild and affected behavior with representative data.
|
||||
|
||||
For protected data or external consumers, preserve the actual contract. Design migration, backup/recovery, compatibility, and verification around demonstrated risks and obtain scoped approval. Shared staging data with valuable manual QA state is protected unless its owner explicitly authorizes replacing it.
|
||||
|
||||
Example: a required field in an authorized throwaway database may need an updated schema and seed data, a rebuild, and a feature check. It does not automatically need tests for upgrading every historic schema. The same field in a customer-backed service may require a staged migration and compatibility checks.
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
## Checkpoint records
|
||||
|
||||
Keep one concise Markdown record in the repository's established plan location; otherwise use `docs/plans/<feature-branch-slug>.md`. Require a unique path before creation. Store status/current phase, request/scope, Git identity, acceptance criteria, exact approved plan and gate decisions, assignment ownership, verification outcomes, review identities/findings/fingerprint, integration, blockers, and the next safe action.
|
||||
Keep one concise Markdown record in the repository's established plan location; otherwise use `docs/plans/<feature-branch-slug>.md`. Require a unique path before creation. Store status/current phase, request/scope, Git identity, acceptance criteria, the plan and its authority (clear user request, standing policy, or explicit decision), ownership, verification outcomes, review identity/findings/fingerprint, integration, blockers, and the next safe action. Retain the template's headings for helper compatibility; use short entries or “not applicable” rather than inventing ceremony for every section.
|
||||
|
||||
Keep transcripts, leases, heartbeats, notification receipts, full command output, and model-authored execution graphs out of the record. Link durable artifacts or commits instead. Update only at meaningful boundaries; task tools hold high-frequency progress.
|
||||
|
||||
@@ -13,7 +13,7 @@ node skills/develop/develop-workflow/scripts/checkpoint-record.mjs \
|
||||
<record.md> --input <candidate.md> --expected-sha256 <current-digest>
|
||||
```
|
||||
|
||||
Omit `--expected-sha256` only for first creation; require an absent target. Commit the initial record and every approved gate before dependent product mutation. The exclusive lock serializes cooperating helper invocations; external mutation of its pathname is outside the contract. A stale digest, lock, temporary artifact with no valid record, validation failure, or filesystem error blocks progress. Preserve artifacts for inspection; never hand-delete recovery evidence to make a command pass.
|
||||
Omit `--expected-sha256` only for first creation; require an absent target. Save the initial plan and scoped decisions before dependent mutation; batch record changes with meaningful commits instead of committing every gate. The exclusive lock serializes cooperating helper invocations; external mutation of its pathname is outside the contract. A stale digest, lock, temporary artifact with no valid record, validation failure, or filesystem error blocks dependent progress. Preserve artifacts for inspection; never hand-delete recovery evidence to make a command pass.
|
||||
|
||||
## Assignment evidence
|
||||
|
||||
@@ -21,11 +21,11 @@ Before every mutating assignment, record:
|
||||
|
||||
- display name, profile, stable child/session IDs when exposed;
|
||||
- deadline and exclusive owned paths;
|
||||
- branch, HEAD, clean/dirty status, changed paths, and patch artifact.
|
||||
- branch, HEAD, clean/dirty status, and changed paths; retain a patch artifact when needed to distinguish concurrent or interrupted edits.
|
||||
|
||||
After delivery, capture the same Git evidence and inspect the patch. Treat the entire ambiguous profile/run as a mutator when stable identity is unavailable. Reject unexpected commits and paths outside ownership.
|
||||
|
||||
On an ownership or Git invariant failure, freeze mutation and present evidence-backed choices: inspect, accept and replan, restore with a non-destructive patch, or abort. Obtain a user decision before proceeding.
|
||||
On an ownership or Git invariant failure, freeze affected mutation and inspect the evidence. Resume automatically only when ownership is re-established, prior children are terminal, and the next action preserves all work within existing authority. Ask with evidence-backed options when recovery would discard work, change agreed behavior, accept uncertain ownership, or broaden scope.
|
||||
|
||||
On deadline expiry or missing delivery, request stop through the available control surface and prove the process/session terminal before reassignment. A completed UI indicator is not output or liveness proof. Keep the workflow blocked for human termination/takeover when liveness remains unknown.
|
||||
|
||||
@@ -37,9 +37,9 @@ Reconstruct observed reality before trusting prose:
|
||||
2. Discover candidate records from observed branches/worktrees and repository conventions.
|
||||
3. Treat a workflow branch/worktree without a record as an orphan. Preserve it; inspect branch, base, reflog, status, commits, and diff; ask whether to adopt it with a record, quarantine it, or remove it after separate cleanup approval.
|
||||
4. Compare the record's base, branch, and checkpoint with Git evidence.
|
||||
5. Preserve atomic-update temporary files. Use the last committed valid record until recovery is approved.
|
||||
5. Preserve atomic-update temporary files. Prefer the latest valid saved record only after reconciling it with Git; distinguish saved from committed checkpoints. When interrupted-write evidence conflicts, use a committed valid record as a fallback if available, then inspect and resolve uncertainty before mutation. Absence of an initial record commit alone is not an orphan or approval gate.
|
||||
6. Rebuild task state from completed evidence and the recorded next action.
|
||||
7. Freeze interrupted mutation until its diff is inspected and prior child liveness is terminal.
|
||||
8. Resume from the last proven checkpoint.
|
||||
|
||||
Ask when evidence conflicts, scope may have changed, child liveness is unknown, or recovery needs risk. Treat the record as a recovery map, not a transaction log.
|
||||
Ask when conflicting evidence remains unresolved, scope needs a new decision, child liveness cannot be proven, or recovery crosses an approval boundary. Resume routine work automatically when inspection restores the evidence within existing authority. Treat the record as a recovery map, not a transaction log.
|
||||
|
||||
@@ -2,39 +2,28 @@
|
||||
|
||||
## Isolation
|
||||
|
||||
Record the source branch and exact base before creating `./.worktrees/<feature>` on `feature/<feature>`. Prove every shell location with explicit `cd` or `git -C`; verify `git rev-parse --show-toplevel` before mutation. Block on collisions, unmerged paths, unexpected ownership, or an unignored worktree root. Never rename, reset, clean, stash, copy source changes, or remove an orphan silently.
|
||||
Record the source branch and exact base before creating `./.worktrees/<feature>` on `feature/<feature>`. Prove shell locations with explicit `cd` or `git -C`; verify `git rev-parse --show-toplevel` before mutation. Block on collisions, unmerged paths, unexpected ownership, or an unignored worktree root. Never rename, reset, clean, stash, copy source changes, or remove an orphan silently.
|
||||
|
||||
A dirty source checkout requires bounded status/path evidence and explicit approval because its changes are omitted. Preserve that evidence in the record.
|
||||
A dirty source checkout requires bounded status/path evidence and explicit approval because its changes are omitted. Preserve that evidence in the record. Do not disturb another workflow's worktree or changes.
|
||||
|
||||
## Synchronize and pin review
|
||||
|
||||
After focused commits and verification:
|
||||
Fetch the latest source branch when a remote exists and rebase onto the intended latest source commit before final verification where practical. Inspect any resulting delta and apply [verification invalidation](verification.md#reuse-and-invalidation); rerun affected checks rather than restarting all phases automatically. Ask before resolving conflicts that change agreed behavior, rewriting shared history, force operations, or non-fast-forward integration.
|
||||
|
||||
1. Fetch the latest source branch when a remote exists.
|
||||
2. Rebase the feature branch onto the latest source commit.
|
||||
3. Rerun affected checks and full required verification whenever rebase changes the effective tree or resolves conflicts.
|
||||
4. Record the review base, product HEAD, clean status, tree hash, and commit list.
|
||||
5. Generate exact review bytes:
|
||||
Create focused commits after their applicable checks pass. Pin the review base, product HEAD, clean status, tree hash, commit list, exact binary diff command, and digest using [review policy](review-and-remediation.md). Keep product approval attached to immutable evidence, including targeted follow-up on subsequent fixes.
|
||||
|
||||
```bash
|
||||
git diff --binary --full-index --find-renames <base>...<product-head> --
|
||||
```
|
||||
|
||||
6. Hash those bytes with SHA-256 and record the command and digest. Regenerate from the durable commits rather than committing a duplicate patch.
|
||||
|
||||
Ask before conflict resolution changes approved behavior, history rewrite of shared work, force operations, or non-fast-forward integration.
|
||||
Update the record at meaningful boundaries and batch it with related commits. Neither an initial record-only commit nor a commit per decision is mandatory. A saved, uncommitted record is recovery evidence, but verify it against Git rather than assuming it was previously committed.
|
||||
|
||||
## Fast-forward integration
|
||||
|
||||
After both review axes approve the exact fingerprint:
|
||||
After independent review and final required verification:
|
||||
|
||||
1. Confirm the source branch still equals the final rebase base. Return to synchronize, verify, and review if it advanced.
|
||||
1. Confirm the source branch still equals the integration base. If it advanced, inspect the delta, synchronize, invalidate affected checks, and refresh review coverage for changed interactions. Reuse unaffected evidence only with justification.
|
||||
2. Confirm the source checkout exists, is clean, and is safe to update. Stop when dirty or missing.
|
||||
3. Fast-forward the source branch to the reviewed feature branch; create no merge commit.
|
||||
4. In the source checkout, update only the workflow record with bound review and integration evidence.
|
||||
5. Prove the closure diff touches only that record, run applicable documentation checks, and create one metadata-only closure commit.
|
||||
6. Run a cheap final sanity check.
|
||||
7. Prove the product and closure commits reachable from the source branch.
|
||||
8. Remove the completed worktree, then delete its feature branch.
|
||||
3. Confirm the final product changes are covered by review and verification. Evidence-only updates to the workflow record may follow review only after proving they change results/status, not scope, acceptance, policy, or executable instructions. Review other changes normally.
|
||||
4. Fast-forward the source branch to the feature branch; create no merge commit.
|
||||
5. Save integration evidence in the record. Batch it with a pending evidence update or use an optional record-only commit when needed to retain final results. Prove any post-review evidence commit touches only the record and run its applicable checks; there is no mandatory separate closure-commit gate.
|
||||
6. Run a cheap sanity check, prove all product and retained evidence commits reachable from the source branch, and verify clean source/worktree state.
|
||||
7. Remove only the workflow-owned worktree, then delete its feature branch.
|
||||
|
||||
Treat cleanup in step 8 as pre-authorized only after every preceding clean-state and reachability proof. Ask before any other destructive cleanup. Pushing, publishing, deployment, and production access remain separate user actions.
|
||||
Cleanup in step 7 is pre-authorized only after the ownership, reachability, and clean-state proofs. Ask before other destructive cleanup. Push, publish, deployment, and production access remain separate scoped actions.
|
||||
|
||||
@@ -1,33 +1,9 @@
|
||||
# Independent review and remediation
|
||||
|
||||
## Identity and capability
|
||||
Use the shared [code-review skill](../../code-review/SKILL.md) as the single source of truth for reviewer selection, pinned evidence, both review axes, findings, and follow-up. One non-author reviewer is the default; additional reviewers require distinct risk or repository policy.
|
||||
|
||||
Exclude every identity that authored or mutated the reviewed diff. Record reviewer display name, profile, child ID, and session ID when available. When stable identity is unavailable, exclude the entire ambiguous profile/run. Keep reviewers read-only and separate for:
|
||||
For workflow reviews, supply the retained record and originating request as the specification, along with repository standards, mutator identities, immutable base/product commits, tree/status evidence, exact binary diff command, and SHA-256. Record the delivered identity and verdict. Leave fixes to an implementer, never the reviewer.
|
||||
|
||||
- **Standards:** repository instructions and documented conventions.
|
||||
- **Specification:** the approved workflow record and linked source plan/spec.
|
||||
Review after focused checks and synchronization, before the final full-suite boundary. For a bounded fix, run affected checks and obtain targeted follow-up from the same independent reviewer covering the new fingerprint and interactions. Broaden review when prior conclusions no longer apply. Keep verification evidence and code-review approval separate: neither substitutes for the other.
|
||||
|
||||
Use Git-capable reviewer profiles with repository read access and no declared `write` or `edit` tools. Shell access can still mutate; capture status, tree, HEAD, and changed paths before and after review. Any mutation invalidates the verdict and marks that identity as a mutator.
|
||||
|
||||
## Review brief
|
||||
|
||||
Generate the brief in the parent. Include:
|
||||
|
||||
- pinned base and product HEAD;
|
||||
- clean status and product tree hash;
|
||||
- commit list;
|
||||
- exact `git diff --binary --full-index --find-renames <base>...<product-head> --` command;
|
||||
- SHA-256 of those exact bytes;
|
||||
- approved record/specification sources;
|
||||
- repository standards sources;
|
||||
- all excluded mutator identities.
|
||||
|
||||
Launch standards and specification reviewers through separate direct subagent calls. Avoid wrapped parallel launch until its delivery defect is proven fixed. Require each reviewer to regenerate the pinned diff, confirm its digest before review, remain read-only, and bind every finding to file/hunk evidence and severity. Treat a missing result or digest mismatch as no verdict.
|
||||
|
||||
Approval binds to base, product HEAD, clean status, tree hash, and diff digest. Permit a later closure commit outside that fingerprint only after proving it changes the workflow record alone.
|
||||
|
||||
## Findings and remediation
|
||||
|
||||
Validate findings against repository evidence. Send accepted fixes to an original implementer or another non-reviewer, never the reviewer. Preserve exclusive path ownership and assignment evidence. Rerun affected checks and the full required suite, create focused commits, and generate a new fingerprint.
|
||||
|
||||
Launch fresh independent reviewers for the changed diff. Escalate after two unsuccessful remediation cycles or whenever a finding requires material scope, acceptance, ownership, verification, or architecture changes; return to plan approval before further production mutation.
|
||||
After review, record-only evidence updates may be excluded only under the narrow [integration rule](git-and-integration.md#fast-forward-integration). Scope, specification, policy, and executable-instruction changes are not evidence-only updates.
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
# Proportionate verification
|
||||
|
||||
## Choose tests by the changed contract
|
||||
|
||||
Cover meaningful behavior, important boundaries, and credible failure modes. Prefer existing test seams and realistic minimal fixtures.
|
||||
|
||||
- Unit tests establish logic and regressions through public interfaces.
|
||||
- Integration tests establish contracts unit tests cannot, such as persistence or process boundaries.
|
||||
- E2E tests establish important user journeys and changed wiring. Avoid replaying every unit-level permutation through the UI.
|
||||
- Validate changed UI in its real interface. One realistic scenario may supply both automated and interface-validation evidence when it actually establishes both.
|
||||
- Simple straightforward code does not need a test for every branch. Avoid assertions that restate implementation, snapshots of incidental details, and regression tests preserving intentionally removed behavior.
|
||||
- For authorized disposable data, verify fresh initialization/reset, representative seed data, and affected behavior. Add historical upgrade/rollback coverage only when data or compatibility must be preserved.
|
||||
|
||||
When test infrastructure threatens to become larger than the feature, reassess the seam and failure cost before expanding it. Do not impose a coverage percentage or time cap that leaves consequential behavior untested. Security, valuable data, and broad shared-code changes can warrant more testing than implementation.
|
||||
|
||||
## Development and final checks
|
||||
|
||||
Run focused checks during implementation and after review fixes. Schedule one full required suite after review/remediation and synchronization, together with applicable static/build checks. Reuse an already completed check instead of running it again merely because a new phase started.
|
||||
|
||||
Before a commit, run the checks appropriate to its changes; earlier review commits may rely on focused checks. The final integration boundary still requires the full suite. Repository or CI rules requiring stricter pre-commit checks take precedence. Pure documentation changes may use documentation checks when repository policy permits; executable skill instructions need policy validation, not arbitrary prose snapshots.
|
||||
|
||||
## Reuse and invalidation
|
||||
|
||||
Record the command, working directory, exit/result, tested commit or tree, and relevant environment/configuration. Evidence is reusable only when the command and relevant code, tests, dependencies, configuration, and environment remain applicable and the process actually completed.
|
||||
|
||||
- A commit-only change, record evidence update, or unrelated prose edit does not automatically invalidate product tests. Check documentation links/formatting as applicable.
|
||||
- A bounded behavior fix requires affected tests and targeted review. Preserve other results only with an evidence-backed impact assessment.
|
||||
- Shared code, dependency/build configuration, conflicting rebases, or uncertain impact require broader checks, including the full suite when its coverage may be affected.
|
||||
- If the source branch advanced, inspect the actual delta. Do not equate a new commit ID with changed test inputs, or assume disjoint filenames imply no interaction.
|
||||
- Failed, aborted, stale, or environmental checks remain visible. Retry with a reason; never convert a failure to a pass by inference.
|
||||
|
||||
“One final full run” removes scheduled duplication, not legitimate reruns after failures or invalidation. If new fixes affect the verified state, rerun the required checks and make the review cover the changed diff before integration.
|
||||
|
||||
## External verification gates
|
||||
|
||||
Inspect repository scripts, CI requirements, and active hook policy before promising evidence reuse. An exact-command hook may require a standalone full command even when a focused command passes. Its requirements remain binding until explicitly changed or waived under its own policy. Record a waiver as skipped, never passed; avoid routine waivers as a substitute for aligning policy. Scope hook/config changes separately rather than inventing a workflow verification cache.
|
||||
@@ -1,10 +1,20 @@
|
||||
---
|
||||
name: browser-harness
|
||||
description: "Always use browser-harness for any web interaction: automation, scraping, testing, or site/app work."
|
||||
description: "Use browser-harness only when explicitly requested for direct CDP browser work or capabilities outside fast-browser's supported scope. Use fast-browser by default for supported local browser tests and regression scenarios."
|
||||
---
|
||||
|
||||
# browser-harness
|
||||
|
||||
## Browser tool routing
|
||||
|
||||
- Use **fast-browser** by default for supported local browser tests and regression scenarios: isolated sessions, bounded click/fill, reads/assertions, navigation/tabs/popups, and private failure evidence. In Pi, use its `fast_browser_*` tools and skill; use its scenario CLI for repeatable tests.
|
||||
- Use **browser-harness only by explicit user opt-in** for direct CDP work or capabilities outside fast-browser's scope. If fast-browser is unavailable or lacks a required capability, explain the limitation and ask before switching.
|
||||
- A failed, timed-out, or unverifiable fast-browser action is **not** permission to retry through browser-harness. Inspect state and resolve the outcome first; do not bypass origin restrictions, profile locks, or other safeguards.
|
||||
- Never let both tools control the same browser session, endpoint, or profile. Close the owned session before changing tools and use a separately owned browser. Do not attach this CLI to a fast-browser-managed session.
|
||||
- Other task-specific routing (such as Firecrawl for web research or Orca for its embedded browser) remains in effect. This skill does not claim all web work.
|
||||
|
||||
The instructions below apply only after browser-harness has been explicitly selected.
|
||||
|
||||
Direct browser control via CDP. For task-specific edits, use `agent-workspace/agent_helpers.py`. For setup, install, or connection problems, read https://github.com/browser-use/browser-harness/blob/main/install.md.
|
||||
|
||||
## When Not to Use
|
||||
|
||||
@@ -19,7 +19,7 @@ Treat [`config/pi-links.json`](../../../config/pi-links.json) as the source of t
|
||||
1. Run commands from the repository root.
|
||||
2. Read the manifest entry and the resource's `SKILL.md` or `index.ts` when diagnosing a mismatch.
|
||||
3. Resolve manifest validation errors before applying changes. Require unique skill names and existing entry points.
|
||||
4. Preserve collisions. The manager refuses regular files, directories, and links whose targets are outside this repository; report these for a human decision instead of replacing them manually.
|
||||
4. Preserve collisions. The manager refuses regular files, directories, and links whose targets are outside this repository; report these for a human decision instead of replacing them manually. For an explicitly approved repository relocation, use `sync --dry-run --migrate-from /absolute/old/root`, inspect the repairs, then `sync --migrate-from /absolute/old/root`. This only repairs manifest-declared destinations whose old target exactly matches the same repository-relative source under that old root. It does not migrate stale/unlisted links or local packages, and cannot be used with `prune`. Finish with a normal `check` without the migration option.
|
||||
5. Apply the authorized operation through `scripts/pi-links.mjs`.
|
||||
6. Run `./scripts/pi-links.mjs check` again. Completion requires exit code zero.
|
||||
7. Tell the user to run `/reload` when skill or extension exposure changed in the active Pi session.
|
||||
|
||||
Reference in New Issue
Block a user