jb/harness-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fast-browser-with-jev
fast-browser scenarios / Offline scenario contract tests (push) Failing after 44s
Details
fast-browser scenarios / Local Chromium headless scenarios (push) Failing after 23s
Details

Browse Source
This commit is contained in:
jb
2026-09-20 12:24:08 -04:00
parent fd0edb1511
commit fd63b0ccec
73 changed files with 21212 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/browser-use-jev-feasibility.md
+206
View File
@@ -0,0 +1,206 @@
# Local browser automation with Pi and Typesafe JEV
Status: feasibility review; proposed architecture, not an implementation decision.
Reviewed: 2026-09-18. No browser tasks or inference benchmarks were run. No API keys were used or changed.
## Verdict
**Worth a bounded prototype for DOM-accessible workflows; not a demonstrated replacement for a general browser agent.** Browser-harness already supports local Chrome control without Browser Use Cloud. JEV can plausibly reduce the cost and latency of semantic choices between known actions. Its usefulness depends on how reliably code can expose the correct action and verify its outcome.
The most important correction: **JEV is text-only, not a screenshot-to-action model.** Use accessibility/DOM observations, not screenshots, on the JEV fast path. A vision-capable Pi model would be a separate fallback for visually grounded tasks.
A skill alone can teach Pi how to use the browser but does not remove main-model turns. The candidate design is a **Pi extension containing a bounded decision loop, with browser-harness as its local execution backend and a companion skill for workflow guidance**. A Chrome extension is not necessary for the initial experiment.
## What the sources establish
### Browser-harness
Inspected upstream commit `afbcc381b963040c19627d788e40c7e7663171ee`, package version `0.1.13`, MIT license.
- Local Chrome/Chromium is supported; cloud browsers are optional. The package requires Python >=3.11; its setup prompt recommends Python 3.12.
- A persistent local daemon owns the CDP connection. Python helpers provide browser operations; the CLI executes Python, and an optional stdio MCP server exposes helpers as tools.
- Upstream recommends accessibility-tree grounding using `Accessibility.getFullAXTree`, `backendDOMNodeId`, box-model coordinates, and outcome verification. Screenshots are a fallback when layout or imagery matters.
- The default local daemon has one mutable current tab. Concurrent tab switching and actions can race. A Pi adapter needs serialized browser operations, not just per-request locking.
- `fill_input()` emits key events per character; `type_text()` uses `Input.insertText`. Long-text entry can dominate latency even with a fast text model. Framework-managed inputs require event handling and exact-value verification; bulk insertion is not a universal replacement.
- Arbitrary Python/JS and self-written helpers are powerful but inappropriate as the automatic JEV action interface. Expose vetted typed operations instead; keep any helper authoring in the supervised slow path.
Local-only configuration needs care:
- Set `BH_TELEMETRY=0`: telemetry is otherwise enabled by default.
- Set `BH_UPDATE_CHECK=0`: the CLI otherwise checks PyPI for updates.
- **Unset `BU_AUTOSPAWN`**, rather than setting it to `0`: source checks whether the environment value is nonempty.
- Explicitly bind to the approved local browser. Do not expose remote-browser start, profile sync, cloud authentication, or proxy-fetch helpers in the fast adapter.
- Do not supply a Browser Use key to this worker. `http_get()` selects a hosted fetch proxy when `BROWSER_USE_API_KEY` is present. Treat environment and workspace `.env` inheritance as part of the audit.
- Keep recordings off unless requested (`BH_RECORD=0` is an explicit override). Treat saved DOM, screenshots, and logs as sensitive.
This establishes that hosted browser infrastructure is unnecessary, not that all dependencies have been exhaustively audited for outbound traffic. An implementation should verify network egress explicitly.
### Typesafe JEV
The documented API is `POST https://api.typesafe.ai/v1/systemone`, authenticated with a bearer key. Requests contain `state`, `model`, and named `questions`.
- Current documented version: `jev-1.13.0`. Pin a version when calibrating thresholds; `jev-latest` can move.
- Inputs: text or structured JSON; no images, audio, or video.
- Outputs: `Choice` among supplied options, `Noul` yes-probability, or `Score` against supplied criteria. This is not arbitrary prose, code, or a general agent action stream.
- Multiple questions about the same state are evaluated independently in parallel. A question cannot rely on another question's answer in the same request.
- Published limits: 64k tokens for the request and 32k for state plus the longest question. These are ceilings, not recommended browser-state sizes.
- Published price: $0.042 per million input tokens, with output tokens free. At 2,000 input tokens per request, 1,000 decisions would cost approximately $0.084 for JEV alone, assuming that token count matches billing. Planner/text calls and retries are additional.
- Typesafe says most queries complete in about 100 ms. **This is a provider claim, not measured local end-to-end latency or a browser-task guarantee.**
- The docs say customer requests/responses are not used for training. They mention enterprise ZDR separately; do not assume a normal API account has zero retention.
- The JavaScript SDK is `@typesafe-ai/sdk` (Node.js >=20), using `TypeSafeClient.systemOne()` and `TYPESAFE_API_KEY`. Keep this as an independent API adapter, not a fake chat model.
- Typesafe explicitly documents weakness with numeric precision, counting, date comparison, multi-hop indirection, irrelevant context, and adversarial input. Parse/count/compare in code; give JEV semantic labels and short, direct questions.
- The documented adversarial-content limitation is especially relevant to websites: page content can steer classifications. Instructions and confidence thresholds alone are not adequate protection.
- Choice confidence and Noul probabilities are not interchangeable. Calibrate each gate separately; a relative Choice can pick a winner even when every option is unsuitable.
Local browser execution still sends selected page-derived text to Typesafe and generated-text context to the configured Pi model provider. Redact passwords, tokens, cookies, and unrelated page content before those calls.
## Proposed execution split
```text
User goal -> Pi main model: bounded subgoal and success criteria
|
Pi extension
|
observe AX/DOM, filter candidates
|
exact deterministic match available?
yes: use code no: JEV Choice
|
policy + freshness + confidence checks
|
typed local browser-harness bridge
|
execute -> verify -> repeat
|
finished / uncertain / budget exhausted
|
return to Pi
Text needed: supplied literal -> insert directly
new prose -> configured small Pi model -> validate -> insert
Visual or novel reasoning needed -> return to Pi, optionally with screenshot
```
### Responsibilities
| Component | Responsibility |
| --- | --- |
| Pi main model | Interpret novel goals, supply bounded subgoals, handle recovery requiring reasoning |
| Extension controller | Budgets, state machine, permissions, cancellation, model routing, traces |
| DOM/AX adapter | Extract relevant controls and labels; generate candidate IDs tied to tab/frame/document |
| JEV | Rank/select candidates, classify known states, flag uncertainty, assess narrow semantic predicates |
| Small Pi model | Generate genuinely new text within supplied facts and constraints |
| Browser bridge | Execute allowlisted operations; verify target identity and resulting browser state |
| Companion skill | Explain usage, escalation rules, and reusable site-specific workflows |
JEV should **replace a main-model microdecision**, not be added before an unchanged main-model turn. The deterministic path should bypass JEV entirely when no semantic judgment is needed.
Candidate sets must include `none_of_the_above`/`needs_more_context`. Selecting the best of an incomplete set can still be wrong. Keep candidate descriptions grounded in visible labels, roles, nearby headings and form structure; filter hidden/disabled elements and preserve frame identity. Treat the correct-target inclusion rate as a first-class metric.
Run independent page-state questions together, but do not confuse speculative decisions with permission to execute alternative actions in parallel. Reobserve after navigation or meaningful DOM changes. Invalidate stale candidates before any click; do not cache coordinates across page transitions.
For literals such as a supplied email address, search phrase, or exact message, **no text-model call is necessary**. A generation model is only justified for composition or rewriting. Validate length, exact required facts, and field-specific constraints. Do not regenerate text when retrying insertion; verify the actual value and framework state.
### Scope and limitations
Good first targets: locating a labeled control, navigating a known menu, choosing a relevant result from a shortlist, filling conventional forms, recognizing a success/error message, and repeating established workflows.
Weak targets: canvas-only applications, image-only controls, novel workflows that require broad planning, CAPTCHAs, ambiguous authorization, and long tasks where the correct action is not represented in the candidate set. Shadow DOM, cross-origin frames, rich-text editors, and changing layouts need explicit coverage rather than assumptions.
## Pi integration findings
The installed Pi API supports this split without changing the main conversation model:
- Register typed tools with `pi.registerTool`; tool execution receives an abort signal and extension context.
- Resolve the auxiliary text model with `ctx.modelRegistry.find(provider, id)` and check `getAvailable()` / `hasConfiguredAuth(model)` at runtime.
- Call `ctx.modelRegistry.complete(model, context, { signal })` for independent text generation. This uses Pi's model registry rather than duplicating provider credential handling.
- Keep Typesafe behind its own SDK/HTTP adapter; its constrained decision API does not need a Pi chat-provider registration.
- Cancellation must propagate to model calls and the local worker. Stopping a tool does not undo an already executed browser action.
Sanitized cached model metadata includes these **candidates, not verified live availability or measured speed rankings**:
| Candidate | Potential role |
| --- | --- |
| `anthropic/claude-haiku-4-5` | Small text-generation baseline; metadata also advertises image input |
| `anthropic/claude-haiku-4-5-20251001` | Pinned Haiku alternative for reproducibility |
| `openai-codex/gpt-5.3-codex-spark` | Text-only alternative to benchmark if runtime authorization permits |
Start by comparing Haiku and Spark on short, constrained field-generation tasks; do not select a winner from model names alone. A local LM Studio provider is another supported integration route, but no particular loaded local model was verified.
Relevant precedents:
- Installed Pi `examples/extensions/qna.ts` and `summarize.ts`: independent registry completions and cancellation.
- `extensions/lm-studio/index.ts`: dynamic local OpenAI-compatible model discovery.
- `extensions/subagents/src/backends/pi.ts`: registry reuse and abort lifecycle handling.
- Adjacent project `/srv/work/personal/code/eden/src/lib/server/assistant/intent.ts` and `tests/intent.test.ts`: JEV Choice routing with pinned version, response validation, confidence gating and failure-closed tests. This is a reference, not a proposed runtime dependency.
Browser-harness exposes Python/CLI/MCP interfaces; no existing TypeScript-native bridge was identified in this investigation. A restricted local Python bridge is the proposed adapter, with the existing MCP server as an alternative transport to evaluate. Keep the controller independent of transport so this choice can change without rewriting decision logic. Measure worker startup/IPC before committing to an additional always-running service.
## Safety and reliability requirements
- Code, not model confidence, enforces allowed domains and actions. Treat all page text as untrusted data, never instructions for the controller.
- Stop for approval before purchases, sending messages, publishing, destructive changes, granting access, or other consequential submissions unless the exact action is already authorized.
- Keep credentials out of model inputs. Stop for passwords, MFA, and ambiguous account selection; prefer an explicitly approved logged-in test profile.
- Bind candidates and approval to the precise browser/tab/frame, target identity, intended action and payload. Revalidate immediately before execution.
- Do not blindly retry non-idempotent actions after timeouts. Inspect the resulting state first: a cancelled request may already have clicked or submitted.
- Bound steps, elapsed time, model requests, and recovery attempts. Detect repeated states/no progress, release the browser lane on stop, and surface a useful partial result.
- Fail closed on malformed responses, missing targets, expired observations, and permission uncertainty. Semantic confidence is not a security boundary.
- Keep the browser debug endpoint local/private. Starting a browser on a remote Pi host does not automatically attach to a desktop browser elsewhere; deployment location needs confirmation before a prototype.
## Benchmark before committing to the architecture
Use a shared local browser backend and comparable observations, test fixtures, warm/cold conditions, and safety policy. Pin model versions and prompts. No live purchases, messages, or destructive tasks in the initial suite.
Compare four paths:
1. Main Pi model chooses each browser action: conventional baseline.
2. Small Pi model chooses from the same bounded candidate list.
3. JEV chooses from that list; small model only generates text when needed.
4. Deterministic/site-specific workflow where applicable: establishes whether AI adds value at all.
Use 30–50 task instances across navigation, result selection, conventional forms, generated text, and recovery. Repeat each several times, randomize run order, reset initial state, and report uncertainty. This is a screening study, not proof of general-web reliability. Use held-out pages/tasks for confirmation.
Measure:
- Verified task success, critical misactions, and human interventions.
- End-to-end p50/p95 latency and time-to-verified-success, including failed runs/timeouts rather than reporting only successful fast runs.
- Observe/filter time, payload size, API RTT, action time, DOM settling, verification, text insertion, and escalation latency separately.
- Candidate recall, selected-action accuracy, no-progress loops, stale-target failures, and fallback frequency.
- Main-model calls eliminated, token usage and total cost per verified successful task.
- Confidence versus empirical action accuracy, by task type. Tune thresholds on calibration data and evaluate on held-out data; do not assume a probability cutoff guarantees browser reliability.
Illustrative latency budget, **not a measurement**:
`T_task = planning + Σ(observe + decide + act + settle + verify) + text generation + recovery`
Replacing a 1,500 ms decision with 200 ms when all other step work takes 500 ms changes a step from 2,000 ms to 700 ms (~2.9× faster). If the site instead spends 3,000 ms loading, the same replacement gives only ~1.4×. Retry and escalation costs can erase either gain.
Suggested promotion gate for discussion: roughly 2× median improvement over the main-model baseline on the target workflows, no observed critical misactions, task success comparable to baseline, and a material advantage over the small-model-only alternative. Tail latency and fallback cost must remain acceptable. These are proposed targets, not established outcomes or statistical guarantees.
## Suggested next experiment
Build only after agreeing on the architecture and deployment target:
1. An offline AX/DOM fixture adapter and JEV-vs-small-model action-selection benchmark. First establish candidate recall and decision quality without any side effects.
2. A bounded local execution prototype using the same candidate schema, with literal form filling and deterministic postconditions.
3. Add generated-text calls and supervised escalation only if the first two stages show a useful advantage.
Do not begin with a general autonomous browser agent, self-modifying helpers, Chrome extension, hosted browser, or custom JEV chat-provider emulation.
## Sources
- [Browser-harness README](https://github.com/browser-use/browser-harness/blob/afbcc381b963040c19627d788e40c7e7663171ee/README.md)
- [Browser workflow and local concurrency](https://github.com/browser-use/browser-harness/blob/afbcc381b963040c19627d788e40c7e7663171ee/src/browser_harness/SKILL.md)
- [CDP and input helpers](https://github.com/browser-use/browser-harness/blob/afbcc381b963040c19627d788e40c7e7663171ee/src/browser_harness/helpers.py)
- [Telemetry controls](https://github.com/browser-use/browser-harness/blob/afbcc381b963040c19627d788e40c7e7663171ee/src/browser_harness/telemetry.py), [cloud autospawn](https://github.com/browser-use/browser-harness/blob/afbcc381b963040c19627d788e40c7e7663171ee/src/browser_harness/run.py), [update checks](https://github.com/browser-use/browser-harness/blob/afbcc381b963040c19627d788e40c7e7663171ee/src/browser_harness/admin.py)
- [Typesafe workflow architecture](https://docs.typesafe.ai/concepts/how-to-build-with-system-one)
- [JEV models, pricing, limits and data handling](https://docs.typesafe.ai/models)
- [Text-only state contract](https://docs.typesafe.ai/concepts/state)
- [Typesafe HTTP API](https://docs.typesafe.ai/api)
- [JEV 1.13 documented limitations](https://docs.typesafe.ai/model-jaggedness/jev-1.13)
- [Typesafe JavaScript SDK](https://docs.typesafe.ai/sdk/javascript)
- Installed Pi API reference: `/home/jb/.bun/install/global/node_modules/@earendil-works/pi-coding-agent/dist/core/model-registry.d.ts`
- Installed Pi extension documentation: `/home/jb/.bun/install/global/node_modules/@earendil-works/pi-coding-agent/docs/extensions.md`
- Installed Pi auxiliary-model examples: `/home/jb/.bun/install/global/node_modules/@earendil-works/pi-coding-agent/examples/extensions/{qna,summarize}.ts`
Research inputs were cached under `/tmp/browser-efficacy/.firecrawl/`; the inspected upstream checkout is `/tmp/browser-efficacy/browser-harness/`. These temporary caches are not deliverables or durable repository dependencies.