feat(browser): inspect, annotate, and grab page elements for the agent (#1708)

* feat(browser): inspect, annotate, and grab page elements for the agent

Build a design-review flow on top of the in-app Electron browser so users
can send page elements to their coding agent with context.

- Annotate: pick an element, write a comment, choose an intent
  (fix/change/question/approve); the element context + intent + comment go
  to the agent as text, and a cropped screenshot rides along as an image.
- Grab: pick an element to copy its info + screenshot straight to the
  system clipboard (no comment), with a toast on success/failure.
- Hover inspector: in select mode each element shows a floating label with
  tag, id/class, React component name, and pixel size.
- Page markers: annotated elements on the current page get numbered badges
  that track scroll/resize.
- Device sizes: a viewport-size menu (responsive + 13 common device presets)
  renders the page centered in a fixed-size frame.
- Toolbar buttons now expose hover tooltips and no longer get squeezed out
  on narrow panes; the element selector is available to all desktop users
  (previously dev-only).

Screenshots and clipboard writes go through new Electron main-process IPCs
(capturePage + clipboard) so they work regardless of webview focus. Element
screenshots are referenced by id in the workspace attachment store and are
protected from the draft-store attachment GC. All new strings are
translated across the six supported locales.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(browser): write element clipboard exactly once

Resolve the image before writing so a combined text+image grab no longer
does a redundant text-only writeText() first (which flashed an intermediate
clipboard state). Addresses greptile review feedback.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* i18n: add browser annotate/grab/devices strings for ja and pt-BR

main added Japanese and Brazilian Portuguese locales after this branch was
created; add the browser annotation, grab, and device-size keys to keep all
locales in sync with en. resources.test.ts parity passes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
huiliaoning
2026-06-30 22:58:16 +02:00
committed by GitHub
co-authored by Claude Opus 4.8
parent 2b46caa20e
commit b8b66816ca
17 changed files with 1313 additions and 94 deletions
@@ -9,6 +9,7 @@ import {
} from "@/attachments/service";
import { useCreateFlowStore } from "@/stores/create-flow-store";
import { useSessionStore, type SessionState } from "@/stores/session-store";
import { useWorkspaceAttachmentsStore } from "@/attachments/workspace-attachments-store";
import {
applyClearDraftRecord,
collectReferencedAttachmentIdsFromState,
@@ -141,6 +142,18 @@ async function runAttachmentGc(): Promise<void> {
collectStreamUserImageIds(session.agentStreamHead, referencedIds);
}
// Browser-element screenshots live in the workspace attachment store, not in
// drafts, so collect their ids here to keep them from being garbage collected
// before the user sends the message.
const attachmentsByScope = useWorkspaceAttachmentsStore.getState().attachmentsByScope;
for (const attachments of Object.values(attachmentsByScope)) {
for (const attachment of attachments) {
if (attachment.kind === "browser_element" && attachment.attachment.screenshot) {
referencedIds.add(attachment.attachment.screenshot.id);
}
}
}
try {
await garbageCollectAttachments({ referencedIds });
} catch (error) {