jb/paseo
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

Investigation: Image attachments are broken

Browse Source 
Tested image attachment flow via Playwright MCP. Found three issues:

1. Agent creation silently drops images (by design - TODO in code)
2. Web platform fails base64 encoding (expo-file-system not available)
3. Server buildAgentPrompt only creates text summary, never sends bytes

Added fix tasks to plan.md and detailed report in
REPORT-image-attachment-investigation.md.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Mohamed Boudra
2025-12-29 09:03:58 +07:00
co-authored by Claude Opus 4.5
parent 62181e1f4c
commit 5d24e0eee8
2 changed files with 195 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plan.md
+31
View File
@@ -118,3 +118,34 @@ Improvements to the new agent screen in the app.
- E2E when possible but no browser tests.
- Add any missing test tasks to the plan.
- **Done (2025-02-10 08:56)**: WHAT: `REPORT-daemon-test-coverage.md:1` documents daemon test coverage review for Iterations 3-9. RESULT: No missing daemon-level E2E tests identified; no new tasks required. EVIDENCE: Review documented in report.
- [x] agent=claude **Investigation**: Test image attachment flow via Playwright MCP.
- Use Playwright MCP to test image uploads at `http://localhost:8081`.
- Create a solid color image using ImageMagick (e.g., `convert -size 100x100 xc:red /tmp/red.png`).
- Upload the image as an attachment to an agent.
- Test with Claude agent: ask "what color is this image?" - verify it responds correctly.
- Test with Codex agent: same test - verify it responds correctly.
- Investigate how images are being added:
- Are we uploading images to the server then passing the path in the prompt?
- Or are we base64 encoding and sending inline?
- Trace the flow from UI upload to agent prompt.
- Document findings in the plan.
- If broken, add fix tasks with debugging details.
- **Done (2025-12-29 09:13)**: WHAT: Tested image attachments via Playwright MCP; traced code in `packages/app/src/app/agent/new.tsx:576-580`, `packages/app/src/contexts/session-context.tsx:1307-1331`, `packages/server/src/server/session.ts:316-334`. RESULT: Image attachments are BROKEN - three issues found: (1) Agent creation silently drops images with console warning, (2) Web platform fails to base64 encode images due to `expo-file-system.readAsStringAsync` not working, (3) Server `buildAgentPrompt` only creates text summary, never passes actual image bytes to LLM. EVIDENCE: Console logs `[WARNING] [DraftAgentScreen] Image attachments on agent creation not yet supported` and `[ERROR] [Session] Failed to convert image: Error: Method readAsStringAsync imported from "expo-file-system"...`; agent responses consistently state "I don't see any image attached". Full report in `REPORT-image-attachment-investigation.md`.
- [ ] **Fix**: Add image support to agent creation API
- Extend `create_agent_request` to accept images array
- Pass images to `handleSendAgentMessage` after agent creation
- Remove warning in `new.tsx:576-580` once implemented
- [ ] **Fix**: Fix web platform image base64 encoding
- Replace `FileSystem.readAsStringAsync` in `session-context.tsx:1312` with cross-platform solution
- Use blob URL + fetch or FileReader API for web
- Test on web, iOS, and Android
- [ ] **Fix**: Implement multimodal prompt building on server
- Modify `buildAgentPrompt` in `session.ts:316-334` to return structured content with images
- Update `AgentPromptInput` type to support content blocks
- Implement Claude-specific image content blocks
- Implement Codex/OpenAI image handling