paseo/REPORT-image-attachment-investigation.md

Image Attachment Flow Investigation Report

Date: 2025-12-29 Task: Test image attachment flow via Playwright MCP

Summary

Image attachments are NOT WORKING for any platform (web, iOS, Android) due to multiple issues in the implementation pipeline.

Issues Found

Issue 1: Agent Creation Does Not Support Images (By Design)

Location: packages/app/src/app/agent/new.tsx:576-580

// TODO: Images in initial agent creation are not yet supported by the server API.
// For now we log a warning. Images can be sent after agent creation via sendAgentMessage.
if (images && images.length > 0) {
  console.warn("[DraftAgentScreen] Image attachments on agent creation not yet supported");
}

Impact: Images selected on the new agent screen are silently dropped when creating an agent.

Evidence: Console warning logged during Playwright test when attaching an image and creating an agent.

---

Issue 2: Web Platform Cannot Base64 Encode Images

Location: packages/app/src/contexts/session-context.tsx:1307-1331

const data = await FileSystem.readAsStringAsync(uri, {
  encoding: "base64",
});

Error:

[ERROR] [Session] Failed to convert image: Error: Method readAsStringAsync imported from "expo-file-system"...

Impact: On web, expo-file-system.readAsStringAsync is not available/deprecated, causing image base64 encoding to fail silently. The message is sent without the image data.

Evidence: Console error logged when trying to send an image attachment to an existing agent on web.

---

Issue 3: Server Does Not Pass Image Bytes to LLM

Location: packages/server/src/server/session.ts:316-334

private buildAgentPrompt(
  text: string,
  images?: Array<{ data: string; mimeType: string }>
): AgentPromptInput {
  const normalized = text?.trim() ?? "";
  if (!images || images.length === 0) {
    return normalized;
  }

  const attachmentSummary = images
    .map((image, index) => {
      const sizeKb = Math.round((image.data.length * 0.75) / 1024);
      return `Attachment ${index + 1}: ${image.mimeType}, ~${sizeKb}KB base64`;
    })
    .join("\n");

  const base = normalized.length > 0 ? normalized : "User shared image attachment(s).";
  return `${base}\n\n[Image attachments]\n${attachmentSummary}\n(Actual image bytes omitted; request a screenshot or file if needed.)`;
}

Impact: Even if images were successfully uploaded to the server, they are NOT sent to the LLM. Instead, a text summary is created that says "(Actual image bytes omitted)".

Evidence: Agent responses consistently state "I don't see any image attached to your message" even when images are selected in the UI.

---

Test Results

Test 1: Claude Agent - Image on Creation

• Action: Attached red.png, asked "what color is this image?"
• Result: FAILED - Image silently dropped, agent asked for image path
• Console: [WARNING] [DraftAgentScreen] Image attachments on agent creation not yet supported

Test 2: Claude Agent - Image on Existing Agent (Web)

• Action: Attached red.png to existing agent, asked "what color is the attached image?"
• Result: FAILED - Base64 encoding failed on web platform
• Console: [ERROR] [Session] Failed to convert image: Error: Method readAsStringAsync imported from "expo-file-system"...

Test 3: Codex Agent

• Not Tested: Would fail for same reasons as Claude

---

Image Flow Architecture

UI (file picker)
    ↓
Image selected → shown as thumbnail ✅
    ↓
[Agent Creation Path]
    → Warning logged, image dropped ❌

[Existing Agent Path]
    → FileSystem.readAsStringAsync(uri, {encoding: 'base64'})
        → FAILS on web ❌
        → May work on native (not tested)
    ↓
WebSocket → send_agent_message with images array
    ↓
Server → buildAgentPrompt()
    → Creates TEXT SUMMARY only ❌
    → Actual bytes NOT passed to LLM
    ↓
LLM receives: "what color is this image?\n\n[Image attachments]\nAttachment 1: image/png, ~1KB base64\n(Actual image bytes omitted)"
    → Agent cannot see image ❌

---

Required Fixes

Fix 1: Implement Image Support for Agent Creation

• Extend create_agent_request server API to accept images
• Pass images from UI through to the initial prompt

Fix 2: Fix Web Platform Base64 Encoding

• Replace expo-file-system.readAsStringAsync with web-compatible alternative
• Use FileReader.readAsDataURL() or similar for web platform
• Or use a cross-platform library that works on all Expo platforms

Fix 3: Pass Actual Image Bytes to LLM

• Modify buildAgentPrompt to return image content blocks, not just text
• Update agent providers (Claude, Codex) to handle multimodal prompts
• For Claude: Use content blocks with type: "image" and base64 data
• For Codex: Check if/how OpenAI API handles image inputs

---

Recommended Task Additions

- [ ] **Fix 1**: Add image support to agent creation API
    - Extend `create_agent_request` to accept images array
    - Pass images to `handleSendAgentMessage` after agent creation
    - Remove warning in `new.tsx` once implemented

- [ ] **Fix 2**: Fix web platform image base64 encoding
    - Replace `FileSystem.readAsStringAsync` with cross-platform solution
    - Use blob URL + fetch or FileReader API for web
    - Test on web, iOS, and Android

- [ ] **Fix 3**: Implement multimodal prompt building on server
    - Modify `buildAgentPrompt` to return structured content with images
    - Update `AgentPromptInput` type to support content blocks
    - Implement Claude-specific image content blocks
    - Implement Codex/OpenAI image handling