paseo/REPORT-claude-streaming-regression.md

Claude Streaming Regression Investigation

Status: ROOT CAUSE IDENTIFIED

Problem

Claude agent text is garbled/corrupted during streaming, with words cut off and merged incorrectly (e.g., "a" + "check error" instead of "a" + "type" + "check error"). This bug ONLY appears in long-running agents during the 2nd+ conversation turn, not during initial agent creation.

Investigation Timeline

Phase 1: App-Side Analysis (Passed)

• Added debug logging to React Native state management
• Verified Zustand state updates are correct
• Confirmed FlatList rendering is correct
• Conclusion: Bug is NOT in the app layer

Phase 2: Server-Side Streaming Analysis (Passed)

• Added E2E test to verify server sends clean chunks during new agent creation
• Test passes - server correctly forwards all text_delta chunks from Claude SDK
• Conclusion: Bug is NOT in baseline streaming code

Phase 3: Long-Running Agent Analysis (IDENTIFIED ROOT CAUSE)

• Realized the E2E test only tested NEW agents, not long-running scenarios with multiple turns
• Examined claude-agent.ts state management across multiple stream() invocations
• ROOT CAUSE FOUND in claude-agent.ts

Root Cause

Location

packages/server/src/server/agent/providers/claude-agent.ts

The Bug: Race Condition with Shared State Flags

File: packages/server/src/server/agent/providers/claude-agent.ts

Critical Lines: 480-482 (fire-and-forget call)

// Line 452-503: stream() method
async *stream(
  prompt: AgentPromptInput,
  _options?: AgentRunOptions
): AsyncGenerator<AgentStreamEvent> {
  const sdkMessage = this.toSdkUserMessage(prompt);
  const queue = new Pushable<AgentStreamEvent>();
  this.eventQueue = queue;
  // ...
  
  // BUG: forwardPromptEvents is not awaited!
  // This is a fire-and-forget call that allows race conditions
  this.forwardPromptEvents(sdkMessage, queue).catch((error) => {
    console.error("[ClaudeAgentSession] Unexpected error in forwardPromptEvents:", error);
  });

  // Immediately starts consuming from queue while forwardPromptEvents
  // is still running asynchronously
  try {
    for await (const event of queue) {
      yield event;
      if (event.type === "turn_completed" || event.type === "turn_failed") {
        finishedNaturally = true;
        break;
      }
    }
  } finally {
    // ...
  }
}

The Mechanism: Shared State Across Turns

In a long-running agent with multiple turns:

1. Turn 1: User sends message

   • stream() called, creates generator instance 1
   • forwardPromptEvents() async method starts running (not awaited)
   • Flags streamedAssistantTextThisTurn and streamedReasoningThisTurn reset to false (line 769-770)
   • Stream events arrive, set flags to true
   • Final "assistant" message arrives, uses flag to suppress duplicate (line 824)

2. Turn 2 (while Turn 1 is still processing): User sends another message

   • stream() called AGAIN, creates generator instance 2
   • Both generator instances share the SAME ClaudeAgentSession object with SAME instance variables
   • forwardPromptEvents() for Turn 2 resets the flags (line 769-770)
   • BUT if Turn 1's forwardPromptEvents() is still running, its stream_events are STILL coming in
   • The flag changes are now SHARED between both stream operations

The Garbled Text Result

When flags get mixed up between concurrent forwardPromptEvents() calls:

1. Turn 2's forwardPromptEvents() resets streamedAssistantTextThisTurn = false (line 769)
2. Turn 1 is still streaming text_deltas and setting streamedAssistantTextThisTurn = true (line 1138)
3. When Turn 1's final "assistant" message arrives, it checks the flag (line 824)
4. The flag value is UNCERTAIN - it depends on which operation modified it last
5. If suppression logic fails, text from different chunks could be:
   • Partially suppressed (causing words to be cut off)
   • Duplicated (causing merging)
   • Out of order

Affected Code Sections

1. The Fire-and-Forget Call (Line 480-482)

this.forwardPromptEvents(sdkMessage, queue).catch((error) => {
  console.error("[ClaudeAgentSession] Unexpected error in forwardPromptEvents:", error);
});

Problem: No await. The async function runs in the background while the queue iteration starts immediately.

2. Shared Flags (Lines 368-369)

private streamedAssistantTextThisTurn = false;
private streamedReasoningThisTurn = false;

Problem: These are instance variables shared across ALL calls to stream() on the same session object.

3. Flag Usage in Suppression Logic (Lines 823-826)

case "assistant": {
  const timelineItems = this.mapBlocksToTimeline(message.message.content, {
    suppressAssistantText: this.streamedAssistantTextThisTurn,  // Uses shared flag!
    suppressReasoning: this.streamedReasoningThisTurn,           // Uses shared flag!
  });

Problem: Reads the shared flag value without synchronization.

4. Flag Mutation During Streaming (Lines 1138, 1149)

if (context === "live") {
  this.streamedAssistantTextThisTurn = true;  // Mutates shared flag
}
// ...
if (context === "live") {
  this.streamedReasoningThisTurn = true;  // Mutates shared flag
}

Problem: Mutates shared state without coordination between concurrent operations.

Why This Wasn't Caught by Tests

The E2E test in commit e587547 creates a NEW Claude agent and sends messages sequentially to the SAME agent. While messages are sent back-to-back, the test waits for each response to complete (the runPrompt() calls don't overlap).

In the actual app during stress testing or rapid input, multiple stream() calls could overlap, causing the race condition.

Proposed Fix

Move the streaming flags from instance variables to local variables scoped to each stream() invocation.

Replace instance variables:

// REMOVE THESE
private streamedAssistantTextThisTurn = false;
private streamedReasoningThisTurn = false;

Add as parameters to forwardPromptEvents() and pass them through the call chain:

async *stream(...) {
  // Local variables scoped to THIS invocation
  let streamedAssistantTextThisTurn = false;
  let streamedReasoningThisTurn = false;
  
  // Pass to forwardPromptEvents
  this.forwardPromptEvents(sdkMessage, queue, {
    streamedAssistantTextThisTurn,
    streamedReasoningThisTurn,
  }).catch(...);
}

private async forwardPromptEvents(
  message: SDKUserMessage, 
  queue: Pushable<AgentStreamEvent>,
  flags: { streamedAssistantTextThisTurn: boolean; streamedReasoningThisTurn: boolean }
) {
  flags.streamedAssistantTextThisTurn = false;  // Reset only this operation's flags
  flags.streamedReasoningThisTurn = false;
  // ... rest of implementation
}

This ensures each stream() invocation has its own isolated flags that can't interfere with concurrent operations.

Verification Needed

Before this fix is applied:

1. Write a test that reproduces the race condition (overlapping stream() calls)
2. Verify the fix prevents the race condition
3. Run existing E2E tests to ensure no regressions

Summary

The bug is a classic race condition caused by:

1. Non-awaited async function call (fire-and-forget)
2. Shared mutable state (instance variables)
3. Multiple concurrent consumers of that state (overlapping stream() calls)

This is a threading/concurrency issue, not a text encoding or protocol problem. The fix is to isolate state per operation rather than sharing it at the instance level.