diff --git a/goals/research/pi-active-turn-reattachment.md b/goals/research/pi-active-turn-reattachment.md new file mode 100644 index 000000000..472f54908 --- /dev/null +++ b/goals/research/pi-active-turn-reattachment.md @@ -0,0 +1,109 @@ +# Pi active-turn reattachment + +Research for **Prepare agent session sandboxing with gVisor**. + +## Question + +Can a Pi turn started through `pi --mode rpc` keep running while the Paseo daemon crashes, then reattach to the replacement daemon with no lost or duplicated stream events? + +This research uses Pi `@earendil-works/pi-coding-agent` **0.84.2**, matching the installed `pi` binary. The primary source was cached with OpenSrc at `/srv/data/opensrc/repos/github.com/earendil-works/pi/0.84.2/packages/coding-agent`. + +## Verdict + +**Not with direct Pi RPC or native session resume. Feasible with a separate supervisor-owned reconnectable bridge.** + +Pi RPC is one process with one stdin command stream and one stdout event stream. It has no attach endpoint, client identity, event acknowledgment, or event replay protocol. Current Paseo owns those pipes in the daemon process and closes Pi during normal daemon shutdown. After an abrupt daemon death, Pi also observes stdin EOF and exits; even if it survived temporarily, a replacement daemon has no handle with which to adopt the pipes. + +A supervisor that outlives the daemon can preserve the Pi process. It must continue owning and draining Pi's pipes, add its own reconnectable IPC protocol, sequence and retain every Pi frame, fence command writers, and replay from a daemon acknowledgment cursor. This is bridge reattachment, not a Pi feature. + +## Three different operations + +| Operation | Result | +| ------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Direct process attachment**: a second RPC client connects to the already-running Pi process | **Unsupported.** Pi exposes only the process's stdin/stdout. There is no socket, attach command, or transferable runtime identifier in the RPC protocol. | +| **Bridge reattachment**: a new daemon reconnects to a supervisor that still owns Pi and its pipes | **Feasible.** The bridge must supply discovery, writer fencing, buffering, replay, and lifecycle semantics that Pi does not supply. | +| **Native session resume after a failed turn**: start a new Pi process with `--session ` | **Conversation recovery only.** It reconstructs persisted messages but does not continue the old provider request, tool execution, or event stream. | + +## Pi RPC findings + +### One process owns one duplex JSONL stream + +Pi documents RPC mode as commands on stdin and responses/events on stdout. Its typed client implements that contract by spawning one child with `stdio: ["pipe", "pipe", "pipe"]`; it stores request promises and event listeners in the client process ([RPC documentation](https://github.com/earendil-works/pi/blob/v0.84.2/packages/coding-agent/docs/rpc.md#protocol-overview), [`rpc-client.ts:56-179`](https://github.com/earendil-works/pi/blob/v0.84.2/packages/coding-agent/src/modes/rpc/rpc-client.ts#L56-L179)). + +The RPC server reads only `process.stdin`, subscribes once to the active `AgentSession`, and writes each resulting frame immediately to `process.stdout`. It does not accept connections or model multiple clients ([`rpc-mode.ts:54-63`](https://github.com/earendil-works/pi/blob/v0.84.2/packages/coding-agent/src/modes/rpc/rpc-mode.ts#L54-L63), [`rpc-mode.ts:351-362`](https://github.com/earendil-works/pi/blob/v0.84.2/packages/coding-agent/src/modes/rpc/rpc-mode.ts#L351-L362), [`rpc-mode.ts:800-813`](https://github.com/earendil-works/pi/blob/v0.84.2/packages/coding-agent/src/modes/rpc/rpc-mode.ts#L800-L813)). The public command union has session switching and history reads, but no attach, detach, claim, or reconnect operation ([`rpc-types.ts:19-76`](https://github.com/earendil-works/pi/blob/v0.84.2/packages/coding-agent/src/modes/rpc/rpc-types.ts#L19-L76)). + +Launching a second `pi --mode rpc --session ` creates a second runtime. It does not attach to the first. Pi's session writer appends directly to the JSONL file and has no cross-process lock in `SessionManager`, so concurrent interactive use of those runtimes is not a reattachment mechanism and is unsafe ([`session-manager.ts:1012-1068`](https://github.com/earendil-works/pi/blob/v0.84.2/packages/coding-agent/src/core/session-manager.ts#L1012-L1068)). + +### Live RPC events are not replayed or acknowledged + +Most RPC events have no request ID. They are emitted as they happen; only direct RPC bash output carries the originating command ID. `message_update` is delta-based, and `message_end.message` is authoritative ([RPC event documentation](https://github.com/earendil-works/pi/blob/v0.84.2/packages/coding-agent/docs/rpc.md#events)). The server keeps no event sequence, subscriber cursor, acknowledgment table, or replay buffer; its output path serializes each subscribed event straight to stdout ([`rpc-mode.ts:60-62`](https://github.com/earendil-works/pi/blob/v0.84.2/packages/coding-agent/src/modes/rpc/rpc-mode.ts#L60-L62), [`rpc-mode.ts:351-362`](https://github.com/earendil-works/pi/blob/v0.84.2/packages/coding-agent/src/modes/rpc/rpc-mode.ts#L351-L362)). The bundled client similarly forwards non-response frames only to listeners currently in memory ([`rpc-client.ts:508-525`](https://github.com/earendil-works/pi/blob/v0.84.2/packages/coding-agent/src/modes/rpc/rpc-client.ts#L508-L525)). + +`get_entries({ since })` is useful catch-up for **persisted session entries**, not replay of the live RPC stream. It returns completed JSONL entries after a durable entry ID. It cannot reproduce `agent_start`, token deltas, partial tool output, queue changes, extension UI requests, or other transient frames ([RPC `get_entries` documentation](https://github.com/earendil-works/pi/blob/v0.84.2/packages/coding-agent/docs/rpc.md#get_entries), [`rpc-mode.ts:634-646`](https://github.com/earendil-works/pi/blob/v0.84.2/packages/coding-agent/src/modes/rpc/rpc-mode.ts#L634-L646)). + +### Stdin EOF and stdout loss terminate the runtime + +Pi treats stdin end as shutdown. Shutdown disposes the runtime, pauses stdin, flushes stdout when applicable, and exits the process ([`rpc-mode.ts:722-813`](https://github.com/earendil-works/pi/blob/v0.84.2/packages/coding-agent/src/modes/rpc/rpc-mode.ts#L722-L813)). A non-retryable stdout write failure also exits the process ([`output-guard.ts:21-31`](https://github.com/earendil-works/pi/blob/v0.84.2/packages/coding-agent/src/core/output-guard.ts#L21-L31), [`output-guard.ts:84-91`](https://github.com/earendil-works/pi/blob/v0.84.2/packages/coding-agent/src/core/output-guard.ts#L84-L91)). + +This makes stdin/stdout ownership part of Pi's lifetime. A replacement daemon cannot reopen anonymous pipes created by the old daemon. Keeping Pi alive requires another process to retain both pipe ends needed by the host, continuously drain stdout, and avoid closing stdin while the daemon is absent. + +### Native session resume does not resume an active turn + +Pi persists messages only when a `message_end` event arrives. A user prompt is emitted as `message_end` before the model call, while the assistant message is persisted only after the provider stream produces its final result ([`agent-loop.ts:102-116`](https://github.com/earendil-works/pi/blob/v0.84.2/packages/agent/src/agent-loop.ts#L102-L116), [`agent-loop.ts:281-371`](https://github.com/earendil-works/pi/blob/v0.84.2/packages/agent/src/agent-loop.ts#L281-L371), [`agent-session.ts:640-667`](https://github.com/earendil-works/pi/blob/v0.84.2/packages/coding-agent/src/core/agent-session.ts#L640-L667)). A hard process loss can therefore leave a durable user message with no completed assistant response; partial deltas are not session entries. + +Starting a new process with `--session` rebuilds completed session context. RPC exposes no command to continue the unfinished provider request or resume an interrupted tool execution; the available recovery action is a new prompt or an explicit retry instruction ([`rpc-types.ts:19-76`](https://github.com/earendil-works/pi/blob/v0.84.2/packages/coding-agent/src/modes/rpc/rpc-types.ts#L19-L76), [Pi sessions documentation](https://github.com/earendil-works/pi/blob/v0.84.2/packages/coding-agent/docs/sessions.md)). This is native conversation resume after a failed turn, not active-turn continuation. + +## Current Paseo behavior + +Paseo builds `pi --mode rpc` and adds `--session ` when resuming a persisted Pi session ([`runtime.ts:75-150`](../../packages/server/src/server/agent/providers/pi/runtime.ts), [`docs/providers.md`](../../docs/providers.md)). Each `PiCliRuntime.startSession()` creates one daemon-owned `JsonlRpcProcess`; that transport spawns Pi with dedicated pipes, owns request IDs and pending promises, parses stdout, and fans non-response frames to in-memory subscribers ([`cli-runtime.ts:49-109`](../../packages/server/src/server/agent/providers/pi/cli-runtime.ts), [`jsonl-rpc-process.ts:101-229`](../../packages/server/src/server/agent/providers/jsonl-rpc-process.ts)). There is no persisted transport endpoint, pipe handle, frame cursor, or replay buffer. + +Paseo converts a Pi process exit into `turn_failed` only while its in-memory `PiRpcAgentSession` still has an active turn ([`agent.ts:2013-2071`](../../packages/server/src/server/agent/providers/pi/agent.ts)). On graceful daemon shutdown, `bootstrap.stop()` closes every managed agent, and Pi session close ends stdin then terminates the child process tree ([`bootstrap.ts:1714-1778`](../../packages/server/src/server/bootstrap.ts), [`agent-manager.ts:1470-1520`](../../packages/server/src/server/agent/agent-manager.ts), [`agent.ts:1532-1544`](../../packages/server/src/server/agent/providers/pi/agent.ts), [`jsonl-rpc-process.ts:205-229`](../../packages/server/src/server/agent/providers/jsonl-rpc-process.ts)). + +Paseo's persisted agent record and Pi's JSONL session handle are enough to rebuild completed history after restart. They do not preserve the live Pi process, active RPC request table, unanswered extension UI requests, or events emitted while the daemon is absent. Paseo's client WebSocket reconnection and authoritative timeline catch-up operate above `AgentManager`; they cannot recover frames that never reached the manager ([`docs/architecture.md`](../../docs/architecture.md), [`docs/agent-lifecycle.md`](../../docs/agent-lifecycle.md)). + +## Bounded local experiments + +The experiments used Pi 0.84.2 and files under `/tmp`; they did not modify Paseo production files. + +1. **Two processes, one session file.** Two simultaneous `pi --mode rpc --session /tmp/pi-rpc-reattach-session.jsonl` processes both answered `get_state` with the same session ID but different PIDs and separate extension UI event streams. This confirms that `--session` opens independent runtimes; it does not attach the second client to the first process. +2. **Parent disappearance.** A Node parent spawned `pi --mode rpc --no-session`, started an RPC `bash` command containing `sleep 30`, then exited without killing the child. The Pi PID was alive immediately after the parent exited and gone within one second, long before the command could complete. This experiment used the direct RPC bash command rather than an LLM turn, so it establishes pipe/EOF lifetime behavior; the source paths above establish that the same RPC shutdown disposes an active agent runtime. + +## Requirements for a throwaway prototype + +Build the prototype outside the production Pi provider first. It should prove the transport and lifecycle contract, not gVisor policy or final API design. + +### Minimum design + +1. Run a small supervisor independently of the Paseo daemon. The supervisor, not the daemon, spawns Pi and owns stdin/stdout/stderr for the full runtime lifetime. +2. Expose a local reconnectable endpoint, preferably a Unix-domain socket, keyed by a durable random bridge runtime ID. Authenticate the caller and reject a second active writer. +3. Frame every Pi stdout JSONL record with a bridge-owned monotonic sequence before forwarding it. Retain frames until the daemon acknowledges a sequence. Fail loudly on buffer or spool exhaustion; never silently skip ahead. +4. Continuously drain Pi stdout while no daemon is attached. Keep Pi stdin open. Preserve unanswered `extension_ui_request` frames and process-exit state for replay. +5. Namespace or translate daemon RPC request IDs. A restarted daemon currently begins again at `req_1`; stale responses from the pre-crash generation must not satisfy new requests. +6. Persist enough discovery state for a replacement daemon to find the bridge: Paseo agent ID, bridge runtime ID, Pi PID/start identity, Pi session file, daemon acknowledgment cursor, process state, and active Paseo turn ID. +7. Define commands with explicit semantics: attach with last acknowledged bridge sequence, write command under an exclusive lease, acknowledge events, inspect state, close Pi, and abandon an orphan. +8. Separate **daemon disconnect** from **agent close**. Disconnect keeps Pi alive for a bounded grace period; Stop, archive, workspace teardown, or explicit provider close must still terminate it. + +### Required demonstration + +Use a controllable fake Pi stream first, then a real Pi turn that executes a long-running tool: + +- kill the Paseo-side prototype client with `SIGKILL` after `turn_start`; +- show that the Pi PID and provider/tool operation remain unchanged; +- reconnect a new client using only persisted bridge identity and its last acknowledgment; +- replay every missed frame in order and deliver the terminal turn event exactly once; +- issue `abort` or steering after reattachment and show that it reaches the original Pi process; +- restart during an unanswered extension UI request and successfully answer the replayed request; +- verify duplicate writers, stale generations, replay-buffer overflow, Pi exit, and supervisor exit produce explicit terminal outcomes. + +The prototype succeeds only if the reconstructed Paseo timeline matches an uninterrupted control run and the same Pi process completes the turn. Reopening the JSONL session in a new Pi process is a failed result for this ticket. + +## Implications for backend and lifecycle tickets + +- **Backend transport:** introduce a reconnectable runtime transport below `PiRpcAgentSession`; do not teach `JsonlRpcProcess` to pretend anonymous pipes are reconnectable. Persist bridge identity separately from Pi's session-file `nativeHandle`. +- **Startup reconciliation:** before normal native resume, ask the supervisor for live runtimes and rebind matching Paseo agents. Restore the same active Paseo turn identity and replay through the normal provider event reducer before accepting new prompts. +- **Lifecycle ownership:** graceful daemon restart must detach from bridge-owned runtimes rather than call `session.close()`. Explicit agent close/archive must still cross the bridge and terminate Pi. Add orphan grace, adoption fencing, and supervisor-loss behavior. +- **Timeline correctness:** Pi's session entries can supplement completed-history reconciliation, but bridge sequence/acknowledgment must be the authority for transient active-turn events. Deduplicate replay before broadcasting to clients. +- **Permissions:** pending extension UI requests are part of the active turn. Their IDs and resolution state must survive daemon replacement, and only one daemon generation may answer them. +- **Failure reporting:** if Pi or the supervisor dies, mark the original turn failed once, then allow native session resume. Do not report native resume as continuation of that turn. +- **Sandbox boundary:** the gVisor-owned supervisor is the natural owner for this bridge if it already outlives daemon processes. Its control socket and spool become security-sensitive host interfaces and need per-runtime authorization, bounded storage, and cleanup. + +Backend and lifecycle implementation tickets should not promise active-turn continuity until the bridge prototype passes the crash/replay demonstration above.