218 lines
6.8 KiB
Markdown
218 lines
6.8 KiB
Markdown
# Storage Revamp Plan
|
|
|
|
Status: active rollout, phases 1 and 2 complete
|
|
|
|
This document now tracks the storage revamp as it exists today, not as a speculative design exercise.
|
|
The DB foundation and the project/workspace identity cutover have landed. What remains is the explicit
|
|
creation/archive surface cleanup, timeline durability cutover, and final removal of legacy paths.
|
|
|
|
## Goals
|
|
|
|
- make structured records durable in Drizzle + SQLite
|
|
- make projects and workspaces explicit first-class records
|
|
- stop deriving project/workspace identity from agent `cwd`
|
|
- keep agent snapshot persistence behind clear ownership
|
|
- move committed timeline history to storage-owned rows
|
|
- remove legacy JSON and in-memory authority once the DB path is proven
|
|
|
|
## Out of scope
|
|
|
|
- moving config, keypairs, push tokens, or server identity into the DB
|
|
- persisting raw provider deltas or transport-only chunk streams
|
|
- designing a hosted/remote database story beyond keeping the schema portable
|
|
- durable reasoning history unless product explicitly asks for it later
|
|
|
|
## Current state
|
|
|
|
The storage revamp is no longer hypothetical.
|
|
|
|
Completed:
|
|
|
|
- Drizzle + SQLite database bootstrap is in place
|
|
- `projects`, `workspaces`, and `agent_snapshots` use integer primary keys
|
|
- `workspaces.project_id` and `agent_snapshots.workspace_id` cascade on delete
|
|
- `agent_snapshots.workspace_id` is `NOT NULL`
|
|
- legacy JSON import feeds the DB-backed structured records
|
|
- project/workspace records use explicit `directory` fields instead of path-as-identity
|
|
- session read paths now use persisted workspace/project rows instead of cwd/git derivation
|
|
- `workspace-reconciliation-service.ts` is deleted
|
|
- `workspace-registry-bootstrap.ts` is deleted
|
|
- `workspace-registry-model.ts` is reduced to `normalizeWorkspaceId`
|
|
|
|
Still pending:
|
|
|
|
- explicit `create_project` / `create_workspace` API cleanup
|
|
- final archive cascade behavior for descendants and live agents
|
|
- committed timeline storage cutover
|
|
- removal of remaining legacy JSON and in-memory committed-history authority
|
|
|
|
## Converged decisions
|
|
|
|
### Structured record authority
|
|
|
|
Projects, workspaces, and agent snapshots are DB-backed structured records.
|
|
The server should not recreate project/workspace identity from:
|
|
|
|
- git remotes
|
|
- worktree main-repo roots
|
|
- normalized cwd strings
|
|
|
|
Temporary exception:
|
|
|
|
- agent creation may still find-or-create a workspace by directory if the UI has not yet provided
|
|
`workspaceId` explicitly
|
|
|
|
That fallback is transitional and should be deleted once the client always sends the workspace id.
|
|
|
|
### Storage seams
|
|
|
|
The useful seams remain concrete and domain-shaped:
|
|
|
|
- `ProjectRegistry`
|
|
- `WorkspaceRegistry`
|
|
- `AgentSnapshotStore`
|
|
- `AgentTimelineStore`
|
|
|
|
There is no reason to reintroduce a reconciliation service layer for project/workspace identity.
|
|
|
|
### Timeline contract
|
|
|
|
The long-term timeline contract remains:
|
|
|
|
- committed rows are durable, canonical history
|
|
- provisional live updates are transient subscription state
|
|
- committed history is fetched by seq
|
|
- provider history replay is not the durability mechanism
|
|
|
|
The structured-record cutover is complete before the timeline cutover so timeline rows can rely on
|
|
stable DB-backed agent and workspace identity.
|
|
|
|
## Remaining phases
|
|
|
|
### Phase 3: Explicit creation and archive cleanup
|
|
|
|
Goal:
|
|
Remove the last transitional write paths that still infer state from directories.
|
|
|
|
Required work:
|
|
|
|
- add explicit `create_project` handling
|
|
- add explicit `create_workspace` handling
|
|
- make agent creation require `workspaceId` once the UI is ready
|
|
- finish archive semantics for workspaces/projects and any descendant agent state
|
|
- remove the temporary find-or-create-by-directory fallback from agent creation
|
|
|
|
Exit gate:
|
|
|
|
- project/workspace creation is explicit end to end
|
|
- no normal creation path infers identity from cwd or git metadata
|
|
- archive flows behave consistently for structured records and live runtime state
|
|
|
|
### Phase 4: Timeline storage cutover
|
|
|
|
Goal:
|
|
Make committed history durable and storage-owned.
|
|
|
|
Required work:
|
|
|
|
- make `AgentTimelineStore` authoritative for committed history
|
|
- write one committed row per finalized logical item
|
|
- support tail, before-seq, and after-seq queries from storage
|
|
- stop treating provider history hydration as the normal refresh/load path
|
|
- keep provisional live updates in memory only
|
|
|
|
Exit gate:
|
|
|
|
- committed history survives daemon restart
|
|
- reconnect uses committed catch-up plus future live events without gaps or duplicates
|
|
- unloaded agents can serve committed history from storage alone
|
|
|
|
### Phase 5: Legacy cleanup
|
|
|
|
Goal:
|
|
Remove compatibility paths after the DB-backed model is fully authoritative.
|
|
|
|
Required work:
|
|
|
|
- remove legacy JSON authority for structured records
|
|
- remove in-memory committed-history ownership
|
|
- remove provider-history rehydrate compatibility paths
|
|
- trim dead protocol and reducer logic from the pre-storage model
|
|
- update architecture docs to match the final model
|
|
|
|
Exit gate:
|
|
|
|
- there is one durable storage path for structured records
|
|
- there is one durable storage path for committed timeline history
|
|
- the runtime no longer depends on the removed JSON/in-memory model
|
|
|
|
## Data model summary
|
|
|
|
### Projects
|
|
|
|
- integer primary key
|
|
- `directory` is unique
|
|
- `display_name`
|
|
- `kind`: `git | directory`
|
|
- optional `git_remote`
|
|
- timestamps and archive state
|
|
|
|
### Workspaces
|
|
|
|
- integer primary key
|
|
- belongs to a project by `project_id`
|
|
- `directory` is unique
|
|
- `display_name`
|
|
- `kind`: `checkout | worktree`
|
|
- timestamps and archive state
|
|
|
|
### Agent snapshots
|
|
|
|
- `agent_id` remains the primary key
|
|
- belongs to a workspace by integer `workspace_id`
|
|
- `workspace_id` is required
|
|
- timestamps, lifecycle state, persistence metadata, attention metadata, archive state
|
|
|
|
### Timeline rows
|
|
|
|
Target shape once Phase 4 lands:
|
|
|
|
- `agent_id`
|
|
- committed `seq`
|
|
- committed timestamp
|
|
- canonical finalized item payload
|
|
|
|
Not part of durable history:
|
|
|
|
- raw streaming chunks
|
|
- provisional assistant text
|
|
- provisional reasoning text
|
|
|
|
## Verification requirements
|
|
|
|
Every remaining phase should keep the same bar:
|
|
|
|
- `npm run typecheck`
|
|
- targeted tests for the touched storage/session/runtime paths
|
|
- migration/import coverage when storage authority changes
|
|
- reconnect and catch-up scenario coverage when timeline behavior changes
|
|
|
|
At minimum, timeline cutover must explicitly prove:
|
|
|
|
- `fetch-after-seq`
|
|
- `fetch-before-seq`
|
|
- restart durability
|
|
- no-gap/no-duplicate reconnect behavior
|
|
|
|
## Main risks
|
|
|
|
- timeline work reintroduces provider-history replay as hidden authority
|
|
- archive behavior diverges between stored records and live in-memory agents
|
|
- explicit creation work leaves the transitional cwd fallback in place too long
|
|
- cleanup stalls after compatibility paths stop being exercised
|
|
|
|
## Rule of thumb
|
|
|
|
If a new change needs to ask "what can we infer from this cwd?" for project or workspace identity,
|
|
it is probably moving in the wrong direction.
|