feat: add sqlite storage and terminal ui
This commit is contained in:
Mohamed Boudra
@@ -0,0 +1,217 @@
|
||||
# Storage Revamp Plan
|
||||
|
||||
Status: active rollout, phases 1 and 2 complete
|
||||
|
||||
This document now tracks the storage revamp as it exists today, not as a speculative design exercise.
|
||||
The DB foundation and the project/workspace identity cutover have landed. What remains is the explicit
|
||||
creation/archive surface cleanup, timeline durability cutover, and final removal of legacy paths.
|
||||
|
||||
## Goals
|
||||
|
||||
- make structured records durable in Drizzle + SQLite
|
||||
- make projects and workspaces explicit first-class records
|
||||
- stop deriving project/workspace identity from agent `cwd`
|
||||
- keep agent snapshot persistence behind clear ownership
|
||||
- move committed timeline history to storage-owned rows
|
||||
- remove legacy JSON and in-memory authority once the DB path is proven
|
||||
|
||||
## Out of scope
|
||||
|
||||
- moving config, keypairs, push tokens, or server identity into the DB
|
||||
- persisting raw provider deltas or transport-only chunk streams
|
||||
- designing a hosted/remote database story beyond keeping the schema portable
|
||||
- durable reasoning history unless product explicitly asks for it later
|
||||
|
||||
## Current state
|
||||
|
||||
The storage revamp is no longer hypothetical.
|
||||
|
||||
Completed:
|
||||
|
||||
- Drizzle + SQLite database bootstrap is in place
|
||||
- `projects`, `workspaces`, and `agent_snapshots` use integer primary keys
|
||||
- `workspaces.project_id` and `agent_snapshots.workspace_id` cascade on delete
|
||||
- `agent_snapshots.workspace_id` is `NOT NULL`
|
||||
- legacy JSON import feeds the DB-backed structured records
|
||||
- project/workspace records use explicit `directory` fields instead of path-as-identity
|
||||
- session read paths now use persisted workspace/project rows instead of cwd/git derivation
|
||||
- `workspace-reconciliation-service.ts` is deleted
|
||||
- `workspace-registry-bootstrap.ts` is deleted
|
||||
- `workspace-registry-model.ts` is reduced to `normalizeWorkspaceId`
|
||||
|
||||
Still pending:
|
||||
|
||||
- explicit `create_project` / `create_workspace` API cleanup
|
||||
- final archive cascade behavior for descendants and live agents
|
||||
- committed timeline storage cutover
|
||||
- removal of remaining legacy JSON and in-memory committed-history authority
|
||||
|
||||
## Converged decisions
|
||||
|
||||
### Structured record authority
|
||||
|
||||
Projects, workspaces, and agent snapshots are DB-backed structured records.
|
||||
The server should not recreate project/workspace identity from:
|
||||
|
||||
- git remotes
|
||||
- worktree main-repo roots
|
||||
- normalized cwd strings
|
||||
|
||||
Temporary exception:
|
||||
|
||||
- agent creation may still find-or-create a workspace by directory if the UI has not yet provided
|
||||
`workspaceId` explicitly
|
||||
|
||||
That fallback is transitional and should be deleted once the client always sends the workspace id.
|
||||
|
||||
### Storage seams
|
||||
|
||||
The useful seams remain concrete and domain-shaped:
|
||||
|
||||
- `ProjectRegistry`
|
||||
- `WorkspaceRegistry`
|
||||
- `AgentSnapshotStore`
|
||||
- `AgentTimelineStore`
|
||||
|
||||
There is no reason to reintroduce a reconciliation service layer for project/workspace identity.
|
||||
|
||||
### Timeline contract
|
||||
|
||||
The long-term timeline contract remains:
|
||||
|
||||
- committed rows are durable, canonical history
|
||||
- provisional live updates are transient subscription state
|
||||
- committed history is fetched by seq
|
||||
- provider history replay is not the durability mechanism
|
||||
|
||||
The structured-record cutover is complete before the timeline cutover so timeline rows can rely on
|
||||
stable DB-backed agent and workspace identity.
|
||||
|
||||
## Remaining phases
|
||||
|
||||
### Phase 3: Explicit creation and archive cleanup
|
||||
|
||||
Goal:
|
||||
Remove the last transitional write paths that still infer state from directories.
|
||||
|
||||
Required work:
|
||||
|
||||
- add explicit `create_project` handling
|
||||
- add explicit `create_workspace` handling
|
||||
- make agent creation require `workspaceId` once the UI is ready
|
||||
- finish archive semantics for workspaces/projects and any descendant agent state
|
||||
- remove the temporary find-or-create-by-directory fallback from agent creation
|
||||
|
||||
Exit gate:
|
||||
|
||||
- project/workspace creation is explicit end to end
|
||||
- no normal creation path infers identity from cwd or git metadata
|
||||
- archive flows behave consistently for structured records and live runtime state
|
||||
|
||||
### Phase 4: Timeline storage cutover
|
||||
|
||||
Goal:
|
||||
Make committed history durable and storage-owned.
|
||||
|
||||
Required work:
|
||||
|
||||
- make `AgentTimelineStore` authoritative for committed history
|
||||
- write one committed row per finalized logical item
|
||||
- support tail, before-seq, and after-seq queries from storage
|
||||
- stop treating provider history hydration as the normal refresh/load path
|
||||
- keep provisional live updates in memory only
|
||||
|
||||
Exit gate:
|
||||
|
||||
- committed history survives daemon restart
|
||||
- reconnect uses committed catch-up plus future live events without gaps or duplicates
|
||||
- unloaded agents can serve committed history from storage alone
|
||||
|
||||
### Phase 5: Legacy cleanup
|
||||
|
||||
Goal:
|
||||
Remove compatibility paths after the DB-backed model is fully authoritative.
|
||||
|
||||
Required work:
|
||||
|
||||
- remove legacy JSON authority for structured records
|
||||
- remove in-memory committed-history ownership
|
||||
- remove provider-history rehydrate compatibility paths
|
||||
- trim dead protocol and reducer logic from the pre-storage model
|
||||
- update architecture docs to match the final model
|
||||
|
||||
Exit gate:
|
||||
|
||||
- there is one durable storage path for structured records
|
||||
- there is one durable storage path for committed timeline history
|
||||
- the runtime no longer depends on the removed JSON/in-memory model
|
||||
|
||||
## Data model summary
|
||||
|
||||
### Projects
|
||||
|
||||
- integer primary key
|
||||
- `directory` is unique
|
||||
- `display_name`
|
||||
- `kind`: `git | directory`
|
||||
- optional `git_remote`
|
||||
- timestamps and archive state
|
||||
|
||||
### Workspaces
|
||||
|
||||
- integer primary key
|
||||
- belongs to a project by `project_id`
|
||||
- `directory` is unique
|
||||
- `display_name`
|
||||
- `kind`: `checkout | worktree`
|
||||
- timestamps and archive state
|
||||
|
||||
### Agent snapshots
|
||||
|
||||
- `agent_id` remains the primary key
|
||||
- belongs to a workspace by integer `workspace_id`
|
||||
- `workspace_id` is required
|
||||
- timestamps, lifecycle state, persistence metadata, attention metadata, archive state
|
||||
|
||||
### Timeline rows
|
||||
|
||||
Target shape once Phase 4 lands:
|
||||
|
||||
- `agent_id`
|
||||
- committed `seq`
|
||||
- committed timestamp
|
||||
- canonical finalized item payload
|
||||
|
||||
Not part of durable history:
|
||||
|
||||
- raw streaming chunks
|
||||
- provisional assistant text
|
||||
- provisional reasoning text
|
||||
|
||||
## Verification requirements
|
||||
|
||||
Every remaining phase should keep the same bar:
|
||||
|
||||
- `npm run typecheck`
|
||||
- targeted tests for the touched storage/session/runtime paths
|
||||
- migration/import coverage when storage authority changes
|
||||
- reconnect and catch-up scenario coverage when timeline behavior changes
|
||||
|
||||
At minimum, timeline cutover must explicitly prove:
|
||||
|
||||
- `fetch-after-seq`
|
||||
- `fetch-before-seq`
|
||||
- restart durability
|
||||
- no-gap/no-duplicate reconnect behavior
|
||||
|
||||
## Main risks
|
||||
|
||||
- timeline work reintroduces provider-history replay as hidden authority
|
||||
- archive behavior diverges between stored records and live in-memory agents
|
||||
- explicit creation work leaves the transitional cwd fallback in place too long
|
||||
- cleanup stalls after compatibility paths stop being exercised
|
||||
|
||||
## Rule of thumb
|
||||
|
||||
If a new change needs to ask "what can we infer from this cwd?" for project or workspace identity,
|
||||
it is probably moving in the wrong direction.
|
||||
Reference in New Issue
Block a user