4.7 KiB
T07 durable asset synchronization
Status: prototype implementation. This record describes the local, disposable queue and does not claim production readiness.
Boundaries
POST /api/sync authenticates the application session and only enqueues
server-derived asset jobs. It does not refresh OAuth credentials or call ESI.
Partial enqueue results carry per-character safe skip reasons; an all-failed
queue response is not reported as 202. The assets and chat surfaces poll
/api/sync/status?job=<id>, expose per-character queued/terminal state and
cancellation, and rerun the report only after every queued job succeeds.
POST /api/sync/cancel fences a job for its owning user. The Convex bridge
checks the session proof, character ownership, active grant and exact dataset
scope again when enqueueing and claiming.
Jobs are user-owned asset_sync rows. The server derives the dedupe key
user:character:assets; only queued and running rows deduplicate. A claim
captures grant revision, authorization epoch, credential version and scopes.
Claims have random lease tokens and a ten-minute default lease, renewed every
minute by the worker. Expired claims are recovered by the next claim
transaction. Selection is bounded to 100 due rows and prefers users with no
running work before due/creation order, providing per-user fairness without a
second scheduler.
Prototype budgets are conservative and explicit (worker lease/poll values are environment-configurable): five attempts per job, 250 assets per staged chunk, 4,096 chunks, 50,000 ESI rows, at most eight active jobs per user, a ten-minute sync lease, and a 24-hour maximum retry delay. ESI request/page/response limits are owned by the T06 transport. Queue backpressure is visible as a bounded failed job, never a silently truncated generation.
Fencing and delivery
A worker starts a generation only through the job lease, stages through a job-token mutation, and commits success or failure through a mutation that rechecks the lease, job fence, character authorization and generation fence. Success and failure therefore cannot mutate state after cancellation, scope loss, credential rotation, quarantine or lease expiry. Staging is contiguous; identical duplicate chunk delivery is idempotent while a changed duplicate is rejected. A successful commit is idempotent if a response is lost. Failed snapshots preserve the previous complete generation.
isSingleton is validated by the T06 ESI parser and is carried unchanged in
every stored asset row; absent legacy values remain unknown.
Refresh coordination
Refresh leases are durable per user/character rows, not process maps. The
worker decrypts a grant only inside the dedicated worker, performs exactly one
remote rotating-token refresh, then commits the replacement with the expected
credential version. The committed credential version/revision/epoch are the
only authority values used for ESI and publication. A lost, timed-out, or
otherwise uncertain refresh result is never replayed: the character becomes
reconnect_required, credentials are removed, active work is quarantined, and
all private datasets are blocked until explicit reconnect.
The worker receives only its dedicated CONVEX_SYNC_WORKER_SECRET,
token-encryption, EVE OAuth and ESI user-agent credentials. Worker mutations do
not accept the general CONVEX_BRIDGE_SECRET; login/session/tenant operations
remain on the web-only bridge. It does not receive application JWT signing
keys, JEV keys, reference paths/artifacts, or deployment-admin credentials. The
web process no longer performs request-bound refresh or asset paging. The image
build bundles workers/sync.ts to build/sync-worker.js, so the runtime never
executes a raw $lib-aliased TypeScript entry. Compose starts that artifact
through scripts/run-sync-worker.ts; the launcher rejects bridge, admin,
JWT-signing and JEV credentials even when bun run sync:worker is invoked
directly. Start it with bun run secrets:run --worker -- docker compose --profile worker up -d --build sync-worker, never with the general web/deployment secret modes. The Compose
service is explicitly profiled so plain docker compose up cannot launch a worker
with empty credentials.
Cooldowns and recovery
Provider cooldowns are persisted in syncCooldowns, keyed by character
context and shared provider group. Asset jobs carry the provider group, and
claim selection plus the worker both honor the group's eligible-at timestamp
before ESI work. ESI remains the single transport retry/cooldown
implementation. No public-feed scheduler is introduced by this task.
Retention, migrations, production quotas, remote revocation, reference extraction and public market orchestration remain deferred as required by the parallel-worktree plan.