feat: add durable asset synchronization worker

This commit is contained in:
jb
2026-09-19 22:15:23 -04:00
parent 2675ea8114
commit fe7d218195
36 changed files with 2218 additions and 189 deletions
+8 -4
View File
@@ -44,11 +44,12 @@ Use [the live deployment guide](docs/live-setup.md) for `dev.evebot.app`, the re
bunx convex dev --once
```
`APP_AUTH_ISSUER`, `APP_AUTH_JWKS` and `CONVEX_BRIDGE_SECRET` must be set in **Convex's environment**, not merely the Docker process environment. The configure script handles these three. Custom JWT audience is `eve-intelligence`; the public JWKS data URI avoids a boot dependency on the web server.
`APP_AUTH_ISSUER`, `APP_AUTH_JWKS`, `CONVEX_BRIDGE_SECRET` and `CONVEX_SYNC_WORKER_SECRET` must be set in **Convex's environment**, not merely the Docker process environment. The direct configure command reads the local private `.env`, including `CONVEX_SELF_HOSTED_ADMIN_KEY`; never put that admin key in a web or worker environment. Custom JWT audience is `eve-intelligence`; the public JWKS data URI avoids a boot dependency on the web server.
6. Register an EVE application and exact callback at <https://developers.eveonline.com/>. Set `EVE_CLIENT_ID`, `EVE_CALLBACK_URL`, and `EVE_CLIENT_SECRET` for a confidential web client. A public PKCE client leaves the secret blank. Initial login requests `esi-assets.read_assets.v1`; later dataset upgrades preserve only server-allowlisted existing scopes.
7. Set `ESI_USER_AGENT` with your actual operator contact. Optional `TYPESAFE_API_KEY` enables server-side JEV routing. Set public/internal Convex URLs as described in `.env.example`.
8. Build the fixture reference database and start `bun run dev`. Connect a character, synchronize it, and add more characters with **Connect EVE Character**. Each connects explicitly to the current application user.
9. In a second terminal, create a private `.env.worker` containing only `CONVEX_URL`, `CONVEX_SYNC_WORKER_SECRET`, `TOKEN_ENCRYPTION_KEY`, `EVE_CLIENT_ID`, optional `EVE_CLIENT_SECRET`, `EVE_CALLBACK_URL`, `ESI_USER_AGENT`, and worker tuning values, then start the worker with `bun --env-file=.env.worker run sync:worker`. Do not source the full web `.env`; the worker launcher intentionally rejects bridge/admin/JWT/JEV credentials.
**Important:** the fixture DB contains only a handful of types. Live assets absent from it remain explicitly unpriced/unknown. Do not describe a fixture-backed report as a complete real-character valuation. See [full-import handoff](docs/reference-operations.md) before ingesting real releases.
@@ -60,10 +61,11 @@ Compose uses Convex's default SQLite backend; it introduces no other datastore.
- `backend`: persistent Convex SQLite volume, API bound to loopback by default (the configured Pangolin deployment explicitly overrides this).
- `web`: Bun web service with read-only reference volume.
- `sync-worker`: dedicated durable asset-sync worker with a bundled runtime and worker-only secret.
- `reference-worker`: dedicated one-shot heavy-ingestion process (`tools` profile).
- `dashboard`: optional loopback-only management UI (`admin` profile).
For local container use, set `APP_ORIGIN` and `APP_AUTH_ISSUER` to `http://localhost:3000` and the EVE callback to `http://localhost:3000/auth/eve/callback`. Repeat `convex:configure` after changing issuer.
For local container use, set `APP_ORIGIN` and `APP_AUTH_ISSUER` to `http://localhost:3000` and the EVE callback to `http://localhost:3000/auth/eve/callback`. Repeat `bun run convex:configure` after changing issuer.
```sh
docker compose up -d backend
@@ -71,6 +73,8 @@ docker compose up -d backend
# Explicit production archive import (~109 MiB for the reviewed releases):
docker compose --profile tools run --rm --build reference-worker
docker compose up -d --build web
# Start the explicitly profiled durable asset worker; Compose forwards only its worker environment:
docker compose --profile worker up -d --build sync-worker
# Optional management UI:
docker compose --profile admin up -d dashboard
```
@@ -109,9 +113,9 @@ docs/research/ current official-source evidence and captured ESI con
## Operational limits
- The configured development deployment has passed live TypeSafe routing and public Convex WebSocket checks; the operator has confirmed real EVE sign-in, asset synchronization, and matching Assets/chat valuation totals for one character. Live multi-character linking and revocation/transfer scenarios remain separate checks.
- This checkpoint records local/disposable prototype verification only. It makes no claim of deployed/live ESI verification or production readiness; live multi-character linking, revocation and transfer scenarios remain deferred.
- Production imports use reviewed source adapters and immutable SQLite artifacts. CCP NPC station IDs, system/type/owner/position facts are imported when `npcStations.jsonl` is present; station names remain optional and the production profile explicitly marks station coverage incomplete until an isolated measured replay. Twenty-one dangling blueprint references in the reviewed release are preserved explicitly rather than fabricated. See reference operations and the verification record for executed import checks and limitations.
- Asset synchronization runs on demand in a single Bun web process. Convex leases/CAS protect staging and token rotation, but durable background character-sync job recovery and scheduled refresh are future work.
- Asset synchronization is enqueued by the web process and completed by the separate durable `sync-worker`. Convex leases/CAS protect staging and token rotation; start the worker explicitly with the worker-scoped launcher before testing live synchronization.
- ESI page collections are not transactional snapshots. Stability checks reduce risk; cache rollover/mutation may require a later retry.
- Location IDs are shown explicitly as unresolved. Fitted/nested assets are distinct ESI items; no contents multiplier is applied to ship prices.
- Conversations/execution summaries persist for authenticated live users; this first UI does not yet provide a history browser. A pending guest request is kept only in tab `sessionStorage` and rechecked after login/sync.