feat: add durable asset synchronization worker
This commit is contained in:
@@ -30,7 +30,7 @@ Dedicated Bun reference worker
|
||||
|
||||
EVE authorization-code + PKCE and encrypted token storage runs in Bun. First login creates an app user keyed to a verified character and owner claim; explicit linking attaches additional characters to the current user. Conflicting ownership never silently merges accounts. Session cookies carry random opaque values; only hashes are stored. Application RS256 JWTs are short-lived and audience-scoped for the Convex browser client. EVE access/refresh tokens are never browser credentials.
|
||||
|
||||
Asset sync stages bounded chunks, then atomically changes a character's active generation after all pages succeed. A failed fetch preserves the previous complete generation. Sync metadata records freshness/error, and the domain gate refuses to present missing/stale coverage as complete. ESI page caches are isolated by authorization identity; no partial paginated result is accepted as a full snapshot. Single-host deployment initially runs one web process; token refresh also uses durable compare-and-swap to prevent a stale refresh from overwriting a rotated token.
|
||||
Asset sync is a durable `asset_sync` job owned by the authenticated user; `/api/sync` only enqueues a server-derived, deduplicated request. A dedicated Bun worker claims bounded due work with renewable tokenized leases, stages validated chunks, then atomically changes a character's active generation after all pages succeed. A failed fetch preserves the previous complete generation. Sync metadata records freshness/error, and the domain gate refuses to present missing/stale coverage as complete. ESI page caches are isolated by authorization identity; no partial paginated result is accepted as a full snapshot. Cooldowns are persisted in Convex and shared by workers. Token refresh is coordinated by a durable per-character lease and compare-and-swap; uncertain rotating-token results fail closed as `reconnect_required` instead of replaying a refresh. The worker receives no application signing, JEV, reference, or deployment-admin secrets. This is a local T07 prototype; it requires `CONVEX_SYNC_WORKER_SECRET` and the explicit worker-scoped startup path. T08 location/containment enrichment remains feature-local, with integration at the post-assets/pre-publication seam still pending.
|
||||
|
||||
## Valuation
|
||||
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
# T07 durable asset synchronization
|
||||
|
||||
Status: prototype implementation. This record describes the local, disposable
|
||||
queue and does not claim production readiness.
|
||||
|
||||
## Boundaries
|
||||
|
||||
`POST /api/sync` authenticates the application session and only enqueues
|
||||
server-derived asset jobs. It does not refresh OAuth credentials or call ESI.
|
||||
Partial enqueue results carry per-character safe skip reasons; an all-failed
|
||||
queue response is not reported as `202`. The assets and chat surfaces poll
|
||||
`/api/sync/status?job=<id>`, expose per-character queued/terminal state and
|
||||
cancellation, and rerun the report only after every queued job succeeds.
|
||||
`POST /api/sync/cancel` fences a job for its owning user. The Convex bridge
|
||||
checks the session proof, character ownership, active grant and exact dataset
|
||||
scope again when enqueueing and claiming.
|
||||
|
||||
Jobs are user-owned `asset_sync` rows. The server derives the dedupe key
|
||||
`user:character:assets`; only `queued` and `running` rows deduplicate. A claim
|
||||
captures grant revision, authorization epoch, credential version and scopes.
|
||||
Claims have random lease tokens and a ten-minute default lease, renewed every
|
||||
minute by the worker. Expired claims are recovered by the next claim
|
||||
transaction. Selection is bounded to 100 due rows and prefers users with no
|
||||
running work before due/creation order, providing per-user fairness without a
|
||||
second scheduler.
|
||||
|
||||
Prototype budgets are conservative and explicit (worker lease/poll values are environment-configurable): five attempts per job,
|
||||
250 assets per staged chunk, 4,096 chunks, 50,000 ESI rows, at most eight
|
||||
active jobs per user, a ten-minute sync lease, and a 24-hour maximum retry
|
||||
delay. ESI request/page/response limits are
|
||||
owned by the T06 transport. Queue backpressure is visible as a bounded failed
|
||||
job, never a silently truncated generation.
|
||||
|
||||
## Fencing and delivery
|
||||
|
||||
A worker starts a generation only through the job lease, stages through a
|
||||
job-token mutation, and commits success or failure through a mutation that
|
||||
rechecks the lease, job fence, character authorization and generation fence.
|
||||
Success and failure therefore cannot mutate state after cancellation, scope
|
||||
loss, credential rotation, quarantine or lease expiry. Staging is contiguous;
|
||||
identical duplicate chunk delivery is idempotent while a changed duplicate is
|
||||
rejected. A successful commit is idempotent if a response is lost. Failed
|
||||
snapshots preserve the previous complete generation.
|
||||
|
||||
`isSingleton` is validated by the T06 ESI parser and is carried unchanged in
|
||||
every stored asset row; absent legacy values remain unknown.
|
||||
|
||||
## Refresh coordination
|
||||
|
||||
Refresh leases are durable per user/character rows, not process maps. The
|
||||
worker decrypts a grant only inside the dedicated worker, performs exactly one
|
||||
remote rotating-token refresh, then commits the replacement with the expected
|
||||
credential version. The committed credential version/revision/epoch are the
|
||||
only authority values used for ESI and publication. A lost, timed-out, or
|
||||
otherwise uncertain refresh result is never replayed: the character becomes
|
||||
`reconnect_required`, credentials are removed, active work is quarantined, and
|
||||
all private datasets are blocked until explicit reconnect.
|
||||
|
||||
The worker receives only its dedicated `CONVEX_SYNC_WORKER_SECRET`,
|
||||
token-encryption, EVE OAuth and ESI user-agent credentials. Worker mutations do
|
||||
not accept the general `CONVEX_BRIDGE_SECRET`; login/session/tenant operations
|
||||
remain on the web-only bridge. It does not receive application JWT signing
|
||||
keys, JEV keys, reference paths/artifacts, or deployment-admin credentials. The
|
||||
web process no longer performs request-bound refresh or asset paging. The image
|
||||
build bundles `workers/sync.ts` to `build/sync-worker.js`, so the runtime never
|
||||
executes a raw `$lib`-aliased TypeScript entry. Compose starts that artifact
|
||||
through `scripts/run-sync-worker.ts`; the launcher rejects bridge, admin,
|
||||
JWT-signing and JEV credentials even when `bun run sync:worker` is invoked
|
||||
directly. Start it with `bun run secrets:run --worker -- docker compose --profile worker up -d
|
||||
--build sync-worker`, never with the general web/deployment secret modes. The Compose
|
||||
service is explicitly profiled so plain `docker compose up` cannot launch a worker
|
||||
with empty credentials.
|
||||
|
||||
## Cooldowns and recovery
|
||||
|
||||
Provider cooldowns are persisted in `syncCooldowns`, keyed by character
|
||||
context and shared provider group. Asset jobs carry the provider group, and
|
||||
claim selection plus the worker both honor the group's eligible-at timestamp
|
||||
before ESI work. ESI remains the single transport retry/cooldown
|
||||
implementation. No public-feed scheduler is introduced by this task.
|
||||
|
||||
Retention, migrations, production quotas, remote revocation, reference
|
||||
extraction and public market orchestration remain deferred as required by the
|
||||
parallel-worktree plan.
|
||||
+4
-3
@@ -37,7 +37,7 @@ bun run secrets:status
|
||||
bun run secrets:run -- bun scripts/check-live-config.ts
|
||||
```
|
||||
|
||||
Expected application keys: `EVE_CLIENT_ID`, optional `EVE_CLIENT_SECRET` for confidential SSO, `TYPESAFE_API_KEY`, `TOKEN_ENCRYPTION_KEY`, `CONVEX_BRIDGE_SECRET`, `APP_AUTH_PRIVATE_KEY`, and `APP_AUTH_JWKS`. Generate the last four **once** using `bun run keys:generate`; import `.env.keys` into the selected Infisical environment without replacing existing API keys. The generator refuses to overwrite an existing file. After verifying the import and key pairing, remove the temporary copy. Back up the vault: changing/loss of the encryption key makes stored EVE credentials unreadable. Do not regenerate keys casually during deployment.
|
||||
Expected application keys: `EVE_CLIENT_ID`, optional `EVE_CLIENT_SECRET` for confidential SSO, `TYPESAFE_API_KEY`, `TOKEN_ENCRYPTION_KEY`, `CONVEX_BRIDGE_SECRET`, `CONVEX_SYNC_WORKER_SECRET`, `APP_AUTH_PRIVATE_KEY`, and `APP_AUTH_JWKS`. Generate the key material **once** using `bun run keys:generate`; import `.env.keys` into the selected Infisical environment without replacing existing API keys. Configure both distinct bridge secrets in Convex with `bun run secrets:run --deployment --configure -- bun run convex:configure`; the sync worker receives only `CONVEX_SYNC_WORKER_SECRET`, while web receives only `CONVEX_BRIDGE_SECRET`. Use `bun run secrets:run --worker -- ...` for worker commands so session/JEV signing and general bridge secrets are not injected. The generator refuses to overwrite an existing file. After verifying the import and key pairing, remove the temporary copy. Back up the vault: changing/loss of the encryption key makes stored EVE credentials unreadable. Do not regenerate keys casually during deployment.
|
||||
|
||||
The helper is an operator launcher, not a sandbox. Commands it launches can read injected secrets: never run `env`, `printenv`, shell tracing, or unrestricted `docker compose config` through it. Use `docker compose config --quiet` for validation. Docker administrators can inspect container environments; Infisical startup injection does not conceal secrets from host administrators. Updating a vault value requires recreating the affected container with fresh injected values. Encryption/signing-key rotation needs a separate migration plan.
|
||||
|
||||
@@ -47,7 +47,7 @@ Run from the repository root:
|
||||
|
||||
```sh
|
||||
bun run secrets:run -- docker compose up -d backend
|
||||
bun run secrets:run --deployment -- bun run convex:configure
|
||||
bun run secrets:run --deployment --configure -- bun run convex:configure
|
||||
bun run secrets:run --deployment -- bun node_modules/convex/bin/main.js dev --once --typecheck disable
|
||||
```
|
||||
|
||||
@@ -61,9 +61,10 @@ Use the production import procedure in [reference operations](reference-operatio
|
||||
|
||||
```sh
|
||||
bun run secrets:run -- docker compose up -d --build web
|
||||
bun run secrets:run --worker -- docker compose --profile worker up -d --build sync-worker
|
||||
```
|
||||
|
||||
The container listens on 3000 internally but publishes **5173** on the host. This is the production SvelteKit/Bun server, not the Vite development server. The web mounts reference data read-only. The one-shot reference worker has the writable volume mount.
|
||||
The web container listens on 3000 internally but publishes **5173** on the host. This is the production SvelteKit/Bun server, not the Vite development server. The web mounts reference data read-only. The dedicated sync-worker is a separate long-running service: its image starts the bundled `build/sync-worker.js` through the restricted launcher and receives only worker-safe secrets. The one-shot reference worker has the writable volume mount.
|
||||
|
||||
## First live verification
|
||||
|
||||
|
||||
+16
-11
@@ -8,17 +8,20 @@
|
||||
|
||||
**Planning baseline:** `fdcc486` (`Initial Commit`). The roadmap records an earlier 97-test gate; each prototype change reruns the current local suite. Existing `jobs`, `userPreferences` and `referenceSyncMetadata` tables are placeholders, not services.
|
||||
|
||||
**T06 evidence:** The centralized ESI client now has fixed typed asset, asset-name,
|
||||
**T06/T07 evidence:** The centralized ESI client has fixed typed asset, asset-name,
|
||||
asset-location, wallet-balance, and market-read strategies with bounded response
|
||||
bytes/rows/pages, bounded retries, abort-first cleanup, validators, canonical int64
|
||||
helper bodies, complete scoped authority-context cache identity, cooldown queries,
|
||||
and runtime schemas. Asset snapshots report bounded non-atomic coverage; page-one
|
||||
revalidation is not presented as a transaction. T07 durable scheduling/worker
|
||||
orchestration, T08 location/service orchestration, and T13 wallet persistence/UI/
|
||||
service orchestration remain unimplemented; production-readiness gates remain
|
||||
deferred. The narrow `beginSync` expected-authority fence is part of T06's atomic
|
||||
bridge contract: it prevents transport cache authority from being published under
|
||||
a different T05 grant/revision/credential fence; it is not T07 durable worker orchestration.
|
||||
and runtime schemas. T07 durable scheduling/worker orchestration is implemented as
|
||||
a local prototype: the current full suite records 204 passing tests, including the
|
||||
bundled dedicated worker, durable leases, fencing, cancellation/recovery, refresh
|
||||
coordination and restricted startup. T08 location/service orchestration and T13
|
||||
wallet persistence/UI/service orchestration remain pending; T07/T08 integration is
|
||||
not complete and production-readiness gates remain deferred. No deployed/live ESI
|
||||
claim is made here. The narrow `beginSync` expected-authority fence is part of T06's
|
||||
atomic bridge contract: it prevents transport cache authority from being published
|
||||
under a different T05 grant/revision/credential fence; it is not T07 durable worker
|
||||
orchestration.
|
||||
|
||||
## Execution model
|
||||
|
||||
@@ -138,13 +141,15 @@ Implement the prototype identity enforcement needed by the pipeline: revalidate
|
||||
|
||||
### T07 — Durable asset queue and worker
|
||||
|
||||
**Own:** proposed `workers/sync.ts`, feature-local job/credential modules, job schema/store operations, `/api/sync` and replacement of request-bound sync orchestration; Compose wiring follows T01.
|
||||
**Status:** implemented local prototype; no deployment, live ESI, or production-readiness claim.
|
||||
|
||||
**Own:** `workers/sync.ts`, feature-local job/credential modules, job schema/store operations, `/api/sync` and replacement of request-bound sync orchestration; Compose wiring and restricted operator startup.
|
||||
|
||||
**Deliver:** enqueue/status instead of long-running request work; typed deduplicated jobs, bounded attempts/due times, cancellation, fair scheduling/backpressure, shared cooldown/error budgets, renewable leases and fenced success **and failure** publication. Stage/validate assets before active-generation publication and retain validated `isSingleton` facts throughout ingestion.
|
||||
|
||||
Share a per-character refresh lease across web and workers, retaining token CAS. At-least-once delivery is not exactly-once remote refresh: an uncertain rotated result produces actionable reauthorization instead of automatic replay. Keep reference extraction separate and minimize worker secrets. Public feeds must be coordinated rather than independently refreshed per process.
|
||||
Share a per-character refresh lease across web and workers, retaining token CAS. At-least-once delivery is not exactly-once remote refresh: an uncertain rotated result produces actionable reauthorization instead of automatic replay. Keep reference extraction separate and minimize worker secrets. Public feeds must be coordinated rather than independently refreshed per process. The worker requires the distinct `CONVEX_SYNC_WORKER_SECRET`; standalone startup uses a worker-only environment and Compose startup uses the explicit `worker` profile.
|
||||
|
||||
**Done:** process kill/restart, long paging beyond the old five-minute lease, duplicate delivery, cancellation, scope loss, stale-worker completion/failure, refresh races and uncertain rotation preserve the newest valid generation. Caps fail visibly. User-supplied jobs cannot gain system authority. Isolated migration/rollback and minimal worker configuration are exercised.
|
||||
**Verified locally:** process kill/restart, long paging beyond the old five-minute lease, duplicate delivery, cancellation, scope loss, stale-worker completion/failure, refresh races, uncertain rotation, restricted runtime startup and configuration boundaries. The full suite currently records 204 passing tests. T08 enrichment integration at the post-assets/pre-publication seam remains pending; production migration/release evidence remains deferred.
|
||||
|
||||
### T08 — Private locations, custom names and containment
|
||||
|
||||
|
||||
@@ -49,7 +49,7 @@ chmod 700 "$work"
|
||||
# Do not let an ambient shell, repository .env, or prior Compose project alter
|
||||
# this rehearsal. The explicit env file below contains only non-secret values;
|
||||
# synthetic key values are loaded separately from the mode-0600 key file.
|
||||
unset COMPOSE_FILE COMPOSE_ENV_FILES COMPOSE_PROJECT_NAME COMPOSE_PROFILES DOCKER_HOST APP_MODE APP_ORIGIN ORIGIN APP_RELEASE_ID APP_CAPABILITY_ALLOWLIST CONVEX_URL PUBLIC_CONVEX_URL CONVEX_SITE_ORIGIN CONVEX_PORT CONVEX_SITE_PORT WEB_PORT CONVEX_IMAGE CONVEX_DASHBOARD_IMAGE EVE_CALLBACK_URL EVE_CLIENT_ID EVE_CLIENT_SECRET ESI_USER_AGENT TYPESAFE_API_KEY APP_AUTH_ISSUER TOKEN_ENCRYPTION_KEY CONVEX_BRIDGE_SECRET APP_AUTH_PRIVATE_KEY APP_AUTH_JWKS CONVEX_SELF_HOSTED_URL CONVEX_SELF_HOSTED_ADMIN_KEY
|
||||
unset COMPOSE_FILE COMPOSE_ENV_FILES COMPOSE_PROJECT_NAME COMPOSE_PROFILES DOCKER_HOST APP_MODE APP_ORIGIN ORIGIN APP_RELEASE_ID APP_CAPABILITY_ALLOWLIST CONVEX_URL PUBLIC_CONVEX_URL CONVEX_SITE_ORIGIN CONVEX_PORT CONVEX_SITE_PORT WEB_PORT CONVEX_IMAGE CONVEX_DASHBOARD_IMAGE EVE_CALLBACK_URL EVE_CLIENT_ID EVE_CLIENT_SECRET ESI_USER_AGENT TYPESAFE_API_KEY APP_AUTH_ISSUER TOKEN_ENCRYPTION_KEY CONVEX_BRIDGE_SECRET CONVEX_SYNC_WORKER_SECRET APP_AUTH_PRIVATE_KEY APP_AUTH_JWKS CONVEX_SELF_HOSTED_URL CONVEX_SELF_HOSTED_ADMIN_KEY
|
||||
export CONVEX_BIND_ADDRESS=127.0.0.1 CONVEX_PORT=$((3200 + RANDOM % 500)) CONVEX_SITE_PORT=$((3700 + RANDOM % 500))
|
||||
export WEB_BIND_ADDRESS=127.0.0.1 WEB_PORT=$((4200 + RANDOM % 500))
|
||||
export APP_MODE=live APP_RELEASE_ID=restore-fixture APP_CAPABILITY_ALLOWLIST=asset_search,asset_valuation,asset_sync,assistant_dispatch,session_token,identity,subscriptions
|
||||
|
||||
+6
-2
@@ -1,7 +1,7 @@
|
||||
# Implementation roadmap
|
||||
|
||||
**Status:** planning proposal, not implementation or a public-launch declaration.
|
||||
**Baseline:** the operator has verified real EVE sign-in, asset synchronization, and matching Assets/chat reference valuations. The earlier implementation gate recorded 97 passing tests; local prototype checks are rerun as features land.
|
||||
**Baseline:** prior checkpoints historically recorded live sign-in, asset synchronization and matching Assets/chat reference valuations for one character (see [verification](verification.md)); that history is not new T07 evidence. This T07 checkpoint records a local, disposable prototype only, with no new deployed/live ESI or production-readiness claim. T07 durable synchronization is implemented and verified locally with the current 204-test suite. T08 location/containment enrichment remains a separate feature whose post-assets/pre-publication integration is still pending.
|
||||
**Evidence:** [architecture](architecture.md), [milestone 1](milestone.md), [verification](verification.md), [ESI research](research/eve.md), [expansion data/permission matrix](research/expansion-data-contracts.md), and [reference operations](reference-operations.md).
|
||||
|
||||
## 1. Decisions confirmed for this roadmap
|
||||
@@ -16,6 +16,10 @@
|
||||
|
||||
New service/table/path names below are proposed implementation boundaries, not existing APIs. Later phases need focused design and current contract verification before coding; they are not one large implementation task.
|
||||
|
||||
### T07 prototype checkpoint
|
||||
|
||||
T07 is implemented as a local prototype, not a deployment claim. It requires the distinct `CONVEX_SYNC_WORKER_SECRET` in Convex and the worker environment, alongside `TOKEN_ENCRYPTION_KEY`, EVE client credentials and `ESI_USER_AGENT`. Start Compose workers only through the explicit `worker` profile; standalone development uses a worker-only `.env.worker`. T08 core modules and their integration at the post-assets/pre-publication seam remain pending.
|
||||
|
||||
## 2. What exists versus what still needs building
|
||||
|
||||
| Area | Current implementation | Important gap |
|
||||
@@ -24,7 +28,7 @@ New service/table/path names below are proposed implementation boundaries, not e
|
||||
| Assets | Complete staged snapshots, tenant gates, exact reference totals | Real search/filtering, paging, containment, custom names, meaningful locations, history |
|
||||
| References | 53,001 types; recipes, dogma and system/gate tables; immutable publication | Runtime reader exposes only types/version; station table currently empty; search and domain-specific readers missing |
|
||||
| Prices | Global ESI average prices, held by the web process | Durable public feed coordination, regional/private books, history, depth allocation, fees |
|
||||
| Sync | On-demand work inside web requests; generation publication and token CAS | `jobs` is schema-only; no durable dispatcher, crash recovery, shared refresh lease or global worker budgets |
|
||||
| Sync | T07 durable queued jobs, dedicated worker, leases, fenced generation publication, cancellation/recovery and shared refresh coordination | T08 enrichment integration and production release evidence remain pending |
|
||||
| Finances | Signed integer-cent money helpers | Wallets, journals, transactions, own orders, history/coverage and reconciliation |
|
||||
| Industry | Recipe data plus placeholder interfaces/economics fields | Owned blueprint instances, skills/jobs, facility assumptions, formulas and optimizer |
|
||||
| Navigation | Exact light-year distance helper and universe data | Gate-distance service, ship-aware capital graph, dynamic constraints and qualified access evidence |
|
||||
|
||||
+2
-2
@@ -7,10 +7,10 @@ These are requirements for the first implementation, not a claim of a production
|
||||
- Connecting another character is an authenticated, explicit operation. Callback must preserve the initiating user/session binding. Character ownership conflicts fail closed; there is no implicit merge or transfer.
|
||||
- Refresh tokens are envelope encrypted using a server-held encryption key with authenticated associated data. No raw refresh token or client secret may be a browser property, public Convex query field, URL parameter, log or exception message.
|
||||
- Browser sessions use high-entropy opaque identifiers, server-side hashes, HttpOnly/SameSite cookies and bounded expiration. Application JWTs use a dedicated issuer/audience/key and short expiry, not EVE bearer tokens. Queries independently enforce the application identity and ownership.
|
||||
- Server bridge authority is distinct from the deployment administrator credential and browser sessions. All privileged Convex entrypoints require it. Deployment keys are not application runtime defaults.
|
||||
- Server bridge authority is distinct from the deployment administrator credential and browser sessions. Web/session operations require the general bridge secret; worker lifecycle operations require a separate `CONVEX_SYNC_WORKER_SECRET` and expose no login, session, tenant-administration or enqueue/cancel authority. Deployment keys are not application runtime defaults.
|
||||
- State-changing browser requests require an exact allowed origin. Normal app pages use `Referrer-Policy: same-origin`: global `no-referrer` would make native POST navigations send `Origin: null` and fail CSRF verification. Auth responses retain `no-referrer` so callback grant URLs never become referrers. Null/missing/foreign origins remain rejected. No arbitrary user-provided fetch URLs, SQL identifiers, executable function names or tool invocations.
|
||||
- Market/ESI errors are redacted before display. Observability reports route, status, timings and cache/rate-limit signals, never Authorization headers, OAuth codes or secrets.
|
||||
- Authenticated ESI caches are keyed by character and authorization context. A partial sync cannot replace a complete snapshot. Token refresh is serialized for a character; stale token rotations cannot overwrite newer credentials.
|
||||
- Authenticated ESI caches are keyed by character and authorization context. A partial sync cannot replace a complete snapshot. Durable jobs use tokenized renewable leases and fenced success/failure publication; duplicate staging is idempotent only for identical payloads. Token refresh is serialized per character with durable CAS; stale token rotations cannot overwrite newer credentials, and uncertain rotation outcomes fail closed as `reconnect_required` rather than replaying.
|
||||
- Fixture/demo data is explicitly opt-in and visibly marked. Live errors never trigger a silent fallback to demo assets or prices; live reference readers require a production-profile artifact.
|
||||
- Reference archives are untrusted input. The production importer reads members without extracting arbitrary filesystem paths, rejects unsafe paths and unsupported member types, and enforces compressed/decompressed/member limits, XZ memory limits, archive checksums, and schema validation. Artifact paths/manifests must remain inside the configured reference directory. Cache reuse and fallback verify canonical hashes and the active artifact’s bound provenance; fixture caches cannot satisfy a production import.
|
||||
- Infisical bootstrap credentials stay in an owner-only local file, separate from application keys. The launcher pins the approved instance and selected project/environment/path, suppresses raw credential-command output, and does not pass bootstrap credentials or the vault token to the application. Application secrets are injected only from an allowlist; deployment admin access requires an explicit operator command. Docker/host administrators can still inspect container environment values.
|
||||
|
||||
+82
-22
@@ -1,6 +1,6 @@
|
||||
# Verification record
|
||||
|
||||
Initial milestone verified locally on 2026-09-18; live development rollout checked on 2026-09-19 UTC with Bun 1.3.14. Dependency versions are pinned in `package.json`/`bun.lock`.
|
||||
This record retains historical verification facts from earlier checkpoints. The current T07 checkpoint is a local/disposable prototype only: it makes no new deployed, live-ESI, or production-readiness claim. Dependency versions are pinned in `package.json`/`bun.lock`.
|
||||
|
||||
## Initial offline/disposable-backend milestone
|
||||
|
||||
@@ -8,19 +8,91 @@ Initial milestone verified locally on 2026-09-18; live development rollout check
|
||||
- `bun test`: offline tests exercise domain ownership/freshness, precise integer-cent accumulation, unknown/unpriced/BPC behavior, typed JEV routing and abstention, cryptographic envelopes, PKCE, SSO JWT checks, Convex ownership/session/snapshot rules, ESI cache/pagination/limiter behavior, and reference promotion/rollback/source failure handling. Final run: **66 passed, 0 failed**, across 10 files.
|
||||
- `bun run build`: production Bun server builds. `svelte-adapter-bun` may emit an unresolved `bun:sqlite` warning while leaving that Bun builtin external. This is not a missing npm dependency: the resulting server and container were executed successfully under Bun.
|
||||
- `scripts/smoke-web.ts`: executed against both the built production server and the final Docker web container in explicit fixture mode. Assistant, Characters and Assets SSR routes succeeded. `/value` and `/assets` returned the same 208,600,000 integer cents (2,086,000.00 ISK), two characters and two excluded/unpriced stacks. Foreign origins and oversized request bodies were rejected.
|
||||
- Browser-bundle inspection: no references to `TOKEN_ENCRYPTION_KEY`, `CONVEX_BRIDGE_SECRET`, `APP_AUTH_PRIVATE_KEY`, `refresh_token` or `TYPESAFE_API_KEY` in `build/client`.
|
||||
- Browser-bundle inspection: no references to `TOKEN_ENCRYPTION_KEY`, `CONVEX_BRIDGE_SECRET`, `CONVEX_SYNC_WORKER_SECRET`, `APP_AUTH_PRIVATE_KEY`, `refresh_token` or `TYPESAFE_API_KEY` in `build/client`.
|
||||
- `docker compose config --quiet` and `docker build`: passed. The containerized fixture worker also executed with `--network none`; no archives were fetched.
|
||||
- Real self-hosted Convex: deployed functions/schema/auth to an isolated disposable backend and executed `scripts/smoke-convex.ts`. Verified session lookup, asset publication, preservation after failed staging, cross-user isolation, safe status projection, correctly signed wrong-audience JWT rejection, and session revocation. This caught and fixed an incorrect attempt to read reserved `aud` as a Convex identity custom claim; audience enforcement belongs in the backend `applicationID` auth configuration.
|
||||
No live EVE credentials, production volumes, or deployed endpoints were used for this T07 checkpoint. Optional disposable Convex smoke procedures remain documented below but are not evidence of a deployment.
|
||||
|
||||
The disposable backend/web containers were stopped and test-only environment files and keys removed after verification. Typecheck and all 66 offline tests were rerun successfully without those files.
|
||||
## Historical verification from prior checkpoints
|
||||
|
||||
Tested backend image:
|
||||
The following evidence was recorded before T07 and is retained as historical context. It is not evidence that this new T07 worker prototype has been deployed or exercised against live ESI.
|
||||
|
||||
- The initial disposable Convex smoke deployed functions/schema/auth to an isolated backend and verified session lookup, asset publication, preservation after failed staging, cross-user isolation, safe status projection, correctly signed wrong-audience JWT rejection, and session revocation. It caught and fixed an incorrect attempt to read reserved `aud` as a Convex identity custom claim; audience enforcement belongs in the backend `applicationID` auth configuration.
|
||||
- The disposable backend/web containers were stopped and test-only environment files and keys removed after that verification. Typecheck and all 66 offline tests were rerun successfully without those files.
|
||||
- Historical tested backend image:
|
||||
|
||||
```text
|
||||
ghcr.io/get-convex/convex-backend@sha256:3aa2ccd6697a75de53948b1ee531d7c9688fe09ef360748d75df46dfc888f69a
|
||||
```
|
||||
|
||||
### Historical live development rollout — 2026-09-19 UTC
|
||||
|
||||
- The integrated gate recorded **97 tests passed, 0 failed, across 16 files**; Svelte/TypeScript check passed with 0 errors/0 warnings, and the Docker web build passed.
|
||||
- An operator-created Infisical project was connected through a read-only Universal Auth identity. Generated app encryption/signing/bridge keys were imported, compared against the staging file, and locally verified for encryption round-trip and signing/JWKS consistency; the temporary `.env.keys` was removed and secret values were not displayed.
|
||||
- Persistent self-hosted Convex schema/functions/auth configuration was deployed. Its administrator credential was derived locally for explicit deployment commands, not stored in the web environment.
|
||||
- Pangolin/UniFi routing was verified: `https://dev-convex.evebot.app/version` returned HTTP 200 and a real WSS upgrade with `Origin: https://dev.evebot.app` passed. Pangolin authentication remained allowed on the main app but was disabled on the Convex API resource by the operator.
|
||||
- One real TypeSafe request through `JevIntentRouter` returned `asset_valuation`, model `jev-1.13.0`, confidence 1, in 915 ms; this proves connectivity/routing for that sample, not calibration.
|
||||
- One real public ESI `/markets/prices` request passed using pinned compatibility `2026-08-18`, returning fresh integer-cent estimates for Tritanium and Rifter. These remain global reference estimates, not Jita bids or executable prices.
|
||||
- Real Reference Data/Hoboleaks/CCP archives totaling **114,291,615 compressed bytes (~109 MiB)** were parsed. The corrected Docker worker compiled them with `--network none`, a 4 GiB memory limit and two CPUs, explicitly targeting `/app/data/reference` in the shared volume. Peak RSS was not measured. The full online download path has fixture-based receipt/validator tests; the real full rebuild used already downloaded archives rather than fetching another copy.
|
||||
- The web container opened the deployed production artifact, resolved Tritanium/Rifter, matched manifest/table counts, and passed SQLite integrity/foreign-key checks: **53,001 types, 5,082 blueprints, 8,490 systems, 13,978 gates, 21 explicitly recorded unresolved blueprint references**. Deployed version: `b9afed7830756a5e617fec31a2a88c917f07b8f64d2bc719e2176fb628c1e98b`.
|
||||
- Live anonymous web smoke passed after deployment: SSR routes, no demo fallback, asset/chat authentication gates, session-token protection, and CSRF rejection. A scan of **13 deployed client artifacts plus anonymous SSR output** found no managed secret values; bootstrap tokens and the deployment-admin key were absent from the web environment.
|
||||
|
||||
A subsequent operator test exposed `ERR_INVALID_RESPONSE` when submitting Connect Character: the global `no-referrer` response policy made native browser form submissions send `Origin: null`, which SvelteKit rejected before OAuth initiation. An isolated temporary-profile Firefox test reproduced that behavior and confirmed that `same-origin` preserves the native form Origin. App pages now use `same-origin` while auth responses retain `no-referrer`; null/missing/foreign origins remain rejected. The rebuilt deployment passed header and anonymous endpoint checks. EVE discovery and a server-to-Convex read also passed; diagnostics wrote no OAuth/character records. The operator then reached EVE login but reported a generic callback failure. Redacted callback diagnostics identified `scopes_invalid` after token verification; the parser was updated to accept EVE's singleton scope-string representation as well as arrays, with signed-token login/refresh regressions and malformed-claim rejection. The revised image and anonymous smoke passed. The operator subsequently confirmed successful character authorization and asset synchronization, including refresh-grant verification and owner continuity before asset publication, and confirmed that live Assets and chat `/value` returned matching totals without sharing the private amount.
|
||||
|
||||
Historical audit regressions covered native-form/CSRF boundaries, shared artifact-directory precedence, live readers rejecting fixture profiles, offline no-network behavior, archive receipts/replay, ZIP/TAR checksums and limits, duplicate records/JSON keys, Hoboleaks metadata, and immutable cache provenance/rollback/reuse. See [live setup](live-setup.md) and [reference operations](reference-operations.md).
|
||||
|
||||
## T07 durable worker boundary regressions
|
||||
|
||||
- Full local `bun test`: **204 passed, 0 failed** across 27 files.
|
||||
- `bun test tests/t07-sync.test.ts tests/t07-worker-runtime.test.ts tests/t07-route-boundaries.test.ts`: deterministic queue/fencing/retry regressions plus runtime alias/startup, missing-job/AbortSignal polling, HTTP 429/503 structured outcomes, forbidden worker-environment, hostile-origin, session-ordering and bounded-body checks (31 passed in the focused run).
|
||||
- `bun build workers/sync.ts --target bun`: produces the same alias-resolved worker artifact copied by the Docker build to `build/sync-worker.js`. A no-cache Docker image build completed, and `docker run --rm --network none` started the image's bundled worker through `scripts/run-sync-worker.ts` with inert Convex/EVE values and `SYNC_WORKER_STARTUP_CHECK=1`; no live keys or network calls were used. `scripts/run-sync-worker.ts` is separately tested to reject general bridge/admin/JWT/JEV credentials without echoing them.
|
||||
- The operational startup is `bun run secrets:run --worker -- docker compose --profile worker up -d --build sync-worker`; `--deployment`/`--configure` is reserved for Convex administration and configuration, not worker startup. Plain `docker compose up` leaves the worker stopped.
|
||||
|
||||
### T07 prototype stage manifest
|
||||
|
||||
The T07 stage includes the following files. T08 core files are intentionally excluded:
|
||||
|
||||
```text
|
||||
ghcr.io/get-convex/convex-backend@sha256:3aa2ccd6697a75de53948b1ee531d7c9688fe09ef360748d75df46dfc888f69a
|
||||
.env.example
|
||||
Dockerfile
|
||||
README.md
|
||||
compose.yaml
|
||||
docs/architecture.md
|
||||
docs/design-records/t07-durable-sync.md
|
||||
docs/live-setup.md
|
||||
docs/parallel-worktrees.md
|
||||
docs/restore-rehearsal.md
|
||||
docs/roadmap.md
|
||||
docs/security.md
|
||||
docs/verification.md
|
||||
package.json
|
||||
scripts/configure-convex.ts
|
||||
scripts/generate-keys.ts
|
||||
scripts/lib/infisical.ts
|
||||
scripts/release-check.ts
|
||||
scripts/run-sync-worker.ts
|
||||
scripts/secrets.ts
|
||||
src/convex/functions.ts
|
||||
src/convex/schema.ts
|
||||
src/lib/client/sync-queue.ts
|
||||
src/lib/domain/model.ts
|
||||
src/lib/server/application.ts
|
||||
src/lib/server/store.ts
|
||||
src/routes/+page.svelte
|
||||
src/routes/api/sync/+server.ts
|
||||
src/routes/api/sync/cancel/+server.ts
|
||||
src/routes/api/sync/status/+server.ts
|
||||
src/routes/assets/+page.svelte
|
||||
tests/infisical.test.ts
|
||||
tests/t07-route-boundaries.test.ts
|
||||
tests/t07-sync.test.ts
|
||||
tests/t07-worker-runtime.test.ts
|
||||
tests/workflow-config.test.ts
|
||||
workers/sync.ts
|
||||
```
|
||||
|
||||
## Reproducing the real Convex smoke
|
||||
Excluded T08 core files: `src/lib/domain/inventory-location.ts`, `src/lib/server/inventory/`, `tests/fixtures/inventory/`, `tests/inventory-containment.test.ts`, and `tests/inventory-location.test.ts`.
|
||||
|
||||
## Optional disposable Convex smoke (not part of this checkpoint)
|
||||
|
||||
Use a **disposable** backend, set its `CONVEX_SELF_HOSTED_URL` and admin key, generate application keys and configure/deploy as in the README, then run:
|
||||
|
||||
@@ -30,25 +102,13 @@ SMOKE_ALLOW_WRITE=1 bun scripts/smoke-convex.ts
|
||||
|
||||
This writes synthetic users/characters and must not be casually pointed at a production deployment. It needs `CONVEX_BRIDGE_SECRET`, `APP_AUTH_ISSUER` and `APP_AUTH_PRIVATE_KEY` in the calling environment. A wrong-audience test is signed locally; no EVE identity is impersonated and no real EVE account is used.
|
||||
|
||||
## Live development rollout — 2026-09-19 UTC
|
||||
## Deferred deployment and live-provider evidence
|
||||
|
||||
- Latest integrated gate (including native-form, callback diagnostics, and singleton-scope compatibility): **97 tests passed, 0 failed, across 16 files**; Svelte/TypeScript check passed with 0 errors/0 warnings. Docker web build passed.
|
||||
- Connected the operator-created Infisical project through a read-only Universal Auth identity. Generated app encryption/signing/bridge keys were imported by the operator, compared against the staging file, and locally verified for encryption round-trip and signing/JWKS consistency. The temporary `.env.keys` was then removed. Secret values were not displayed.
|
||||
- Persistent self-hosted Convex schema/functions/auth configuration deployed. Its administrator credential is derived locally for explicit deployment commands, not stored in the web environment.
|
||||
- Pangolin/UniFi routing was verified: `https://dev-convex.evebot.app/version` returned HTTP 200 and a real WSS upgrade with `Origin: https://dev.evebot.app` passed. Pangolin authentication remains allowed on the main app but was disabled on the Convex API resource by the operator.
|
||||
- One real TypeSafe request through `JevIntentRouter` returned `asset_valuation`, model `jev-1.13.0`, confidence 1, in 915 ms. This proves connectivity/routing for that sample, not calibration.
|
||||
- One real public ESI `/markets/prices` request passed using pinned compatibility `2026-08-18`, returning fresh integer-cent estimates for Tritanium and Rifter. These remain global reference estimates, not Jita bids or executable prices.
|
||||
- Real Reference Data/Hoboleaks/CCP archives totaling **114,291,615 compressed bytes (~109 MiB)** were parsed. The corrected Docker worker compiled them with `--network none`, a 4 GiB memory limit and two CPUs, explicitly targeting `/app/data/reference` in the shared volume. Peak RSS was not measured. The full online download path has fixture-based receipt/validator tests; the real full rebuild used the already downloaded archives rather than fetching another copy.
|
||||
- The web container's own read-only repository successfully opened the deployed production artifact, resolved Tritanium/Rifter, matched manifest/table counts, and passed SQLite integrity/foreign-key checks: **53,001 types, 5,082 blueprints, 8,490 systems, 13,978 gates, 21 explicitly recorded unresolved blueprint references**. Deployed version: `b9afed7830756a5e617fec31a2a88c917f07b8f64d2bc719e2176fb628c1e98b`.
|
||||
- Live anonymous web smoke passed after deployment: SSR routes, no demo fallback, asset/chat authentication gates, session-token protection, and CSRF rejection. A scan of **13 deployed client artifacts plus anonymous SSR output** found no managed secret values; bootstrap tokens and the deployment-admin key were absent from the web environment.
|
||||
|
||||
A subsequent operator test exposed `ERR_INVALID_RESPONSE` when submitting Connect Character. The global `no-referrer` response policy made native browser form submissions send `Origin: null`; SvelteKit rejected them before OAuth initiation. An isolated, temporary-profile Firefox test reproduced that behavior and confirmed that `same-origin` preserves the native form Origin. App pages now use `same-origin` while auth responses retain `no-referrer`; null/missing/foreign origins remain rejected. The rebuilt deployment passed header and anonymous endpoint checks. EVE discovery and a server-to-Convex read also passed; diagnostics wrote no OAuth/character records. The operator subsequently reached EVE login but reported a generic failure after callback. Redacted callback-stage diagnostics were added and deployed: only fixed failure categories and allowlisted protocol status/claim names are retained; provider descriptions, JWT payloads, credentials, codes, cookies, and user identifiers are excluded. The next redacted callback diagnostic identified `scopes_invalid` after token verification. The parser now accepts EVE's singleton scope-string representation as well as arrays, with signed-token login/refresh regressions and malformed-claim rejection. The revised image and anonymous smoke passed. After that deployment, the operator confirmed successful character authorization and asset synchronization. Explicit sync exercises refresh-grant verification and owner continuity before asset publication. The operator also confirmed that the live Assets page and chat `/value` return matching totals, without sharing the private amount. This completes the initial live sign-in/sync/reference-valuation verification for one character.
|
||||
|
||||
Audit regressions cover native-form header/CSRF boundaries, shared artifact-directory precedence, live readers rejecting fixture profiles, offline no-network behavior, archive receipts/replay, ZIP CRC/length/duplicate records, TAR checksums/end markers, duplicate JSON keys, Hoboleaks file metadata, and immutable cache provenance/rollback/reuse. See [live setup](live-setup.md) and [reference operations](reference-operations.md).
|
||||
No deployed endpoint, live ESI request, live EVE credential, or production rollout is claimed for this T07 checkpoint. Existing operational procedures remain reference material only; deployment, migration/rollback, live-provider, browser failure-injection and production-readiness evidence are deferred. See [live setup](live-setup.md) and [reference operations](reference-operations.md).
|
||||
|
||||
## Not claimed as verified
|
||||
|
||||
- Live revocation/character-transfer scenarios and additional-character linking; successful sign-in/sync for one real character is now operator-confirmed.
|
||||
- Live revocation/character-transfer scenarios, additional-character linking, live ESI behavior and deployed-artifact evidence are not claimed for this prototype.
|
||||
- Broad TypeSafe inference quality or the proposed 0.80 confidence threshold's calibration on EVE requests.
|
||||
- Compatibility with arbitrary future archive releases, station-data coverage, large-scale memory/performance benchmarks, or production-size market analytics.
|
||||
- End-to-end browser automation/visual accessibility audit; the web smoke verifies the built server and endpoints, not all browser interactions.
|
||||
|
||||
Reference in New Issue
Block a user