68 lines
9.7 KiB
Markdown
68 lines
9.7 KiB
Markdown
# Reference data operations
|
|
|
|
The reference compiler builds immutable SQLite artifacts from checked-in fixtures or explicitly supplied source adapters. It does not download an archive by default.
|
|
|
|
## Fixture build
|
|
|
|
```sh
|
|
REFERENCE_DIR=data/reference bun workers/reference.ts --fixture
|
|
# or
|
|
bun run reference:fixture
|
|
```
|
|
|
|
## Production import
|
|
|
|
Production is explicit and uses the pinned public EVE Ref Reference Data, Hoboleaks, and CCP JSONL SDE releases. `--check` performs only independent HTTP metadata checks (no archive bytes); `--production` streams each archive to a bounded staging file, hashes/counts transport bytes, validates archive members, and publishes only after cross-source SQLite validation.
|
|
|
|
```sh
|
|
bun workers/reference.ts --check --production
|
|
bun workers/reference.ts --production
|
|
# Re-run an already downloaded, offline import:
|
|
bun workers/reference.ts --production --offline --archive-dir data/reference-live
|
|
```
|
|
|
|
The default production archive and artifact directory is `data/reference-live` (ignored by git); production publishes `data/reference-live/manifest.json`, while fixture builds remain under `data/reference`. An explicit `--directory` wins over `REFERENCE_DIR`, which wins over these defaults. `REFERENCE_LIVE_DIR` controls archive storage only. Compose explicitly publishes to `/app/data/reference`, the shared worker/web volume; the production web reader rejects fixture-profile artifacts. Runtime images must provide the `xz` executable (Debian package `xz-utils`) for streaming `.tar.xz` decompression; ZIP deflate uses the Bun runtime. No npm archive dependency is required. Production validation requires more than 5,000 systems, at least one station, and imports CCP's `mapSolarSystems`, `mapConstellations`, `mapRegions`, `mapStargates`, and `npcStations` JSONL members. A production candidate missing `npcStations.jsonl` or containing no station rows fails closed rather than claiming any station rows. The importer parses official station ID/system/type/owner/position facts; names remain unknown unless a separately verified name-bearing adapter is selected. The production profile records station coverage as explicitly `incomplete` until a measured release replay proves source-consistent coverage; `stationsMinimum: 1` is only a non-empty safety gate, not a completeness claim. The current worktree does not download or promote a production artifact, so deployed station coverage remains unverified until an isolated replay validates the member, its joins, and its measured count.
|
|
|
|
`fixtures/reference/` contains small schema-faithful EVE Ref keyed records for categories, groups, types, and blueprints (including types 34 Tritanium, 35 Pyerite, 587 Rifter, and blueprint 687). The Rifter recipe is synthetic and is labelled as such. Hoboleaks `industryactivities`, `repackagedvolumes`, and `meta` are independent raw maps. `systems.json` is explicitly an ESI-universe fixture: it preserves the official system fields and XYZ coordinate object and is not presented as a Reference Data `/systems` record.
|
|
|
|
The fixture profile is intentionally small. It uses a one-system threshold and required categories 4, 6, and 9. Those thresholds are never silently used for production: production validation requires more than 5,000 systems, at least one station, categories 4/6/9, and at least one blueprint. On the inspected 2026-09-13/2026-09-10 releases, the production candidate contained 48 categories, 1,610 groups, 53,001 types, 5,082 blueprints, 8,490 systems, 1,184 constellations, 114 regions, and 13,978 gates. Twenty-one official blueprint material/product IDs are absent from the current Ref type map; those rows are preserved and explicitly listed in `unresolved_references` rather than dropped or fabricated.
|
|
|
|
## Active artifact and read handles
|
|
|
|
`manifest.json` is the sole publication pointer. It contains `active` and `previous`, source provenance, and the last attempt. Database files are immutable under `databases/`; source caches are immutable, content-addressed/versioned files under `sources/<source>/<fingerprint>-<importer-key>.json`. Downloaded archives are content-addressed under `archives/<source>/<sha256>.archive` and bound by `.production-archive-receipts.json`; offline replay verifies the receipt, byte count, and SHA-256 before parsing. Each active artifact records the exact source cache path and local payload hash in `sourceInputs`, so rollback selects the old inputs rather than a mutable latest cache. Cache format 2 binds canonical hashes and source-check provenance to the active artifact before reuse or fallback. When a same-identity immutable cache is reused, its verified metadata is adopted before publication, while the latest metadata-check observation remains separate. The manifest stores archive HTTP ETags as opaque validators, not SHA-256 values. `localSha256`/`archiveByteLength` describe the downloaded transport artifact; `canonicalSha256` describes the canonical JSON serialization of the source-cache payload. The local archive hash is computed locally and is not published authenticity evidence.
|
|
|
|
```ts
|
|
import { SqliteReferenceRepository } from '$lib/server/reference';
|
|
|
|
const repository = new SqliteReferenceRepository(process.env.REFERENCE_DIR ?? 'data/reference');
|
|
try {
|
|
const [types, version] = await Promise.all([
|
|
repository.types([34, 35, 587, 999999]),
|
|
repository.version(),
|
|
]);
|
|
// types contains the three fixture records; unknown IDs are omitted.
|
|
} finally {
|
|
repository.close();
|
|
}
|
|
```
|
|
|
|
Construction snapshots the manifest and opens that database read-only. It performs only a cheap schema/header check (including the versioned reader schema, FTS5 type-search table, and required normalized-name index); full SQLite integrity and foreign-key scans belong to the compiler and are not repeated on every request. It therefore keeps `types()` and `version()` consistent if another process promotes a build. Type lookups chunk large ID requests into bounded `IN` queries; category/group/location ID selections reject oversized batches visibly. Create one repository (or `repository.pin()`/`snapshot()`) per request and call `close()` in `finally`; do not retain a closed handle. A new constructor sees the next active artifact. A prior schema-version artifact is rejected until a compatible compiler rebuilds it; rollback validates schema/profile/importer/count/provenance compatibility before swapping immutable artifacts.
|
|
|
|
## Promotion lifecycle
|
|
|
|
A build acquires `.reference-build.lock`, checks each source independently, fingerprints it, and loads only changed sources. Unchanged sources come from their last successful cache. Inputs are schema-validated and normalized before a fresh staging SQLite file is created. The candidate gets indexes, foreign-key checks, `PRAGMA integrity_check`, cross-source joins, smoke queries, and profile thresholds. The database is fsynced, renamed into `databases/`, then one fsynced manifest rename publishes `active` and retains `previous`.
|
|
|
|
An invalid candidate never replaces ACTIVE or overwrites a successful source cache. A source-level failure is recorded independently: if that source's last committed cache verifies, another changed source may advance in a newly validated candidate against that fallback; the failed source remains visibly `failed`/`reused` and its active fingerprint stays pinned. Failures are recorded independently in `manifest.sources` and `lastAttempt`; successful-import timestamps are updated only after candidate validation and manifest promotion. A stale Hoboleaks file is visible and rejected by the promotion policy. A failed source does not get hidden by a successful other source.
|
|
|
|
The lock prevents concurrent builds. A crash can leave a lock, staging directory, or a fully validated database whose manifest rename did not happen; after confirming no compiler is running, remove the lock, then the next build removes `.staging-*` directories and may recover an existing artifact only after rechecking integrity, schema, source provenance hashes, and canonical counts. It never overwrites an existing database. Completed immutable database files are safe to retain. Manifest replacement is a same-filesystem rename with directory fsync; an interrupted build therefore leaves the old manifest active.
|
|
|
|
## Rollback and checks
|
|
|
|
```sh
|
|
REFERENCE_DIR=data/reference bun workers/reference.ts --fixture --rollback
|
|
REFERENCE_DIR=data/reference bun workers/reference.ts --check --fixture
|
|
bun workers/reference.ts --help
|
|
```
|
|
|
|
`--check` performs source metadata checks only and never archive fetches. With `--fixture` it checks the local fixture adapters; without `--fixture` it performs independent HEAD requests to the official Reference Data, Hoboleaks, and CCP SDE release URLs and reports metadata fingerprints with the top-level `archiveFetch: false` marker. That metadata fingerprint is not a published archive checksum. A default invocation prints help. Production archive provenance is recorded independently for each source: resolved URL, HTTP validators, transport byte count, locally computed archive SHA-256, source build/revision, and canonical payload/cache SHA-256. Offline mode is fail-closed and never falls back to HTTP. TAR headers, XZ memory limits, two-block TAR end markers, ZIP advertised sizes, and CRC-32 values are validated before promotion. Selected Hoboleaks files must match their published MD5, carry a non-future per-file revision (older revisions are valid when the file is unchanged), and be explicitly non-deprecated. An ETag is never treated as a checksum. Archive payloads are content-addressed in immutable source caches; `manifest.json` remains the sole publication pointer. Hoboleaks stale metadata for the selected imported files rejects promotion, while unrelated stale/deprecated files remain visible in the source archive metadata and do not get silently imported.
|