8.5 KiB
Reference data operations
The reference compiler builds immutable SQLite artifacts from checked-in fixtures or explicitly supplied source adapters. It does not download an archive by default.
Fixture build
REFERENCE_DIR=data/reference bun workers/reference.ts --fixture
# or
bun run reference:fixture
Production import
Production is explicit and uses the pinned public EVE Ref Reference Data, Hoboleaks, and CCP JSONL SDE releases. --check performs only independent HTTP metadata checks (no archive bytes); --production streams each archive to a bounded staging file, hashes/counts transport bytes, validates archive members, and publishes only after cross-source SQLite validation.
bun workers/reference.ts --check --production
bun workers/reference.ts --production
# Re-run an already downloaded, offline import:
bun workers/reference.ts --production --offline --archive-dir data/reference-live
The default production archive and artifact directory is data/reference-live (ignored by git); production publishes data/reference-live/manifest.json, while fixture builds remain under data/reference. An explicit --directory wins over REFERENCE_DIR, which wins over these defaults. REFERENCE_LIVE_DIR controls archive storage only. Compose explicitly publishes to /app/data/reference, the shared worker/web volume; the production web reader rejects fixture-profile artifacts. Runtime images must provide the xz executable (Debian package xz-utils) for streaming .tar.xz decompression; ZIP deflate uses the Bun runtime. No npm archive dependency is required. Production validation requires more than 5,000 systems and currently imports CCP's mapSolarSystems, mapConstellations, mapRegions, and mapStargates JSONL members.
fixtures/reference/ contains small schema-faithful EVE Ref keyed records for categories, groups, types, and blueprints (including types 34 Tritanium, 35 Pyerite, 587 Rifter, and blueprint 687). The Rifter recipe is synthetic and is labelled as such. Hoboleaks industryactivities, repackagedvolumes, and meta are independent raw maps. systems.json is explicitly an ESI-universe fixture: it preserves the official system fields and XYZ coordinate object and is not presented as a Reference Data /systems record.
The fixture profile is intentionally small. It uses a one-system threshold and required categories 4, 6, and 9. Those thresholds are never silently used for production: production validation requires more than 5,000 systems, categories 4/6/9, and at least one blueprint. On the inspected 2026-09-13/2026-09-10 releases, the production candidate contained 48 categories, 1,610 groups, 53,001 types, 5,082 blueprints, 8,490 systems, 1,184 constellations, 114 regions, and 13,978 gates. Twenty-one official blueprint material/product IDs are absent from the current Ref type map; those rows are preserved and explicitly listed in unresolved_references rather than dropped or fabricated.
Active artifact and read handles
manifest.json is the sole publication pointer. It contains active and previous, source provenance, and the last attempt. Database files are immutable under databases/; source caches are immutable, content-addressed/versioned files under sources/<source>/<fingerprint>-<importer-key>.json. Downloaded archives are content-addressed under archives/<source>/<sha256>.archive and bound by .production-archive-receipts.json; offline replay verifies the receipt, byte count, and SHA-256 before parsing. Each active artifact records the exact source cache path and local payload hash in sourceInputs, so rollback selects the old inputs rather than a mutable latest cache. Cache format 2 binds canonical hashes and source-check provenance to the active artifact before reuse or fallback. When a same-identity immutable cache is reused, its verified metadata is adopted before publication, while the latest metadata-check observation remains separate. The manifest stores archive HTTP ETags as opaque validators, not SHA-256 values. localSha256/archiveByteLength describe the downloaded transport artifact; canonicalSha256 describes the canonical JSON serialization of the source-cache payload. The local archive hash is computed locally and is not published authenticity evidence.
import { SqliteReferenceRepository } from '$lib/server/reference';
const repository = new SqliteReferenceRepository(process.env.REFERENCE_DIR ?? 'data/reference');
try {
const [types, version] = await Promise.all([
repository.types([34, 35, 587, 999999]),
repository.version(),
]);
// types contains the three fixture records; unknown IDs are omitted.
} finally {
repository.close();
}
Construction snapshots the manifest and opens that database read-only. It performs only a cheap schema/header check; full SQLite integrity and foreign-key scans belong to the compiler and are not repeated on every request. It therefore keeps types() and version() consistent if another process promotes a build. Type lookups chunk large ID lists into bounded IN queries. Create one repository (or repository.pin()/snapshot()) per request and call close() in finally; do not retain a closed handle. A new constructor sees the next active artifact.
Promotion lifecycle
A build acquires .reference-build.lock, checks each source independently, fingerprints it, and loads only changed sources. Unchanged sources come from their last successful cache. Inputs are schema-validated and normalized before a fresh staging SQLite file is created. The candidate gets indexes, foreign-key checks, PRAGMA integrity_check, cross-source joins, smoke queries, and profile thresholds. The database is fsynced, renamed into databases/, then one fsynced manifest rename publishes active and retains previous.
An invalid candidate never replaces ACTIVE or overwrites a successful source cache. A source-level failure is recorded independently: if that source's last committed cache verifies, another changed source may advance in a newly validated candidate against that fallback; the failed source remains visibly failed/reused and its active fingerprint stays pinned. Failures are recorded independently in manifest.sources and lastAttempt; successful-import timestamps are updated only after candidate validation and manifest promotion. A stale Hoboleaks file is visible and rejected by the promotion policy. A failed source does not get hidden by a successful other source.
The lock prevents concurrent builds. A crash can leave a lock, staging directory, or a fully validated database whose manifest rename did not happen; after confirming no compiler is running, remove the lock, then the next build removes .staging-* directories and may recover an existing artifact only after rechecking integrity, schema, source provenance hashes, and canonical counts. It never overwrites an existing database. Completed immutable database files are safe to retain. Manifest replacement is a same-filesystem rename with directory fsync; an interrupted build therefore leaves the old manifest active.
Rollback and checks
REFERENCE_DIR=data/reference bun workers/reference.ts --rollback
REFERENCE_DIR=data/reference bun workers/reference.ts --check --fixture
bun workers/reference.ts --help
--check performs source metadata checks only and never archive fetches. With --fixture it checks the local fixture adapters; without --fixture it performs independent HEAD requests to the official Reference Data, Hoboleaks, and CCP SDE release URLs and reports metadata fingerprints with the top-level archiveFetch: false marker. That metadata fingerprint is not a published archive checksum. A default invocation prints help. Production archive provenance is recorded independently for each source: resolved URL, HTTP validators, transport byte count, locally computed archive SHA-256, source build/revision, and canonical payload/cache SHA-256. Offline mode is fail-closed and never falls back to HTTP. TAR headers, XZ memory limits, two-block TAR end markers, ZIP advertised sizes, and CRC-32 values are validated before promotion. Selected Hoboleaks files must match their published MD5, carry a non-future per-file revision (older revisions are valid when the file is unchanged), and be explicitly non-deprecated. An ETag is never treated as a checksum. Archive payloads are content-addressed in immutable source caches; manifest.json remains the sole publication pointer. Hoboleaks stale metadata for the selected imported files rejects promotion, while unrelated stale/deprecated files remain visible in the source archive metadata and do not get silently imported.