feat(reference): add indexed readers and station source

This commit is contained in:
jb
2026-09-19 09:53:46 -04:00
parent 9ddd12fbc5
commit f7b604f40f
10 changed files with 429 additions and 25 deletions
+3 -3
View File
@@ -21,11 +21,11 @@ bun workers/reference.ts --production
bun workers/reference.ts --production --offline --archive-dir data/reference-live
```
The default production archive and artifact directory is `data/reference-live` (ignored by git); production publishes `data/reference-live/manifest.json`, while fixture builds remain under `data/reference`. An explicit `--directory` wins over `REFERENCE_DIR`, which wins over these defaults. `REFERENCE_LIVE_DIR` controls archive storage only. Compose explicitly publishes to `/app/data/reference`, the shared worker/web volume; the production web reader rejects fixture-profile artifacts. Runtime images must provide the `xz` executable (Debian package `xz-utils`) for streaming `.tar.xz` decompression; ZIP deflate uses the Bun runtime. No npm archive dependency is required. Production validation requires more than 5,000 systems and currently imports CCP's `mapSolarSystems`, `mapConstellations`, `mapRegions`, and `mapStargates` JSONL members.
The default production archive and artifact directory is `data/reference-live` (ignored by git); production publishes `data/reference-live/manifest.json`, while fixture builds remain under `data/reference`. An explicit `--directory` wins over `REFERENCE_DIR`, which wins over these defaults. `REFERENCE_LIVE_DIR` controls archive storage only. Compose explicitly publishes to `/app/data/reference`, the shared worker/web volume; the production web reader rejects fixture-profile artifacts. Runtime images must provide the `xz` executable (Debian package `xz-utils`) for streaming `.tar.xz` decompression; ZIP deflate uses the Bun runtime. No npm archive dependency is required. Production validation requires more than 5,000 systems, at least one station, and imports CCP's `mapSolarSystems`, `mapConstellations`, `mapRegions`, `mapStargates`, and `npcStations` JSONL members. A production candidate missing `npcStations.jsonl` or containing no station rows fails closed rather than claiming station coverage. The importer parses official station ID/system/type/owner/position facts; names remain unknown unless a separately verified name-bearing adapter is selected. The current worktree does not download or promote a production artifact, so deployed station coverage remains unverified until an isolated replay validates the member and its joins.
`fixtures/reference/` contains small schema-faithful EVE Ref keyed records for categories, groups, types, and blueprints (including types 34 Tritanium, 35 Pyerite, 587 Rifter, and blueprint 687). The Rifter recipe is synthetic and is labelled as such. Hoboleaks `industryactivities`, `repackagedvolumes`, and `meta` are independent raw maps. `systems.json` is explicitly an ESI-universe fixture: it preserves the official system fields and XYZ coordinate object and is not presented as a Reference Data `/systems` record.
The fixture profile is intentionally small. It uses a one-system threshold and required categories 4, 6, and 9. Those thresholds are never silently used for production: production validation requires more than 5,000 systems, categories 4/6/9, and at least one blueprint. On the inspected 2026-09-13/2026-09-10 releases, the production candidate contained 48 categories, 1,610 groups, 53,001 types, 5,082 blueprints, 8,490 systems, 1,184 constellations, 114 regions, and 13,978 gates. Twenty-one official blueprint material/product IDs are absent from the current Ref type map; those rows are preserved and explicitly listed in `unresolved_references` rather than dropped or fabricated.
The fixture profile is intentionally small. It uses a one-system threshold and required categories 4, 6, and 9. Those thresholds are never silently used for production: production validation requires more than 5,000 systems, at least one station, categories 4/6/9, and at least one blueprint. On the inspected 2026-09-13/2026-09-10 releases, the production candidate contained 48 categories, 1,610 groups, 53,001 types, 5,082 blueprints, 8,490 systems, 1,184 constellations, 114 regions, and 13,978 gates. Twenty-one official blueprint material/product IDs are absent from the current Ref type map; those rows are preserved and explicitly listed in `unresolved_references` rather than dropped or fabricated.
## Active artifact and read handles
@@ -46,7 +46,7 @@ try {
}
```
Construction snapshots the manifest and opens that database read-only. It performs only a cheap schema/header check; full SQLite integrity and foreign-key scans belong to the compiler and are not repeated on every request. It therefore keeps `types()` and `version()` consistent if another process promotes a build. Type lookups chunk large ID lists into bounded `IN` queries. Create one repository (or `repository.pin()`/`snapshot()`) per request and call `close()` in `finally`; do not retain a closed handle. A new constructor sees the next active artifact.
Construction snapshots the manifest and opens that database read-only. It performs only a cheap schema/header check (including the versioned reader schema, FTS5 type-search table, and required normalized-name index); full SQLite integrity and foreign-key scans belong to the compiler and are not repeated on every request. It therefore keeps `types()` and `version()` consistent if another process promotes a build. Type lookups chunk large ID requests into bounded `IN` queries; category/group/location ID selections reject oversized batches visibly. Create one repository (or `repository.pin()`/`snapshot()`) per request and call `close()` in `finally`; do not retain a closed handle. A new constructor sees the next active artifact. A prior schema-version artifact is rejected until a compatible compiler rebuilds it; rollback validates schema/profile/importer/count/provenance compatibility before swapping immutable artifacts.
## Promotion lifecycle