jb/eden
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(reference): add indexed readers and station source

Browse Source
This commit is contained in:
jb
2026-09-19 09:53:46 -04:00
parent 9ddd12fbc5
commit f7b604f40f
10 changed files with 429 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/reference-operations.md
+3 -3
View File
@@ -21,11 +21,11 @@ bun workers/reference.ts --production
bun workers/reference.ts --production --offline --archive-dir data/reference-live
```
The default production archive and artifact directory is `data/reference-live` (ignored by git); production publishes `data/reference-live/manifest.json`, while fixture builds remain under `data/reference`. An explicit `--directory` wins over `REFERENCE_DIR`, which wins over these defaults. `REFERENCE_LIVE_DIR` controls archive storage only. Compose explicitly publishes to `/app/data/reference`, the shared worker/web volume; the production web reader rejects fixture-profile artifacts. Runtime images must provide the `xz` executable (Debian package `xz-utils`) for streaming `.tar.xz` decompression; ZIP deflate uses the Bun runtime. No npm archive dependency is required. Production validation requires more than 5,000 systems and currently imports CCP's `mapSolarSystems`, `mapConstellations`, `mapRegions`, and `mapStargates` JSONL members.
The default production archive and artifact directory is `data/reference-live` (ignored by git); production publishes `data/reference-live/manifest.json`, while fixture builds remain under `data/reference`. An explicit `--directory` wins over `REFERENCE_DIR`, which wins over these defaults. `REFERENCE_LIVE_DIR` controls archive storage only. Compose explicitly publishes to `/app/data/reference`, the shared worker/web volume; the production web reader rejects fixture-profile artifacts. Runtime images must provide the `xz` executable (Debian package `xz-utils`) for streaming `.tar.xz` decompression; ZIP deflate uses the Bun runtime. No npm archive dependency is required. Production validation requires more than 5,000 systems, at least one station, and imports CCP's `mapSolarSystems`, `mapConstellations`, `mapRegions`, `mapStargates`, and `npcStations` JSONL members. A production candidate missing `npcStations.jsonl` or containing no station rows fails closed rather than claiming station coverage. The importer parses official station ID/system/type/owner/position facts; names remain unknown unless a separately verified name-bearing adapter is selected. The current worktree does not download or promote a production artifact, so deployed station coverage remains unverified until an isolated replay validates the member and its joins.
`fixtures/reference/` contains small schema-faithful EVE Ref keyed records for categories, groups, types, and blueprints (including types 34 Tritanium, 35 Pyerite, 587 Rifter, and blueprint 687). The Rifter recipe is synthetic and is labelled as such. Hoboleaks `industryactivities`, `repackagedvolumes`, and `meta` are independent raw maps. `systems.json` is explicitly an ESI-universe fixture: it preserves the official system fields and XYZ coordinate object and is not presented as a Reference Data `/systems` record.
The fixture profile is intentionally small. It uses a one-system threshold and required categories 4, 6, and 9. Those thresholds are never silently used for production: production validation requires more than 5,000 systems, categories 4/6/9, and at least one blueprint. On the inspected 2026-09-13/2026-09-10 releases, the production candidate contained 48 categories, 1,610 groups, 53,001 types, 5,082 blueprints, 8,490 systems, 1,184 constellations, 114 regions, and 13,978 gates. Twenty-one official blueprint material/product IDs are absent from the current Ref type map; those rows are preserved and explicitly listed in `unresolved_references` rather than dropped or fabricated.
The fixture profile is intentionally small. It uses a one-system threshold and required categories 4, 6, and 9. Those thresholds are never silently used for production: production validation requires more than 5,000 systems, at least one station, categories 4/6/9, and at least one blueprint. On the inspected 2026-09-13/2026-09-10 releases, the production candidate contained 48 categories, 1,610 groups, 53,001 types, 5,082 blueprints, 8,490 systems, 1,184 constellations, 114 regions, and 13,978 gates. Twenty-one official blueprint material/product IDs are absent from the current Ref type map; those rows are preserved and explicitly listed in `unresolved_references` rather than dropped or fabricated.
## Active artifact and read handles
@@ -46,7 +46,7 @@ try {
}
```
Construction snapshots the manifest and opens that database read-only. It performs only a cheap schema/header check; full SQLite integrity and foreign-key scans belong to the compiler and are not repeated on every request. It therefore keeps `types()` and `version()` consistent if another process promotes a build. Type lookups chunk large ID lists into bounded `IN` queries. Create one repository (or `repository.pin()`/`snapshot()`) per request and call `close()` in `finally`; do not retain a closed handle. A new constructor sees the next active artifact.
Construction snapshots the manifest and opens that database read-only. It performs only a cheap schema/header check (including the versioned reader schema, FTS5 type-search table, and required normalized-name index); full SQLite integrity and foreign-key scans belong to the compiler and are not repeated on every request. It therefore keeps `types()` and `version()` consistent if another process promotes a build. Type lookups chunk large ID requests into bounded `IN` queries; category/group/location ID selections reject oversized batches visibly. Create one repository (or `repository.pin()`/`snapshot()`) per request and call `close()` in `finally`; do not retain a closed handle. A new constructor sees the next active artifact. A prior schema-version artifact is rejected until a compatible compiler rebuilds it; rollback validates schema/profile/importer/count/provenance compatibility before swapping immutable artifacts.
## Promotion lifecycle
docs/research/production-import.md
+12 -3
View File
@@ -162,7 +162,15 @@ The list is metadata-driven, not a hard-coded schema promise: if a future `meta.
Hoboleaks metadata also carries per-file `md5`, `revision`, `stale`, and `deprecated`. A file can be retrievable but stale. Record this as source state; do not convert it to a transport-successful fresh input.
## 4. Universe XYZ source choice
## 4. NPC station source and explicit name coverage
The CCP JSON Lines SDE includes `npcStations.jsonl`; CCP's [schema changelog](https://developers.eveonline.com/static-data/tranquility/schema-changelog.yaml) records the station-schema migration and the [Map Data guide](https://developers.eveonline.com/docs/guides/map-data/) establishes the map hierarchy. A bounded inspection of the build-3503375 station member found 5,210 records with `_key`, `solarSystemID`, `ownerID`, `typeID`, and nested `position` (`x`, `y`, `z`); none carried a `name` field. The third-party schema page was used only as a cross-check, not as authority; this worktree did not download or promote the production archive.
The importer therefore treats CCP station IDs, system relationships, owner/type IDs, and coordinates as source facts, while station names remain unknown for the official SDE. The normalized station contract permits an optional `name` only when a separately verified name-bearing adapter supplies it; the production CCP adapter ignores all name-like fields and never derives a name from an ID or type. A station record must reference a system in the same candidate; malformed position/IDs or mismatched supplied system station lists fail closed. The selected production CCP parser requires `npcStations.jsonl` and the production validation profile requires at least one parsed station row; a missing member or empty station set fails closed instead of degrading to fabricated or incomplete station coverage.
This worktree has not downloaded or replayed the production ZIP, so actual release member presence and station row coverage remain an acceptance blocker for deployed NPC station coverage.
## 5. Universe XYZ source choice
### Comparison
@@ -196,7 +204,7 @@ The real line has additional fields. Preserve unknown fields or explicitly ignor
A current ESI list request returned 8,490 IDs (checked 2026-09-18), confirming that a production profile limited to synthetic fixtures or a handful of systems is not sufficient. The CCP SDE map file is the efficient immutable complement: one current archive download rather than thousands of ESI calls.
## 5. Download and streaming plan (after this phase)
## 6. Download and streaming plan (after this phase)
### Preflight
@@ -221,7 +229,7 @@ The ESI-scrape alternative would total 92,227,356 bytes with Reference Data + Ho
1. Create a unique staging directory and `.part` file per source. Stream each HTTP body to disk while hashing and counting; do not place bytes in a JS/Bun `Buffer` for the whole archive.
2. Close and verify the staged file before opening it. Keep the immutable staged artifact until the candidate DB and manifest validate.
3. For Reference Data and Hoboleaks, stream XZ → TAR sequentially. Parse each selected root member as it arrives; close the TAR/XZ reader on any error. These archives are small enough to retain on disk but not to trust as an in-memory blob.
4. For CCP SDE JSONL, keep the completed ZIP staged, inspect the ZIP central directory, and stream only `mapSolarSystems.jsonl` through the JSON Lines parser. Do not extract the whole ~99 MB ZIP merely to build the universe complement. Keep the ZIP hash in provenance even if only one member is consumed.
4. For CCP SDE JSONL, keep the completed ZIP staged, inspect the ZIP central directory, and stream only `mapSolarSystems.jsonl`, `mapConstellations.jsonl`, `mapRegions.jsonl`, `mapStargates.jsonl`, and `npcStations.jsonl` through the JSON Lines parser. Do not extract the whole ~99 MB ZIP merely to build the universe complement. Keep the ZIP hash in provenance even though only selected members are consumed.
5. Parse JSONL line-by-line: reject malformed lines, require integer `_key`, require finite numeric `position.x/y/z`, reject duplicate IDs, and count systems. Preserve the full source line or unknown fields if later features may need them.
6. Build a new immutable SQLite candidate, validate all cross-source references and the real universe count (expected to be >5,000), close it, then atomically publish its manifest. Keep the previous manifest/DB untouched on any source or validation failure.
@@ -253,6 +261,7 @@ A successful Reference Data archive must not mask a failed SDE universe compleme
- [EVE Ref `ScrapeHoboleaks.java`](https://github.com/autonomouslogic/eve-ref/blob/17948f63ad35b3417ad888a72f6d4fcc49e2b72d/src/main/java/com/autonomouslogic/everef/cli/ScrapeHoboleaks.java) — metadata-driven root member creation.
- [Hoboleaks `tq/meta.json`](https://sde.hoboleaks.space/tq/meta.json) — current revision, file list, MD5, stale/deprecated state.
- [CCP Static Data docs](https://developers.eveonline.com/docs/services/static-data/) — JSONL format, automation routes, build metadata, HTTP caching.
- [CCP schema changelog](https://developers.eveonline.com/static-data/tranquility/schema-changelog.yaml) and [Map Data guide](https://developers.eveonline.com/docs/guides/map-data/) — station-schema migration and station/system/region hierarchy evidence.
- [CCP current build metadata](https://developers.eveonline.com/static-data/tranquility/latest.jsonl) and [build 3503375 JSONL release](https://developers.eveonline.com/static-data/tranquility/eve-online-static-data-3503375-jsonl.zip) — current build and release size.
- [CCP Map Data guide](https://developers.eveonline.com/docs/guides/map-data/) — `mapSolarSystems.jsonl`, `_key`, XYZ fields, all-space coverage, and coordinate precision.
- [CCP System Security guide](https://developers.eveonline.com/docs/guides/system-security/) — SDE system source and `securityStatus` field.