feat(reference): add indexed readers and station source

This commit is contained in:
jb
2026-09-19 09:53:46 -04:00
parent 9ddd12fbc5
commit f7b604f40f
10 changed files with 429 additions and 25 deletions
+12 -3
View File
@@ -162,7 +162,15 @@ The list is metadata-driven, not a hard-coded schema promise: if a future `meta.
Hoboleaks metadata also carries per-file `md5`, `revision`, `stale`, and `deprecated`. A file can be retrievable but stale. Record this as source state; do not convert it to a transport-successful fresh input.
## 4. Universe XYZ source choice
## 4. NPC station source and explicit name coverage
The CCP JSON Lines SDE includes `npcStations.jsonl`; CCP's [schema changelog](https://developers.eveonline.com/static-data/tranquility/schema-changelog.yaml) records the station-schema migration and the [Map Data guide](https://developers.eveonline.com/docs/guides/map-data/) establishes the map hierarchy. A bounded inspection of the build-3503375 station member found 5,210 records with `_key`, `solarSystemID`, `ownerID`, `typeID`, and nested `position` (`x`, `y`, `z`); none carried a `name` field. The third-party schema page was used only as a cross-check, not as authority; this worktree did not download or promote the production archive.
The importer therefore treats CCP station IDs, system relationships, owner/type IDs, and coordinates as source facts, while station names remain unknown for the official SDE. The normalized station contract permits an optional `name` only when a separately verified name-bearing adapter supplies it; the production CCP adapter ignores all name-like fields and never derives a name from an ID or type. A station record must reference a system in the same candidate; malformed position/IDs or mismatched supplied system station lists fail closed. The selected production CCP parser requires `npcStations.jsonl` and the production validation profile requires at least one parsed station row; a missing member or empty station set fails closed instead of degrading to fabricated or incomplete station coverage.
This worktree has not downloaded or replayed the production ZIP, so actual release member presence and station row coverage remain an acceptance blocker for deployed NPC station coverage.
## 5. Universe XYZ source choice
### Comparison
@@ -196,7 +204,7 @@ The real line has additional fields. Preserve unknown fields or explicitly ignor
A current ESI list request returned 8,490 IDs (checked 2026-09-18), confirming that a production profile limited to synthetic fixtures or a handful of systems is not sufficient. The CCP SDE map file is the efficient immutable complement: one current archive download rather than thousands of ESI calls.
## 5. Download and streaming plan (after this phase)
## 6. Download and streaming plan (after this phase)
### Preflight
@@ -221,7 +229,7 @@ The ESI-scrape alternative would total 92,227,356 bytes with Reference Data + Ho
1. Create a unique staging directory and `.part` file per source. Stream each HTTP body to disk while hashing and counting; do not place bytes in a JS/Bun `Buffer` for the whole archive.
2. Close and verify the staged file before opening it. Keep the immutable staged artifact until the candidate DB and manifest validate.
3. For Reference Data and Hoboleaks, stream XZ → TAR sequentially. Parse each selected root member as it arrives; close the TAR/XZ reader on any error. These archives are small enough to retain on disk but not to trust as an in-memory blob.
4. For CCP SDE JSONL, keep the completed ZIP staged, inspect the ZIP central directory, and stream only `mapSolarSystems.jsonl` through the JSON Lines parser. Do not extract the whole ~99 MB ZIP merely to build the universe complement. Keep the ZIP hash in provenance even if only one member is consumed.
4. For CCP SDE JSONL, keep the completed ZIP staged, inspect the ZIP central directory, and stream only `mapSolarSystems.jsonl`, `mapConstellations.jsonl`, `mapRegions.jsonl`, `mapStargates.jsonl`, and `npcStations.jsonl` through the JSON Lines parser. Do not extract the whole ~99 MB ZIP merely to build the universe complement. Keep the ZIP hash in provenance even though only selected members are consumed.
5. Parse JSONL line-by-line: reject malformed lines, require integer `_key`, require finite numeric `position.x/y/z`, reject duplicate IDs, and count systems. Preserve the full source line or unknown fields if later features may need them.
6. Build a new immutable SQLite candidate, validate all cross-source references and the real universe count (expected to be >5,000), close it, then atomically publish its manifest. Keep the previous manifest/DB untouched on any source or validation failure.
@@ -253,6 +261,7 @@ A successful Reference Data archive must not mask a failed SDE universe compleme
- [EVE Ref `ScrapeHoboleaks.java`](https://github.com/autonomouslogic/eve-ref/blob/17948f63ad35b3417ad888a72f6d4fcc49e2b72d/src/main/java/com/autonomouslogic/everef/cli/ScrapeHoboleaks.java) — metadata-driven root member creation.
- [Hoboleaks `tq/meta.json`](https://sde.hoboleaks.space/tq/meta.json) — current revision, file list, MD5, stale/deprecated state.
- [CCP Static Data docs](https://developers.eveonline.com/docs/services/static-data/) — JSONL format, automation routes, build metadata, HTTP caching.
- [CCP schema changelog](https://developers.eveonline.com/static-data/tranquility/schema-changelog.yaml) and [Map Data guide](https://developers.eveonline.com/docs/guides/map-data/) — station-schema migration and station/system/region hierarchy evidence.
- [CCP current build metadata](https://developers.eveonline.com/static-data/tranquility/latest.jsonl) and [build 3503375 JSONL release](https://developers.eveonline.com/static-data/tranquility/eve-online-static-data-3503375-jsonl.zip) — current build and release size.
- [CCP Map Data guide](https://developers.eveonline.com/docs/guides/map-data/) — `mapSolarSystems.jsonl`, `_key`, XYZ fields, all-space coverage, and coordinate precision.
- [CCP System Security guide](https://developers.eveonline.com/docs/guides/system-security/) — SDE system source and `securityStatus` field.