feat(reference): add indexed readers and station source
This commit is contained in:
@@ -162,7 +162,15 @@ The list is metadata-driven, not a hard-coded schema promise: if a future `meta.
|
||||
|
||||
Hoboleaks metadata also carries per-file `md5`, `revision`, `stale`, and `deprecated`. A file can be retrievable but stale. Record this as source state; do not convert it to a transport-successful fresh input.
|
||||
|
||||
## 4. Universe XYZ source choice
|
||||
## 4. NPC station source and explicit name coverage
|
||||
|
||||
The CCP JSON Lines SDE includes `npcStations.jsonl`; CCP's [schema changelog](https://developers.eveonline.com/static-data/tranquility/schema-changelog.yaml) records the station-schema migration and the [Map Data guide](https://developers.eveonline.com/docs/guides/map-data/) establishes the map hierarchy. A bounded inspection of the build-3503375 station member found 5,210 records with `_key`, `solarSystemID`, `ownerID`, `typeID`, and nested `position` (`x`, `y`, `z`); none carried a `name` field. The third-party schema page was used only as a cross-check, not as authority; this worktree did not download or promote the production archive.
|
||||
|
||||
The importer therefore treats CCP station IDs, system relationships, owner/type IDs, and coordinates as source facts, while station names remain unknown for the official SDE. The normalized station contract permits an optional `name` only when a separately verified name-bearing adapter supplies it; the production CCP adapter ignores all name-like fields and never derives a name from an ID or type. A station record must reference a system in the same candidate; malformed position/IDs or mismatched supplied system station lists fail closed. The selected production CCP parser requires `npcStations.jsonl` and the production validation profile requires at least one parsed station row; a missing member or empty station set fails closed instead of degrading to fabricated or incomplete station coverage.
|
||||
|
||||
This worktree has not downloaded or replayed the production ZIP, so actual release member presence and station row coverage remain an acceptance blocker for deployed NPC station coverage.
|
||||
|
||||
## 5. Universe XYZ source choice
|
||||
|
||||
### Comparison
|
||||
|
||||
@@ -196,7 +204,7 @@ The real line has additional fields. Preserve unknown fields or explicitly ignor
|
||||
|
||||
A current ESI list request returned 8,490 IDs (checked 2026-09-18), confirming that a production profile limited to synthetic fixtures or a handful of systems is not sufficient. The CCP SDE map file is the efficient immutable complement: one current archive download rather than thousands of ESI calls.
|
||||
|
||||
## 5. Download and streaming plan (after this phase)
|
||||
## 6. Download and streaming plan (after this phase)
|
||||
|
||||
### Preflight
|
||||
|
||||
@@ -221,7 +229,7 @@ The ESI-scrape alternative would total 92,227,356 bytes with Reference Data + Ho
|
||||
1. Create a unique staging directory and `.part` file per source. Stream each HTTP body to disk while hashing and counting; do not place bytes in a JS/Bun `Buffer` for the whole archive.
|
||||
2. Close and verify the staged file before opening it. Keep the immutable staged artifact until the candidate DB and manifest validate.
|
||||
3. For Reference Data and Hoboleaks, stream XZ → TAR sequentially. Parse each selected root member as it arrives; close the TAR/XZ reader on any error. These archives are small enough to retain on disk but not to trust as an in-memory blob.
|
||||
4. For CCP SDE JSONL, keep the completed ZIP staged, inspect the ZIP central directory, and stream only `mapSolarSystems.jsonl` through the JSON Lines parser. Do not extract the whole ~99 MB ZIP merely to build the universe complement. Keep the ZIP hash in provenance even if only one member is consumed.
|
||||
4. For CCP SDE JSONL, keep the completed ZIP staged, inspect the ZIP central directory, and stream only `mapSolarSystems.jsonl`, `mapConstellations.jsonl`, `mapRegions.jsonl`, `mapStargates.jsonl`, and `npcStations.jsonl` through the JSON Lines parser. Do not extract the whole ~99 MB ZIP merely to build the universe complement. Keep the ZIP hash in provenance even though only selected members are consumed.
|
||||
5. Parse JSONL line-by-line: reject malformed lines, require integer `_key`, require finite numeric `position.x/y/z`, reject duplicate IDs, and count systems. Preserve the full source line or unknown fields if later features may need them.
|
||||
6. Build a new immutable SQLite candidate, validate all cross-source references and the real universe count (expected to be >5,000), close it, then atomically publish its manifest. Keep the previous manifest/DB untouched on any source or validation failure.
|
||||
|
||||
@@ -253,6 +261,7 @@ A successful Reference Data archive must not mask a failed SDE universe compleme
|
||||
- [EVE Ref `ScrapeHoboleaks.java`](https://github.com/autonomouslogic/eve-ref/blob/17948f63ad35b3417ad888a72f6d4fcc49e2b72d/src/main/java/com/autonomouslogic/everef/cli/ScrapeHoboleaks.java) — metadata-driven root member creation.
|
||||
- [Hoboleaks `tq/meta.json`](https://sde.hoboleaks.space/tq/meta.json) — current revision, file list, MD5, stale/deprecated state.
|
||||
- [CCP Static Data docs](https://developers.eveonline.com/docs/services/static-data/) — JSONL format, automation routes, build metadata, HTTP caching.
|
||||
- [CCP schema changelog](https://developers.eveonline.com/static-data/tranquility/schema-changelog.yaml) and [Map Data guide](https://developers.eveonline.com/docs/guides/map-data/) — station-schema migration and station/system/region hierarchy evidence.
|
||||
- [CCP current build metadata](https://developers.eveonline.com/static-data/tranquility/latest.jsonl) and [build 3503375 JSONL release](https://developers.eveonline.com/static-data/tranquility/eve-online-static-data-3503375-jsonl.zip) — current build and release size.
|
||||
- [CCP Map Data guide](https://developers.eveonline.com/docs/guides/map-data/) — `mapSolarSystems.jsonl`, `_key`, XYZ fields, all-space coverage, and coordinate precision.
|
||||
- [CCP System Security guide](https://developers.eveonline.com/docs/guides/system-security/) — SDE system source and `securityStatus` field.
|
||||
|
||||
Reference in New Issue
Block a user