jb/eden
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
eden/docs/research/production-import.md
T

24 KiB
Raw Permalink Blame History

Production reference-data import plan

Checked: 2026-09-18
Scope: full real-data import is now authorized, but this phase performed no archive download. Only release indexes, HTTP HEAD metadata, official documentation/source, and small metadata/API files were inspected. Credentials and Infisical were not touched.

Decision summary

  1. Download the current Reference Data and Hoboleaks archives, then parse their exporter-defined root members. Their current compressed sizes are 14,512,812 bytes and 675,868 bytes.
  2. Use CCP's current JSON Lines SDE for the universe complement, specifically mapSolarSystems.jsonl. CCP's current build is 3503375 (released 2026-09-10), and the official map guide directly documents reading _key and position.x/y/z from that member.
  3. Do not use the EVE Ref ESI-scrape archive as the production universe source: its latest published artifact is dated 2025-11-26, while the CCP SDE is current to 2026-09-10. The ESI scrape is still useful as a shape/reference comparison, not as the freshness authority.
  4. The selected compressed download budget is 114,291,615 bytes (~108.997 MiB) before temporary extraction/database space: Reference Data + Hoboleaks + CCP JSONL SDE.

1. Current release metadata (preflight; no archive bytes read)

The values below came from the public JSON indexes and final archive HEAD responses. ETags are opaque HTTP validators, not published SHA-256 checksums.

Input Immutable/current URL Bytes Approx. compressed size Last-Modified ETag / version
EVE Ref Reference Data reference-data-latest.tar.xz (dated route: reference-data-2026-09-13.tar.xz) 14,512,812 13.84 MiB 2026-09-13 11:16:26Z a4b2dd0325ffac28158f2b93500a0d52
EVE Ref Hoboleaks mirror hoboleaks-sde-latest.tar.xz (dated route: hoboleaks-sde-2026-09-13.tar.xz) 675,868 660.03 KiB 2026-09-13 11:05:28Z 5681550f6fe5f2927b055dfe33883655
EVE Ref ESI scrape (not recommended for production systems) eve-ref-esi-scrape-latest.tar.xz 77,038,676 73.47 MiB 2025-11-26 15:42:24Z 08fe439c74509e521620f588a13cfb37-10
CCP SDE JSONL, build 3503375 eve-online-static-data-3503375-jsonl.zip 99,102,935 94.51 MiB 2026-09-10 11:21:36Z fce5dcb01dbe357402ace2848297f556-12

The CCP automation endpoint currently returns:

{"_key":"sde","buildNumber":3503375,"releaseDate":"2026-09-10T11:09:04Z"}

The EVE Ref archive indexes are reference-data/index.json, hoboleaks-sde/index.json, and esi-scrape/index.json. CCP's official automation documentation says the latest alias redirects to the build-numbered URL and that all resources support ETag/Last-Modified. Record both the alias and the resolved immutable URL in the import manifest.

EVE Ref's running /meta currently reports build time 2026-09-13T11:10:03Z and source hashes:

{
  "build_time": "2026-09-13T11:10:03Z",
  "esi": {"sha256": "645b2902174fe52561b946a5c6e39e257358803956238f0750798e1cde5e4077"},
  "hoboleaks": {"sha256": "477799f37c9275c3ad408117cce8e7913688be41b17fe404cd48e3b501be55cd"},
  "sde": {"sha256": "56f5f7eba4b401aeb2fc1090d2db5ab9dd55cc08a396871e764df45017fa70af"}
}

These are hashes of EVE Ref's three input archives, not checksums of reference-data-latest.tar.xz. Current EVE Ref source downloads the YAML SDE input (SDE_V2_YAML) when building Reference Data, so /meta.sde.sha256 must not be presented as the hash of the separately selected CCP JSONL SDE complement.

2. Reference Data archive: exporter-defined member layout

The current EVE Ref exporter source is the strongest available archive contract:

  • BuildRefData.java at commit 17948f63 writes meta.json, then one .json member for every entry in refdata.yaml, into a TAR and XZ-compresses it.
  • DataUtil.java shows that the builder's SDE input is the configured SDE_V2_YAML ZIP, while the separate production universe complement below deliberately uses CCP's current JSONL ZIP.
  • refdata.yaml at commit ac663020 defines the member basenames, source filenames, ID fields, and output-store relationships.
  • RefDataUtil.java confirms that meta.json is a standalone object and every other JSON member is parsed as an object whose property names are decimal IDs. Its config loader defaults absent outputFile to lower-snake-case (marketGroups → market_groups) and absent outputStore to the config ID (source).

Exact member names

The exporter emits these 19 root members in config order:

meta.json
categories.json
groups.json
market_groups.json
types.json
type_materials.json
type_bonus.json
masteries.json
type_dogma.json
dogma_attributes.json
dogma_effects.json
meta_groups.json
mutaplasmids.json
skills.json
units.json
blueprints.json
icons.json
regions.json
schematics.json

meta.json is a RefDataMeta object containing build_time and sha256 objects for sde, esi, and hoboleaks. Every other member is written as:

{
  "<decimal-id>": {"<normalized record fields>": "..."}
}

The writer serializes the map key as a JSON string and the value as the normalized EVE Ref record. It does not write one file per ID inside the archive. A loader should preserve the string key as the canonical ID while validating that it parses as a non-negative 64-bit integer.

Member/source map

Member Config ID / output store Input members named by the exporter Import role
categories.json categories / categories SDE categories.yaml; ESI universe/categories Canonical category map.
groups.json groups / groups SDE groups.yaml; ESI universe/groups Canonical group map.
market_groups.json marketGroups / marketGroups SDE marketGroups.yaml; ESI markets/groups Canonical market-group map.
types.json types / types SDE types.yaml; ESI universe/types Canonical inventory-type map.
type_materials.json typeMaterials / types SDE typeMaterials.yaml; Hoboleaks typematerials.json Supplemental type merge input; values belong in types.
type_bonus.json typeBonus / types SDE typeBonus.yaml Supplemental type merge input; values belong in types.
masteries.json masteries / types SDE masteries.yaml Supplemental type merge input; values belong in types.
type_dogma.json typeDogma / types SDE typeDogma.yaml Supplemental type merge input; values belong in types.
dogma_attributes.json dogmaAttributes / dogmaAttributes SDE dogmaAttributes.yaml; ESI dogma/attributes Canonical dogma-attribute map.
dogma_effects.json dogmaEffects / dogmaEffects SDE dogmaEffects.yaml; ESI dogma/effects Canonical dogma-effect map.
meta_groups.json metaGroups / metaGroups SDE metaGroups.yaml Canonical meta-group map.
mutaplasmids.json mutaplasmids / mutaplasmids Hoboleaks dynamicitemattributes.json Canonical mutaplasmid map.
skills.json skills / skills Derived by EVE Ref decorators Canonical skill map.
units.json units / units Hoboleaks dogmaunits.json Canonical unit map.
blueprints.json blueprints / blueprints SDE blueprints.yaml; Hoboleaks blueprints.json Canonical blueprint map.
icons.json icons / icons SDE icons.yaml Canonical icon map.
regions.json regions / regions SDE mapRegions.yaml; ESI universe/regions Canonical region map.
schematics.json schematics / schematics SDE planetSchematics.yaml; ESI universe/schematics Canonical schematic map.

The four type supplemental configs have dedicatedOutput: false and outputStore: types in the current config. The exporter still iterates every config and names a member from config.outputFile; therefore, treat types.json as the canonical merged type table, and treat those four sidecar members as exporter outputs whose content must be checked after download rather than assumed to be separate canonical tables. Do not invent records from a sidecar if the authoritative merged types.json does not contain them.

Reference Data parser invariants

  • Read TAR entries sequentially through XZ decompression; there is no need to materialize an uncompressed archive.
  • Require meta.json exactly once.
  • For each other .json member, require a top-level JSON object; every property key must be a decimal ID; reject duplicate IDs and malformed values.
  • Skip no member silently except an explicitly documented non-JSON compatibility member. The current EVE Ref parser skips non-JSON entries but the exporter contract above emits JSON only.
  • Preserve source/build metadata separately from the local archive URL, byte count, ETag, Last-Modified, and locally computed SHA-256.

3. Hoboleaks archive: exact root member layout

The current archive builder is ScrapeHoboleaks.java at commit 17948f63: it downloads Hoboleaks meta.json, writes that as root meta.json, iterates meta.files, downloads each named file from the Hoboleaks root, and writes that file at the archive root with the same filename. There are no nested tq/ directories inside the EVE Ref mirror archive.

The current tq/meta.json has revision 3503375, timestamp 2026-09-13T10:41:34, and 28 file entries. Thus the expected current archive has 29 root members:

meta.json
accountingentrytypes.json
agenttypes.json
attributeorders.json
clonestates.json
compressibletypes.json
dbuffs.json
dogmaeffectcategories.json
dogmaunits.json
dynamicitemattributes.json
expertsystems.json
graphicmaterialsets.json
industryactivities.json
industryassemblylines.json
industryinstallationtypes.json
industrymodifiersources.json
industrytargetfilters.json
localization_dgmattributes.json
localization_languages.json
repackagedvolumes.json
schoolmap.json
schools.json
skillplans.json
skinmaterialnames.json
skinmaterials.json
skins.json
stationstandingsrestrictions.json
typematerials.json
blueprints.json

The list is metadata-driven, not a hard-coded schema promise: if a future meta.json adds/removes a file, the archive member set changes with it. The source files retain Hoboleaks casing and shape (for example, activityID/activityName, not normalized activity_id/activity_name). The prior reference contract records small current examples for industryactivities.json, agenttypes.json, and repackagedvolumes.json.

Hoboleaks metadata also carries per-file md5, revision, stale, and deprecated. A file can be retrievable but stale. Record this as source state; do not convert it to a transport-successful fresh input.

4. NPC station source and explicit name coverage

The CCP JSON Lines SDE includes npcStations.jsonl; CCP's schema changelog records the station-schema migration and the Map Data guide establishes the map hierarchy. A bounded inspection of the build-3503375 station member found 5,210 records with _key, solarSystemID, ownerID, typeID, and nested position (x, y, z); none carried a name field. The third-party schema page was used only as a cross-check, not as authority; this worktree did not download or promote the production archive.

The importer therefore treats CCP station IDs, system relationships, owner/type IDs, and coordinates as source facts, while station names remain unknown for the official SDE. The normalized station contract permits an optional name only when a separately verified name-bearing adapter supplies it; the production CCP adapter ignores all name-like fields and never derives a name from an ID or type. A station record must reference a system in the same candidate; malformed position/IDs or mismatched supplied system station lists fail closed. The selected production CCP parser requires npcStations.jsonl and the production validation profile requires at least one verified station row; a missing member or empty station set fails closed instead of degrading to fabricated coverage. This is explicitly an incomplete station-coverage profile until an isolated replay measures a source-consistent release count; it is not a universal completeness threshold.

This worktree has not downloaded or replayed the production ZIP, so actual release member presence and station row coverage remain an acceptance blocker for deployed NPC station coverage.

5. Universe XYZ source choice

Comparison

Source Current release Member/path shape Freshness and scale Decision
EVE Ref ESI scrape eve-ref-esi-scrape-latest.tar.xz, 77,038,676 bytes, last modified 2025-11-26 EVE Ref's source table names the file universe/systems.yaml; ESI universe records use system_id, name, position: {x,y,z}, and related fields One complete scrape archive, but almost ten months behind the current SDE and not the freshness authority Do not select for production universe data.
CCP SDE JSONL build 3503375, 99,102,935 bytes, released 2026-09-10 Root member mapSolarSystems.jsonl; each line has _key (solar-system ID) and object fields such as position.x/y/z Current official SDE; JSONL is explicitly documented as streamable and the map guide covers all space types Select this source.
ESI live per-system routes Current API; current ID list observed at 8,490 systems /universe/systems/{system_id} response; one request per system Fresh but ~8,490 API requests, rate/cache policy, and no single immutable snapshot Use only for spot checks, not bulk import.

The official CCP Map Data guide says map data is in SDE files prefixed map, explicitly names mapSolarSystems, and its executable example opens ./mapSolarSystems.jsonl, parses each line, reads system["_key"], and reads system["position"]["x"], ["y"], and ["z"]. It also says SDE/ESI include known, wormhole, and abyssal space. The current schema-changelog.yaml records mapSolarSystems as a current file and records position2D as added after build 3096543.

The official System Security guide independently identifies mapSolarSystems.jsonl as the SDE source and securityStatus as a full-precision floating-point field. The official Static Data docs define JSONL handling: an integer-keyed object record is represented as one line with _key; object values remain object fields (the map guide's system["position"] access is the concrete example).

Minimal exact SDE system line contract

Do not claim an entire system schema from the small guide example. The safe import contract is the fields directly documented by the official examples:

{
  "_key": 30000142,
  "position": {
    "x": -129064861735000000.0,
    "y": 60755306910000000.0,
    "z": 117469227060000000.0
  },
  "securityStatus": 0.9459131360054016
}

The real line has additional fields. Preserve unknown fields or explicitly ignore them only after validating the fields needed by the application. Use _key as system_id; do not expect the ESI system_id field in this SDE member. Keep XYZ as 64-bit floating-point values; the official guide warns that 32-bit precision is insufficient for interstellar and planetary coordinates.

A current ESI list request returned 8,490 IDs (checked 2026-09-18), confirming that a production profile limited to synthetic fixtures or a handful of systems is not sufficient. The CCP SDE map file is the efficient immutable complement: one current archive download rather than thousands of ESI calls.

6. Download and streaming plan (after this phase)

Preflight

  1. Fetch only the small JSON indexes, CCP latest.jsonl, Hoboleaks tq/meta.json, and HEAD metadata as done here.
  2. Resolve each latest alias and record the final URL, status, content length, ETag, Last-Modified, media type, source revision/build, and retrieval time.
  3. Require the expected sizes above for this checked release, but treat a changed release as a new candidate rather than a fatal global condition. Refresh metadata and use the new immutable URL.
  4. Never use an ETag as a SHA-256. On the actual download, stream bytes through a local SHA-256 digest and compare byte count with Content-Length.

Selected transfers

Transfer Compressed bytes
Reference Data 14,512,812
Hoboleaks SDE 675,868
CCP SDE JSONL 99,102,935
Total 114,291,615 (~108.997 MiB)

The ESI-scrape alternative would total 92,227,356 bytes with Reference Data + Hoboleaks, but it is stale and is not selected. Do not download both universe complements unless a deliberate cross-check is needed.

Safe extraction order

  1. Create a unique staging directory and .part file per source. Stream each HTTP body to disk while hashing and counting; do not place bytes in a JS/Bun Buffer for the whole archive.
  2. Close and verify the staged file before opening it. Keep the immutable staged artifact until the candidate DB and manifest validate.
  3. For Reference Data and Hoboleaks, stream XZ → TAR sequentially. Parse each selected root member as it arrives; close the TAR/XZ reader on any error. These archives are small enough to retain on disk but not to trust as an in-memory blob.
  4. For CCP SDE JSONL, keep the completed ZIP staged, inspect the ZIP central directory, and stream only mapSolarSystems.jsonl, mapConstellations.jsonl, mapRegions.jsonl, mapStargates.jsonl, and npcStations.jsonl through the JSON Lines parser. Do not extract the whole ~99 MB ZIP merely to build the universe complement. Keep the ZIP hash in provenance even though only selected members are consumed.
  5. Parse JSONL line-by-line: reject malformed lines, require integer _key, require finite numeric position.x/y/z, reject duplicate IDs, and count systems. Preserve the full source line or unknown fields if later features may need them.
  6. Build a new immutable SQLite candidate, validate all cross-source references and the real universe count (expected to be >5,000), close it, then atomically publish its manifest. Keep the previous manifest/DB untouched on any source or validation failure.

Independent source status

Track three source outcomes independently:

  • reference_data: HTTP/download/hash/archive/member validation;
  • hoboleaks: HTTP/download/hash/archive/member validation plus per-file stale/deprecated state;
  • ccp_sde_jsonl: HTTP/download/hash/ZIP member/JSONL validation and build number.

A successful Reference Data archive must not mask a failed SDE universe complement. A stale Hoboleaks file is a distinct policy state from a failed download. Promotion should require the source set selected by the production profile to be successful and policy-acceptable, while retaining the previous published candidate for rollback.

Gaps and non-claims

  • The archives were not downloaded in this phase, so the emitted member list was not independently enumerated from the TAR/ZIP central directories. Reference Data and Hoboleaks member paths are derived from the current EVE Ref exporter source and metadata-driven builder, which is stronger than guessing but remains subject to exporter changes.
  • CCP's official docs document the mapSolarSystems.jsonl path and fields used by the map example, but do not publish a complete machine-readable schema for every field on that page. The import should not require unlisted optional fields.
  • The EVE Ref ESI-scrape archive's internal universe/systems.yaml path is established by the current EVE Ref data-source table; its current artifact is stale for a production freshness requirement.
  • No credentials, Infisical data, or application secrets were accessed.

Sources

Reference in New Issue View Git Blame Copy Permalink