jb/eden
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial Commit

Browse Source
This commit is contained in:
jb
2026-09-18 23:15:05 -04:00
commit fdcc4863f5
118 changed files with 16868 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/research/production-import.md
+260
View File
@@ -0,0 +1,260 @@
# Production reference-data import plan
**Checked:** 2026-09-18
**Scope:** full real-data import is now authorized, but this phase performed **no archive download**. Only release indexes, HTTP `HEAD` metadata, official documentation/source, and small metadata/API files were inspected. Credentials and Infisical were not touched.
## Decision summary
1. Download the current Reference Data and Hoboleaks archives, then parse their exporter-defined root members. Their current compressed sizes are 14,512,812 bytes and 675,868 bytes.
2. Use CCP's current JSON Lines SDE for the universe complement, specifically `mapSolarSystems.jsonl`. CCP's current build is `3503375` (released 2026-09-10), and the official map guide directly documents reading `_key` and `position.x/y/z` from that member.
3. Do **not** use the EVE Ref ESI-scrape archive as the production universe source: its latest published artifact is dated 2025-11-26, while the CCP SDE is current to 2026-09-10. The ESI scrape is still useful as a shape/reference comparison, not as the freshness authority.
4. The selected compressed download budget is **114,291,615 bytes (~108.997 MiB)** before temporary extraction/database space: Reference Data + Hoboleaks + CCP JSONL SDE.
## 1. Current release metadata (preflight; no archive bytes read)
The values below came from the public JSON indexes and final archive `HEAD` responses. ETags are opaque HTTP validators, not published SHA-256 checksums.
| Input | Immutable/current URL | Bytes | Approx. compressed size | Last-Modified | ETag / version |
| --- | --- | ---: | ---: | --- | --- |
| EVE Ref Reference Data | [`reference-data-latest.tar.xz`](https://data.everef.net/reference-data/reference-data-latest.tar.xz) (dated route: [`reference-data-2026-09-13.tar.xz`](https://data.everef.net/reference-data/history/2026/reference-data-2026-09-13.tar.xz)) | 14,512,812 | 13.84 MiB | 2026-09-13 11:16:26Z | `a4b2dd0325ffac28158f2b93500a0d52` |
| EVE Ref Hoboleaks mirror | [`hoboleaks-sde-latest.tar.xz`](https://data.everef.net/hoboleaks-sde/hoboleaks-sde-latest.tar.xz) (dated route: [`hoboleaks-sde-2026-09-13.tar.xz`](https://data.everef.net/hoboleaks-sde/history/2026/hoboleaks-sde-2026-09-13.tar.xz)) | 675,868 | 660.03 KiB | 2026-09-13 11:05:28Z | `5681550f6fe5f2927b055dfe33883655` |
| EVE Ref ESI scrape (not recommended for production systems) | [`eve-ref-esi-scrape-latest.tar.xz`](https://data.everef.net/esi-scrape/eve-ref-esi-scrape-latest.tar.xz) | 77,038,676 | 73.47 MiB | 2025-11-26 15:42:24Z | `08fe439c74509e521620f588a13cfb37-10` |
| CCP SDE JSONL, build 3503375 | [`eve-online-static-data-3503375-jsonl.zip`](https://developers.eveonline.com/static-data/tranquility/eve-online-static-data-3503375-jsonl.zip) | 99,102,935 | 94.51 MiB | 2026-09-10 11:21:36Z | `fce5dcb01dbe357402ace2848297f556-12` |
The CCP automation endpoint currently returns:
```json
{"_key":"sde","buildNumber":3503375,"releaseDate":"2026-09-10T11:09:04Z"}
```
The EVE Ref archive indexes are [`reference-data/index.json`](https://data.everef.net/reference-data/index.json), [`hoboleaks-sde/index.json`](https://data.everef.net/hoboleaks-sde/index.json), and [`esi-scrape/index.json`](https://data.everef.net/esi-scrape/index.json). CCP's official automation documentation says the latest alias redirects to the build-numbered URL and that all resources support ETag/Last-Modified. Record both the alias and the resolved immutable URL in the import manifest.
EVE Ref's running [`/meta`](https://ref-data.everef.net/meta) currently reports build time `2026-09-13T11:10:03Z` and source hashes:
```json
{
"build_time": "2026-09-13T11:10:03Z",
"esi": {"sha256": "645b2902174fe52561b946a5c6e39e257358803956238f0750798e1cde5e4077"},
"hoboleaks": {"sha256": "477799f37c9275c3ad408117cce8e7913688be41b17fe404cd48e3b501be55cd"},
"sde": {"sha256": "56f5f7eba4b401aeb2fc1090d2db5ab9dd55cc08a396871e764df45017fa70af"}
}
```
These are hashes of EVE Ref's three **input archives**, not checksums of `reference-data-latest.tar.xz`. Current EVE Ref source downloads the YAML SDE input (`SDE_V2_YAML`) when building Reference Data, so `/meta.sde.sha256` must not be presented as the hash of the separately selected CCP JSONL SDE complement.
## 2. Reference Data archive: exporter-defined member layout
The current EVE Ref exporter source is the strongest available archive contract:
- [`BuildRefData.java` at commit 17948f63](https://github.com/autonomouslogic/eve-ref/blob/17948f63ad35b3417ad888a72f6d4fcc49e2b72d/src/main/java/com/autonomouslogic/everef/cli/refdata/BuildRefData.java#L335-L390) writes `meta.json`, then one `.json` member for every entry in `refdata.yaml`, into a TAR and XZ-compresses it.
- [`DataUtil.java`](https://github.com/autonomouslogic/eve-ref/blob/75b17f51e8880b474e44236049b50198a58d0332/src/main/java/com/autonomouslogic/everef/util/DataUtil.java#L1-L85) shows that the builder's SDE input is the configured `SDE_V2_YAML` ZIP, while the separate production universe complement below deliberately uses CCP's current JSONL ZIP.
- [`refdata.yaml` at commit ac663020](https://github.com/autonomouslogic/eve-ref/blob/ac6630205f41404c7635f9efcfd8d6646e45370f/src/main/resources/refdata.yaml) defines the member basenames, source filenames, ID fields, and output-store relationships.
- [`RefDataUtil.java`](https://github.com/autonomouslogic/eve-ref/blob/17948f63ad35b3417ad888a72f6d4fcc49e2b72d/src/main/java/com/autonomouslogic/everef/util/RefDataUtil.java#L103-L136) confirms that `meta.json` is a standalone object and every other JSON member is parsed as an object whose property names are decimal IDs. Its config loader defaults absent `outputFile` to lower-snake-case (`marketGroups` → `market_groups`) and absent `outputStore` to the config ID ([source](https://github.com/autonomouslogic/eve-ref/blob/17948f63ad35b3417ad888a72f6d4fcc49e2b72d/src/main/java/com/autonomouslogic/everef/util/RefDataUtil.java#L177-L200)).
### Exact member names
The exporter emits these 19 root members in config order:
```text
meta.json
categories.json
groups.json
market_groups.json
types.json
type_materials.json
type_bonus.json
masteries.json
type_dogma.json
dogma_attributes.json
dogma_effects.json
meta_groups.json
mutaplasmids.json
skills.json
units.json
blueprints.json
icons.json
regions.json
schematics.json
```
`meta.json` is a `RefDataMeta` object containing `build_time` and `sha256` objects for `sde`, `esi`, and `hoboleaks`. Every other member is written as:
```json
{
"<decimal-id>": {"<normalized record fields>": "..."}
}
```
The writer serializes the map key as a JSON string and the value as the normalized EVE Ref record. It does not write one file per ID inside the archive. A loader should preserve the string key as the canonical ID while validating that it parses as a non-negative 64-bit integer.
### Member/source map
| Member | Config ID / output store | Input members named by the exporter | Import role |
| --- | --- | --- | --- |
| `categories.json` | `categories` / `categories` | SDE `categories.yaml`; ESI `universe/categories` | Canonical category map. |
| `groups.json` | `groups` / `groups` | SDE `groups.yaml`; ESI `universe/groups` | Canonical group map. |
| `market_groups.json` | `marketGroups` / `marketGroups` | SDE `marketGroups.yaml`; ESI `markets/groups` | Canonical market-group map. |
| `types.json` | `types` / `types` | SDE `types.yaml`; ESI `universe/types` | Canonical inventory-type map. |
| `type_materials.json` | `typeMaterials` / `types` | SDE `typeMaterials.yaml`; Hoboleaks `typematerials.json` | Supplemental type merge input; values belong in `types`. |
| `type_bonus.json` | `typeBonus` / `types` | SDE `typeBonus.yaml` | Supplemental type merge input; values belong in `types`. |
| `masteries.json` | `masteries` / `types` | SDE `masteries.yaml` | Supplemental type merge input; values belong in `types`. |
| `type_dogma.json` | `typeDogma` / `types` | SDE `typeDogma.yaml` | Supplemental type merge input; values belong in `types`. |
| `dogma_attributes.json` | `dogmaAttributes` / `dogmaAttributes` | SDE `dogmaAttributes.yaml`; ESI `dogma/attributes` | Canonical dogma-attribute map. |
| `dogma_effects.json` | `dogmaEffects` / `dogmaEffects` | SDE `dogmaEffects.yaml`; ESI `dogma/effects` | Canonical dogma-effect map. |
| `meta_groups.json` | `metaGroups` / `metaGroups` | SDE `metaGroups.yaml` | Canonical meta-group map. |
| `mutaplasmids.json` | `mutaplasmids` / `mutaplasmids` | Hoboleaks `dynamicitemattributes.json` | Canonical mutaplasmid map. |
| `skills.json` | `skills` / `skills` | Derived by EVE Ref decorators | Canonical skill map. |
| `units.json` | `units` / `units` | Hoboleaks `dogmaunits.json` | Canonical unit map. |
| `blueprints.json` | `blueprints` / `blueprints` | SDE `blueprints.yaml`; Hoboleaks `blueprints.json` | Canonical blueprint map. |
| `icons.json` | `icons` / `icons` | SDE `icons.yaml` | Canonical icon map. |
| `regions.json` | `regions` / `regions` | SDE `mapRegions.yaml`; ESI `universe/regions` | Canonical region map. |
| `schematics.json` | `schematics` / `schematics` | SDE `planetSchematics.yaml`; ESI `universe/schematics` | Canonical schematic map. |
The four type supplemental configs have `dedicatedOutput: false` and `outputStore: types` in the current config. The exporter still iterates every config and names a member from `config.outputFile`; therefore, treat `types.json` as the canonical merged type table, and treat those four sidecar members as exporter outputs whose content must be checked after download rather than assumed to be separate canonical tables. Do not invent records from a sidecar if the authoritative merged `types.json` does not contain them.
### Reference Data parser invariants
- Read TAR entries sequentially through XZ decompression; there is no need to materialize an uncompressed archive.
- Require `meta.json` exactly once.
- For each other `.json` member, require a top-level JSON object; every property key must be a decimal ID; reject duplicate IDs and malformed values.
- Skip no member silently except an explicitly documented non-JSON compatibility member. The current EVE Ref parser skips non-JSON entries but the exporter contract above emits JSON only.
- Preserve source/build metadata separately from the local archive URL, byte count, ETag, Last-Modified, and locally computed SHA-256.
## 3. Hoboleaks archive: exact root member layout
The current archive builder is [`ScrapeHoboleaks.java` at commit 17948f63](https://github.com/autonomouslogic/eve-ref/blob/17948f63ad35b3417ad888a72f6d4fcc49e2b72d/src/main/java/com/autonomouslogic/everef/cli/ScrapeHoboleaks.java#L87-L143): it downloads Hoboleaks `meta.json`, writes that as root `meta.json`, iterates `meta.files`, downloads each named file from the Hoboleaks root, and writes that file at the archive root with the same filename. There are no nested `tq/` directories inside the EVE Ref mirror archive.
The current [`tq/meta.json`](https://sde.hoboleaks.space/tq/meta.json) has revision `3503375`, timestamp `2026-09-13T10:41:34`, and 28 file entries. Thus the expected current archive has **29 root members**:
```text
meta.json
accountingentrytypes.json
agenttypes.json
attributeorders.json
clonestates.json
compressibletypes.json
dbuffs.json
dogmaeffectcategories.json
dogmaunits.json
dynamicitemattributes.json
expertsystems.json
graphicmaterialsets.json
industryactivities.json
industryassemblylines.json
industryinstallationtypes.json
industrymodifiersources.json
industrytargetfilters.json
localization_dgmattributes.json
localization_languages.json
repackagedvolumes.json
schoolmap.json
schools.json
skillplans.json
skinmaterialnames.json
skinmaterials.json
skins.json
stationstandingsrestrictions.json
typematerials.json
blueprints.json
```
The list is metadata-driven, not a hard-coded schema promise: if a future `meta.json` adds/removes a file, the archive member set changes with it. The source files retain Hoboleaks casing and shape (for example, `activityID`/`activityName`, not normalized `activity_id`/`activity_name`). The prior [reference contract](https://github.com/autonomouslogic/eve-ref/blob/main/docs/src/datasets/reference-data.md) records small current examples for `industryactivities.json`, `agenttypes.json`, and `repackagedvolumes.json`.
Hoboleaks metadata also carries per-file `md5`, `revision`, `stale`, and `deprecated`. A file can be retrievable but stale. Record this as source state; do not convert it to a transport-successful fresh input.
## 4. Universe XYZ source choice
### Comparison
| Source | Current release | Member/path shape | Freshness and scale | Decision |
| --- | --- | --- | --- | --- |
| EVE Ref ESI scrape | `eve-ref-esi-scrape-latest.tar.xz`, 77,038,676 bytes, last modified 2025-11-26 | EVE Ref's source table names the file `universe/systems.yaml`; ESI universe records use `system_id`, `name`, `position: {x,y,z}`, and related fields | One complete scrape archive, but almost ten months behind the current SDE and not the freshness authority | Do not select for production universe data. |
| CCP SDE JSONL | build `3503375`, 99,102,935 bytes, released 2026-09-10 | Root member `mapSolarSystems.jsonl`; each line has `_key` (solar-system ID) and object fields such as `position.x/y/z` | Current official SDE; JSONL is explicitly documented as streamable and the map guide covers all space types | **Select this source.** |
| ESI live per-system routes | Current API; current ID list observed at 8,490 systems | `/universe/systems/{system_id}` response; one request per system | Fresh but ~8,490 API requests, rate/cache policy, and no single immutable snapshot | Use only for spot checks, not bulk import. |
The official CCP [Map Data guide](https://developers.eveonline.com/docs/guides/map-data/) says map data is in SDE files prefixed `map`, explicitly names `mapSolarSystems`, and its executable example opens `./mapSolarSystems.jsonl`, parses each line, reads `system["_key"]`, and reads `system["position"]["x"]`, `["y"]`, and `["z"]`. It also says SDE/ESI include known, wormhole, and abyssal space. The current [`schema-changelog.yaml`](https://developers.eveonline.com/static-data/tranquility/schema-changelog.yaml) records `mapSolarSystems` as a current file and records `position2D` as added after build `3096543`.
The official [System Security guide](https://developers.eveonline.com/docs/guides/system-security/) independently identifies `mapSolarSystems.jsonl` as the SDE source and `securityStatus` as a full-precision floating-point field. The official [Static Data docs](https://developers.eveonline.com/docs/services/static-data/) define JSONL handling: an integer-keyed object record is represented as one line with `_key`; object values remain object fields (the map guide's `system["position"]` access is the concrete example).
### Minimal exact SDE system line contract
Do not claim an entire system schema from the small guide example. The safe import contract is the fields directly documented by the official examples:
```json
{
"_key": 30000142,
"position": {
"x": -129064861735000000.0,
"y": 60755306910000000.0,
"z": 117469227060000000.0
},
"securityStatus": 0.9459131360054016
}
```
The real line has additional fields. Preserve unknown fields or explicitly ignore them only after validating the fields needed by the application. Use `_key` as `system_id`; do not expect the ESI `system_id` field in this SDE member. Keep XYZ as 64-bit floating-point values; the official guide warns that 32-bit precision is insufficient for interstellar and planetary coordinates.
A current ESI list request returned 8,490 IDs (checked 2026-09-18), confirming that a production profile limited to synthetic fixtures or a handful of systems is not sufficient. The CCP SDE map file is the efficient immutable complement: one current archive download rather than thousands of ESI calls.
## 5. Download and streaming plan (after this phase)
### Preflight
1. Fetch only the small JSON indexes, CCP `latest.jsonl`, Hoboleaks `tq/meta.json`, and `HEAD` metadata as done here.
2. Resolve each `latest` alias and record the final URL, status, content length, ETag, Last-Modified, media type, source revision/build, and retrieval time.
3. Require the expected sizes above for this checked release, but treat a changed release as a new candidate rather than a fatal global condition. Refresh metadata and use the new immutable URL.
4. Never use an ETag as a SHA-256. On the actual download, stream bytes through a local SHA-256 digest and compare byte count with `Content-Length`.
### Selected transfers
| Transfer | Compressed bytes |
| --- | ---: |
| Reference Data | 14,512,812 |
| Hoboleaks SDE | 675,868 |
| CCP SDE JSONL | 99,102,935 |
| **Total** | **114,291,615 (~108.997 MiB)** |
The ESI-scrape alternative would total 92,227,356 bytes with Reference Data + Hoboleaks, but it is stale and is not selected. Do not download both universe complements unless a deliberate cross-check is needed.
### Safe extraction order
1. Create a unique staging directory and `.part` file per source. Stream each HTTP body to disk while hashing and counting; do not place bytes in a JS/Bun `Buffer` for the whole archive.
2. Close and verify the staged file before opening it. Keep the immutable staged artifact until the candidate DB and manifest validate.
3. For Reference Data and Hoboleaks, stream XZ → TAR sequentially. Parse each selected root member as it arrives; close the TAR/XZ reader on any error. These archives are small enough to retain on disk but not to trust as an in-memory blob.
4. For CCP SDE JSONL, keep the completed ZIP staged, inspect the ZIP central directory, and stream only `mapSolarSystems.jsonl` through the JSON Lines parser. Do not extract the whole ~99 MB ZIP merely to build the universe complement. Keep the ZIP hash in provenance even if only one member is consumed.
5. Parse JSONL line-by-line: reject malformed lines, require integer `_key`, require finite numeric `position.x/y/z`, reject duplicate IDs, and count systems. Preserve the full source line or unknown fields if later features may need them.
6. Build a new immutable SQLite candidate, validate all cross-source references and the real universe count (expected to be >5,000), close it, then atomically publish its manifest. Keep the previous manifest/DB untouched on any source or validation failure.
### Independent source status
Track three source outcomes independently:
- `reference_data`: HTTP/download/hash/archive/member validation;
- `hoboleaks`: HTTP/download/hash/archive/member validation plus per-file `stale`/`deprecated` state;
- `ccp_sde_jsonl`: HTTP/download/hash/ZIP member/JSONL validation and build number.
A successful Reference Data archive must not mask a failed SDE universe complement. A stale Hoboleaks file is a distinct policy state from a failed download. Promotion should require the source set selected by the production profile to be successful and policy-acceptable, while retaining the previous published candidate for rollback.
## Gaps and non-claims
- The archives were not downloaded in this phase, so the emitted member list was not independently enumerated from the TAR/ZIP central directories. Reference Data and Hoboleaks member paths are derived from the current EVE Ref exporter source and metadata-driven builder, which is stronger than guessing but remains subject to exporter changes.
- CCP's official docs document the `mapSolarSystems.jsonl` path and fields used by the map example, but do not publish a complete machine-readable schema for every field on that page. The import should not require unlisted optional fields.
- The EVE Ref ESI-scrape archive's internal `universe/systems.yaml` path is established by the current EVE Ref data-source table; its current artifact is stale for a production freshness requirement.
- No credentials, Infisical data, or application secrets were accessed.
## Sources
- [EVE Ref Reference Data docs](https://docs.everef.net/datasets/reference-data) — source combination, archive/API routes, data-source table, and normalization.
- [EVE Ref Reference Data archive index](https://data.everef.net/reference-data/index.json) and [Hoboleaks archive index](https://data.everef.net/hoboleaks-sde/index.json) — current sizes, timestamps, ETags, and URLs.
- [EVE Ref ESI scrape index](https://data.everef.net/esi-scrape/index.json) — current published ESI-scrape size and age.
- [EVE Ref `BuildRefData.java`](https://github.com/autonomouslogic/eve-ref/blob/17948f63ad35b3417ad888a72f6d4fcc49e2b72d/src/main/java/com/autonomouslogic/everef/cli/refdata/BuildRefData.java) — exact Reference Data TAR member writer and `meta.json` behavior.
- [EVE Ref `RefDataUtil.java`](https://github.com/autonomouslogic/eve-ref/blob/17948f63ad35b3417ad888a72f6d4fcc49e2b72d/src/main/java/com/autonomouslogic/everef/util/RefDataUtil.java) — archive parser expectations.
- [EVE Ref `refdata.yaml`](https://github.com/autonomouslogic/eve-ref/blob/ac6630205f41404c7635f9efcfd8d6646e45370f/src/main/resources/refdata.yaml) — current member basenames, source paths, and output stores.
- [EVE Ref `ScrapeHoboleaks.java`](https://github.com/autonomouslogic/eve-ref/blob/17948f63ad35b3417ad888a72f6d4fcc49e2b72d/src/main/java/com/autonomouslogic/everef/cli/ScrapeHoboleaks.java) — metadata-driven root member creation.
- [Hoboleaks `tq/meta.json`](https://sde.hoboleaks.space/tq/meta.json) — current revision, file list, MD5, stale/deprecated state.
- [CCP Static Data docs](https://developers.eveonline.com/docs/services/static-data/) — JSONL format, automation routes, build metadata, HTTP caching.
- [CCP current build metadata](https://developers.eveonline.com/static-data/tranquility/latest.jsonl) and [build 3503375 JSONL release](https://developers.eveonline.com/static-data/tranquility/eve-online-static-data-3503375-jsonl.zip) — current build and release size.
- [CCP Map Data guide](https://developers.eveonline.com/docs/guides/map-data/) — `mapSolarSystems.jsonl`, `_key`, XYZ fields, all-space coverage, and coordinate precision.
- [CCP System Security guide](https://developers.eveonline.com/docs/guides/system-security/) — SDE system source and `securityStatus` field.
- [CCP schema changelog](https://developers.eveonline.com/static-data/tranquility/schema-changelog.yaml) — current `mapSolarSystems` file and `position2D` history.
- [Official ESI OpenAPI](https://esi.evetech.net/meta/openapi.json) and [ESI system ID list](https://esi.evetech.net/latest/universe/systems/) — ESI comparison shape and current system-count sanity check.