jb/eden
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
1371995cbeddc5da65e74bb89f6077972e1a9ffd
eden/docs/research/reference.md
T
jb fdcc4863f5 Initial Commit
2026-09-18 23:15:05 -04:00

18 KiB
Raw Blame History

EVE Ref reference-data and Hoboleaks source contract

Checked: 2026-09-18
Scope: current EVE Ref documentation/OpenAPI and the two current data.everef.net archive release routes named below. No full archive was downloaded. Firecrawl CLI was used for the documentation and representative API pages, sequentially; current machine-readable indexes, metadata, and small Hoboleaks JSON files were fetched directly.

Summary

EVE Ref combines CCP SDE, an ESI scrape, and Hoboleaks into Reference Data. The Reference Data contract is explicitly still in development and its OpenAPI description is unstable, so the compiler must validate the incoming schema rather than assume that an omitted optional field is present. The current archive artifacts are published through directory index.json routes, while the running Reference Data API exposes independent input provenance at /meta.

The latest observed artifacts are:

Artifact Release route Bytes Last modified ETag (opaque HTTP validator)
Reference Data reference-data/index.json / reference-data-latest.tar.xz 14,512,812 2026-09-13 11:16:26.504Z a4b2dd0325ffac28158f2b93500a0d52
Hoboleaks SDE hoboleaks-sde/index.json / hoboleaks-sde-latest.tar.xz 675,868 2026-09-13 11:05:28.403Z 5681550f6fe5f2927b055dfe33883655

The dated, immutable-looking release routes are listed by reference-data/history/index.json and hoboleaks-sde/history/index.json. Treat the ETag as an HTTP cache validator, not as a published SHA-256 checksum.

1. EVE Ref data model and endpoint contract

The Reference Data documentation says the set merges the latest SDE, ESI scrape, and Hoboleaks export; the complete download is linked from that page. It also states the current normalization rules:

  • fields and URLs use snake_case;
  • the JSON layout is mostly ESI-shaped;
  • keyed objects are preferred over arrays to make merging deterministic;
  • names and descriptions are language maps, as in the SDE.

The pinned Reference Data OpenAPI document (commit 58dafd29, checked 2026-09-18) declares openapi: 3.0.1, server https://ref-data.everef.net, and says the spec is unstable. The examples below use the documented schema plus the live example endpoints linked by the docs. OpenAPI properties are not marked required for these records; a fixture may therefore be a valid minimal object, but a parser must not turn missing optional fields into invented defaults.

Categories

InventoryCategory has these documented properties:

category_id: int64
name: map<string, string>
published: boolean
icon_id: int64
group_ids: array<int64>       # added by EVE Ref

A current small representative fixture is the live category 4 response, with the join-bearing fields retained:

{
  "category_id": 4,
  "name": {"en": "Material"},
  "published": true,
  "icon_id": 22,
  "group_ids": [18]
}

The live response currently contains additional language-map entries and additional group IDs; the fixture above is a deliberately small projection, not a claim that the production response contains only one group.

Groups

InventoryGroup has these documented properties:

anchorable: boolean
anchored: boolean
category_id: int64
fittable_non_singleton: boolean
group_id: int64
icon_id: int64
name: map<string, string>
published: boolean
type_ids: array<int64>        # added by EVE Ref
use_base_price: boolean

A current group 18 response can be reduced for a join fixture to:

{
  "category_id": 4,
  "group_id": 18,
  "name": {"en": "Mineral"},
  "published": true,
  "anchorable": false,
  "anchored": false,
  "fittable_non_singleton": false,
  "icon_id": 22,
  "use_base_price": true,
  "type_ids": [34]
}

The production response has more language entries and more type IDs. Preserve the integer IDs as IDs; do not infer them from names.

Types

InventoryType is intentionally broad. Its documented properties include the identity/join fields type_id, group_id, and EVE Ref-added category_id; language maps name and description; booleans such as published, is_blueprint, is_skill, is_ore, and is_repackable; numeric physical/economic fields such as volume, packaged_volume, portion_size, mass, capacity, radius, and base_price; and keyed maps including dogma_attributes, dogma_effects, type_materials, required_skills, and produced_by_blueprints.

A minimal current type 34 response sufficient to exercise category/group/type joins is:

{
  "type_id": 34,
  "name": {"en": "Tritanium"},
  "group_id": 18,
  "category_id": 4,
  "published": true,
  "portion_size": 1,
  "volume": 0.01,
  "packaged_volume": 0.01,
  "mass": 0,
  "base_price": 2.0
}

This is a projection of the current response; fields not needed by this fixture are omitted, not declared unsupported. In particular, do not require every optional InventoryType property to exist on every record.

Blueprints, activities, materials, and products

The OpenAPI schemas are exact about the important nesting:

Blueprint {
  activities: map<string, BlueprintActivity>
  blueprint_type_id: int64
  max_production_limit: int32
}

BlueprintActivity {
  materials: map<string, BlueprintMaterial>
  products: map<string, BlueprintMaterial>
  required_skills: map<string, int32>
  time: int64
}

BlueprintMaterial {
  probability: double       # optional; present for probabilistic products
  quantity: int64
  type_id: int64
}

The activity/material/product maps are keyed objects, not arrays. The key is a serialized ID for the material/product maps and an activity name for activities; the schema does not publish a closed activity enum. A current blueprint 683 response demonstrates both deterministic manufacturing and probabilistic invention. A tiny exact manufacturing slice for a fixture is:

{
  "blueprint_type_id": 683,
  "activities": {
    "manufacturing": {
      "materials": {
        "34": {"quantity": 24000, "type_id": 34}
      },
      "products": {
        "582": {"quantity": 1, "type_id": 582}
      },
      "required_skills": {"3380": 1},
      "time": 6000
    }
  },
  "max_production_limit": 30
}

The live object also has other activities and manufacturing materials; the slice above retains only the fields needed to test keyed material/product lookup and a required-skill map. Do not flatten these maps into an array without preserving their IDs.

Systems and XYZ: explicit scope boundary

The EVE Ref data-source table lists systems under SDE mapSolarSystems.yaml and ESI universe/systems.yaml; it does not list a Reference Data /systems endpoint. The Reference Data OpenAPI has Coordinate {x, y, z} and uses it for Region.position, but it does not define a solar-system schema. Therefore, a systems fixture must not be presented as a Reference Data record.

For a compiler that also ingests the documented ESI system source, the official ESI schema requires this shape:

{
  "system_id": 30000142,
  "name": "Jita",
  "constellation_id": 20000020,
  "position": {
    "x": -129064861735000000.0,
    "y": 60755306910000000.0,
    "z": 117469227060000000.0
  },
  "security_status": 0.9459131360054016
}

The position object is the requested XYZ fixture. system_id, name, position, security_status, and constellation_id are required by the current ESI OpenAPI; planets, star, gates, stations, and security class are optional. The EVE Ref source table is the authority for the fact that systems come from those SDE/ESI paths; the official ESI machine-readable schema is the authority for the exact ESI record shape. This distinction avoids inventing a missing Reference Data /systems schema.

2. Hoboleaks SDE contract

The EVE Ref Hoboleaks SDE page identifies data.everef.net/hoboleaks-sde/ as the archive mirror for files published by Hoboleaks SDE. The Hoboleaks landing page says these are JSON complements for data missing from the official SDE and/or ESI, and warns that they are updated after TQ patches and can temporarily lag the SDE.

There is no formal JSON Schema linked for every Hoboleaks file. The following are exact current small-file structures observed on 2026-09-18; preserve their source casing and top-level map shape when ingesting raw Hoboleaks. EVE Ref's normalized API shape is separate and uses snake_case.

Hoboleaks tq/meta.json: per-file provenance and stale state

The landing page explicitly documents tq/meta.json as update metadata, including a timestamp, revision, per-file MD5, and a stale flag. A current excerpt is:

{
  "timestamp": "2026-09-13T10:41:34",
  "revision": 3503375,
  "files": {
    "industryactivities.json": {
      "deprecated": false,
      "stale": false,
      "md5": "bfeffbb198389876e9b7b8f70c81615d",
      "revision": 3503375
    },
    "attributeorders.json": {
      "deprecated": false,
      "stale": true,
      "md5": "72906f5a2415bbdca45fc06936b20096",
      "revision": 2911964
    }
  }
}

stale: true means Hoboleaks says that file's update was not successful and its contents may be outdated; it is not equivalent to a transport failure. The compiler should record the file as present-but-stale and make the promotion policy explicit. The md5 is a content-change detector supplied by Hoboleaks; it is not a substitute for a locally computed SHA-256 of an archive.

Small useful raw maps

industryactivities.json is a small keyed object. Its current entries use numeric string keys and camelCase fields:

{
  "1": {
    "activityID": 1,
    "description": "Manufacturing of things",
    "activityName": "Manufacturing"
  },
  "9": {
    "activityID": 9,
    "description": "The process of combining raw and intermediate materials to create advanced components",
    "activityName": "Reactions"
  }
}

agenttypes.json is even smaller and maps numeric string keys directly to strings:

{
  "1": "NonAgent",
  "4": "ResearchAgent",
  "13": "HeraldryAgent"
}

repackagedvolumes.json is a keyed map from a type ID string to a numeric volume:

{
  "52907": 1300000.0,
  "34828": 5000.0,
  "22544": 3750.0
}

The EVE Ref data-source table specifically identifies repackagedvolumes.json as an addition to inventory types. These files are useful smoke-test inputs because they exercise object maps, string keys, source-specific camelCase, booleans, strings, integers, and floating-point values without requiring a large dataset.

The table also names Hoboleaks blueprints.json, typematerials.json, and other large files, but no raw schema claim is made here for those files: their full payloads were not downloaded, and the normalized EVE Ref blueprint contract above is the only blueprint/material/product contract asserted in this document.

3. Independent provenance and compiler policy

Use independent observations for each layer; do not treat one successful endpoint as proof that all sources succeeded.

What to record Route/evidence Meaning
Archive artifact identity data.everef.net/{reference-data,hoboleaks-sde}/index.json URL, byte size, last_modified, opaque ETag, and archive type.
Immutable release history .../history/index.json and dated year index Dated artifact URL, file_time, byte size, last-modified, and opaque ETag.
Reference build/source provenance https://ref-data.everef.net/meta Current build_time and SHA-256 values for esi, hoboleaks, and sde. Observed: build 2026-09-13T11:10:03Z; ESI 645b2902174fe52561b946a5c6e39e257358803956238f0750798e1cde5e4077; Hoboleaks 477799f37c9275c3ad408117cce8e7913688be41b17fe404cd48e3b501be55cd; SDE 56f5f7eba4b401aeb2fc1090d2db5ab9dd55cc08a396871e764df45017fa70af.
Hoboleaks per-file state https://sde.hoboleaks.space/tq/meta.json Per-file revision, MD5, stale, and deprecated, plus Hoboleaks timestamp/revision.
Independent HTTP check HEAD on the exact archive and metadata URLs Transport status, content length, ETag, and Last-Modified; use to detect a changed artifact before parsing.

The two archive indexes publish no SHA-256 field. For a future downloader, compute SHA-256 after downloading to a staging path and store it separately from the server ETag and from /meta's input-source hashes. Do not label the archive ETag as a checksum.

The /meta payload itself is a small useful provenance fixture:

{
  "build_time": "2026-09-13T11:10:03Z",
  "esi": {"sha256": "645b2902174fe52561b946a5c6e39e257358803956238f0750798e1cde5e4077"},
  "hoboleaks": {"sha256": "477799f37c9275c3ad408117cce8e7913688be41b17fe404cd48e3b501be55cd"},
  "sde": {"sha256": "56f5f7eba4b401aeb2fc1090d2db5ab9dd55cc08a396871e764df45017fa70af"}
}

This is source/build provenance for the running Reference Data service, not a cryptographic checksum of either .tar.xz artifact.

Recommended staged promotion behavior (design guidance, not an EVE schema)

  • Build each candidate in a fresh immutable SQLite database; never mutate the currently served database while parsing.
  • Capture one source result for Reference Data and one for Hoboleaks: success, stale, or failed with URL, HTTP metadata, bytes, local SHA-256, source revision/build, and an error detail. Keep source results independent so a Hoboleaks failure is not hidden by a successful Reference Data download, and vice versa.
  • Validate schema, IDs, keyed-map joins, duplicate keys, numeric ranges, and the expected fixture set before the candidate is eligible for promotion. A transport-successful but stale: true Hoboleaks file is visibly stale, not silently “fresh”.
  • Write and fsync the candidate database and its manifest before publishing a single atomic pointer/manifest to it. Keep the previous manifest and immutable DB available for immediate rollback; a failed source or validation must leave the previous pointer untouched.
  • Store both archive provenance (artifact URL/ETag/Last-Modified/local hash) and EVE Ref provenance (/meta build time and input hashes). They answer different questions and must not be merged into one version field.

The local manifest fields in this subsection are implementation recommendations only; they are not claims about a published EVE Ref schema.

Gaps and non-claims

  • Reference Data is explicitly marked “currently in development”; the OpenAPI spec may change without a compatibility promise.
  • reference-data-latest.tar.xz and hoboleaks-sde-latest.tar.xz are release artifacts; their directory indexes do not document an internal tar member manifest or a published archive SHA-256. No full archive was downloaded to infer one.
  • Reference Data currently has no documented /systems endpoint. Systems XYZ are covered here only as the ESI/SDE source shape identified by EVE Ref, not as a Reference Data record.
  • Hoboleaks documents file names and metadata semantics but does not publish a formal JSON Schema for every raw file. No unsupported raw schema is inferred for large blueprints.json or typematerials.json.
  • A stale Hoboleaks file can be outdated while still retrievable. Promotion policy must decide whether that is allowed; the source does not define the application's policy.

Sources

Kept sources are limited to current EVE Ref docs/schema, the linked official ESI machine-readable schema for the explicit systems boundary, and the linked current Hoboleaks metadata/files.

Reference in New Issue View Git Blame Copy Permalink