jb/eden
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial Commit

Browse Source
This commit is contained in:
jb
2026-09-18 23:15:05 -04:00
commit fdcc4863f5
118 changed files with 16868 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/research/eve-changelog.headers
+18
View File
@@ -0,0 +1,18 @@
HTTP/2 200
date: Fri, 18 Sep 2026 19:44:13 GMT
content-type: application/json
access-control-allow-origin: *
access-control-expose-headers: Etag, Retry-After, X-Compatibility-Date, X-Esi-Error-Limit-Remain, X-Esi-Error-Limit-Reset, X-Pages, X-Ratelimit-Group, X-Ratelimit-Limit, X-Ratelimit-Remaining, X-Ratelimit-Used
cache-control: public, max-age=600
content-language: en
expires: Fri, 18 Sep 2026 19:54:13 GMT
strict-transport-security: max-age=31536000
vary: X-Tenant
vary: Accept-Language
vary: X-Compatibility-Date
x-compatibility-date: 2025-09-26
x-ratelimit-group: meta
x-ratelimit-limit: 150/15m
x-ratelimit-remaining: 130
x-ratelimit-used: 2
docs/research/eve-changelog.json
+1
View File
File diff suppressed because one or more lines are too long
docs/research/eve-compatibility.headers
+19
View File
@@ -0,0 +1,19 @@
HTTP/2 200
date: Fri, 18 Sep 2026 19:44:12 GMT
content-type: application/json
content-length: 208
access-control-allow-origin: *
access-control-expose-headers: Etag, Retry-After, X-Compatibility-Date, X-Esi-Error-Limit-Remain, X-Esi-Error-Limit-Reset, X-Pages, X-Ratelimit-Group, X-Ratelimit-Limit, X-Ratelimit-Remaining, X-Ratelimit-Used
cache-control: public, max-age=600
content-language: en
expires: Fri, 18 Sep 2026 19:54:12 GMT
strict-transport-security: max-age=31536000
vary: X-Tenant
vary: Accept-Language
vary: X-Compatibility-Date
x-compatibility-date: 2020-01-01
x-ratelimit-group: meta
x-ratelimit-limit: 150/15m
x-ratelimit-remaining: 138
x-ratelimit-used: 2
docs/research/eve-compatibility.json
+1
View File
@@ -0,0 +1 @@
{"compatibility_dates":["2026-08-18","2026-08-04","2026-07-21","2026-07-17","2026-06-09","2026-05-19","2025-12-16","2025-11-06","2025-09-30","2025-09-26","2025-08-26","2025-04-02","2025-04-01","2020-01-01"]}
docs/research/eve-openapi-evidence.json
+4783
View File
File diff suppressed because it is too large Load Diff
docs/research/eve-openapi.headers
+17
View File
@@ -0,0 +1,17 @@
HTTP/2 200
date: Fri, 18 Sep 2026 19:44:14 GMT
content-type: application/openapi+json
access-control-allow-origin: *
access-control-expose-headers: Etag, Retry-After, X-Compatibility-Date, X-Esi-Error-Limit-Remain, X-Esi-Error-Limit-Reset, X-Pages, X-Ratelimit-Group, X-Ratelimit-Limit, X-Ratelimit-Remaining, X-Ratelimit-Used
cache-control: public, max-age=600
content-language: en
strict-transport-security: max-age=31536000
vary: X-Tenant
vary: Accept-Language
vary: X-Compatibility-Date
x-compatibility-date: 2026-08-18
x-ratelimit-group: meta
x-ratelimit-limit: 150/15m
x-ratelimit-remaining: 128
x-ratelimit-used: 2
docs/research/eve.md
+298
View File
@@ -0,0 +1,298 @@
# EVE ESI/SSO research brief
**Checked:** 2026-09-18 (UTC)
**Scope:** current official ESI/SSO documentation and live ESI OpenAPI/metadata only. No SDE archive was downloaded. The captured OpenAPI evidence is [`eve-openapi-evidence.json`](./eve-openapi-evidence.json); live compatibility/changelog responses are [`eve-compatibility.json`](./eve-compatibility.json) and [`eve-changelog.json`](./eve-changelog.json).
## Executive contract
- Use `https://esi.evetech.net` and send `X-Compatibility-Date: 2026-08-18`. This is the newest value in the live compatibility-date list on this date; `2026-09-18` is not yet a supported value. Pinning the header avoids silently receiving the oldest behavior. [ESI versioning](https://developers.eveonline.com/docs/services/esi/overview/#versioning) · [live compatibility dates](https://esi.evetech.net/meta/compatibility-dates)
- Character-scoped ESI tokens are character-specific. A local EVE app user may link multiple characters, but each character needs its own authorization/token record and the token's granted scopes apply only to that character. [SSO](https://developers.eveonline.com/docs/services/sso/#how-it-works)
- The asset route is paged (`page`, 1-based; `X-Pages`) and has a 3,600-second server/client cache. Market reference prices are one unpaged global array, also cached 3,600 seconds. [Current OpenAPI](https://esi.evetech.net/meta/openapi.json)
- An Access List membership is **not** proof of docking: the current Access List routes return list membership but no structure/profile association. A successful authenticated `GET /universe/structures/{structure_id}` is character-scoped ACL/access evidence, not an unconditional docking guarantee; a `structure_id` in `GET /characters/{character_id}/location` is evidence the character was docked there at the cached observation time. [Resolve Structure IDs](https://docs.esi.evetech.net/docs/scenarios/resolve_structure_ids.html) · [Current OpenAPI](https://esi.evetech.net/meta/openapi.json)
## ESI versioning and current OpenAPI
The live API exposes OpenAPI 3.1 at [`/meta/openapi.json`](https://esi.evetech.net/meta/openapi.json). It was fetched with `X-Compatibility-Date: 2026-08-18`; the response's `X-Compatibility-Date` and `info.version` were both `2026-08-18`. The live [`/meta/compatibility-dates`](https://esi.evetech.net/meta/compatibility-dates) response lists, newest first:
```text
2026-08-18, 2026-08-04, 2026-07-21, 2026-07-17, 2026-06-09,
2026-05-19, 2025-12-16, 2025-11-06, 2025-09-30, 2025-09-26,
2025-08-26, 2025-04-02, 2025-04-01, 2020-01-01
```
ESI documents `X-Compatibility-Date: YYYY-MM-DD` as the reviewed API behavior date; `compatibility_date` is the fallback for clients unable to set headers. It also says the API changes date at 11:00 UTC and a date cannot be future or below the oldest available. The live endpoint is the authoritative check for the value to pin.
The 2026-05-19 changelog release added Access Lists, Mercenary Dens/MTOs, Skyhooks, Sovereignty Hubs, raidable Skyhooks, and `/sovereignty/systems`; it removed `/sovereignty/map` and `/sovereignty/structures` at that compatibility date. [Live changelog](https://esi.evetech.net/meta/changelog) · [Equinox ESI announcement](https://developers.eveonline.com/blog/equinox-on-esi-structures-sovereignty-and-access-lists)
## Assets: exact current contract
### Character assets
```http
GET https://esi.evetech.net/characters/{character_id}/assets?page={page}
Authorization: Bearer <character-token>
X-Compatibility-Date: 2026-08-18
```
- Scope: `esi-assets.read_assets.v1`.
- `page` is optional, integer, minimum `1`; response header `X-Pages` is the total number of pages. Fetch page 1, read `X-Pages`, then pages 2..N. ESI's X-Pages guidance warns that cache expiry between pages can duplicate items; de-duplicate by `item_id` (and consider a stable page-fetch strategy). [X-Pages pagination](https://developers.eveonline.com/docs/services/esi/pagination/x-pages/)
- Cache: `x-cache-age: 3600`, `x-client-cache-ttl: 3600`, `x-server-cache-mode: ttl-based`, `x-server-cache-ttl: 3600`; response supports `Cache-Control`, `ETag`, `Last-Modified`, and requests support `If-None-Match` / `If-Modified-Since`.
- Rate extension: group `char-asset`, `1800` tokens per `15m` (where the new limiter is active).
- Response is an array. Each item has this exact shape:
| Field | Type | Required | Contract |
|---|---|---:|---|
| `is_blueprint_copy` | boolean | no | Whether the asset is a blueprint copy |
| `is_singleton` | boolean | yes | Singleton flag |
| `item_id` | int64 | yes | Asset item ID |
| `location_flag` | enum string | yes | Exact character enum below |
| `location_id` | int64 | yes | Parent/container/location ID |
| `location_type` | enum string | yes | `station`, `solar_system`, `item`, `other` |
| `quantity` | int64 | yes | Quantity |
| `type_id` | int64 | yes | EVE type ID; no type name or capital classification is included |
Character `location_flag` enum (exact OpenAPI values):
```text
AssetSafety, AutoFit, BoosterBay, CapsuleerDeliveries, Cargo,
CorporationGoalDeliveries, CorpseBay, Deliveries, DroneBay, ExpeditionHold,
FighterBay, FighterTube0, FighterTube1, FighterTube2, FighterTube3, FighterTube4,
FleetHangar, FrigateEscapeBay, Hangar, HangarAll, HiSlot0, HiSlot1, HiSlot2,
HiSlot3, HiSlot4, HiSlot5, HiSlot6, HiSlot7, HiddenModifiers, Implant,
InfrastructureHangar, LoSlot0, LoSlot1, LoSlot2, LoSlot3, LoSlot4, LoSlot5,
LoSlot6, LoSlot7, Locked, MedSlot0, MedSlot1, MedSlot2, MedSlot3, MedSlot4,
MedSlot5, MedSlot6, MedSlot7, MobileDepotHold, MoonMaterialBay, QuafeBay,
RigSlot0, RigSlot1, RigSlot2, RigSlot3, RigSlot4, RigSlot5, RigSlot6, RigSlot7,
ShipHangar, Skill, SpecializedAmmoHold, SpecializedAsteroidHold,
SpecializedCommandCenterHold, SpecializedFuelBay, SpecializedGasHold,
SpecializedIceHold, SpecializedIndustrialShipHold, SpecializedLargeShipHold,
SpecializedMaterialBay, SpecializedMediumShipHold, SpecializedMineralHold,
SpecializedOreHold, SpecializedPlanetaryCommoditiesHold, SpecializedSalvageHold,
SpecializedShipHold, SpecializedSmallShipHold, StructureDeedBay, SubSystemBay,
SubSystemSlot0, SubSystemSlot1, SubSystemSlot2, SubSystemSlot3, SubSystemSlot4,
SubSystemSlot5, SubSystemSlot6, SubSystemSlot7, Unlocked, Wardrobe
```
### Corporation assets
```http
GET https://esi.evetech.net/corporations/{corporation_id}/assets?page={page}
```
- Scope: `esi-assets.read_corporation_assets.v1`; current OpenAPI declares required role `Director`.
- Same page/cache/header contract, with rate group `corp-asset`, `1800/15m`.
- Same fields and requiredness as character assets. The exact corporation `location_flag` enum is:
```text
AssetSafety, AutoFit, Bonus, Booster, BoosterBay, Capsule, CapsuleerDeliveries,
Cargo, CorpDeliveries, CorpSAG1, CorpSAG2, CorpSAG3, CorpSAG4, CorpSAG5,
CorpSAG6, CorpSAG7, CorporationGoalDeliveries, CrateLoot, Deliveries, DroneBay,
DustBattle, DustDatabank, ExpeditionHold, FighterBay, FighterTube0, FighterTube1,
FighterTube2, FighterTube3, FighterTube4, FleetHangar, FrigateEscapeBay, Hangar,
HangarAll, HiSlot0, HiSlot1, HiSlot2, HiSlot3, HiSlot4, HiSlot5, HiSlot6, HiSlot7,
HiddenModifiers, Implant, Impounded, InfrastructureHangar, JunkyardReprocessed,
JunkyardTrashed, LoSlot0, LoSlot1, LoSlot2, LoSlot3, LoSlot4, LoSlot5, LoSlot6,
LoSlot7, Locked, MedSlot0, MedSlot1, MedSlot2, MedSlot3, MedSlot4, MedSlot5,
MedSlot6, MedSlot7, MobileDepotHold, MoonMaterialBay, OfficeFolder, Pilot,
PlanetSurface, QuafeBay, QuantumCoreRoom, Reward, RigSlot0, RigSlot1, RigSlot2,
RigSlot3, RigSlot4, RigSlot5, RigSlot6, RigSlot7, SecondaryStorage,
ServiceSlot0, ServiceSlot1, ServiceSlot2, ServiceSlot3, ServiceSlot4, ServiceSlot5,
ServiceSlot6, ServiceSlot7, ShipHangar, ShipOffline, Skill, SkillInTraining,
SpecializedAmmoHold, SpecializedAsteroidHold, SpecializedCommandCenterHold,
SpecializedFuelBay, SpecializedGasHold, SpecializedIceHold,
SpecializedIndustrialShipHold, SpecializedLargeShipHold, SpecializedMaterialBay,
SpecializedMediumShipHold, SpecializedMineralHold, SpecializedOreHold,
SpecializedPlanetaryCommoditiesHold, SpecializedSalvageHold, SpecializedShipHold,
SpecializedSmallShipHold, StructureActive, StructureFuel, StructureInactive,
StructureOffline, SubSystemBay, SubSystemSlot0, SubSystemSlot1, SubSystemSlot2,
SubSystemSlot3, SubSystemSlot4, SubSystemSlot5, SubSystemSlot6, SubSystemSlot7,
Unlocked, Wallet, Wardrobe
```
### Asset helper routes
The same asset scope/rate group applies to the character helper routes:
- `POST /characters/{character_id}/assets/locations` accepts a required JSON array of **1..1000 unique int64 `item_id`s** and returns `{item_id, position:{x,y,z}}`; coordinates for items in hangars/stations are `(0,0,0)`.
- `POST /characters/{character_id}/assets/names` accepts the same 1..1000 unique-ID array and returns `{item_id, name}` for custom-namable items.
- Corporation equivalents use `esi-assets.read_corporation_assets.v1`, require the current OpenAPI's `Director` role, and use the `corp-asset` group.
- These POST responses expose the same cache validators but have no page parameter. [Current OpenAPI evidence](./eve-openapi-evidence.json)
## Reference-price valuation
```http
GET https://esi.evetech.net/markets/prices
X-Compatibility-Date: 2026-08-18
```
- Public; no OAuth scope and no query parameters.
- Response is an unpaged array. Item schema is exactly `{type_id:int64}` plus optional `adjusted_price:number(double)` and optional `average_price:number(double)`; only `type_id` is required by OpenAPI. Do not assume either price field exists for every type.
- Cache: `x-cache-age: 3600`, client/server TTL 3600, server mode `ttl-based`; response advertises `Cache-Control`, `ETag`, and `Last-Modified`, and accepts `If-None-Match` / `If-Modified-Since`. The live response is global rather than region/structure-specific.
- The endpoint does not provide a price timestamp, region, order depth, or valuation semantics. Store fetch time and compatibility date with a valuation snapshot. Choose and document a missing-price policy; the ESI schema does **not** authorize an automatic `adjusted_price` fallback for a missing `average_price`.
- For each asset item with a known price, a transparent reference valuation is `quantity × selected price`; blueprint-copy, fitted/container, and location semantics remain application policy.
For actual structure-market orders, use `GET /markets/structures/{structure_id}?page=N` with `esi-markets.structure_markets.v1`; it is paged by `X-Pages`, cached for 300 seconds, and is not the same data as global reference prices.
## Rate limiting and cache headers
ESI currently documents two mutually exclusive limiter header families:
**New floating-window/token limiter** (only on routes that advertise `x-rate-limit` in OpenAPI):
- `X-Ratelimit-Group`: route bucket, e.g. `char-asset` or `meta`.
- `X-Ratelimit-Limit`: total tokens/window, e.g. `150/15m`.
- `X-Ratelimit-Remaining`: available tokens.
- `X-Ratelimit-Used`: tokens consumed by the request.
- `Retry-After` on `429`, in seconds.
The documented token costs are 2 for 2xx, 1 for 3xx, 5 for 4xx (excluding 429), and 0 for 5xx. Buckets are per rate-limit group and user ID; authenticated user ID is `<applicationID>:<characterID>`, not a global app-only bucket. [Rate limiting](https://developers.eveonline.com/docs/services/esi/rate-limiting/)
**Legacy error limiter** (still active for routes without the new limiter):
- `X-ESI-Error-Limit-Remain`: errors left in the current fixed window.
- `X-ESI-Error-Limit-Reset`: seconds until that window resets.
- It allows at most 100 non-2xx/3xx responses per minute; once reached, ESI returns 420s on routes. [Best practices](https://developers.eveonline.com/docs/services/esi/best-practices/#error-limit)
A live `GET /markets/prices` response on the check date exposed the legacy `X-ESI-*` headers (and ETag/Last-Modified); a live `GET /meta/openapi.json` response with the pinned date exposed the new `X-Ratelimit-*` family (`meta`, `150/15m`). Header values such as remaining/reset are volatile. The ESI `Access-Control-Expose-Headers` response also lists both families, `Retry-After`, `X-Compatibility-Date`, and `X-Pages`.
Use ETag validators first, respect endpoint TTLs, and do not poll at the limiter edge. A 304 is the expected cache-saving path where validators are honored.
## SSO: web login, PKCE, JWT, and refresh
### Discovery and endpoints
Fetch and cache the OAuth metadata document at [`https://login.eveonline.com/.well-known/oauth-authorization-server`](https://login.eveonline.com/.well-known/oauth-authorization-server). On the check date it returned:
```json
{
"issuer": "https://login.eveonline.com",
"authorization_endpoint": "https://login.eveonline.com/v2/oauth/authorize",
"token_endpoint": "https://login.eveonline.com/v2/oauth/token",
"userinfo_endpoint": "https://login.eveonline.com/v2/oauth/verify",
"jwks_uri": "https://login.eveonline.com/oauth/jwks",
"revocation_endpoint": "https://login.eveonline.com/v2/oauth/revoke",
"code_challenge_methods_supported": ["S256"],
"token_endpoint_auth_methods_supported": ["client_secret_basic", "client_secret_post", "client_secret_jwt"]
}
```
Use metadata URLs rather than hard-coding them; the current SSO page explicitly recommends this. The current official deprecation notice says the old `oauth/*` endpoints are obsolete: use `v2/oauth/authorize` and `v2/oauth/token`, send a form-encoded body, and do not send JSON or token parameters in the query string. [SSO docs](https://developers.eveonline.com/docs/services/sso/) · [SSO endpoint deprecations](https://developers.eveonline.com/blog/sso-endpoint-deprecations-2)
### Authorization Code (server/web)
1. Register the app and exact callback URL in the EVE Developers Portal. Generate a random `state` and retain it in the login session.
2. Redirect to `authorization_endpoint` with URL-encoded parameters:
```text
response_type=code
client_id=<client id>
redirect_uri=<registered exact URL>
scope=<space-separated requested scopes>
state=<random CSRF value>
```
3. Verify callback `state`, then POST form-urlencoded to `token_endpoint` with `grant_type=authorization_code`, `code=<one-time-code>`, and the registered `redirect_uri`, using HTTP Basic credentials `client_id:client_secret`. The current strict-conformance notice shows `redirect_uri` in the token body; the current SSO page's simplified Python example omits it, so send the registered value and test against the registered app.
4. The response contains an access token and refresh token. Keep the refresh token server-side; access tokens are time-limited and character/scope-bound.
### Authorization Code + PKCE (native/public client)
- Generate a high-entropy `code_verifier`; send `code_challenge = base64url(SHA-256(code_verifier))` without `=` padding, with `code_challenge_method=S256`.
- Add `code_challenge` and `code_challenge_method=S256` to the authorization query above. Do not use a client secret in this flow.
- Exchange the callback code with a form-urlencoded POST body containing `grant_type=authorization_code`, `code`, `client_id`, `code_verifier`, and the registered `redirect_uri`. Retain the verifier until exchange; verify `state` exactly as in the web flow. The strict-conformance notice is the reason to include `redirect_uri`; its newer SSO-page sample omits that field.
The current ESI OpenAPI OAuth2 security scheme also declares the same `v2` authorize/token URLs and all ESI scopes. For this app's chosen data, the primary scopes are `esi-assets.read_assets.v1`; add `esi-location.read_location.v1` for current docking, `esi-location.read_ship_type.v1` for current ship/capital type ID, and `esi-universe.read_structures.v1` for authenticated structure resolution. [Current OpenAPI](https://esi.evetech.net/meta/openapi.json)
### Refresh
The retained official refresh-token page documents the exact current v2 request shape: web apps POST form-urlencoded `grant_type=refresh_token&refresh_token=<token>` with Basic `client_id:secret`; native/public apps send `grant_type=refresh_token`, `refresh_token`, and `client_id` without Basic. Optional `scope` may narrow to a subset of the original grant. URL-encode the token. The response shape is:
```json
{
"access_token": "<jwt>",
"token_type": "Bearer",
"expires_in": 1200,
"refresh_token": "<possibly-new-token>"
}
```
The current SSO page describes refresh tokens as long-lived while access is not revoked, and the official deprecation notice says refresh-token rotation may change the returned token: always replace the stored refresh token with the returned one. [Official refresh contract (retained page)](https://docs.esi.evetech.net/docs/sso/refreshing_access_tokens.html) · [Rotation notice](https://developers.eveonline.com/blog/sso-endpoint-deprecations-2)
### JWT verification and local user/character identity
The access token is a JWT. Fetch `jwks_uri` from metadata, select the key by JWT header `kid`/algorithm, verify the signature with a standard JWT library, validate expiration, issuer, and audience, and only then trust claims. The current SSO docs specify:
- `iss` should be the EVE issuer (`https://login.eveonline.com/` or, in some cases, `login.eveonline.com`; use metadata and accept only the documented EVE forms).
- `aud` is an array containing the app `client_id` and static `EVE Online`; check both.
- `sub` is `CHARACTER:EVE:<character-id>`; `name` is the character name; `scp` enumerates granted scopes. The official guide describes an array, but singleton scope claims can also be scalar strings. Normalize one valid scope string into a one-element array only after JWT verification; do not infer permissions from missing or malformed claims.
Runtime compatibility follow-up (2026-09-19): a live callback reached verified token processing but failed the original array-only scope parser. The maintained [`eveauth` implementation](https://github.com/ErikKalkoken/eveauth/blob/main/jose.go) explicitly handles both scalar strings and arrays in `extractScopes`, with a [singleton regression test](https://github.com/ErikKalkoken/eveauth/blob/main/jose_test.go). Our signed-token regression reproduced the scalar failure and now verifies both login and refresh normalization. Invalid/missing claims and whitespace-delimited ambiguity remain rejected; all signature, issuer, audience, owner/session and exact scope-membership checks remain in force.
The current JWKS observed `RS256`, but the official deprecation notice warns signature methods can change; do not hard-code the observed key/algorithm. A local app user should be an application identity with a child record per `character_id`, token, refresh token, scopes, and last refresh. SSO selects one character per authorization, so linking another character is another flow. [JWT guidance](https://developers.eveonline.com/docs/services/sso/#validating-jwt-tokens)
## Upwell structures, Access Lists, and docking evidence
### General structure routes
| Route | Auth/scope | Current contract and cache |
|---|---|---|
| `GET /universe/structures?filter=market\|manufacturing_basic` | Public | Unique array of public structure int64 IDs; `filter` only selects structures with that service online; TTL 3600. |
| `GET /universe/structures/{structure_id}` | `esi-universe.read_structures.v1` | 200 returns `name`, `solar_system_id`, `owner_id`, optional `type_id`/position; default error includes 403 when not on ACL; TTL 3600. |
| `GET /corporations/{corporation_id}/structures?page=N` | `esi-corporations.read_structures.v1`; current OpenAPI role `Station_Manager` | Paged array with structure ID/type/corporation/system/profile, name, services, state, fuel/reinforcement fields; `X-Pages`; TTL 3600; `corp-structure` 300/15m. |
| `GET /markets/structures/{structure_id}?page=N` | `esi-markets.structure_markets.v1` | Paged order array (`X-Pages`); TTL 300; no claim that market visibility means docking access. |
The current OpenAPI explicitly describes `/universe/structures/{id}` as returning information only if the caller is “on the ACL”, otherwise “Forbidden”. The retained official structure scenario is more explicit: `/universe/structures` lists only **fully public** structures (an ACL that prevents even one character from docking excludes it), while the authenticated ID route requires a token for a character with docking access. Treat the scenario as corroborating operational guidance because its page warns that it is no longer updated; the pinned OpenAPI is the current field/scope source.
### 2026-05-19 routes
These routes are present in the current (2026-08-18) OpenAPI and have route compatibility date `2026-05-19`:
| Route | Scope | Shape/cache/limit |
|---|---|---|
| `GET /characters/{character_id}/access-lists` | `esi-access.read_lists.v1` | `{access_lists:[{id:int64}]}`; character must manage/admin an ACL; event cache 300; `char-access` 600/15m. |
| `GET /characters/{character_id}/access-lists/{access_list_id}` | same | `{id,name,description,membership}`; event cache 300; same limiter. |
| `GET /characters/{character_id}/structures/mercenary-dens` and `/{mercenary_den_id}` | `esi-structures.read_character.v1` | Character structure data; event cache 3600; `char-structure` 30/15m. |
| `GET /characters/{character_id}/mercenary-tactical-operations` and `/{operation_id}` | `esi-activities.read_character.v1` | MTO data; event cache 300; `char-activity` 150/15m. |
| `GET /corporations/{corporation_id}/structures/skyhooks` and `/{skyhook_id}` | `esi-structures.read_corporation.v1` | Listing wrapper `{skyhooks:[...]}`; detail includes id/planet/state/activity/reagents/timers; event cache 3600; `corp-structure` 300/15m; current OpenAPI says `Station_Manager`. |
| `GET /corporations/{corporation_id}/structures/sovereignty-hubs` and `/{sovereignty_hub_id}` | same | Listing wrapper `{sovereignty_hubs:[...]}`; detail includes id/system/upgrades/reagent bay/resources/transport; event cache 3600; same limiter; detail declares `Station_Manager`. |
| `GET /skyhooks/raidable` | Public | Upcoming/current raidable Skyhooks; TTL cache 300/client, server TTL 60; `activity` 30/15m. |
| `GET /sovereignty/systems` | Public | Combined K-space sovereignty details; TTL 300; `sovereignty` 600/15m. |
### What proves what
1. **Public visibility/discovery only:** an ID in `/universe/structures` proves the structure is in ESI's fully-public list (and, with `filter`, that the selected service is online). It does not enumerate private/ACL structures and does not prove this particular character can dock.
2. **Character-scoped ACL/access evidence:** a successful 200 from `/universe/structures/{structure_id}` using that character's token is the direct ESI structure-access check; a non-authorized caller receives the default error/403. Current OpenAPI says “on the ACL”; the older scenario interprets this as docking access. Preserve that evidence with its timestamp and confidence, but do not promote metadata visibility into an unconditional `canDock` boolean. Ship class, structure state, subsequent ACL changes and cache age still matter, especially for capitals.
3. **Observed docking:** `/characters/{character_id}/location` requires `esi-location.read_location.v1`, is event-based with a 5-second cache, and returns required `solar_system_id` plus optional `station_id` or `structure_id`. A matching `structure_id` is evidence of docking at the observation time, not guaranteed present/future docking permission for a different ship. A mere ACL membership or a 200 on structure metadata is not a docking observation.
4. **ACL membership:** the Access List detail route returns `allow_everyone` plus character/corporation/alliance entries whose access is `Unspecified`, `Allowed`, `Blocked`, `Manager`, or `Admin`. It does not return `structure_id`, structure profile, docking service, or a computed per-character permission. Therefore membership alone cannot prove docking.
5. **Corporation ownership/management:** the corporation-structures route proves the corporation's structures and exposes `profile_id`/operational state to an appropriately authorized corporation character. It does not prove that an arbitrary linked character can dock.
## Capital/intel-specific limits
- `GET /characters/{character_id}/ship` (`esi-location.read_ship_type.v1`) returns only `ship_type_id`, `ship_item_id`, and `ship_name`; it is 5-second TTL-cached and uses `char-location` 1200/15m. Assets likewise expose `type_id`, not a `capital` boolean.
- Capital/supercapital classification therefore requires joining `ship_type_id`/asset `type_id` to official static data; ESI itself does not provide that classification. The official static-data page documents SDE as the static game data source and JSON Lines/YAML formats; no archive was downloaded for this research. [Static Data](https://developers.eveonline.com/docs/services/static-data/)
- A ship `type_id` plus a market price is not proof of current location or docking: use the location route for current presence and structure metadata/ACL check for access.
## Confidence, blockers, and implementation handoff
**High confidence:** current asset/market schemas, scopes, page headers, cache extensions, 2026-08-18 supported compatibility value, 2026-05-19 route additions, SSO endpoints/PKCE parameters, JWT claims, and live rate-header names; all were taken from the pinned OpenAPI, current EVE docs, or live official metadata.
**Medium confidence:** translating a successful structure metadata request into “docking access” relies on the retained official scenario page; current OpenAPI says “on the ACL” and 403 otherwise, but does not name the ACL permission as `docking`.
**Blockers/decisions:**
- There is no current ESI route that joins an Access List to a structure profile or computes a character's docking permission. Do not infer that relationship from Access List membership.
- Asset pages can change while being fetched; use validators/TTL and de-duplicate by `item_id`. Asset `location_id` and `type_id` need separate name/static-data joins.
- `/markets/prices` is a global cached reference snapshot with optional fields, not a region/structure order-book valuation. Define missing-price, blueprint, fitted-ship, and stale-price policy before showing totals.
- Refresh tokens can rotate. Persist the replacement returned by every successful refresh, and store credentials per linked character.
- The current SSO page's simplified token examples omit `redirect_uri` at exchange, while the official strict-conformance notice includes the registered `redirect_uri`; use the latter and verify in a registered test app before launch.
## Sources kept
- [ESI overview/versioning](https://developers.eveonline.com/docs/services/esi/overview/) — compatibility-date and API contract rules.
- [Current ESI OpenAPI](https://esi.evetech.net/meta/openapi.json) — exact route, schema, scope, cache, pagination, and limiter extensions; captured with `X-Compatibility-Date: 2026-08-18`.
- [Compatibility dates](https://esi.evetech.net/meta/compatibility-dates) and [changelog](https://esi.evetech.net/meta/changelog) — live supported-date and 2026 route-release evidence.
- [ESI rate limiting](https://developers.eveonline.com/docs/services/esi/rate-limiting/) and [best practices/error limit](https://developers.eveonline.com/docs/services/esi/best-practices/) — new and legacy headers/costs.
- [X-Pages pagination](https://developers.eveonline.com/docs/services/esi/pagination/x-pages/) — page and cache-expiry behavior.
- [Current SSO docs](https://developers.eveonline.com/docs/services/sso/), [OAuth metadata](https://login.eveonline.com/.well-known/oauth-authorization-server), and [SSO deprecation/rotation notice](https://developers.eveonline.com/blog/sso-endpoint-deprecations-2) — auth code, PKCE, JWT, v2 endpoints, and refresh rotation.
- [Equinox ESI structures/access lists](https://developers.eveonline.com/blog/equinox-on-esi-structures-sovereignty-and-access-lists) — current 2026 structure/access-list release context.
- [Retained structure scenario](https://docs.esi.evetech.net/docs/scenarios/resolve_structure_ids.html) and [retained refresh contract](https://docs.esi.evetech.net/docs/sso/refreshing_access_tokens.html) — exact docking interpretation and refresh body; both pages warn that the old docs site is no longer updated, so current OpenAPI/current SSO docs take precedence.
- [Official static-data docs](https://developers.eveonline.com/docs/services/static-data/) — capital/type classification dependency; no archive downloaded.
docs/research/expansion-data-contracts.md
+84
View File
@@ -0,0 +1,84 @@
# Data contracts for the remaining-feature roadmap
**Research date:** 2026-09-19 UTC. **Compatibility pin:** `X-Compatibility-Date: 2026-08-18`.
**Scope:** official public documentation/OpenAPI research for planning. No private ESI requests, credentials, user data, archives, or service changes were used. These adapters are not implemented merely because their contracts are listed here.
Primary source: [official ESI OpenAPI](https://esi.evetech.net/meta/openapi.json), requested with the compatibility header above. Additional sources: [versioning](https://developers.eveonline.com/docs/services/esi/overview/), [pagination](https://developers.eveonline.com/docs/services/esi/pagination/x-pages/), [rate limiting](https://developers.eveonline.com/docs/services/esi/rate-limiting/), [best practices](https://developers.eveonline.com/docs/services/esi/best-practices/), and [SSO](https://developers.eveonline.com/docs/services/sso/).
Existing captured evidence in `eve-openapi-evidence.json` covers the first milestone's assets/helpers/structure topics, **not** all expansion routes below. Capture focused, versioned fixtures for each new adapter before implementation. Recheck the pinned contract at that point; advancing the compatibility date is a separate deliberate change.
## Personal-character datasets
All methods in this table are GET. Paths use the trusted `https://esi.evetech.net` base.
| Dataset | Route | OAuth scope | Pagination / historical coverage | Documented cache |
|---|---|---|---|---|
| Wallet balance | `/characters/{character_id}/wallet` | `esi-wallet.read_character_wallet.v1` | Single numeric balance | 120 s |
| Wallet journal | `/characters/{character_id}/wallet/journal` | Same wallet scope | `page` + `X-Pages`; past **30 days** | 3600 s |
| Wallet transactions | `/characters/{character_id}/wallet/transactions` | Same wallet scope | `from_id`, records before the given ID; retention/page size not specified | 3600 s |
| Open orders | `/characters/{character_id}/orders` | `esi-markets.read_character_orders.v1` | Unpaged current orders; includes `is_corporation` | 1200 s |
| Order history | `/characters/{character_id}/orders/history` | Same order scope | `page` + `X-Pages`; cancelled/expired orders up to **90 days** | 3600 s |
| Owned blueprints | `/characters/{character_id}/blueprints` | `esi-characters.read_blueprints.v1` | `page` + `X-Pages`; item/type/location, ME/TE and runs | 3600 s |
| Skills | `/characters/{character_id}/skills` | `esi-skills.read_skills.v1` | Unpaged; documented possibility of stale levels after training completion | 60 s, event-based |
| Skill queue | `/characters/{character_id}/skillqueue` | `esi-skills.read_skillqueue.v1` | Unpaged; needed to reconcile completed training with stale skill responses | 60 s, event-based |
| Industry jobs | `/characters/{character_id}/industry/jobs` | `esi-industry.read_character_jobs.v1` | Unpaged; `include_completed`; completed jobs only **90 days** | 300 s |
Planning consequences:
- Personal wallets share one scope, but balance/journal/transaction freshness and collection strategy differ.
- Do not treat `from_id` as a page number or invent a fixed transaction page size, cursor depth, or retention horizon.
- ESI retention is not application retention. Data may be retained after collection under an approved policy, but missing earlier history cannot be reconstructed by assumption.
- Do not assume order history describes every possible fill or disappearance; reconcile documented state/quantities with transaction evidence and preserve uncertainty.
- Exclude or explicitly partition `is_corporation` orders from personal financial totals. A character endpoint is not proof that every returned economic position is personally owned. Check any similar personal/corporation flags in transaction schemas during adapter implementation.
- Skills and skill queue are separate grants. Reconcile according to the documented completion behavior and timestamps; future queue entries are not trained skills, and conflicting/stale evidence must not manufacture eligibility.
- Owned blueprint instances are distinct from immutable recipe types and from the current asset-copy flag.
## Public and authenticated market feeds
| Dataset | GET route | Access | Pagination / cache |
|---|---|---|---|
| Active region types | `/markets/{region_id}/types` | Public | `page` + `X-Pages`; 600 s |
| Region orders | `/markets/{region_id}/orders` | Public; required `order_type` (`buy`, `sell`, or `all`), optional `type_id` | `page` + `X-Pages`; 300 s |
| Region history | `/markets/{region_id}/history` | Public; required `type_id` | Unpaged daily statistics; daily expiry at 11:05; retention horizon not specified |
| Structure orders | `/markets/structures/{structure_id}` | `esi-markets.structure_markets.v1` and authorized character context | `page` + `X-Pages`; 300 s |
Regional history contains date, volume, order count and high/average/low statistics. It is not a fill forecast, order book, or proof of executable liquidity. Structure-market access is not docking permission and must not become an unrestricted shared private-data cache.
The existing `/markets/prices` provider remains a different, global reference-price dataset; see [the initial ESI research](eve.md#reference-price-valuation).
## Later corporation datasets
Role declarations below come from the pinned contract, not a live corporation-role test. Verify effective role semantics, division restrictions, membership and application-workspace permissions at implementation time. A corporation scope alone is insufficient.
| Dataset | GET route | OAuth scope | Declared role requirement | Pagination / cache / history |
|---|---|---|---|---|
| Assets | `/corporations/{corporation_id}/assets` | `esi-assets.read_corporation_assets.v1` | `Director` | `X-Pages`; 3600 s |
| Wallet balances | `/corporations/{corporation_id}/wallets` | `esi-wallet.read_corporation_wallets.v1` | `Accountant` or `Junior_Accountant` | Division/balance array; 300 s |
| Wallet journal | `/corporations/{corporation_id}/wallets/{division}/journal` | Same wallet scope | Same wallet roles | `X-Pages`; 30 days; 3600 s |
| Wallet transactions | `/corporations/{corporation_id}/wallets/{division}/transactions` | Same wallet scope | Same wallet roles | `from_id`; retention unspecified; 3600 s |
| Open orders | `/corporations/{corporation_id}/orders` | `esi-markets.read_corporation_orders.v1` | `Accountant` or `Trader` | `X-Pages`; 1200 s |
| Order history | `/corporations/{corporation_id}/orders/history` | Same order scope | Same order roles | `X-Pages`; 90 days; 3600 s |
| Blueprints | `/corporations/{corporation_id}/blueprints` | `esi-corporations.read_blueprints.v1` | `Director` | `X-Pages`; 3600 s |
| Industry jobs | `/corporations/{corporation_id}/industry/jobs` | `esi-industry.read_corporation_jobs.v1` | `Factory_Manager` | `X-Pages`; completed jobs 90 days; 300 s |
No corporation-skills equivalent was verified: skills remain character-scoped. Corporation industry-job records must not be confused with the application's own `jobs` orchestration table.
## Scheduling and consistency implications
The pinned extensions identify `char-wallet` as 150 tokens/15 minutes, `char-industry` as 600/15 minutes, `char-detail` as 600/15 minutes, and regional orders as `market-order` 12,000/15 minutes. These are **tokens**, not necessarily one token per request. Confirm route-specific headers and token costs at implementation time.
Character order routes have no route-specific limiter extension in the inspected contract; that does not mean unlimited access. Respect the legacy error limiter and response caching. TTLs are scheduling hints bounded by actual response validators/cache headers, not promises that upstream content is fresh to the second.
X-Pages guidance warns that cache expiry between pages can duplicate records. Do not assume one asset-specific pagination algorithm establishes stable snapshots for every route. Preserve source IDs, idempotent checkpoints, partial/backfill status and coverage windows. Prices, journals, assets, skills and jobs do not form one atomic EVE-wide transaction.
## Concrete code integration points
- `src/lib/server/sso.ts`: authorization currently requests only assets; add server-allowlisted capability scope upgrades and state binding.
- `src/lib/server/esi/index.ts`: only assets and global reference prices currently have public client methods; implement separate typed adapters/strategies for these new routes.
- `src/lib/domain/model.ts`: add dataset/finance/owned-blueprint/skill/job contracts; existing `BlueprintRepository`/`OrderBook` are narrow placeholders.
- `src/convex/schema.ts`: add bounded records, generations/checkpoints and indexes. Its current generic `jobs` placeholder is not ESI industry-job data.
- `src/lib/server/application.ts`: existing synchronization orchestrates assets only; new datasets belong in the durable job/capability design.
## Unresolved and explicitly not promised
Wallet-transaction retention, market-history retention, exact page sizes, maximum cursor depth, complete consistency guarantees across pages/endpoints, real corporation-role behavior, fee formulas, industry economics and dynamic docking authority are not established by this research. Verify those separately; do not infer them from endpoint availability.
docs/research/jev.md
+163
View File
@@ -0,0 +1,163 @@
# TypeSafe AI / JEV integration research
**Checked:** 2026-09-18
**Scope:** current official TypeSafe AI/Jev documentation only. Research was gathered with Firecrawl CLI, using sequential scrapes of the official documentation index/full corpus and JavaScript SDK page (global scrape concurrency was 2). No implementation was added.
## Summary
Use JEV as a small, typed intent router in front of the existing asset workflows—not as a chat model, valuation calculator, search engine, authenticator, or function executor. The recommended first call is one JavaScript SDK `Choice` question with exactly `asset_valuation`, `asset_search`, and `unsupported` options. Treat low `Choice.confidence` as an application-level abstention and route to a deterministic clarification/manual path; JEV has no native abstain answer.
For an asset-valuation result, code should call the reference-data/valuation service and perform any arithmetic. For an asset-search result, code should query the asset index. For `unsupported`, code should return a fixed product response. Keep authentication, authorization, tool execution, response text, and all side effects in application code.
## Findings
1. **System One is bounded decision infrastructure, not an agent or chat model.** TypeSafe says System One evaluates a `state` and returns typed answers and probabilities; it does not write replies, generate code, explain reasoning, or choose its own next action. The intended architecture is ordinary software control flow with narrow AI decisions embedded in it. [System One](https://docs.typesafe.ai/concepts/system-one) · [How to build with System One](https://docs.typesafe.ai/concepts/how-to-build-with-system-one)
Implications for this integration:
- **Not generative chat:** use deterministic/canned UI responses or another separately approved component for user-facing text; do not ask JEV to compose the clarification or unsupported message.
- **Not calculation:** do not ask JEV to compute a reference valuation, normalize currency, or perform exact arithmetic. Keep those operations in code. TypeSafe explicitly recommends using code for work it can compute exactly. [Advanced structure/failure modes](https://docs.typesafe.ai/primitives/advanced)
- **Not authentication or authorization:** the SDK API key authenticates calls to TypeSafe; it does not authenticate the end user or authorize an asset action. Perform application identity, entitlements, and permissions before/after routing in code.
- **Not execution:** the function-calling cookbook maps closed-set values to ordinary typed functions, but the functions remain application code. [Function calling cookbook](https://docs.typesafe.ai/cookbooks/function_calling)
2. **State is shared, questions are independent.** A request evaluates one `state` against one or more questions. State may be a string or structured JSON object/array containing the text and relevant application context; JEV currently supports text input and does not support image, audio, or video. Every question sees the same state and one question's answer does not become context for another. [State](https://docs.typesafe.ai/concepts/state) · [API request body](https://docs.typesafe.ai/api#request-body)
For the router, send only the user request plus any text/JSON metadata required to disambiguate it, for example `{ message, asset_context }`. Do not send unrelated records. Current docs warn that context rot lowers accuracy and recommend decomposing both state and questions.
3. **The primitives have distinct, typed semantics.**
- **Choice:** selects exactly one member of a fixed criteria map and returns `choice`, all option `probabilities`, and `confidence`. It is the correct primitive for the three-way router. Include an explicit `unsupported` option rather than forcing every request into a supported route; TypeSafe recommends an `other`/`none of the above` option when a list may not cover every input. A Choice supports up to 255 options. [Choice](https://docs.typesafe.ai/primitives/choice)
- **Noul:** returns `noul`, the probability that a yes/no statement is true. It has no separate `confidence`; values near 0.5 indicate an ambiguous yes/no judgment. Use it for a separate boolean such as “does the request mention a specific asset identifier?”, not as a replacement for the three-way router. [Noul](https://docs.typesafe.ai/primitives/noul)
- **Score:** returns an expected numeric position over an ordered list of 2–10 descriptive levels, the level probabilities, a legend, and confidence. It is appropriate for a dimension such as evidence quality or urgency, not for exact valuation arithmetic. [Score](https://docs.typesafe.ai/primitives/score)
- **Confidence:** is derived from the Choice/Score probability distribution. It is a useful uncertainty signal, not a correctness guarantee. Full probabilities should be retained for auditing and policy decisions. Noul answers do not carry `confidence`. [Confidence](https://docs.typesafe.ai/confidence)
4. **Exact recommended router labels and boundaries.** Use one `Choice` whose criteria are deliberately contrastive:
- `asset_valuation`: the user asks to estimate, compare, look up, or understand the reference worth/valuation of an identified asset or asset candidate. The downstream valuation implementation owns data freshness, methodology, and arithmetic.
- `asset_search`: the user asks to find, browse, list, filter, or retrieve assets or reference records; valuation is not the primary requested operation.
- `unsupported`: the request is neither supported asset browsing nor reference valuation (including unrelated chat, unsupported transactions, or unrelated domains).
Avoid overlapping descriptions such as “anything involving prices” in both supported options. In particular, “find assets priced below X” is primarily `asset_search`; “what is this asset worth?” is `asset_valuation`. A low-confidence result is not a fourth JEV label: it is an application-level abstention.
5. **Confirmed JavaScript package and API.** The official JavaScript SDK docs specify Node.js 20+, package `@typesafe-ai/sdk`, ESM/CommonJS support, and TypeScript declarations. The current SDK reference exposes `VERSION` `0.6.0` and the changelog records v0.6.0 on 2026-09-15. [JavaScript SDK](https://docs.typesafe.ai/sdk/javascript) · [JavaScript changelog](https://docs.typesafe.ai/sdk/javascript/changelog)
Confirmed API/type surface:
- `new TypeSafeClient(config?)`; `TypeSafeClientConfig` includes `apiKey`, `baseURL`, `defaultModel`, `defaultHeaders`, `fetch`, `logger`, `logLevel`, `retry`, `timeout`, and `dangerouslyAllowBrowser`.
- `client.systemOne<Q>(request, options?)` returns `APIPromise<SystemOneResult<Q>>`.
- `systemOne` request fields are `state: EntryType`, `questions: Q`, and optional `model: string`. `EntryType` is a string, JSON object, JSON array, or `null`.
- `choice<T>(instructions, criteria)` returns `ChoiceQuestion<T>`; `noul(instructions?, criteria?)` returns `NoulQuestion`; `score<T>(instructions, criteria)` returns `ScoreQuestion<T>`.
- `SystemOneResult<Q>.answers` maps each question key to `ResultFor<Q[K]>`; `ResultFor` preserves the criteria keys. `ChoiceResponse<T>.choice` is typed as `keyof T & string`, with typed `probabilities`, `confidence`, and `type: "choice"`.
- Environment defaults are `TYPESAFE_API_KEY`, `TYPESAFE_BASE_URL`, `TYPESAFE_DEFAULT_MODEL`, and `TYPESAFE_LOG_LEVEL`; default base URL is `https://api.typesafe.ai`, default model is `jev-latest`, default log level is `warn`, and default per-attempt timeout is 10,000 ms. [TypeSafeClient](https://docs.typesafe.ai/sdk/javascript/api/classes/TypeSafeClient) · [choice()](https://docs.typesafe.ai/sdk/javascript/api/functions/choice) · [SystemOneRequest](https://docs.typesafe.ai/sdk/javascript/api/interfaces/SystemOneRequest) · [SystemOneResult](https://docs.typesafe.ai/sdk/javascript/api/interfaces/SystemOneResult) · [ResultFor](https://docs.typesafe.ai/sdk/javascript/api/type-aliases/ResultFor) · [ChoiceResponse](https://docs.typesafe.ai/sdk/javascript/api/interfaces/ChoiceResponse) · [TypeSafeClientConfig](https://docs.typesafe.ai/sdk/javascript/api/interfaces/TypeSafeClientConfig)
6. **Exact typed-router call (SDK-confirmed).** This is the integration shape to implement later; the snippet uses the confirmed package and methods, not an inferred SDK name:
```ts
import { choice, TypeSafeClient } from "@typesafe-ai/sdk";
const client = new TypeSafeClient({
// Pin while calibrating thresholds; jev-latest is the SDK default.
defaultModel: "jev-1.13.0",
});
const result = await client.systemOne({
state: { message: userMessage },
questions: {
intent: choice("What is the user's primary supported intent?", {
asset_valuation:
"The user wants a reference valuation or wants to understand the worth of an identified asset.",
asset_search:
"The user wants to find, browse, list, filter, or retrieve assets or reference records; valuation is not the primary request.",
unsupported:
"The request is outside asset browsing and reference valuation, including unrelated chat or unsupported actions.",
}),
},
});
const answer = result.answers.intent;
// answer.choice is inferred as the criteria-key union:
// "asset_valuation" | "asset_search" | "unsupported".
// answer.probabilities and answer.confidence are also available.
```
The current model page says `jev-latest` currently resolves to `jev-1.13.0`. Use the pinned ID while calibrating the abstention policy; use the alias only when accepting model movement is intentional. The response `model` field reports the version that actually answered. [Models](https://docs.typesafe.ai/models)
7. **Recommended dispatch and abstention policy.** JEV should return a classification, after which code owns the route:
- if `answer.confidence < CONFIDENCE_FLOOR`, return an application-level `abstain` result and use a deterministic clarification/manual-review flow;
- otherwise, route `asset_valuation` to the reference valuation service, `asset_search` to the asset index, and `unsupported` to a fixed unsupported response.
Start with a configurable `CONFIDENCE_FLOOR = 0.80` for this medium/high-consequence router, then calibrate it against labeled requests and the cost of each routing error. This is an application policy, not a TypeSafe default. The official guidance gives examples of 0.5/0.6 floors for human routing and >0.85/>0.9 for higher-risk actions, and says thresholds must match risk and be tested on the application's data. [Confidence](https://docs.typesafe.ai/confidence) · [Confidence-gated routing](https://docs.typesafe.ai/patterns/confidence-routing) · [Intent routing](https://docs.typesafe.ai/patterns/intent-routing)
Recommended local result shape (not a JEV response shape): `abstain | { route: "asset_valuation" | "asset_search" | "unsupported", confidence, probabilities, model }`. Do not pretend a low-confidence `choice` is a reliable route. Preserve the raw answer, probability map, model version, usage, and request ID for evaluation.
8. **There is no native abstention field.** A Choice always selects one supplied option. Therefore:
- `unsupported` is the explicit semantic escape hatch for out-of-scope input;
- low confidence is the uncertainty escape hatch and must be converted to `abstain` by application code;
- if a future requirement needs a separate “in-scope but missing details” class, add a fourth criteria key or use the same low-confidence abstention path; do not infer that behavior from a missing SDK field.
This is preferable to using an independent Noul for each route: separate Nouls are absolute yes/no judgments and are not guaranteed to be mutually exclusive or comparable. A Choice is the relative one-of-N classifier needed here. [Choice](https://docs.typesafe.ai/primitives/choice) · [Noul](https://docs.typesafe.ai/primitives/noul) · [Confidence](https://docs.typesafe.ai/confidence)
9. **Transport and SDK failure behavior is distinct from semantic abstention.** The HTTP API documents `POST https://api.typesafe.ai/v1/systemone`, bearer authentication, and these error meanings: 401 missing/invalid key, 422 invalid request body, 429 rate limit, and 529 temporary overload. [API reference](https://docs.typesafe.ai/api)
The JavaScript SDK exposes:
- `AuthenticationError` (401), `BadRequestError` (400), `PermissionDeniedError` (403), `NotFoundError` (404), `RateLimitError` (429), `UnprocessableEntityError` (422), and `InternalServerError` (5xx), all carrying the parsed `body`, headers, status, and usually `requestId` through `APIError`.
- `APIConnectionError` for connection/response delivery failures, `APITimeoutError` for a response that exceeds the configured timeout, and `APIUserAbortError` for an `AbortSignal` cancellation.
- Default retry policy: connection errors and timeouts are retryable; HTTP statuses 408, 429, and 500–599 are retried, up to 2 retries after the initial attempt, with exponential backoff starting at 500 ms, capped at 5,000 ms, 25% jitter, and `Retry-After` honored up to 60,000 ms. Per-call `timeout`, `retry`, `headers`, and `signal` are supported via `RequestOptions`. [APIError](https://docs.typesafe.ai/sdk/javascript/api/classes/APIError) · [RateLimitError](https://docs.typesafe.ai/sdk/javascript/api/classes/RateLimitError) · [APITimeoutError](https://docs.typesafe.ai/sdk/javascript/api/classes/APITimeoutError) · [RequestOptions](https://docs.typesafe.ai/sdk/javascript/api/interfaces/RequestOptions) · [RetryPolicy](https://docs.typesafe.ai/sdk/javascript/api/interfaces/RetryPolicy)
Handle 401/403/422/configuration failures as non-retryable integration failures; let the SDK handle transient 408/429/5xx/connection/timeout retries, then fail closed to an application fallback. A transport error is not an `unsupported` classification and must not be converted into one. Never expose the API key in a browser: the SDK defaults `dangerouslyAllowBrowser` to false.
10. **Fan-out is the right composition strategy for independent router features.** TypeSafe recommends sending independent questions about one state together: they are evaluated in parallel, and code may ignore answers that do not apply. If a later question genuinely depends on the first answer or requires newly fetched state, make a second request in code. [Speculative fan-out](https://docs.typesafe.ai/patterns/fan-out) · [Parallel questions cookbook](https://docs.typesafe.ai/cookbooks/parallel_questions)
For the first implementation, keep the request to the single intent Choice. If later needed, add independent questions in the same request—for example a Noul for whether an asset identifier is explicitly present—but do not add an extraction/calculation question merely because it is available. The parallel-questions cookbook reports that batching 13 questions over one document avoids resending the document and, in its dated example, was 12.2× cheaper and 10.0× faster than 13 serial calls with no answer change. The exact benchmark is workload-specific, but the architectural lesson applies.
11. **Composite scoring belongs after classification and remains deterministic.** Break a broad judgment into independent Score dimensions, normalize each score to 0–1, and combine with weights in application code. Preserve each component and its probabilities rather than asking JEV for one opaque “overall valuation.” [Composite scoring](https://docs.typesafe.ai/patterns/composite-scoring) · [Score](https://docs.typesafe.ai/primitives/score)
This can support a later reference-ranking policy (for example evidence quality, recency, and match quality), but it must not replace the authoritative valuation formula. Scores are judgments over descriptive levels; they are not exact financial numbers.
12. **Function-calling cookbook: closed sets only, with code in control.** The official example maps a function selector to a Choice, fixed-list arguments to Choice, `bool` flags to Noul, and set-valued `list[Literal]` arguments to one Noul per member. Optional arguments use a separate “was this argument stated?” Noul so the function default can remain in effect. The cookbook reports call confidence as the least certain judgment rather than a product of probabilities. [Function calling](https://docs.typesafe.ai/cookbooks/function_calling)
This pattern is useful later for deterministic asset-search/valuation handlers, but it is not a JEV tool-execution API. Do not pass open-ended numeric calculations, authorization decisions, or arbitrary generated function arguments to JEV. Validate every route and argument in application code.
13. **Reranking is a candidate-scoring pattern, not an asset search replacement.** The official reranking cookbook first uses a fast search method (BM25) to produce a shortlist, then scores each query/candidate pair with a Noul and sorts by the returned probability. TypeSafe cannot add a candidate omitted by the fast search stage. The dated example reranked 30 candidates for each of 40 legal queries and improved top-1 from 5% to 18% and top-10 from 38% to 62%. [Reranking cookbook](https://docs.typesafe.ai/cookbooks/rerank_typesafe)
If asset search eventually needs semantic ordering, keep the asset index/retrieval system first, then use one Noul per query-candidate pair (or batch independent signals per pair) and sort in code. Do not invoke JEV against the entire asset corpus.
14. **Extraction should be candidate-first and lossless.** The official pre-parsed extraction cookbook uses regex to over-find candidate emails, phone numbers, or amounts; a Choice selects one candidate verbatim; code copies and normalizes it. It adds a `none` Choice option as an escape hatch and notes the 255-option limit. It explicitly says JEV should choose among candidates, not invent an unbounded value or transpose digits. [Pre-parsed value extraction](https://docs.typesafe.ai/cookbooks/pre_parsed_value_extraction_cookbook)
Apply this to asset IDs/tickers only when the candidate finder is owned by code or another approved extractor: provide the candidate strings as Choice options, include `none`, copy the selected identifier exactly, and validate it against the asset index. Do not ask JEV to generate a new identifier or exact valuation.
15. **Routing cookbook evidence supports graceful degradation.** The official classification-with-confidence cookbook classifies into a fine-grained Choice taxonomy, then reports a broader parent label when confidence is below a threshold rather than forcing a specific answer. The intent-routing pattern similarly sends low-confidence requests to a human and routes high-confidence labels to deterministic handlers or specialist systems. [Classification using confidence](https://docs.typesafe.ai/cookbooks/classification_using_confidence) · [Intent routing](https://docs.typesafe.ai/patterns/intent-routing)
For this scope, the equivalent broader fallback is the local `abstain` route, which can request required asset details through a fixed form or send to manual review. There is no official evidence that JEV itself should generate the clarification text.
## Exact integration recommendation
1. Run the router server-side with `@typesafe-ai/sdk` on Node.js 20+.
2. Pin `model: "jev-1.13.0"` (or `defaultModel`) while measuring; log the returned `model`. Move to `jev-latest` only with an explicit threshold-recalibration policy.
3. Send a minimal state containing the incoming user text and only relevant text/JSON context.
4. Ask one `choice()` question with the three contrastive keys `asset_valuation`, `asset_search`, and `unsupported`.
5. Convert `confidence < 0.80` to a local typed `abstain`; validate/recalibrate this floor with a labeled evaluation set. Keep all probabilities, not only the winner.
6. Dispatch high-confidence results in ordinary code: valuation service, asset index, or fixed unsupported response. Keep user auth, authorization, exact calculations, data fetching, and side effects outside JEV.
7. Treat SDK/API failures as failures, not as semantic `unsupported`. Retry only according to the SDK policy; fail closed after retries and retain the request ID/error body for diagnosis.
8. Add fan-out questions only when independent and useful. Add extraction/reranking/composite scoring as separate, deterministic stages—not inside the initial router.
## Gaps and caveats
- **No native abstain result:** the SDK/API always returns a Choice label when the request succeeds. Local `abstain` is required.
- **No official asset-domain rubric or benchmark:** current docs provide generic routing examples, not accuracy for asset browsing/reference valuation. Build a labeled test set containing near-boundary requests, missing asset details, price-search wording, unsupported requests, and adversarial/ambiguous text.
- **Threshold calibration is application-specific:** official docs give guidance and examples but no universal confidence threshold. Validate the proposed 0.80 floor and separately measure false valuation routing versus false search routing.
- **No clarification-label requirement in the chosen scope:** the three requested labels do not include `clarification`. Low-confidence in-domain requests will abstain; if product analytics need a distinct typed `needs_details` outcome, add it deliberately rather than overloading `unsupported`.
- **No JEV-generated response or valuation:** current docs do not provide a chat-generation method, calculator, finance data source, valuation methodology, or authorization system. Those are integration responsibilities.
- **Current SDK documentation is precise, but cookbook code is mostly Python:** the official JavaScript API confirms the client and types, while the requested cookbooks show Python examples. Porting cookbook architecture to TypeScript will require local implementation and tests; do not infer undocumented helper classes from Python cookbook names.
- **Alias movement:** `jev-latest` can point to a future release; thresholds tuned to a pinned model must be retested after any model change.
## Sources
All kept sources below are official TypeSafe AI documentation and were current at the checked date.
- [System One](https://docs.typesafe.ai/concepts/system-one) — model purpose and distinction from LLM agents/generative output.
- [How to build with System One](https://docs.typesafe.ai/concepts/how-to-build-with-system-one) — code-controlled architecture, decomposition, fan-out, confidence gates.
- [State](https://docs.typesafe.ai/concepts/state) — supported state shapes and text-only Jev input.
- [Choice](https://docs.typesafe.ai/primitives/choice), [Noul](https://docs.typesafe.ai/primitives/noul), [Score](https://docs.typesafe.ai/primitives/score), [Confidence](https://docs.typesafe.ai/confidence) — primitive and uncertainty semantics.
- [API reference](https://docs.typesafe.ai/api) and [Models](https://docs.typesafe.ai/models) — HTTP shape, errors, current model ID/alias.
- [JavaScript SDK](https://docs.typesafe.ai/sdk/javascript), [JavaScript changelog](https://docs.typesafe.ai/sdk/javascript/changelog), [TypeSafeClient](https://docs.typesafe.ai/sdk/javascript/api/classes/TypeSafeClient), [choice()](https://docs.typesafe.ai/sdk/javascript/api/functions/choice), [SystemOneRequest](https://docs.typesafe.ai/sdk/javascript/api/interfaces/SystemOneRequest), [SystemOneResult](https://docs.typesafe.ai/sdk/javascript/api/interfaces/SystemOneResult), [ChoiceResponse](https://docs.typesafe.ai/sdk/javascript/api/interfaces/ChoiceResponse), [ResultFor](https://docs.typesafe.ai/sdk/javascript/api/type-aliases/ResultFor) — confirmed package/API/types.
- [Speculative fan-out](https://docs.typesafe.ai/patterns/fan-out), [Composite scoring](https://docs.typesafe.ai/patterns/composite-scoring), [Confidence-gated routing](https://docs.typesafe.ai/patterns/confidence-routing), [Intent routing](https://docs.typesafe.ai/patterns/intent-routing) — required architecture patterns.
- [Function calling](https://docs.typesafe.ai/cookbooks/function_calling), [Parallel questions](https://docs.typesafe.ai/cookbooks/parallel_questions), [Reranking](https://docs.typesafe.ai/cookbooks/rerank_typesafe), [Pre-parsed value extraction](https://docs.typesafe.ai/cookbooks/pre_parsed_value_extraction_cookbook), [Classification using confidence](https://docs.typesafe.ai/cookbooks/classification_using_confidence) — required cookbook coverage.
**Dropped by scope:** third-party articles, reviews, hosted-provider documentation, social posts, and non-official examples were not used as evidence.
docs/research/platform.md
+20
View File
@@ -0,0 +1,20 @@
# Platform evidence
Checked 2026-09-18 against live official documentation. Raw working copies are in ignored `.firecrawl/`.
## Convex
- [Self hosting](https://docs.convex.dev/self-hosting) and [official deployment guide](https://github.com/get-convex/convex-backend/blob/main/self-hosted/README.md): backend defaults to SQLite; no PostgreSQL/Redis is necessary. Backend API uses 3210, HTTP actions 3211, dashboard 6791. Persistent `/convex/data` is essential. Generate a deployment admin key with `./generate_admin_key.sh`; CLI uses `CONVEX_SELF_HOSTED_URL` and `CONVEX_SELF_HOSTED_ADMIN_KEY`, not cloud project login. Admin key is deployment-only, never a browser environment value.
- [Official Compose](https://github.com/get-convex/convex-backend/blob/main/self-hosted/docker/docker-compose.yml): health probe `/version`; backend SIGINT shutdown; distinct cloud/site origins. Upstream example uses `latest`; operators should pin tested image digests and back up before upgrades.
- [Svelte quickstart](https://docs.convex.dev/quickstart/svelte), [Svelte overview](https://docs.convex.dev/client/svelte/overview): use the actual `convex-svelte` package, Svelte 5; put Convex functions in `src/convex` via `convex.json`. Root `setupConvex(url)`; declarative `useQuery(api.function, () => args)` for subscriptions. Keep domain calculations in Bun server services, not separate browser/Convex copies.
- [Custom JWT](https://docs.convex.dev/auth/advanced/custom-jwt): supports RS256/ES256, `kid/alg/typ`, `sub/iss/exp` and client refresh `iat`. Enforce application audience. JWKS may be a data URI, avoiding a boot-time callback from Convex to the web container. This permits an EVE-login-backed application identity without handing EVE tokens to the browser. An application JWT is NOT an EVE access token.
## Bun
- [SQLite](https://bun.sh/docs/runtime/sqlite): synchronous `bun:sqlite` driver; parameterized prepared statements, transactions, read-only connections, strict bindings, integrity pragmas. SQLite integers are 64-bit but JS numbers only exact through 2^53−1: external EVE instance/location IDs should be decimal strings at domain boundaries. Reference type/system IDs fit safe integers and are validated.
- Transactions must contain synchronous writes, not awaited parsing/downloads. Immutable artifact compiler can use DELETE journaling to avoid promoting a DB without a WAL sidecar. Complete/close staging file before publishing its manifest. Serving uses read-only handles to a version, not an in-place changing database.
- Local available runtime: Bun 1.3.14. Live docs advertise a newer release; use only APIs verified by repository tests on 1.3.14 and explicitly pin the tested runtime in deployment.
## Evidence caveat
Documentation evolves. URLs above are live sources, not promises of permanent API compatibility. Dependency lockfile, ESI contract evidence and offline regression tests capture the implemented contract. Deployment and live-auth testing need separately configured services/credentials.
docs/research/production-import.md
+260
View File
@@ -0,0 +1,260 @@
# Production reference-data import plan
**Checked:** 2026-09-18
**Scope:** full real-data import is now authorized, but this phase performed **no archive download**. Only release indexes, HTTP `HEAD` metadata, official documentation/source, and small metadata/API files were inspected. Credentials and Infisical were not touched.
## Decision summary
1. Download the current Reference Data and Hoboleaks archives, then parse their exporter-defined root members. Their current compressed sizes are 14,512,812 bytes and 675,868 bytes.
2. Use CCP's current JSON Lines SDE for the universe complement, specifically `mapSolarSystems.jsonl`. CCP's current build is `3503375` (released 2026-09-10), and the official map guide directly documents reading `_key` and `position.x/y/z` from that member.
3. Do **not** use the EVE Ref ESI-scrape archive as the production universe source: its latest published artifact is dated 2025-11-26, while the CCP SDE is current to 2026-09-10. The ESI scrape is still useful as a shape/reference comparison, not as the freshness authority.
4. The selected compressed download budget is **114,291,615 bytes (~108.997 MiB)** before temporary extraction/database space: Reference Data + Hoboleaks + CCP JSONL SDE.
## 1. Current release metadata (preflight; no archive bytes read)
The values below came from the public JSON indexes and final archive `HEAD` responses. ETags are opaque HTTP validators, not published SHA-256 checksums.
| Input | Immutable/current URL | Bytes | Approx. compressed size | Last-Modified | ETag / version |
| --- | --- | ---: | ---: | --- | --- |
| EVE Ref Reference Data | [`reference-data-latest.tar.xz`](https://data.everef.net/reference-data/reference-data-latest.tar.xz) (dated route: [`reference-data-2026-09-13.tar.xz`](https://data.everef.net/reference-data/history/2026/reference-data-2026-09-13.tar.xz)) | 14,512,812 | 13.84 MiB | 2026-09-13 11:16:26Z | `a4b2dd0325ffac28158f2b93500a0d52` |
| EVE Ref Hoboleaks mirror | [`hoboleaks-sde-latest.tar.xz`](https://data.everef.net/hoboleaks-sde/hoboleaks-sde-latest.tar.xz) (dated route: [`hoboleaks-sde-2026-09-13.tar.xz`](https://data.everef.net/hoboleaks-sde/history/2026/hoboleaks-sde-2026-09-13.tar.xz)) | 675,868 | 660.03 KiB | 2026-09-13 11:05:28Z | `5681550f6fe5f2927b055dfe33883655` |
| EVE Ref ESI scrape (not recommended for production systems) | [`eve-ref-esi-scrape-latest.tar.xz`](https://data.everef.net/esi-scrape/eve-ref-esi-scrape-latest.tar.xz) | 77,038,676 | 73.47 MiB | 2025-11-26 15:42:24Z | `08fe439c74509e521620f588a13cfb37-10` |
| CCP SDE JSONL, build 3503375 | [`eve-online-static-data-3503375-jsonl.zip`](https://developers.eveonline.com/static-data/tranquility/eve-online-static-data-3503375-jsonl.zip) | 99,102,935 | 94.51 MiB | 2026-09-10 11:21:36Z | `fce5dcb01dbe357402ace2848297f556-12` |
The CCP automation endpoint currently returns:
```json
{"_key":"sde","buildNumber":3503375,"releaseDate":"2026-09-10T11:09:04Z"}
```
The EVE Ref archive indexes are [`reference-data/index.json`](https://data.everef.net/reference-data/index.json), [`hoboleaks-sde/index.json`](https://data.everef.net/hoboleaks-sde/index.json), and [`esi-scrape/index.json`](https://data.everef.net/esi-scrape/index.json). CCP's official automation documentation says the latest alias redirects to the build-numbered URL and that all resources support ETag/Last-Modified. Record both the alias and the resolved immutable URL in the import manifest.
EVE Ref's running [`/meta`](https://ref-data.everef.net/meta) currently reports build time `2026-09-13T11:10:03Z` and source hashes:
```json
{
"build_time": "2026-09-13T11:10:03Z",
"esi": {"sha256": "645b2902174fe52561b946a5c6e39e257358803956238f0750798e1cde5e4077"},
"hoboleaks": {"sha256": "477799f37c9275c3ad408117cce8e7913688be41b17fe404cd48e3b501be55cd"},
"sde": {"sha256": "56f5f7eba4b401aeb2fc1090d2db5ab9dd55cc08a396871e764df45017fa70af"}
}
```
These are hashes of EVE Ref's three **input archives**, not checksums of `reference-data-latest.tar.xz`. Current EVE Ref source downloads the YAML SDE input (`SDE_V2_YAML`) when building Reference Data, so `/meta.sde.sha256` must not be presented as the hash of the separately selected CCP JSONL SDE complement.
## 2. Reference Data archive: exporter-defined member layout
The current EVE Ref exporter source is the strongest available archive contract:
- [`BuildRefData.java` at commit 17948f63](https://github.com/autonomouslogic/eve-ref/blob/17948f63ad35b3417ad888a72f6d4fcc49e2b72d/src/main/java/com/autonomouslogic/everef/cli/refdata/BuildRefData.java#L335-L390) writes `meta.json`, then one `.json` member for every entry in `refdata.yaml`, into a TAR and XZ-compresses it.
- [`DataUtil.java`](https://github.com/autonomouslogic/eve-ref/blob/75b17f51e8880b474e44236049b50198a58d0332/src/main/java/com/autonomouslogic/everef/util/DataUtil.java#L1-L85) shows that the builder's SDE input is the configured `SDE_V2_YAML` ZIP, while the separate production universe complement below deliberately uses CCP's current JSONL ZIP.
- [`refdata.yaml` at commit ac663020](https://github.com/autonomouslogic/eve-ref/blob/ac6630205f41404c7635f9efcfd8d6646e45370f/src/main/resources/refdata.yaml) defines the member basenames, source filenames, ID fields, and output-store relationships.
- [`RefDataUtil.java`](https://github.com/autonomouslogic/eve-ref/blob/17948f63ad35b3417ad888a72f6d4fcc49e2b72d/src/main/java/com/autonomouslogic/everef/util/RefDataUtil.java#L103-L136) confirms that `meta.json` is a standalone object and every other JSON member is parsed as an object whose property names are decimal IDs. Its config loader defaults absent `outputFile` to lower-snake-case (`marketGroups` → `market_groups`) and absent `outputStore` to the config ID ([source](https://github.com/autonomouslogic/eve-ref/blob/17948f63ad35b3417ad888a72f6d4fcc49e2b72d/src/main/java/com/autonomouslogic/everef/util/RefDataUtil.java#L177-L200)).
### Exact member names
The exporter emits these 19 root members in config order:
```text
meta.json
categories.json
groups.json
market_groups.json
types.json
type_materials.json
type_bonus.json
masteries.json
type_dogma.json
dogma_attributes.json
dogma_effects.json
meta_groups.json
mutaplasmids.json
skills.json
units.json
blueprints.json
icons.json
regions.json
schematics.json
```
`meta.json` is a `RefDataMeta` object containing `build_time` and `sha256` objects for `sde`, `esi`, and `hoboleaks`. Every other member is written as:
```json
{
"<decimal-id>": {"<normalized record fields>": "..."}
}
```
The writer serializes the map key as a JSON string and the value as the normalized EVE Ref record. It does not write one file per ID inside the archive. A loader should preserve the string key as the canonical ID while validating that it parses as a non-negative 64-bit integer.
### Member/source map
| Member | Config ID / output store | Input members named by the exporter | Import role |
| --- | --- | --- | --- |
| `categories.json` | `categories` / `categories` | SDE `categories.yaml`; ESI `universe/categories` | Canonical category map. |
| `groups.json` | `groups` / `groups` | SDE `groups.yaml`; ESI `universe/groups` | Canonical group map. |
| `market_groups.json` | `marketGroups` / `marketGroups` | SDE `marketGroups.yaml`; ESI `markets/groups` | Canonical market-group map. |
| `types.json` | `types` / `types` | SDE `types.yaml`; ESI `universe/types` | Canonical inventory-type map. |
| `type_materials.json` | `typeMaterials` / `types` | SDE `typeMaterials.yaml`; Hoboleaks `typematerials.json` | Supplemental type merge input; values belong in `types`. |
| `type_bonus.json` | `typeBonus` / `types` | SDE `typeBonus.yaml` | Supplemental type merge input; values belong in `types`. |
| `masteries.json` | `masteries` / `types` | SDE `masteries.yaml` | Supplemental type merge input; values belong in `types`. |
| `type_dogma.json` | `typeDogma` / `types` | SDE `typeDogma.yaml` | Supplemental type merge input; values belong in `types`. |
| `dogma_attributes.json` | `dogmaAttributes` / `dogmaAttributes` | SDE `dogmaAttributes.yaml`; ESI `dogma/attributes` | Canonical dogma-attribute map. |
| `dogma_effects.json` | `dogmaEffects` / `dogmaEffects` | SDE `dogmaEffects.yaml`; ESI `dogma/effects` | Canonical dogma-effect map. |
| `meta_groups.json` | `metaGroups` / `metaGroups` | SDE `metaGroups.yaml` | Canonical meta-group map. |
| `mutaplasmids.json` | `mutaplasmids` / `mutaplasmids` | Hoboleaks `dynamicitemattributes.json` | Canonical mutaplasmid map. |
| `skills.json` | `skills` / `skills` | Derived by EVE Ref decorators | Canonical skill map. |
| `units.json` | `units` / `units` | Hoboleaks `dogmaunits.json` | Canonical unit map. |
| `blueprints.json` | `blueprints` / `blueprints` | SDE `blueprints.yaml`; Hoboleaks `blueprints.json` | Canonical blueprint map. |
| `icons.json` | `icons` / `icons` | SDE `icons.yaml` | Canonical icon map. |
| `regions.json` | `regions` / `regions` | SDE `mapRegions.yaml`; ESI `universe/regions` | Canonical region map. |
| `schematics.json` | `schematics` / `schematics` | SDE `planetSchematics.yaml`; ESI `universe/schematics` | Canonical schematic map. |
The four type supplemental configs have `dedicatedOutput: false` and `outputStore: types` in the current config. The exporter still iterates every config and names a member from `config.outputFile`; therefore, treat `types.json` as the canonical merged type table, and treat those four sidecar members as exporter outputs whose content must be checked after download rather than assumed to be separate canonical tables. Do not invent records from a sidecar if the authoritative merged `types.json` does not contain them.
### Reference Data parser invariants
- Read TAR entries sequentially through XZ decompression; there is no need to materialize an uncompressed archive.
- Require `meta.json` exactly once.
- For each other `.json` member, require a top-level JSON object; every property key must be a decimal ID; reject duplicate IDs and malformed values.
- Skip no member silently except an explicitly documented non-JSON compatibility member. The current EVE Ref parser skips non-JSON entries but the exporter contract above emits JSON only.
- Preserve source/build metadata separately from the local archive URL, byte count, ETag, Last-Modified, and locally computed SHA-256.
## 3. Hoboleaks archive: exact root member layout
The current archive builder is [`ScrapeHoboleaks.java` at commit 17948f63](https://github.com/autonomouslogic/eve-ref/blob/17948f63ad35b3417ad888a72f6d4fcc49e2b72d/src/main/java/com/autonomouslogic/everef/cli/ScrapeHoboleaks.java#L87-L143): it downloads Hoboleaks `meta.json`, writes that as root `meta.json`, iterates `meta.files`, downloads each named file from the Hoboleaks root, and writes that file at the archive root with the same filename. There are no nested `tq/` directories inside the EVE Ref mirror archive.
The current [`tq/meta.json`](https://sde.hoboleaks.space/tq/meta.json) has revision `3503375`, timestamp `2026-09-13T10:41:34`, and 28 file entries. Thus the expected current archive has **29 root members**:
```text
meta.json
accountingentrytypes.json
agenttypes.json
attributeorders.json
clonestates.json
compressibletypes.json
dbuffs.json
dogmaeffectcategories.json
dogmaunits.json
dynamicitemattributes.json
expertsystems.json
graphicmaterialsets.json
industryactivities.json
industryassemblylines.json
industryinstallationtypes.json
industrymodifiersources.json
industrytargetfilters.json
localization_dgmattributes.json
localization_languages.json
repackagedvolumes.json
schoolmap.json
schools.json
skillplans.json
skinmaterialnames.json
skinmaterials.json
skins.json
stationstandingsrestrictions.json
typematerials.json
blueprints.json
```
The list is metadata-driven, not a hard-coded schema promise: if a future `meta.json` adds/removes a file, the archive member set changes with it. The source files retain Hoboleaks casing and shape (for example, `activityID`/`activityName`, not normalized `activity_id`/`activity_name`). The prior [reference contract](https://github.com/autonomouslogic/eve-ref/blob/main/docs/src/datasets/reference-data.md) records small current examples for `industryactivities.json`, `agenttypes.json`, and `repackagedvolumes.json`.
Hoboleaks metadata also carries per-file `md5`, `revision`, `stale`, and `deprecated`. A file can be retrievable but stale. Record this as source state; do not convert it to a transport-successful fresh input.
## 4. Universe XYZ source choice
### Comparison
| Source | Current release | Member/path shape | Freshness and scale | Decision |
| --- | --- | --- | --- | --- |
| EVE Ref ESI scrape | `eve-ref-esi-scrape-latest.tar.xz`, 77,038,676 bytes, last modified 2025-11-26 | EVE Ref's source table names the file `universe/systems.yaml`; ESI universe records use `system_id`, `name`, `position: {x,y,z}`, and related fields | One complete scrape archive, but almost ten months behind the current SDE and not the freshness authority | Do not select for production universe data. |
| CCP SDE JSONL | build `3503375`, 99,102,935 bytes, released 2026-09-10 | Root member `mapSolarSystems.jsonl`; each line has `_key` (solar-system ID) and object fields such as `position.x/y/z` | Current official SDE; JSONL is explicitly documented as streamable and the map guide covers all space types | **Select this source.** |
| ESI live per-system routes | Current API; current ID list observed at 8,490 systems | `/universe/systems/{system_id}` response; one request per system | Fresh but ~8,490 API requests, rate/cache policy, and no single immutable snapshot | Use only for spot checks, not bulk import. |
The official CCP [Map Data guide](https://developers.eveonline.com/docs/guides/map-data/) says map data is in SDE files prefixed `map`, explicitly names `mapSolarSystems`, and its executable example opens `./mapSolarSystems.jsonl`, parses each line, reads `system["_key"]`, and reads `system["position"]["x"]`, `["y"]`, and `["z"]`. It also says SDE/ESI include known, wormhole, and abyssal space. The current [`schema-changelog.yaml`](https://developers.eveonline.com/static-data/tranquility/schema-changelog.yaml) records `mapSolarSystems` as a current file and records `position2D` as added after build `3096543`.
The official [System Security guide](https://developers.eveonline.com/docs/guides/system-security/) independently identifies `mapSolarSystems.jsonl` as the SDE source and `securityStatus` as a full-precision floating-point field. The official [Static Data docs](https://developers.eveonline.com/docs/services/static-data/) define JSONL handling: an integer-keyed object record is represented as one line with `_key`; object values remain object fields (the map guide's `system["position"]` access is the concrete example).
### Minimal exact SDE system line contract
Do not claim an entire system schema from the small guide example. The safe import contract is the fields directly documented by the official examples:
```json
{
"_key": 30000142,
"position": {
"x": -129064861735000000.0,
"y": 60755306910000000.0,
"z": 117469227060000000.0
},
"securityStatus": 0.9459131360054016
}
```
The real line has additional fields. Preserve unknown fields or explicitly ignore them only after validating the fields needed by the application. Use `_key` as `system_id`; do not expect the ESI `system_id` field in this SDE member. Keep XYZ as 64-bit floating-point values; the official guide warns that 32-bit precision is insufficient for interstellar and planetary coordinates.
A current ESI list request returned 8,490 IDs (checked 2026-09-18), confirming that a production profile limited to synthetic fixtures or a handful of systems is not sufficient. The CCP SDE map file is the efficient immutable complement: one current archive download rather than thousands of ESI calls.
## 5. Download and streaming plan (after this phase)
### Preflight
1. Fetch only the small JSON indexes, CCP `latest.jsonl`, Hoboleaks `tq/meta.json`, and `HEAD` metadata as done here.
2. Resolve each `latest` alias and record the final URL, status, content length, ETag, Last-Modified, media type, source revision/build, and retrieval time.
3. Require the expected sizes above for this checked release, but treat a changed release as a new candidate rather than a fatal global condition. Refresh metadata and use the new immutable URL.
4. Never use an ETag as a SHA-256. On the actual download, stream bytes through a local SHA-256 digest and compare byte count with `Content-Length`.
### Selected transfers
| Transfer | Compressed bytes |
| --- | ---: |
| Reference Data | 14,512,812 |
| Hoboleaks SDE | 675,868 |
| CCP SDE JSONL | 99,102,935 |
| **Total** | **114,291,615 (~108.997 MiB)** |
The ESI-scrape alternative would total 92,227,356 bytes with Reference Data + Hoboleaks, but it is stale and is not selected. Do not download both universe complements unless a deliberate cross-check is needed.
### Safe extraction order
1. Create a unique staging directory and `.part` file per source. Stream each HTTP body to disk while hashing and counting; do not place bytes in a JS/Bun `Buffer` for the whole archive.
2. Close and verify the staged file before opening it. Keep the immutable staged artifact until the candidate DB and manifest validate.
3. For Reference Data and Hoboleaks, stream XZ → TAR sequentially. Parse each selected root member as it arrives; close the TAR/XZ reader on any error. These archives are small enough to retain on disk but not to trust as an in-memory blob.
4. For CCP SDE JSONL, keep the completed ZIP staged, inspect the ZIP central directory, and stream only `mapSolarSystems.jsonl` through the JSON Lines parser. Do not extract the whole ~99 MB ZIP merely to build the universe complement. Keep the ZIP hash in provenance even if only one member is consumed.
5. Parse JSONL line-by-line: reject malformed lines, require integer `_key`, require finite numeric `position.x/y/z`, reject duplicate IDs, and count systems. Preserve the full source line or unknown fields if later features may need them.
6. Build a new immutable SQLite candidate, validate all cross-source references and the real universe count (expected to be >5,000), close it, then atomically publish its manifest. Keep the previous manifest/DB untouched on any source or validation failure.
### Independent source status
Track three source outcomes independently:
- `reference_data`: HTTP/download/hash/archive/member validation;
- `hoboleaks`: HTTP/download/hash/archive/member validation plus per-file `stale`/`deprecated` state;
- `ccp_sde_jsonl`: HTTP/download/hash/ZIP member/JSONL validation and build number.
A successful Reference Data archive must not mask a failed SDE universe complement. A stale Hoboleaks file is a distinct policy state from a failed download. Promotion should require the source set selected by the production profile to be successful and policy-acceptable, while retaining the previous published candidate for rollback.
## Gaps and non-claims
- The archives were not downloaded in this phase, so the emitted member list was not independently enumerated from the TAR/ZIP central directories. Reference Data and Hoboleaks member paths are derived from the current EVE Ref exporter source and metadata-driven builder, which is stronger than guessing but remains subject to exporter changes.
- CCP's official docs document the `mapSolarSystems.jsonl` path and fields used by the map example, but do not publish a complete machine-readable schema for every field on that page. The import should not require unlisted optional fields.
- The EVE Ref ESI-scrape archive's internal `universe/systems.yaml` path is established by the current EVE Ref data-source table; its current artifact is stale for a production freshness requirement.
- No credentials, Infisical data, or application secrets were accessed.
## Sources
- [EVE Ref Reference Data docs](https://docs.everef.net/datasets/reference-data) — source combination, archive/API routes, data-source table, and normalization.
- [EVE Ref Reference Data archive index](https://data.everef.net/reference-data/index.json) and [Hoboleaks archive index](https://data.everef.net/hoboleaks-sde/index.json) — current sizes, timestamps, ETags, and URLs.
- [EVE Ref ESI scrape index](https://data.everef.net/esi-scrape/index.json) — current published ESI-scrape size and age.
- [EVE Ref `BuildRefData.java`](https://github.com/autonomouslogic/eve-ref/blob/17948f63ad35b3417ad888a72f6d4fcc49e2b72d/src/main/java/com/autonomouslogic/everef/cli/refdata/BuildRefData.java) — exact Reference Data TAR member writer and `meta.json` behavior.
- [EVE Ref `RefDataUtil.java`](https://github.com/autonomouslogic/eve-ref/blob/17948f63ad35b3417ad888a72f6d4fcc49e2b72d/src/main/java/com/autonomouslogic/everef/util/RefDataUtil.java) — archive parser expectations.
- [EVE Ref `refdata.yaml`](https://github.com/autonomouslogic/eve-ref/blob/ac6630205f41404c7635f9efcfd8d6646e45370f/src/main/resources/refdata.yaml) — current member basenames, source paths, and output stores.
- [EVE Ref `ScrapeHoboleaks.java`](https://github.com/autonomouslogic/eve-ref/blob/17948f63ad35b3417ad888a72f6d4fcc49e2b72d/src/main/java/com/autonomouslogic/everef/cli/ScrapeHoboleaks.java) — metadata-driven root member creation.
- [Hoboleaks `tq/meta.json`](https://sde.hoboleaks.space/tq/meta.json) — current revision, file list, MD5, stale/deprecated state.
- [CCP Static Data docs](https://developers.eveonline.com/docs/services/static-data/) — JSONL format, automation routes, build metadata, HTTP caching.
- [CCP current build metadata](https://developers.eveonline.com/static-data/tranquility/latest.jsonl) and [build 3503375 JSONL release](https://developers.eveonline.com/static-data/tranquility/eve-online-static-data-3503375-jsonl.zip) — current build and release size.
- [CCP Map Data guide](https://developers.eveonline.com/docs/guides/map-data/) — `mapSolarSystems.jsonl`, `_key`, XYZ fields, all-space coverage, and coordinate precision.
- [CCP System Security guide](https://developers.eveonline.com/docs/guides/system-security/) — SDE system source and `securityStatus` field.
- [CCP schema changelog](https://developers.eveonline.com/static-data/tranquility/schema-changelog.yaml) — current `mapSolarSystems` file and `position2D` history.
- [Official ESI OpenAPI](https://esi.evetech.net/meta/openapi.json) and [ESI system ID list](https://esi.evetech.net/latest/universe/systems/) — ESI comparison shape and current system-count sanity check.
docs/research/reference.md
+318
View File
@@ -0,0 +1,318 @@
# EVE Ref reference-data and Hoboleaks source contract
**Checked:** 2026-09-18
**Scope:** current EVE Ref documentation/OpenAPI and the two current `data.everef.net` archive release routes named below. No full archive was downloaded. Firecrawl CLI was used for the documentation and representative API pages, sequentially; current machine-readable indexes, metadata, and small Hoboleaks JSON files were fetched directly.
## Summary
EVE Ref combines CCP SDE, an ESI scrape, and Hoboleaks into Reference Data. The Reference Data contract is explicitly still in development and its OpenAPI description is unstable, so the compiler must validate the incoming schema rather than assume that an omitted optional field is present. The current archive artifacts are published through directory `index.json` routes, while the running Reference Data API exposes independent input provenance at `/meta`.
The latest observed artifacts are:
| Artifact | Release route | Bytes | Last modified | ETag (opaque HTTP validator) |
| --- | --- | ---: | --- | --- |
| Reference Data | [`reference-data/index.json`](https://data.everef.net/reference-data/index.json) / [`reference-data-latest.tar.xz`](https://data.everef.net/reference-data/reference-data-latest.tar.xz) | 14,512,812 | 2026-09-13 11:16:26.504Z | `a4b2dd0325ffac28158f2b93500a0d52` |
| Hoboleaks SDE | [`hoboleaks-sde/index.json`](https://data.everef.net/hoboleaks-sde/index.json) / [`hoboleaks-sde-latest.tar.xz`](https://data.everef.net/hoboleaks-sde/hoboleaks-sde-latest.tar.xz) | 675,868 | 2026-09-13 11:05:28.403Z | `5681550f6fe5f2927b055dfe33883655` |
The dated, immutable-looking release routes are listed by [`reference-data/history/index.json`](https://data.everef.net/reference-data/history/index.json) and [`hoboleaks-sde/history/index.json`](https://data.everef.net/hoboleaks-sde/history/index.json). Treat the ETag as an HTTP cache validator, not as a published SHA-256 checksum.
## 1. EVE Ref data model and endpoint contract
The [Reference Data documentation](https://docs.everef.net/datasets/reference-data) says the set merges the latest SDE, ESI scrape, and Hoboleaks export; the complete download is linked from that page. It also states the current normalization rules:
- fields and URLs use `snake_case`;
- the JSON layout is mostly ESI-shaped;
- keyed objects are preferred over arrays to make merging deterministic;
- names and descriptions are language maps, as in the SDE.
The [pinned Reference Data OpenAPI document](https://github.com/autonomouslogic/eve-ref/blob/58dafd29ea55ad78fd62cbf820460cb79eb66202/spec/reference-data.yaml) (commit `58dafd29`, checked 2026-09-18) declares `openapi: 3.0.1`, server `https://ref-data.everef.net`, and says the spec is unstable. The examples below use the documented schema plus the live example endpoints linked by the docs. OpenAPI properties are not marked required for these records; a fixture may therefore be a valid minimal object, but a parser must not turn missing optional fields into invented defaults.
### Categories
`InventoryCategory` has these documented properties:
```text
category_id: int64
name: map<string, string>
published: boolean
icon_id: int64
group_ids: array<int64> # added by EVE Ref
```
A current small representative fixture is the live [category 4 response](https://ref-data.everef.net/categories/4), with the join-bearing fields retained:
```json
{
"category_id": 4,
"name": {"en": "Material"},
"published": true,
"icon_id": 22,
"group_ids": [18]
}
```
The live response currently contains additional language-map entries and additional group IDs; the fixture above is a deliberately small projection, not a claim that the production response contains only one group.
### Groups
`InventoryGroup` has these documented properties:
```text
anchorable: boolean
anchored: boolean
category_id: int64
fittable_non_singleton: boolean
group_id: int64
icon_id: int64
name: map<string, string>
published: boolean
type_ids: array<int64> # added by EVE Ref
use_base_price: boolean
```
A current [group 18 response](https://ref-data.everef.net/groups/18) can be reduced for a join fixture to:
```json
{
"category_id": 4,
"group_id": 18,
"name": {"en": "Mineral"},
"published": true,
"anchorable": false,
"anchored": false,
"fittable_non_singleton": false,
"icon_id": 22,
"use_base_price": true,
"type_ids": [34]
}
```
The production response has more language entries and more type IDs. Preserve the integer IDs as IDs; do not infer them from names.
### Types
`InventoryType` is intentionally broad. Its documented properties include the identity/join fields `type_id`, `group_id`, and EVE Ref-added `category_id`; language maps `name` and `description`; booleans such as `published`, `is_blueprint`, `is_skill`, `is_ore`, and `is_repackable`; numeric physical/economic fields such as `volume`, `packaged_volume`, `portion_size`, `mass`, `capacity`, `radius`, and `base_price`; and keyed maps including `dogma_attributes`, `dogma_effects`, `type_materials`, `required_skills`, and `produced_by_blueprints`.
A minimal current [type 34 response](https://ref-data.everef.net/types/34) sufficient to exercise category/group/type joins is:
```json
{
"type_id": 34,
"name": {"en": "Tritanium"},
"group_id": 18,
"category_id": 4,
"published": true,
"portion_size": 1,
"volume": 0.01,
"packaged_volume": 0.01,
"mass": 0,
"base_price": 2.0
}
```
This is a projection of the current response; fields not needed by this fixture are omitted, not declared unsupported. In particular, do not require every optional `InventoryType` property to exist on every record.
### Blueprints, activities, materials, and products
The OpenAPI schemas are exact about the important nesting:
```text
Blueprint {
activities: map<string, BlueprintActivity>
blueprint_type_id: int64
max_production_limit: int32
}
BlueprintActivity {
materials: map<string, BlueprintMaterial>
products: map<string, BlueprintMaterial>
required_skills: map<string, int32>
time: int64
}
BlueprintMaterial {
probability: double # optional; present for probabilistic products
quantity: int64
type_id: int64
}
```
The activity/material/product maps are keyed objects, not arrays. The key is a serialized ID for the material/product maps and an activity name for `activities`; the schema does not publish a closed activity enum. A current [blueprint 683 response](https://ref-data.everef.net/blueprints/683) demonstrates both deterministic manufacturing and probabilistic invention. A tiny exact manufacturing slice for a fixture is:
```json
{
"blueprint_type_id": 683,
"activities": {
"manufacturing": {
"materials": {
"34": {"quantity": 24000, "type_id": 34}
},
"products": {
"582": {"quantity": 1, "type_id": 582}
},
"required_skills": {"3380": 1},
"time": 6000
}
},
"max_production_limit": 30
}
```
The live object also has other activities and manufacturing materials; the slice above retains only the fields needed to test keyed material/product lookup and a required-skill map. Do not flatten these maps into an array without preserving their IDs.
### Systems and XYZ: explicit scope boundary
The EVE Ref data-source table lists systems under SDE `mapSolarSystems.yaml` and ESI `universe/systems.yaml`; it does **not** list a Reference Data `/systems` endpoint. The Reference Data OpenAPI has `Coordinate {x, y, z}` and uses it for `Region.position`, but it does not define a solar-system schema. Therefore, a systems fixture must not be presented as a Reference Data record.
For a compiler that also ingests the documented ESI system source, the official ESI schema requires this shape:
```json
{
"system_id": 30000142,
"name": "Jita",
"constellation_id": 20000020,
"position": {
"x": -129064861735000000.0,
"y": 60755306910000000.0,
"z": 117469227060000000.0
},
"security_status": 0.9459131360054016
}
```
The `position` object is the requested XYZ fixture. `system_id`, `name`, `position`, `security_status`, and `constellation_id` are required by the current ESI OpenAPI; planets, star, gates, stations, and security class are optional. The [EVE Ref source table](https://docs.everef.net/datasets/reference-data#data-sources) is the authority for the fact that systems come from those SDE/ESI paths; the [official ESI machine-readable schema](https://esi.evetech.net/meta/openapi.json) is the authority for the exact ESI record shape. This distinction avoids inventing a missing Reference Data `/systems` schema.
## 2. Hoboleaks SDE contract
The EVE Ref [Hoboleaks SDE page](https://docs.everef.net/datasets/hoboleaks-sde.html) identifies [`data.everef.net/hoboleaks-sde/`](https://data.everef.net/hoboleaks-sde/) as the archive mirror for files published by [Hoboleaks SDE](https://sde.hoboleaks.space/). The [Hoboleaks landing page](https://sde.hoboleaks.space/) says these are JSON complements for data missing from the official SDE and/or ESI, and warns that they are updated after TQ patches and can temporarily lag the SDE.
There is no formal JSON Schema linked for every Hoboleaks file. The following are exact current small-file structures observed on 2026-09-18; preserve their source casing and top-level map shape when ingesting raw Hoboleaks. EVE Ref's normalized API shape is separate and uses `snake_case`.
### Hoboleaks `tq/meta.json`: per-file provenance and stale state
The landing page explicitly documents `tq/meta.json` as update metadata, including a timestamp, revision, per-file MD5, and a `stale` flag. A current excerpt is:
```json
{
"timestamp": "2026-09-13T10:41:34",
"revision": 3503375,
"files": {
"industryactivities.json": {
"deprecated": false,
"stale": false,
"md5": "bfeffbb198389876e9b7b8f70c81615d",
"revision": 3503375
},
"attributeorders.json": {
"deprecated": false,
"stale": true,
"md5": "72906f5a2415bbdca45fc06936b20096",
"revision": 2911964
}
}
}
```
`stale: true` means Hoboleaks says that file's update was not successful and its contents may be outdated; it is not equivalent to a transport failure. The compiler should record the file as present-but-stale and make the promotion policy explicit. The `md5` is a content-change detector supplied by Hoboleaks; it is not a substitute for a locally computed SHA-256 of an archive.
### Small useful raw maps
[`industryactivities.json`](https://sde.hoboleaks.space/tq/industryactivities.json) is a small keyed object. Its current entries use numeric string keys and **camelCase** fields:
```json
{
"1": {
"activityID": 1,
"description": "Manufacturing of things",
"activityName": "Manufacturing"
},
"9": {
"activityID": 9,
"description": "The process of combining raw and intermediate materials to create advanced components",
"activityName": "Reactions"
}
}
```
[`agenttypes.json`](https://sde.hoboleaks.space/tq/agenttypes.json) is even smaller and maps numeric string keys directly to strings:
```json
{
"1": "NonAgent",
"4": "ResearchAgent",
"13": "HeraldryAgent"
}
```
[`repackagedvolumes.json`](https://sde.hoboleaks.space/tq/repackagedvolumes.json) is a keyed map from a type ID string to a numeric volume:
```json
{
"52907": 1300000.0,
"34828": 5000.0,
"22544": 3750.0
}
```
The EVE Ref data-source table specifically identifies `repackagedvolumes.json` as an addition to inventory types. These files are useful smoke-test inputs because they exercise object maps, string keys, source-specific camelCase, booleans, strings, integers, and floating-point values without requiring a large dataset.
The table also names Hoboleaks `blueprints.json`, `typematerials.json`, and other large files, but no raw schema claim is made here for those files: their full payloads were not downloaded, and the normalized EVE Ref blueprint contract above is the only blueprint/material/product contract asserted in this document.
## 3. Independent provenance and compiler policy
Use independent observations for each layer; do not treat one successful endpoint as proof that all sources succeeded.
| What to record | Route/evidence | Meaning |
| --- | --- | --- |
| Archive artifact identity | `data.everef.net/{reference-data,hoboleaks-sde}/index.json` | URL, byte size, `last_modified`, opaque ETag, and archive type. |
| Immutable release history | `.../history/index.json` and dated year index | Dated artifact URL, `file_time`, byte size, last-modified, and opaque ETag. |
| Reference build/source provenance | [`https://ref-data.everef.net/meta`](https://ref-data.everef.net/meta) | Current `build_time` and SHA-256 values for `esi`, `hoboleaks`, and `sde`. Observed: build `2026-09-13T11:10:03Z`; ESI `645b2902174fe52561b946a5c6e39e257358803956238f0750798e1cde5e4077`; Hoboleaks `477799f37c9275c3ad408117cce8e7913688be41b17fe404cd48e3b501be55cd`; SDE `56f5f7eba4b401aeb2fc1090d2db5ab9dd55cc08a396871e764df45017fa70af`. |
| Hoboleaks per-file state | [`https://sde.hoboleaks.space/tq/meta.json`](https://sde.hoboleaks.space/tq/meta.json) | Per-file revision, MD5, `stale`, and `deprecated`, plus Hoboleaks timestamp/revision. |
| Independent HTTP check | `HEAD` on the exact archive and metadata URLs | Transport status, content length, ETag, and Last-Modified; use to detect a changed artifact before parsing. |
The two archive indexes publish no SHA-256 field. For a future downloader, compute SHA-256 after downloading to a staging path and store it separately from the server ETag and from `/meta`'s *input-source* hashes. Do not label the archive ETag as a checksum.
The `/meta` payload itself is a small useful provenance fixture:
```json
{
"build_time": "2026-09-13T11:10:03Z",
"esi": {"sha256": "645b2902174fe52561b946a5c6e39e257358803956238f0750798e1cde5e4077"},
"hoboleaks": {"sha256": "477799f37c9275c3ad408117cce8e7913688be41b17fe404cd48e3b501be55cd"},
"sde": {"sha256": "56f5f7eba4b401aeb2fc1090d2db5ab9dd55cc08a396871e764df45017fa70af"}
}
```
This is source/build provenance for the running Reference Data service, not a cryptographic checksum of either `.tar.xz` artifact.
### Recommended staged promotion behavior (design guidance, not an EVE schema)
- Build each candidate in a fresh immutable SQLite database; never mutate the currently served database while parsing.
- Capture one source result for Reference Data and one for Hoboleaks: `success`, `stale`, or `failed` with URL, HTTP metadata, bytes, local SHA-256, source revision/build, and an error detail. Keep source results independent so a Hoboleaks failure is not hidden by a successful Reference Data download, and vice versa.
- Validate schema, IDs, keyed-map joins, duplicate keys, numeric ranges, and the expected fixture set before the candidate is eligible for promotion. A transport-successful but `stale: true` Hoboleaks file is visibly stale, not silently “fresh”.
- Write and fsync the candidate database and its manifest before publishing a single atomic pointer/manifest to it. Keep the previous manifest and immutable DB available for immediate rollback; a failed source or validation must leave the previous pointer untouched.
- Store both archive provenance (artifact URL/ETag/Last-Modified/local hash) and EVE Ref provenance (`/meta` build time and input hashes). They answer different questions and must not be merged into one version field.
The local manifest fields in this subsection are implementation recommendations only; they are not claims about a published EVE Ref schema.
## Gaps and non-claims
- Reference Data is explicitly marked “currently in development”; the OpenAPI spec may change without a compatibility promise.
- `reference-data-latest.tar.xz` and `hoboleaks-sde-latest.tar.xz` are release artifacts; their directory indexes do not document an internal tar member manifest or a published archive SHA-256. No full archive was downloaded to infer one.
- Reference Data currently has no documented `/systems` endpoint. Systems XYZ are covered here only as the ESI/SDE source shape identified by EVE Ref, not as a Reference Data record.
- Hoboleaks documents file names and metadata semantics but does not publish a formal JSON Schema for every raw file. No unsupported raw schema is inferred for large `blueprints.json` or `typematerials.json`.
- A `stale` Hoboleaks file can be outdated while still retrievable. Promotion policy must decide whether that is allowed; the source does not define the application's policy.
## Sources
Kept sources are limited to current EVE Ref docs/schema, the linked official ESI machine-readable schema for the explicit systems boundary, and the linked current Hoboleaks metadata/files.
- [EVE Ref Reference Data docs](https://docs.everef.net/datasets/reference-data) — source combination, download/API routes, data-source table, normalization rules, and development warning.
- [EVE Ref Reference Data OpenAPI (pinned commit)](https://github.com/autonomouslogic/eve-ref/blob/58dafd29ea55ad78fd62cbf820460cb79eb66202/spec/reference-data.yaml) — exact `InventoryCategory`, `InventoryGroup`, `InventoryType`, `Blueprint`, `BlueprintActivity`, `BlueprintMaterial`, `Coordinate`, and `RefDataMeta` schemas.
- [Reference Data API metadata](https://ref-data.everef.net/meta) — current build time and independent SHA-256 hashes for ESI, Hoboleaks, and SDE inputs.
- [Reference Data category 4](https://ref-data.everef.net/categories/4), [group 18](https://ref-data.everef.net/groups/18), [type 34](https://ref-data.everef.net/types/34), and [blueprint 683](https://ref-data.everef.net/blueprints/683) — live representative records used for fixture slices.
- [Reference Data archive index](https://data.everef.net/reference-data/index.json) and [Hoboleaks archive index](https://data.everef.net/hoboleaks-sde/index.json) — current archive URL, size, last-modified, and ETag metadata.
- [EVE Ref Hoboleaks SDE docs](https://docs.everef.net/datasets/hoboleaks-sde.html) — archive mirror route and provenance link.
- [Hoboleaks SDE landing page](https://sde.hoboleaks.space/) and [Hoboleaks `tq/meta.json`](https://sde.hoboleaks.space/tq/meta.json) — raw file inventory, update caveats, revision/MD5/stale semantics.
- [Hoboleaks `industryactivities.json`](https://sde.hoboleaks.space/tq/industryactivities.json), [agenttypes.json](https://sde.hoboleaks.space/tq/agenttypes.json), and [repackagedvolumes.json](https://sde.hoboleaks.space/tq/repackagedvolumes.json) — small current raw-map fixtures.
- [Official ESI OpenAPI](https://esi.evetech.net/meta/openapi.json) and [Jita system response](https://esi.evetech.net/latest/universe/systems/30000142/) — exact ESI systems required fields and XYZ example; used only because EVE Ref maps systems to the ESI/SDE sources rather than defining a Reference Data systems endpoint.