Define migration, backfill, and compatibility policy #7

Closed
opened 2026-08-23 13:08:37 -04:00 by jb · 2 comments
Owner

Part of #1

Blocked by: #2, #4

Question

How should existing descriptions, tier projections, captured-only rows, executable programs, and currently forced-model records migrate into the new lifecycle; which records can be classified or backfilled without guessing; how are source hashes and stale state initialized; and what backup, snapshot-version, execution, audit, and verification guarantees prevent schema-only migration from leaving derived data incomplete?

Part of #1 Blocked by: #2, #4 ## Question How should existing descriptions, tier projections, captured-only rows, executable programs, and currently forced-model records migrate into the new lifecycle; which records can be classified or backfilled without guessing; how are source hashes and stale state initialized; and what backup, snapshot-version, execution, audit, and verification guarantees prevent schema-only migration from leaving derived data incomplete?
jb added the wayfinder:grilling label 2026-08-23 13:08:37 -04:00
jb self-assigned this 2026-08-23 14:52:27 -04:00
Author
Owner

Migration, backfill, and compatibility policy

Read-only preflight

A preflight command reads the current database without changing it and classifies every mechanic owner as lossless, absent, or blocked. It writes a machine-readable report with owner identity, required variants, template and projection checks, current readiness, program validation, proposed source fingerprint, and each blocking reason.

The preflight applies owner-specific proof rules:

  • Talent: derive authored slots from current numbered outcomes and duration or stack pools, render every owner-required Tier variant, and require exact equality with talent_tiers.description.
  • OS Protocol Talent: require the template, base outcome values, and os_protocol_talent_effects projection to agree, then render the base prose.
  • Skill Chip Talent: require matching minimum and maximum outcome values and effect projections, then render both variants.
  • Specialization tree-rank effect: require rank description, reusable effect identity and template, attachment values, and the base program to form one unambiguous source projection.

The preflight does not infer missing values from prose, pair outcomes by similar names, repair placeholders, or promote approximate legacy behavior. A record with no Mechanic Program may still classify as absent when its authored source can migrate exactly.

A populated captured_only program may classify as lossless when it validates and its authored projections round-trip. An empty captured placeholder becomes absent. Invalid graphs, missing required variants, projection disagreements, unresolved placeholders, and unsupported legacy actions are blocked.

Repair gate

The cutover cannot start while the preflight report contains a blocked record. A Registered Agent must repair the authored source or explicitly clear the incompatible model after reviewing the preserved evidence. Each repair uses the normal owner action and audit event.

The team reruns preflight after every repair set. The final report must classify every owner and must match the database state selected for backup. Migration does not maintain a legacy reader for unresolved records.

Initial authored state and model revisions

The migration stores each proven template, independent authored slots, ordered variants, and materialized prose. It computes source fingerprint version 1 as SHA-256 over the versioned canonical JSON contract defined by issue #2.

For each proven non-empty program, migration creates model revision 1 with the current readiness, Scenario Inputs, pools, clauses, outcomes, mechanic contract version, and initial source fingerprint, then selects it as current. Migration labels this revision as imported legacy state rather than claiming a historical generation source.

Records without a program and empty captured placeholders receive no selected revision and begin absent. Migration does not fabricate deleted program history, earlier source fingerprints, or stale periods. Existing audit events and the verified backup remain the evidence for pre-cutover state.

Atomic cutover

Implementation migrates every reader and writer before the data cutover. The repeatable migration runs in one database transaction and performs these steps:

  1. create authored-source, slot, materialization, model-revision, and selected-revision structures;
  2. assert that the live database matches the final preflight classifications;
  3. backfill authored state, source fingerprints, and proven initial revisions;
  4. validate counts, required variants, hashes, foreign keys, model graphs, and selected revisions;
  5. switch all owner relationships to the new structures;
  6. drop legacy description and effect projections and the mutable conditional-rule graph after proving that no reader or foreign key uses them;
  7. commit only when every assertion passes.

The cutover removes talent_tiers.description, OS Protocol and Skill Chip compatibility effect tables, the mutable conditional_effect_rules graph, and equivalent obsolete projections in the same release. The application performs no dual writes and provides no fallback parsing after cutover.

Model revisions remain immutable by application contract and audit convention. Mutation services create new revisions and never update or delete an existing graph. The database does not add immutability triggers.

Snapshot and backup policy

ISAC snapshot format becomes version 6. Version 6 exports authored sources, slots, materialized variants, model revisions, selected revisions, and their contract versions in dependency order. Import accepts version 6 only and requires an empty compatible target under the existing replacement rules. Version 5 snapshots fail with an explicit unsupported-version error; no compatibility reader or conversion branch remains in the application.

The generic full JSON backup format may remain unchanged if it already enumerates all database tables without schema-specific decoding. Its backup manifest must include the new tables and row counts.

Before production cutover, operators stop writers and run npm run db:backup. They verify the timestamped JSON file, record row counts for every affected and critical table, and record the identities of representative Talent, Protocol, Skill Chip, Specialization, authored-source, model-revision, and audit rows. Backups remain outside Git.

After migration, verification compares total owners, relationships, authored variants, selected revisions, readiness counts, source fingerprints, and critical entity identities with the preflight and backup records. It also renders every materialized description again and validates every selected Mechanic Program.

A migration assertion failure rolls back the transaction. A post-commit verification failure keeps writers stopped, preserves the failed database and logs, and restores the verified backup only through the documented empty-target restore procedure before the old release resumes.

Audit and reports

The migration does not invent a human actor or append ordinary ISAC mutation events for mechanical backfill. It preserves existing audit history and produces the preflight report, backup manifest, migration summary, and post-cutover verification report as operational evidence.

Repairs performed before migration and later model selections remain normal audited owner mutations. Migration reports must not contain credentials, private profile data unrelated to the affected records, or hidden model reasoning.

Acceptance checks

  • Fixture databases cover every owner type, standard and optional Exotic variants, populated and empty captured-only rows, executable programs, mismatched projections, and unsupported legacy actions.
  • Preflight is read-only, deterministic, and blocks every unprovable record.
  • Source fingerprint and rendered prose fixtures remain stable across repeated runs.
  • Migration is repeatable and aborts without partial backfill on changed source data.
  • Every migrated current revision validates and matches its source fingerprint.
  • Empty captured placeholders become absent.
  • No legacy projection table, dual write, version 5 importer, or fallback reader remains.
  • Version 6 export/import round-trips the complete authored and model revision graph.
  • Backup, row-count, critical-entity, description-render, and program-validation checks pass before writers restart.
# Migration, backfill, and compatibility policy ## Read-only preflight A preflight command reads the current database without changing it and classifies every mechanic owner as `lossless`, `absent`, or `blocked`. It writes a machine-readable report with owner identity, required variants, template and projection checks, current readiness, program validation, proposed source fingerprint, and each blocking reason. The preflight applies owner-specific proof rules: - Talent: derive authored slots from current numbered outcomes and duration or stack pools, render every owner-required Tier variant, and require exact equality with `talent_tiers.description`. - OS Protocol Talent: require the template, base outcome values, and `os_protocol_talent_effects` projection to agree, then render the base prose. - Skill Chip Talent: require matching minimum and maximum outcome values and effect projections, then render both variants. - Specialization tree-rank effect: require rank description, reusable effect identity and template, attachment values, and the base program to form one unambiguous source projection. The preflight does not infer missing values from prose, pair outcomes by similar names, repair placeholders, or promote approximate legacy behavior. A record with no Mechanic Program may still classify as `absent` when its authored source can migrate exactly. A populated `captured_only` program may classify as lossless when it validates and its authored projections round-trip. An empty captured placeholder becomes `absent`. Invalid graphs, missing required variants, projection disagreements, unresolved placeholders, and unsupported legacy actions are blocked. ## Repair gate The cutover cannot start while the preflight report contains a blocked record. A Registered Agent must repair the authored source or explicitly clear the incompatible model after reviewing the preserved evidence. Each repair uses the normal owner action and audit event. The team reruns preflight after every repair set. The final report must classify every owner and must match the database state selected for backup. Migration does not maintain a legacy reader for unresolved records. ## Initial authored state and model revisions The migration stores each proven template, independent authored slots, ordered variants, and materialized prose. It computes source fingerprint version 1 as SHA-256 over the versioned canonical JSON contract defined by issue #2. For each proven non-empty program, migration creates model revision 1 with the current readiness, Scenario Inputs, pools, clauses, outcomes, mechanic contract version, and initial source fingerprint, then selects it as current. Migration labels this revision as imported legacy state rather than claiming a historical generation source. Records without a program and empty captured placeholders receive no selected revision and begin `absent`. Migration does not fabricate deleted program history, earlier source fingerprints, or stale periods. Existing audit events and the verified backup remain the evidence for pre-cutover state. ## Atomic cutover Implementation migrates every reader and writer before the data cutover. The repeatable migration runs in one database transaction and performs these steps: 1. create authored-source, slot, materialization, model-revision, and selected-revision structures; 2. assert that the live database matches the final preflight classifications; 3. backfill authored state, source fingerprints, and proven initial revisions; 4. validate counts, required variants, hashes, foreign keys, model graphs, and selected revisions; 5. switch all owner relationships to the new structures; 6. drop legacy description and effect projections and the mutable conditional-rule graph after proving that no reader or foreign key uses them; 7. commit only when every assertion passes. The cutover removes `talent_tiers.description`, OS Protocol and Skill Chip compatibility effect tables, the mutable `conditional_effect_rules` graph, and equivalent obsolete projections in the same release. The application performs no dual writes and provides no fallback parsing after cutover. Model revisions remain immutable by application contract and audit convention. Mutation services create new revisions and never update or delete an existing graph. The database does not add immutability triggers. ## Snapshot and backup policy ISAC snapshot format becomes version 6. Version 6 exports authored sources, slots, materialized variants, model revisions, selected revisions, and their contract versions in dependency order. Import accepts version 6 only and requires an empty compatible target under the existing replacement rules. Version 5 snapshots fail with an explicit unsupported-version error; no compatibility reader or conversion branch remains in the application. The generic full JSON backup format may remain unchanged if it already enumerates all database tables without schema-specific decoding. Its backup manifest must include the new tables and row counts. Before production cutover, operators stop writers and run `npm run db:backup`. They verify the timestamped JSON file, record row counts for every affected and critical table, and record the identities of representative Talent, Protocol, Skill Chip, Specialization, authored-source, model-revision, and audit rows. Backups remain outside Git. After migration, verification compares total owners, relationships, authored variants, selected revisions, readiness counts, source fingerprints, and critical entity identities with the preflight and backup records. It also renders every materialized description again and validates every selected Mechanic Program. A migration assertion failure rolls back the transaction. A post-commit verification failure keeps writers stopped, preserves the failed database and logs, and restores the verified backup only through the documented empty-target restore procedure before the old release resumes. ## Audit and reports The migration does not invent a human actor or append ordinary ISAC mutation events for mechanical backfill. It preserves existing audit history and produces the preflight report, backup manifest, migration summary, and post-cutover verification report as operational evidence. Repairs performed before migration and later model selections remain normal audited owner mutations. Migration reports must not contain credentials, private profile data unrelated to the affected records, or hidden model reasoning. ## Acceptance checks - Fixture databases cover every owner type, standard and optional Exotic variants, populated and empty captured-only rows, executable programs, mismatched projections, and unsupported legacy actions. - Preflight is read-only, deterministic, and blocks every unprovable record. - Source fingerprint and rendered prose fixtures remain stable across repeated runs. - Migration is repeatable and aborts without partial backfill on changed source data. - Every migrated current revision validates and matches its source fingerprint. - Empty captured placeholders become absent. - No legacy projection table, dual write, version 5 importer, or fallback reader remains. - Version 6 export/import round-trips the complete authored and model revision graph. - Backup, row-count, critical-entity, description-render, and program-validation checks pass before writers restart.
Author
Owner

Decision specification recorded above. It defines proof-based preflight, explicit repair gates, imported initial revisions, one atomic cutover, immediate legacy projection removal, version 6-only snapshots, verified backup and count checks, and no compatibility reader. Closing as resolved.

Decision specification recorded above. It defines proof-based preflight, explicit repair gates, imported initial revisions, one atomic cutover, immediate legacy projection removal, version 6-only snapshots, verified backup and count checks, and no compatibility reader. Closing as resolved.
jb closed this issue 2026-08-23 18:26:15 -04:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: jb/shd.build#7