eden/docs/restore-rehearsal.md

# Isolated application/Convex restore rehearsal (T01)

This is a runbook and evidence template, not an executed restore. It is
intended for a disposable self-hosted Convex deployment and synthetic users
only. Never point it at the persistent development deployment, copy a live
`.env`, or use production Infisical credentials.

## Scope and ownership

T01 owns the application/Compose release procedure, Convex data and instance
credential backup/restore, configuration/key backup handling, and post-restore
session/grant invalidation evidence. T03 owns immutable reference artifact
rollback and profile-compatible `PREVIOUS` handling; this runbook does not
change or claim that path.

The current repository has no automated Convex migration rollback command and
no implemented global post-restore session/grant invalidation migration. Those
are explicit blockers below, not steps that may be silently skipped.

## Prerequisites and isolation

- A clean T01 worktree at a recorded commit; Bun 1.3.14 and Docker/Compose.
- A unique project name, for example `eden-restore-$USER-$RANDOM`, unique
  volume names, and unique loopback ports. Keep all published ports on
  loopback and use a disposable backend.
- Free disk equal to **source volume bytes + backup archive bytes + restored
  volume bytes + 20% headroom**. The archive size is unknown until backup, so
  record `du -sb` before starting and recheck before extraction; a simple 2x
  estimate is not sufficient.
- Budget **2 CPUs, 4 GiB RAM, and 30 minutes** for the disposable backend and
  application rehearsal. A reference rebuild is a separate T03 operation;
  the reviewed production worker budget is 2 CPUs/4 GiB and must not be
  assumed as Convex restore evidence.
- A temporary mode-0600 directory outside the repository for synthetic keys,
  backup archives, logs, and evidence. Do not print key values or put them in
  command arguments.
- Host root access for direct Docker-volume `tar` operations, or a separately
  approved, digest-pinned helper image. The commands below use direct host
  tar and must be run only on the disposable volumes.

```bash
#!/usr/bin/env bash
set -euo pipefail
repo="$PWD"
project="eden-restore-${USER:-operator}-$(date +%s)-$$"
restore_project="${project}-restored"
work="$(mktemp -d "${TMPDIR:-/tmp}/eden-restore.XXXXXX")"
chmod 700 "$work"
# Do not let an ambient shell, repository .env, or prior Compose project alter
# this rehearsal. The explicit env file below contains only non-secret values;
# synthetic key values are loaded separately from the mode-0600 key file.
unset COMPOSE_FILE COMPOSE_ENV_FILES COMPOSE_PROJECT_NAME COMPOSE_PROFILES DOCKER_HOST APP_MODE APP_ORIGIN ORIGIN APP_RELEASE_ID APP_CAPABILITY_ALLOWLIST CONVEX_URL PUBLIC_CONVEX_URL CONVEX_SITE_ORIGIN CONVEX_PORT CONVEX_SITE_PORT WEB_PORT CONVEX_IMAGE CONVEX_DASHBOARD_IMAGE EVE_CALLBACK_URL EVE_CLIENT_ID EVE_CLIENT_SECRET ESI_USER_AGENT TYPESAFE_API_KEY APP_AUTH_ISSUER TOKEN_ENCRYPTION_KEY CONVEX_BRIDGE_SECRET CONVEX_SYNC_WORKER_SECRET APP_AUTH_PRIVATE_KEY APP_AUTH_JWKS CONVEX_SELF_HOSTED_URL CONVEX_SELF_HOSTED_ADMIN_KEY
export CONVEX_BIND_ADDRESS=127.0.0.1 CONVEX_PORT=$((3200 + RANDOM % 500)) CONVEX_SITE_PORT=$((3700 + RANDOM % 500))
export WEB_BIND_ADDRESS=127.0.0.1 WEB_PORT=$((4200 + RANDOM % 500))
export APP_MODE=live APP_RELEASE_ID=restore-fixture APP_CAPABILITY_ALLOWLIST=asset_search,asset_valuation,asset_sync,assistant_dispatch,session_token,identity,subscriptions
export APP_ORIGIN="http://127.0.0.1:$WEB_PORT" ORIGIN="$APP_ORIGIN" APP_AUTH_ISSUER="$APP_ORIGIN"
export PUBLIC_CONVEX_URL="http://127.0.0.1:$CONVEX_PORT" CONVEX_SITE_ORIGIN="http://127.0.0.1:$CONVEX_SITE_PORT"
export EVE_CALLBACK_URL="$APP_ORIGIN/auth/eve/callback"
write_compose_env() {
  cat > "$work/compose.env" <<EOF
COMPOSE_PROJECT_NAME=$project
CONVEX_BIND_ADDRESS=127.0.0.1
CONVEX_PORT=$CONVEX_PORT
CONVEX_SITE_PORT=$CONVEX_SITE_PORT
WEB_BIND_ADDRESS=127.0.0.1
WEB_PORT=$WEB_PORT
APP_MODE=$APP_MODE
APP_RELEASE_ID=$APP_RELEASE_ID
APP_CAPABILITY_ALLOWLIST=$APP_CAPABILITY_ALLOWLIST
APP_ORIGIN=$APP_ORIGIN
ORIGIN=$ORIGIN
APP_AUTH_ISSUER=$APP_AUTH_ISSUER
PUBLIC_CONVEX_URL=$PUBLIC_CONVEX_URL
CONVEX_SITE_ORIGIN=$CONVEX_SITE_ORIGIN
EVE_CALLBACK_URL=$EVE_CALLBACK_URL
EOF
}
write_compose_env
compose() { docker compose --project-directory "$repo" --file "$repo/compose.yaml" --env-file "$work/compose.env" "$@"; }
cleanup() {
  compose -p "$project" down -v >/dev/null 2>&1 || true
  compose -p "$restore_project" down -v >/dev/null 2>&1 || true
  # Export/copy the evidence bundle before this cleanup; the temporary key and
  # archives are intentionally removed only after retention is recorded.
  rm -rf "$work"
}
trap cleanup EXIT
(cd "$work" && bun "$repo/scripts/generate-keys.ts")
chmod 600 "$work/.env.keys"
set -a
. "$work/.env.keys"
set +a
```

The bootstrap above loads only synthetic values into a temporary shell
environment and writes the explicit Compose env file. It allocates loopback
backend, site, web, issuer, public URL, and callback origins together; when
restore ports change, update all corresponding exported values before starting
the restored project. The Compose web service intentionally uses the
service-internal `http://backend:3210`; only host/public settings use the
allocated loopback ports. Do not use a production callback or provider key.

## Backup rehearsal

1. Start a disposable backend with a unique Compose project and record the
   backend image digest, git commit, volume name, and container ID.

   ```sh
   export COMPOSE_PROJECT_NAME="$project"
   compose -p "$project" up -d backend
   volume="${project}_convex-data"
   mount="$(docker volume inspect -f '{{ .Mountpoint }}' "$volume")"
   source_bytes_before="$(du -sb "$mount" | awk '{print $1}')"
   printf 'source_bytes_before=%s\n' "$source_bytes_before" > "$work/volume-metrics.txt"
   ```

2. Obtain the disposable instance admin key through the operator-approved
   secret mechanism for the exact pinned `CONVEX_IMAGE`; hold it in memory
   only. Configure the synthetic Convex environment using the temporary key
   file, synthetic issuer/JWKS, bridge secret, and allowlist. Do not invent a
   key fingerprint or assume an image-specific helper command:

   ```sh
   read -r -s admin_key
   CONVEX_SELF_HOSTED_ADMIN_KEY="$admin_key" \
     CONVEX_SELF_HOSTED_URL="http://127.0.0.1:$CONVEX_PORT" \
     bun "$repo/scripts/configure-convex.ts" >/dev/null
   ```

   Use the existing configure path only with this disposable URL/admin key;
   never pass a persistent admin key through this procedure. Successful
   configuration against the restored instance using this same in-memory key
   is the credential continuity check.

3. The legacy `scripts/smoke-convex.ts` exercises useful synthetic writes,
   but currently accepts an arbitrary `CONVEX_SELF_HOSTED_URL` when write is
   enabled. Do **not** execute it as part of this runbook; it remains
   unexecuted evidence until it is wrapped with the same loopback target,
   explicit disposable confirmation, and key scoping as the marker tool.
   Then write a deterministic marker that can be looked up after restore. Use
   a fresh marker and synthetic character ID for every rehearsal; do not use a
   random-only smoke result as proof of record survival:

   ```sh
   export RESTORE_MARKER="$(date -u +%Y%m%dT%H%M%SZ)-synthetic"
   export RESTORE_MARKER_CHARACTER_ID=4000000001 # reserved synthetic range
   export RESTORE_MARKER_SESSION_EXPIRES_AT=$(( $(date +%s) * 1000 + 7 * 24 * 60 * 60 * 1000 ))
   SMOKE_ALLOW_WRITE=1 RESTORE_MARKER_CONFIRMATION=DISPOSABLE_LOOPBACK_ONLY \
     RESTORE_MARKER_PHASE=write RESTORE_MARKER="$RESTORE_MARKER" \
     RESTORE_MARKER_CHARACTER_ID="$RESTORE_MARKER_CHARACTER_ID" \
     RESTORE_MARKER_SESSION_EXPIRES_AT="$RESTORE_MARKER_SESSION_EXPIRES_AT" \
     CONVEX_SELF_HOSTED_URL="http://127.0.0.1:$CONVEX_PORT" \
     bun "$repo/scripts/smoke-restore-marker.ts" | tee "$work/marker-write.log"
   ```

   The marker tool defaults to non-writing verification, rejects non-loopback
   targets before constructing a backend client, and requires both the explicit
   write phase and `DISPOSABLE_LOOPBACK_ONLY` confirmation. Never reuse the
   marker command against a shared, staging, or production URL.

4. Create a consistent filesystem backup while the disposable backend is
   stopped. The Convex data volume contains the backend data and instance
   credential material; back up it as one unit. A host administrator may need
   root access to the Docker volume mount.

   ```sh
   compose -p "$project" stop backend
   tar --xattrs --acls --numeric-owner -C "$mount" \
     -czf "$work/convex-data.tar.gz" .
   sha256sum "$work/convex-data.tar.gz" > "$work/convex-data.tar.gz.sha256"
   archive_bytes="$(stat -c '%s' "$work/convex-data.tar.gz")"
   printf 'archive_bytes=%s\n' "$archive_bytes" >> "$work/volume-metrics.txt"
   cp --preserve=mode "$work/.env.keys" "$work/application-keys.synthetic"
   sha256sum "$work/application-keys.synthetic" > "$work/application-keys.synthetic.sha256"
   ```

   Record archive byte size, SHA-256, volume size, and stop/start timestamps.
   Do not include raw keys in the evidence bundle; retain the mode-0600 file in
   the protected temporary directory until cleanup. Before extraction, require
   free disk of at least `source_bytes_before + archive_bytes + estimated
   restored_bytes + 20% headroom`.

## Restore and isolation checks

1. Create a second unique Compose project and volume. Extract only the
   recorded archive into the new volume, preserving ownership and xattrs.

   ```sh
   restore_volume="${restore_project}_convex-data"
   docker volume create "$restore_volume" >/dev/null
   restore_mount="$(docker volume inspect -f '{{ .Mountpoint }}' "$restore_volume")"
   tar --xattrs --acls --numeric-owner -C "$restore_mount" \
     -xzf "$work/convex-data.tar.gz"
   export CONVEX_PORT=$((5200 + RANDOM % 500)) CONVEX_SITE_PORT=$((5700 + RANDOM % 500)) WEB_PORT=$((6200 + RANDOM % 500))
   export APP_ORIGIN="http://127.0.0.1:$WEB_PORT" ORIGIN="$APP_ORIGIN" APP_AUTH_ISSUER="$APP_ORIGIN"
   export PUBLIC_CONVEX_URL="http://127.0.0.1:$CONVEX_PORT" CONVEX_SITE_ORIGIN="http://127.0.0.1:$CONVEX_SITE_PORT"
   export EVE_CALLBACK_URL="$APP_ORIGIN/auth/eve/callback"
   write_compose_env
   ```

2. Start the restored backend under the second project and configure the web
   container with **only** the synthetic application keys. Reuse the same
   in-memory `CONVEX_SELF_HOSTED_ADMIN_KEY` captured before backup; the
   configure command must succeed against the restored URL. Failure means
   instance-credential continuity is unproven and traffic must remain closed:

   ```sh
   compose -p "$restore_project" up -d backend
   CONVEX_SELF_HOSTED_ADMIN_KEY="$admin_key" \
     CONVEX_SELF_HOSTED_URL="http://127.0.0.1:$CONVEX_PORT" \
     bun "$repo/scripts/configure-convex.ts" >/dev/null
   ```

3. Verify the deterministic marker before any other restore evidence. The
   legacy synthetic Convex smoke remains unexecuted for the safety reason
   above; do not substitute its arbitrary-target behavior for restore proof:

   ```sh
   RESTORE_MARKER="$RESTORE_MARKER" \
     RESTORE_MARKER_CHARACTER_ID="$RESTORE_MARKER_CHARACTER_ID" \
     RESTORE_MARKER_SESSION_EXPIRES_AT="$RESTORE_MARKER_SESSION_EXPIRES_AT" \
     CONVEX_SELF_HOSTED_URL="http://127.0.0.1:$CONVEX_PORT" \
     RESTORE_MARKER_PHASE=verify bun "$repo/scripts/smoke-restore-marker.ts"
   ```

   Verify:

   - restored schema/functions start without a migration error;
   - a newly created synthetic user cannot read the source project's records;
   - restored records remain tenant-isolated;
   - the deterministic pre-backup marker is present and unchanged;
   - the old pre-backup session/JWT is rejected after the approved invalidation
     step; and
   - the old encrypted grant cannot service synchronization until explicit
     reauthorization.

   Reference-volume separation, read-only web mounting, and reference artifact
   rollback are outside this procedure and belong to T03's separate rehearsal.
   Do not record those as T01 restore evidence.

4. **Current blocker:** the repository does not yet provide a global
   post-restore session/grant invalidation command. Do not claim this check
   passed. The operator must choose and approve one of: a Convex migration that
   revokes all sessions and quarantines token records, a deployment-specific
   credential-version fence, or an equivalent bounded mechanism. Record the
   maximum stale-authority window and the decision before reopening traffic.
   If implemented for the rehearsal, run the marker check before its explicit
   expiry window:

   ```sh
   RESTORE_MARKER="$RESTORE_MARKER" RESTORE_MARKER_CHARACTER_ID="$RESTORE_MARKER_CHARACTER_ID" \
     RESTORE_MARKER_SESSION_EXPIRES_AT="$RESTORE_MARKER_SESSION_EXPIRES_AT" \
     CONVEX_SELF_HOSTED_URL="http://127.0.0.1:$CONVEX_PORT" \
     RESTORE_MARKER_PHASE=expect-invalidated bun "$repo/scripts/smoke-restore-marker.ts"
   ```

   Grant rotation/quarantine behavior is unexecuted until that mechanism
   exists; no synthetic provider result can substitute for it.

5. Test application rollback by stopping the restored web service, starting the
   immediately previous digest-pinned web image against the restored backend,
   and exercising read-only synthetic status/assets paths. Do not use a mutable
   `latest` tag. If the schema is not backward compatible, record rollback as
   blocked; do not mutate the source volume to make it pass.

## Migration rollback rehearsal

There is no migration history/rollback command in this baseline. A valid
rehearsal therefore requires a parent-approved migration fixture and a chosen
rollback mechanism. Before starting, record the exact **baseline commit/image
and candidate commit/image**, the migration command, the backend image digest,
and the compatibility contract. The evidence must include commands equivalent
to these (with the operator's approved migration tool substituted explicitly):

```sh
baseline_commit=$(git rev-parse HEAD)
candidate_commit=<recorded-candidate-commit>
baseline_image=<recorded-digest-pinned-image>
candidate_image=<recorded-digest-pinned-image>
# approved migration command against the disposable candidate deployment:
<deploy-candidate-and-run-migration>
# restore pre-migration archive into a fresh volume/project:
<restore-pre-migration-volume>
# start baseline image against the restored volume:
<deploy-baseline-image-and-run-compatibility-checks>
```

At minimum:

1. back up the disposable volume before migration;
2. deploy the candidate migration to the disposable backend;
3. run schema/functions and tenant-isolation checks;
4. restore the pre-migration volume into a fresh project (never in place);
5. start the previous application image and rerun the checks; and
6. record whether the migration is backward compatible and which data loss,
   RPO, and RTO limits apply.

Placeholder commands are an evidence gap, not a passing rehearsal. A
`docker compose down -v` or a successful process restart is not migration
rollback evidence. T03's reference artifact rollback is a separate test and
must verify profile, importer version, manifest integrity, and live-reader
compatibility.

## Retention and cleanup

Before leaving the shell (and before the `trap` removes `$work`), export only
redacted logs, checksums, metrics, image digests, and the evidence table to a
protected location outside the temporary directory. Never copy either key
file or an admin key into that bundle:

```sh
evidence_dir="${EVIDENCE_DIR:?Set an approved protected evidence directory}"
umask 077
mkdir -p "$evidence_dir/$project"
cp "$work"/*.log "$work"/*.sha256 "$work"/*metrics* "$evidence_dir/$project/" 2>/dev/null || true
printf 'source_project=%s\nrestored_project=%s\n' "$project" "$restore_project" > "$evidence_dir/$project/projects.txt"
sha256sum "$evidence_dir/$project"/* > "$evidence_dir/$project/evidence.sha256"
printf 'retained_until_utc=<operator-set-expiry>\n' > "$evidence_dir/$project/retention.txt"
```

Confirm the bundle contains no `.env`, `application-keys`, admin key, token,
or unredacted provider output. Only after export and operator approval may the
cleanup trap remove both Compose projects, both volumes, and `$work`.

## Evidence template

Copy this table into the release record and attach only redacted logs/checksums:

| Field | Record |
|---|---|
| Worktree commit / image digests | |
| Source and restored Compose project names | |
| Convex source/restored volume names | |
| Synthetic key IDs/fingerprints (not values) | |
| Backup archive bytes / SHA-256 | |
| Pre/post volume byte counts | |
| Schema/migration version and rollback mechanism | |
| Session/grant invalidation decision and stale window | |
| Commands and timestamps (UTC) | |
| Synthetic smoke result | |
| Tenant-isolation result | |
| Old-session rejection result | |
| Old-grant sync rejection result | |
| Browser/provider/restore steps not exercised | |
| Cleanup confirmation and retained artifacts/expiry | |
| Operator approval and unresolved blockers | |

The evidence must label mock, fixture, disposable-backend, and consenting-live
observations separately. Passing local tests, CI, or a fixture container does
not constitute a restore, browser, provider, or production-readiness claim.