jb/eden
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
01b8b4642b0bbe9d7b3d1b8f7e623bbcf4ff1bfa
eden/docs/restore-rehearsal.md
T

17 KiB
Raw Blame History

Isolated application/Convex restore rehearsal (T01)

This is a runbook and evidence template, not an executed restore. It is intended for a disposable self-hosted Convex deployment and synthetic users only. Never point it at the persistent development deployment, copy a live .env, or use production Infisical credentials.

Scope and ownership

T01 owns the application/Compose release procedure, Convex data and instance credential backup/restore, configuration/key backup handling, and post-restore session/grant invalidation evidence. T03 owns immutable reference artifact rollback and profile-compatible PREVIOUS handling; this runbook does not change or claim that path.

The current repository has no automated Convex migration rollback command and no implemented global post-restore session/grant invalidation migration. Those are explicit blockers below, not steps that may be silently skipped.

Prerequisites and isolation

  • A clean T01 worktree at a recorded commit; Bun 1.3.14 and Docker/Compose.
  • A unique project name, for example eden-restore-$USER-$RANDOM, unique volume names, and unique loopback ports. Keep all published ports on loopback and use a disposable backend.
  • Free disk equal to source volume bytes + backup archive bytes + restored volume bytes + 20% headroom. The archive size is unknown until backup, so record du -sb before starting and recheck before extraction; a simple 2x estimate is not sufficient.
  • Budget 2 CPUs, 4 GiB RAM, and 30 minutes for the disposable backend and application rehearsal. A reference rebuild is a separate T03 operation; the reviewed production worker budget is 2 CPUs/4 GiB and must not be assumed as Convex restore evidence.
  • A temporary mode-0600 directory outside the repository for synthetic keys, backup archives, logs, and evidence. Do not print key values or put them in command arguments.
  • Host root access for direct Docker-volume tar operations, or a separately approved, digest-pinned helper image. The commands below use direct host tar and must be run only on the disposable volumes.
#!/usr/bin/env bash
set -euo pipefail
repo="$PWD"
project="eden-restore-${USER:-operator}-$(date +%s)-$$"
restore_project="${project}-restored"
work="$(mktemp -d "${TMPDIR:-/tmp}/eden-restore.XXXXXX")"
chmod 700 "$work"
# Do not let an ambient shell, repository .env, or prior Compose project alter
# this rehearsal. The explicit env file below contains only non-secret values;
# synthetic key values are loaded separately from the mode-0600 key file.
unset COMPOSE_FILE COMPOSE_ENV_FILES COMPOSE_PROJECT_NAME COMPOSE_PROFILES DOCKER_HOST APP_MODE APP_ORIGIN ORIGIN APP_RELEASE_ID APP_CAPABILITY_ALLOWLIST CONVEX_URL PUBLIC_CONVEX_URL CONVEX_SITE_ORIGIN CONVEX_PORT CONVEX_SITE_PORT WEB_PORT CONVEX_IMAGE CONVEX_DASHBOARD_IMAGE EVE_CALLBACK_URL EVE_CLIENT_ID EVE_CLIENT_SECRET ESI_USER_AGENT TYPESAFE_API_KEY APP_AUTH_ISSUER TOKEN_ENCRYPTION_KEY CONVEX_BRIDGE_SECRET APP_AUTH_PRIVATE_KEY APP_AUTH_JWKS CONVEX_SELF_HOSTED_URL CONVEX_SELF_HOSTED_ADMIN_KEY
export CONVEX_BIND_ADDRESS=127.0.0.1 CONVEX_PORT=$((3200 + RANDOM % 500)) CONVEX_SITE_PORT=$((3700 + RANDOM % 500))
export WEB_BIND_ADDRESS=127.0.0.1 WEB_PORT=$((4200 + RANDOM % 500))
export APP_MODE=live APP_RELEASE_ID=restore-fixture APP_CAPABILITY_ALLOWLIST=asset_search,asset_valuation,asset_sync,assistant_dispatch,session_token,identity,subscriptions
export APP_ORIGIN="http://127.0.0.1:$WEB_PORT" ORIGIN="$APP_ORIGIN" APP_AUTH_ISSUER="$APP_ORIGIN"
export PUBLIC_CONVEX_URL="http://127.0.0.1:$CONVEX_PORT" CONVEX_SITE_ORIGIN="http://127.0.0.1:$CONVEX_SITE_PORT"
export EVE_CALLBACK_URL="$APP_ORIGIN/auth/eve/callback"
write_compose_env() {
  cat > "$work/compose.env" <<EOF
COMPOSE_PROJECT_NAME=$project
CONVEX_BIND_ADDRESS=127.0.0.1
CONVEX_PORT=$CONVEX_PORT
CONVEX_SITE_PORT=$CONVEX_SITE_PORT
WEB_BIND_ADDRESS=127.0.0.1
WEB_PORT=$WEB_PORT
APP_MODE=$APP_MODE
APP_RELEASE_ID=$APP_RELEASE_ID
APP_CAPABILITY_ALLOWLIST=$APP_CAPABILITY_ALLOWLIST
APP_ORIGIN=$APP_ORIGIN
ORIGIN=$ORIGIN
APP_AUTH_ISSUER=$APP_AUTH_ISSUER
PUBLIC_CONVEX_URL=$PUBLIC_CONVEX_URL
CONVEX_SITE_ORIGIN=$CONVEX_SITE_ORIGIN
EVE_CALLBACK_URL=$EVE_CALLBACK_URL
EOF
}
write_compose_env
compose() { docker compose --project-directory "$repo" --file "$repo/compose.yaml" --env-file "$work/compose.env" "$@"; }
cleanup() {
  compose -p "$project" down -v >/dev/null 2>&1 || true
  compose -p "$restore_project" down -v >/dev/null 2>&1 || true
  # Export/copy the evidence bundle before this cleanup; the temporary key and
  # archives are intentionally removed only after retention is recorded.
  rm -rf "$work"
}
trap cleanup EXIT
(cd "$work" && bun "$repo/scripts/generate-keys.ts")
chmod 600 "$work/.env.keys"
set -a
. "$work/.env.keys"
set +a

The bootstrap above loads only synthetic values into a temporary shell environment and writes the explicit Compose env file. It allocates loopback backend, site, web, issuer, public URL, and callback origins together; when restore ports change, update all corresponding exported values before starting the restored project. The Compose web service intentionally uses the service-internal http://backend:3210; only host/public settings use the allocated loopback ports. Do not use a production callback or provider key.

Backup rehearsal

  1. Start a disposable backend with a unique Compose project and record the backend image digest, git commit, volume name, and container ID.

    export COMPOSE_PROJECT_NAME="$project"
compose -p "$project" up -d backend
volume="${project}_convex-data"
mount="$(docker volume inspect -f '{{ .Mountpoint }}' "$volume")"
source_bytes_before="$(du -sb "$mount" | awk '{print $1}')"
printf 'source_bytes_before=%s\n' "$source_bytes_before" > "$work/volume-metrics.txt"

  2. Obtain the disposable instance admin key through the operator-approved secret mechanism for the exact pinned CONVEX_IMAGE; hold it in memory only. Configure the synthetic Convex environment using the temporary key file, synthetic issuer/JWKS, bridge secret, and allowlist. Do not invent a key fingerprint or assume an image-specific helper command:

    read -r -s admin_key
CONVEX_SELF_HOSTED_ADMIN_KEY="$admin_key" \
  CONVEX_SELF_HOSTED_URL="http://127.0.0.1:$CONVEX_PORT" \
  bun "$repo/scripts/configure-convex.ts" >/dev/null

    Use the existing configure path only with this disposable URL/admin key; never pass a persistent admin key through this procedure. Successful configuration against the restored instance using this same in-memory key is the credential continuity check.

  3. Run the existing synthetic write smoke against this backend, with SMOKE_ALLOW_WRITE=1, and record its output without recording secrets. It exercises login/session, staged publication, failed publication preserving the previous generation, tenant isolation, JWT audience checks, and local revocation:

    SMOKE_ALLOW_WRITE=1 CONVEX_SELF_HOSTED_URL="http://127.0.0.1:$CONVEX_PORT" \
  bun "$repo/scripts/smoke-convex.ts" | tee "$work/convex-smoke.log"

    Then write a deterministic marker that can be looked up after restore. Use a fresh marker and synthetic character ID for every rehearsal; do not use a random-only smoke result as proof of record survival:

    export RESTORE_MARKER="$(date -u +%Y%m%dT%H%M%SZ)-synthetic"
export RESTORE_MARKER_CHARACTER_ID=4000000001 # reserved synthetic range
export RESTORE_MARKER_SESSION_EXPIRES_AT=$(( $(date +%s) * 1000 + 7 * 24 * 60 * 60 * 1000 ))
SMOKE_ALLOW_WRITE=1 RESTORE_MARKER_CONFIRMATION=DISPOSABLE_LOOPBACK_ONLY \
  RESTORE_MARKER_PHASE=write RESTORE_MARKER="$RESTORE_MARKER" \
  RESTORE_MARKER_CHARACTER_ID="$RESTORE_MARKER_CHARACTER_ID" \
  RESTORE_MARKER_SESSION_EXPIRES_AT="$RESTORE_MARKER_SESSION_EXPIRES_AT" \
  CONVEX_SELF_HOSTED_URL="http://127.0.0.1:$CONVEX_PORT" \
  bun "$repo/scripts/smoke-restore-marker.ts" | tee "$work/marker-write.log"

    The marker tool defaults to non-writing verification, rejects non-loopback targets before constructing a backend client, and requires both the explicit write phase and DISPOSABLE_LOOPBACK_ONLY confirmation. Never reuse the marker command against a shared, staging, or production URL.

  4. Create a consistent filesystem backup while the disposable backend is stopped. The Convex data volume contains the backend data and instance credential material; back up it as one unit. A host administrator may need root access to the Docker volume mount.

    compose -p "$project" stop backend
tar --xattrs --acls --numeric-owner -C "$mount" \
  -czf "$work/convex-data.tar.gz" .
sha256sum "$work/convex-data.tar.gz" > "$work/convex-data.tar.gz.sha256"
archive_bytes="$(stat -c '%s' "$work/convex-data.tar.gz")"
printf 'archive_bytes=%s\n' "$archive_bytes" >> "$work/volume-metrics.txt"
cp --preserve=mode "$work/.env.keys" "$work/application-keys.synthetic"
sha256sum "$work/application-keys.synthetic" > "$work/application-keys.synthetic.sha256"

    Record archive byte size, SHA-256, volume size, and stop/start timestamps. Do not include raw keys in the evidence bundle; retain the mode-0600 file in the protected temporary directory until cleanup. Before extraction, require free disk of at least source_bytes_before + archive_bytes + estimated restored_bytes + 20% headroom.

Restore and isolation checks

  1. Create a second unique Compose project and volume. Extract only the recorded archive into the new volume, preserving ownership and xattrs.

    restore_volume="${restore_project}_convex-data"
docker volume create "$restore_volume" >/dev/null
restore_mount="$(docker volume inspect -f '{{ .Mountpoint }}' "$restore_volume")"
tar --xattrs --acls --numeric-owner -C "$restore_mount" \
  -xzf "$work/convex-data.tar.gz"
export CONVEX_PORT=$((5200 + RANDOM % 500)) CONVEX_SITE_PORT=$((5700 + RANDOM % 500)) WEB_PORT=$((6200 + RANDOM % 500))
export APP_ORIGIN="http://127.0.0.1:$WEB_PORT" ORIGIN="$APP_ORIGIN" APP_AUTH_ISSUER="$APP_ORIGIN"
export PUBLIC_CONVEX_URL="http://127.0.0.1:$CONVEX_PORT" CONVEX_SITE_ORIGIN="http://127.0.0.1:$CONVEX_SITE_PORT"
export EVE_CALLBACK_URL="$APP_ORIGIN/auth/eve/callback"
write_compose_env

  2. Start the restored backend under the second project and configure the web container with only the synthetic application keys. Reuse the same in-memory CONVEX_SELF_HOSTED_ADMIN_KEY captured before backup; the configure command must succeed against the restored URL. Failure means instance-credential continuity is unproven and traffic must remain closed:

    compose -p "$restore_project" up -d backend
CONVEX_SELF_HOSTED_ADMIN_KEY="$admin_key" \
  CONVEX_SELF_HOSTED_URL="http://127.0.0.1:$CONVEX_PORT" \
  bun "$repo/scripts/configure-convex.ts" >/dev/null

  3. Verify the deterministic marker before running any fresh smoke, then run the synthetic Convex smoke with fresh IDs:

    RESTORE_MARKER="$RESTORE_MARKER" \
  RESTORE_MARKER_CHARACTER_ID="$RESTORE_MARKER_CHARACTER_ID" \
  RESTORE_MARKER_SESSION_EXPIRES_AT="$RESTORE_MARKER_SESSION_EXPIRES_AT" \
  CONVEX_SELF_HOSTED_URL="http://127.0.0.1:$CONVEX_PORT" \
  RESTORE_MARKER_PHASE=verify bun "$repo/scripts/smoke-restore-marker.ts"
SMOKE_ALLOW_WRITE=1 CONVEX_SELF_HOSTED_URL="http://127.0.0.1:$CONVEX_PORT" \
  bun "$repo/scripts/smoke-convex.ts" | tee "$work/convex-restore-smoke.log"

    Verify:

    • restored schema/functions start without a migration error;
    • a newly created synthetic user cannot read the source project's records;
    • restored records remain tenant-isolated;
    • the deterministic pre-backup marker is present and unchanged;
    • the old pre-backup session/JWT is rejected after the approved invalidation step; and
    • the old encrypted grant cannot service synchronization until explicit reauthorization.

    Reference-volume separation, read-only web mounting, and reference artifact rollback are outside this procedure and belong to T03's separate rehearsal. Do not record those as T01 restore evidence.

  4. Current blocker: the repository does not yet provide a global post-restore session/grant invalidation command. Do not claim this check passed. The operator must choose and approve one of: a Convex migration that revokes all sessions and quarantines token records, a deployment-specific credential-version fence, or an equivalent bounded mechanism. Record the maximum stale-authority window and the decision before reopening traffic. If implemented for the rehearsal, run the marker check before its explicit expiry window:

    RESTORE_MARKER="$RESTORE_MARKER" RESTORE_MARKER_CHARACTER_ID="$RESTORE_MARKER_CHARACTER_ID" \
  RESTORE_MARKER_SESSION_EXPIRES_AT="$RESTORE_MARKER_SESSION_EXPIRES_AT" \
  CONVEX_SELF_HOSTED_URL="http://127.0.0.1:$CONVEX_PORT" \
  RESTORE_MARKER_PHASE=expect-invalidated bun "$repo/scripts/smoke-restore-marker.ts"

    Grant rotation/quarantine behavior is unexecuted until that mechanism exists; no synthetic provider result can substitute for it.

  5. Test application rollback by stopping the restored web service, starting the immediately previous digest-pinned web image against the restored backend, and exercising read-only synthetic status/assets paths. Do not use a mutable latest tag. If the schema is not backward compatible, record rollback as blocked; do not mutate the source volume to make it pass.

Migration rollback rehearsal

There is no migration history/rollback command in this baseline. A valid rehearsal therefore requires a parent-approved migration fixture and a chosen rollback mechanism. Before starting, record the exact baseline commit/image and candidate commit/image, the migration command, the backend image digest, and the compatibility contract. The evidence must include commands equivalent to these (with the operator's approved migration tool substituted explicitly):

baseline_commit=$(git rev-parse HEAD)
candidate_commit=<recorded-candidate-commit>
baseline_image=<recorded-digest-pinned-image>
candidate_image=<recorded-digest-pinned-image>
# approved migration command against the disposable candidate deployment:
<deploy-candidate-and-run-migration>
# restore pre-migration archive into a fresh volume/project:
<restore-pre-migration-volume>
# start baseline image against the restored volume:
<deploy-baseline-image-and-run-compatibility-checks>

At minimum:

  1. back up the disposable volume before migration;
  2. deploy the candidate migration to the disposable backend;
  3. run schema/functions and tenant-isolation checks;
  4. restore the pre-migration volume into a fresh project (never in place);
  5. start the previous application image and rerun the checks; and
  6. record whether the migration is backward compatible and which data loss, RPO, and RTO limits apply.

Placeholder commands are an evidence gap, not a passing rehearsal. A docker compose down -v or a successful process restart is not migration rollback evidence. T03's reference artifact rollback is a separate test and must verify profile, importer version, manifest integrity, and live-reader compatibility.

Retention and cleanup

Before leaving the shell (and before the trap removes $work), export only redacted logs, checksums, metrics, image digests, and the evidence table to a protected location outside the temporary directory. Never copy either key file or an admin key into that bundle:

evidence_dir="${EVIDENCE_DIR:?Set an approved protected evidence directory}"
umask 077
mkdir -p "$evidence_dir/$project"
cp "$work"/*.log "$work"/*.sha256 "$work"/*metrics* "$evidence_dir/$project/" 2>/dev/null || true
printf 'source_project=%s\nrestored_project=%s\n' "$project" "$restore_project" > "$evidence_dir/$project/projects.txt"
sha256sum "$evidence_dir/$project"/* > "$evidence_dir/$project/evidence.sha256"
printf 'retained_until_utc=<operator-set-expiry>\n' > "$evidence_dir/$project/retention.txt"

Confirm the bundle contains no .env, application-keys, admin key, token, or unredacted provider output. Only after export and operator approval may the cleanup trap remove both Compose projects, both volumes, and $work.

Evidence template

Copy this table into the release record and attach only redacted logs/checksums:

Field Record
Worktree commit / image digests
Source and restored Compose project names
Convex source/restored volume names
Synthetic key IDs/fingerprints (not values)
Backup archive bytes / SHA-256
Pre/post volume byte counts
Schema/migration version and rollback mechanism
Session/grant invalidation decision and stale window
Commands and timestamps (UTC)
Synthetic smoke result
Tenant-isolation result
Old-session rejection result
Old-grant sync rejection result
Browser/provider/restore steps not exercised
Cleanup confirmation and retained artifacts/expiry
Operator approval and unresolved blockers

The evidence must label mock, fixture, disposable-backend, and consenting-live observations separately. Passing local tests, CI, or a fixture container does not constitute a restore, browser, provider, or production-readiness claim.

Reference in New Issue View Git Blame Copy Permalink