test: add rollout smoke and restore evidence runbook
This commit is contained in:
@@ -0,0 +1,184 @@
|
||||
# Isolated application/Convex restore rehearsal (T01)
|
||||
|
||||
This is a runbook and evidence template, not an executed restore. It is
|
||||
intended for a disposable self-hosted Convex deployment and synthetic users
|
||||
only. Never point it at the persistent development deployment, copy a live
|
||||
`.env`, or use production Infisical credentials.
|
||||
|
||||
## Scope and ownership
|
||||
|
||||
T01 owns the application/Compose release procedure, Convex data and instance
|
||||
credential backup/restore, configuration/key backup handling, and post-restore
|
||||
session/grant invalidation evidence. T03 owns immutable reference artifact
|
||||
rollback and profile-compatible `PREVIOUS` handling; this runbook does not
|
||||
change or claim that path.
|
||||
|
||||
The current repository has no automated Convex migration rollback command and
|
||||
no implemented global post-restore session/grant invalidation migration. Those
|
||||
are explicit blockers below, not steps that may be silently skipped.
|
||||
|
||||
## Prerequisites and isolation
|
||||
|
||||
- A clean T01 worktree at a recorded commit; Bun 1.3.14 and Docker/Compose.
|
||||
- A unique project name, for example `eden-restore-$USER-$RANDOM`, and unique
|
||||
volume names. Keep published ports on loopback and use a disposable backend.
|
||||
- At least **2x the current Convex data volume bytes plus 10%** free disk for
|
||||
source volume, backup archive, and restored volume. Record `du -sb` before
|
||||
starting.
|
||||
- Budget **2 CPUs, 4 GiB RAM, and 30 minutes** for the disposable backend and
|
||||
application rehearsal. A reference rebuild is a separate T03 operation;
|
||||
the reviewed production worker budget is 2 CPUs/4 GiB and must not be
|
||||
assumed as Convex restore evidence.
|
||||
- A temporary mode-0600 directory outside the repository for synthetic keys,
|
||||
backup archives, logs, and evidence. Do not print key values or put them in
|
||||
command arguments.
|
||||
|
||||
```sh
|
||||
set -euo pipefail
|
||||
repo="$PWD"
|
||||
project="eden-restore-${USER:-operator}-$(date +%s)-$$"
|
||||
work="$(mktemp -d "${TMPDIR:-/tmp}/eden-restore.XXXXXX")"
|
||||
chmod 700 "$work"
|
||||
trap 'docker compose -p "$project" down -v >/dev/null 2>&1 || true; rm -rf "$work"' EXIT
|
||||
(cd "$work" && bun "$repo/scripts/generate-keys.ts")
|
||||
chmod 600 "$work/.env.keys"
|
||||
```
|
||||
|
||||
Load only those synthetic values into a temporary shell environment. Set
|
||||
`APP_MODE=demo`, `APP_RELEASE_ID=restore-fixture`,
|
||||
`APP_CAPABILITY_ALLOWLIST=asset_search,asset_valuation,asset_sync,assistant_dispatch,session_token,identity,subscriptions`,
|
||||
and an issuer such as `http://127.0.0.1:18080`. Do not use a production
|
||||
callback or provider key.
|
||||
|
||||
## Backup rehearsal
|
||||
|
||||
1. Start a disposable backend with a unique Compose project and record the
|
||||
backend image digest, git commit, volume name, and container ID.
|
||||
|
||||
```sh
|
||||
export COMPOSE_PROJECT_NAME="$project"
|
||||
docker compose -p "$project" up -d backend
|
||||
volume="${project}_convex-data"
|
||||
mount="$(docker volume inspect -f '{{ .Mountpoint }}' "$volume")"
|
||||
du -sb "$mount"
|
||||
```
|
||||
|
||||
2. Configure the synthetic Convex environment using the temporary key file,
|
||||
synthetic issuer/JWKS, bridge secret, and allowlist. Use the existing
|
||||
`bun run convex:configure` path only with the disposable URL/admin key;
|
||||
never pass a persistent admin key through this procedure.
|
||||
|
||||
3. Run the existing synthetic write smoke against this backend, with
|
||||
`SMOKE_ALLOW_WRITE=1`, and record its output without recording secrets. It
|
||||
exercises login/session, staged publication, failed publication preserving
|
||||
the previous generation, tenant isolation, JWT audience checks, and local
|
||||
revocation.
|
||||
|
||||
4. Create a consistent filesystem backup while the disposable backend is
|
||||
stopped. The Convex data volume contains the backend data and instance
|
||||
credential material; back up it as one unit. A host administrator may need
|
||||
root access to the Docker volume mount.
|
||||
|
||||
```sh
|
||||
docker compose -p "$project" stop backend
|
||||
tar --xattrs --acls --numeric-owner -C "$mount" \
|
||||
-czf "$work/convex-data.tar.gz" .
|
||||
sha256sum "$work/convex-data.tar.gz" > "$work/convex-data.tar.gz.sha256"
|
||||
cp --preserve=mode "$work/.env.keys" "$work/application-keys.synthetic"
|
||||
sha256sum "$work/application-keys.synthetic" > "$work/application-keys.synthetic.sha256"
|
||||
```
|
||||
|
||||
Record archive byte size, SHA-256, volume size, and stop/start timestamps.
|
||||
Do not include raw keys in the evidence bundle; retain the mode-0600 file in
|
||||
the protected temporary directory until cleanup.
|
||||
|
||||
## Restore and isolation checks
|
||||
|
||||
1. Create a second unique Compose project and volume. Extract only the
|
||||
recorded archive into the new volume, preserving ownership and xattrs.
|
||||
|
||||
```sh
|
||||
restore_project="${project}-restored"
|
||||
restore_volume="${restore_project}_convex-data"
|
||||
docker volume create "$restore_volume" >/dev/null
|
||||
restore_mount="$(docker volume inspect -f '{{ .Mountpoint }}' "$restore_volume")"
|
||||
tar --xattrs --acls --numeric-owner -C "$restore_mount" \
|
||||
-xzf "$work/convex-data.tar.gz"
|
||||
```
|
||||
|
||||
2. Start the restored backend under the second project and configure the web
|
||||
container with **only** the synthetic application keys. Confirm that the
|
||||
restored instance credential is the backed-up credential, not a newly
|
||||
generated one; record only a redacted fingerprint/metadata, never its
|
||||
value.
|
||||
|
||||
3. Repeat the synthetic Convex smoke with fresh synthetic IDs. Verify:
|
||||
|
||||
- restored schema/functions start without a migration error;
|
||||
- a newly created synthetic user cannot read the source project's records;
|
||||
- restored records remain tenant-isolated;
|
||||
- the restored reference/config volume is separate and mounted read-only by
|
||||
web;
|
||||
- the old pre-backup session/JWT is rejected after the approved invalidation
|
||||
step; and
|
||||
- the old encrypted grant cannot service synchronization until explicit
|
||||
reauthorization.
|
||||
|
||||
4. **Current blocker:** the repository does not yet provide a global
|
||||
post-restore session/grant invalidation command. Do not claim this check
|
||||
passed. The operator must choose and approve one of: a Convex migration that
|
||||
revokes all sessions and quarantines token records, a deployment-specific
|
||||
credential-version fence, or an equivalent bounded mechanism. Record the
|
||||
maximum stale-authority window and the decision before reopening traffic.
|
||||
|
||||
5. Test application rollback by stopping the restored web service, starting the
|
||||
immediately previous digest-pinned web image against the restored backend,
|
||||
and exercising read-only synthetic status/assets paths. Do not use a mutable
|
||||
`latest` tag. If the schema is not backward compatible, record rollback as
|
||||
blocked; do not mutate the source volume to make it pass.
|
||||
|
||||
## Migration rollback rehearsal
|
||||
|
||||
There is no migration history/rollback command in this baseline. A valid
|
||||
rehearsal therefore requires a parent-approved migration fixture and a chosen
|
||||
rollback mechanism. At minimum:
|
||||
|
||||
1. back up the disposable volume before migration;
|
||||
2. deploy the migration to the disposable backend;
|
||||
3. run schema/functions and tenant-isolation checks;
|
||||
4. restore the pre-migration volume into a fresh project (never in place);
|
||||
5. start the previous application image and rerun the checks; and
|
||||
6. record whether the migration is backward compatible and which data loss,
|
||||
RPO, and RTO limits apply.
|
||||
|
||||
A `docker compose down -v` or a successful process restart is not migration
|
||||
rollback evidence. T03's reference artifact rollback is a separate test and
|
||||
must verify profile, importer version, manifest integrity, and live-reader
|
||||
compatibility.
|
||||
|
||||
## Evidence template
|
||||
|
||||
Copy this table into the release record and attach only redacted logs/checksums:
|
||||
|
||||
| Field | Record |
|
||||
|---|---|
|
||||
| Worktree commit / image digests | |
|
||||
| Source and restored Compose project names | |
|
||||
| Convex source/restored volume names | |
|
||||
| Synthetic key IDs/fingerprints (not values) | |
|
||||
| Backup archive bytes / SHA-256 | |
|
||||
| Pre/post volume byte counts | |
|
||||
| Schema/migration version and rollback mechanism | |
|
||||
| Session/grant invalidation decision and stale window | |
|
||||
| Commands and timestamps (UTC) | |
|
||||
| Synthetic smoke result | |
|
||||
| Tenant-isolation result | |
|
||||
| Old-session rejection result | |
|
||||
| Old-grant sync rejection result | |
|
||||
| Browser/provider/restore steps not exercised | |
|
||||
| Cleanup confirmation and retained artifacts/expiry | |
|
||||
| Operator approval and unresolved blockers | |
|
||||
|
||||
The evidence must label mock, fixture, disposable-backend, and consenting-live
|
||||
observations separately. Passing local tests, CI, or a fixture container does
|
||||
not constitute a restore, browser, provider, or production-readiness claim.
|
||||
Reference in New Issue
Block a user