jb/eden
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: fail closed backend capabilities and isolate gate smoke

Browse Source
This commit is contained in:
jb
2026-09-19 10:01:23 -04:00
parent 1371995cbe
commit 5b0076cb20
17 changed files with 344 additions and 81 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/release-safety.md
+29 -9
View File
@@ -40,6 +40,25 @@ provider. The current character status subscription is gated in Convex; no
durable user-job service exists in this baseline, so future job kinds remain
closed until they have reviewed enqueue and worker entry points.
### Policy authority and update order
The web and Convex processes receive the same value, but parse it in separate
processes. The web parser preserves the private baseline when its setting is
omitted for development compatibility; the Convex/backend parser requires an
explicit, valid setting and fails closed when propagation is missing, empty,
or malformed. Convex is authoritative for persisted subscriptions and bridge
mutations; the web gate is an earlier ingress/provider guard, not a replacement
for backend authorization.
Update the backend setting first, validate a status query and representative
bridge mutation in an isolated deployment, then update the web setting and
verify the HTTP gate. A web-only narrowing does **not** revoke existing
Convex subscriptions, sessions, or grants, and this lane makes no such claim.
A backend-only narrowing may cause already-running web requests to receive
backend denials until the web setting converges. Record both settings and the
validation timestamps in the release evidence; rollback must restore the
backend policy before reopening web entry points.
## Immutable release metadata
`APP_RELEASE_ID` is carried into the runtime image and returned as the
@@ -89,15 +108,16 @@ lane.
## Evidence boundaries
Read-only `tea` inspection on 2026-09-19 reported no Gitea workflows or runs
for `jb/eden`; runner labels, Docker availability, and action availability are
therefore not validated. The workflow is prepared locally but has not run
remotely. The source job runs a strict staging release check after building
the client. The separate container job is restricted to trusted pushes,
validates Compose expansion, builds the digest-pinned web image, compiles a
synthetic reference into a private volume, and smoke-tests the runtime with
`--network none`; these steps still need an actual compatible Gitea runner.
No browser automation, OAuth callback failure injection, Convex migration
rollback, instance-credential backup/restore, key rotation, or live
for `jb/eden`; runner labels, Docker availability, protected-ref enforcement,
and action availability were therefore not validated. The workflow is prepared
locally but has not run remotely. The source job runs a strict staging release
check after building the client. The separate container job is currently
**disabled** (`if: ${{ false }}`), not merely push-filtered: its rootless
runner and protected-ref policy require operator verification and approval
before enablement. Its draft commands use per-run image, volume, and container
names and clean up only those names, but are not release evidence while the job
is disabled. No browser automation, OAuth callback failure injection, Convex
migration rollback, instance-credential backup/restore, key rotation, or live
revocation/transfer evidence is claimed by this lane. Those remain explicit
operator/integration gates and must use synthetic or separately consented
isolated environments.
docs/restore-rehearsal.md
+132 -24
View File
@@ -20,11 +20,13 @@ are explicit blockers below, not steps that may be silently skipped.
## Prerequisites and isolation
- A clean T01 worktree at a recorded commit; Bun 1.3.14 and Docker/Compose.
- A unique project name, for example `eden-restore-$USER-$RANDOM`, and unique
volume names. Keep published ports on loopback and use a disposable backend.
- At least **2x the current Convex data volume bytes plus 10%** free disk for
source volume, backup archive, and restored volume. Record `du -sb` before
starting.
- A unique project name, for example `eden-restore-$USER-$RANDOM`, unique
volume names, and unique loopback ports. Keep all published ports on
loopback and use a disposable backend.
- Free disk equal to **source volume bytes + backup archive bytes + restored
volume bytes + 20% headroom**. The archive size is unknown until backup, so
record `du -sb` before starting and recheck before extraction; a simple 2x
estimate is not sufficient.
- Budget **2 CPUs, 4 GiB RAM, and 30 minutes** for the disposable backend and
application rehearsal. A reference rebuild is a separate T03 operation;
the reviewed production worker budget is 2 CPUs/4 GiB and must not be
@@ -32,14 +34,28 @@ are explicit blockers below, not steps that may be silently skipped.
- A temporary mode-0600 directory outside the repository for synthetic keys,
backup archives, logs, and evidence. Do not print key values or put them in
command arguments.
- Host root access for direct Docker-volume `tar` operations, or a separately
approved, digest-pinned helper image. The commands below use direct host
tar and must be run only on the disposable volumes.
```sh
```bash
#!/usr/bin/env bash
set -euo pipefail
repo="$PWD"
project="eden-restore-${USER:-operator}-$(date +%s)-$$"
restore_project="${project}-restored"
work="$(mktemp -d "${TMPDIR:-/tmp}/eden-restore.XXXXXX")"
chmod 700 "$work"
trap 'docker compose -p "$project" down -v >/dev/null 2>&1 || true; rm -rf "$work"' EXIT
export CONVEX_BIND_ADDRESS=127.0.0.1 CONVEX_PORT=$((3200 + RANDOM % 500)) CONVEX_SITE_PORT=$((3700 + RANDOM % 500))
export WEB_BIND_ADDRESS=127.0.0.1 WEB_PORT=$((4200 + RANDOM % 500))
cleanup() {
docker compose -p "$project" down -v >/dev/null 2>&1 || true
docker compose -p "$restore_project" down -v >/dev/null 2>&1 || true
# Export/copy the evidence bundle before this cleanup; the temporary key and
# archives are intentionally removed only after retention is recorded.
rm -rf "$work"
}
trap cleanup EXIT
(cd "$work" && bun "$repo/scripts/generate-keys.ts")
chmod 600 "$work/.env.keys"
```
@@ -60,13 +76,27 @@ callback or provider key.
docker compose -p "$project" up -d backend
volume="${project}_convex-data"
mount="$(docker volume inspect -f '{{ .Mountpoint }}' "$volume")"
du -sb "$mount"
source_bytes_before="$(du -sb "$mount" | awk '{print $1}')"
printf 'source_bytes_before=%s\n' "$source_bytes_before" > "$work/volume-metrics.txt"
```
2. Configure the synthetic Convex environment using the temporary key file,
synthetic issuer/JWKS, bridge secret, and allowlist. Use the existing
`bun run convex:configure` path only with the disposable URL/admin key;
never pass a persistent admin key through this procedure.
2. Obtain the disposable instance admin key through the operator-approved
secret mechanism for the exact pinned `CONVEX_IMAGE`; hold it in memory
only. Configure the synthetic Convex environment using the temporary key
file, synthetic issuer/JWKS, bridge secret, and allowlist. Do not invent a
key fingerprint or assume an image-specific helper command:
```sh
read -r -s CONVEX_SELF_HOSTED_ADMIN_KEY
export CONVEX_SELF_HOSTED_ADMIN_KEY
CONVEX_SELF_HOSTED_URL=http://127.0.0.1:"$CONVEX_PORT" \
bun "$repo/scripts/configure-convex.ts" >/dev/null
```
Use the existing configure path only with this disposable URL/admin key;
never pass a persistent admin key through this procedure. Successful
configuration against the restored instance using this same in-memory key
is the credential continuity check.
3. Run the existing synthetic write smoke against this backend, with
`SMOKE_ALLOW_WRITE=1`, and record its output without recording secrets. It
@@ -74,6 +104,19 @@ callback or provider key.
the previous generation, tenant isolation, JWT audience checks, and local
revocation.
Then write a deterministic marker that can be looked up after restore. Use
a fresh marker and synthetic character ID for every rehearsal; do not use a
random-only smoke result as proof of record survival:
```sh
export RESTORE_MARKER="$(date -u +%Y%m%dT%H%M%SZ)-synthetic"
export RESTORE_MARKER_CHARACTER_ID=998000001
SMOKE_ALLOW_WRITE=1 RESTORE_MARKER="$RESTORE_MARKER" \
RESTORE_MARKER_CHARACTER_ID="$RESTORE_MARKER_CHARACTER_ID" \
CONVEX_SELF_HOSTED_URL=http://127.0.0.1:"$CONVEX_PORT" \
bun "$repo/scripts/smoke-restore-marker.ts" | tee "$work/marker-write.log"
```
4. Create a consistent filesystem backup while the disposable backend is
stopped. The Convex data volume contains the backend data and instance
credential material; back up it as one unit. A host administrator may need
@@ -84,13 +127,17 @@ callback or provider key.
tar --xattrs --acls --numeric-owner -C "$mount" \
-czf "$work/convex-data.tar.gz" .
sha256sum "$work/convex-data.tar.gz" > "$work/convex-data.tar.gz.sha256"
archive_bytes="$(stat -c '%s' "$work/convex-data.tar.gz")"
printf 'archive_bytes=%s\n' "$archive_bytes" >> "$work/volume-metrics.txt"
cp --preserve=mode "$work/.env.keys" "$work/application-keys.synthetic"
sha256sum "$work/application-keys.synthetic" > "$work/application-keys.synthetic.sha256"
```
Record archive byte size, SHA-256, volume size, and stop/start timestamps.
Do not include raw keys in the evidence bundle; retain the mode-0600 file in
the protected temporary directory until cleanup.
the protected temporary directory until cleanup. Before extraction, require
free disk of at least `source_bytes_before + archive_bytes + estimated
restored_bytes + 20% headroom`.
## Restore and isolation checks
@@ -98,38 +145,59 @@ callback or provider key.
recorded archive into the new volume, preserving ownership and xattrs.
```sh
restore_project="${project}-restored"
restore_volume="${restore_project}_convex-data"
docker volume create "$restore_volume" >/dev/null
restore_mount="$(docker volume inspect -f '{{ .Mountpoint }}' "$restore_volume")"
tar --xattrs --acls --numeric-owner -C "$restore_mount" \
-xzf "$work/convex-data.tar.gz"
export CONVEX_PORT=$((5200 + RANDOM % 500)) CONVEX_SITE_PORT=$((5700 + RANDOM % 500)) WEB_PORT=$((6200 + RANDOM % 500))
```
2. Start the restored backend under the second project and configure the web
container with **only** the synthetic application keys. Confirm that the
restored instance credential is the backed-up credential, not a newly
generated one; record only a redacted fingerprint/metadata, never its
value.
container with **only** the synthetic application keys. Reuse the same
in-memory `CONVEX_SELF_HOSTED_ADMIN_KEY` captured before backup; the
configure command must succeed against the restored URL. Failure means
instance-credential continuity is unproven and traffic must remain closed:
3. Repeat the synthetic Convex smoke with fresh synthetic IDs. Verify:
```sh
docker compose -p "$restore_project" up -d backend
CONVEX_SELF_HOSTED_URL=http://127.0.0.1:"$CONVEX_PORT" \
bun "$repo/scripts/configure-convex.ts" >/dev/null
```
3. Verify the deterministic marker before running any fresh smoke, then run
the synthetic Convex smoke with fresh IDs:
```sh
SMOKE_ALLOW_WRITE=1 RESTORE_MARKER="$RESTORE_MARKER" \
RESTORE_MARKER_CHARACTER_ID="$RESTORE_MARKER_CHARACTER_ID" \
CONVEX_SELF_HOSTED_URL=http://127.0.0.1:"$CONVEX_PORT" \
RESTORE_MARKER_PHASE=verify bun "$repo/scripts/smoke-restore-marker.ts"
```
Verify:
- restored schema/functions start without a migration error;
- a newly created synthetic user cannot read the source project's records;
- restored records remain tenant-isolated;
- the restored reference/config volume is separate and mounted read-only by
web;
- the deterministic pre-backup marker is present and unchanged;
- the old pre-backup session/JWT is rejected after the approved invalidation
step; and
- the old encrypted grant cannot service synchronization until explicit
reauthorization.
Reference-volume separation, read-only web mounting, and reference artifact
rollback are outside this procedure and belong to T03's separate rehearsal.
Do not record those as T01 restore evidence.
4. **Current blocker:** the repository does not yet provide a global
post-restore session/grant invalidation command. Do not claim this check
passed. The operator must choose and approve one of: a Convex migration that
revokes all sessions and quarantines token records, a deployment-specific
credential-version fence, or an equivalent bounded mechanism. Record the
maximum stale-authority window and the decision before reopening traffic.
Grant rotation/quarantine behavior is unexecuted until that mechanism
exists; no synthetic provider result can substitute for it.
5. Test application rollback by stopping the restored web service, starting the
immediately previous digest-pinned web image against the restored backend,
@@ -141,21 +209,61 @@ callback or provider key.
There is no migration history/rollback command in this baseline. A valid
rehearsal therefore requires a parent-approved migration fixture and a chosen
rollback mechanism. At minimum:
rollback mechanism. Before starting, record the exact **baseline commit/image
and candidate commit/image**, the migration command, the backend image digest,
and the compatibility contract. The evidence must include commands equivalent
to these (with the operator's approved migration tool substituted explicitly):
```sh
baseline_commit=$(git rev-parse HEAD)
candidate_commit=<recorded-candidate-commit>
baseline_image=<recorded-digest-pinned-image>
candidate_image=<recorded-digest-pinned-image>
# approved migration command against the disposable candidate deployment:
<deploy-candidate-and-run-migration>
# restore pre-migration archive into a fresh volume/project:
<restore-pre-migration-volume>
# start baseline image against the restored volume:
<deploy-baseline-image-and-run-compatibility-checks>
```
At minimum:
1. back up the disposable volume before migration;
2. deploy the migration to the disposable backend;
2. deploy the candidate migration to the disposable backend;
3. run schema/functions and tenant-isolation checks;
4. restore the pre-migration volume into a fresh project (never in place);
5. start the previous application image and rerun the checks; and
6. record whether the migration is backward compatible and which data loss,
RPO, and RTO limits apply.
A `docker compose down -v` or a successful process restart is not migration
Placeholder commands are an evidence gap, not a passing rehearsal. A
`docker compose down -v` or a successful process restart is not migration
rollback evidence. T03's reference artifact rollback is a separate test and
must verify profile, importer version, manifest integrity, and live-reader
compatibility.
## Retention and cleanup
Before leaving the shell (and before the `trap` removes `$work`), export only
redacted logs, checksums, metrics, image digests, and the evidence table to a
protected location outside the temporary directory. Never copy either key
file or an admin key into that bundle:
```sh
evidence_dir="${EVIDENCE_DIR:?Set an approved protected evidence directory}"
umask 077
mkdir -p "$evidence_dir/$project"
cp "$work"/*.log "$work"/*.sha256 "$work"/*metrics* "$evidence_dir/$project/" 2>/dev/null || true
printf 'source_project=%s\nrestored_project=%s\n' "$project" "$restore_project" > "$evidence_dir/$project/projects.txt"
sha256sum "$evidence_dir/$project"/* > "$evidence_dir/$project/evidence.sha256"
printf 'retained_until_utc=<operator-set-expiry>\n' > "$evidence_dir/$project/retention.txt"
```
Confirm the bundle contains no `.env`, `application-keys`, admin key, token,
or unredacted provider output. Only after export and operator approval may the
cleanup trap remove both Compose projects, both volumes, and `$work`.
## Evidence template
Copy this table into the release record and attach only redacted logs/checksums: