jb/shd.build
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: migrate secrets to Infisical
Validate and deploy production / validate (push) Canceled after 0s
Details
Validate and deploy production / deploy (push) Canceled after 0s
Details
Validate and deploy staging / validate (push) Failing after 22s
Details
Validate and deploy staging / deploy (push) Skipped
Details

Browse Source
This commit is contained in:
jb
2026-08-02 01:19:41 -04:00
parent 9089f4a257
commit 25d9241b51
34 changed files with 2353 additions and 691 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/DEPLOYMENT.md
+149 -73
View File
@@ -2,8 +2,8 @@
SHD.build runs in a Node 24 container through the official SvelteKit Node
adapter. The image includes application code, migration and notification
scripts, and native runtime dependencies. Environment files, databases, backups,
and user uploads remain outside the image.
scripts, the Infisical CLI, and native runtime dependencies. Databases, backups,
user uploads, and Universal Auth bootstrap credentials remain outside the image.
## Persistent data
@@ -30,7 +30,7 @@ Staging defaults to:
- Adapter canonical origin: `https://stage.shd.build`
Guide ratings and reports use Cloudflare Turnstile after repeated anonymous
activity. Configure both values in each encrypted environment:
activity. Configure both values under `/app` in each Infisical environment:
```text
TURNSTILE_SITE_KEY=<public site key>
@@ -44,12 +44,13 @@ and the request address to Cloudflare for verification; it never exposes
valid secret, a visitor who reaches the challenge threshold cannot submit more
anonymous ratings or Guide reports.
Edit its encrypted configuration locally:
Edit staging configuration in Infisical project
`2ee94b27-e599-4194-9797-3372d42e53f6`:
```sh
npm run env:decrypt:staging
# Fill in Discord and SMTP values.
npm run env:encrypt:staging
```text
Environment: staging
Application path: /app
Scheduler path: /shared/scheduler
```
Add `https://stage.shd.build/auth/discord/callback` to the Discord application's
@@ -59,9 +60,43 @@ and configure the host's TLS reverse proxy to forward `stage.shd.build` to port
a containerized reverse proxy or tunnel can reach it; firewall the port to
trusted LAN/tunnel sources when the host is reachable from other networks.
The trusted staging runner decrypts `.env.staging` into a mode-`0600` runtime
file. Gitea Actions holds the decryption key; Git and the container image do
not.
The trusted staging runner writes separate mode-`0600` Universal Auth client-ID
and client-secret files for the app and scheduler. Compose mounts each pair only
into its assigned service. Each container exchanges those credentials for a
short-lived access token and writes the assigned paths to mode-`0600` temporary
JSON files. The entrypoint rejects unknown secret names, deletes the token,
transfers the exports to the dedicated `shd-runtime` UID 10001, and drops root
before parsing them. The launcher deletes the files before validation,
migration, or startup. Startup fails if the mounted credential files are
readable by UID 10001 or by their group or other users. Runtime-created data
retains group access for rollback to the previous container UID.
| Machine identity | Environment | Injected paths | Gitea variable | Gitea secret |
| -------------------------------- | ----------- | --------------------------- | ------------------------------------------ | ---------------------------------------------- |
| `shd-build-staging-app` | `staging` | `/app`, `/shared/scheduler` | `INFISICAL_STAGING_APP_CLIENT_ID` | `INFISICAL_STAGING_APP_CLIENT_SECRET` |
| `shd-build-staging-scheduler` | `staging` | `/shared/scheduler` | `INFISICAL_STAGING_SCHEDULER_CLIENT_ID` | `INFISICAL_STAGING_SCHEDULER_CLIENT_SECRET` |
| `shd-build-production-app` | `prod` | `/app`, `/shared/scheduler` | `INFISICAL_PRODUCTION_APP_CLIENT_ID` | `INFISICAL_PRODUCTION_APP_CLIENT_SECRET` |
| `shd-build-production-scheduler` | `prod` | `/shared/scheduler` | `INFISICAL_PRODUCTION_SCHEDULER_CLIENT_ID` | `INFISICAL_PRODUCTION_SCHEDULER_CLIENT_SECRET` |
The self-hosted Free plan allows five machine identities and does not include
Custom Project Roles. The four runtime identities use the built-in Viewer role,
which can read the complete project. Service-specific paths limit injected
values, but they do not restrict what a stolen root credential could request.
Use an Enterprise license with Custom Project Roles, or split runtime scopes
into separate projects, before treating identity authorization as least
privilege.
An Infisical outage does not stop a running container. A new container or direct
restart fails closed until `https://secrets.jb.codes` becomes available and the
identity can authenticate.
Keep the old `.env.*.runtime` files and `DOTENV_PRIVATE_KEY_*` Gitea secrets
only through the first successful staging and production cutovers. The rollback
path may need them to start the previous image. After both environments pass
direct restart checks, rotate every migrated provider credential in Infisical,
remove the old dotenv private keys and `SCHEDULER_TOKEN_PRODUCTION` from Gitea,
and delete local and hosted dotenv key material. Removing encrypted files from
the current Git tree does not revoke values stored in Git history.
The staging container disables default-user seeding. On a brand-new staging
database, the existing bootstrap policy grants Admin to the first real account
@@ -99,31 +134,51 @@ Restart the runner session after adding it to the Docker group.
In `https://git.jb.codes/jb/shd.build/settings/actions`:
1. Add repository secret `DOTENV_PRIVATE_KEY_STAGING`, using the matching value
from the ignored local `.env.keys`.
2. Optionally add repository variable `STAGE_DEPLOY_PATH`. It defaults to
1. Add repository secrets `INFISICAL_STAGING_APP_CLIENT_SECRET` and
`INFISICAL_STAGING_SCHEDULER_CLIENT_SECRET`.
2. Add repository variables `INFISICAL_STAGING_APP_CLIENT_ID` and
`INFISICAL_STAGING_SCHEDULER_CLIENT_ID`.
3. Optionally add repository variable `STAGE_DEPLOY_PATH`. It defaults to
`/opt/shd-build-stage`.
3. Optionally add repository variable `STAGE_PORT`. It defaults to `5569`.
4. Ensure Actions are enabled and the `stage-deploy` runner is online.
4. Optionally add repository variable `STAGE_PORT`. It defaults to `5569`.
5. Ensure Actions are enabled and the `stage-deploy` runner is online.
`.gitea/workflows/deploy-staging.yml` runs for pushes to `main` and manual
dispatches. The trusted staging runner builds each commit-tagged image locally,
so the workflow needs no registry credentials. Gitea assigns `gitea.run_number`
to `SHD_BUILD_NUMBER`; each new workflow run receives the next integer, while
retries of the same run retain their number.
dispatches. The trusted staging runner builds each image locally with a tag that
contains the commit, workflow run, and attempt, so the workflow needs no
registry credentials and a retry cannot overwrite the active image. Gitea
assigns `gitea.run_number` to `SHD_BUILD_NUMBER`; each new workflow run receives
the next integer, while retries of the same run retain their number.
Staging and production deployments, ISAC exports, and production-to-staging
synchronization share the `shd-build-host-maintenance` concurrency group. Gitea
runs only one of these database maintenance workflows on the deployment host at
a time and never cancels one in progress.
## Deployment lifecycle
The workflow:
The workflow stages Compose, credentials, and release metadata in an isolated
candidate directory. It leaves the canonical deployment files unchanged until
the candidate passes preflight, backup, and migration. The workflow then:
1. installs dependencies, runs `npm run check`, and produces the Node build;
1. installs dependencies, runs `npm run check` and `npm test`, and produces the
Node build;
2. builds the candidate container while the current site remains online;
3. stops only the staging application and notification scheduler;
4. creates a timestamped JSON backup if the staging database exists;
5. applies additive, repeatable migrations;
6. starts the application and scheduler with the new image;
7. waits for `/api/health` to report healthy;
8. retains the logs and exits unsuccessfully if health verification fails.
3. starts one-off app and scheduler containers to verify Infisical access and
validate the candidate environment;
4. stops only the staging application and notification scheduler;
5. creates a timestamped JSON backup if the staging database exists;
6. applies additive, repeatable migrations in a one-off container;
7. promotes the candidate files to the canonical deployment paths;
8. starts the application and scheduler with the new image;
9. validates the app environment and reruns the repeatable migration in the app
entrypoint before serving;
10. verifies `/api/health` and scheduler liveness;
11. restores the previous canonical files and verifies the previous app and
scheduler after a failed stop, backup, migration, startup, or health check.
On a failed health check, the deploy script prints the final 150 application log
lines and exits unsuccessfully. It does not create a separate log artifact.
The outage covers backup, migration, and container startup. Migrations may also
remove records that use retired data formats when a release explicitly ends
@@ -145,11 +200,10 @@ feed remain published; the importer performs no automatic removal. The scheduler
runs notification delivery before it starts the bounded Ubisoft request, so a
slow provider does not postpone notifications for the current tick.
`deploy/compose.production.yml` declares `UBISOFT_NEWS_IMPORT_ENABLED=true`,
while the staging Compose file declares `false`. Development does not set the
flag and remains disabled because the importer requires the exact value `true`.
The application process evaluates the flag when it handles
`/api/internal/scheduler`.
The runtime launcher sets `UBISOFT_NEWS_IMPORT_ENABLED=true` for production and
`false` for staging. Development does not set the flag, so the importer remains
disabled because it requires the exact value `true`. The application process
evaluates the flag when it handles `/api/internal/scheduler`.
## Prelaunch operation
@@ -216,10 +270,11 @@ channel, keep it out of Git, then copy it to the staging host and run:
/secure/path/shd-stage-bootstrap-YYYY-MM-DDTHH-MM-SSZ.json
```
The bootstrap flow stops the app and scheduler, creates a safety backup of the
target, verifies that staging is unused, imports the entire development database
and uploads, and restarts both services. Delete the transfer copy from the
staging host after verifying the imported record counts and critical entities.
The bootstrap flow requires both services to stop, copies the bundle into the
private data volume, assigns it to the runtime UID, creates a safety backup,
verifies that staging is unused, imports the entire development database and
uploads, deletes the volume copy, and restarts both services. Delete the host
transfer copy after verifying the imported record counts and critical entities.
The copy under the local ignored backup directory remains the recovery source.
Because this is a full initial clone, development session tokens and account
@@ -279,16 +334,21 @@ environment, port, and deployment workflow:
- Database: `/app/data/shd-build-production.db`
- Persistent volume: `shd-build-production-data`
Routine pushes leave production data unchanged. The production deploy stops its
application and scheduler, creates a timestamped JSON backup, applies repeatable
migrations, restarts, and verifies database health. Keep
`docker compose down --volumes` away from production.
Routine pushes preserve production data unless an included migration removes a
retired prototype format. The production deploy stops its application and
scheduler, creates a timestamped JSON backup, applies repeatable migrations,
restarts, and verifies database health. Keep `docker compose down --volumes`
away from production.
Add these repository Action secrets:
- `DOTENV_PRIVATE_KEY_PRODUCTION`, matching the ignored local `.env.keys`
- `SCHEDULER_TOKEN_PRODUCTION`, a long random value used only by the internal
scheduler endpoint
- `INFISICAL_PRODUCTION_APP_CLIENT_SECRET`
- `INFISICAL_PRODUCTION_SCHEDULER_CLIENT_SECRET`
Add the matching repository variables:
- `INFISICAL_PRODUCTION_APP_CLIENT_ID`
- `INFISICAL_PRODUCTION_SCHEDULER_CLIENT_ID`
Optional repository variables are `PRODUCTION_DEPLOY_PATH` (default
`/opt/shd-build-production`) and `PRODUCTION_PORT` (default `3102`). The
@@ -299,12 +359,12 @@ workflow run number supplies the production `SHD_BUILD_NUMBER`.
Prepare the host directory once from an interactive SSH session:
```sh
sudo install -d -o jb -g jb -m 0750 /opt/shd-build-production
sudo install -d -o act_runner -g act_runner -m 0750 /opt/shd-build-production
```
Generate the scheduler secret locally without printing it into a shared log,
then enter it directly in Gitea's repository-secret form. Do not commit it or
reuse the staging scheduler token.
Store a separate production `SCHEDULER_TOKEN` under Infisical path
`/shared/scheduler`. The application and scheduler fetch the same value from
that path. Do not reuse the staging token.
### Initial production bootstrap
@@ -322,9 +382,11 @@ Copy the resulting ignored JSON bundle to the deployment host over SSH, then:
/secure/path/shd-production-bootstrap-YYYY-MM-DDTHH-MM-SSZ.json
```
The importer verifies the bundle target, creates a production safety backup, and
refuses to replace a production database containing users, builds, audit events,
or proposals. Remove the remote transfer copy after verification.
The bootstrap script requires both services to stop, stages the bundle in the
private data volume with runtime-only access, and removes that copy after the
import. The importer verifies the bundle target, creates a production safety
backup, and refuses to replace a production database containing users, builds,
audit events, or proposals. Remove the remote host copy after verification.
### Promote one News article
@@ -340,13 +402,14 @@ managed-image metadata, and image bytes. It excludes comments, reports, user
records, Builds, and ISAC data.
After deploying the matching application schema, stop the production app and
scheduler and mount the transferred bundle into a one-off application container.
Run `scripts/article-transfer.mjs import` with `ARTICLE_IMPORT_ENABLED=true` and
scheduler. Copy the transferred bundle into `/app/data` with `docker cp`, assign
it to `shd-runtime:node` with mode `0600` through a root one-off container, then
run `scripts/article-transfer.mjs import` with `ARTICLE_IMPORT_ENABLED=true` and
`--confirm-article-import`. The importer creates a production database backup
before opening the database, resolves the article owner by stable ID or
username, rejects ID and slug conflicts, verifies image checksums, and appends
`editorial_content / news.promoted`. Restart and health-check both services,
then remove the transferred bundle from the host.
`editorial_content / news.promoted`. Remove both bundle copies, restart the app
and scheduler, and health-check both services.
### Pull production ISAC data into development
@@ -373,11 +436,11 @@ safety backups before replacement. Deployments do not run this flow.
Image Library assets use stable IDs and can be promoted additively from local
development. Export a bundle after finishing local uploads and metadata edits:
Set the server-only `REMOVE_BG_API_KEY` in each encrypted environment that
accepts ISAC uploads. It is sent only from the server in the `X-Api-Key` header.
Transparent icon uploads skip remove.bg. Opaque sources use its preview output,
then pass through local trimming, white normalization, WebP conversion, and
storage.
Set the server-only `REMOVE_BG_API_KEY` under `/app` in each Infisical
environment that accepts ISAC uploads. The server sends it only in the
`X-Api-Key` header. Transparent icon uploads skip remove.bg. Opaque sources use
its preview output, then pass through local trimming, white normalization, WebP
conversion, and storage.
The application stores a remove.bg capacity snapshot and refreshes it from the
internal scheduler. `REMOVE_BG_CREDIT_CHECK_HOURS` controls the account lookup
@@ -399,19 +462,33 @@ embedded database is unlocked, and run the merge using the target image:
```sh
# Staging
cd /opt/shd-build-stage
docker compose -f compose.staging.yml stop scheduler app
docker compose -f compose.staging.yml run --rm \
-v /secure/path:/import:ro app \
node scripts/managed-images-import.mjs /import/managed-images.json --confirm-merge
docker compose -f compose.staging.yml up -d app scheduler
docker compose --env-file .env.deploy -f compose.staging.yml stop scheduler app
docker compose --env-file .env.deploy -f compose.staging.yml create app
app_container=$(docker compose --env-file .env.deploy -f compose.staging.yml ps --all --quiet app)
docker cp /secure/path/managed-images.json "$app_container:/app/data/.managed-images-import.json"
docker compose --env-file .env.deploy -f compose.staging.yml run --rm --no-deps \
--entrypoint sh --user root app -c \
'chown shd-runtime:node /app/data/.managed-images-import.json && chmod 600 /app/data/.managed-images-import.json'
docker compose --env-file .env.deploy -f compose.staging.yml run --rm --no-deps app \
node scripts/managed-images-import.mjs /app/data/.managed-images-import.json --confirm-merge
docker compose --env-file .env.deploy -f compose.staging.yml run --rm --no-deps \
--entrypoint rm --user root app -f /app/data/.managed-images-import.json
docker compose --env-file .env.deploy -f compose.staging.yml up -d app scheduler
# Production
cd /opt/shd-build-production
docker compose -f compose.production.yml stop scheduler app
docker compose -f compose.production.yml run --rm \
-v /secure/path:/import:ro app \
node scripts/managed-images-import.mjs /import/managed-images.json --confirm-merge
docker compose -f compose.production.yml up -d app scheduler
docker compose --env-file .env.deploy -f compose.production.yml stop scheduler app
docker compose --env-file .env.deploy -f compose.production.yml create app
app_container=$(docker compose --env-file .env.deploy -f compose.production.yml ps --all --quiet app)
docker cp /secure/path/managed-images.json "$app_container:/app/data/.managed-images-import.json"
docker compose --env-file .env.deploy -f compose.production.yml run --rm --no-deps \
--entrypoint sh --user root app -c \
'chown shd-runtime:node /app/data/.managed-images-import.json && chmod 600 /app/data/.managed-images-import.json'
docker compose --env-file .env.deploy -f compose.production.yml run --rm --no-deps app \
node scripts/managed-images-import.mjs /app/data/.managed-images-import.json --confirm-merge
docker compose --env-file .env.deploy -f compose.production.yml run --rm --no-deps \
--entrypoint rm --user root app -f /app/data/.managed-images-import.json
docker compose --env-file .env.deploy -f compose.production.yml up -d app scheduler
```
Each import creates a full target database backup before changes. The importer
@@ -422,9 +499,8 @@ After reviewing an intentional metadata or file replacement, add
verifying both environments because it may contain unpublished visual assets and
creator identifiers.
The `.env.staging` and `.env.production` npm import commands are intended for
maintenance performed directly in an environment where those database and
storage paths are mounted:
The staging and production npm import commands fetch `/app` from Infisical. Run
them only where the target database and storage paths are mounted:
```sh
npm run images:import:staging -- /path/to/bundle.json --confirm-merge
docs/architecture.md
+7 -15
View File
@@ -911,12 +911,9 @@ The notification router sends Data Change review items to agents who hold
delivery for each authorized queue. Submission creates Immediate alerts;
`oversight_notification_queue` stores consolidated alerts.
Production must run the idempotent dispatcher once per minute from one scheduler
instance:
```text
* * * * * cd /path/to/shd.build && npm run notifications:oversight:production
```
The production Compose scheduler calls the internal scheduler endpoint once per
minute. That endpoint runs the idempotent dispatcher in the application process,
which owns the embedded-database connection. Do not run a second host cron job.
The dispatcher groups due rows by reviewer, queue, cadence, and delivery window.
Daily rows use the next 10:00 AM wall-clock boundary in `America/New_York`,
@@ -941,12 +938,7 @@ default welcome introduces News, Database, Builds, Guides, Builder, ISAC
Contributor Mode, and the SHD Discord community.
The registration request delivers Immediate definitions.
`automated_system_message_deliveries` stores definitions with a delay.
Production must run the delayed dispatcher once per minute:
```text
* * * * * cd /path/to/shd.build && npm run notifications:system:production
```
Only one dispatcher instance should run for the embedded database. Delivered
rows retain both their delivery timestamp and generated notification ID.
`automated_system_message_deliveries` stores definitions with a delay. The same
Compose scheduler tick runs the delayed dispatcher. Only one scheduler instance
should call the embedded-database application. Delivered rows retain both their
delivery timestamp and generated notification ID.