shd.build/docs/DEPLOYMENT.md

# Container, staging, and production deployment

SHD.build runs in a Node 24 container through the official SvelteKit Node
adapter. The image includes application code, migration and notification
scripts, and native runtime dependencies. Environment files, databases, backups,
and user uploads remain outside the image.

## Persistent data

The staging compose project uses the named volume `shd-build-stage-data`,
mounted at `/app/data`. It contains:

- `shd-build-staging.db` and its embedded-database support files;
- timestamped JSON backups under `backups/`;
- normalized user avatars under `avatars/`;
- uploaded game-data icons under `game-icons/`.

Keep this volume during routine deployments. `docker compose down` without
`--volumes` preserves it. Before a manual backup or restore, stop the `app` and
`scheduler` services because both access the embedded database.

## Staging environment

Staging defaults to:

- Public origin: `https://stage.shd.build`
- Reverse-proxy upstream: host port `5569`
- Database: `/app/data/shd-build-staging.db`
- Passkey relying-party ID: `shd.build`
- Adapter canonical origin: `https://stage.shd.build`

Guide ratings and reports use Cloudflare Turnstile after repeated anonymous
activity. Configure both values in each encrypted environment:

```text
TURNSTILE_SITE_KEY=<public site key>
TURNSTILE_SECRET_KEY=<server verification secret>
```

Register the exact staging and production hostnames in the Turnstile widget. The
browser receives only `TURNSTILE_SITE_KEY`. The server sends a challenge token
and the request address to Cloudflare for verification; it never exposes
`TURNSTILE_SECRET_KEY` or stores the raw address as visitor identity. Without a
valid secret, a visitor who reaches the challenge threshold cannot submit more
anonymous ratings or Guide reports.

Edit its encrypted configuration locally:

```sh
npm run env:decrypt:staging
# Fill in Discord and SMTP values.
npm run env:encrypt:staging
```

Add `https://stage.shd.build/auth/discord/callback` to the Discord application's
allowed OAuth redirect URIs. Point the staging DNS record at the deployment host
and configure the host's TLS reverse proxy to forward `stage.shd.build` to port
`5569` on the staging host. Docker publishes this port on the host interfaces so
a containerized reverse proxy or tunnel can reach it; firewall the port to
trusted LAN/tunnel sources when the host is reachable from other networks.

The trusted staging runner decrypts `.env.staging` into a mode-`0600` runtime
file. Gitea Actions holds the decryption key; Git and the container image do
not.

The staging container disables default-user seeding. On a brand-new staging
database, the existing bootstrap policy grants Admin to the first real account
that signs in instead of writing generated credentials into deployment logs.

## Gitea Actions runner

Gitea Actions requires an `act_runner`. Register a trusted runner on the staging
host from the repository's **Settings → Actions → Runners** page. The host needs
Node/npm, Docker Engine, and Docker Compose v2. The runner account must be able
to use Docker and write to the deployment directory.

Use runner labels that provide an isolated validation job and a host deployment
job:

```text
ubuntu-latest:docker://node:24-bookworm,stage-deploy:host
```

The `stage-deploy:host` label is intentional: the deployment step needs the
staging host's Docker daemon and persistent named volume. Only attach this
trusted runner to repositories allowed to deploy to that host. Gitea warns that
Docker daemon access is effectively host-level authority.

Prepare the default deployment directory:

```sh
sudo install -d -o act_runner -g act_runner -m 0750 /opt/shd-build-stage
sudo usermod -aG docker act_runner
```

Restart the runner session after adding it to the Docker group.

## Repository Actions configuration

In `https://git.jb.codes/jb/shd.build/settings/actions`:

1. Add repository secret `DOTENV_PRIVATE_KEY_STAGING`, using the matching value
   from the ignored local `.env.keys`.
2. Optionally add repository variable `STAGE_DEPLOY_PATH`. It defaults to
   `/opt/shd-build-stage`.
3. Optionally add repository variable `STAGE_PORT`. It defaults to `5569`.
4. Ensure Actions are enabled and the `stage-deploy` runner is online.

`.gitea/workflows/deploy-staging.yml` runs for pushes to `main` and manual
dispatches. The trusted staging runner builds each commit-tagged image locally,
so the workflow needs no registry credentials. Gitea assigns `gitea.run_number`
to `SHD_BUILD_NUMBER`; each new workflow run receives the next integer, while
retries of the same run retain their number.

## Deployment lifecycle

The workflow:

1. installs dependencies, runs `npm run check`, and produces the Node build;
2. builds the candidate container while the current site remains online;
3. stops only the staging application and notification scheduler;
4. creates a timestamped JSON backup if the staging database exists;
5. applies additive, repeatable migrations;
6. starts the application and scheduler with the new image;
7. waits for `/api/health` to report healthy;
8. retains the logs and exits unsuccessfully if health verification fails.

The outage covers backup, migration, and container startup. Migrations may also
remove records that use retired data formats when a release explicitly ends
compatibility. The deployment backup remains the recovery source for those
records; the application does not repair or reinterpret them at runtime. The
scheduler calls a bearer-authenticated endpoint on the application service once
per minute. Scheduled message delivery therefore uses the web process's existing
embedded-database connection instead of opening the locked database from a
second process. The endpoint runs automated system messages, Oversight
consolidation, email digest checks, and the Ubisoft News importer. Docker does
not publish the endpoint through a host port.

When enabled in production, the Ubisoft importer keeps its own persisted due
time, so the one-minute tick starts a feed synchronization no more than once per
hour after a successful run. When a synchronization fails, the importer stores
the last error and consecutive failure count, then retries after 15 minutes with
exponential backoff capped at six hours. Source items that disappear from the
feed remain published; the importer performs no automatic removal. The scheduler
runs notification delivery before it starts the bounded Ubisoft request, so a
slow provider does not postpone notifications for the current tick.

`deploy/compose.production.yml` declares `UBISOFT_NEWS_IMPORT_ENABLED=true`,
while the staging Compose file declares `false`. Development does not set the
flag and remains disabled because the importer requires the exact value `true`.
The application process evaluates the flag when it handles
`/api/internal/scheduler`.

## Prelaunch operation

The launch-control migration creates `site_launch_settings` with these defaults
for a database that does not contain the row:

- Coming Soon: enabled
- Account creation: enabled
- News, Database, Builds, Guides, and Builder: enabled

The migration uses an insert-if-missing seed. Later deployments preserve the
operator's settings, and configuration changes take effect without a restart. A
full environment bootstrap copies the source row. Canonical ISAC game-data
imports leave the target environment's launch settings unchanged.

Use this operator flow after the first deployment:

1. Open `/sign-in`; the launch gate keeps that route available.
2. Sign in with an existing Administrator account.
3. Open ISAC → System Configuration → Site Controls.
4. Set account creation and public-section availability.
5. Review the complete change set, then confirm it.
6. Use `PREVIEW PAGE` to inspect `/coming-soon` with the authenticated
   `ENTER FULL SITE` return action. The preview lists each section's current
   availability.
7. Verify `/coming-soon` and the affected public routes in a signed-out browser
   session before public release. Sign in and verify the same disabled routes
   remain unavailable.

Keep Coming Soon enabled while preparing production. Section controls can be set
in advance because the launch gate takes precedence for signed-out traffic. Set
Account Creation to Disabled if prelaunch access must remain limited to existing
identities. Switch the Launch Gate to Public Launch after verification. Disabled
sections remain absent from navigation and unavailable by direct URL for every
user, including authenticated agents. Authentication bypasses only the Coming
Soon gate.

Turn Coming Soon on to restore the promotional gate. Site-control changes append
`system_configuration / site_access.updated` audit events, including the
operator and complete before/after settings.

## Initial development-data bootstrap

Initialize the first staging release from the development database, including
users, roles, builds, guides, proposals, audit history, avatars, and uploaded
game-data icons. The staging importer refuses a target database that contains
users, builds, audits, or proposals. It also requires the source and target
table sets to match exactly; snapshots with retired or unknown tables are
rejected.

First stop this project's development server. The embedded database must not be
open while its complete transfer bundle is created:

```sh
npm run stage:bootstrap:export
```

The command writes a timestamped, ignored JSON bundle under `data/backups/`. The
bundle includes user records and password hashes. Transfer it over an encrypted
channel, keep it out of Git, then copy it to the staging host and run:

```sh
/opt/shd-build-stage/bootstrap-staging.sh \
  /secure/path/shd-stage-bootstrap-YYYY-MM-DDTHH-MM-SSZ.json
```

The bootstrap flow stops the app and scheduler, creates a safety backup of the
target, verifies that staging is unused, imports the entire development database
and uploads, and restarts both services. Delete the transfer copy from the
staging host after verifying the imported record counts and critical entities.
The copy under the local ignored backup directory remains the recovery source.

Because this is a full initial clone, development session tokens and account
data also enter staging. Staging and development remain independent from that
point forward.

## Pulling canonical ISAC data into development

Use the manually triggered **Export staging ISAC data** workflow in Gitea
Actions. It briefly stops staging database writers, exports only canonical
builder-source tables plus uploaded game-data icons, restarts staging, and
publishes a `shd-stage-isac-data` artifact for 14 days.

Download `isac-stage.json` from that workflow run, stop this project's
development server, and run:

```sh
npm run isac:import -- /path/to/isac-stage.json --confirm-replace-isac-data
```

The importer automatically creates both a complete development database backup
and an ISAC-only snapshot before replacing canonical data. It does **not**
replace local users, roles, sessions, builds, guides, comments, notifications,
proposals, contributor agreements, audit history, or system-message settings.
Restart development only after the command reports both backup paths and the
imported row/icon counts.

For a local read-only snapshot without importing:

```sh
npm run isac:export
```

## Manual staging operations

From the deployment directory:

```sh
docker compose --env-file .env.deploy -f compose.staging.yml ps
docker compose --env-file .env.deploy -f compose.staging.yml logs -f app
docker compose --env-file .env.deploy -f compose.staging.yml stop scheduler app
docker compose --env-file .env.deploy -f compose.staging.yml run --rm --no-deps app node scripts/db-backup.mjs
docker compose --env-file .env.deploy -f compose.staging.yml up -d app scheduler
```

Do not use `docker compose down --volumes` for staging. That deletes the named
volume containing the embedded database and uploads.

## Production promotion

Production uses a separate compose project, volume, database filename, runtime
environment, port, and deployment workflow:

- Public origin: `https://shd.build`
- Reverse-proxy upstream: host port `3102`
- Deployment directory: `/opt/shd-build-production`
- Database: `/app/data/shd-build-production.db`
- Persistent volume: `shd-build-production-data`

Routine pushes leave production data unchanged. The production deploy stops its
application and scheduler, creates a timestamped JSON backup, applies repeatable
migrations, restarts, and verifies database health. Keep
`docker compose down --volumes` away from production.

Add these repository Action secrets:

- `DOTENV_PRIVATE_KEY_PRODUCTION`, matching the ignored local `.env.keys`
- `SCHEDULER_TOKEN_PRODUCTION`, a long random value used only by the internal
  scheduler endpoint

Optional repository variables are `PRODUCTION_DEPLOY_PATH` (default
`/opt/shd-build-production`) and `PRODUCTION_PORT` (default `3102`). The
`deploy-production.yml` workflow deploys production on pushes to `main` and
manual dispatches using the trusted `stage-deploy` host runner. Its Gitea
workflow run number supplies the production `SHD_BUILD_NUMBER`.

Prepare the host directory once from an interactive SSH session:

```sh
sudo install -d -o jb -g jb -m 0750 /opt/shd-build-production
```

Generate the scheduler secret locally without printing it into a shared log,
then enter it directly in Gitea's repository-secret form. Do not commit it or
reuse the staging scheduler token.

### Initial production bootstrap

The production bootstrap clones current development data and uploads. Stop this
project's development server before creating the bundle:

```sh
npm run production:bootstrap:export
```

Copy the resulting ignored JSON bundle to the deployment host over SSH, then:

```sh
/opt/shd-build-production/bootstrap-production.sh \
  /secure/path/shd-production-bootstrap-YYYY-MM-DDTHH-MM-SSZ.json
```

The importer verifies the bundle target, creates a production safety backup, and
refuses to replace a production database containing users, builds, audit events,
or proposals. Remove the remote transfer copy after verification.

### Promote one News article

Export a locally authored News article by ID or slug while the development
server is stopped:

```sh
npm run article:export -- <article-id-or-slug>
```

The ignored bundle contains the article draft, immutable revisions, search row,
managed-image metadata, and image bytes. It excludes comments, reports, user
records, Builds, and ISAC data.

After deploying the matching application schema, stop the production app and
scheduler and mount the transferred bundle into a one-off application container.
Run `scripts/article-transfer.mjs import` with `ARTICLE_IMPORT_ENABLED=true` and
`--confirm-article-import`. The importer creates a production database backup
before opening the database, resolves the article owner by stable ID or
username, rejects ID and slug conflicts, verifies image checksums, and appends
`editorial_content / news.promoted`. Restart and health-check both services,
then remove the transferred bundle from the host.

### Pull production ISAC data into development

Manually run **Export production ISAC data** in Gitea Actions and download its
`shd-production-isac-data` artifact. Stop the local development server, then:

```sh
npm run isac:import -- /path/to/isac-production.json --confirm-replace-isac-data
```

The importer creates a complete local database backup and an ISAC-only snapshot
before replacing canonical game data. It leaves accounts, content, proposals,
notifications, settings, and audit history untouched.

### Pull production ISAC data into staging

Manually run **Replace staging ISAC data from production** in Gitea Actions. The
workflow exports a consistent production snapshot, restarts production, then
stops staging and imports it. The staging importer creates full and ISAC-only
safety backups before replacement. Deployments do not run this flow.

### Promote managed images without replacing other data

Image Library assets use stable IDs and can be promoted additively from local
development. Export a bundle after finishing local uploads and metadata edits:

Set the server-only `REMOVE_BG_API_KEY` in each encrypted environment that
accepts ISAC uploads. It is sent only from the server in the `X-Api-Key` header.
Transparent icon uploads skip remove.bg. Opaque sources use its preview output,
then pass through local trimming, white normalization, WebP conversion, and
storage.

The application stores a remove.bg capacity snapshot and refreshes it from the
internal scheduler. `REMOVE_BG_CREDIT_CHECK_HOURS` controls the account lookup
interval and defaults to six hours. Low-capacity alerts require both remaining
pools to cross their thresholds: `REMOVE_BG_LOW_CREDIT_THRESHOLD` defaults to
five paid credits, and `REMOVE_BG_LOW_FREE_CALL_THRESHOLD` defaults to ten free
calls. Alerts are mandatory ISAC notifications for active Administrators and are
deduplicated for each warning or critical episode.

```sh
npm run images:export
```

The command writes a timestamped JSON bundle under `data/backups/`. It contains
only managed-image metadata and processed WebP files. Transfer that bundle to
the deployment host, stop the target `app` and `scheduler` services so the
embedded database is unlocked, and run the merge using the target image:

```sh
# Staging
cd /opt/shd-build-stage
docker compose -f compose.staging.yml stop scheduler app
docker compose -f compose.staging.yml run --rm \
  -v /secure/path:/import:ro app \
  node scripts/managed-images-import.mjs /import/managed-images.json --confirm-merge
docker compose -f compose.staging.yml up -d app scheduler

# Production
cd /opt/shd-build-production
docker compose -f compose.production.yml stop scheduler app
docker compose -f compose.production.yml run --rm \
  -v /secure/path:/import:ro app \
  node scripts/managed-images-import.mjs /import/managed-images.json --confirm-merge
docker compose -f compose.production.yml up -d app scheduler
```

Each import creates a full target database backup before changes. The importer
adds new stable IDs, skips identical assets, and blocks conflicting records.
After reviewing an intentional metadata or file replacement, add
`--update-existing`. Imported changes append `image.imported` or
`image.import_updated` audit events. Remove the transferred bundle after
verifying both environments because it may contain unpublished visual assets and
creator identifiers.

The `.env.staging` and `.env.production` npm import commands are intended for
maintenance performed directly in an environment where those database and
storage paths are mounted:

```sh
npm run images:import:staging -- /path/to/bundle.json --confirm-merge
npm run images:import:production -- /path/to/bundle.json --confirm-merge
```