shd.build/docs/CONTENT_MODERATION.md

Community content moderation

SHD.build applies one moderation lifecycle to public comments, profile biographies, custom avatars, guides, and build descriptions. The shared server service lives in src/lib/server/community-moderation.ts. Content save actions must use that service before publication.

Publication decisions

The moderation service combines local rules with the configured provider result:

• Clean content publishes without human review when automatic publication is enabled.
• Banned terms, PII checks, link limits, provider flags, and configured score thresholds send content to Community Oversight.
• Provider failures send content to Oversight when fail-closed behavior is enabled.
• An allow term can suppress the matching local term rule. It cannot override a provider flag or threshold.
• A user report sends visible content to Oversight. One report does not hide the content because reports can be mistaken or abusive.

The service records every automated scan in content_moderation_scans. Automated scans do not write to the human ISAC audit log. Human policy changes and review decisions use the shared audit utility.

Content storage

Comments use an explicit status:

• visible: public and approved
• pending_review: hidden while Oversight decides
• removed: unavailable after author removal or an Oversight decision

Long-form content uses immutable pending revisions. Profile and guide edits that need review go into moderated_content_revisions; the public row keeps its last approved text. Approval copies the reviewed revision into the public row. Rejection preserves the revision and review evidence without publishing it.

Custom avatar uploads remain active after a clean image scan. A flagged image or fail-closed scan error creates an avatar_reviews item and is unavailable through the public avatar route until approval. An Agent report also creates a review item, but leaves an already-published avatar visible while Oversight decides.

ISAC responsibilities

ISAC Site Controls stores the site-wide policy:

• provider scanning and per-content enablement;
• clean-content and provider-failure behavior;
• provider model;
• PII, link, and duplicate checks;
• category review and priority thresholds;
• allowed and banned local terms.

The term registry accepts single rules and batch imports. Batch imports use one term or phrase per line, accept commas and semicolons as separators, infer word or phrase matching, and add no more than 250 unique rules. Word rules match exact forms: kill does not match kills, killed, or killing. The import skips duplicates, records each created rule in the ISAC audit log, and leaves the batch editor open for the next list. The registry provides server-side search and displays 25 rules per page by default.

Each Site Controls category has its own save action. A save validates and updates only that category, increments the moderation policy version when applicable, and records a specific ISAC audit event with before and after evidence. Unsaved changes in other categories remain local drafts.

Site Controls requires site.settings.manage. Community Oversight handles text decisions through the existing comment and profile permissions. Avatar Oversight handles custom images through avatars.review.

Moderators must record a reason when they approve, reject, remove, restore, dismiss, or resolve a queued item. The action must append an audit event after the content and review records change.

User reports

Authenticated Agents can report visible content they do not own. The report control must identify the content type and stable content ID, validate that the content remains public, and create a pending user_content_reports row. Duplicate pending reports from the same Agent for the same content are rejected.

A reviewer either dismisses the report and keeps the content or resolves the report through a moderation action. Report details must not copy authentication secrets or unnecessary personal information into the audit log.

Adding a public UGC field

Guide authoring, build descriptions, and later public text fields must complete this checklist:

1. Normalize and validate the submitted text before scanning.
2. Call moderateCommunityText with a supported content type.
3. Place the shared ModerationDisclosure beside the submission control.
4. Publish only an approved result.
5. Store held long-form text as an immutable revision so the last approved version remains public.
6. Call recordCommunityModerationScan with the stable content ID, author, hash, policy version, provider evidence, and local-rule matches.
7. Add the content type to the Community Oversight queue and permission checks.
8. Add the shared report control to the public rendering.
9. Delay follower, owner, or reply notifications until the content becomes public.
10. Add approval, rejection, report dismissal, author notification, and audit tests.
11. Confirm every public query selects approved content explicitly.

Do not add a route-specific word filter, provider client, threshold table, or moderation status vocabulary. Extend the shared service and schema when a new content type needs more evidence or a different review presentation.

Operations and calibration

The API key stays in OPENAI_MODERATION_API_KEY. The server can use OPENAI_MECHANIC_API_KEY as a temporary fallback, but production should use a separate moderation credential.

Public submission notices link to the Terms & Privacy page instead of summarizing provider data handling beside each form. The Terms & Privacy page must not promise that OpenAI never retains or uses moderation inputs. It states that OpenAI controls downstream processing under its then-current terms, data controls, and account settings. If SHD.build enables an OpenAI data-sharing option, update the legal copy before sending user content under that setting.

Threshold changes increment the policy version. Stored scans retain their model and policy version so reviewers can interpret older decisions. Test threshold changes against representative Division terminology before activation, including weapon names, damage descriptions, kills, status effects, and build strategy language.

Database migrations that add moderation states or revision storage follow the repository database-preservation procedure: stop this project's development server, verify the database path, create and verify a JSON backup, migrate, check foreign keys and critical record counts, then restart and health-check.