138 lines
6.4 KiB
Markdown
138 lines
6.4 KiB
Markdown
# Community content moderation
|
|
|
|
SHD.build applies one moderation lifecycle to public comments, profile
|
|
biographies, custom avatars, guides, and build descriptions. The shared server
|
|
service lives in `src/lib/server/community-moderation.ts`. Content save actions
|
|
must use that service before publication.
|
|
|
|
## Publication decisions
|
|
|
|
The moderation service combines local rules with the configured provider
|
|
result:
|
|
|
|
- Clean content publishes without human review when automatic publication is
|
|
enabled.
|
|
- Banned terms, PII checks, link limits, provider flags, and configured score
|
|
thresholds send content to Community Oversight.
|
|
- Provider failures send content to Oversight when fail-closed behavior is
|
|
enabled.
|
|
- An allow term can suppress the matching local term rule. It cannot override a
|
|
provider flag or threshold.
|
|
- A user report sends visible content to Oversight. One report does not hide
|
|
the content because reports can be mistaken or abusive.
|
|
|
|
The service records every automated scan in `content_moderation_scans`.
|
|
Automated scans do not write to the human ISAC audit log. Human policy changes
|
|
and review decisions use the shared audit utility.
|
|
|
|
## Content storage
|
|
|
|
Comments use an explicit status:
|
|
|
|
- `visible`: public and approved
|
|
- `pending_review`: hidden while Oversight decides
|
|
- `removed`: unavailable after author removal or an Oversight decision
|
|
|
|
Long-form content uses immutable pending revisions. Profile and guide edits that
|
|
need review go into `moderated_content_revisions`; the public row keeps its last
|
|
approved text. Approval copies the reviewed revision into the public row.
|
|
Rejection preserves the revision and review evidence without publishing it.
|
|
|
|
Custom avatar uploads remain active after a clean image scan. A flagged image
|
|
or fail-closed scan error creates an `avatar_reviews` item and is unavailable
|
|
through the public avatar route until approval. An Agent report also creates a
|
|
review item, but leaves an already-published avatar visible while Oversight
|
|
decides.
|
|
|
|
## ISAC responsibilities
|
|
|
|
ISAC Site Controls stores the site-wide policy:
|
|
|
|
- provider scanning and per-content enablement;
|
|
- clean-content and provider-failure behavior;
|
|
- provider model;
|
|
- PII, link, and duplicate checks;
|
|
- category review and priority thresholds;
|
|
- allowed and banned local terms.
|
|
|
|
The term registry accepts single rules and batch imports. Batch imports use one
|
|
term or phrase per line, accept commas and semicolons as separators, infer word
|
|
or phrase matching, and add no more than 250 unique rules. Word rules match
|
|
exact forms: `kill` does not match `kills`, `killed`, or `killing`. The import
|
|
skips duplicates, records each created rule in the ISAC audit log, and leaves
|
|
the batch editor open for the next list. The registry provides server-side
|
|
search and displays 25 rules per page by default.
|
|
|
|
Each Site Controls category has its own save action. A save validates and
|
|
updates only that category, increments the moderation policy version when
|
|
applicable, and records a specific ISAC audit event with before and after
|
|
evidence. Unsaved changes in other categories remain local drafts.
|
|
|
|
Site Controls requires `site.settings.manage`. Community Oversight handles text
|
|
decisions through the existing comment and profile permissions. Avatar
|
|
Oversight handles custom images through `avatars.review`.
|
|
|
|
Moderators must record a reason when they approve, reject, remove, restore,
|
|
dismiss, or resolve a queued item. The action must append an audit event after
|
|
the content and review records change.
|
|
|
|
## User reports
|
|
|
|
Authenticated Agents can report visible content they do not own. The report
|
|
control must identify the content type and stable content ID, validate that the
|
|
content remains public, and create a pending `user_content_reports` row.
|
|
Duplicate pending reports from the same Agent for the same content are rejected.
|
|
|
|
A reviewer either dismisses the report and keeps the content or resolves the
|
|
report through a moderation action. Report details must not copy authentication
|
|
secrets or unnecessary personal information into the audit log.
|
|
|
|
## Adding a public UGC field
|
|
|
|
Guide authoring, build descriptions, and later public text fields must complete
|
|
this checklist:
|
|
|
|
1. Normalize and validate the submitted text before scanning.
|
|
2. Call `moderateCommunityText` with a supported content type.
|
|
3. Place the shared `ModerationDisclosure` beside the submission control.
|
|
4. Publish only an `approved` result.
|
|
5. Store held long-form text as an immutable revision so the last approved
|
|
version remains public.
|
|
6. Call `recordCommunityModerationScan` with the stable content ID, author, hash,
|
|
policy version, provider evidence, and local-rule matches.
|
|
7. Add the content type to the Community Oversight queue and permission checks.
|
|
8. Add the shared report control to the public rendering.
|
|
9. Delay follower, owner, or reply notifications until the content becomes
|
|
public.
|
|
10. Add approval, rejection, report dismissal, author notification, and audit
|
|
tests.
|
|
11. Confirm every public query selects approved content explicitly.
|
|
|
|
Do not add a route-specific word filter, provider client, threshold table, or
|
|
moderation status vocabulary. Extend the shared service and schema when a new
|
|
content type needs more evidence or a different review presentation.
|
|
|
|
## Operations and calibration
|
|
|
|
The API key stays in `OPENAI_MODERATION_API_KEY`. The server can use
|
|
`OPENAI_MECHANIC_API_KEY` as a temporary fallback, but production should use a
|
|
separate moderation credential.
|
|
|
|
Public submission notices link to the Terms & Privacy page instead of
|
|
summarizing provider data handling beside each form. The Terms & Privacy page
|
|
must not promise that OpenAI never retains or uses moderation inputs. It states
|
|
that OpenAI controls downstream processing under its then-current terms, data
|
|
controls, and account settings. If SHD.build enables an OpenAI data-sharing
|
|
option, update the legal copy before sending user content under that setting.
|
|
|
|
Threshold changes increment the policy version. Stored scans retain their model
|
|
and policy version so reviewers can interpret older decisions. Test threshold
|
|
changes against representative Division terminology before activation,
|
|
including weapon names, damage descriptions, kills, status effects, and build
|
|
strategy language.
|
|
|
|
Database migrations that add moderation states or revision storage follow the
|
|
repository database-preservation procedure: stop this project's development
|
|
server, verify the database path, create and verify a JSON backup, migrate,
|
|
check foreign keys and critical record counts, then restart and health-check.
|