feat: add automated community moderation
This commit is contained in:
@@ -0,0 +1,124 @@
|
||||
# Community content moderation
|
||||
|
||||
SHD.build applies one moderation lifecycle to public comments, profile
|
||||
biographies, custom avatars, guides, and build descriptions. The shared server
|
||||
service lives in `src/lib/server/community-moderation.ts`. Content save actions
|
||||
must use that service before publication.
|
||||
|
||||
## Publication decisions
|
||||
|
||||
The moderation service combines local rules with the configured provider
|
||||
result:
|
||||
|
||||
- Clean content publishes without human review when automatic publication is
|
||||
enabled.
|
||||
- Banned terms, PII checks, link limits, provider flags, and configured score
|
||||
thresholds send content to Community Oversight.
|
||||
- Provider failures send content to Oversight when fail-closed behavior is
|
||||
enabled.
|
||||
- An allow term can suppress the matching local term rule. It cannot override a
|
||||
provider flag or threshold.
|
||||
- A user report sends visible content to Oversight. One report does not hide
|
||||
the content because reports can be mistaken or abusive.
|
||||
|
||||
The service records every automated scan in `content_moderation_scans`.
|
||||
Automated scans do not write to the human ISAC audit log. Human policy changes
|
||||
and review decisions use the shared audit utility.
|
||||
|
||||
## Content storage
|
||||
|
||||
Comments use an explicit status:
|
||||
|
||||
- `visible`: public and approved
|
||||
- `pending_review`: hidden while Oversight decides
|
||||
- `removed`: unavailable after author removal or an Oversight decision
|
||||
|
||||
Long-form content uses immutable pending revisions. Profile and guide edits that
|
||||
need review go into `moderated_content_revisions`; the public row keeps its last
|
||||
approved text. Approval copies the reviewed revision into the public row.
|
||||
Rejection preserves the revision and review evidence without publishing it.
|
||||
|
||||
Custom avatar uploads remain active after a clean image scan. A flagged image
|
||||
or fail-closed scan error creates an `avatar_reviews` item and is unavailable
|
||||
through the public avatar route until approval. An Agent report also creates a
|
||||
review item, but leaves an already-published avatar visible while Oversight
|
||||
decides.
|
||||
|
||||
## ISAC responsibilities
|
||||
|
||||
ISAC Site Controls stores the site-wide policy:
|
||||
|
||||
- provider scanning and per-content enablement;
|
||||
- clean-content and provider-failure behavior;
|
||||
- provider model;
|
||||
- PII, link, and duplicate checks;
|
||||
- category review and priority thresholds;
|
||||
- allowed and banned local terms.
|
||||
|
||||
Site Controls requires `site.settings.manage`. Community Oversight handles text
|
||||
decisions through the existing comment and profile permissions. Avatar
|
||||
Oversight handles custom images through `avatars.review`.
|
||||
|
||||
Moderators must record a reason when they approve, reject, remove, restore,
|
||||
dismiss, or resolve a queued item. The action must append an audit event after
|
||||
the content and review records change.
|
||||
|
||||
## User reports
|
||||
|
||||
Authenticated Agents can report visible content they do not own. The report
|
||||
control must identify the content type and stable content ID, validate that the
|
||||
content remains public, and create a pending `user_content_reports` row.
|
||||
Duplicate pending reports from the same Agent for the same content are rejected.
|
||||
|
||||
A reviewer either dismisses the report and keeps the content or resolves the
|
||||
report through a moderation action. Report details must not copy authentication
|
||||
secrets or unnecessary personal information into the audit log.
|
||||
|
||||
## Adding a public UGC field
|
||||
|
||||
Guide authoring, build descriptions, and later public text fields must complete
|
||||
this checklist:
|
||||
|
||||
1. Normalize and validate the submitted text before scanning.
|
||||
2. Call `moderateCommunityText` with a supported content type.
|
||||
3. Place the shared `ModerationDisclosure` beside the submission control.
|
||||
4. Publish only an `approved` result.
|
||||
5. Store held long-form text as an immutable revision so the last approved
|
||||
version remains public.
|
||||
6. Call `recordCommunityModerationScan` with the stable content ID, author, hash,
|
||||
policy version, provider evidence, and local-rule matches.
|
||||
7. Add the content type to the Community Oversight queue and permission checks.
|
||||
8. Add the shared report control to the public rendering.
|
||||
9. Delay follower, owner, or reply notifications until the content becomes
|
||||
public.
|
||||
10. Add approval, rejection, report dismissal, author notification, and audit
|
||||
tests.
|
||||
11. Confirm every public query selects approved content explicitly.
|
||||
|
||||
Do not add a route-specific word filter, provider client, threshold table, or
|
||||
moderation status vocabulary. Extend the shared service and schema when a new
|
||||
content type needs more evidence or a different review presentation.
|
||||
|
||||
## Operations and calibration
|
||||
|
||||
The API key stays in `OPENAI_MODERATION_API_KEY`. The server can use
|
||||
`OPENAI_MECHANIC_API_KEY` as a temporary fallback, but production should use a
|
||||
separate moderation credential.
|
||||
|
||||
Public submission notices link to the Terms & Privacy page instead of
|
||||
summarizing provider data handling beside each form. The Terms & Privacy page
|
||||
must not promise that OpenAI never retains or uses moderation inputs. It states
|
||||
that OpenAI controls downstream processing under its then-current terms, data
|
||||
controls, and account settings. If SHD.build enables an OpenAI data-sharing
|
||||
option, update the legal copy before sending user content under that setting.
|
||||
|
||||
Threshold changes increment the policy version. Stored scans retain their model
|
||||
and policy version so reviewers can interpret older decisions. Test threshold
|
||||
changes against representative Division terminology before activation,
|
||||
including weapon names, damage descriptions, kills, status effects, and build
|
||||
strategy language.
|
||||
|
||||
Database migrations that add moderation states or revision storage follow the
|
||||
repository database-preservation procedure: stop this project's development
|
||||
server, verify the database path, create and verify a JSON backup, migrate,
|
||||
check foreign keys and critical record counts, then restart and health-check.
|
||||
Reference in New Issue
Block a user