Adversarial refresh

The Growth section of docs/PRODUCT.md promises that detection catalogs stay current through adversarial refresh. This page is the procedure. It is what an agent runs to make that promise real. Nothing here runs on a schedule (see docs/DECISIONS.md, "Agent-invoked only"); it runs when an agent, or the person driving one, decides to check.

1. Check staleness

python3 scripts/refresh_status.py

Stdlib, offline, always exits 0. It reports three signals as JSON:

wiki_sync: days since the local Wikipedia sync state was last updated.
parity_bench: days since the model-parity matrix in references/pipeline.md was last recorded.
newest_pattern_row: days since evals/adversarial-evals.json last gained a row.

Each carries last (an ISO date or null), days (or null), and a stale flag; the script never fails a build over it, and the flag is a prompt for judgment rather than a rule to obey mechanically. The thresholds it applies (wiki 90 days, bench 180 days, rows 60 days) are commented in the script itself as starting points.

2. If `wiki_sync` is stale

python3 scripts/wiki_sync.py check   # exit 0 = no updates, 1 = updates available
python3 scripts/wiki_sync.py diff    # structured JSON diff, when check exits 1

For each new word or phrase in the diff, follow "Add a banned phrase" in references/maintenance.md: eval rows first, then the scanner, then the catalog. Skip Wikipedia-specific patterns, such as broken wikitext, DOI formatting, or citation graffiti. Only genuine AI tells in general prose belong here.

3. If `parity_bench` is stale

Treat staleness as a prompt to double-check the recorded table against reality. It is not an automatic re-run trigger by itself: the binding rule in references/pipeline.md (Model Parity) already forces a re-run whenever the co-writer, mimic, or detector-pack model features change, independent of elapsed time. When a re-run is warranted, run the live matrix across both spectrums:

python3 evals/run_model_parity.py --task both --format both

Update the tiering table and the "Recorded results" tables in references/pipeline.md from the output, and update the standing conclusions in docs/PRODUCT.md's Model tiering section if they changed. The PARITY-* rows in evals/adversarial-evals.json gate the grader itself in dry-run mode. They do not substitute for the live run above:

python3 evals/run_model_parity.py --dry-run \
  --responses evals/fixtures/parity/canned_responses.json

4. Every miss becomes a row

A miss can come from a live bench output, a wiki-sync diff, or something caught by hand. Whichever it is, route it through references/commands/contribute.md. Contribute turns the exact specimen into a red-first eval row and a structured PR, so the miss becomes a permanent regression test instead of a one-off fix.

5. Record results

Model-parity runs: update references/pipeline.md's dated tables (the "Live matrix recorded YYYY-MM-DD" line and the two results tables under it) with the new date and numbers.
Behavioral or skill runs: follow the conventions already set in evals/TUNE-RESULTS.md. Name the harness version, split, runner, and judge. Then report per-case deltas rather than aggregate lift, per CLAUDE.md's Interpreting Results section.

Cadence guidance

There is no scheduler. Use refresh_status.py's thresholds as the trigger for a manual or agent-invoked check instead: wiki sync roughly quarterly, the parity bench roughly every six months (sooner on any model-feature change, per the binding rule above), and a look at the eval catalog if two months pass with no new row.

3.6 KiB Raw Blame History