global setup and configuration.
This commit is contained in:
@@ -0,0 +1,122 @@
|
||||
# Teach Calibration Game (A/B Preference Elicitation)
|
||||
|
||||
An agent-hosted micro-game that calibrates voice preferences the user can state but
|
||||
never bothered to write samples for. Far lower effort than `teach`: harvest bootstraps
|
||||
samples, the A/B game calibrates dimension preferences, and both feed the voice card.
|
||||
`teach` offers this game (`references/commands/teach.md`, step 3) when sample coverage
|
||||
or confidence stays thin; this file is the canonical flow and rationale.
|
||||
|
||||
## Deterministic set
|
||||
|
||||
Five dimensions are reachable by reversible, seed-deterministic transforms of a base
|
||||
passage (`scripts/calibrate_pairs.py`): `contractions`, `em_dash`, `sentence_length`,
|
||||
`connectives`, `staccato`. Every transform preserves the base passage's must-preserve
|
||||
constraints (`scripts/extract_constraints.py`) or the command exits 3 rather than
|
||||
silently dropping a fact.
|
||||
|
||||
Register and warmth are NOT in this set — they need LLM generation, not a mechanical
|
||||
transform, and are out of scope for `calibrate_pairs.py`. If the game wants to cover
|
||||
those dimensions, generate the pair with a model and skip the determinism/constraint
|
||||
guarantees this script gives for free.
|
||||
|
||||
## Game flow
|
||||
|
||||
1. Pick base passages from the user's own harvested samples (`references/harvest.md`)
|
||||
when available; otherwise fall back to neutral seed texts. Paragraph-sized (60-150
|
||||
words) so a round takes seconds to read.
|
||||
2. For the dimension needing the most attention, run:
|
||||
```bash
|
||||
python3 scripts/calibrate_pairs.py generate --base passage.txt --dimension DIM --seed N
|
||||
```
|
||||
Increment the seed each round for that dimension so repeated rounds against the same
|
||||
base passage still vary. If the command exits 3 ("dimension not expressible in this
|
||||
passage"), pick a different base passage or move to the next dimension — do not force
|
||||
a transform through.
|
||||
|
||||
The output also carries `a_flags`/`b_flags`: the `banned_phrase_scan.py` category names
|
||||
each variant trips (empty when clean). A generated variant tripping the scanner is not a
|
||||
reason to decline the pair — see "Voice overrides defaults" below.
|
||||
3. Present `a_text` and `b_text` to the user as "is A or B closer to how you'd write
|
||||
this?" without naming the dimension or which side is which pole (avoid anchoring).
|
||||
Accept "neither" as a valid answer.
|
||||
|
||||
Randomize which pole is shown as "A" and which as "B" each round — a seeded
|
||||
shuffle keyed on the round (e.g. `pair_id`) so it is reproducible, not a fresh
|
||||
coin flip you can't reconstruct later. Always displaying `calibrate_pairs.py`'s
|
||||
own `a_text` as "A" would let the user learn the game's fixed pole ordering
|
||||
after a few rounds, reintroducing the anchoring this step is trying to avoid.
|
||||
Record the mapping — which of the script's `a_text`/`b_text` was actually shown
|
||||
as "A" this round — alongside the round so the recorded `choice` and
|
||||
`a_label`/`b_label` in step 4 resolve to the correct pole regardless of what
|
||||
was displayed on screen.
|
||||
4. Append one line to `.unslop/voice/<name>/preferences.jsonl`:
|
||||
```json
|
||||
{"pair_id": "...", "dimension": "...", "choice": "a|b|neither", "ts": "...",
|
||||
"a_label": "<A's pole from transform_applied>", "b_label": "<B's pole>"}
|
||||
```
|
||||
`a_label`/`b_label` come straight off `calibrate_pairs.py`'s `transform_applied`
|
||||
(`"<dimension>:<pole>"` names B's pole; A sits at the dimension's other named pole).
|
||||
Recording them is what lets `calibrate_score.py` report a semantic direction instead
|
||||
of a bare "a"/"b" tally, and what makes conflict detection possible.
|
||||
5. After each round, run:
|
||||
```bash
|
||||
python3 scripts/calibrate_score.py --preferences preferences.jsonl
|
||||
```
|
||||
Stop playing a dimension once its `confidence >= 0.7` OR it has reached `k >= 9`
|
||||
observations, whichever comes first. Below `k = 5` observations a dimension is always
|
||||
reported `"insufficient"` regardless of how lopsided the tally looks — five is the
|
||||
floor before a lean means anything. A dimension can also come back `"tied"` with
|
||||
`preferred: null` when its top two tallies are exactly equal at `k >= 5` — that's a
|
||||
real "no lean yet" result, not insufficient data, and not a coin flip the aggregator
|
||||
should silently resolve one way. Play it a couple more rounds. Replaying the same
|
||||
round (same `pair_id`) is safe: the aggregator dedups by `pair_id`, keeping only the
|
||||
latest row by `ts`, so a retried write never double-counts or lets a stale choice win.
|
||||
6. Use `--next` to pick which dimension to play next:
|
||||
```bash
|
||||
python3 scripts/calibrate_score.py --preferences preferences.jsonl --next
|
||||
```
|
||||
Ordering: fewest observations first, then lowest confidence, tie-broken by the fixed
|
||||
dimension order (`contractions, em_dash, sentence_length, connectives, staccato`) so
|
||||
the choice is reproducible from the same preferences file.
|
||||
7. When a sample-derived profile exists (`profile.json`, WP10a), run:
|
||||
```bash
|
||||
python3 scripts/calibrate_score.py --preferences preferences.jsonl --profile profile.json
|
||||
```
|
||||
Any confident (`>= 0.7`) stated preference that contradicts the profile's measured
|
||||
value comes back as a `conflicts[]` record naming both values and their provenance,
|
||||
e.g. "stated preference for 'short' (confidence 0.70) contradicts
|
||||
avg_sentence_length=22.4 measured from samples." Surface every conflict to the user
|
||||
verbatim — do not silently pick a winner. Let the user say which one wins; record
|
||||
that choice's provenance on the card.
|
||||
8. Hand the final per-dimension summary to the teach-card step. Every claim on the card
|
||||
sourced from this game carries provenance `"stated-preference"`, distinct from
|
||||
`"measured-from-samples"` values pulled from the profile directly.
|
||||
|
||||
## Voice overrides defaults
|
||||
|
||||
A generated variant can legitimately trip `banned_phrase_scan.py` — a staccato pole reads
|
||||
as `anti_slop_register` (three short fragments in a row is the scanner's own AI-cadence
|
||||
tell), a plain connective can occasionally read as `filler_opener`. Do not decline these
|
||||
pairs or quietly steer the user away from the flagged pole; the scanner encodes a default
|
||||
register guard, and this game exists specifically to find out when a user's real voice
|
||||
sits outside that default.
|
||||
|
||||
When a user's choices consistently land on a pole that `a_flags`/`b_flags` marks as
|
||||
flagged (e.g. they keep picking the staccato side even though it trips
|
||||
`anti_slop_register`), record the preference together with its flags. Surface the tension
|
||||
to the user once, not on every round — something like: "you prefer staccato; the default
|
||||
register guard flags it as an AI cadence tell — your voice wins in your profile." Do not
|
||||
re-litigate it every round once the user has answered.
|
||||
|
||||
On the final per-dimension summary (step 8), mark that dimension's provenance
|
||||
`"user-preference overrides register guard"` instead of a bare `"stated-preference"`, so
|
||||
the teach-card step (and anyone reading the card later) can see that the scanner's default
|
||||
was deliberately overridden, not silently ignored. This is the same conflict-surfacing
|
||||
philosophy as the profile conflicts below: voice beats defaults, but only visibly.
|
||||
|
||||
## Provenance discipline
|
||||
|
||||
The card must always be able to say, for any voice claim, whether it came from the
|
||||
user's own samples (measured) or from a choice the user made in this game (stated).
|
||||
Conflicts are a feature, not a bug to smooth over — surfacing them is the whole point
|
||||
of running both a profile and a preference game.
|
||||
@@ -0,0 +1,59 @@
|
||||
# /unslop cleanup
|
||||
|
||||
Surface AI patterns as reviewable edits the user accepts or rejects, rather than
|
||||
a finished rewrite. Agent-invoked, never a background daemon. Route here on
|
||||
`/unslop cleanup` or when the user wants suggestions instead of a silent rewrite.
|
||||
Set `$INPUT` to the source.
|
||||
|
||||
## Report-only variant
|
||||
|
||||
When the user says "flag only", "just detect", "audit only", "don't change
|
||||
anything", or passes `--report`, change nothing. Scan and report:
|
||||
|
||||
```bash
|
||||
python3 scripts/banned_phrase_scan.py <<< "$INPUT"
|
||||
python3 scripts/structure_scan.py <<< "$INPUT"
|
||||
python3 scripts/silhouette_scan.py <<< "$INPUT"
|
||||
python3 scripts/readability_metrics.py <<< "$INPUT"
|
||||
```
|
||||
|
||||
Report each issue by span, category, severity, and why it reads as machine-
|
||||
written, separating clear problems from register-dependent judgment calls (the
|
||||
audit-only shape in SKILL.md **Output Format**). Re-read negations, scope, and
|
||||
certainty yourself; the scanners are necessary, not sufficient.
|
||||
|
||||
## Suggestion flow
|
||||
|
||||
```bash
|
||||
python3 scripts/suggest.py document.md # emit suggestions
|
||||
python3 scripts/suggest.py doc.md --apply-replacements repl.json # merge model replacements
|
||||
python3 scripts/check_suggestions.py suggestions.json # blocking contract gates
|
||||
```
|
||||
|
||||
- **Detection is cheap and deterministic.** `suggest.py` runs both scanners and
|
||||
emits LSP-style suggestions `{span, severity, category, rationale,
|
||||
suggested_replacement, phrased_as_question}`, ordered and non-overlapping.
|
||||
- **Replacement generation is delegated to a stronger model.** `suggest.py`
|
||||
leaves `suggested_replacement` null; the stronger model fills them, merged back
|
||||
via `--apply-replacements`.
|
||||
- **Hard findings** become direct replacements; **soft findings** are register-
|
||||
dependent, so their rationale is phrased as a question
|
||||
(`phrased_as_question: true`).
|
||||
- **Suggestions are surfaced to the user, never silently applied.**
|
||||
|
||||
## Contract gates
|
||||
|
||||
`check_suggestions.py` is the blocking contract. Every replacement clears all
|
||||
four gates or it is rejected:
|
||||
|
||||
- `span-minimality`: the edit changes only its span (a whole-sentence rewrite
|
||||
fails).
|
||||
- `replacement-scanner`: each replacement passes both scanners in isolation and
|
||||
adds no new violation in context.
|
||||
- `accept-all`: applying every suggestion yields a document that passes both
|
||||
scanners with `validate_preservation` exit 0 versus the original.
|
||||
- `span-overlap`: spans must not overlap.
|
||||
|
||||
Non-English input is declined the way the scanners decline it: `suggest.py`
|
||||
returns `non_english: true` with no suggestions. For a full rewrite instead of
|
||||
suggestions, route to `references/commands/rewrite.md`.
|
||||
@@ -0,0 +1,51 @@
|
||||
# /unslop contribute
|
||||
|
||||
Turn one AI-ism caught in the wild into a scanner contract example and a structured PR. Route
|
||||
here on "contribute a pattern", "add this tell", or "I found a new AI-ism".
|
||||
Everything stays offline until the user approves both publication and the final
|
||||
PR body. Read `references/contribute.md` for the precheck exit codes, redaction
|
||||
discipline, row-diff verification, and the non-maintainer fork path.
|
||||
|
||||
## Flow
|
||||
|
||||
1. **Precheck.** `python3 scripts/contribute.py precheck /abs/path/snippet.txt`.
|
||||
Stop if it exits 3. The tell is already covered, so add a REC row rather
|
||||
than opening a fresh false-negative row.
|
||||
|
||||
2. **User confirmation gate #1 — publication.** Show the exact snippet and ask
|
||||
whether it may go in a public repo. Offer redaction hints from
|
||||
`python3 scripts/extract_constraints.py < snippet.txt`. Redact only with
|
||||
explicit approval, and keep the tell byte-for-byte intact.
|
||||
|
||||
3. **Scaffold** the working bundle under `.unslop/contrib/<slug>/` (do not
|
||||
commit it):
|
||||
```bash
|
||||
python3 scripts/contribute.py scaffold --snippet /abs/path/snippet.txt \
|
||||
--tell "exact substring" --category significance_inflation \
|
||||
--pattern-name exact-substring-slug --redact "Alice=NAME"
|
||||
```
|
||||
|
||||
4. **Implement eval-first**, following `references/maintenance.md`: copy
|
||||
`row_fn.json` into `evals/fixtures/contracts/scanner-examples.json`, renumber from the live
|
||||
maxima, then set `exact_total` to the new `examples` length. Add the
|
||||
literal-use FP example (and a REC example if a word is gated
|
||||
behind a collocation). Confirm the FN example is **red** before you touch the
|
||||
scanner, then update `scripts/banned_phrase_scan.py` and
|
||||
`references/taboo-phrases.md` until it goes **green** while the FP example holds.
|
||||
|
||||
5. **Verify:** `python3 scripts/contribute.py verify --bundle .unslop/contrib/<slug>`.
|
||||
It refuses TODO markers, checks specimen fidelity, and captures the
|
||||
red-to-green transition.
|
||||
|
||||
6. **Full deterministic gate battery:** `python3 evals/check.py --full`.
|
||||
|
||||
7. **Render the PR body:**
|
||||
`python3 scripts/contribute.py report --bundle .unslop/contrib/<slug> > pr-body.md`.
|
||||
|
||||
8. **User confirmation gate #2 — publication of the PR.** Show the final PR
|
||||
body, get approval, then branch, `git add` the three changed files, commit,
|
||||
and use `gh pr create` (or the non-maintainer fork path in
|
||||
`references/contribute.md`). Network access belongs to the host agent.
|
||||
|
||||
The resulting branch contains the contract example, scanner change, and taboo-catalog
|
||||
entry.
|
||||
@@ -0,0 +1,66 @@
|
||||
# /unslop mimic
|
||||
|
||||
Single-pass voiced writing. Draft or rewrite in a taught voice, then clear the
|
||||
full gate battery. Route here on "write this like me", "match my voice", or
|
||||
"mimic this author" once a voice card exists. Read `references/mimic.md` for card
|
||||
anatomy, scoring, and failure modes.
|
||||
|
||||
## Flow
|
||||
|
||||
1. Load the situation sheet the task needs. Read `card.md` for the always-on core
|
||||
(rhythm, contraction habits, punctuation, openers, the Never list) and the
|
||||
one `card/<situation>.md` sheet that matches what you are writing. Load only
|
||||
what the task needs; the card is layered so generation stays cheap.
|
||||
2. Draft or rewrite in that voice. Follow the card's measured markers and
|
||||
verbatim snippets; do not invent a habit the samples cannot support.
|
||||
3. Run the output through **every** removal gate:
|
||||
```bash
|
||||
python3 scripts/banned_phrase_scan.py <<< "$OUTPUT"
|
||||
python3 scripts/structure_scan.py <<< "$OUTPUT"
|
||||
python3 scripts/readability_metrics.py <<< "$OUTPUT"
|
||||
python3 scripts/validate_preservation.py original.txt "$OUTPUT_FILE"
|
||||
python3 scripts/diff_check.py original.txt "$OUTPUT_FILE"
|
||||
```
|
||||
4. Score the voice:
|
||||
```bash
|
||||
python3 scripts/voice_score.py --profile profile.json "$OUTPUT_FILE"
|
||||
```
|
||||
|
||||
## The rule
|
||||
|
||||
A mimic that scores well on voice but trips a slop gate is **rejected**. Voice
|
||||
never buys an exemption from the constitution — the removal gates are hard, the
|
||||
voice score is a guide. Meaning preservation versus the original draft is its own
|
||||
hard gate, never blended into the voice score.
|
||||
|
||||
## Refine — when one pass is not enough
|
||||
|
||||
Most users teach once and mimic cheaply thereafter. When a single pass keeps
|
||||
scoring short of the target voice, hill-climb with the refine loop — worth the
|
||||
cost only then:
|
||||
|
||||
```bash
|
||||
python3 evals/run_mimic_refine.py --samples samples/ --draft draft.md --out OUT
|
||||
```
|
||||
|
||||
LIVE (default) calls `--generate-cmd` (default `claude -p`) once per beam per
|
||||
iteration; `--candidates-dir DIR` runs the loop logic with no model calls. Hard
|
||||
gates (banned-phrase, structure, draft→candidate preservation, copy-gate, 150-
|
||||
word floor) discard a candidate regardless of score; survivors are scored against
|
||||
a held-out DEV split with the full `voice_score` composite, and the best-so-far
|
||||
never regresses. The **divergence guard** halts and sets
|
||||
`reward_hacking_warning` when the A composite improves while DEV worsens for two
|
||||
iterations — the reward-hacking signature. Confirm a win only when
|
||||
`mimic_stats.py` shows the CI lower bound > 0 and p < 0.05. Internals, baselines,
|
||||
and the failure table are in `references/mimic.md`.
|
||||
|
||||
## Voice check
|
||||
|
||||
"Does this sound like me?" — score only, change nothing. Run
|
||||
`python3 scripts/voice_score.py --profile .unslop/voice/<name>/profile.json
|
||||
--impostors evals/fixtures/voice/impostors --seed 7 <draft>` and report the
|
||||
composite (lower is more them), the GI rank, and the two or three metric
|
||||
deltas that explain the score, in plain words ("your sentences run longer than
|
||||
usual; contractions match"). No rewrite unless asked. This is the cheapest
|
||||
voice interaction and the usual one after teach: check drafts often, commission
|
||||
rewrites rarely.
|
||||
@@ -0,0 +1,46 @@
|
||||
# /unslop rewrite
|
||||
|
||||
The default command. Read `references/core-contract.md`; it is authoritative.
|
||||
Use two passes: diagnose, then repair confirmed findings and validate.
|
||||
|
||||
Input arrives as an argument, a file path, or stdin. Set `$INPUT` to the source
|
||||
and `$OUTPUT` to your rewrite.
|
||||
|
||||
## Pass 1 — diagnose
|
||||
|
||||
1. Extract the facts and constraints that must survive:
|
||||
```bash
|
||||
python3 scripts/extract_constraints.py <<< "$INPUT"
|
||||
```
|
||||
2. Scan the source so the rewrite answers evidence, not a hunch:
|
||||
```bash
|
||||
python3 scripts/banned_phrase_scan.py <<< "$INPUT"
|
||||
python3 scripts/structure_scan.py <<< "$INPUT"
|
||||
python3 scripts/silhouette_scan.py <<< "$INPUT"
|
||||
python3 scripts/readability_metrics.py <<< "$INPUT"
|
||||
```
|
||||
Pass `--genre docs` or `--genre social` only when the input truly belongs to
|
||||
that genre. Add `banned_phrase_scan.py --include-quoted` only when the user
|
||||
wants quoted examples audited too.
|
||||
3. Read the selected preset only for voice. It cannot authorize a finding.
|
||||
4. Classify exact spans as confirmed findings or protected source. Do not
|
||||
fact-check, use the current date, or treat missing support as an AI tell.
|
||||
Scanner rows remain candidates until contextual review confirms a defect.
|
||||
|
||||
## Pass 2 — reconstruct
|
||||
|
||||
Apply the core contract. Edit only a sentence containing a confirmed finding,
|
||||
make the smallest repair, and copy all other sentences byte-for-byte. If there
|
||||
are no findings, return the source exactly. Undo any edit that changes a fact,
|
||||
meaning, register, attribution, or protected domain phrase.
|
||||
|
||||
## Validate
|
||||
|
||||
Run the gate battery and apply the blocking semantics in
|
||||
`references/core-contract.md` **Validation**.
|
||||
|
||||
Return the cleaned text only, unless the user asked for the strict analysis
|
||||
block in SKILL.md **Output Format**.
|
||||
|
||||
For multi-agent execution, use `references/pipeline.md`; for orchestrated
|
||||
detector packs, use `references/packs/`.
|
||||
@@ -0,0 +1,77 @@
|
||||
# /unslop teach
|
||||
|
||||
Build a reusable voice from a writer's samples. **Agent-driven end to end**: the
|
||||
user supplies only approvals and answers, never a directory or a command. Route
|
||||
here on "teach my voice", "learn how I write", or when the user offers samples to
|
||||
learn from. Any samples the user provides are fair game; there are no rights or
|
||||
attestation checks.
|
||||
|
||||
Read `references/mimic.md` for card anatomy, sample requirements, and scoring
|
||||
before you finish; this file is the flow.
|
||||
|
||||
## 1. Gather samples (harvest)
|
||||
|
||||
If the user offers none, default to bootstrapping. Harvest their transcripts and
|
||||
writing folders per `references/harvest.md`:
|
||||
|
||||
```bash
|
||||
python3 scripts/harvest_samples.py SOURCE [SOURCE...] -o candidates.json
|
||||
python3 scripts/harvest_classify.py --candidates candidates.json --mode heuristic
|
||||
```
|
||||
|
||||
Present the ranked candidates with their sources and flags; the user approves or
|
||||
rejects each. Treat `suspect_ai` candidates skeptically — that human review is
|
||||
the contamination defense, so never auto-approve a flagged paste. Copy only
|
||||
approved samples into `.unslop/voice/<name>/samples/`. The adapter internals,
|
||||
contamination tripwire, and privacy rules live in `references/harvest.md`.
|
||||
|
||||
## 2. Build the profile and card
|
||||
|
||||
Create `.unslop/voice/<name>/` yourself (it is gitignored) and run:
|
||||
|
||||
```bash
|
||||
python3 scripts/voice_profile.py samples/ -o profile.json
|
||||
python3 scripts/voice_card.py --profile profile.json --samples samples/ --out . --name <name> --provenance
|
||||
```
|
||||
|
||||
`voice_profile.py` is the machine fingerprint the scorer refers to.
|
||||
`voice_card.py` writes the layered card the generating model follows (`card.md`
|
||||
plus `card/<situation>.md` sheets) and, with `--provenance`, a `provenance.json`
|
||||
audit trail. Both scripts read only `.txt` and `.md` files; convert samples
|
||||
first. `voice_card.py` refuses (exit 2) when the profile does not describe the
|
||||
supplied samples, so a stale profile can never drive a card.
|
||||
|
||||
## 3. Surface confidence and gaps
|
||||
|
||||
Show the user `card.md`. If the profile is `low_confidence` (thin or cross-genre
|
||||
samples), say the voice is provisional and ask for more same-genre samples before
|
||||
trusting a mimic. For an **Uncovered** dimension the target writing will need,
|
||||
ask for one short sample that exercises it — and only those the task requires, not
|
||||
all ten. Prompt templates:
|
||||
|
||||
| Dimension | Prompt |
|
||||
|-----------|--------|
|
||||
| explaining-technical | "Share something you wrote explaining how a thing works." |
|
||||
| anecdote | "Share a few sentences telling a small story that happened to you." |
|
||||
| argument | "Share something where you defended a claim." |
|
||||
| disagreement | "Share something where you pushed back on an idea." |
|
||||
| praise | "Share something where you recommended something you liked." |
|
||||
| hedging-uncertainty | "Share something where you were unsure and said so." |
|
||||
| numbers-data | "Share something you wrote that works with quantities." |
|
||||
| addressing-reader | "Share a note or instructions written to a reader." |
|
||||
|
||||
`openings` and `closings` are structural and covered as soon as one document
|
||||
exists, so they need no prompt. The full canonical table lives in
|
||||
`references/mimic.md`.
|
||||
|
||||
When coverage or confidence stays thin and the user cannot supply more samples,
|
||||
offer the A/B calibration game (`references/calibrate.md`): it elicits voice
|
||||
preferences through reversible pair choices and feeds them to the card as
|
||||
`stated-preference` provenance, distinct from `measured-from-samples`.
|
||||
|
||||
## 4. Close with a scored demo
|
||||
|
||||
Prove the loop before the user trusts it: mimic one paragraph per
|
||||
`references/commands/mimic.md`, run `scripts/voice_score.py` and the scanners,
|
||||
and show the results. A voiced demo that trips a slop gate fails the loop; do not
|
||||
ship it with an excuse.
|
||||
@@ -0,0 +1,106 @@
|
||||
# Contributing a New AI-ism — internals
|
||||
|
||||
The routed procedure lives in `references/commands/contribute.md`. This
|
||||
exhaustive companion covers every command flag, the redaction discipline, the
|
||||
row-diff verification, and the non-maintainer fork path. Keep it offline until
|
||||
the user has approved both publication and the final PR body.
|
||||
|
||||
## 1. Precheck
|
||||
|
||||
```bash
|
||||
python3 scripts/contribute.py precheck /absolute/path/to/snippet.txt
|
||||
```
|
||||
|
||||
If the snippet is already flagged, the command exits 3 and prints the covering
|
||||
patterns/categories. Stop there. Add a REC example instead of a new
|
||||
false-negative example.
|
||||
|
||||
## 2. User Confirmation Gate #1
|
||||
|
||||
Show the exact snippet to the user and ask whether it may be published in a
|
||||
public repo. Offer redaction hints from:
|
||||
|
||||
```bash
|
||||
python3 scripts/extract_constraints.py < /absolute/path/to/snippet.txt
|
||||
```
|
||||
|
||||
Redact names, numbers, or private details only with explicit approval. The tell
|
||||
must remain byte-for-byte intact.
|
||||
|
||||
## 3. Scaffold
|
||||
|
||||
```bash
|
||||
python3 scripts/contribute.py scaffold \
|
||||
--snippet /absolute/path/to/snippet.txt \
|
||||
--tell "exact substring" \
|
||||
--category significance_inflation \
|
||||
--pattern-name exact-substring-slug \
|
||||
--redact "Alice=NAME"
|
||||
```
|
||||
|
||||
The bundle is written under `.unslop/contrib/<slug>/`. Do not publish it. It is
|
||||
working material for the agent and reviewer.
|
||||
|
||||
## 4. Implement the Pattern
|
||||
|
||||
Follow `references/maintenance.md` in this order:
|
||||
|
||||
1. Copy `.unslop/contrib/<slug>/row_fn.json` into
|
||||
`evals/fixtures/contracts/scanner-examples.json`.
|
||||
2. Renumber the copied example from the live maxima in that table; do not keep
|
||||
the `CONTRIB-FN-*` bundle id in the committed contract. Set `exact_total` to
|
||||
the resulting `examples` array length.
|
||||
3. Add the literal-use FP example for the category, and add a REC example if an existing
|
||||
word is being gated behind collocations.
|
||||
4. Run the scanner contract before implementation and confirm the FN example is
|
||||
red while the FP and any REC examples encode the intended boundary.
|
||||
5. Update the scanner and `references/taboo-phrases.md`.
|
||||
6. Re-run `python3 scripts/contribute.py verify --bundle .unslop/contrib/<slug>`.
|
||||
7. Before treating the example as ready, diff it against
|
||||
`.unslop/contrib/<slug>/row_fn.json` and verify that only expected suite
|
||||
fields changed, such as the id, category grouping, or row ordering; the
|
||||
specimen stdin and assertion intent must still match because redaction or
|
||||
implementation work does not authorize a silent change to the reported tell.
|
||||
|
||||
## 5. Verify
|
||||
|
||||
```bash
|
||||
python3 scripts/contribute.py verify --bundle .unslop/contrib/<slug>
|
||||
```
|
||||
|
||||
Verify refuses reports with TODO markers, checks specimen fidelity, captures the
|
||||
red-to-green transition for the FN row, and records offline gate tails.
|
||||
|
||||
## 6. Run the Full Gate Battery
|
||||
|
||||
```bash
|
||||
python3 evals/check.py --full
|
||||
```
|
||||
|
||||
## 7. Render the Report
|
||||
|
||||
```bash
|
||||
python3 scripts/contribute.py report --bundle .unslop/contrib/<slug> > /tmp/pr-body.md
|
||||
```
|
||||
|
||||
## 8. User Confirmation Gate #2
|
||||
|
||||
Show the final PR body to the user. Only after the user approves publication:
|
||||
|
||||
```bash
|
||||
git switch -c add-<slug>
|
||||
git add evals/fixtures/contracts/scanner-examples.json scripts/banned_phrase_scan.py references/taboo-phrases.md
|
||||
git commit -m "Add <slug> AI-ism pattern"
|
||||
gh pr create --title "Add <category> pattern: <tell>" --body-file /tmp/pr-body.md
|
||||
```
|
||||
|
||||
For non-maintainers:
|
||||
|
||||
```bash
|
||||
gh repo fork --remote
|
||||
git push --set-upstream <fork-remote> add-<slug>
|
||||
gh pr create --repo <upstream-owner>/unslop --head <fork-owner>:add-<slug> --body-file /tmp/pr-body.md
|
||||
```
|
||||
|
||||
The `gh` commands are for the host agent after approval. Scripts and evals must
|
||||
never call `gh` or any network command.
|
||||
@@ -0,0 +1,91 @@
|
||||
# UNSLOP core rewrite contract
|
||||
|
||||
## Goal
|
||||
|
||||
Repair concrete AI-writing or clarity defects. Leave everything else unchanged;
|
||||
prefer a no-op to an uncertain edit.
|
||||
|
||||
## Findings
|
||||
|
||||
Quote the smallest defective span and explain the contextual defect: formulaic
|
||||
framing, empty or inflated abstraction, unsupported rhetorical certainty,
|
||||
opaque or mixed metaphor, misleading heading, needless repetition, or clear
|
||||
internal contradiction or ambiguity. Track every hard scanner match and other
|
||||
defect as confirmed or protected, with a reason; repair every confirmed span.
|
||||
One repair does not excuse another nearby defect.
|
||||
|
||||
Do not fact-check or infer truth from the date or outside knowledge. Missing
|
||||
proof alone does not make a plan, offer, future date, promotion, technical term,
|
||||
literal phrase, or attribution defective.
|
||||
|
||||
Compare the source's claims. Qualify a status contradicted by observations.
|
||||
Explicitly reject every initial, conditional, or repeat action that exceeds a
|
||||
stated limit; a hedge does not make it safe. Preserve evidence and quantities.
|
||||
Keep attributed claims intact and state conflicts separately. Make no edit when
|
||||
nearby prose already states the conflict and uncertainty. Attribution does not
|
||||
protect a separate operative recommendation.
|
||||
|
||||
A scanner match alone never authorizes an edit. Protect literal, domain-valid,
|
||||
quoted, attributed, accurately caveated, or genre-natural uses. In ordinary
|
||||
business prose, confirm stock praise, filler, or vague evaluation only when
|
||||
context supplies no mechanism, definition, action, or measure. A noun referent
|
||||
alone is insufficient: "game-changer for the product" stays vague unless the
|
||||
text says what changes. "Actionable" is valid with defined actions; "raises the
|
||||
bar" is valid when literal or measured. Soft cadence and document-shape scores
|
||||
never authorize edits alone.
|
||||
|
||||
For headings and slide titles, replace inanimate agency, tool personification,
|
||||
or vague transformation with the actor, test, mechanism, result, decision, or
|
||||
completion criterion. Renaming the same abstraction is not a repair. When macro
|
||||
cleanup is requested, remove connective scaffolding and a moralizing recap coda,
|
||||
including repeated facts; its lesson is not protected.
|
||||
|
||||
Inspect slogans and closing calls to action even when tools are quiet. Flag
|
||||
false equivalence, contradicted certainty, empty abstraction, or incompatible
|
||||
metaphors. Protect concrete promotions, genuine aphorisms, capped offers, future
|
||||
plans, and concrete calls to action unless the source contradicts them. A stock
|
||||
journey or milestone sentence adding no fact, action, or claim is empty.
|
||||
|
||||
## Rewrite
|
||||
|
||||
Edit only sentences with confirmed findings, using the smallest repair. Copy
|
||||
every other sentence byte-for-byte in its original order and paragraph. With no
|
||||
findings, return the source exactly.
|
||||
|
||||
Preserve facts, quantities, dates, names, quotations, citations, code, units,
|
||||
scope, uncertainty, attribution, register, and meaning. Add no claims, advice,
|
||||
personality, anecdote, certainty, or conclusion. Do not substitute stock phrases
|
||||
or create staccato anti-slop prose. Preserve force-bearing "never", "must", and
|
||||
"all" exactly in safety, security, legal, and technical rules. Presets change
|
||||
delivery, not facts. For lists, return one validated replacement per item.
|
||||
|
||||
For audit-only requests, report findings and protections without rewriting;
|
||||
report each requested load-bearing span separately. Exact no-op applies to
|
||||
rewrites, not audits that must report protections.
|
||||
|
||||
## Decision rule
|
||||
|
||||
Return a finding only when the contextual defect is clearer than preservation.
|
||||
|
||||
## Validation
|
||||
|
||||
After every rewrite or voiced draft, run:
|
||||
|
||||
```bash
|
||||
python3 scripts/validate_preservation.py original.txt transformed.txt
|
||||
python3 scripts/banned_phrase_scan.py <<< "$OUTPUT"
|
||||
python3 scripts/structure_scan.py <<< "$OUTPUT"
|
||||
python3 scripts/silhouette_scan.py <<< "$OUTPUT"
|
||||
python3 scripts/readability_metrics.py <<< "$OUTPUT"
|
||||
python3 scripts/diff_check.py original.txt transformed.txt
|
||||
```
|
||||
|
||||
Block preservation loss, introduced hard or `anti_slop_register` hits,
|
||||
unjustified structural damage, corroborated or hard silhouette damage, staccato,
|
||||
and strict-mode scores below 32/40. Use preservation `--strict` for legal,
|
||||
medical, security, or scientific text. Preexisting soft cadence and
|
||||
uncorroborated soft silhouette warnings are advisory.
|
||||
|
||||
For remaining structure damage, pass exact findings to
|
||||
`evals/run_structure_climb.py`. Re-read negations, conditions, scope, certainty,
|
||||
and party relationships.
|
||||
@@ -0,0 +1,415 @@
|
||||
# Edit Library
|
||||
|
||||
Before/after transformation examples showing specific edit patterns.
|
||||
|
||||
---
|
||||
|
||||
## Pattern 1: Throat-Clearing Removal
|
||||
|
||||
### Before
|
||||
> Here's the thing: building products is hard. Not because the technology is complex. Because people are complex. Let that sink in.
|
||||
|
||||
### After
|
||||
> Building products is hard, and the reason is people, not technology.
|
||||
|
||||
### Changes
|
||||
- Removed opener "Here's the thing:"
|
||||
- Stated the real reason in one flowing sentence — not a clipped "Not X. Y." fragment pair (that's its own AI tell)
|
||||
- Deleted "Let that sink in" emphasis crutch
|
||||
- 23 words → 12 words (48% reduction)
|
||||
|
||||
---
|
||||
|
||||
## Pattern 2: Filler + Unnecessary Reassurance
|
||||
|
||||
### Before
|
||||
> It turns out that most teams struggle with alignment. The uncomfortable truth is that nobody wants to admit they're confused. And that's okay.
|
||||
|
||||
### After
|
||||
> Teams struggle with alignment because nobody admits they're confused.
|
||||
|
||||
### Changes
|
||||
- Cut "It turns out that"
|
||||
- Cut "The uncomfortable truth is"
|
||||
- Removed permission-granting "And that's okay"
|
||||
- Joined cause and effect into one sentence instead of two clipped fragments
|
||||
- 26 words → 9 words (65% reduction)
|
||||
|
||||
---
|
||||
|
||||
## Pattern 3: Business Jargon Stack
|
||||
|
||||
### Before
|
||||
> In today's fast-paced landscape, we need to lean into discomfort and navigate uncertainty with clarity while leveraging our core competencies.
|
||||
|
||||
### After
|
||||
> Move fast, handle uncertainty, and use what you're actually good at.
|
||||
|
||||
### Changes
|
||||
- "In today's fast-paced landscape" → cut
|
||||
- "lean into discomfort" → cut (filler motivation)
|
||||
- "navigate uncertainty" → "handle uncertainty"
|
||||
- "leveraging core competencies" → "use what you're actually good at"
|
||||
- Kept it as one sentence — a staccato imperative triplet ("Move fast. Handle uncertainty. …") is its own AI cadence
|
||||
- 22 words → 11 words (50% reduction)
|
||||
|
||||
---
|
||||
|
||||
## Pattern 4: Dramatic Fragmentation
|
||||
|
||||
### Before
|
||||
> Speed. Quality. Cost. You can only pick two. That's it. That's the tradeoff.
|
||||
|
||||
### After
|
||||
> Speed, quality, cost. Pick two.
|
||||
|
||||
### Changes
|
||||
- Collapsed fragmented list into single sentence
|
||||
- Removed performative "That's it. That's the tradeoff."
|
||||
- 14 words → 5 words (64% reduction)
|
||||
|
||||
---
|
||||
|
||||
## Pattern 5: Rhetorical Setup
|
||||
|
||||
### Before
|
||||
> What if I told you that the best teams don't optimize for productivity? Here's what I mean: they optimize for learning.
|
||||
|
||||
### After
|
||||
> The best teams optimize for learning, not productivity.
|
||||
|
||||
### Changes
|
||||
- Removed "What if I told you" rhetorical question
|
||||
- Removed "Here's what I mean:" scaffolding
|
||||
- Direct statement instead of reveal structure
|
||||
- 22 words → 9 words (59% reduction)
|
||||
|
||||
---
|
||||
|
||||
## Pattern 6: Hedging Stack
|
||||
|
||||
### Before
|
||||
> I think it's probably fair to say that perhaps most organizations might benefit from potentially considering a more agile approach.
|
||||
|
||||
### After
|
||||
> Most organizations should try agile.
|
||||
|
||||
### Changes
|
||||
- "I think" → cut
|
||||
- "probably fair to say" → cut
|
||||
- "perhaps" → cut
|
||||
- "might benefit from potentially considering" → "should try"
|
||||
- 21 words → 6 words (71% reduction)
|
||||
|
||||
---
|
||||
|
||||
## Pattern 7: Absolute Claims Softened
|
||||
|
||||
### Before
|
||||
> Everyone knows that nobody ever succeeds without failing first. It's always been this way and it always will be.
|
||||
|
||||
### After
|
||||
> Success usually follows failure.
|
||||
|
||||
### Changes
|
||||
- "Everyone knows" → cut (false absolute)
|
||||
- "nobody ever" → implicit
|
||||
- "always been/always will be" → "usually"
|
||||
- 19 words → 4 words (79% reduction)
|
||||
|
||||
---
|
||||
|
||||
## Pattern 8: Meta-Commentary Removal
|
||||
|
||||
### Before
|
||||
> Hint: the answer isn't what you think. Plot twist: it's actually quite simple. Let me explain. Think about it: when you remove complexity, things get easier.
|
||||
|
||||
### After
|
||||
> The answer is simple: remove complexity.
|
||||
|
||||
### Changes
|
||||
- Removed "Hint:" meta-commentary
|
||||
- Removed "Plot twist:" performative
|
||||
- Removed "Let me explain" / "Think about it:"
|
||||
- Direct statement instead of theatrical reveal
|
||||
- 27 words → 6 words (78% reduction)
|
||||
|
||||
---
|
||||
|
||||
## Pattern 9: Binary Contrast Collapse
|
||||
|
||||
### Before
|
||||
> It's not about working harder. It's about working smarter. It feels like the former is the answer. It's actually the latter. Make no mistake about it.
|
||||
|
||||
### After
|
||||
> Work smarter, not harder.
|
||||
|
||||
### Changes
|
||||
- Collapsed repetitive binary structure
|
||||
- Removed "feels like / actually" false drama
|
||||
- Removed "Make no mistake about it" emphasis
|
||||
- 28 words → 4 words (86% reduction)
|
||||
|
||||
---
|
||||
|
||||
## Pattern 10: List Compression
|
||||
|
||||
### Before
|
||||
> There are three things every leader needs: First, they need clarity. Second, they need courage. Third, they need consistency. These three C's are the foundation of effective leadership.
|
||||
|
||||
### After
|
||||
> Leaders need clarity, courage, and consistency.
|
||||
|
||||
### Changes
|
||||
- Collapsed numbered list into single sentence
|
||||
- Removed "There are three things" setup
|
||||
- Removed "these three C's" meta-reference
|
||||
- Removed "foundation of effective leadership" filler
|
||||
- 31 words → 7 words (77% reduction)
|
||||
|
||||
---
|
||||
|
||||
## Pattern 11: Intensifier Reduction
|
||||
|
||||
### Before
|
||||
> This is incredibly important and truly fundamental to deeply understanding the absolutely essential nature of really effective communication.
|
||||
|
||||
### After
|
||||
> This matters for effective communication.
|
||||
|
||||
### Changes
|
||||
- "incredibly" → cut
|
||||
- "truly fundamental" → cut
|
||||
- "deeply understanding" → cut
|
||||
- "absolutely essential" → cut
|
||||
- "really effective" → "effective"
|
||||
- 17 words → 6 words (65% reduction)
|
||||
|
||||
---
|
||||
|
||||
## Pattern 12: Passive to Active
|
||||
|
||||
### Before
|
||||
> The decision was made by the team after the options were considered by leadership and the risks were evaluated by the analysts.
|
||||
|
||||
### After
|
||||
> Leadership weighed the options, the analysts assessed the risks, and the team made the call.
|
||||
|
||||
### Changes
|
||||
- "decision was made by the team" → "the team made the call"
|
||||
- "options were considered by leadership" → "Leadership weighed the options"
|
||||
- "risks were evaluated by analysts" → "the analysts assessed the risks"
|
||||
- Passive → active throughout, joined as one sentence (three 3-word sentences in a row is staccato)
|
||||
- 23 words → 15 words (35% reduction)
|
||||
|
||||
---
|
||||
|
||||
## Pattern 13: Fact-Heavy Preservation
|
||||
|
||||
### Before
|
||||
> It's worth noting that in Q3 2024, Acme Corp reported revenue of $47.3M, which represents a truly impressive 23% year-over-year increase. This is a game-changing result for the industry.
|
||||
|
||||
### After
|
||||
> Acme Corp reported $47.3M revenue in Q3 2024, up 23% year-over-year.
|
||||
|
||||
### Changes
|
||||
- "It's worth noting that" → cut
|
||||
- "truly impressive" → cut (editorializing)
|
||||
- "game-changing result for the industry" → cut (filler)
|
||||
- All facts preserved: Q3 2024, Acme Corp, $47.3M, 23%
|
||||
- 33 words → 12 words (64% reduction)
|
||||
|
||||
---
|
||||
|
||||
## Pattern 14: Conditional Simplification
|
||||
|
||||
### Before
|
||||
> If you want to succeed in the modern marketplace, you need to ensure that you're providing value to your customers in a way that differentiates you from the competition.
|
||||
|
||||
### After
|
||||
> Differentiate through customer value.
|
||||
|
||||
### Changes
|
||||
- "If you want to succeed in the modern marketplace" → cut (obvious)
|
||||
- "ensure that you're providing value to your customers" → "customer value"
|
||||
- "in a way that differentiates you from the competition" → "Differentiate through"
|
||||
- 30 words → 4 words (87% reduction)
|
||||
|
||||
---
|
||||
|
||||
## Pattern 15: Question + Immediate Answer
|
||||
|
||||
### Before
|
||||
> What's the key to retention? It's simple: make users feel valued. How do you do that? By listening. And why does that matter? Because people want to be heard.
|
||||
|
||||
### After
|
||||
> Retain users by making them feel valued. Listen to them.
|
||||
|
||||
### Changes
|
||||
- Collapsed Q&A theatrical structure
|
||||
- Direct statements instead of rhetorical questions
|
||||
- "Because people want to be heard" → implicit in "Listen"
|
||||
- 32 words → 11 words (66% reduction)
|
||||
|
||||
---
|
||||
|
||||
## Pattern 16: Significance Inflation
|
||||
|
||||
### Before
|
||||
> The Apollo program stands as a testament to human ingenuity, leaving an indelible mark on the history of space exploration and serving as a cornerstone of scientific achievement.
|
||||
|
||||
### After
|
||||
> The Apollo program put twelve people on the moon between 1969 and 1972.
|
||||
|
||||
### Changes
|
||||
- "stands as a testament to human ingenuity" → cut (inflated copula + significance)
|
||||
- "leaving an indelible mark" → cut (cliché)
|
||||
- "cornerstone of scientific achievement" → cut (inflation)
|
||||
- Replaced abstract praise with the specific fact
|
||||
- 30 words → 13 words (57% reduction)
|
||||
|
||||
---
|
||||
|
||||
## Pattern 17: Promotional Language
|
||||
|
||||
### Before
|
||||
> Nestled in the heart of downtown, the restaurant boasts a world-class menu featuring breathtaking flavors. A hidden gem that's a must-visit for any food lover.
|
||||
|
||||
### After
|
||||
> The restaurant, on Main Street downtown, serves Southern-French fusion. The duck confit tacos are worth the trip.
|
||||
|
||||
### Changes
|
||||
- "Nestled in the heart of" → specific location
|
||||
- "boasts a world-class menu" → describes the actual cuisine
|
||||
- "breathtaking flavors" → named a specific dish
|
||||
- "hidden gem" / "must-visit" → concrete recommendation
|
||||
- 27 words → 18 words (33% reduction)
|
||||
|
||||
---
|
||||
|
||||
## Pattern 18: Superficial -ing Analysis
|
||||
|
||||
### Before
|
||||
> The company reported 40% revenue growth, highlighting the effectiveness of its strategy, showcasing the strength of its market position, and underscoring the importance of innovation in the tech sector.
|
||||
|
||||
### After
|
||||
> The company's revenue grew 40%. Its new API product drove most of that — enterprise contracts tripled.
|
||||
|
||||
### Changes
|
||||
- Deleted all three participial clauses (highlighting, showcasing, underscoring)
|
||||
- Replaced fake analysis with actual causal explanation
|
||||
- 31 words → 17 words (45% reduction)
|
||||
|
||||
---
|
||||
|
||||
## Pattern 19: Copula Avoidance
|
||||
|
||||
### Before
|
||||
> The library serves as a community hub that functions as a gathering place and represents a vital resource for residents.
|
||||
|
||||
### After
|
||||
> The library is where the neighborhood meets. Residents use it for everything from job searches to story time.
|
||||
|
||||
### Changes
|
||||
- "serves as" → "is"
|
||||
- "functions as" → cut (redundant with "serves as")
|
||||
- "represents a vital resource" → concrete examples of use
|
||||
- 20 words → 19 words (same length, more information)
|
||||
|
||||
---
|
||||
|
||||
## Pattern 20: Negative Parallelism
|
||||
|
||||
### Before
|
||||
> It's not just about building a product. It's about building a community. Not only does the platform connect users, but it also fosters meaningful relationships and drives engagement.
|
||||
|
||||
### After
|
||||
> The platform's value is its community. Users keep coming back because they've built relationships there.
|
||||
|
||||
### Changes
|
||||
- "It's not just about X, it's about Y" → stated Y directly
|
||||
- "Not only...but also..." → collapsed into cause-effect
|
||||
- "fosters meaningful relationships and drives engagement" → concrete behavior
|
||||
- 30 words → 16 words (47% reduction)
|
||||
|
||||
---
|
||||
|
||||
## Pattern 21: Communication Artifacts
|
||||
|
||||
### Before
|
||||
> Great question! I'd be happy to help with that. Certainly, there are several approaches you could take. I hope this helps — let me know if you need anything else!
|
||||
|
||||
### After
|
||||
> Three approaches work here: [approach 1], [approach 2], [approach 3].
|
||||
|
||||
### Changes
|
||||
- "Great question!" → cut (chatbot flattery)
|
||||
- "I'd be happy to help" → cut (chatbot compliance)
|
||||
- "Certainly" → cut (chatbot filler)
|
||||
- "I hope this helps" / "let me know if..." → cut (chatbot sign-off)
|
||||
- 31 words → 9 words (71% reduction)
|
||||
|
||||
---
|
||||
|
||||
## Pattern 22: False Ranges
|
||||
|
||||
### Before
|
||||
> The festival attracts visitors from young professionals to retirees, from local residents to international tourists, spanning everything from music enthusiasts to casual attendees.
|
||||
|
||||
### After
|
||||
> The festival draws 50,000 people annually, mostly locals and tourists from neighboring states.
|
||||
|
||||
### Changes
|
||||
- Collapsed three stacked "from X to Y" ranges into specific data
|
||||
- Replaced vague breadth claim with actual numbers
|
||||
- 25 words → 13 words (48% reduction)
|
||||
|
||||
---
|
||||
|
||||
## Pattern 23: Generic Positive Conclusion
|
||||
|
||||
### Before
|
||||
> As the industry continues to evolve, the future looks bright. Exciting times lie ahead for all stakeholders involved. One thing is certain: innovation will continue to shape the landscape.
|
||||
|
||||
### After
|
||||
> Two trends to watch: edge computing costs dropping 30% annually, and enterprise AI adoption doubling each quarter.
|
||||
|
||||
### Changes
|
||||
- "continues to evolve" → cut (says nothing)
|
||||
- "the future looks bright" → cut (empty optimism)
|
||||
- "exciting times lie ahead" → cut (content-free)
|
||||
- "one thing is certain" → replaced with specific predictions
|
||||
- 30 words → 16 words (47% reduction)
|
||||
|
||||
---
|
||||
|
||||
## Pattern 24: Compound Example (Multiple Pattern Types)
|
||||
|
||||
### Before
|
||||
> In today's rapidly evolving digital landscape, the groundbreaking initiative serves as a testament to the company's commitment to innovation. Not only does it showcase their robust capabilities, but it also underscores the importance of fostering collaboration. Experts argue this represents a pivotal moment. The future looks bright.
|
||||
|
||||
### After
|
||||
> The company shipped a new collaboration tool. Early numbers: 2,000 teams signed up in the first week, 85% retention at day 30.
|
||||
|
||||
### Changes
|
||||
- "In today's rapidly evolving digital landscape" → cut (filler)
|
||||
- "groundbreaking initiative serves as a testament to" → named the actual thing
|
||||
- "Not only...but also..." → cut (negative parallelism)
|
||||
- "showcasing robust capabilities" / "underscoring importance" → cut (superficial -ing)
|
||||
- "fostering collaboration" → named the product category
|
||||
- "Experts argue" → cut (vague attribution)
|
||||
- "represents a pivotal moment" → replaced with data
|
||||
- "The future looks bright" → cut (generic conclusion)
|
||||
- 47 words → 22 words (53% reduction), information density dramatically higher
|
||||
|
||||
---
|
||||
|
||||
## Transformation Principles
|
||||
|
||||
1. **Cut setup, keep substance** - Remove everything before the actual point
|
||||
2. **One structure, not three** - If it can be one sentence, make it one
|
||||
3. **Let facts speak** - Numbers don't need emphasis crutches
|
||||
4. **Active beats passive** - Who did what, not what was done by whom
|
||||
5. **Trust the reader** - They'll get it without "Think about it:"
|
||||
6. **Verbs do the work** - Concrete actions, not abstract processes
|
||||
7. **Vary the rhythm** - Concise is not the same as staccato. After cutting, check the result isn't a run of tiny sentences or a bare "Not X. Y." contrast — that clipped cadence is its own AI tell. Mix short sentences with a longer one.
|
||||
@@ -0,0 +1,169 @@
|
||||
# Fact Preservation Rules
|
||||
|
||||
Certain content must survive transformation unchanged. This reference defines what to protect.
|
||||
|
||||
---
|
||||
|
||||
## Absolute Preservation (Never Modify)
|
||||
|
||||
These elements must appear in the output exactly as they appear in the input.
|
||||
|
||||
### Numbers & Quantities
|
||||
- Specific figures: `$47.3M`, `23%`, `1,500 users`
|
||||
- Dates: `Q3 2024`, `March 15, 2024`, `2024-03-15`
|
||||
- Times: `9:00 AM PST`, `14:30 UTC`
|
||||
- Measurements: `5.2kg`, `100ms`, `2TB`
|
||||
- Ranges: `$50-75K`, `3-5 years`, `10-15%`
|
||||
- Counts: `3 engineers`, `5 steps`, `47 countries`
|
||||
|
||||
### Proper Nouns
|
||||
- Company names: `Acme Corp`, `OpenAI`, `Google`
|
||||
- Product names: `GPT-4`, `iPhone 15`, `TensorFlow`
|
||||
- Person names: `John Smith`, `Dr. Chen`
|
||||
- Place names: `San Francisco`, `Building 43`
|
||||
- Brand terms: `React`, `Kubernetes`, `AWS Lambda`
|
||||
|
||||
### Technical Terms
|
||||
- API endpoints: `/api/v2/users`
|
||||
- Code references: `handleSubmit()`, `UserContext`
|
||||
- Config values: `max_retries: 3`, `timeout: 30s`
|
||||
- Version numbers: `v2.1.0`, `Node 20.x`
|
||||
- File paths: `/src/components/Button.tsx`
|
||||
|
||||
### Quoted Material
|
||||
- Direct quotes: `"We're just getting started" - CEO`
|
||||
- Cited text: anything in quotation marks attributed to a source
|
||||
- Code snippets: preserve exactly, including whitespace
|
||||
|
||||
### URLs & References
|
||||
- Links: `https://example.com/page`
|
||||
- DOIs: `10.1234/journal.2024.001`
|
||||
- ISBNs: `978-0-123456-78-9`
|
||||
- Citations: `[1]`, `(Smith et al., 2024)`
|
||||
|
||||
---
|
||||
|
||||
## Semantic Preservation (Keep Meaning)
|
||||
|
||||
These can be rephrased but meaning must stay identical.
|
||||
|
||||
### Causal Claims
|
||||
- Original: "A caused B"
|
||||
- OK: "B resulted from A"
|
||||
- NOT OK: "A influenced B" (weaker claim)
|
||||
|
||||
### Comparisons
|
||||
- Original: "X is 3x faster than Y"
|
||||
- OK: "X outperforms Y by 3x"
|
||||
- NOT OK: "X is faster than Y" (lost quantifier)
|
||||
|
||||
### Conditional Statements
|
||||
- Original: "If A, then B"
|
||||
- OK: "B happens when A"
|
||||
- NOT OK: "A and B" (lost conditionality)
|
||||
|
||||
### Negations
|
||||
- Original: "X does not support Y"
|
||||
- OK: "X lacks Y support"
|
||||
- NOT OK: "X partially supports Y" (opposite meaning)
|
||||
|
||||
### Scope Qualifiers
|
||||
- Original: "Most users (73%) prefer X"
|
||||
- OK: "73% of users prefer X"
|
||||
- NOT OK: "Users prefer X" (lost scope)
|
||||
|
||||
---
|
||||
|
||||
## Constraint Extraction
|
||||
|
||||
Before transforming, extract these as must-preserve spans:
|
||||
|
||||
```python
|
||||
CONSTRAINT_PATTERNS = {
|
||||
# Numbers with context
|
||||
"currency": r"\$[\d,]+\.?\d*[KMB]?",
|
||||
"percentage": r"\d+\.?\d*%",
|
||||
"date_iso": r"\d{4}-\d{2}-\d{2}",
|
||||
"date_natural": r"(Q[1-4]\s+\d{4}|(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)[a-z]*\s+\d{1,2},?\s+\d{4})",
|
||||
"time": r"\d{1,2}:\d{2}\s*(?:AM|PM|UTC|PST|EST)?",
|
||||
"measurement": r"\d+\.?\d*\s*(?:ms|s|min|hr|KB|MB|GB|TB|kg|lb|m|km|mi)",
|
||||
|
||||
# Identifiers
|
||||
"url": r"https?://[^\s]+",
|
||||
"email": r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}",
|
||||
"code_ref": r"`[^`]+`",
|
||||
"quote": r'"[^"]{10,}"',
|
||||
|
||||
# Proper nouns (NER-detected)
|
||||
"org": "ORGANIZATION entities",
|
||||
"person": "PERSON entities",
|
||||
"product": "PRODUCT entities",
|
||||
"location": "GPE/LOC entities"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification Process
|
||||
|
||||
After transformation:
|
||||
|
||||
1. **Extract all constraints from original**
|
||||
2. **Check each constraint exists in output**
|
||||
3. **Flag any missing or altered constraints**
|
||||
|
||||
### Pass Criteria
|
||||
- All absolute preservation items present and unchanged
|
||||
- All semantic preservation items equivalent in meaning
|
||||
- No fact inversions or significant alterations
|
||||
|
||||
### Fail Examples
|
||||
| Original | Output | Problem |
|
||||
|----------|--------|---------|
|
||||
| "$47.3M revenue" | "$47M revenue" | Precision lost |
|
||||
| "23% increase" | "significant increase" | Number removed |
|
||||
| "John Smith, CEO" | "the CEO" | Name removed |
|
||||
| "https://example.com" | "[link]" | URL removed |
|
||||
| "Q3 2024" | "recently" | Date removed |
|
||||
| "does not support" | "partially supports" | Meaning inverted |
|
||||
|
||||
---
|
||||
|
||||
## Special Cases
|
||||
|
||||
### Approximations in Original
|
||||
If original uses approximate language, preserve the approximation:
|
||||
- "about 50%" → OK to keep as "about 50%" or "roughly 50%"
|
||||
- NOT OK to change to "50%" (implies false precision)
|
||||
|
||||
### Ranges
|
||||
Preserve both bounds:
|
||||
- "$50-75K" must keep both $50K and $75K
|
||||
- "3-5 years" must keep both bounds
|
||||
|
||||
### Lists of Items
|
||||
All items must survive:
|
||||
- "Teams in SF, NYC, and London" → all three cities required
|
||||
- Can reorder, but can't drop any
|
||||
|
||||
### Attributed Quotes
|
||||
Quote text and attribution both required:
|
||||
- `"We're excited" - Jane Doe, CEO`
|
||||
- Both the quote and "Jane Doe, CEO" must survive
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Type | Example | Modifiable? |
|
||||
|------|---------|-------------|
|
||||
| Currency | $47.3M | NO - exact |
|
||||
| Percentage | 23% | NO - exact |
|
||||
| Date | Q3 2024 | NO - exact |
|
||||
| Person | John Smith | NO - exact |
|
||||
| Company | Acme Corp | NO - exact |
|
||||
| URL | https://... | NO - exact |
|
||||
| Code | `function()` | NO - exact |
|
||||
| Quote | "..." | NO - exact |
|
||||
| Causal | A caused B | YES - preserve meaning |
|
||||
| Comparison | 3x faster | YES - preserve numbers |
|
||||
@@ -0,0 +1,41 @@
|
||||
# Harvesting Voice Samples — internals
|
||||
|
||||
Harvest bootstraps teach samples from local chat transcripts and folders the user
|
||||
declares as their own writing. The routed flow (harvest → classify → approve →
|
||||
copy) is step 1 of `references/commands/teach.md`; this file holds the adapter
|
||||
internals, the contamination tripwire, and the privacy rules that flow links to.
|
||||
|
||||
The transcript adapters only accept entries with explicit user authorship. Assistant
|
||||
turns are dropped before candidate filtering, and unknown JSONL schemas are skipped with
|
||||
a warning rather than guessed. This authorship separation is the product requirement:
|
||||
assistant-authored transcript text must never enter candidates.
|
||||
|
||||
The `codex-jsonl` adapter reads Codex CLI/Desktop sessions, stored under
|
||||
`~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl` and auto-detected by shape alongside
|
||||
`claude-jsonl`. It harvests `event_msg` user_message text and user-role
|
||||
`response_item` message content, and drops assistant/developer-role turns, tool
|
||||
and reasoning plumbing, and `session_meta.base_instructions` outright. Codex also
|
||||
injects non-user content into user-role turns: a repo's AGENTS.md dumped verbatim,
|
||||
an `<environment_context>` banner, and similar wrapper tags. The adapter filters
|
||||
those by structural marker, never by guessing at content, and counts them under
|
||||
the `instruction-injection` drop reason instead of silently discarding them.
|
||||
|
||||
The contamination tripwire runs each kept candidate through `banned_phrase_scan` and
|
||||
`structure_scan`. A hard violation, or two or more distinct scanner categories, marks
|
||||
the sample as `suspect_ai: true`. Suspect samples are kept for human review, ranked last,
|
||||
and must never be auto-approved.
|
||||
A clean-styled, unmarked assistant paste inside a user turn is undetectable by parsing
|
||||
and only weakly covered by the tripwire; mandatory human approval is the real defense.
|
||||
|
||||
The folder adapter treats `.md` and `.txt` files in a designated folder as user-authored
|
||||
by declaration, but still runs the same tripwire. Dictated text with heavy `um`/`uh`
|
||||
fillers is marked `dictated: true` and kept because spoken phrasing can still be useful
|
||||
voice evidence.
|
||||
|
||||
Privacy: harvesting is local. Candidate files, classifier task files, and approved
|
||||
`.unslop/voice/<name>/` samples should stay out of git unless the user explicitly chooses
|
||||
otherwise.
|
||||
|
||||
Tiering: parsing and filtering are Tier 0 deterministic work. Situation/register
|
||||
classification can be fanned out to the cheapest capable model through the emitted task
|
||||
files. Final approval stays human.
|
||||
@@ -0,0 +1,91 @@
|
||||
# Maintenance Procedures
|
||||
|
||||
Every change here is eval-first: the example lands in
|
||||
`evals/fixtures/contracts/scanner-examples.json` and fails before any scanner or
|
||||
catalog edit makes it pass. Run `python3 evals/check.py --full` for the full
|
||||
repository check.
|
||||
|
||||
## Add a banned phrase (`--add-phrase`)
|
||||
|
||||
1. Add a false-negative example (the phrase flags) and a false-positive
|
||||
example (a literal or domain use stays clean) to the scanner contract table.
|
||||
Run `python3 evals/check_scanner_contract.py` and confirm it fails.
|
||||
2. Add the phrase to `scripts/banned_phrase_scan.py` `BANNED_PHRASES` with
|
||||
category, severity, and suggestion. If the word has a literal sense, gate it
|
||||
behind collocations in `STRUCTURAL_PATTERNS` instead, and add a REC recall
|
||||
example proving the jargon use still flags.
|
||||
3. Document it in `references/taboo-phrases.md` (parity is enforced by
|
||||
`python3 evals/check_taboo_parity.py`).
|
||||
4. Run `python3 evals/check.py --full`; expect green with no new xfail.
|
||||
|
||||
## Add a structural pattern (`--add-structure`)
|
||||
|
||||
Same procedure, but the entry goes in `STRUCTURAL_PATTERNS` with a regex, and
|
||||
the false-positive example must cover the nearest legitimate construction the regex
|
||||
could clip.
|
||||
|
||||
## Prefer a pair
|
||||
|
||||
When a pattern has literal senses or only becomes visible in context, add a
|
||||
minimal pair under `evals/fixtures/pairs/` instead of relying only on one-line
|
||||
stdin rows. The `_with` twin should contain one controlled tell, the `_without`
|
||||
twin should preserve the facts while scanning clean, and the manifest should
|
||||
name the target category or structure metric. Run `python3 evals/check_pairs.py`
|
||||
before changing the scanner.
|
||||
|
||||
## Coverage gate and category protection (enforced)
|
||||
|
||||
`python3 evals/check_pattern_coverage.py` makes the two examples above mandatory, not
|
||||
conventional:
|
||||
|
||||
- **Coverage.** Every `STRUCTURAL_PATTERNS` regex must match at least one contract
|
||||
example's corpus, and every `BANNED_PHRASES`
|
||||
key must appear in at least one corpus text. A new scanner entry with no eval
|
||||
example fails this gate (DOC-09) while every other gate is green. There is no
|
||||
grandfather list: an uncovered pattern is a hard failure. If a batch of legacy
|
||||
keys is uncovered, add `REC`-style coverage packs (8-12 phrases per stdin, one
|
||||
natural sentence each) rather than exempting them.
|
||||
- **Protection.** Every violation category the scanner can emit must be claimed by
|
||||
a `scanner_false_positive` example carrying `"protects": "<category>"`. Backfill the
|
||||
field on that category's FP example; add a new FP example
|
||||
(asserting `total_violations == 0` on tempting-but-clean prose) for any category
|
||||
with no protector. The runner ignores the `protects` key, so it is safe on any
|
||||
FP example.
|
||||
|
||||
## Rehearse the procedure (the kata)
|
||||
|
||||
`python3 evals/kata_add_pattern.py --run` is a meta-eval (DOC-10): in a temp copy
|
||||
of the repo it adds a throwaway pattern and asserts each safety net fires in order
|
||||
— coverage catches the example-less entry, parity catches the missing catalog line,
|
||||
the contract goes green once both exist, and the example goes red if the scanner entry
|
||||
is later removed. Run it before touching the maintenance tooling; if a refactor
|
||||
breaks a safety net, this kata turns red.
|
||||
|
||||
## List current patterns
|
||||
|
||||
```bash
|
||||
python3 - <<'PY'
|
||||
from scripts.banned_phrase_scan import BANNED_PHRASES
|
||||
for k in sorted(BANNED_PHRASES): print(k)
|
||||
PY
|
||||
rg -n '"pattern":' scripts/banned_phrase_scan.py # structural patterns
|
||||
```
|
||||
|
||||
## Wiki sync (`--wiki-sync`)
|
||||
|
||||
Syncs pattern rules with Wikipedia's
|
||||
[Signs of AI writing](https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing) page.
|
||||
|
||||
1. Check for updates: `python3 scripts/wiki_sync.py check` (exit 0 = no updates).
|
||||
2. Get the structured diff: `python3 scripts/wiki_sync.py diff` (JSON with
|
||||
change type, section, words).
|
||||
3. For each new word or phrase, follow "Add a banned phrase" above — eval rows
|
||||
first, then scanner, then catalog.
|
||||
4. Verify: `python3 scripts/banned_phrase_scan.py < /dev/null` (no syntax
|
||||
errors), then the full suite.
|
||||
|
||||
Only adopt phrases that are genuine AI tells in general prose. Skip
|
||||
Wikipedia-specific patterns (broken wikitext, DOI formatting, citation graffiti).
|
||||
|
||||
For the full agent-runnable refresh procedure, including the staleness
|
||||
reporter and the parity-bench cadence, see `references/refresh.md`.
|
||||
@@ -0,0 +1,194 @@
|
||||
# Teach & Mimic — internals
|
||||
|
||||
The routed flows live in `references/commands/teach.md`,
|
||||
`references/commands/mimic.md`, and the refine section of that mimic command
|
||||
file. This file holds the deep material those flows link to: card anatomy,
|
||||
sample requirements, scoring, baselines, statistics, and failure modes.
|
||||
|
||||
Reconstruction runs under detection's constitution: a mimic or refined output
|
||||
that reintroduces slop is a failure. Every artifact here is deterministic and
|
||||
testable; the model is used only to generate prose, never to score itself.
|
||||
|
||||
## Teach — the artifacts
|
||||
|
||||
`teach` distills a writer's samples into a reusable voice, stored under
|
||||
`.unslop/voice/<name>/` (gitignored by default):
|
||||
|
||||
1. `scripts/voice_profile.py samples/ -o profile.json` — the machine
|
||||
fingerprint (char 3-grams, function-word delta, sentence-length EMD,
|
||||
punctuation, contractions, MTLD, word-length, impostor z-scores, GI rank).
|
||||
This is the referee the scorer uses.
|
||||
2. `scripts/voice_card.py --profile profile.json --samples samples/ --out .
|
||||
--name <name>` — the **layered voice card** the generating model actually
|
||||
follows. `--name` labels the card (`# Voice card: <name>`); default `voice`.
|
||||
voice_card recomputes the profile from `--samples` and refuses (exit 2, named
|
||||
field) if the supplied `--profile` does not describe those samples, so a stale
|
||||
profile can never silently drive a card.
|
||||
3. Add `--provenance` to write `provenance.json` (per-sample sha256, word
|
||||
counts, doc count, genre note, low-confidence flag) so a teach run is
|
||||
auditable.
|
||||
|
||||
**Sample files.** Both scripts read only `.txt` and `.md` files, recursively,
|
||||
under the samples directory (nothing else — a directory of `.docx` or extensionless
|
||||
files reads as empty and both scripts exit 2 with a diagnostic naming the
|
||||
requirement). Convert samples to `.txt`/`.md` first.
|
||||
|
||||
**Sample requirements.** Use at least 5 documents, 2-3k words total, in the
|
||||
**same genre** as the target. Fewer than ~2000 words or cross-genre samples set
|
||||
`low_confidence` in the profile and provenance; the card still builds but its
|
||||
claims are noisier. **Surface `low_confidence` to the user** — tell them the voice
|
||||
is provisional and ask for more same-genre samples before trusting a mimic.
|
||||
Cross-domain style attribution degrades sharply, so do not teach on blog posts
|
||||
and score legal memos.
|
||||
|
||||
## The voice card (layered)
|
||||
|
||||
The card is a directory, not one file, so the generator loads only what the
|
||||
current writing task needs:
|
||||
|
||||
- `card.md` — the always-loaded core, kept under 300 words: rhythm (median
|
||||
sentence length, IQR, burstiness), contraction habits with real examples,
|
||||
punctuation the writer uses and avoids, top sentence openers, a **Never**
|
||||
list of at/near-zero features, and an **index table** — "when writing X, read
|
||||
`card/X.md`".
|
||||
- `card/<situation>.md` — one sheet per **covered** situation from the taxonomy
|
||||
(explaining-technical, anecdote, argument, disagreement, praise,
|
||||
hedging-uncertainty, numbers-data, addressing-reader, openings, closings).
|
||||
Each sheet gives how this author handles that situation, 1-3 verbatim sample
|
||||
snippets, and measured markers.
|
||||
|
||||
**No fabrication.** A dimension with no sample evidence gets **no sheet**; it is
|
||||
named under "Uncovered" in `card.md`. The card never invents a voice the
|
||||
samples cannot support.
|
||||
|
||||
**Coverage classification.** `voice_card.py --coverage` emits a deterministic
|
||||
lexical coverage matrix over the taxonomy. It is intentionally coarse: it
|
||||
DRIVES which sheets get written and which teach prompts to ask the user, and a
|
||||
misclassified sentence can only add or drop a sheet — it can never change a card
|
||||
claim, because claims come from measured facts and verbatim snippets. So
|
||||
misclassification is low-stakes by construction. A sample set missing numeric
|
||||
writing, for instance, leaves `numbers-data` uncovered and named as a gap.
|
||||
|
||||
**Coverage → prompt procedure.** When a dimension the target writing will need is
|
||||
uncovered, ask the user for a short sample that exercises it. **Only ask for
|
||||
dimensions the task requires** — do not fish for all ten. Templates:
|
||||
|
||||
| Dimension | Prompt to the user |
|
||||
|-----------|--------------------|
|
||||
| explaining-technical | "Share something you wrote that explains how a thing works or why it behaves the way it does." |
|
||||
| anecdote | "Share a few sentences where you tell a small story about something that happened to you." |
|
||||
| argument | "Share something you wrote that argues a position — a claim you defended." |
|
||||
| disagreement | "Share something where you pushed back on or disagreed with an idea." |
|
||||
| praise | "Share something where you praised or recommended something you liked." |
|
||||
| hedging-uncertainty | "Share something where you were unsure and said so — thinking out loud." |
|
||||
| numbers-data | "Share something you wrote that works with numbers, quantities, or measurements." |
|
||||
| addressing-reader | "Share something written directly to a reader — instructions or a note to someone." |
|
||||
|
||||
`openings` and `closings` are structural (first/last sentence of every document)
|
||||
and are covered as soon as there is one document, so they need no prompt.
|
||||
|
||||
The card is pack-sized (~1-2k tokens loaded) so it rides in the generation
|
||||
context cheaply. Same inputs give byte-identical files.
|
||||
|
||||
## `--refine` — the internal hill-climb
|
||||
|
||||
`evals/run_mimic_refine.py` iterates when one pass is not enough. It is worth
|
||||
the cost only when a single mimic keeps landing short of the target voice; most
|
||||
users teach once and mimic cheaply thereafter.
|
||||
|
||||
Protocol (climb on A, accept on DEV, report on a sealed split):
|
||||
|
||||
- Split samples by document (seeded): **A** (~60%, retrieval/context pool) and
|
||||
**DEV** (~40%, acceptance). Fewer than 5 docs → refuse (exit 2).
|
||||
- **LIVE (default):** each iteration assembles B prompts — the draft, the
|
||||
A-split voice card, the k=2 nearest-A samples by char-3gram cosine, and the
|
||||
current directives — and invokes `--generate-cmd` (default `claude -p`) once
|
||||
per beam, piping the prompt on stdin and reading the candidate on stdout.
|
||||
`--baseline zero|few|retrieval` selects which samples ride in the prompt (none
|
||||
/ first-k / k nearest-A) and generates its candidate set through this same
|
||||
path, so the honest retrieval-few-shot competitor and the refine loop share
|
||||
one pipeline.
|
||||
- **DRY-RUN:** pass `--candidates-dir DIR` and the loop reads
|
||||
`DIR/iter<i>/*.md` as that iteration's batch instead of generating them, so
|
||||
the loop logic runs with no model calls (this is how the MIMIC-* rows exercise
|
||||
it; the LIVE path is covered by a mock generator).
|
||||
- **Hard gates** discard a candidate regardless of score: banned-phrase clean,
|
||||
structure clean, draft→candidate preservation, no copy-gate violation against
|
||||
A, and a 150-word floor.
|
||||
- Survivors are scored against DEV (and recorded against A for the divergence
|
||||
guard) with the **full `voice_score` composite** — `0.5·(1−GI) + 0.5·` clipped
|
||||
weighted impostor-z distance against a same-genre impostor pool (`--impostors`,
|
||||
default the committed pool), seeded. This is deliberately NOT a raw weighted
|
||||
distance: a marker-stuffed candidate can minimize a raw distance and clear
|
||||
every hard gate yet be exposed by GI, so raw-distance acceptance would accept
|
||||
it over honest prose. The generator never scores itself.
|
||||
- Accept an iteration only if the DEV composite improves by ≥ `min-delta`; the
|
||||
best-so-far never regresses.
|
||||
|
||||
### Stopping and the divergence guard
|
||||
|
||||
Earliest of: iteration cap, patience (N iterations without a DEV gain), or
|
||||
**divergence** — the A composite improving while DEV worsens for two
|
||||
consecutive iterations. Divergence is the reward-hacking signature (the model
|
||||
is fitting the retrieval pool's topics, not the author's style); the loop halts
|
||||
immediately and sets `reward_hacking_warning`.
|
||||
|
||||
### Directives and card refinement
|
||||
|
||||
Each iteration derives up to four ranked directives from the DEV metric deltas,
|
||||
and each directive carries a proposed **card amendment** line. The best
|
||||
candidate is copied to `OUT/final.md` and the amended card to
|
||||
`OUT/voice-card.refined.md`.
|
||||
|
||||
## Scoring
|
||||
|
||||
The `voice_score` **composite** the refine loop accepts on is
|
||||
`0.5·(1−GI) + 0.5·zsum` (lower = more author-like), where `GI` is the General
|
||||
Impostors rank — the fraction of random feature-subset trials in which the
|
||||
candidate beats every sampled impostor at looking like the profile — and `zsum`
|
||||
is the clipped, weighted **impostor-z** distance (each per-feature distance
|
||||
normalized by the impostor pool's mean/std, clipped to ±3, weighted by the WP10a
|
||||
research weights). Both halves are calibrated against a same-genre **impostor
|
||||
pool**, so the number rewards *beating plausible other authors*, not merely
|
||||
matching aggregate surface stats. That is what defeats a marker-stuffed
|
||||
candidate: it can drive a raw weighted distance down and still be far worse under
|
||||
GI. Meaning preservation vs the original draft is a **separate** hard gate, never
|
||||
blended into the composite. The composite is a **guide, not an oracle** — even
|
||||
strong authorship verification tops out near a 93-94% ceiling, so treat
|
||||
per-candidate deltas as the signal and under-claim.
|
||||
|
||||
## Baselines
|
||||
|
||||
Score refine output against paired baselines on the same drafts: zero-shot
|
||||
("write like this person"), static few-shot, and **retrieval few-shot** (k
|
||||
nearest-A samples — the honest competitor). Refine has to beat retrieval
|
||||
few-shot to earn its cost.
|
||||
|
||||
## Statistics
|
||||
|
||||
`evals/mimic_stats.py` runs the small-n discipline: per-item paired deltas, a
|
||||
BCa bootstrap CI (n=2000, seeded), and a sign-flip permutation test (exact for
|
||||
≤12 items, else 20k seeded). Claim a win only when the CI lower bound > 0 **and**
|
||||
p < 0.05. Anything degenerate (too few items, zero variance) reports
|
||||
not-improved.
|
||||
|
||||
## Cost
|
||||
|
||||
| Step | Calls |
|
||||
|------|-------|
|
||||
| teach (profile + card) | free / deterministic |
|
||||
| mimic | one strong generation call |
|
||||
| refine | iterations × beam strong calls, plus free gate/score passes |
|
||||
|
||||
## Failure modes and the gate that catches each
|
||||
|
||||
| Failure | Symptom | Caught by |
|
||||
|---------|---------|-----------|
|
||||
| Topic bleed | importing what samples are *about*, not *how* they read | divergence guard (A improves, DEV worsens) |
|
||||
| Verbatim copying | lifting sample phrasing | copy-gate (4-gram / LCS vs A) |
|
||||
| Register lock | latching one sample's register | DEV composite vs held-out split |
|
||||
| Reward hacking | self-assessed quality rising while DEV falls | divergence guard + `reward_hacking_warning` |
|
||||
| Slop reintroduced | AI tells creep back into voiced prose | banned-phrase / structure gates |
|
||||
|
||||
For chunking long documents and the tiered execution architecture around these
|
||||
steps, see `pipeline.md`.
|
||||
@@ -0,0 +1,75 @@
|
||||
{
|
||||
"packs": {
|
||||
"pack-phrases-core": [
|
||||
"academic_excess",
|
||||
"acknowledgment_loop",
|
||||
"ai_vocabulary",
|
||||
"assistant_artifact",
|
||||
"communication_artifact",
|
||||
"conclusion_scaffold",
|
||||
"copula_avoidance",
|
||||
"editorializing",
|
||||
"emphasis_crutch",
|
||||
"false_agency",
|
||||
"false_concession",
|
||||
"false_range",
|
||||
"filler",
|
||||
"filler_opener",
|
||||
"formulaic_challenges",
|
||||
"generic_conclusion",
|
||||
"jargon",
|
||||
"knowledge_cutoff",
|
||||
"meta",
|
||||
"novelty_inflation",
|
||||
"numbered_list_inflation",
|
||||
"performative",
|
||||
"promotional",
|
||||
"reader_addressing",
|
||||
"reader_steering",
|
||||
"reasoning_chain",
|
||||
"rhetorical_question",
|
||||
"rhetorical_setup",
|
||||
"significance_inflation",
|
||||
"superficial_ing",
|
||||
"throat_clearing",
|
||||
"unnecessary_permission",
|
||||
"vague_attribution",
|
||||
"vague_endorsement"
|
||||
],
|
||||
"pack-structure": [
|
||||
"macro_both_sidesism",
|
||||
"macro_redemption_arc",
|
||||
"macro_preview_recap",
|
||||
"macro_over_determination",
|
||||
"macro_emotional_flatness"
|
||||
],
|
||||
"pack-register-guards": [
|
||||
"hedge_stack",
|
||||
"parenthetical_hedging"
|
||||
],
|
||||
"pack-voice": [
|
||||
"anthropomorphic_shipping",
|
||||
"anti_slop_register",
|
||||
"binary_contrast",
|
||||
"colon_reveal",
|
||||
"contrastive_definition",
|
||||
"dramatic_fragment",
|
||||
"em_dash_overuse",
|
||||
"exclamation_overuse",
|
||||
"formatting",
|
||||
"fragment_template",
|
||||
"headline_cadence",
|
||||
"headline_container_agency",
|
||||
"imperative_slogan",
|
||||
"negative_listing",
|
||||
"negative_parallelism",
|
||||
"numeric_parallelism",
|
||||
"slogan_fragment",
|
||||
"spec_fragment",
|
||||
"staccato_fragmentation",
|
||||
"tool_agency_reflexive",
|
||||
"tool_agency_volitional"
|
||||
],
|
||||
"pack-facts": []
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
# Pack: Facts
|
||||
|
||||
Use this pack for facts and constraints that must survive a rewrite. Do not judge style or phrase quality.
|
||||
|
||||
## Look For
|
||||
|
||||
- Constraints from `extract_constraints.py`: numbers, dates, times, names, URLs, emails, quoted text, code identifiers, currencies, percentages, units, ranges.
|
||||
- Negations, conditionals, and scope words surfaced by preservation validation: "not", "unless", "except", "only", "must", "may", "can", "at least", "no more than".
|
||||
- Approximation traps: "roughly 60%" is not "60%"; "Q2" is not "June"; "under 500 ms" is not "500 ms".
|
||||
- Proper-noun and party relationships: Apple sued Qualcomm is not Qualcomm sued Apple.
|
||||
- Technical literals: API names, flags, file paths, version numbers, protocol names.
|
||||
|
||||
## Emit
|
||||
|
||||
Return JSON findings only:
|
||||
|
||||
```json
|
||||
{"span":"roughly 60%","rule":"approximate_quantity","pack":"pack-facts","severity":"hard","note":"Keep the approximation marker with the number."}
|
||||
```
|
||||
|
||||
Use `hard` for exact facts, negations, scope, and quantities. Use `soft` for phrasing that appears factual but may be compressible.
|
||||
|
||||
## Examples
|
||||
|
||||
- "p99 latency fell to 180 ms on June 18" -> report p99, 180 ms, June 18.
|
||||
- "roughly 60%" -> report the approximation and number together.
|
||||
- "The team improved performance." -> no finding; style claim without a concrete constraint.
|
||||
- "`--strict` failed on v2.1.0" -> report flag and version.
|
||||
@@ -0,0 +1,30 @@
|
||||
# Pack: Phrases Core
|
||||
|
||||
Use this pack only for scanner-backed phrase, jargon, and scaffold tells. Do not judge facts, register-preservation, macro structure, or the rewriter's house style here.
|
||||
|
||||
## Look For
|
||||
|
||||
- Throat-clearing and reader steering: "here's the thing", "let's dive in", "to answer your question".
|
||||
- Emphasis crutches and inflated significance: "let that sink in", "underscores the importance", "a testament to".
|
||||
- Business, academic, and promotional vocabulary used as generic polish: "leverage", "seamless", "vibrant", "rich cultural heritage".
|
||||
- Vague attribution or false agency: unnamed "experts", numbers that "speak for themselves", data that "tells a story".
|
||||
- Formulaic scaffolds: ordinal sequencing, generic conclusions, rhetorical setup questions, novelty claims, numbered-list hype.
|
||||
|
||||
Use `references/taboo-phrases.md` as the authoritative phrase catalog. Report only spans this pack owns.
|
||||
|
||||
## Emit
|
||||
|
||||
Return JSON findings only:
|
||||
|
||||
```json
|
||||
{"span":"...","rule":"throat_clearing","pack":"pack-phrases-core","severity":"hard","note":"Cut the opener and start with the claim."}
|
||||
```
|
||||
|
||||
Use `hard` for always-bad boilerplate and `soft` for context-dependent wording. Leave legitimate literal, legal, medical, code, or domain-specific uses unreported.
|
||||
|
||||
## Examples
|
||||
|
||||
- "Here's the thing: the data speaks for itself." -> report "Here's the thing" and "speaks for itself".
|
||||
- "The loan used 3:1 leverage." -> no finding; finance sense is literal.
|
||||
- "This update unlocks seamless cross-functional synergy." -> report "seamless" and "synergy".
|
||||
- "Some critics argue the policy failed." -> report vague attribution unless the critics are named nearby.
|
||||
@@ -0,0 +1,30 @@
|
||||
# Pack: Register Guards
|
||||
|
||||
Use this pack to find wording that must survive a rewrite. Do not flag ordinary slop phrases and do not rewrite. This pack prevents harm.
|
||||
|
||||
## Look For
|
||||
|
||||
- Legal scope and negation: "arguably", "may", "shall", "must", "unless", "except", "not", "does not", section references, liability limits, indemnity terms.
|
||||
- Medical and scientific uncertainty: "may cause", "can increase", "studies suggest", "preliminary", "observational", "does not establish causation", confidence limits, cohort limits.
|
||||
- Security and safety absolutes: "never", "always", "must", "all input", "no secrets", "do not mix", when they define a rule or warning.
|
||||
- Scope words and quantities: "most", "some", "only", "at least", "no more than", percentages, ranges, dates, named parties.
|
||||
- Anti-dehedging traps: a phrase that looks cautious but carries the claim's certainty, exception, or legal burden.
|
||||
|
||||
Run on the whole text, not paragraph chunks, because scope can be established earlier than the sentence being rewritten.
|
||||
|
||||
## Emit
|
||||
|
||||
Return JSON findings only:
|
||||
|
||||
```json
|
||||
{"span":"arguably does not rise to gross negligence","rule":"legal_negation_scope","pack":"pack-register-guards","severity":"hard","note":"Hedge and negation are load-bearing; preserve them exactly."}
|
||||
```
|
||||
|
||||
Every finding is a preservation constraint. Use `hard` when changing it could invert or strengthen the claim; `soft` when it is likely load-bearing but needs human review.
|
||||
|
||||
## Examples
|
||||
|
||||
- "The incident arguably does not rise to gross negligence under Section 12(b)." -> report the full legal hedge/negation span.
|
||||
- "Users should never store secrets in client-side code." -> report "never store secrets".
|
||||
- "It is worth noting that the button is blue." -> no finding; ordinary filler belongs to another pack.
|
||||
- "The trial may cause drowsiness in some patients." -> report "may" and "some patients".
|
||||
@@ -0,0 +1,31 @@
|
||||
# Pack: Structure
|
||||
|
||||
Use this pack for document-level AI structure. Do not report isolated phrases unless they create a macro pattern.
|
||||
|
||||
## Look For
|
||||
|
||||
- `structure_scan.py` flags: `sentence_burstiness`, `conclusion_coda`, `bold_colon_listicle`, `one_line_staccato`, `connective_paragraph_openers`, `signpost_density`, `opener_repetition`, `participial_closer_share`.
|
||||
- Concrete thresholds to self-check: average sentence length under 8 or over 34 words is suspicious; sentence-length variance under 18 on 5+ sentences suggests uniform rhythm; 3+ one-line paragraphs under 8 words suggests staccato; 3+ consecutive paragraphs opened by connectives/signposts suggests scaffolding; 4+ repeated paragraph openers is blocking; 35%+ participial sentence endings suggests templated cadence.
|
||||
- Judge-only macro tells from `taboo-phrases.md`: both-sidesism, templated redemption arc, preview/recap symmetry, over-determination, uniform emotional register.
|
||||
- `silhouette_scan.py` flags idea arrangement above the surface: `scaffold_opener_share`, `callback_content` (recap loop), `role_entropy_bits`, `preview_fulfillment`, `heading_preview`, composited into `silhouette_penalty` (flags at `>= 1.0`).
|
||||
- A high `silhouette_penalty` means the document follows a template outline (preview-then-fulfill, rotating scaffold openers, closing recap); cite the specific flagged metric, not just the composite.
|
||||
- Cue-deletion evades silhouette but not surface repetition: pair `silhouette_scan` with `structure_scan` (`opener_repetition`, `one_line_staccato`) so a stripped-connective attack still fails one of the two fences.
|
||||
|
||||
Genre matters. `--genre docs` can allow reference-doc structure; `--genre social` can allow deliberate social cadence. Output still fails if the genre excuse is false.
|
||||
|
||||
## Emit
|
||||
|
||||
Return JSON findings only:
|
||||
|
||||
```json
|
||||
{"span":"paragraphs 2-5","rule":"connective_paragraph_openers","pack":"pack-structure","severity":"hard","note":"Paragraphs advance by scaffold words instead of content."}
|
||||
```
|
||||
|
||||
Quote the smallest span that shows the macro pattern. For whole-document patterns, use paragraph ranges.
|
||||
|
||||
## Examples
|
||||
|
||||
- "However... In addition... Consequently... In the end..." -> report connective scaffold and coda.
|
||||
- A technical comparison with "In Rails / In Django / In raw SQL" once -> no finding.
|
||||
- A story that always moves problem -> lesson -> uplift -> moral -> report `macro_redemption_arc`.
|
||||
- A report whose conclusion only recaps the intro -> report `macro_preview_recap`.
|
||||
@@ -0,0 +1,33 @@
|
||||
# Pack: Voice
|
||||
|
||||
Use this pack for manufactured voice and the skill's own replacement tells. Do not report core phrase boilerplate or factual/register constraints.
|
||||
|
||||
## Look For
|
||||
|
||||
- Anti-slop register: bare fragment contrasts such as "Not the tool. The team."; forced punch endings; repeated two- or three-word sentences.
|
||||
- Elegant variation and synonym cycling that calls attention to itself instead of clarifying.
|
||||
- Dramatic fragments, binary contrasts, negative parallelism, colon reveals, and rhetorical setups.
|
||||
- Contrastive-definition tails ("X is a build step, not an output."), two-beat imperative slogans ("Emit tokens. Ship bytes."), repeated "<plural noun> that <verb>." fragments, "One X, N Y." numeric parallelism, and abstractions that "ship inside" things.
|
||||
- Standalone slogan/spec fragment lines and headers: "N X, one Y." slogan cadence ("Four presets, one input.") and "N noun-phrase, past-participle ..." spec-sheet fragments ("Eight criteria, scored 1 to 5."). Flagged only when the whole line is the fragment; the same shape embedded in a sentence is a literal count and stays clean. Soft.
|
||||
- Headline slogan cadence: the "Short statement. Short statement." two-beat rhythm repeated across a document's headlines ("One command. A real URL." / "Reviewers click. The agent fixes."). One is voice; three or more is template grammar. Frequency-gated, soft.
|
||||
- Headline container agency: a calendar or program container delivers an artifact or outcome ("Week 1 ends with a calibrated eval"; "Demo day closes the cohort"). Name the completion criterion, mechanism, result, or decision instead. Literal boundaries such as "Week 1 ends on Friday" stay clean. Soft.
|
||||
- Tool anthropomorphism: an inanimate tool-noun given human agency. Reflexive self-agency fires anywhere ("The suite defends itself." / "The rules update themselves." / "It graded its own reflection."); strongly-volitional verbs (decides, hunts, wants, knows, believes, cares, refuses, judges, thinks) fire only on a standalone headline line ("The bench decides which model does which job." / "It hunts instances, not word lists."). Ordinary technical register stays clean ("the parser reads the file", "the model learns the distribution", "the test cleans up after itself"). Soft.
|
||||
- Punctuation performance: em-dash clusters, exclamation overuse, decorative bolding, title case inside body prose.
|
||||
- Warmth stripped into telegraphese: an email or narrative that loses natural softeners and sounds colder than the source.
|
||||
|
||||
## Emit
|
||||
|
||||
Return JSON findings only:
|
||||
|
||||
```json
|
||||
{"span":"Not the tooling. The conversations.","rule":"anti_slop_register","pack":"pack-voice","severity":"hard","note":"Bare fragment contrast is the skill's own tell; rewrite as varied prose."}
|
||||
```
|
||||
|
||||
Mark `hard` when the cadence is itself a recognizable formula. Mark `soft` when it may be genre-appropriate but needs review.
|
||||
|
||||
## Examples
|
||||
|
||||
- "Not the strategy. The execution." -> report `anti_slop_register`.
|
||||
- "We tested the API, fixed the retry path, and shipped." -> no finding.
|
||||
- "The result: alignment. The cost: trust." -> report colon reveal / fragment cadence.
|
||||
- "Thanks for sending this over. Could you send the figures by Friday?" -> no finding; natural warmth.
|
||||
@@ -0,0 +1,110 @@
|
||||
# Personality & Voice Guide
|
||||
|
||||
Removing AI patterns is necessary but not sufficient. Text that's clean of AI tells can still read as sterile, voiceless, and institutional. This guide covers the positive dimension: what makes writing feel like a person wrote it.
|
||||
|
||||
> **Two guardrails before you use any of this.**
|
||||
> 1. **Don't invent a person.** The first-person voice, opinions, and anecdotes below belong to writing that *has* an author (a personal post, a founder's note, an opinion piece). For impersonal copy — technical docs, reference text, third-party announcements — do **not** fabricate an "I"/"we" or a lived experience. An invented anecdote is a louder tell than the slop you removed.
|
||||
> 2. **Don't mistake de-hedging for voice.** "Show uncertainty honestly" means drop *habitual* hedging, not load-bearing scope or certainty. In technical, legal, medical, or safety writing, "may", "preliminary", "does not establish causation", and "most (73%)" are the content — keep them.
|
||||
|
||||
---
|
||||
|
||||
## Signs of Soulless Writing
|
||||
|
||||
Text that passes all taboo-phrase checks but still feels off:
|
||||
|
||||
| Signal | What It Looks Like |
|
||||
|--------|--------------------|
|
||||
| No stance | States facts without interpretation or opinion |
|
||||
| Uniform rhythm | Every sentence 12-16 words, same structure |
|
||||
| Zero first person | Never says "I" or "we" — hides behind passive |
|
||||
| No specificity | "Many companies" instead of naming one |
|
||||
| Overly balanced | Presents all sides equally, never commits |
|
||||
| No surprise | Every paragraph follows the expected arc |
|
||||
| Sanitized vocabulary | Only uses "safe" words — nothing vivid or unusual |
|
||||
| Perfect grammar | No fragments, contractions, or colloquialisms |
|
||||
|
||||
---
|
||||
|
||||
## How to Add Voice
|
||||
|
||||
### 1. Have Opinions
|
||||
|
||||
Voiceless: "There are advantages and disadvantages to remote work."
|
||||
Human: "Remote work is better for deep work. Open offices are where focus goes to die."
|
||||
|
||||
State what you believe. Qualify only when accuracy demands it, not out of habit.
|
||||
|
||||
### 2. Vary Rhythm Deliberately
|
||||
|
||||
Voiceless: "The team reviewed the metrics. The team discussed the findings. The team decided on next steps."
|
||||
Human: "The team reviewed metrics. Three hours of argument later, they had a plan. Not a great one, but a plan."
|
||||
|
||||
Mix short punches with longer flowing sentences. Break patterns. A fragment works. So does a 30-word sentence if it earns the length.
|
||||
|
||||
### 3. Be Specific
|
||||
|
||||
Voiceless: "The company experienced significant growth in the past year."
|
||||
Human: "Revenue hit $4.2M, up from $1.8M twelve months ago."
|
||||
|
||||
Names, numbers, dates. Replace every "significant" with the actual data.
|
||||
|
||||
### 4. Use First Person When Natural
|
||||
|
||||
Voiceless: "It is recommended to ship early and iterate."
|
||||
Human: "I ship early. Half the time the first version is wrong, but I learn faster than if I'd planned for six months."
|
||||
|
||||
"I" and "we" are not unprofessional. They signal a real person is speaking.
|
||||
|
||||
### 5. Leave Some Mess
|
||||
|
||||
Voiceless: "The approach combines agile methodology with user-centered design principles to deliver optimal outcomes."
|
||||
Human: "We mashed agile and UCD together. It's messy but it works. Users get what they need faster."
|
||||
|
||||
Real writing has parenthetical asides, self-corrections, and imperfect phrasing. Overpolished prose reads as synthetic.
|
||||
|
||||
### 6. Show Uncertainty Honestly
|
||||
|
||||
Voiceless: "Further research is needed to determine the optimal approach."
|
||||
Human: "I don't know the right answer yet. Two options look promising, and we'll know more after the pilot."
|
||||
|
||||
Admitting you don't know something is more human than hedging behind passive constructions. But this is for opinion and personal writing. In regulated or scientific content, "further research is needed" may be the precise, required claim — don't trade real epistemic caution for casual guesswork.
|
||||
|
||||
### 7. Use Concrete Sensory Language
|
||||
|
||||
Voiceless: "The office environment was suboptimal for productivity."
|
||||
Human: "Fluorescent lights, a printer that jammed every third page, and a thermostat war that never ended."
|
||||
|
||||
Let the reader see, hear, feel the situation instead of categorizing it abstractly.
|
||||
|
||||
---
|
||||
|
||||
## Before/After: Full Passage
|
||||
|
||||
### Before (Clean but Soulless)
|
||||
> Remote work has both benefits and challenges. Many organizations have adopted hybrid models. Research suggests that employee satisfaction increases with flexibility. Companies should consider their specific needs when designing work policies. The key is finding the right balance.
|
||||
|
||||
### After (Has Voice)
|
||||
> I've worked remote for three years. Some weeks I get more done by Thursday than I used to in a full office week. Other weeks I don't put on pants until noon and accomplish nothing. The trick isn't policy, it's trust. Let people figure out where they work best and hold them to outcomes, not hours.
|
||||
|
||||
### What Changed
|
||||
- Added first person and lived experience
|
||||
- Replaced "many organizations" with a specific perspective
|
||||
- Dropped the balanced-to-a-fault framing
|
||||
- "Finding the right balance" (corporate nothing) became a concrete recommendation
|
||||
- Included honest admission of unproductive days (human texture)
|
||||
|
||||
---
|
||||
|
||||
## Scoring Integration
|
||||
|
||||
When evaluating Human Authenticity (Rubric criterion 5), voice counts:
|
||||
|
||||
| Score | Voice Indicators |
|
||||
|-------|-----------------|
|
||||
| 5 | Clear authorial voice. Opinions stated. Rhythm varies. Could identify the writer. |
|
||||
| 4 | Personality visible. Mostly specific. Minor generic patches. |
|
||||
| 3 | Functional but anonymous. Could be anyone. No glaring AI tells. |
|
||||
| 2 | Corporate-neutral. All hedged. No specificity. Reads as template. |
|
||||
| 1 | Actively soulless. Reads like a policy document generated by committee. |
|
||||
|
||||
A score of 3 means the text is clean but not human. Aim for 4+.
|
||||
@@ -0,0 +1,357 @@
|
||||
# Tiered Execution Pipeline
|
||||
|
||||
## Core release protocol: Luna versus Luna
|
||||
|
||||
The product comparison uses `gpt-5.6-luna` for both arms on the same unfamiliar
|
||||
source. The treatment arm receives the frozen UNSLOP contract, scanner candidates,
|
||||
span-bounded rewrite flow, and validation gates. The raw arm receives neutral
|
||||
editorial guidance and no UNSLOP rules, repository access, scanner output, or
|
||||
validation feedback. A blinded `gpt-5.6-sol` judge sees randomized candidate
|
||||
labels. Report per-case calls, uncached input tokens, output tokens, elapsed time,
|
||||
detection precision and recall, repair success, preservation, damage, clean no-op,
|
||||
net improvement, and paired wins.
|
||||
|
||||
Run a fresh public split only after its corpus, annotations, thresholds, model
|
||||
controls, runner, scorer, acceptance code, and per-case budgets are frozen. Open
|
||||
the sealed holdback only after public acceptance passes. The current beta has no
|
||||
fresh valid paired result: the newest independently annotated corpus failed its
|
||||
preregistered composition floor before either arm ran. Earlier paired runs are
|
||||
directional evidence only.
|
||||
|
||||
The cross-provider tables below are ancillary engineering history. They do not
|
||||
answer the core release question and do not substitute for Luna-vs-Luna evidence.
|
||||
|
||||
The skill still works with one agent, but orchestrators should use the cheapest executor that can do each job.
|
||||
|
||||
## Tier 0: Deterministic Gates
|
||||
|
||||
Run first on the source text:
|
||||
|
||||
```bash
|
||||
python3 scripts/banned_phrase_scan.py <<< "$INPUT"
|
||||
python3 scripts/structure_scan.py <<< "$INPUT"
|
||||
python3 scripts/extract_constraints.py <<< "$INPUT"
|
||||
python3 scripts/readability_metrics.py <<< "$INPUT"
|
||||
```
|
||||
|
||||
Run again on output:
|
||||
|
||||
```bash
|
||||
python3 scripts/banned_phrase_scan.py <<< "$OUTPUT"
|
||||
python3 scripts/structure_scan.py <<< "$OUTPUT"
|
||||
python3 scripts/validate_preservation.py original.txt transformed.txt
|
||||
python3 scripts/readability_metrics.py <<< "$OUTPUT"
|
||||
python3 scripts/diff_check.py original.txt transformed.txt
|
||||
```
|
||||
|
||||
Output gates are blocking. Use `structure_scan.py --genre docs` or `--genre social` only when the final text truly belongs to that genre.
|
||||
|
||||
## Tier 1: Small Detector Agents
|
||||
|
||||
Run one small detector per `(pack x chunk)`. Use paragraph chunks up to about 500 words. Run `pack-register-guards` on the whole text because scope, negation, and legal/security force need context.
|
||||
|
||||
Detector agents only read their assigned pack from `references/packs/` and return findings:
|
||||
|
||||
```json
|
||||
{"span":"...","rule":"...","pack":"...","severity":"hard|soft","note":"..."}
|
||||
```
|
||||
|
||||
They do not rewrite, score, or import rules from other packs.
|
||||
|
||||
## Tier 2: Rewriter
|
||||
|
||||
Send one strong-enough rewriter the original text, merged Tier 0/Tier 1 findings, extracted constraints, and the selected preset. It rewrites from findings instead of re-running detection in prose. It must preserve register-guard and fact findings, remove phrase/structure/voice findings, and avoid introducing anti-slop-register tells.
|
||||
|
||||
## Model Tiers
|
||||
|
||||
| Job | Default executor | Escalate when |
|
||||
|---|---|---|
|
||||
| Tier 0 scripts | local deterministic | never; fix the script or input |
|
||||
| Detection packs | cheapest tier (see Model Parity) | JSON is malformed or pack scope is violated |
|
||||
| Span replacement / short rewrite | cheapest tier; gates carry safety (parity 2026-07-06: 8/8) | output fails a blocking gate twice |
|
||||
| Full rewrite of register-sensitive text (legal, medical, security, load-bearing hedges) | strongest practical model + mandatory Tier-0 re-scan | start here; cheap tiers erode register |
|
||||
| Macro structure (restructuring, coda/preview removal) | machine-gated AND machine-corrected via the structure climb (generate→scan→directive→regenerate); both frontier tiers converge fast, cheapest-that-converges is model-dependent (Anthropic haiku-4-5 at a slightly larger round cap; OpenAI's cheap tier not yet shown to converge) | never trust a model's own macro self-check — feed the scanners' directives back and re-scan (see Macro structure under the climb) |
|
||||
| Judge/eval | model specified by `evals/BEHAVIORAL-EVALS.md` | benchmark protocol changes |
|
||||
|
||||
The model-dependent rows above are not set by taste. They are set by
|
||||
`evals/run_model_parity.py` (see Model Parity), whose live matrix was recorded 2026-07-06:
|
||||
span replacement clears on the cheapest tier because the output gates carry safety, while
|
||||
full rewrites of register-sensitive text and any macro restructuring escalate to the
|
||||
strongest practical model with a Tier-0 re-scan. Re-run the harness when the model features
|
||||
change and update these rows from its output.
|
||||
|
||||
If Tier 2 output fails the same blocking gate twice, escalate one model tier. Do not add more rules to the prompt; the failure is execution quality.
|
||||
|
||||
## Model Parity
|
||||
|
||||
The pipeline depends on a model in exactly two places: Tier-1 pack **detection** and
|
||||
Tier-2 **replacement** generation. Everything else is deterministic. Whether a cheap model
|
||||
may own those surfaces is a measured question, not an assumption.
|
||||
|
||||
`evals/run_model_parity.py` measures it against a fixed seeded corpus:
|
||||
|
||||
- **Task A (detection).** Each model reads one pack file plus a short seeded chunk (the
|
||||
Tier-1 contract above) and returns JSON findings. Grading is deterministic: recall of the
|
||||
seeded findings and a count of false findings, scored against a frozen manifest. Six
|
||||
fixtures span the phrases-core, voice, and register-guard families (two are register-guard
|
||||
cases, where the load-bearing hedge must be caught).
|
||||
- **Task B (replacement).** Each model is handed one seeded finding (span + rationale) and
|
||||
asked for a span-minimal replacement. Grading applies the co-writer contract: the
|
||||
replacement removes the flagged tell and adds no new one (both scanners), preserves every
|
||||
fact/number/negation, and stays span-minimal.
|
||||
|
||||
The matrix is config-driven — entries are `{name, kind, model_id}` where `kind` is
|
||||
`claude-cli` (`claude -p --model <id>`, the Anthropic spectrum) or `openrouter` (the GPT
|
||||
spectrum and others, key read from the macOS keychain service `OPENROUTER_API_KEY`). The
|
||||
harness emits a per-model / per-task score table (JSON and a markdown summary) plus a
|
||||
`--dry-run` mode that grades canned response fixtures with no network for the `PARITY-*`
|
||||
eval rows.
|
||||
|
||||
**Rule (binding).** Any change to the co-writer, mimic, or detector-pack model features
|
||||
must run this harness across both the GPT and Anthropic spectrums before merge, and the
|
||||
detection/replacement rows of the tiering table above must be updated from its output. If
|
||||
cheap models match the strong reference, use cheap everywhere; if they do not, keep the
|
||||
strong model for the passes where the gap appears and record which. The `PARITY-*` rows in
|
||||
`evals/adversarial-evals.json` gate the grader itself (dry-run, no network); they do not
|
||||
substitute for the live run.
|
||||
|
||||
### Recorded results
|
||||
|
||||
Live matrix recorded **2026-07-06**. `evals/run_model_parity.py` re-measures on demand;
|
||||
rerun it after any change to the co-writer, mimic, or detector-pack model features and
|
||||
update these tables from its output.
|
||||
|
||||
**Task B — span replacement.** One seeded finding (span + rationale) per trial, graded on
|
||||
the co-writer contract: the replacement removes the flagged tell, adds no new one (both
|
||||
scanners), preserves every fact/number/negation, and stays span-minimal. Eight seeded
|
||||
findings per model.
|
||||
|
||||
| Model | Kind | Task B pass rate |
|
||||
|---|---|---:|
|
||||
| claude-haiku | claude-cli | 8/8 |
|
||||
| claude-sonnet | claude-cli | 8/8 |
|
||||
| gpt-5.4-mini | openrouter | 8/8 |
|
||||
| gpt-5.5 | openrouter | 8/8 |
|
||||
| claude-opus | claude-cli | 7/8 |
|
||||
|
||||
The single miss was opus dropping the fact "8:30" from a replacement — caught by
|
||||
`validate_preservation`, not by model choice. On the mechanical span contract the cheapest
|
||||
tier ties the flagships; the gates carry safety, the tier does not.
|
||||
|
||||
**Behavioral parity matrix — full rewrites.** Eight register/structure cases per model,
|
||||
graded deterministically across both the Anthropic and GPT spectrums.
|
||||
|
||||
| Model | Kind | Full-rewrite pass rate |
|
||||
|---|---|---:|
|
||||
| claude-opus | claude-cli | 7/8 |
|
||||
| gpt-5.4-mini | openrouter | 7/8 |
|
||||
| claude-sonnet | claude-cli | 6/8 |
|
||||
| gpt-5.5 | openrouter | 6/8 |
|
||||
| claude-haiku | claude-cli | 5/8 |
|
||||
| gpt-5.4-nano | openrouter | 5/8 |
|
||||
|
||||
`MACRO-01` (`structure_clean`) failed for all six models, opus included: each kept a
|
||||
conclusion coda the prose instruction told it to drop. No model self-checks macro structure
|
||||
from prose. The cheap-tier misses were register and fact erosion in full rewrites — softened
|
||||
`never`/`all`, a dropped `roughly`, a dropped legal `arguably`/negation, an emitted
|
||||
`X, not Y` contrastive tail. The lab ladders are symmetric (7/6/5 on both spectrums).
|
||||
|
||||
**Measured conclusions.**
|
||||
|
||||
- **Span replacement = cheapest tier, with gates on.** The output gates enforce fact and
|
||||
tell safety, so the smallest model is safe for span-minimal replacement.
|
||||
- **Full rewrites of register-sensitive text = frontier.** Legal, medical, security, or any
|
||||
text with load-bearing hedges/negation goes to the strongest practical model with a
|
||||
mandatory Tier-0 re-scan; cheap models erode register in unsupervised full rewrites.
|
||||
- **Macro structure = always machine-gated, never self-checked — but machine-correctable.**
|
||||
A Tier-0 scan surfaces it as an explicit directive, because no model, flagship included,
|
||||
reliably catches it from prose instructions. Fed those directives back in a scan-regenerate
|
||||
loop (`evals/run_structure_climb.py`), both frontier tiers converge to clean fast; whether
|
||||
the cheap tier converges is model-dependent, not a uniform "cheap tier" property -- see
|
||||
"Macro structure under the climb" for the recorded before/after and the follow-up
|
||||
experiments.
|
||||
|
||||
### Open-weights spot check — 2026-07-07
|
||||
|
||||
Live matrix run against the current open-weights flagships of three Chinese labs, each the
|
||||
newest general chat release of its family on OpenRouter: DeepSeek V4 Pro
|
||||
(`deepseek/deepseek-v4-pro`), Kimi K2.6 (`moonshotai/kimi-k2.6`), and GLM 5.2
|
||||
(`z-ai/glm-5.2`). Same harness and contracts as above; six detection fixtures (Task A) and
|
||||
six replacement fixtures (Task B) per model. Models file:
|
||||
`evals/fixtures/parity/models_openweights.json`.
|
||||
|
||||
| Model | Kind | Task A mean recall | Task A false findings | Task B pass rate |
|
||||
|---|---|---:|---:|---:|
|
||||
| glm-5.2 | openrouter | 1.00 | 0 | 6/6 |
|
||||
| deepseek-v4-pro | openrouter | 0.92 | 1 | 6/6 |
|
||||
| kimi-k2.6 | openrouter | 0.92 | 0 | 6/6 |
|
||||
|
||||
All three clear the span-replacement contract at 6/6, and the only detection misses were
|
||||
one seeded voice span apiece (fixture A4) for DeepSeek and Kimi; on both model-dependent
|
||||
surfaces the open-weights flagships sit level with the recorded GPT and Anthropic ladders.
|
||||
|
||||
## Macro structure under the climb — 2026-07-07
|
||||
|
||||
The recorded parity matrix above closed on a flat verdict: single-pass macro-structure
|
||||
self-checking failed on every model. Told in prose to drop the coda / connective scaffold /
|
||||
outline echo, each one shipped it anyway. But the deterministic scanners SEE those failures
|
||||
precisely, so the failure is correctable even when it is not self-checkable. `evals/run_structure_climb.py`
|
||||
closes that loop:
|
||||
|
||||
generate -> scan (structure_scan + silhouette_scan) -> if clean, done
|
||||
-> else turn each finding into a TARGETED directive that names WHERE and
|
||||
WHAT -> feed the directives + current draft back -> regenerate, capped at
|
||||
N rounds (default 4).
|
||||
|
||||
The core is the **directive builder** (`build_directives`): a deterministic map from scanner
|
||||
findings to instructions that locate the tell, because the model cannot see it. It re-reads
|
||||
the draft with the scanners' own helpers, so a `conclusion_coda` flag becomes *"the final
|
||||
paragraph (`Ultimately, ...`) restates the opening; end on a new fact"*, a
|
||||
`connective_paragraph_openers` flag names the offending paragraph numbers and their opener
|
||||
words, and a silhouette `callback_content` flag names the exact opening vocabulary the ending
|
||||
loops back to. Every macro flag both scanners can emit maps to a directive (gated by
|
||||
`CLIMB-06`). A preservation guard (`validate_preservation`, anchored to `--source-file` when
|
||||
given) validates every round, so climbing toward clean structure can never eat a fact; a
|
||||
regression aborts the climb. Honest terminal states (`converged` / `capped` /
|
||||
`preservation_violation`) carry distinct exit codes. The mock path (`CLIMB-*` rows,
|
||||
`evals/fixtures/climb/`) exercises all of this offline; the live path drives real models via
|
||||
`--generate-cmd`.
|
||||
|
||||
Live run on the same `SKILL-MACRO-01` robotics-essay fixture as Task C, `--max-rounds 4`,
|
||||
preservation anchored to the source essay:
|
||||
|
||||
| Model | Tier | Single pass (round 0) | Climbed | Rounds | Terminal | Facts |
|
||||
|---|---|---|---|---:|---|---|
|
||||
| claude-sonnet-4-5 | frontier | dirty: `signpost_density`, `callback_content` (penalty 7.5) | both scanners clean | 3 | converged | preserved every round |
|
||||
| claude-haiku-4-5 | cheap | dirty: `callback_content` (penalty 2.5) | 1 residual structure flag | 4 | capped | preserved every round |
|
||||
| gpt-5.5 | frontier | pending | pending | n/a | pending (OpenRouter HTTP 402) | pending |
|
||||
| glm-5.2 | open-weights | pending | pending | n/a | pending (OpenRouter HTTP 402) | pending |
|
||||
|
||||
Read honestly:
|
||||
|
||||
- **Single pass ships macro tells on both Claude tiers**, reconfirming the thesis: told to
|
||||
remove macro tells, the model does not catch its own document shape.
|
||||
- **The climb recovers macro on the frontier model where the single pass failed.** Sonnet
|
||||
converged to both-scanners-clean in three rounds (violation trajectory 2 → 1 → 0) with every
|
||||
fact preserved. This is the doctrine headline: with the loop, macro structure becomes
|
||||
machine-*corrected*, a step past machine-*gated*.
|
||||
- **On the cheap tier the recovery is partial, not full.** haiku-4-5 consumed the silhouette
|
||||
recap loop on the first directive and held silhouette clean from round 1 on, but it
|
||||
oscillated on surface structure (re-introducing a coda, then a uniform cadence) and did not
|
||||
converge within four rounds, capping at one residual structure flag. The loop improves the
|
||||
cheap tier but does not guarantee it at cap 4; a residual macro flag on a cheap model still
|
||||
routes to a stronger model or another round, not to shipping.
|
||||
- **The preservation guard held live throughout** (source-anchored): no round on either model
|
||||
eroded a fact.
|
||||
- **The OpenRouter spectrum (GPT + open-weights) is pending on account billing** (HTTP 402
|
||||
Payment Required at run time), not on the harness. Exact reproduction commands:
|
||||
|
||||
```bash
|
||||
# Anthropic spectrum (native claude -p; run above):
|
||||
python3 evals/run_structure_climb.py --prompt-file <macro-prompt> --source-file <essay> \
|
||||
--out out/haiku --generate-cmd "claude -p --model claude-haiku-4-5" --max-rounds 4
|
||||
# OpenRouter spectrum (needs OPENROUTER_API_KEY credit):
|
||||
python3 evals/run_structure_climb.py --prompt-file <macro-prompt> --source-file <essay> \
|
||||
--out out/gpt --generate-cmd "python3 evals/model_generate.py --kind openrouter --model openai/gpt-5.5" --max-rounds 4
|
||||
python3 evals/run_structure_climb.py --prompt-file <macro-prompt> --source-file <essay> \
|
||||
--out out/glm --generate-cmd "python3 evals/model_generate.py --kind openrouter --model z-ai/glm-5.2" --max-rounds 4
|
||||
```
|
||||
|
||||
### Two follow-up experiments — 2026-07-07
|
||||
|
||||
The OpenRouter 402 blocked the GPT spectrum above, and the cap-4 haiku row left one open
|
||||
question (does the cheap tier ever converge, or does it just stall). Two follow-ups, both run
|
||||
against the identical `SKILL-MACRO-01` fixture and source-anchored preservation:
|
||||
|
||||
**A. A local alternative to OpenRouter for the GPT side.** The `codex` CLI (OpenAI's agentic
|
||||
coding CLI) is installed locally and can drive one model call per round the same way
|
||||
`claude -p` does, sidestepping the billing block. `codex exec` is an AGENT CLI, though: its
|
||||
stdout is an interleaved transcript (tool calls, hook lines, token counts), not a clean
|
||||
document, so it cannot be piped straight into the climb loop. Its `-o/--output-last-message
|
||||
FILE` flag writes only the agent's final text turn with no wrapping -- verified by hand against
|
||||
this fixture before wiring it in. `evals/model_generate.py` gained a `codex` kind
|
||||
(`call_codex`) that runs `codex exec --sandbox read-only --ephemeral --skip-git-repo-check -o
|
||||
FILE -m <model> -`, reads that file, and discards the transcript. KNOWN RISK: `codex exec` has
|
||||
been observed to hang silently in this environment; `call_codex` runs it in its own process
|
||||
group with a hard wall-clock timeout (180s default) and SIGKILLs the whole group on expiry, so
|
||||
a hang surfaces as an honest failed round instead of blocking the climb forever. `CLIMB-07`
|
||||
exercises this offline against a fake `codex` binary (success, nonzero-exit, empty-output, and
|
||||
hang paths; the hang case is SIGKILLed in about a second against a 1s test timeout).
|
||||
|
||||
**B. Does the cheap Anthropic tier ever converge, or does cap 4 just cut it off early.**
|
||||
`claude-haiku-4-5` re-run with `--max-rounds 8` (same fixture, same generator) to see whether
|
||||
the one residual flag at cap 4 was a hard ceiling or a round short.
|
||||
|
||||
Results, same fixture and settings as the recorded matrix above except where noted:
|
||||
|
||||
| Model | Tier | Kind | Single pass (round 0) | Climbed | Rounds (cap) | Terminal | Facts |
|
||||
|---|---|---|---|---|---:|---|---|
|
||||
| gpt-5.5 | frontier | codex exec (default model) | dirty: `sentence_burstiness`, `conclusion_coda`, `preview_fulfillment`, `callback_content` (4 flags) | both scanners clean | 4 (4) | converged | preserved every round |
|
||||
| gpt-5.4-mini | cheap | codex exec | dirty: `sentence_burstiness`, `opener_repetition`, `preview_fulfillment`, `callback_content` (4 flags) | 2 residual flags, oscillating (4 → 1 → 5 → 2) | 4 (4) | capped | preserved every round |
|
||||
| gpt-5.4-mini | cheap | codex exec, retried | dirty: `sentence_burstiness`, `preview_fulfillment`, `callback_content` (3 flags) | 5 residual flags, oscillating (3 → 3 → 2 → 1 → 6 → 2 → 1 → 5) | 8 (8) | capped | preserved every round |
|
||||
| claude-haiku-4-5 | cheap | claude-cli, retried | dirty: `callback_content` (1 flag) | both scanners clean | 5 (8) | converged | preserved every round |
|
||||
|
||||
Read honestly:
|
||||
|
||||
- **The `-o` extraction is clean and codex is a real generator for this loop.** No stray
|
||||
transcript text ever reached the scanners; every round's draft was bare prose. `gpt-5.5`
|
||||
through `codex exec` converged in the same 4-round budget sonnet used (3 rounds), matching
|
||||
the frontier-converges finding on the Anthropic side.
|
||||
- **Codex's cheap tier does not converge, even given double the round budget.** `gpt-5.4-mini`
|
||||
was run twice -- once at the recorded cap of 4, once at cap 8 -- and never reached clean
|
||||
either time. Its violation trajectory does not fall monotonically the way sonnet's or
|
||||
haiku's did; it swings between 1 and 6 flags round to round. The loop still holds every fact
|
||||
(preservation passed all 12 rounds across the two runs), but macro structure on this model
|
||||
is not machine-*correctable* at any round budget tested here, only machine-*gated*.
|
||||
- **The cheap Anthropic tier was one round short of converging, not permanently capped.**
|
||||
`claude-haiku-4-5` re-run at `--max-rounds 8` converged in 5 rounds (violation trajectory
|
||||
1 → 1 → 1 → 1 → 0), the same residual `callback_content` flag from the cap-4 run finally
|
||||
clearing once the loop got one more pass. Every round preserved every fact.
|
||||
- **This changes the doctrine.** "Cheap tiers partially recover" (the prior wording) was true
|
||||
of the one cheap model measured so far, but it is not a property of "cheap" in general --
|
||||
it is model-dependent. `claude-haiku-4-5` fully recovers with a slightly larger round cap;
|
||||
`gpt-5.4-mini` does not recover with an even larger one. Read the two model families
|
||||
separately, not as one "cheap tier" line.
|
||||
- **Cheapest model that does this well: `claude-haiku-4-5`, with `--max-rounds` raised from 4
|
||||
to 6** (5 rounds measured to converge here; 6 gives one round of margin). On the GPT side,
|
||||
the floor that has actually been shown to converge is `gpt-5.5` (frontier) via codex, not
|
||||
`gpt-5.4-mini` -- the OpenAI cheap tier is not yet a substitute for this task on this
|
||||
evidence.
|
||||
- **The OpenRouter spectrum itself (`gpt-5.5` via API, and `glm-5.2`) is still pending on
|
||||
account billing** (HTTP 402), not on the harness. Codex gave a working local alternative for
|
||||
measuring OpenAI's frontier and cheap models on this task, but it drives the model through
|
||||
Codex's own agent scaffolding and system prompt, not the raw chat-completions endpoint --
|
||||
read the codex-CLI numbers above as "the GPT models, as codex CLI calls them," not as an
|
||||
exact stand-in for a raw OpenRouter measurement once billing clears. `glm-5.2` has no local
|
||||
CLI alternative and remains unmeasured.
|
||||
|
||||
Exact reproduction commands:
|
||||
|
||||
```bash
|
||||
# Anthropic spectrum, cheap tier, extended round cap:
|
||||
python3 evals/run_structure_climb.py --prompt-file <macro-prompt> --source-file <essay> \
|
||||
--out out/haiku8 --generate-cmd "claude -p --model claude-haiku-4-5" --max-rounds 8
|
||||
# Codex CLI, GPT spectrum (frontier default, then cheapest):
|
||||
python3 evals/run_structure_climb.py --prompt-file <macro-prompt> --source-file <essay> \
|
||||
--out out/codex-gpt5.5 --generate-cmd "python3 evals/model_generate.py --kind codex --model gpt-5.5" --max-rounds 4
|
||||
python3 evals/run_structure_climb.py --prompt-file <macro-prompt> --source-file <essay> \
|
||||
--out out/codex-mini --generate-cmd "python3 evals/model_generate.py --kind codex --model gpt-5.4-mini" --max-rounds 8
|
||||
# OpenRouter spectrum (still needs OPENROUTER_API_KEY credit; unchanged from above):
|
||||
python3 evals/run_structure_climb.py --prompt-file <macro-prompt> --source-file <essay> \
|
||||
--out out/gpt --generate-cmd "python3 evals/model_generate.py --kind openrouter --model openai/gpt-5.5" --max-rounds 4
|
||||
python3 evals/run_structure_climb.py --prompt-file <macro-prompt> --source-file <essay> \
|
||||
--out out/glm --generate-cmd "python3 evals/model_generate.py --kind openrouter --model z-ai/glm-5.2" --max-rounds 4
|
||||
```
|
||||
|
||||
## Cost Note
|
||||
|
||||
For a 1000-word document:
|
||||
|
||||
| Path | Calls | Approx tokens |
|
||||
|---|---:|---:|
|
||||
| Tier 0 | local scripts | free |
|
||||
| Tier 1 | about 5 small detector calls | about 10k small-model tokens |
|
||||
| Tier 2 | one rewrite call | source + findings + preset |
|
||||
| Monolithic | one large call with full skill and all references | often 8k-15k strong-model tokens before output |
|
||||
|
||||
The tiered path spends cheap tokens on detection, reserves the stronger model for rewriting, and gives the final Tier 0 gates authority.
|
||||
@@ -0,0 +1,90 @@
|
||||
# Adversarial refresh
|
||||
|
||||
The Growth section of `docs/PRODUCT.md` promises that detection catalogs
|
||||
stay current through adversarial refresh. This page is the procedure. It is
|
||||
what an agent runs to make that promise real. Nothing here runs on a
|
||||
schedule (see `docs/DECISIONS.md`, "Agent-invoked only"); it runs when an
|
||||
agent, or the person driving one, decides to check.
|
||||
|
||||
## 1. Check staleness
|
||||
|
||||
```bash
|
||||
python3 scripts/refresh_status.py
|
||||
```
|
||||
|
||||
Stdlib, offline, always exits 0. It reports three signals as JSON:
|
||||
|
||||
- `wiki_sync`: days since the local Wikipedia sync state was last updated.
|
||||
- `parity_bench`: days since the model-parity matrix in
|
||||
`references/pipeline.md` was last recorded.
|
||||
- `newest_pattern_row`: days since `evals/adversarial-evals.json` last
|
||||
gained a row.
|
||||
|
||||
Each carries `last` (an ISO date or `null`), `days` (or `null`), and a
|
||||
`stale` flag; the script never fails a build over it, and the flag is a
|
||||
prompt for judgment rather than a rule to obey mechanically. The thresholds
|
||||
it applies (wiki 90 days, bench 180 days, rows 60 days) are commented in the
|
||||
script itself as starting points.
|
||||
|
||||
## 2. If `wiki_sync` is stale
|
||||
|
||||
```bash
|
||||
python3 scripts/wiki_sync.py check # exit 0 = no updates, 1 = updates available
|
||||
python3 scripts/wiki_sync.py diff # structured JSON diff, when check exits 1
|
||||
```
|
||||
|
||||
For each new word or phrase in the diff, follow "Add a banned phrase" in
|
||||
`references/maintenance.md`: eval rows first, then the scanner, then the
|
||||
catalog. Skip Wikipedia-specific patterns, such as broken wikitext, DOI
|
||||
formatting, or citation graffiti. Only genuine AI tells in general prose
|
||||
belong here.
|
||||
|
||||
## 3. If `parity_bench` is stale
|
||||
|
||||
Treat staleness as a prompt to double-check the recorded table against
|
||||
reality. It is not an automatic re-run trigger by itself: the binding rule
|
||||
in `references/pipeline.md` (Model Parity) already forces a re-run whenever
|
||||
the co-writer, mimic, or detector-pack model features change, independent of
|
||||
elapsed time. When a re-run is warranted, run the live matrix across both
|
||||
spectrums:
|
||||
|
||||
```bash
|
||||
python3 evals/run_model_parity.py --task both --format both
|
||||
```
|
||||
|
||||
Update the tiering table and the "Recorded results" tables in
|
||||
`references/pipeline.md` from the output, and update the standing
|
||||
conclusions in `docs/PRODUCT.md`'s Model tiering section if they changed.
|
||||
The `PARITY-*` rows in `evals/adversarial-evals.json` gate the grader itself
|
||||
in dry-run mode. They do not substitute for the live run above:
|
||||
|
||||
```bash
|
||||
python3 evals/run_model_parity.py --dry-run \
|
||||
--responses evals/fixtures/parity/canned_responses.json
|
||||
```
|
||||
|
||||
## 4. Every miss becomes a row
|
||||
|
||||
A miss can come from a live bench output, a wiki-sync diff, or something
|
||||
caught by hand. Whichever it is, route it through
|
||||
`references/commands/contribute.md`. Contribute turns the exact specimen
|
||||
into a red-first eval row and a structured PR, so the miss becomes a
|
||||
permanent regression test instead of a one-off fix.
|
||||
|
||||
## 5. Record results
|
||||
|
||||
- Model-parity runs: update `references/pipeline.md`'s dated tables (the
|
||||
"Live matrix recorded **YYYY-MM-DD**" line and the two results tables
|
||||
under it) with the new date and numbers.
|
||||
- Behavioral or skill runs: follow the conventions already set in
|
||||
`evals/TUNE-RESULTS.md`. Name the harness version, split, runner, and
|
||||
judge. Then report per-case deltas rather than aggregate lift, per
|
||||
`CLAUDE.md`'s Interpreting Results section.
|
||||
|
||||
## Cadence guidance
|
||||
|
||||
There is no scheduler. Use `refresh_status.py`'s thresholds as the trigger
|
||||
for a manual or agent-invoked check instead: wiki sync roughly quarterly,
|
||||
the parity bench roughly every six months (sooner on any model-feature
|
||||
change, per the binding rule above), and a look at the eval catalog if two
|
||||
months pass with no new row.
|
||||
@@ -0,0 +1,17 @@
|
||||
# Rewrite examples
|
||||
|
||||
These compact examples are executable documentation. They illustrate removal
|
||||
of stock framing and inflated abstraction; they do not override the core
|
||||
contract.
|
||||
|
||||
**Input:**
|
||||
> Here's the thing: building products is hard. Not because the technology is complex. Because people are complex. Let that sink in.
|
||||
|
||||
**Output (crisp):**
|
||||
> Building products is hard, and not because the technology is complex. People are complex, and they are the part you cannot refactor.
|
||||
|
||||
**Input:**
|
||||
> In today's fast-paced business environment, it's becoming increasingly important for organizations to leverage their core competencies while navigating the complex landscape of digital transformation.
|
||||
|
||||
**Output (crisp):**
|
||||
> Companies need to use their strengths while going digital.
|
||||
@@ -0,0 +1,171 @@
|
||||
# Scoring Rubric
|
||||
|
||||
Score transformed text on 8 criteria (1-5 scale each). Minimum passing: 32/40 (80%).
|
||||
|
||||
---
|
||||
|
||||
## 1. Directness (1-5)
|
||||
|
||||
Does the text get to the point without hedging or softening?
|
||||
|
||||
| Score | Criteria |
|
||||
|-------|----------|
|
||||
| 5 | Opens with main point. No qualifiers, no throat-clearing. Every sentence advances the argument. |
|
||||
| 4 | Minimal hedging. One or two soft qualifiers. Gets to point within first sentence. |
|
||||
| 3 | Some unnecessary setup. A few hedging phrases. Main point clear but delayed. |
|
||||
| 2 | Multiple hedges per paragraph. Excessive qualifiers. Main point buried. |
|
||||
| 1 | Constant hedging. Throat-clearing openers. Reader must hunt for the point. |
|
||||
|
||||
**Red flags:** "It's worth noting," "To be fair," "I think," "Perhaps," "Maybe," "Kind of," "Sort of"
|
||||
|
||||
---
|
||||
|
||||
## 2. Natural Rhythm (1-5)
|
||||
|
||||
Does sentence length vary naturally? Does it read like speech, not template?
|
||||
|
||||
| Score | Criteria |
|
||||
|-------|----------|
|
||||
| 5 | Sentence lengths vary 8-25 words naturally. Mix of structures. No mechanical patterns. |
|
||||
| 4 | Good variance. One or two repetitive structures. Flows well overall. |
|
||||
| 3 | Some variance but noticeable patterns. Occasional monotony. |
|
||||
| 2 | Most sentences similar length. Obvious templated rhythm. |
|
||||
| 1 | Robotic uniformity. Every sentence same structure. Machine-generated feel. |
|
||||
|
||||
**Test:** Read aloud. Does it sound like a person talking?
|
||||
|
||||
---
|
||||
|
||||
## 3. Concrete Verbs (1-5)
|
||||
|
||||
Are verbs specific and active, or abstract and passive?
|
||||
|
||||
| Score | Criteria |
|
||||
|-------|----------|
|
||||
| 5 | All verbs concrete and specific. Actions you can visualize. No abstract placeholders. |
|
||||
| 4 | Mostly concrete verbs. One or two abstract (leverage, utilize). |
|
||||
| 3 | Mix of concrete and abstract. Some passive voice. |
|
||||
| 2 | Dominated by abstract verbs. Heavy passive voice. |
|
||||
| 1 | Almost entirely abstract verbs. "Utilize," "leverage," "navigate" throughout. |
|
||||
|
||||
**Red flags:** leverage, utilize, facilitate, implement, optimize, navigate, unpack, foster
|
||||
|
||||
**Good verbs:** build, cut, fix, send, write, call, test, ship, break
|
||||
|
||||
---
|
||||
|
||||
## 4. Reader Trust (1-5)
|
||||
|
||||
Does the text trust the reader, or over-explain and patronize?
|
||||
|
||||
| Score | Criteria |
|
||||
|-------|----------|
|
||||
| 5 | Assumes reader competence. No stating the obvious. Respects intelligence. |
|
||||
| 4 | Mostly trusting. One instance of over-explanation. |
|
||||
| 3 | Some hand-holding. Occasional obvious statements. |
|
||||
| 2 | Frequent over-explanation. Multiple obvious statements. |
|
||||
| 1 | Constant patronizing. "Think about it," "Let that sink in," "In other words." |
|
||||
|
||||
**Red flags:** "Think about it," "Let that sink in," "In other words," "To put it simply," "As you know"
|
||||
|
||||
---
|
||||
|
||||
## 5. Human Authenticity (1-5)
|
||||
|
||||
Does the text sound like a person wrote it, or an AI performing humanity? This criterion evaluates both the absence of AI patterns and the presence of human voice.
|
||||
|
||||
| Score | Criteria |
|
||||
|-------|----------|
|
||||
| 5 | Could have been written by a specific person. Has voice, opinion, and lived experience. Rhythm varies naturally. Reader can sense the author's personality. |
|
||||
| 4 | Sounds human. Clear perspective. Minor generic patches. Mostly specific with occasional abstraction. |
|
||||
| 3 | Clean but anonymous. No glaring AI tells, but no personality either. Could be anyone. Reads as competent but soulless. |
|
||||
| 2 | Generic tone. Multiple AI patterns. Corporate-neutral. Hedges everything. No first person, no specificity, no stance. |
|
||||
| 1 | Obvious AI. Performative emphasis, binary contrasts, dramatic fragmentation. Significance inflation, promotional language, chatbot artifacts. |
|
||||
|
||||
**Red flags (AI tells):** "Full stop." "Period." "Let me be clear." Stacked short sentences for effect. Rhetorical questions answered immediately. Significance inflation ("stands as a testament"). Promotional language ("nestled", "boasts"). Superficial -ing analyses. Vague attributions ("experts argue"). Copula avoidance ("serves as a"). Communication artifacts ("I hope this helps").
|
||||
|
||||
**Red flags (soulless):** No opinions stated. No first person. Every claim hedged. Uniform sentence rhythm. No specific names, numbers, or dates. Balanced-to-a-fault framing. Sanitized vocabulary only. See `references/personality-guide.md` for detailed guidance.
|
||||
|
||||
---
|
||||
|
||||
## 6. Content Density (1-5)
|
||||
|
||||
What's the ratio of substance to filler?
|
||||
|
||||
| Score | Criteria |
|
||||
|-------|----------|
|
||||
| 5 | Every word necessary. No filler. Dense with information or insight. |
|
||||
| 4 | Minimal filler. One or two unnecessary phrases. Tight overall. |
|
||||
| 3 | Some padding. A few filler phrases. Content present but diluted. |
|
||||
| 2 | Significant padding. Multiple filler phrases per paragraph. |
|
||||
| 1 | More filler than content. Adverb-heavy. "In today's landscape" type phrases throughout. |
|
||||
|
||||
**Filler indicators:** "At its core," "In today's [X]," "When it comes to," "At the end of the day," "Moving forward"
|
||||
|
||||
---
|
||||
|
||||
## 7. Fact Preservation (1-5)
|
||||
|
||||
Are all factual elements from the original intact?
|
||||
|
||||
| Score | Criteria |
|
||||
|-------|----------|
|
||||
| 5 | All facts preserved exactly. Numbers, names, dates, URLs unchanged. |
|
||||
| 4 | All facts preserved. Minor rephrasing of non-critical details. |
|
||||
| 3 | Most facts preserved. One minor fact altered. |
|
||||
| 2 | Some facts missing or changed. |
|
||||
| 1 | Significant fact loss or alteration. |
|
||||
|
||||
**Must preserve:** Numbers, percentages, dates, proper nouns, URLs, quotes, technical terms, citations
|
||||
|
||||
---
|
||||
|
||||
## 8. Template Avoidance (1-5)
|
||||
|
||||
Is the structure organic, or following AI templates?
|
||||
|
||||
| Score | Criteria |
|
||||
|-------|----------|
|
||||
| 5 | Unique structure. No recognizable AI patterns. Organic flow. |
|
||||
| 4 | Mostly organic. One minor template echo. |
|
||||
| 3 | Some template patterns. Binary contrasts or lists visible. |
|
||||
| 2 | Multiple template patterns. Predictable structure. |
|
||||
| 1 | Full AI template. Three-item lists, binary contrasts, dramatic fragments throughout. |
|
||||
|
||||
**Template patterns:**
|
||||
- "Not X. Y." binary contrasts
|
||||
- "X. That's it. That's the [thing]." fragments
|
||||
- "Think about it:" setups
|
||||
- Exactly three items in every list
|
||||
- Every paragraph ending with punch
|
||||
|
||||
---
|
||||
|
||||
## Scoring Guide
|
||||
|
||||
| Total | Assessment |
|
||||
|-------|------------|
|
||||
| 40 | Perfect—indistinguishable from skilled human writer |
|
||||
| 36-39 | Excellent—occasional minor tells |
|
||||
| 32-35 | Passing—clearly human, some AI influence |
|
||||
| 28-31 | Needs work—AI patterns visible |
|
||||
| 24-27 | Significant issues—AI-forward |
|
||||
| <24 | Fail—reads as AI-generated |
|
||||
|
||||
**Minimum passing score: 32/40 (80%)**
|
||||
|
||||
---
|
||||
|
||||
## Quick Scoring Checklist
|
||||
|
||||
```
|
||||
□ Directness: Opens with point? No hedging? [ /5]
|
||||
□ Rhythm: Sentence lengths vary 8-25 words? [ /5]
|
||||
□ Verbs: Concrete, active, specific? [ /5]
|
||||
□ Trust: No over-explaining? Respects reader? [ /5]
|
||||
□ Authenticity: Sounds like a person? [ /5]
|
||||
□ Density: Substance over filler? [ /5]
|
||||
□ Facts: All facts preserved exactly? [ /5]
|
||||
□ Structure: No AI templates? [ /5]
|
||||
TOTAL: [ /40]
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user