195 lines
10 KiB
Markdown
195 lines
10 KiB
Markdown
# Teach & Mimic — internals
|
||
|
||
The routed flows live in `references/commands/teach.md`,
|
||
`references/commands/mimic.md`, and the refine section of that mimic command
|
||
file. This file holds the deep material those flows link to: card anatomy,
|
||
sample requirements, scoring, baselines, statistics, and failure modes.
|
||
|
||
Reconstruction runs under detection's constitution: a mimic or refined output
|
||
that reintroduces slop is a failure. Every artifact here is deterministic and
|
||
testable; the model is used only to generate prose, never to score itself.
|
||
|
||
## Teach — the artifacts
|
||
|
||
`teach` distills a writer's samples into a reusable voice, stored under
|
||
`.unslop/voice/<name>/` (gitignored by default):
|
||
|
||
1. `scripts/voice_profile.py samples/ -o profile.json` — the machine
|
||
fingerprint (char 3-grams, function-word delta, sentence-length EMD,
|
||
punctuation, contractions, MTLD, word-length, impostor z-scores, GI rank).
|
||
This is the referee the scorer uses.
|
||
2. `scripts/voice_card.py --profile profile.json --samples samples/ --out .
|
||
--name <name>` — the **layered voice card** the generating model actually
|
||
follows. `--name` labels the card (`# Voice card: <name>`); default `voice`.
|
||
voice_card recomputes the profile from `--samples` and refuses (exit 2, named
|
||
field) if the supplied `--profile` does not describe those samples, so a stale
|
||
profile can never silently drive a card.
|
||
3. Add `--provenance` to write `provenance.json` (per-sample sha256, word
|
||
counts, doc count, genre note, low-confidence flag) so a teach run is
|
||
auditable.
|
||
|
||
**Sample files.** Both scripts read only `.txt` and `.md` files, recursively,
|
||
under the samples directory (nothing else — a directory of `.docx` or extensionless
|
||
files reads as empty and both scripts exit 2 with a diagnostic naming the
|
||
requirement). Convert samples to `.txt`/`.md` first.
|
||
|
||
**Sample requirements.** Use at least 5 documents, 2-3k words total, in the
|
||
**same genre** as the target. Fewer than ~2000 words or cross-genre samples set
|
||
`low_confidence` in the profile and provenance; the card still builds but its
|
||
claims are noisier. **Surface `low_confidence` to the user** — tell them the voice
|
||
is provisional and ask for more same-genre samples before trusting a mimic.
|
||
Cross-domain style attribution degrades sharply, so do not teach on blog posts
|
||
and score legal memos.
|
||
|
||
## The voice card (layered)
|
||
|
||
The card is a directory, not one file, so the generator loads only what the
|
||
current writing task needs:
|
||
|
||
- `card.md` — the always-loaded core, kept under 300 words: rhythm (median
|
||
sentence length, IQR, burstiness), contraction habits with real examples,
|
||
punctuation the writer uses and avoids, top sentence openers, a **Never**
|
||
list of at/near-zero features, and an **index table** — "when writing X, read
|
||
`card/X.md`".
|
||
- `card/<situation>.md` — one sheet per **covered** situation from the taxonomy
|
||
(explaining-technical, anecdote, argument, disagreement, praise,
|
||
hedging-uncertainty, numbers-data, addressing-reader, openings, closings).
|
||
Each sheet gives how this author handles that situation, 1-3 verbatim sample
|
||
snippets, and measured markers.
|
||
|
||
**No fabrication.** A dimension with no sample evidence gets **no sheet**; it is
|
||
named under "Uncovered" in `card.md`. The card never invents a voice the
|
||
samples cannot support.
|
||
|
||
**Coverage classification.** `voice_card.py --coverage` emits a deterministic
|
||
lexical coverage matrix over the taxonomy. It is intentionally coarse: it
|
||
DRIVES which sheets get written and which teach prompts to ask the user, and a
|
||
misclassified sentence can only add or drop a sheet — it can never change a card
|
||
claim, because claims come from measured facts and verbatim snippets. So
|
||
misclassification is low-stakes by construction. A sample set missing numeric
|
||
writing, for instance, leaves `numbers-data` uncovered and named as a gap.
|
||
|
||
**Coverage → prompt procedure.** When a dimension the target writing will need is
|
||
uncovered, ask the user for a short sample that exercises it. **Only ask for
|
||
dimensions the task requires** — do not fish for all ten. Templates:
|
||
|
||
| Dimension | Prompt to the user |
|
||
|-----------|--------------------|
|
||
| explaining-technical | "Share something you wrote that explains how a thing works or why it behaves the way it does." |
|
||
| anecdote | "Share a few sentences where you tell a small story about something that happened to you." |
|
||
| argument | "Share something you wrote that argues a position — a claim you defended." |
|
||
| disagreement | "Share something where you pushed back on or disagreed with an idea." |
|
||
| praise | "Share something where you praised or recommended something you liked." |
|
||
| hedging-uncertainty | "Share something where you were unsure and said so — thinking out loud." |
|
||
| numbers-data | "Share something you wrote that works with numbers, quantities, or measurements." |
|
||
| addressing-reader | "Share something written directly to a reader — instructions or a note to someone." |
|
||
|
||
`openings` and `closings` are structural (first/last sentence of every document)
|
||
and are covered as soon as there is one document, so they need no prompt.
|
||
|
||
The card is pack-sized (~1-2k tokens loaded) so it rides in the generation
|
||
context cheaply. Same inputs give byte-identical files.
|
||
|
||
## `--refine` — the internal hill-climb
|
||
|
||
`evals/run_mimic_refine.py` iterates when one pass is not enough. It is worth
|
||
the cost only when a single mimic keeps landing short of the target voice; most
|
||
users teach once and mimic cheaply thereafter.
|
||
|
||
Protocol (climb on A, accept on DEV, report on a sealed split):
|
||
|
||
- Split samples by document (seeded): **A** (~60%, retrieval/context pool) and
|
||
**DEV** (~40%, acceptance). Fewer than 5 docs → refuse (exit 2).
|
||
- **LIVE (default):** each iteration assembles B prompts — the draft, the
|
||
A-split voice card, the k=2 nearest-A samples by char-3gram cosine, and the
|
||
current directives — and invokes `--generate-cmd` (default `claude -p`) once
|
||
per beam, piping the prompt on stdin and reading the candidate on stdout.
|
||
`--baseline zero|few|retrieval` selects which samples ride in the prompt (none
|
||
/ first-k / k nearest-A) and generates its candidate set through this same
|
||
path, so the honest retrieval-few-shot competitor and the refine loop share
|
||
one pipeline.
|
||
- **DRY-RUN:** pass `--candidates-dir DIR` and the loop reads
|
||
`DIR/iter<i>/*.md` as that iteration's batch instead of generating them, so
|
||
the loop logic runs with no model calls (this is how the MIMIC-* rows exercise
|
||
it; the LIVE path is covered by a mock generator).
|
||
- **Hard gates** discard a candidate regardless of score: banned-phrase clean,
|
||
structure clean, draft→candidate preservation, no copy-gate violation against
|
||
A, and a 150-word floor.
|
||
- Survivors are scored against DEV (and recorded against A for the divergence
|
||
guard) with the **full `voice_score` composite** — `0.5·(1−GI) + 0.5·` clipped
|
||
weighted impostor-z distance against a same-genre impostor pool (`--impostors`,
|
||
default the committed pool), seeded. This is deliberately NOT a raw weighted
|
||
distance: a marker-stuffed candidate can minimize a raw distance and clear
|
||
every hard gate yet be exposed by GI, so raw-distance acceptance would accept
|
||
it over honest prose. The generator never scores itself.
|
||
- Accept an iteration only if the DEV composite improves by ≥ `min-delta`; the
|
||
best-so-far never regresses.
|
||
|
||
### Stopping and the divergence guard
|
||
|
||
Earliest of: iteration cap, patience (N iterations without a DEV gain), or
|
||
**divergence** — the A composite improving while DEV worsens for two
|
||
consecutive iterations. Divergence is the reward-hacking signature (the model
|
||
is fitting the retrieval pool's topics, not the author's style); the loop halts
|
||
immediately and sets `reward_hacking_warning`.
|
||
|
||
### Directives and card refinement
|
||
|
||
Each iteration derives up to four ranked directives from the DEV metric deltas,
|
||
and each directive carries a proposed **card amendment** line. The best
|
||
candidate is copied to `OUT/final.md` and the amended card to
|
||
`OUT/voice-card.refined.md`.
|
||
|
||
## Scoring
|
||
|
||
The `voice_score` **composite** the refine loop accepts on is
|
||
`0.5·(1−GI) + 0.5·zsum` (lower = more author-like), where `GI` is the General
|
||
Impostors rank — the fraction of random feature-subset trials in which the
|
||
candidate beats every sampled impostor at looking like the profile — and `zsum`
|
||
is the clipped, weighted **impostor-z** distance (each per-feature distance
|
||
normalized by the impostor pool's mean/std, clipped to ±3, weighted by the WP10a
|
||
research weights). Both halves are calibrated against a same-genre **impostor
|
||
pool**, so the number rewards *beating plausible other authors*, not merely
|
||
matching aggregate surface stats. That is what defeats a marker-stuffed
|
||
candidate: it can drive a raw weighted distance down and still be far worse under
|
||
GI. Meaning preservation vs the original draft is a **separate** hard gate, never
|
||
blended into the composite. The composite is a **guide, not an oracle** — even
|
||
strong authorship verification tops out near a 93-94% ceiling, so treat
|
||
per-candidate deltas as the signal and under-claim.
|
||
|
||
## Baselines
|
||
|
||
Score refine output against paired baselines on the same drafts: zero-shot
|
||
("write like this person"), static few-shot, and **retrieval few-shot** (k
|
||
nearest-A samples — the honest competitor). Refine has to beat retrieval
|
||
few-shot to earn its cost.
|
||
|
||
## Statistics
|
||
|
||
`evals/mimic_stats.py` runs the small-n discipline: per-item paired deltas, a
|
||
BCa bootstrap CI (n=2000, seeded), and a sign-flip permutation test (exact for
|
||
≤12 items, else 20k seeded). Claim a win only when the CI lower bound > 0 **and**
|
||
p < 0.05. Anything degenerate (too few items, zero variance) reports
|
||
not-improved.
|
||
|
||
## Cost
|
||
|
||
| Step | Calls |
|
||
|------|-------|
|
||
| teach (profile + card) | free / deterministic |
|
||
| mimic | one strong generation call |
|
||
| refine | iterations × beam strong calls, plus free gate/score passes |
|
||
|
||
## Failure modes and the gate that catches each
|
||
|
||
| Failure | Symptom | Caught by |
|
||
|---------|---------|-----------|
|
||
| Topic bleed | importing what samples are *about*, not *how* they read | divergence guard (A improves, DEV worsens) |
|
||
| Verbatim copying | lifting sample phrasing | copy-gate (4-gram / LCS vs A) |
|
||
| Register lock | latching one sample's register | DEV composite vs held-out split |
|
||
| Reward hacking | self-assessed quality rising while DEV falls | divergence guard + `reward_hacking_warning` |
|
||
| Slop reintroduced | AI tells creep back into voiced prose | banned-phrase / structure gates |
|
||
|
||
For chunking long documents and the tiered execution architecture around these
|
||
steps, see `pipeline.md`.
|