global setup and configuration.
This commit is contained in:
@@ -0,0 +1,194 @@
|
||||
# Teach & Mimic — internals
|
||||
|
||||
The routed flows live in `references/commands/teach.md`,
|
||||
`references/commands/mimic.md`, and the refine section of that mimic command
|
||||
file. This file holds the deep material those flows link to: card anatomy,
|
||||
sample requirements, scoring, baselines, statistics, and failure modes.
|
||||
|
||||
Reconstruction runs under detection's constitution: a mimic or refined output
|
||||
that reintroduces slop is a failure. Every artifact here is deterministic and
|
||||
testable; the model is used only to generate prose, never to score itself.
|
||||
|
||||
## Teach — the artifacts
|
||||
|
||||
`teach` distills a writer's samples into a reusable voice, stored under
|
||||
`.unslop/voice/<name>/` (gitignored by default):
|
||||
|
||||
1. `scripts/voice_profile.py samples/ -o profile.json` — the machine
|
||||
fingerprint (char 3-grams, function-word delta, sentence-length EMD,
|
||||
punctuation, contractions, MTLD, word-length, impostor z-scores, GI rank).
|
||||
This is the referee the scorer uses.
|
||||
2. `scripts/voice_card.py --profile profile.json --samples samples/ --out .
|
||||
--name <name>` — the **layered voice card** the generating model actually
|
||||
follows. `--name` labels the card (`# Voice card: <name>`); default `voice`.
|
||||
voice_card recomputes the profile from `--samples` and refuses (exit 2, named
|
||||
field) if the supplied `--profile` does not describe those samples, so a stale
|
||||
profile can never silently drive a card.
|
||||
3. Add `--provenance` to write `provenance.json` (per-sample sha256, word
|
||||
counts, doc count, genre note, low-confidence flag) so a teach run is
|
||||
auditable.
|
||||
|
||||
**Sample files.** Both scripts read only `.txt` and `.md` files, recursively,
|
||||
under the samples directory (nothing else — a directory of `.docx` or extensionless
|
||||
files reads as empty and both scripts exit 2 with a diagnostic naming the
|
||||
requirement). Convert samples to `.txt`/`.md` first.
|
||||
|
||||
**Sample requirements.** Use at least 5 documents, 2-3k words total, in the
|
||||
**same genre** as the target. Fewer than ~2000 words or cross-genre samples set
|
||||
`low_confidence` in the profile and provenance; the card still builds but its
|
||||
claims are noisier. **Surface `low_confidence` to the user** — tell them the voice
|
||||
is provisional and ask for more same-genre samples before trusting a mimic.
|
||||
Cross-domain style attribution degrades sharply, so do not teach on blog posts
|
||||
and score legal memos.
|
||||
|
||||
## The voice card (layered)
|
||||
|
||||
The card is a directory, not one file, so the generator loads only what the
|
||||
current writing task needs:
|
||||
|
||||
- `card.md` — the always-loaded core, kept under 300 words: rhythm (median
|
||||
sentence length, IQR, burstiness), contraction habits with real examples,
|
||||
punctuation the writer uses and avoids, top sentence openers, a **Never**
|
||||
list of at/near-zero features, and an **index table** — "when writing X, read
|
||||
`card/X.md`".
|
||||
- `card/<situation>.md` — one sheet per **covered** situation from the taxonomy
|
||||
(explaining-technical, anecdote, argument, disagreement, praise,
|
||||
hedging-uncertainty, numbers-data, addressing-reader, openings, closings).
|
||||
Each sheet gives how this author handles that situation, 1-3 verbatim sample
|
||||
snippets, and measured markers.
|
||||
|
||||
**No fabrication.** A dimension with no sample evidence gets **no sheet**; it is
|
||||
named under "Uncovered" in `card.md`. The card never invents a voice the
|
||||
samples cannot support.
|
||||
|
||||
**Coverage classification.** `voice_card.py --coverage` emits a deterministic
|
||||
lexical coverage matrix over the taxonomy. It is intentionally coarse: it
|
||||
DRIVES which sheets get written and which teach prompts to ask the user, and a
|
||||
misclassified sentence can only add or drop a sheet — it can never change a card
|
||||
claim, because claims come from measured facts and verbatim snippets. So
|
||||
misclassification is low-stakes by construction. A sample set missing numeric
|
||||
writing, for instance, leaves `numbers-data` uncovered and named as a gap.
|
||||
|
||||
**Coverage → prompt procedure.** When a dimension the target writing will need is
|
||||
uncovered, ask the user for a short sample that exercises it. **Only ask for
|
||||
dimensions the task requires** — do not fish for all ten. Templates:
|
||||
|
||||
| Dimension | Prompt to the user |
|
||||
|-----------|--------------------|
|
||||
| explaining-technical | "Share something you wrote that explains how a thing works or why it behaves the way it does." |
|
||||
| anecdote | "Share a few sentences where you tell a small story about something that happened to you." |
|
||||
| argument | "Share something you wrote that argues a position — a claim you defended." |
|
||||
| disagreement | "Share something where you pushed back on or disagreed with an idea." |
|
||||
| praise | "Share something where you praised or recommended something you liked." |
|
||||
| hedging-uncertainty | "Share something where you were unsure and said so — thinking out loud." |
|
||||
| numbers-data | "Share something you wrote that works with numbers, quantities, or measurements." |
|
||||
| addressing-reader | "Share something written directly to a reader — instructions or a note to someone." |
|
||||
|
||||
`openings` and `closings` are structural (first/last sentence of every document)
|
||||
and are covered as soon as there is one document, so they need no prompt.
|
||||
|
||||
The card is pack-sized (~1-2k tokens loaded) so it rides in the generation
|
||||
context cheaply. Same inputs give byte-identical files.
|
||||
|
||||
## `--refine` — the internal hill-climb
|
||||
|
||||
`evals/run_mimic_refine.py` iterates when one pass is not enough. It is worth
|
||||
the cost only when a single mimic keeps landing short of the target voice; most
|
||||
users teach once and mimic cheaply thereafter.
|
||||
|
||||
Protocol (climb on A, accept on DEV, report on a sealed split):
|
||||
|
||||
- Split samples by document (seeded): **A** (~60%, retrieval/context pool) and
|
||||
**DEV** (~40%, acceptance). Fewer than 5 docs → refuse (exit 2).
|
||||
- **LIVE (default):** each iteration assembles B prompts — the draft, the
|
||||
A-split voice card, the k=2 nearest-A samples by char-3gram cosine, and the
|
||||
current directives — and invokes `--generate-cmd` (default `claude -p`) once
|
||||
per beam, piping the prompt on stdin and reading the candidate on stdout.
|
||||
`--baseline zero|few|retrieval` selects which samples ride in the prompt (none
|
||||
/ first-k / k nearest-A) and generates its candidate set through this same
|
||||
path, so the honest retrieval-few-shot competitor and the refine loop share
|
||||
one pipeline.
|
||||
- **DRY-RUN:** pass `--candidates-dir DIR` and the loop reads
|
||||
`DIR/iter<i>/*.md` as that iteration's batch instead of generating them, so
|
||||
the loop logic runs with no model calls (this is how the MIMIC-* rows exercise
|
||||
it; the LIVE path is covered by a mock generator).
|
||||
- **Hard gates** discard a candidate regardless of score: banned-phrase clean,
|
||||
structure clean, draft→candidate preservation, no copy-gate violation against
|
||||
A, and a 150-word floor.
|
||||
- Survivors are scored against DEV (and recorded against A for the divergence
|
||||
guard) with the **full `voice_score` composite** — `0.5·(1−GI) + 0.5·` clipped
|
||||
weighted impostor-z distance against a same-genre impostor pool (`--impostors`,
|
||||
default the committed pool), seeded. This is deliberately NOT a raw weighted
|
||||
distance: a marker-stuffed candidate can minimize a raw distance and clear
|
||||
every hard gate yet be exposed by GI, so raw-distance acceptance would accept
|
||||
it over honest prose. The generator never scores itself.
|
||||
- Accept an iteration only if the DEV composite improves by ≥ `min-delta`; the
|
||||
best-so-far never regresses.
|
||||
|
||||
### Stopping and the divergence guard
|
||||
|
||||
Earliest of: iteration cap, patience (N iterations without a DEV gain), or
|
||||
**divergence** — the A composite improving while DEV worsens for two
|
||||
consecutive iterations. Divergence is the reward-hacking signature (the model
|
||||
is fitting the retrieval pool's topics, not the author's style); the loop halts
|
||||
immediately and sets `reward_hacking_warning`.
|
||||
|
||||
### Directives and card refinement
|
||||
|
||||
Each iteration derives up to four ranked directives from the DEV metric deltas,
|
||||
and each directive carries a proposed **card amendment** line. The best
|
||||
candidate is copied to `OUT/final.md` and the amended card to
|
||||
`OUT/voice-card.refined.md`.
|
||||
|
||||
## Scoring
|
||||
|
||||
The `voice_score` **composite** the refine loop accepts on is
|
||||
`0.5·(1−GI) + 0.5·zsum` (lower = more author-like), where `GI` is the General
|
||||
Impostors rank — the fraction of random feature-subset trials in which the
|
||||
candidate beats every sampled impostor at looking like the profile — and `zsum`
|
||||
is the clipped, weighted **impostor-z** distance (each per-feature distance
|
||||
normalized by the impostor pool's mean/std, clipped to ±3, weighted by the WP10a
|
||||
research weights). Both halves are calibrated against a same-genre **impostor
|
||||
pool**, so the number rewards *beating plausible other authors*, not merely
|
||||
matching aggregate surface stats. That is what defeats a marker-stuffed
|
||||
candidate: it can drive a raw weighted distance down and still be far worse under
|
||||
GI. Meaning preservation vs the original draft is a **separate** hard gate, never
|
||||
blended into the composite. The composite is a **guide, not an oracle** — even
|
||||
strong authorship verification tops out near a 93-94% ceiling, so treat
|
||||
per-candidate deltas as the signal and under-claim.
|
||||
|
||||
## Baselines
|
||||
|
||||
Score refine output against paired baselines on the same drafts: zero-shot
|
||||
("write like this person"), static few-shot, and **retrieval few-shot** (k
|
||||
nearest-A samples — the honest competitor). Refine has to beat retrieval
|
||||
few-shot to earn its cost.
|
||||
|
||||
## Statistics
|
||||
|
||||
`evals/mimic_stats.py` runs the small-n discipline: per-item paired deltas, a
|
||||
BCa bootstrap CI (n=2000, seeded), and a sign-flip permutation test (exact for
|
||||
≤12 items, else 20k seeded). Claim a win only when the CI lower bound > 0 **and**
|
||||
p < 0.05. Anything degenerate (too few items, zero variance) reports
|
||||
not-improved.
|
||||
|
||||
## Cost
|
||||
|
||||
| Step | Calls |
|
||||
|------|-------|
|
||||
| teach (profile + card) | free / deterministic |
|
||||
| mimic | one strong generation call |
|
||||
| refine | iterations × beam strong calls, plus free gate/score passes |
|
||||
|
||||
## Failure modes and the gate that catches each
|
||||
|
||||
| Failure | Symptom | Caught by |
|
||||
|---------|---------|-----------|
|
||||
| Topic bleed | importing what samples are *about*, not *how* they read | divergence guard (A improves, DEV worsens) |
|
||||
| Verbatim copying | lifting sample phrasing | copy-gate (4-gram / LCS vs A) |
|
||||
| Register lock | latching one sample's register | DEV composite vs held-out split |
|
||||
| Reward hacking | self-assessed quality rising while DEV falls | divergence guard + `reward_hacking_warning` |
|
||||
| Slop reintroduced | AI tells creep back into voiced prose | banned-phrase / structure gates |
|
||||
|
||||
For chunking long documents and the tiered execution architecture around these
|
||||
steps, see `pipeline.md`.
|
||||
Reference in New Issue
Block a user