Initial authoring system prototype

SPEC.md

@@ -0,0 +1,392 @@
+# SPEC.md — Drafting Engine Brief Specification
+
+**Version:** 0.5.3-windowed-draft
+**Terminology:** per AUTHORING.md Appendix A.
+**Status:** DRAFT — freeze before any SFT data generation
+**Rule:** Once frozen, changes require a version bump and a retrain. Argue here, not later.
+
+---
+
+## 1. Purpose
+
+Defines the exact input package the drafting engine receives and the output it returns. This document is simultaneously:
+
+- the SFT dataset format (every training pair mirrors it exactly),
+- the API contract for `POST /draft` and `draft.py`,
+- the authoring guide for whoever writes briefs.
+
+The engine honors everything inside the package and nothing outside it. Producing the package (timeline maintenance, character snapshots, tail selection, notes content) is upstream work — author + authoring system — and out of the engine's scope.
+
+v0.4 extended the format to a multi-author training corpus (Lovecraft, Galsworthy, Dreiser, Maugham, Hapa Girls). Voice is an explicit, conditioned input via the STYLE section: the engine learns to map a style reference and/or description onto prose. A STYLE may name an author, describe observable techniques, or do both; it does not carry source passages.
+
+v0.5 makes the package **bounded and closure-aware**, driven by a genre review against epic fantasy but general in effect:
+
+- **LEXICON** (new): the controlled vocabulary of the book — how invented or specialized terms are spelled, capitalized, and used. The project maintains the complete lexicon; the author selects the bounded subset sent with a package.
+- **Windowing:** STORY is limited to the five most recent scenes; CHARACTERS to those in the scene. Older material reaches the engine through RECALL, at the author's discretion, so package size stops growing with book length.
+- **RECALL generalized:** full verbatim material *or* author summaries, each typed on its source line.
+- **Closure handling:** DIRECTIVE declares how the scene should end — whether it lands, breaks in motion, closes a thread, or ends a book — because those are different prose behaviors at the last two hundred words and are invisible to the engine unless stated. Exact controlled wording remains open pending training results.
+
+## 2. Package format
+
+Plain UTF-8 text. Labeled sections, fixed order, exact delimiter strings, one blank line between sections. All delimiters begin at column 0. All sections are always present; empty sections carry an explicit `(none …)` line, never omission.
+
+```
+### STYLE
+{voice_description}
+
+### LEXICON
+{controlled_vocabulary}
+
+### STORY
+{recent_scene_summaries}
+
+### CHARACTERS
+{character_states}
+
+### RECALL
+{referenced_material}
+
+### PREVIOUS
+{previous_scene_tail}
+
+### NOTES
+{world_facts}
+
+### BRIEF
+{scene_brief}
+
+### DIRECTIVE
+{directive}
+
+### SCENE
+```
+
+Generation begins immediately after `### SCENE` and a newline. The serving layer validates section order and presence on every call and rejects malformed packages loudly (never silently reorder or repair).
+
+## 3. Section boundaries — one home per fact
+
+Every piece of information has exactly one section that owns it. When filling a package, place each item by this table and nowhere else:
+
+| Information | Owner |
+|---|---|
+| Enduring voice of the book or declared Part (POV convention, register, rhythm, dialogue density) | STYLE |
+| Surface form of invented or specialized terms: spelling, capitalization, plural, form of address | LEXICON |
+| Events of the five scenes preceding the one PREVIOUS owns | STORY |
+| The preceding scene | PREVIOUS (verbatim tail; it does not also get a STORY sentence) |
+| Everything about a character in this scene: identity, relationships, emotional/mental state, what they know or don't know | CHARACTERS |
+| Material older than the STORY window that this scene draws on — full text or summary | RECALL |
+| World facts: location, objects, time, weather, physical continuity | NOTES |
+| What happens in this scene | BRIEF |
+| Scene-local handling: closure depth, length, pacing, this scene's particular register | DIRECTIVE |
+
+Three rules make the partition unambiguous:
+
+**One scene, one section.** A given scene's events appear in exactly one of STORY, PREVIOUS, or RECALL — never two. The timeline is partitioned with no overlap: RECALL covers older material the author elects to carry forward; STORY covers the five scenes before the preceding one; PREVIOUS *is* the preceding scene; SCENE is the current one.
+
+**No chapters in the package.** The engine's units are the scene and the thread. A chapter is presentation, owned by the author (`AUTHORING.md` §3), and it carries no continuity semantics the engine could act on — so **no package field refers to one.** Not RECALL source markers, not NOTES provenance, not CHARACTERS status cites. Where a fact came from is the authoring system's bookkeeping: the world-fact envelope records its establishing scene, the Scene record records its chapter assignment, and neither is projected. What reaches the package is the information itself.
+
+This is not only tidiness. §4.10 forbids the engine from producing headers; feeding it chapter references teaches it that chapter numbering is part of the narrative frame, which is exactly the habit that prohibition exists to prevent. Books are different and may be named — a prior volume is a real boundary in the state model (`AUTHORING.md` §3.2) — but a chapter never is.
+
+**Happened versus is now true.** STORY and RECALL record that something *happened*. CHARACTERS records what is *now true because it happened*. "He realizes what hiding it had cost her" is an event and belongs to STORY; "he knows about the letter, and has forgiven her" is state and belongs to CHARACTERS. The same fact has two aspects and one home each. Character information never appears in NOTES; world information never appears in CHARACTERS; nothing enduring goes in DIRECTIVE and nothing scene-local goes in STYLE or LEXICON.
+
+Throughout this document, "preceding" means preceding **in the scene's thread** (its run of narrative continuity — AUTHORING.md §3): usually the on-page predecessor, but when a thread resumes after a pause it may be a scene several chapters back. The engine sees no difference — it only ever receives a tail; thread selection is the authoring system's projection decision.
+
+**The scene is the unit.** One invocation of the engine produces exactly one scene. A long set piece is written as several consecutive scenes on one thread, each carrying the last one's tail in PREVIOUS; it is not requested as a single long generation. Scene order is tracked by the authoring system, which realigns derived state when the author reorders.
+
+### 3.1 Precedence — DIRECTIVE commands, state informs
+
+**When DIRECTIVE presupposes something the state fields contradict, DIRECTIVE wins.** This is measured behavior, not a design preference, and packages must be built on the assumption that it holds.
+
+The evidence (`STATE_PERTURBATION_REPORT.md`, Round 1) is unusually clean. Perturb a state field the DIRECTIVE says nothing about — swap the decor of a room — and every model tested registered the change: **1.00 responsiveness, four arms, three architectures.** Perturb a state field whose behavior the DIRECTIVE explicitly names — mark a character deceased while the DIRECTIVE still calls for that character's discovery of the handkerchief and his silent protection of Irene — and every model resurrects him: **0.00 responsiveness, four arms, three architectures, three independent scoring passes, no dissent.** The models read state. They lose to DIRECTIVE when the two disagree.
+
+**Why this is structural rather than a defect to be trained away.** Reverse instruction derives every training directive backward from the scene that actually occurred, so a corpus package's DIRECTIVE can never contradict the CHARACTERS or STORY fields describing that same moment. Agreement holds in 100% of training pairs. The model therefore has no examples of state overriding directive, and the conflict case is out of distribution — which is exactly why it resolves in favor of the channel carrying an explicit instruction.
+
+The alternative — constructing SFT pairs whose response defies its own directive — would teach directive disobedience to buy state fidelity. That is a worse trade for an engine whose whole contract is to do what the package says.
+
+**So the rule is a contract, and the obligation sits upstream.** State fields inform; DIRECTIVE commands; **keeping the two consistent is the authoring system's job, not the engine's.** A package whose DIRECTIVE presupposes a dead character's action is a malformed package that no validator can detect and the engine will not refuse — it will produce fluent, plausible prose in which the character is alive. Silent, credible, and wrong is the worst failure class available, and the only place it can be caught is at projection time (`AUTHORING.md` §7.1).
+
+## 4. Field definitions
+
+### 4.1 STYLE — required
+
+- *The enduring voice of the book*, or an explicitly declared alternate voice for a Part. It may name an author, describe the desired techniques, or do both. The description should cover POV convention, register, sentence rhythm, dialogue density, and any signature handling (e.g., `Close third person, one viewpoint per scene. Warm domestic realism; humor lives in the dialogue. Short paragraphs, dialogue-heavy, contemporary American diction.`).
+- Written once per book and **reused verbatim** across every package from that book. An alternate STYLE is an exception, declared at a Part boundary and reused across the scenes in that Part. Consistency of a given STYLE is what makes the conditioning learnable.
+- Never scene-local: pacing or tonal instructions for this one scene belong in DIRECTIVE.
+
+### 4.2 LEXICON — required (may be empty)
+
+- *The controlled vocabulary in force for this scene*: how invented or specialized terms are written. Surface form only — **not** what the term means, which is a world fact (NOTES) or a character fact (CHARACTERS).
+- One term per line: the term, an em dash, and its usage note. Cover whatever is not self-evident — plural form, capitalization, article, possessive, form of address, and any homonym or near-miss to avoid.
+  - `Aes Sedai — same form singular and plural; both words capitalized; never possessive.`
+  - `Tar Valon — the city; takes no article.`
+  - `Warder — capitalized when it denotes the bond, lowercase in ordinary use.`
+- **Project versus package scope:** the authoring project maintains a complete book lexicon. The package carries the author-selected subset relevant to the forthcoming scene. The system may suggest entries found in BRIEF or established continuity, but it must not select them silently.
+- The selected subset carries forward as the default for succeeding scenes until the author alters it. The project-level scope of this carry-forward and the master-list threshold that makes subset selection mandatory remain open decisions (§9).
+- **Package cap: open.** The prior 25-line cap is retained as a working target, not a frozen rule. If nothing applies: `(none)`.
+- A new term found in an accepted scene may be proposed for addition to the complete lexicon. Text from rejected or uncommitted drafts never creates a lexicon entry.
+- Spelling conformance is additionally checked downstream (AUTHORING.md §7). Conditioning reduces the error rate; the check catches the remainder.
+
+### 4.3 STORY — required (may be marked as new thread)
+
+- *What happened in the five scenes preceding the one PREVIOUS owns*, told the way the books are written: in scenes. Strictly event summary — no verbatim text, no constraints, no character description.
+- **Window: up to five scenes**, in story order, oldest first, ending just before the preceding scene, which PREVIOUS owns. Fewer when the thread is younger than five scenes. Anything older reaches the engine only through RECALL, and only because the author put it there.
+- **One sentence per scene, and the sentence is dense.** House form: a single sentence carrying the scene's beats in causal order, semicolon-delimited, typically **25–45 words**. The pattern:
+
+  > `Ella reveals she went off birth control without telling her husband; he is initially angry; he realizes what hiding that had cost her; he wanted a baby anyway; and he forgave her.`
+
+  A terse single-beat summary ("Ella and her husband argue in the kitchen") is *not* conformant. Density is what makes the section useful, and a corpus built to the thinner reading would diverge from what the author writes at inference — a mismatch no validator catches.
+- Scene designators are stripped; only the sentences remain.
+- Written upstream and maintained as a rolling record; the engine never updates it.
+- If the scene opens a new thread, unrelated to the scenes that came before it (including the opening of the first book), the literal line: `(none - new storyline)`.
+- The author chooses the thread for a scene. The authoring system defaults that choice to the thread of the preceding canonical scene, but the author may resume any existing thread or create a new one. Summaries are derived from accepted scenes and may be edited by the author; an author-edited approved summary is the one used in future packages.
+
+### 4.4 CHARACTERS — required
+
+- *The state of each character present or referenced in this scene, immediately before the scene begins.* This section owns **all** character information: identity, relationships, emotional and mental state, and knowledge state (what they know or don't yet know).
+- **Scope: this scene only.** Characters not in the scene and not referenced by it get no entry, regardless of their importance elsewhere. Characters referenced but absent do get an entry — the engine needs to know who they are to write the mention correctly.
+- **Paragraph form (50–150 words)** on a character's **first appearance in the story**, and again on **re-introduction**: any return after an absence of ten or more scenes in this thread. Without the re-introduction rule, a character returning late in a book arrives with one sentence of state and no other information anywhere in the package, because their defining paragraph fell out of the window long ago. The introduction and re-introduction thresholds remain open for revision before freeze (§9).
+- **Otherwise:** one sentence of current state (e.g., `Emma Hayes — still does not know about the letter; cheerful, packing for the trip.`).
+- No knowledge from the character's future; nothing the story has not yet established.
+- Label each entry with the character's name on its own line.
+- **Status entries are required for characters who are dead or otherwise permanently removed** but still referenced: `James Forsyte — deceased; present only in memory.` Promoted from a v0.6 candidate on the strength of the confirmed character-resurrection defect (`SFT_V2_TRAINING_REPORT.md` §5, finding 2).
+  **The entry alone is not sufficient.** Round 1 perturbation marked a character deceased in exactly this form and every model tested resurrected him, because the DIRECTIVE still called for his actions (§3.1). The status line and the projection-time conflict check (`AUTHORING.md` §7.1) work only as a pair; either one without the other buys very little.
+- If no entry applies (rare): `(none)`.
+- The author selects the characters for a scene; the preceding scene's selected roster is the default. The system may propose current-state text from committed canon, but the author may edit it. A character may be planned before generation, introduced in BRIEF, or proposed from an accepted scene; no character introduced only in a rejected or uncommitted draft enters the register.
+
+### 4.5 RECALL — required (may be empty)
+
+- *Material older than the STORY window that this scene draws on.* This is the overflow channel for a bounded package: when the author needs the engine to have something the window no longer covers, it goes here.
+- Two admissible forms, chosen per entry by the author:
+  - **Full text** — the verbatim text of an earlier committed scene, from this book or a previous one. Prose to draw on: phrasing, sensory detail, rhythm are all available for reuse.
+  - **Summary** — the author's compressed account of earlier material. Fact only; not prose to echo.
+- **Each entry opens with a typed source line**, and the type is load-bearing: without it the engine cannot tell prose-to-draw-on from fact-only, and summary prose becomes echo bait.
+  - The **type** is required and is the last element: `— full text)` or `— summary)`.
+  - An optional preceding phrase may place the material, in terms the engine's model of the work contains — earlier in this book, or a named or numbered prior volume. **Never a chapter** (§3).
+  - `(earlier in this book — full text)`
+  - `(Book 1 — summary)`
+  - `(full text)` — bare type, when placement adds nothing
+- Reference material, never instruction. The brief may point to it ("she remembers the night in RECALL") without carrying the material itself.
+- A scene carried in RECALL does not also appear in STORY (§3, one scene one section).
+- Otherwise: `(none)`.
+- **RECALL is the primary large-context section.** Full recalled scenes can dominate a package, while author-flexible BRIEF, PREVIOUS, and exact-text NOTES can also vary in size. The total package budget (§7 rule 11) remains the governing limit.
+- **Cross-volume access:** RECALL is the sole channel through which earlier books reach the engine. Prior volumes are read-only archives of committed scenes (AUTHORING.md §8); no other state crosses a book boundary.
+- The author selects recalled material from any thread in the current book or from an earlier book in the series. The system may offer candidates and may generate an editable summary on request, but it does not insert RECALL silently.
+
+### 4.6 PREVIOUS — required (may be empty)
+
+- Verbatim tail of the **preceding scene in this thread, only**: by default, roughly **300–500 words**, ending on a paragraph boundary exactly where the preceding scene ended. The author may select a shorter or longer complete-paragraph tail. No summarization, no editing.
+- Carries flow: closing rhythm, emotional temperature, time and place, who is present. This section *owns* the preceding scene — it is not summarized in STORY.
+- When the preceding scene broke off mid-action, the tail carries that unresolved motion; the engine reads the state of play from the prose. Whether the *new* scene resolves anything is DIRECTIVE's business (§4.9), not PREVIOUS's.
+- When STORY is `(none - new storyline)`, PREVIOUS is `(none)`.
+
+### 4.7 NOTES — required (may be sparse)
+
+- Reminders of **significant world facts needed for this scene**: location, objects, time, weather, physical continuity. Standalone facts, one per line, **stated as facts rather than as citations** — where a fact was established lives in the ledger's envelope (`AUTHORING.md` §3) and is not projected.
+- Character information does not belong here — that is CHARACTERS' job, including knowledge state. Term spelling does not belong here — that is LEXICON's job.
+- Line types, by example:
+  - Location fact: `The kitchen has been repainted yellow — keep it so.`
+  - Time/weather: `It has been raining since yesterday.`
+  - Standing physical fact: `The Hayes house has no air conditioning.`
+  - A fact carried from an earlier volume reads no differently: it is stated, not cited.
+- Exact text may be supplied when it must appear word-for-word, such as a quotation, Bible verse, inscription, letter wording, prayer, or motto. Mark it explicitly as exact text; the authoring system verifies the required verbatim text after generation. The exact markup remains open before freeze (§9).
+- **Maximum 15 lines** is a working target pending review of exact-text markup and package budget. If nothing applies: `(none)`.
+- Normally `(none)` when STORY is `(none - new storyline)`.
+- NOT for plot instructions — those belong in BRIEF. Notes constrain; the brief directs.
+
+### 4.8 BRIEF — required
+
+- Author-authored and length-flexible. The first sentence normally declares the **narrative stance**: either a single named viewpoint character (e.g., `This scene is told in third person from the viewpoint of Lizzie Hayes.`), or an omniscient/authorial stance (e.g., `Told omnisciently, attention moving among the three friends and the social world around them.`). The stance must be consistent with the POV convention declared in STYLE.
+- Describes what happens in the scene: setting, participants, movement of events, the emotional arc, and where the scene ends.
+- House style: written as flowing description, not a bullet list; present tense preferred.
+- May specify beats that must occur and their order, including a required closing beat.
+- **Purely directive.** All context material lives in the sections above; a brief never restates context, though it may point to it.
+- The authoring system may ask for confirmation before drafting when a BRIEF is unusually short or long, but it imposes no length limit. Training-pair BRIEF limits are a separate corpus-design decision.
+
+### 4.9 DIRECTIVE — required
+
+The stable requirement is semantic: DIRECTIVE states a maximum word count and tells the engine how the scene should end. It may add scene-local handling for pacing, register, dialogue density, or scope. Its final controlled wording and validator rule remain open pending training results (§9).
+
+**Terminal depth (current candidate, not frozen).** Every invocation produces one scene, so every scene ends. What the marker declares is what *else* ends with it — and the four cases are different prose behaviors over the closing two hundred words, invisible to the engine unless stated. The following five opening strings are the current candidate set:
+
+| String | Meaning |
+|---|---|
+| `Break off mid-action; the story continues directly into the next scene.` | No cadence, no summarizing beat, no landing. The next scene on this thread picks up in motion. |
+| `Bring the scene to a close; the thread continues.` | Standard scene-closing landing. The thread resumes later, after a jump in time or place. |
+| `Bring the scene to a close, and the thread with it.` | Stronger closure: the thread's arc resolves and never resumes. The last lines carry valedictory weight for that storyline. |
+| `Bring the scene to a close, and the thread and the book with it.` | Strongest closure. |
+| `Bring the scene to a close and end the book; the thread remains open.` | Volume ending on an unresolved thread — the cliffhanger convention. |
+
+*(Decision to revisit: a terse tag such as `Ends: thread` is an alternative. The candidate sentences remain useful because they validate by literal match while remaining in the register the base model responds to; their final form will be chosen through training and evaluation.)*
+
+**Handling.** After any terminal sentence, add scene-local handling: pacing, and this scene's particular register where it departs from or sharpens STYLE. Length is stated as a maximum in words. Alternative wording such as `Do not extend past the BRIEF` remains admissible while the candidate set is under evaluation. NOTE: the model learns length as a distribution from training data, not by counting — keep training-pair lengths consistent with their directives, and expect ±15% in practice.
+
+**DIRECTIVE is the highest-authority field in the package (§3.1).** Anything it names, the engine will attempt regardless of what CHARACTERS, STORY, or NOTES say — measured at 0.00 state responsiveness under conflict, across four checkpoints and three architectures. Two consequences for whoever writes one:
+
+- **Never name a specific character action in DIRECTIVE.** Actions belong in BRIEF. A directive reading *let his discovery of the handkerchief carry real moral weight* silently overrides a CHARACTERS entry marking him dead; the same instruction phrased as manner — *let the discovery carry real moral weight* — does not, because it presupposes no actor.
+- **Keep DIRECTIVE to manner, pacing, register, and closure.** The narrower its scope, the fewer facts it can contradict. Its final length guidance remains open, but it should stay narrowly scene-local.
+
+Examples:
+
+- `Bring the scene to a close; the thread continues. Expand to a maximum of 1500 words. Keep it fun and full of dialog.`
+- `Break off mid-action; the story continues directly into the next scene. Expand to a maximum of 1200 words. The argument is still rising when the scene stops.`
+- `Bring the scene to a close, and the thread with it. Expand to a maximum of 1000 words. Quiet; let the tension stay under the surface.`
+
+### 4.10 SCENE — output
+
+- The engine writes **one scene** and stops. The stop token marks the end of the scene — which is the end of the requested span, not necessarily a break in the reader's sense. How much closure precedes the stop is governed by DIRECTIVE.
+- The engine must not: continue past the scene it was asked for, append commentary or summary, echo the package, or produce headers.
+- Target length per DIRECTIVE. Prose only.
+
+## 5. Sampler parameters (accompany the package; not part of the text)
+
+Passed as API/CLI arguments, logged with every call:
+
+| Param | Range | Default | Meaning |
+|---|---|---|---|
+| `sampler` | `entropy` \| `plain` | `entropy` | plain = standard sampling baseline, always available |
+| `alpha` | −1.0 … +1.0 | `+0.2` | −1 conventional ↔ +1 entropy-chasing |
+| `wave_period` | 0 or tokens | `0` | 0 = constant α; else sinusoidal modulation period |
+| `n_candidates` | 5–40 | `10` | lookahead candidates per token (linear cost) |
+| `min_p` | 0.0–0.2 | `0.05` | pre-filter before entropy weighting |
+| `max_tokens` | — | `2200` | hard generation cap above directive target |
+
+Per-scene sampler settings are a *directing instruction* chosen by the author alongside the brief; they are not encoded in the package text in v0.5. (Candidate feature: an optional `### SAMPLER` section.)
+
+**Repetition control (binding on any serving implementation):** soft, multiplicative penalties only — e.g., llama.cpp `repeat_penalty` ≈ 1.05–1.15 with `repeat_last_n` ≈ 64. **Never hard n-gram bans** (`no_repeat_ngram_size` or equivalent): a hard ban forces misspellings of any proper noun that recurs more than a few times in a scene, corrupting names while leaving prose apparently fluent. This was diagnosed and confirmed empirically (SFT_V2_TRAINING_REPORT.md §4–5); the artifact was severe enough to mask real quality differences between checkpoints. Temperature/top_p sampling alone prevents degenerate repetition; the hard ban is both unnecessary and harmful. The hazard compounds in any book with a dense field of invented proper nouns.
+
+## 6. Worked example — REPLACE WITH REAL EXAMPLES BEFORE FREEZE
+
+> Authoring task: build 2–3 complete packages from real scenes and paste
+> them here before freeze — at least one with a populated RECALL and one
+> with a populated LEXICON. The skeleton below shows shape only.
+
+```
+### STYLE
+Close third person, one viewpoint per scene. Warm domestic realism; humor
+lives in the dialogue. Short paragraphs, dialogue-heavy, contemporary
+American diction.
+
+### LEXICON
+(none)
+
+### STORY
+[Up to five dense sentences, oldest first, no scene designators,
+stopping before the preceding scene:]
+Ella finds the letter in the mailbox; she reads it twice; she hides it in
+the flour tin; and she says nothing at dinner.
+Andy books the flight to Honolulu over Elikapeka's objections; ...
+...
+
+### CHARACTERS
+Ella Hayes
+[First appearance or re-introduction: a paragraph — identity,
+relationships, and her state just before the scene begins, including what
+she knows. 50–150 words.]
+
+Emma Hayes
+Still does not know about the letter; cheerful, packing for the trip.
+
+### RECALL
+(Book 1 — summary)
+[The author's compressed account of material older than the window.]
+
+### PREVIOUS
+The screen door banged twice behind her, and Ella let it... [300–500 words
+of the actual tail of the preceding scene, ending at its true final
+sentence.]
+
+### NOTES
+It has been raining since yesterday.
+The kitchen has been repainted yellow — keep it so.
+
+### BRIEF
+From Ella's viewpoint. She comes back into the kitchen intending to
+apologize and instead... [author-authored description of the scene's
+movement and its closing beat; no authoring-time length limit.]
+
+### DIRECTIVE
+Bring the scene to a close; the thread continues. Expand to a maximum of
+1200 words. Keep it warm and quick; the apology never quite lands until
+the last exchange.
+
+### SCENE
+```
+
+## 7. Validation and advisory rules
+
+1. All nine input sections present, in order, delimiters exact.
+2. STYLE 2–4 lines; author names are permitted.
+3. LEXICON: one term per line in `term — usage` form, or `(none)`. The 25-line cap is advisory pending freeze.
+4. STORY: at most five scene sentences, in order, no scene designators; or the literal `(none - new storyline)` line.
+5. CHARACTERS entries labeled by name: a paragraph on first appearance or re-introduction, one sentence thereafter, or `(none)`. Exact word and absence thresholds are advisory pending freeze.
+6. RECALL: every entry opens with a typed source line ending in `— full text)` or `— summary)`; or `(none)`.
+6a. **No chapter references in any field** (§3): reject a package whose RECALL source lines, NOTES, or CHARACTERS entries cite a chapter. Book references are permitted.
+7. PREVIOUS: verbatim, paragraph-boundary tail of the preceding scene in the thread, or `(none)`. Must be `(none)` when STORY is `(none - new storyline)`; its default 300–500-word target is advisory.
+8. NOTES: no character-state lines or term-spelling lines. The 15-line cap and exact-text markup are advisory pending freeze.
+9. BRIEF: present and author-authored; no hard length constraint.
+10. DIRECTIVE contains a maximum word count and closure instruction. Exact controlled wording is advisory pending freeze.
+11. Total package ≤ 16,000 tokens (working limit pending §9). RECALL is normally the largest contributor, but the total applies to every section together.
+12. Reject a failure of any binding rule with a specific error naming that rule; never repair silently. Report an advisory-rule failure clearly but do not block the author.
+
+**Validation checks form, not fit.** A package can pass every rule above and still be a bad package — a brief referencing a character with no entry, a directive that contradicts how PREVIOUS ends, a brief turning on an event outside the window. Those are the failures that actually produce bad drafts, and they are the authoring system's advisory diagnostics to catch (AUTHORING.md §7.1), not the serving layer's to reject.
+
+## 8. SFT dataset conformance
+
+Every training pair mirrors this format byte-for-byte: same delimiters, same order, same field constraints. Additionally:
+
+- **Strip chapter references when reverse-instructing.** Source novels cite their own chapters constantly, and a package builder working from that text will carry the habit into NOTES, RECALL markers, and CHARACTERS entries. Every such reference is a training example teaching the engine that chapter numbering belongs in the narrative frame — the opposite of §4.10. Audit for it; it is the kind of leak that passes every other check.
+- **Thread segmentation is a required pre-pass.** Chunks are sequential in a book but *not* sequential in a thread: multi-POV novels alternate threads chapter to chapter, so the on-page predecessor is frequently a different storyline. Building PREVIOUS from the previous chunk regardless would train the engine that PREVIOUS is routinely discontinuous with the BRIEF — active mistraining, and the dominant pattern in any multi-POV corpus. A frontier model labels each chunk with its POV/thread first; PREVIOUS and STORY then project along thread order (AUTHORING.md §5).
+- **Terminal-depth labels are a current candidate, not frozen.** Once chunks carry thread IDs, the five candidate DIRECTIVE values are derivable mechanically: last chunk of a scene, of a thread, of a book, and whether the thread was still open at the book's end. Evaluate their wording alongside alternative closure instructions before freezing the corpus.
+- **STORY density must be audited, not assumed.** The house form is a dense multi-clause sentence (§4.3). A package builder reading "one sentence per scene" naively produces terse single-beat summaries that pass every validator while diverging from what the author writes at inference. Spot-check a sample against the 25–45 word band and the causal-chain pattern.
+- **RECALL must be populated at inference-like rates.** As the overflow channel for a five-scene window it will be used constantly in real authoring; if the corpus leaves it near-empty the model meets a populated RECALL out of distribution. Derivable rule: when a chunk references an entity or event outside the window, add a summary entry for its establishing scene. This needs entity tracking layered on the thread pass. Include both entry types; set and measure a target ratio.
+- **LEXICON per book.** Extract the book's invented or distinctive terms and their usage into a complete book-level lexicon. Build each package from a selected bounded subset, `(none)` where none applies. Do not let LEXICON become a genre tell: populate it for realist books too (surnames, place names, honorifics) so the section is not entangled with secondary-world settings.
+- **STYLE conditioning:** each book gets one STYLE, written once and reused verbatim across its packages; declared Part-level alternates are the exception. A STYLE may name an author, describe techniques, or do both. The corpus must include each permitted form at inference-like rates and make descriptions distinct enough that the model can learn the mapping from STYLE to voice. **Watch for collision within a genre:** a hundred novels of one genre will produce a hundred descriptions that all read close-third, multi-POV, elevated register — and when descriptions cluster, the model learns genre-average voice with STYLE as a weak dial. Either raise the STYLE budget so descriptions can separate, or rely on tier weighting to make the target voice the attractor.
+- **Reverse-instructed corpus packages:** chunks are treated as scenes. STORY (five dense sentences, thread-ordered) and CHARACTERS (scene-scoped, with re-introduction) are generated mechanically by a frontier model during package-building — the corpus teaches the full format, not a subset. BRIEF/DIRECTIVE/NOTES are reverse-instructed from the chunk text.
+- **New threads:** `(none - new storyline)` pairs (with matching `(none)` PREVIOUS and normally-`(none)` NOTES) appear at their natural ratio in the source novels — build pairs faithfully from the books and the ratio takes care of itself.
+- **Sparse context:** target **15–20%** sparse-context pairs — new threads, `(none)` NOTES, minimal CHARACTERS — deliberately constructed from book and part openings, so empty fields remain in-distribution.
+- **Eval holdout:** one full book held out per author or series, so evaluation probes both in-voice generalization and cross-style following.
+- Response side ends at the end of the scene with the stop token — no trailing content.
+
+## 9. Open package decisions before freeze
+
+The following are deliberately unresolved. They are not serving-layer rejection rules and must be settled, versioned, and represented in the SFT corpus before freeze:
+
+1. **LEXICON selection and budget.** Set the maximum package-entry count, the master-lexicon threshold that requires author selection, and whether the rolling selected subset follows global Manuscript order or individual threads.
+2. **STORY house form.** Confirm or revise the five-scene window, single-sentence-per-scene format, and 25–45-word density target.
+3. **CHARACTERS thresholds.** Confirm or revise the 50–150-word introduction, ten-scene re-introduction trigger, and any corresponding validation.
+4. **RECALL budget.** Set the total package/token limit and any limit on the number or total size of recalled scenes.
+5. **NOTES exact-text format.** Define the exact package markup for text that must appear verbatim and revise the NOTES size cap if necessary.
+6. **DIRECTIVE syntax.** Test and choose controlled closure wording, including whether any fixed terminal-depth markers are retained. The final form must still require a maximum word count and an ending instruction.
+7. **STYLE syntax.** Define the exact allowed forms for author-name references and the package-level indication of a declared Part-level alternate style.
+
+## 10. Candidates for v0.6 (require corpus changes and a retrain — do NOT apply to v0.5 packages)
+
+Collected here per the header rule: argue in this document, version-bump, retrain. Each of these changes the training distribution and is worthless (or harmful) applied at inference against a v0.5-trained model alone.
+
+1. **DIRECTIVE as narrative transformation.** Beyond terminal depth: directives could name the intended narrative movement (increase tension, strengthen the friendship, delay the conflict), with BRIEF carrying movement + target state and DIRECTIVE carrying manner + constraints — the four-part mapping (current state / movement / target state / constraints) from the narrative-state analysis. **Weigh against §3.1:** DIRECTIVE is already the field that wins every conflict, so widening its semantic load widens the surface on which it can silently override state. Movement language that names actors is the specific hazard.
+2. **Field-dropout augmentation.** Deliberately include package variants with individual sections at `(none)` beyond the natural sparse ratio, so the model degrades gracefully when fields are omitted rather than becoming interface-dependent.
+3. **No package echo.** Audit corpus packages whose SCENE opens with exposition recycled from their own STORY/CHARACTERS/RECALL fields. Dense STORY sentences and RECALL summaries read more like prose than terse ones do, so windowing raises this risk rather than lowering it.
+4. **Evaluation: state-perturbation, split into two families.** Round 1 (`STATE_PERTURBATION_REPORT.md`) merged two distinct measurements and could separate neither. Future rounds run them apart:
+   - **State-reading** — perturb a field the DIRECTIVE is silent about. Round 1's only conditions of this kind were the decor swap (1.00 for every arm) and the cosmetic-adjective rung; everything else was contaminated. This is the family the checkpoint rubric should adopt.
+   - **Precedence** — construct the DIRECTIVE/state conflict deliberately and score which channel won. Round 1 measured this in four conditions without intending to, and the answer was unanimous (§3.1). Now the more interesting of the two for engine design, though not a checkpoint discriminator.
+
+   Also carried forward: fix `n_predict` generously (the `word_max × 1.6` cap truncated before scene climaxes on all three architectures), and **pre-register a prediction before generating**. Round 1's pre-registration is what caught the confound; without it a shared floor would have read as a legitimate tie.
+5. **Story-time and travel.** Scenes carry no in-world date or elapsed time, so parallel threads in different places cannot be checked against each other and travel duration cannot be checked at all — a continuity class this system otherwise exists to catch mechanically. Would need a scene-level story-time field and a matching invariant; it is a state-model change (AUTHORING.md §3, §7) before it is a package change.
+6. **Rules distinguished from facts.** A magic system, a legal code, or any hard constraint is an always-binding invariant, never superseded and never filtered out; the world-fact ledger holds situational, supersedable, per-scene-selected facts. NOTES cannot carry both. Would need a Rules record and a distinct always-on projection.
+7. **Cross-thread knowledge.** STORY projects the scene's own thread, so what the reader knows from another thread — dramatic irony, converging events — reaches the package only when the author remembers to put it there. Candidate-surfacing could propose it; the format may or may not need a home for it.
+
+## 11. Change log
+
+| Version | Date | Change |
+|---|---|---|
+| 0.1-pilot | 2026-07-12 | Initial skeleton drafted (short-story / Lovecraft pilot) |
+| 0.2-novel | 2026-07-15 | Novel/series format: added STORY, CHARACTERS, RECALL; PREVIOUS three-state marker; NOTES raised to 15 lines; BRIEF purely directive; budget 4k → 16k tokens |
+| 0.3-novel | 2026-07-15 | Trimmed after 4-scene pilot run: STORY = one-sentence scene summaries (no numbers) + paragraph per previous book, `(none - new storyline)`; CHARACTERS = paragraph then one sentence, state before scene; RECALL = full referenced scene only; PREVIOUS = preceding scene only; thread-shift markers removed |
+| 0.4-multi-author | 2026-07-18 | Multi-author corpus: added required STYLE section (description-based voice conditioning, no author names); section-boundary table — one home per fact; conformance rules for reverse-instructed corpus packages, per-author eval holdouts, deliberate sparse-context construction |
+| 0.4.1-multi-author | 2026-07-21 | Consistency pass against AUTHORING.md: "preceding" defined by thread order, not page order; terminology bound to AUTHORING.md Appendix A |
+| 0.4.2-multi-author | 2026-07-24 | BRIEF stance relaxed (named-viewpoint **or** omniscient/authorial). Binding sampler rule: soft repetition penalties only, never hard n-gram bans. New §9: v0.5 candidates requiring retrain |
+| 0.5.0-windowed | 2026-07-28 | Genre review (epic fantasy) drove five changes, all general in effect. **LEXICON** section added (thread-active controlled vocabulary, always-on, ≤25 lines). **STORY windowed** to five scenes, with the dense multi-clause house sentence made explicit (25–45 words, causal beats) and previous-book paragraphs demoted to authored prose. **CHARACTERS scoped** to the scene, with a re-introduction rule after ten scenes' absence. **RECALL generalized** to full text *or* author summary with typed source lines, established as the overflow channel and the only variable-size section. **DIRECTIVE gains terminal depth**: five sanctioned opening sentences declaring what ends with the scene; §4.10 rewritten so the stop token means end-of-scene rather than end-of-narrative-unit. Partition rules stated explicitly (one scene one section; happened vs. is now true). Scene defined as the unit of one invocation. §8 requires a thread-segmentation pre-pass, RECALL population at inference-like rates, STORY density audit, and warns on intra-genre STYLE collision. §9 renumbered to v0.6; adopted items removed, four new candidates added from the genre review |
+| 0.5.2-windowed | 2026-08-02 | **Chapters removed from the package.** The engine's units are the scene and the thread; a chapter is presentation owned by the author (`AUTHORING.md` §3) and carries nothing the engine can act on, so no field refers to one. New rule in §3 with the reason stated — §4.10 forbids the engine from producing headers, and feeding it chapter references teaches the habit that prohibition exists to prevent. RECALL source markers restructured: the type suffix stays mandatory, placement becomes an optional phrase in engine-meaningful terms (`(earlier in this book — full text)`, `(Book 1 — summary)`, bare `(full text)`). NOTES state facts rather than citing where they were established — provenance lives in the world-fact envelope and is not projected. CHARACTERS status example de-cited. Validation rule 6a added; §8 gains a reverse-instruction audit, since source novels cite their own chapters constantly and a package builder will carry the habit. Books remain citable — a prior volume is a real boundary in the state model |
+| 0.5.1-windowed | 2026-08-02 | Round 1 state-perturbation results folded in (`STATE_PERTURBATION_REPORT.md`). **New §3.1 — precedence:** DIRECTIVE overrides the state fields whenever they conflict, established by measurement (1.00 responsiveness when DIRECTIVE is silent on the perturbed field, 0.00 when it names the affected behavior; four checkpoints, three architectures, three scoring passes, no dissent). Recorded as a contract rather than a defect, with the structural cause named — reverse instruction makes DIRECTIVE/state agreement hold in 100% of training pairs, so conflict is out of distribution — and the obligation assigned upstream to the authoring system. §4.4: **CHARACTERS death/status rule promoted from v0.6 candidate to required**, with the finding that the entry alone is insufficient against a competing directive. §4.9: DIRECTIVE named as the highest-authority field, with the rule that it must never name a specific character action (actions belong in BRIEF, manner to DIRECTIVE). §9 renumbered; the narrative-transformation candidate now carries a §3.1 caution, and the state-perturbation entry rewritten to split future rounds into state-reading and precedence families, carrying forward the generous `n_predict` and the pre-registration discipline that caught this round's confound |
+| 0.5.3-windowed-draft | 2026-08-23 | Authoring-system review. STYLE may name an author and may vary only through declared Part-level alternatives. LEXICON is now a complete project-level register projected into packages by author selection, with a rolling selected subset; new terms are proposed only from accepted scenes. STORY thread selection defaults to the preceding canonical scene's thread but remains author-selectable; author-approved generated summaries become its source. CHARACTERS uses an author-selected rolling roster and admits planned, BRIEF-introduced, and accepted-scene-proposed characters. RECALL is explicitly author-selected across threads and earlier series volumes. PREVIOUS defaults to a paragraph-boundary 300–500-word tail but is author-adjustable. NOTES may carry explicitly marked required verbatim text. BRIEF has no authoring-time length limit; unusual lengths receive confirmation. DIRECTIVE requires word limit plus closure semantics, while its fixed wording remains under evaluation. §9 records all remaining package decisions before freeze. |
+| | | *(freeze entry goes here: "0.5.3 FROZEN — SFT generation begins")* |