Document validated authoring prototype

AUTHORING_SYSTEM_SPEC.md

@@ -1,28 +1,32 @@
 # Authoring System Specification
 
-**Version:** 0.1-prototype-draft  
+**Version:** 0.2-prototype-validated
-**Status:** DRAFT  
+**Status:** VALIDATED REFERENCE PROTOTYPE; FUTURE ARCHITECTURE OPEN
-**Companion contract:** `SPEC.md` defines the exact drafting-engine package. This document defines the authoring system that helps an author create, review, archive, and project that package from a Scrivener project.
+**Companion contract:** `SPEC.md` defines the exact drafting-engine package. This document defines the authoring-system invariants for creating, reviewing, archiving, and projecting that package from authoring records; the validated prototype used a Scrivener project.
 
 ---
 
 ## 1. Purpose
 
-The authoring system is a local, interactive assistant for fiction drafting. It keeps the author focused on the `BRIEF`, maintains project continuity, builds a valid package for `SPEC.md`, and manages the draft-to-canon workflow.
+The authoring system is an interactive assistant for fiction drafting. It keeps the author focused on the `BRIEF`, maintains project continuity, builds a valid package for `SPEC.md`, and manages the draft-to-canon workflow.
 
-The first implementation is a quick local prototype for one author on one Mac. It uses a frontier-model API as a temporary stand-in for the eventual dedicated drafting engine.
+The first implementation was a quick local prototype for one author on one Mac. It used a frontier-model API as a temporary stand-in for the eventual dedicated drafting engine. It successfully demonstrated the complete authoring loop against a live Scrivener project; that success validates the workflow and package design, not this particular implementation form.
 
 ## 2. Scope and non-goals
 
-### In scope for the prototype
+### Validated by the prototype
 
 - Read one configured Scrivener project.
+- Create a planned Manuscript scene and its paired blank BRIEF.
 - Assemble and display a package before generation.
 - Call a configurable frontier-model provider with a temporary adapter instruction.
 - Support reject, retry, accept-for-editing, collaborative revision, and accept-into-book.
 - Preserve package snapshots and accepted-for-editing drafts.
 - Propose continuity updates after a scene becomes canonical.
 - Write accepted canonical prose back into Scrivener.
+- Restore a selected canonical scene’s saved package context, approved summary, and any accepted working copy for later review.
+- Display scene state and explicit narrative-thread links as an author-readable continuity dashboard.
+- Select a rolling package subset from a complete project lexicon.
 
 ### Explicitly out of scope
 
@@ -43,17 +47,17 @@ The first implementation is a quick local prototype for one author on one Mac. I
 7. **Visible autonomy.** The author chooses review or blanket-approval behavior by category. Every system-generated proposal retains provenance and status.
 8. **Local-first.** Book content remains on the Mac. The only external transmission is an author-authorized package sent to the configured model provider.
 
-## 4. Prototype architecture
+## 4. Validated reference prototype — not a future architecture requirement
 
-The prototype is a local web application:
+The validated prototype was a local web application:
 
 - **Browser UI:** runs at `http://127.0.0.1:<port>`; no public listener.
-- **Local service:** Python/FastAPI process that owns project access, workflow logic, package construction, and model calls.
+- **Local service:** dependency-light Python process that owned project access, workflow logic, package construction, and model calls.
 - **Scrivener adapter:** reads and writes the `.scriv` bundle only while Scrivener is closed.
-- **Operational sidecar:** a local SQLite database or equivalent adjacent to the project. It records stable bindings, package snapshots, draft history, model-call metadata, and derived-proposal provenance. It is not the source of canonical prose.
+- **Prototype records:** stable scene/BRIEF and narrative-thread bindings were stored in a project-local JSON file; accepted drafts and package snapshots were stored in the Scrivener project. No SQL sidecar was required for the proof of concept.
 - **Frontier-model adapter:** takes a versioned temporary system instruction plus the exact package as the request content. It is replaceable by the dedicated drafting-engine adapter.
 
-The adapter boundary is deliberate: the UI and continuity workflow must not depend on a particular model vendor or API.
+The adapter boundary is deliberate: the UI and continuity workflow must not depend on a particular model vendor or API. A future authoring system may be a native Mac application, a local or hosted web application, or another interface; it may use Scrivener, a database, or both as storage. It must preserve the invariants in this document and the package contract in `SPEC.md`, rather than reproduce this prototype’s technology choices.
 
 ## 5. Scrivener project contract
 
@@ -103,17 +107,17 @@ A blank Manuscript scene paired with a brief is a **planned** scene. It has no c
 - `Places` and `World Notes`: reusable setting and physical-continuity information.
 - `Master Lexicon`: complete book-level controlled vocabulary; the package receives an author-selected rolling subset.
 - `Scene Summaries`: one approved dense event sentence per canonical scene.
-- `Narrative Threads`: records the author-selected thread identity and active order for each scene.
+- `Narrative Threads`: author-visible home for thread material. The prototype stored the actual UUID predecessor links in its project-local binding record; a later implementation may choose a different representation.
 
-Detailed document templates and fields remain to be specified as the prototype exposes the real authoring workflow.
+The prototype confirmed that one `term — definition` lexicon entry per paragraph is a workable author-facing convention and that summaries, characters, places, BRIEFs, and scenes can remain ordinary Scrivener text records. Detailed future record templates remain open.
 
 ## 6. Project access and safety
 
 1. Before every write, the service checks for Scrivener's active project lock. If present, it refuses the operation and tells the author to close Scrivener.
 2. The prototype treats Scrivener as closed for all write operations. It may later support cautious read-only inspection while open, but this is not a v0.1 requirement.
-3. Every Scrivener mutation is non-destructive: create a new document, revise the intended active scene, or move a prior version to `Superseded & Branches`.
+3. Every Scrivener mutation is non-destructive: create a new document, revise the intended active scene, or preserve a prior version outside active Manuscript. The prototype created archive records; full branch restoration remains future work.
 4. The service creates a recoverable project backup before a structural Binder write.
-5. Package snapshots and model-call records are immutable once a request has been made.
+5. Package snapshots and the first accepted-for-editing draft are immutable once saved. The prototype also retained an editable working-copy record that can be restored after restarting the app.
 
 ## 7. Scene lifecycle
 
@@ -137,9 +141,9 @@ planned scene + BRIEF
 - **Abandon after editing acceptance:** retain the preserved snapshot as uncommitted/abandoned; it never affects package context.
 - **Accept into book:** write final edited prose to the paired Manuscript scene and mark it canonical.
 
-### 7.2 Branches and restoration
+### 7.2 Future: branches and restoration
 
-If the author replaces a canonical scene or abandons a continuation, preserve the old text and its bindings in `Superseded & Branches`. The app chooses exactly one active canonical path for package construction. Restoring an earlier path marks dependent derived context stale and reprojects it from the selected point.
+If the author replaces a canonical scene or abandons a continuation, a future implementation should preserve the old text and its bindings in `Superseded & Branches` (or equivalent durable history). It should choose exactly one active canonical path for package construction. Restoring an earlier path should mark dependent derived context stale and reproject it from the selected point. The prototype created the reserved location but did not implement full branch restoration.
 
 ## 8. Package preparation
 
@@ -148,16 +152,16 @@ The app builds the exact nine-section package defined by `SPEC.md`. The author m
 | Section | Prototype source and control |
 |---|---|
 | STYLE | Book setup record; declared Part-level alternate when applicable. |
-| LEXICON | Author-selected rolling subset of Master Lexicon; the app may suggest terms found in BRIEF. |
+| LEXICON | Author-selected rolling subset of Master Lexicon. The validated prototype exposed `term — definition` entries as checkboxes, loaded the selected entries into an editable package field, and carried the subset forward until changed. |
 | STORY | Up to five approved summaries on the selected thread, excluding PREVIOUS. |
 | CHARACTERS | Author-selected rolling roster with proposed current-state snapshots. |
 | RECALL | Author-selected committed scenes from any thread or prior series book, as full text or an editable summary. |
-| PREVIOUS | Author-adjustable, paragraph-boundary verbatim tail from the preceding selected-thread scene. |
+| PREVIOUS | Author-adjustable, paragraph-boundary verbatim tail from any selected canonical scene. |
 | NOTES | Author-selected world constraints and explicitly marked required verbatim text. |
 | BRIEF | Paired author-owned brief. |
 | DIRECTIVE | Author-selected word maximum, ending instruction, and local handling. |
 
-The author selects the narrative thread freely; the default is continuation of the preceding canonical scene's thread. Selecting an existing older thread resumes it. Marking a new thread makes STORY and PREVIOUS empty.
+The author selects narrative-thread membership freely; the default is continuation of the preceding canonical scene's thread. Selecting an existing older thread resumes it. Thread membership is distinct from package context: a new thread may still draw STORY, PREVIOUS, or RECALL from any author-selected canonical scene, and a continuation may omit any of them.
 
 ## 9. Frontier-model stand-in
 
@@ -166,18 +170,18 @@ For the prototype, a model call contains two distinct inputs:
 1. **Frontier-model instructions:** a versioned, project-level adapter prompt stored in `Book Setup → Frontier-Model Instructions`. It explains the temporary provider's role and requires prose-only output.
 2. **Drafting package:** the exact text defined by `SPEC.md`, ending at `### SCENE`.
 
-The adapter records provider, model identifier, instruction-version ID, sampler settings, request time, response, and package snapshot. The temporary instruction is never part of the SFT training pairs or the eventual dedicated-engine package.
+The prototype preserved an immutable package snapshot and accepted draft records in Scrivener, and displayed the exact package before generation. A production implementation should additionally retain provider, model identifier, instruction version, sampler settings, request time, and response provenance in whichever durable storage it adopts. The temporary instruction is never part of the SFT training pairs or the eventual dedicated-engine package.
 
-## 10. Initial browser workflows
+## 10. Validated prototype workflows
 
 ### Project dashboard
 
 - Choose/configure one Scrivener project.
-- Display lock status, active manuscript path, planned scenes, canonical scenes, and stale derived context.
+- Display lock status, planned and canonical scene counts, scene/BRIEF binding health, and a continuity dashboard of explicit thread links and approved summaries.
 
 ### Prepare scene
 
-- Select a planned or existing scene/BRIEF pair.
+- Select a planned or existing scene/BRIEF pair, or create a planned scene and its paired BRIEF directly.
 - Choose or create a narrative thread.
 - Select characters, lexicon entries, recall material, notes, PREVIOUS tail length, and DIRECTIVE.
 - View system-generated suggestions separately from author-approved data.
@@ -185,60 +189,79 @@ The adapter records provider, model identifier, instruction-version ID, sampler
 
 ### Draft and revise
 
-- Display the generated draft alongside the BRIEF and package.
+- Display the generated draft alongside the BRIEF and package; restore a selected scene’s saved package context and working copy when available.
-- Reject, retry, accept for editing, directly edit, or request a frontier-model edit.
+- Reject, retry, accept for editing, and directly edit the accepted working copy.
 - Preserve the first accepted-for-editing draft before any edits.
 - Accept final prose into the paired Manuscript scene.
 
 ### Continuity review
 
-- After acceptance, display proposed summary, character updates, lexicon additions, world notes, and thread-state updates.
+- After acceptance, display proposed summary, character updates, lexicon additions, and world notes; thread membership remains an explicit author choice.
 - Approve, edit, reject, or apply configured blanket approval per category.
 - Show provenance: source scene, method/model, time, and approval status.
 
-## 11. Implementation phases
+## 11. Prototype delivery record
 
-### Phase 1 — project reader and package preview
+### Completed: project reader and package preview
 
 - Load the test Scrivener project and validate the reserved Binder structure.
 - Bind existing `Scene X` briefs, manuscript placeholders, characters, and places.
 - Display them locally; make no Scrivener writes.
 - Assemble a manually selected package and export/display it.
 
-### Phase 2 — frontier generation and draft archive
+### Completed: frontier generation and draft archive
 
 - Add provider configuration and the temporary instruction adapter.
 - Generate, reject, retry, and accept-for-editing drafts.
-- Write immutable draft and package snapshots to the archive/sidecar.
+- Write immutable draft and package snapshots to the Scrivener archive.
 
-### Phase 3 — commit to Manuscript
+### Completed: commit to Manuscript
 
 - Add direct editing and accept-into-book.
 - Safely write final prose to the paired Scrivener scene.
 - Create backups and enforce lock checks.
 
-### Phase 4 — continuity assistance
+### Completed: continuity assistance
 
-- Propose dense scene summaries, character updates, lexicon additions, notes, and thread updates after acceptance.
+- Propose dense scene summaries, character updates, lexicon additions, and world-note additions after acceptance.
-- Add author approval and re-projection for reordered/restored scenes.
+- Add author approval for summaries and continuity updates; full re-projection for reordered/restored branches remains future work.
 
-## 12. Prototype acceptance criteria
+## 12. Prototype acceptance evidence
 
-The prototype is successful if an author can use the Hapa Girls test project to:
+The prototype met these criteria in the Hapa Girls test project:
 
 1. Select one of the prepared scene briefs and its planned scene.
 2. Assemble a visible package from the relevant Scrivener records.
 3. Send it to the temporary frontier-model adapter.
 4. Reject/retry drafts and preserve the first draft accepted for editing.
 5. Edit a selected draft and accept the result into its paired Manuscript scene.
-6. Review and approve a proposed dense summary without the system silently changing canon.
+6. Review and approve a proposed dense summary and continuity updates without the system silently changing canon.
-7. Reopen the Scrivener project and see the accepted scene and archive records intact.
+7. Create a new scene and paired BRIEF in the app, then use it in the same workflow.
+8. Reopen the Scrivener project and see accepted scenes, package snapshots, working copies, summaries, and continuity records intact.
+9. Represent separate and branching narrative threads independently from STORY, PREVIOUS, and RECALL package selections.
 
-## 13. Open design decisions
+## 13. What the prototype proved—and what it did not
+
+### Proven
+
+- The `SPEC.md` package is practical for interactive, scene-by-scene book drafting.
+- Scrivener can hold the author-facing source records, canonical prose, accepted drafts, package snapshots, summaries, and continuity updates required for a useful workflow.
+- The author can retain complete control while receiving model-generated drafts and proposed continuity updates.
+- Stable IDs, not scene labels or chapter placement, can preserve scene/BRIEF and narrative-thread relationships through reorganization.
+- A small local interface is sufficient to make package assembly, review, and restoration manageable for an author.
+
+### Not implied
+
+- Scrivener need not be the sole or eventual storage system.
+- A browser UI, Python service, JSON binding file, OpenRouter, or a Mac-only deployment is not mandated for the next implementation.
+- The frontier-model stand-in is not evidence that the eventual dedicated drafting engine will have identical behaviors or API requirements.
+- Prototype success does not freeze package syntax or resolve the corpus/training decisions in `SPEC.md`.
+
+## 14. Open design decisions for future implementations
 
 - Exact text templates for Characters, Places, Narrative Threads, and archive records.
-- Whether the first prototype stores every archive artifact in Scrivener, the sidecar, or both.
+- Durable storage and synchronization design when Scrivener is not the only store.
-- SQLite schema and backup/restore mechanics.
+- Database or sidecar schema and backup/restore mechanics, if a future implementation needs one.
 - Frontier-model provider and API credentials/configuration approach.
 - Detailed permissions for per-category blanket approval.
 - How the app creates and tracks Part boundaries and style assignments.

SPEC.md

@@ -1,6 +1,6 @@
 # SPEC.md — Drafting Engine Brief Specification
 
-**Version:** 0.5.3-windowed-draft
+**Version:** 0.5.4-windowed-prototype-validated
 **Terminology:** per AUTHORING.md Appendix A.
 **Status:** DRAFT — freeze before any SFT data generation
 **Rule:** Once frozen, changes require a version bump and a retrain. Argue here, not later.
@@ -17,6 +17,12 @@ Defines the exact input package the drafting engine receives and the output it r
 
 The engine honors everything inside the package and nothing outside it. Producing the package (timeline maintenance, character snapshots, tail selection, notes content) is upstream work — author + authoring system — and out of the engine's scope.
 
+### Prototype validation note — 2026-08-27
+
+The complete package was assembled, inspected, sent to a frontier-model stand-in, edited by an author, and accepted into a live Scrivener manuscript through the companion prototype. This demonstrates that the nine-section package is usable as an authoring interface: its fields can be selected independently, its context can be made visible before drafting, and it supports repeatable scene creation without embedding chapter labels in the package.
+
+This is **operational validation, not a format freeze or a training result**. The open decisions in §9 still require corpus and dedicated-engine evaluation before SFT generation.
+
 v0.4 extended the format to a multi-author training corpus (Lovecraft, Galsworthy, Dreiser, Maugham, Hapa Girls). Voice is an explicit, conditioned input via the STYLE section: the engine learns to map a style reference and/or description onto prose. A STYLE may name an author, describe observable techniques, or do both; it does not carry source passages.
 
 v0.5 makes the package **bounded and closure-aware**, driven by a genre review against epic fantasy but general in effect:
@@ -122,6 +128,7 @@ The alternative — constructing SFT pairs whose response defies its own directi
   - `Warder — capitalized when it denotes the bond, lowercase in ordinary use.`
 - **Project versus package scope:** the authoring project maintains a complete book lexicon. The package carries the author-selected subset relevant to the forthcoming scene. The system may suggest entries found in BRIEF or established continuity, but it must not select them silently.
 - The selected subset carries forward as the default for succeeding scenes until the author alters it. The project-level scope of this carry-forward and the master-list threshold that makes subset selection mandatory remain open decisions (§9).
+- The prototype validated a practical author control: a master lexicon maintained as one `term — definition` entry per paragraph can be presented as selectable entries, loaded into an editable package field, and then carried forward. This is an implementation convenience, not additional package syntax.
 - **Package cap: open.** The prior 25-line cap is retained as a working target, not a frozen rule. If nothing applies: `(none)`.
 - A new term found in an accepted scene may be proposed for addition to the complete lexicon. Text from rejected or uncommitted drafts never creates a lexicon entry.
 - Spelling conformance is additionally checked downstream (AUTHORING.md §7). Conditioning reduces the error rate; the check catches the remainder.
@@ -139,6 +146,7 @@ The alternative — constructing SFT pairs whose response defies its own directi
 - Written upstream and maintained as a rolling record; the engine never updates it.
 - If the scene opens a new thread, unrelated to the scenes that came before it (including the opening of the first book), the literal line: `(none - new storyline)`.
 - The author chooses the thread for a scene. The authoring system defaults that choice to the thread of the preceding canonical scene, but the author may resume any existing thread or create a new one. Summaries are derived from accepted scenes and may be edited by the author; an author-edited approved summary is the one used in future packages.
+- **Thread membership is independent of package context.** A scene may start a new thread while drawing STORY summaries, PREVIOUS text, or RECALL from any author-selected earlier scene; conversely, a continuation need not use every available context source. The recorded thread link is continuity bookkeeping, while STORY, PREVIOUS, and RECALL are the author’s separate package choices.
 
 ### 4.4 CHARACTERS — required
 
@@ -389,4 +397,5 @@ Collected here per the header rule: argue in this document, version-bump, retrai
 | 0.5.2-windowed | 2026-08-02 | **Chapters removed from the package.** The engine's units are the scene and the thread; a chapter is presentation owned by the author (`AUTHORING.md` §3) and carries nothing the engine can act on, so no field refers to one. New rule in §3 with the reason stated — §4.10 forbids the engine from producing headers, and feeding it chapter references teaches the habit that prohibition exists to prevent. RECALL source markers restructured: the type suffix stays mandatory, placement becomes an optional phrase in engine-meaningful terms (`(earlier in this book — full text)`, `(Book 1 — summary)`, bare `(full text)`). NOTES state facts rather than citing where they were established — provenance lives in the world-fact envelope and is not projected. CHARACTERS status example de-cited. Validation rule 6a added; §8 gains a reverse-instruction audit, since source novels cite their own chapters constantly and a package builder will carry the habit. Books remain citable — a prior volume is a real boundary in the state model |
 | 0.5.1-windowed | 2026-08-02 | Round 1 state-perturbation results folded in (`STATE_PERTURBATION_REPORT.md`). **New §3.1 — precedence:** DIRECTIVE overrides the state fields whenever they conflict, established by measurement (1.00 responsiveness when DIRECTIVE is silent on the perturbed field, 0.00 when it names the affected behavior; four checkpoints, three architectures, three scoring passes, no dissent). Recorded as a contract rather than a defect, with the structural cause named — reverse instruction makes DIRECTIVE/state agreement hold in 100% of training pairs, so conflict is out of distribution — and the obligation assigned upstream to the authoring system. §4.4: **CHARACTERS death/status rule promoted from v0.6 candidate to required**, with the finding that the entry alone is insufficient against a competing directive. §4.9: DIRECTIVE named as the highest-authority field, with the rule that it must never name a specific character action (actions belong in BRIEF, manner to DIRECTIVE). §9 renumbered; the narrative-transformation candidate now carries a §3.1 caution, and the state-perturbation entry rewritten to split future rounds into state-reading and precedence families, carrying forward the generous `n_predict` and the pre-registration discipline that caught this round's confound |
 | 0.5.3-windowed-draft | 2026-08-23 | Authoring-system review. STYLE may name an author and may vary only through declared Part-level alternatives. LEXICON is now a complete project-level register projected into packages by author selection, with a rolling selected subset; new terms are proposed only from accepted scenes. STORY thread selection defaults to the preceding canonical scene's thread but remains author-selectable; author-approved generated summaries become its source. CHARACTERS uses an author-selected rolling roster and admits planned, BRIEF-introduced, and accepted-scene-proposed characters. RECALL is explicitly author-selected across threads and earlier series volumes. PREVIOUS defaults to a paragraph-boundary 300–500-word tail but is author-adjustable. NOTES may carry explicitly marked required verbatim text. BRIEF has no authoring-time length limit; unusual lengths receive confirmation. DIRECTIVE requires word limit plus closure semantics, while its fixed wording remains under evaluation. §9 records all remaining package decisions before freeze. |
+| 0.5.4-windowed-prototype-validated | 2026-08-27 | A local authoring prototype successfully exercised the package against a live Scrivener manuscript and a frontier-model stand-in. Added the operational-validation note: package fields are practical to select and inspect; lexicon entries can be selected from a project register; and narrative-thread membership is independent of STORY, PREVIOUS, and RECALL context choices. No package delimiter, field order, or training/freeze decision changed. |
 | | | *(freeze entry goes here: "0.5.3 FROZEN — SFT generation begins")* |