Document validated authoring prototype

AUTHORING_SYSTEM_SPEC.md

@@ -1,28 +1,32 @@
 # Authoring System Specification
 
-**Version:** 0.1-prototype-draft  
-**Status:** DRAFT  
-**Companion contract:** `SPEC.md` defines the exact drafting-engine package. This document defines the authoring system that helps an author create, review, archive, and project that package from a Scrivener project.
+**Version:** 0.2-prototype-validated
+**Status:** VALIDATED REFERENCE PROTOTYPE; FUTURE ARCHITECTURE OPEN
+**Companion contract:** `SPEC.md` defines the exact drafting-engine package. This document defines the authoring-system invariants for creating, reviewing, archiving, and projecting that package from authoring records; the validated prototype used a Scrivener project.
 
 ---
 
 ## 1. Purpose
 
-The authoring system is a local, interactive assistant for fiction drafting. It keeps the author focused on the `BRIEF`, maintains project continuity, builds a valid package for `SPEC.md`, and manages the draft-to-canon workflow.
+The authoring system is an interactive assistant for fiction drafting. It keeps the author focused on the `BRIEF`, maintains project continuity, builds a valid package for `SPEC.md`, and manages the draft-to-canon workflow.
 
-The first implementation is a quick local prototype for one author on one Mac. It uses a frontier-model API as a temporary stand-in for the eventual dedicated drafting engine.
+The first implementation was a quick local prototype for one author on one Mac. It used a frontier-model API as a temporary stand-in for the eventual dedicated drafting engine. It successfully demonstrated the complete authoring loop against a live Scrivener project; that success validates the workflow and package design, not this particular implementation form.
 
 ## 2. Scope and non-goals
 
-### In scope for the prototype
+### Validated by the prototype
 
 - Read one configured Scrivener project.
+- Create a planned Manuscript scene and its paired blank BRIEF.
 - Assemble and display a package before generation.
 - Call a configurable frontier-model provider with a temporary adapter instruction.
 - Support reject, retry, accept-for-editing, collaborative revision, and accept-into-book.
 - Preserve package snapshots and accepted-for-editing drafts.
 - Propose continuity updates after a scene becomes canonical.
 - Write accepted canonical prose back into Scrivener.
+- Restore a selected canonical scene’s saved package context, approved summary, and any accepted working copy for later review.
+- Display scene state and explicit narrative-thread links as an author-readable continuity dashboard.
+- Select a rolling package subset from a complete project lexicon.
 
 ### Explicitly out of scope
 
@@ -43,17 +47,17 @@ The first implementation is a quick local prototype for one author on one Mac. I
 7. **Visible autonomy.** The author chooses review or blanket-approval behavior by category. Every system-generated proposal retains provenance and status.
 8. **Local-first.** Book content remains on the Mac. The only external transmission is an author-authorized package sent to the configured model provider.
 
-## 4. Prototype architecture
+## 4. Validated reference prototype — not a future architecture requirement
 
-The prototype is a local web application:
+The validated prototype was a local web application:
 
 - **Browser UI:** runs at `http://127.0.0.1:<port>`; no public listener.
-- **Local service:** Python/FastAPI process that owns project access, workflow logic, package construction, and model calls.
+- **Local service:** dependency-light Python process that owned project access, workflow logic, package construction, and model calls.
 - **Scrivener adapter:** reads and writes the `.scriv` bundle only while Scrivener is closed.
-- **Operational sidecar:** a local SQLite database or equivalent adjacent to the project. It records stable bindings, package snapshots, draft history, model-call metadata, and derived-proposal provenance. It is not the source of canonical prose.
+- **Prototype records:** stable scene/BRIEF and narrative-thread bindings were stored in a project-local JSON file; accepted drafts and package snapshots were stored in the Scrivener project. No SQL sidecar was required for the proof of concept.
 - **Frontier-model adapter:** takes a versioned temporary system instruction plus the exact package as the request content. It is replaceable by the dedicated drafting-engine adapter.
 
-The adapter boundary is deliberate: the UI and continuity workflow must not depend on a particular model vendor or API.
+The adapter boundary is deliberate: the UI and continuity workflow must not depend on a particular model vendor or API. A future authoring system may be a native Mac application, a local or hosted web application, or another interface; it may use Scrivener, a database, or both as storage. It must preserve the invariants in this document and the package contract in `SPEC.md`, rather than reproduce this prototype’s technology choices.
 
 ## 5. Scrivener project contract
 
@@ -103,17 +107,17 @@ A blank Manuscript scene paired with a brief is a **planned** scene. It has no c
 - `Places` and `World Notes`: reusable setting and physical-continuity information.
 - `Master Lexicon`: complete book-level controlled vocabulary; the package receives an author-selected rolling subset.
 - `Scene Summaries`: one approved dense event sentence per canonical scene.
-- `Narrative Threads`: records the author-selected thread identity and active order for each scene.
+- `Narrative Threads`: author-visible home for thread material. The prototype stored the actual UUID predecessor links in its project-local binding record; a later implementation may choose a different representation.
 
-Detailed document templates and fields remain to be specified as the prototype exposes the real authoring workflow.
+The prototype confirmed that one `term — definition` lexicon entry per paragraph is a workable author-facing convention and that summaries, characters, places, BRIEFs, and scenes can remain ordinary Scrivener text records. Detailed future record templates remain open.
 
 ## 6. Project access and safety
 
 1. Before every write, the service checks for Scrivener's active project lock. If present, it refuses the operation and tells the author to close Scrivener.
 2. The prototype treats Scrivener as closed for all write operations. It may later support cautious read-only inspection while open, but this is not a v0.1 requirement.
-3. Every Scrivener mutation is non-destructive: create a new document, revise the intended active scene, or move a prior version to `Superseded & Branches`.
+3. Every Scrivener mutation is non-destructive: create a new document, revise the intended active scene, or preserve a prior version outside active Manuscript. The prototype created archive records; full branch restoration remains future work.
 4. The service creates a recoverable project backup before a structural Binder write.
-5. Package snapshots and model-call records are immutable once a request has been made.
+5. Package snapshots and the first accepted-for-editing draft are immutable once saved. The prototype also retained an editable working-copy record that can be restored after restarting the app.
 
 ## 7. Scene lifecycle
 
@@ -137,9 +141,9 @@ planned scene + BRIEF
 - **Abandon after editing acceptance:** retain the preserved snapshot as uncommitted/abandoned; it never affects package context.
 - **Accept into book:** write final edited prose to the paired Manuscript scene and mark it canonical.
 
-### 7.2 Branches and restoration
+### 7.2 Future: branches and restoration
 
-If the author replaces a canonical scene or abandons a continuation, preserve the old text and its bindings in `Superseded & Branches`. The app chooses exactly one active canonical path for package construction. Restoring an earlier path marks dependent derived context stale and reprojects it from the selected point.
+If the author replaces a canonical scene or abandons a continuation, a future implementation should preserve the old text and its bindings in `Superseded & Branches` (or equivalent durable history). It should choose exactly one active canonical path for package construction. Restoring an earlier path should mark dependent derived context stale and reproject it from the selected point. The prototype created the reserved location but did not implement full branch restoration.
 
 ## 8. Package preparation
 
@@ -148,16 +152,16 @@ The app builds the exact nine-section package defined by `SPEC.md`. The author m
 | Section | Prototype source and control |
 |---|---|
 | STYLE | Book setup record; declared Part-level alternate when applicable. |
-| LEXICON | Author-selected rolling subset of Master Lexicon; the app may suggest terms found in BRIEF. |
+| LEXICON | Author-selected rolling subset of Master Lexicon. The validated prototype exposed `term — definition` entries as checkboxes, loaded the selected entries into an editable package field, and carried the subset forward until changed. |
 | STORY | Up to five approved summaries on the selected thread, excluding PREVIOUS. |
 | CHARACTERS | Author-selected rolling roster with proposed current-state snapshots. |
 | RECALL | Author-selected committed scenes from any thread or prior series book, as full text or an editable summary. |
-| PREVIOUS | Author-adjustable, paragraph-boundary verbatim tail from the preceding selected-thread scene. |
+| PREVIOUS | Author-adjustable, paragraph-boundary verbatim tail from any selected canonical scene. |
 | NOTES | Author-selected world constraints and explicitly marked required verbatim text. |
 | BRIEF | Paired author-owned brief. |
 | DIRECTIVE | Author-selected word maximum, ending instruction, and local handling. |
 
-The author selects the narrative thread freely; the default is continuation of the preceding canonical scene's thread. Selecting an existing older thread resumes it. Marking a new thread makes STORY and PREVIOUS empty.
+The author selects narrative-thread membership freely; the default is continuation of the preceding canonical scene's thread. Selecting an existing older thread resumes it. Thread membership is distinct from package context: a new thread may still draw STORY, PREVIOUS, or RECALL from any author-selected canonical scene, and a continuation may omit any of them.
 
 ## 9. Frontier-model stand-in
 
@@ -166,18 +170,18 @@ For the prototype, a model call contains two distinct inputs:
 1. **Frontier-model instructions:** a versioned, project-level adapter prompt stored in `Book Setup → Frontier-Model Instructions`. It explains the temporary provider's role and requires prose-only output.
 2. **Drafting package:** the exact text defined by `SPEC.md`, ending at `### SCENE`.
 
-The adapter records provider, model identifier, instruction-version ID, sampler settings, request time, response, and package snapshot. The temporary instruction is never part of the SFT training pairs or the eventual dedicated-engine package.
+The prototype preserved an immutable package snapshot and accepted draft records in Scrivener, and displayed the exact package before generation. A production implementation should additionally retain provider, model identifier, instruction version, sampler settings, request time, and response provenance in whichever durable storage it adopts. The temporary instruction is never part of the SFT training pairs or the eventual dedicated-engine package.
 
-## 10. Initial browser workflows
+## 10. Validated prototype workflows
 
 ### Project dashboard
 
 - Choose/configure one Scrivener project.
-- Display lock status, active manuscript path, planned scenes, canonical scenes, and stale derived context.
+- Display lock status, planned and canonical scene counts, scene/BRIEF binding health, and a continuity dashboard of explicit thread links and approved summaries.
 
 ### Prepare scene
 
-- Select a planned or existing scene/BRIEF pair.
+- Select a planned or existing scene/BRIEF pair, or create a planned scene and its paired BRIEF directly.
 - Choose or create a narrative thread.
 - Select characters, lexicon entries, recall material, notes, PREVIOUS tail length, and DIRECTIVE.
 - View system-generated suggestions separately from author-approved data.
@@ -185,60 +189,79 @@ The adapter records provider, model identifier, instruction-version ID, sampler
 
 ### Draft and revise
 
-- Display the generated draft alongside the BRIEF and package.
-- Reject, retry, accept for editing, directly edit, or request a frontier-model edit.
+- Display the generated draft alongside the BRIEF and package; restore a selected scene’s saved package context and working copy when available.
+- Reject, retry, accept for editing, and directly edit the accepted working copy.
 - Preserve the first accepted-for-editing draft before any edits.
 - Accept final prose into the paired Manuscript scene.
 
 ### Continuity review
 
-- After acceptance, display proposed summary, character updates, lexicon additions, world notes, and thread-state updates.
+- After acceptance, display proposed summary, character updates, lexicon additions, and world notes; thread membership remains an explicit author choice.
 - Approve, edit, reject, or apply configured blanket approval per category.
 - Show provenance: source scene, method/model, time, and approval status.
 
-## 11. Implementation phases
+## 11. Prototype delivery record
 
-### Phase 1 — project reader and package preview
+### Completed: project reader and package preview
 
 - Load the test Scrivener project and validate the reserved Binder structure.
 - Bind existing `Scene X` briefs, manuscript placeholders, characters, and places.
 - Display them locally; make no Scrivener writes.
 - Assemble a manually selected package and export/display it.
 
-### Phase 2 — frontier generation and draft archive
+### Completed: frontier generation and draft archive
 
 - Add provider configuration and the temporary instruction adapter.
 - Generate, reject, retry, and accept-for-editing drafts.
-- Write immutable draft and package snapshots to the archive/sidecar.
+- Write immutable draft and package snapshots to the Scrivener archive.
 
-### Phase 3 — commit to Manuscript
+### Completed: commit to Manuscript
 
 - Add direct editing and accept-into-book.
 - Safely write final prose to the paired Scrivener scene.
 - Create backups and enforce lock checks.
 
-### Phase 4 — continuity assistance
+### Completed: continuity assistance
 
-- Propose dense scene summaries, character updates, lexicon additions, notes, and thread updates after acceptance.
-- Add author approval and re-projection for reordered/restored scenes.
+- Propose dense scene summaries, character updates, lexicon additions, and world-note additions after acceptance.
+- Add author approval for summaries and continuity updates; full re-projection for reordered/restored branches remains future work.
 
-## 12. Prototype acceptance criteria
+## 12. Prototype acceptance evidence
 
-The prototype is successful if an author can use the Hapa Girls test project to:
+The prototype met these criteria in the Hapa Girls test project:
 
 1. Select one of the prepared scene briefs and its planned scene.
 2. Assemble a visible package from the relevant Scrivener records.
 3. Send it to the temporary frontier-model adapter.
 4. Reject/retry drafts and preserve the first draft accepted for editing.
 5. Edit a selected draft and accept the result into its paired Manuscript scene.
-6. Review and approve a proposed dense summary without the system silently changing canon.
-7. Reopen the Scrivener project and see the accepted scene and archive records intact.
+6. Review and approve a proposed dense summary and continuity updates without the system silently changing canon.
+7. Create a new scene and paired BRIEF in the app, then use it in the same workflow.
+8. Reopen the Scrivener project and see accepted scenes, package snapshots, working copies, summaries, and continuity records intact.
+9. Represent separate and branching narrative threads independently from STORY, PREVIOUS, and RECALL package selections.
 
-## 13. Open design decisions
+## 13. What the prototype proved—and what it did not
+
+### Proven
+
+- The `SPEC.md` package is practical for interactive, scene-by-scene book drafting.
+- Scrivener can hold the author-facing source records, canonical prose, accepted drafts, package snapshots, summaries, and continuity updates required for a useful workflow.
+- The author can retain complete control while receiving model-generated drafts and proposed continuity updates.
+- Stable IDs, not scene labels or chapter placement, can preserve scene/BRIEF and narrative-thread relationships through reorganization.
+- A small local interface is sufficient to make package assembly, review, and restoration manageable for an author.
+
+### Not implied
+
+- Scrivener need not be the sole or eventual storage system.
+- A browser UI, Python service, JSON binding file, OpenRouter, or a Mac-only deployment is not mandated for the next implementation.
+- The frontier-model stand-in is not evidence that the eventual dedicated drafting engine will have identical behaviors or API requirements.
+- Prototype success does not freeze package syntax or resolve the corpus/training decisions in `SPEC.md`.
+
+## 14. Open design decisions for future implementations
 
 - Exact text templates for Characters, Places, Narrative Threads, and archive records.
-- Whether the first prototype stores every archive artifact in Scrivener, the sidecar, or both.
-- SQLite schema and backup/restore mechanics.
+- Durable storage and synchronization design when Scrivener is not the only store.
+- Database or sidecar schema and backup/restore mechanics, if a future implementation needs one.
 - Frontier-model provider and API credentials/configuration approach.
 - Detailed permissions for per-category blanket approval.
 - How the app creates and tracks Part boundaries and style assignments.