diff --git a/docs/superpowers/specs/2026-07-11-content-canon-build-tool-design.md b/docs/superpowers/specs/2026-07-11-content-canon-build-tool-design.md new file mode 100644 index 0000000..bfe930d --- /dev/null +++ b/docs/superpowers/specs/2026-07-11-content-canon-build-tool-design.md @@ -0,0 +1,299 @@ +# Content & Canon Build Tool — Design + +- **Date:** 2026-07-11 +- **Status:** Design approved; not yet planned/implemented. +- **Implements:** the build tool whose *contract* was fixed by + [`2026-07-11-content-canon-schema-design.md`](2026-07-11-content-canon-schema-design.md) + (the locked schema — envelope, two categories, NPC layer, gate logic, + role-based split, canon home, promotion/validation). **That schema is locked; + this doc does not re-litigate it** — it specifies the compiler that realizes it. +- **Specimen:** `candidate-town.md` (Duncarrow) at repo root — the bible the tool + parses and, as its acceptance test, promotes. + +--- + +## Purpose + +Build the **compiler** that turns the authored Markdown bible +(`content/lore/*.md`, prose + fenced `yaml` blocks) into validated runtime JSON, +split by role into a client tree and a server-only tree, with validation as a +hard gate. Promote Duncarrow as the tool's first real use and end-to-end proof. + +**Charter alignment:** §2 (code owns state, AI owns text — this is a build-time +transform, no runtime state), §4 (the server/ split keeps spoilers + personas off +the client), §6 (the gate skeleton feeds `available_moves`), §16 (`/content` +layout, "prompts/content are source, reviewed and versioned"), §18 (branch off +`dev`). + +--- + +## Locked decisions (from the brainstorm) + +| # | Decision | Choice | +|---|---|---| +| L1 | Language / runtime | **Python**, standalone package under `tools/` — reuses the repo's Python + `jsonschema`/pytest stack; no Godot dependency to run a build; not bundled into the `api` proxy image. | +| L2 | Output layout | **Per-category dirs** that fit the existing readers. Minimal, additive loader change; no per-town bundle rewrite. | +| L3 | When it runs | **Manual command + CI `--check`.** Built JSON is committed; CI fails on stale-or-invalid. | +| L4 | `start_disposition` → integer | **Band midpoint.** *Refinement (L4a):* only the client needs band integers at runtime, so the midpoint mapping lives client-side; the build stays **number-free** and emits the rung name. | +| L5 | Promote Duncarrow this cycle | **Yes** — the tool's first real use and acceptance test. | + +### Refinements settled in the brainstorm (explicitly approved) + +- **L4a — the build needs zero disposition numbers.** The only runtime consumer + of band *integers* is the client (it owns disposition state and computes + `available_moves`). The build validates gate **names** only. So band numbers + + the rung→midpoint mapping live in **one** client GDScript const (the + `palette.gd` pattern); the build emits `start_disposition` as the **rung name** + and passes it through. Legal rung names are read from a structured `rungs: [...]` + field on `rule.disposition-ladder` in the bible — the bible owns names/order + (D1), code owns numbers, and there is no cross-language table to keep in sync. +- **L5a — the validator rejects `knows → canon-entity`.** Spec category A: canon + entities never appear in a `knows` list. The specimen's Mayor and Harn carry + `knows: town.duncarrow`; the validator enforces the rule and would reject them. + Promotion **drops those links** (the town is public/tier-0, spoken freely, and + needs no gated knows-link). This is intended — the validator catching real + authoring drift. + +--- + +## Scope boundary + +**In scope (this cycle):** + +- The `tools/content_build/` compiler: parse → resolve/validate → emit → split. +- The validation gate (all rules below), abort-on-failure with source-line context. +- A `--check` mode for CI (stale-or-invalid → nonzero exit). +- Promoting Duncarrow: `candidate-town.md` → `content/lore/duncarrow.md`, + reformat blocks to valid YAML, drop the `knows → canon-entity` links, add + `rungs:` to the ladder, flip entries `candidate → canon`, add + `content/lore/canon-roadmap.md`. +- Committing the built `content/world/**` + `content/server/**` artifacts. +- A **minimal, additive** `content_db.gd` change so the client loader tolerates + the new `canon/` and `topics/` dirs (existing dirs + legacy `fenn.json` + untouched). + +**Out of scope (next cycle / deferred):** + +- **Runtime consumption** — the client computing gated `available_moves` from the + new npc skeletons, the api reading personas from `content/server/`, and voicing + Mera Fenn end-to-end. Promotion proves the **compiler**, not the runtime. +- **Legacy Fenn migration** — `content/world/npcs/fenn.json`'s old + persona+knowledge+capabilities shape stays as-is; it is superseded when Fenn + gets his own bible (flagged deferred in the schema doc). +- **Godot export packaging** — deliberately including `world/` and excluding + `server/` from the client export. `/content` already sits *outside* `res://`, + so `server/` is out of the export by default; the packaging step is the + pre-existing debt, **unblocked** by this clean split but not built here. + +--- + +## Architecture + +`tools/content_build/` — a Python package, run as `python -m content_build`. + +``` +tools/content_build/ + __init__.py + __main__.py # CLI entry: build [--check] [--lore DIR] [--world DIR] [--server DIR] + parse.py # Markdown → list of raw entries (dict + source line) + model.py # Entry / NpcLayer / KnowsLink dataclasses; envelope + category rules + resolve.py # id map, cross-ref resolution, the validation gate + rungs.py # reads ladder `rungs:`; legal-gate set = rungs + {never} + emit.py # route fields per D4; serialize the two JSON trees + errors.py # BuildError carrying (entry id, source file:line, message) + tests/ + fixtures/ # tiny valid + deliberately-broken bibles + test_parse.py + test_resolve.py # one test per gate rule (each broken fixture) + test_emit.py # split routing + secrecy safety-check + test_duncarrow.py # the real end-to-end round-trip +``` + +Dependencies: `PyYAML` (block parsing). Added to a **dev** requirements file, +not `api/requirements.txt` — the proxy runtime never imports the build tool. + +Each module has one job and a narrow interface: + +- **parse** — `parse_bible(path) -> list[RawEntry]`. Knows Markdown fences and + YAML; knows nothing about the schema's meaning. +- **model** — the dataclasses + envelope/category predicates. Pure data + rules, + no I/O. +- **resolve** — `resolve(entries) -> World | raises BuildError`. The gate. Pure + function over parsed entries; no file writes. +- **rungs** — the one place that knows the legal gate vocabulary, sourced from the + ladder entry. +- **emit** — `emit(world, world_dir, server_dir)`. Routing + serialization only. +- **__main__** — orchestration + `--check` diffing + exit codes. + +--- + +## Pipeline + +### 1. Parse + +For each `content/lore/*.md`: extract every ` ```yaml `-fenced block, `yaml.safe_load` +each into a dict, attach source `file:line`. Prose between blocks is ignored. One +block = one entry. + +The promoted bible **must contain valid YAML** — the specimen's current +`- fact: x gate: y` one-liners are *not* valid YAML and become +`{fact: x, gate: y}` (or the two-line block form) during promotion (spec D3: +"formalized as real YAML so parsing is trivial"). + +### 2. Resolve & validate — the gate + +Build the id → entry map, then enforce every rule below. **Any failure aborts the +build, emits nothing, and prints the offending id + source `file:line`.** + +- **No id collision** — two entries with the same `id` → fail. +- **id/type agreement** — the `id` prefix must be a **legal namespace** from the + known set `{town, place, region, faction, person, rule, rumor, fact, secret, + npc}`, and `type` must be **consistent** with it: prefix `npc` → `type` is + `person` (the interactive layer vs the `person.*` world-record); otherwise the + prefix equals `type`. A non-namespace prefix (e.g. the specimen's `shrine.`) → + fail; such ids are renamed to their type namespace at promotion + (`shrine.the-white-antlers` → `place.the-white-antlers`). +- **`related` resolves** — every id in every `related` list is a real entry + **defined in the built world** (strict, per schema §Build). The specimen + references `region.the-tallow-reach` and `faction.elves` without defining them; + promotion adds minimal canon stubs for both so the graph stays closed (there is + no forward-ref-to-pending concept — the closed graph is the invariant). +- **NPC-layer shape** — `npc.*` entries carry `knows` **and** `start_disposition`; + `person.*` records carry **neither**. +- **`knows` targets** — every `knows[].fact` resolves to a **knowledge entry** + (`rumor`/`fact`/`secret`). A canon-entity id in a `knows` list → fail (L5a). +- **Gate legality** — every `knows[].gate` ∈ `rungs` (read from + `rule.disposition-ladder`) ∪ `{never}`. +- **Ladder coverage** — exactly one `rule.disposition-ladder` entry exists and + carries a non-empty `rungs:` list. +- **Secrecy safety-check** — after routing (stage 4), no `secrecy ≥ 3` body may + appear in any client (`content/world/**`) artifact → fail. Defense-in-depth + behind the structural routing, not the router itself (schema D4). +- **`start_disposition` rung legality** — the rung is one of `rungs`. + +Contradiction-with-existing-canon is **author-asserted**, not auto-checked — it is +resolved by the human at the `candidate → canon` promotion (schema §Build). + +### 3. Emit + +Only `status: canon` entries build into artifacts. `candidate` entries are +dev-visible (they parse and validate) but are **not emitted** — with an +all-`candidate` bible the artifacts are empty, which is why L5 promotes Duncarrow +to `canon` this cycle. + +### 4. Split (routing per schema D4) + +**Client — `content/world/`** (bundled in the export; also readable by the server): + +| Dir | Contents | +|---|---| +| `npcs/.json` | npc **skeleton**: `{id, type:"person", start_disposition:"", related, knows:[{fact_id, gate}]}` — no persona/body | +| `canon/.json` | each public canon entity (`town/place/region/faction/person/rule`) **whole**, including its public `body` and (for the ladder) `rungs` | +| `topics/.json` | knowledge-entry **skeleton**: `{id, type, related}` — **no body** | + +**Server — `content/server/`** (server-only; excluded from the Godot export): + +| Dir | Contents | +|---|---| +| `npcs/.json` | `{id, persona:, disposition_notes}` | +| `topics/.json` | `{id, body, secrecy}` — the knowledge bodies `/npc/speak` voices | + +- Canon-entity `body` is **public** (category A, `secrecy:0`) → ships in + `world/canon/`. +- Knowledge-entry `body` and NPC persona/`disposition_notes` → `content/server/` + only. +- The server process reads **both** trees (it has full filesystem access); only + the **client export** excludes `content/server/`. + +`` = the id with its `.`/`npc.` prefix stripped +(`npc.mera-fenn` → `mera-fenn`, `town.duncarrow` → `duncarrow`). + +--- + +## Client loader change (minimal, additive) + +`client/scripts/content/content_db.gd` gains two dirs: + +- `load_from()` also loads `world/canon/` and `world/topics/` into generic + id-keyed dictionaries (`canon`, `topics`), mirroring the existing + `_load_dir` pattern. +- Existing `locations/npcs/quests/items` loading is untouched; the legacy + `fenn.json` still loads (it has an `id`, which is all `_load_dir` requires). +- Add `canon(id)` / `topic(id)` / `has_canon` / `has_topic` accessors. + +No api change this cycle — runtime consumption of the new shapes is next cycle +(scope boundary). + +--- + +## CLI & CI + +- `python -m content_build` — rebuild `content/world/**` + `content/server/**` + from `content/lore/*.md`. +- `python -m content_build --check` — build to memory, diff against the committed + JSON, exit **nonzero** if the tree is **stale** (would-differ) **or** validation + fails. This is the CI gate. +- Dir overrides (`--lore/--world/--server`) exist for tests to run against + fixtures in a temp dir. + +Built JSON is **committed** (schema/prompt "content is source, reviewed" culture); +`--check` keeps the committed artifacts honest. + +--- + +## Testing + +- **Parse** — a fixture bible with prose + multiple `yaml` blocks parses to the + expected entry list with source lines. +- **Resolve** — **one deliberately-broken fixture per gate rule** (id collision, + bad id/type, dangling `related`, `person.*` with `knows`, `knows → canon-entity`, + illegal gate, missing ladder, `secrecy ≥ 3` body routed to client), each + asserting the specific `BuildError` + that nothing is emitted. +- **Emit** — a valid fixture routes every field to the correct tree; the + secrecy safety-check passes for a legal body and fails for an illegal one. +- **Duncarrow round-trip** (`test_duncarrow.py`) — the real acceptance test: the + promoted `content/lore/duncarrow.md` builds green, and the emitted client/server + JSON matches expectations (npc skeletons carry no persona; the `secret.crell-*` + body lands only in `content/server/topics/`; `world/canon/` holds the public + entities incl. the ladder `rungs`). + +--- + +## Promotion checklist (Duncarrow — the acceptance run) + +1. `git mv candidate-town.md content/lore/duncarrow.md`. +2. Reformat every fenced block to a ` ```yaml ` fence with valid YAML (fix the + `knows` one-liners to `{fact, gate}`). +3. Drop the `knows: town.duncarrow` links from Mayor and Harn (L5a). +4. Rename `shrine.the-white-antlers` → `place.the-white-antlers` and update every + `related` reference to it (id/type-agreement rule). +5. Add canon stubs for `region.the-tallow-reach` and `faction.elves` so all + `related` ids resolve (closed-graph rule). +6. Drop `secret.crell-runs-slave-trade` from the **public** `person.mayor-oswin-crell` + record's `related` — a public record must not point at a secret; that graph + edge lives on the secret entry (server-side). The `npc.*` layer still holds the + secret at `gate: never`, which is the actual bounded-move test. +7. Add `rungs: [hostile, cold, neutral, warm, trusted]` to + `rule.disposition-ladder`. +8. Add `content/lore/canon-roadmap.md` (the authored-vs-pending index, schema D5). +9. Run the tool; fix validation errors; flip each entry `candidate → canon` as it + passes. +10. Commit the promoted bible + built `content/world/**` + `content/server/**`. +11. `--check` is green in CI. + +--- + +## Open / deferred (carried, not decided here) + +- **Band numbers** — tunable placeholder in the client const; settle by playtest + (schema D1). The build never sees them (L4a). +- **Runtime consumption / legacy Fenn migration / export packaging** — the scope + boundary above; each is its own later cycle. +- **`make`/`just` wrapper** — a convenience target for the build command can be + added when a task runner lands; not required for this cycle. +- **ids are self-describing (spoiler property, schema-level).** Per D4 the gate + skeleton ships each NPC's `knows:[{fact_id, gate}]` to the client, so a + descriptive slug like `secret.crell-runs-slave-trade` reaches the client binary + even when held at `gate: never` (the body never does). This is a property of the + **locked** schema, not the build tool — opaque/hashed topic ids are a possible + future hardening if datamining becomes a concern. Noted, not reopened.