code_of_conquest_dnd/docs/superpowers/specs/2026-07-11-content-canon-build-tool-design.md

# Content & Canon Build Tool — Design

- **Date:** 2026-07-11
- **Status:** Design approved; not yet planned/implemented.
- **Implements:** the build tool whose *contract* was fixed by
  [`2026-07-11-content-canon-schema-design.md`](2026-07-11-content-canon-schema-design.md)
  (the locked schema — envelope, two categories, NPC layer, gate logic,
  role-based split, canon home, promotion/validation). **That schema is locked;
  this doc does not re-litigate it** — it specifies the compiler that realizes it.
- **Specimen:** `candidate-town.md` (Duncarrow) at repo root — the bible the tool
  parses and, as its acceptance test, promotes.

---

## Purpose

Build the **compiler** that turns the authored Markdown bible
(`content/lore/*.md`, prose + fenced `yaml` blocks) into validated runtime JSON,
split by role into a client tree and a server-only tree, with validation as a
hard gate. Promote Duncarrow as the tool's first real use and end-to-end proof.

**Charter alignment:** §2 (code owns state, AI owns text — this is a build-time
transform, no runtime state), §4 (the server/ split keeps spoilers + personas off
the client), §6 (the gate skeleton feeds `available_moves`), §16 (`/content`
layout, "prompts/content are source, reviewed and versioned"), §18 (branch off
`dev`).

---

## Locked decisions (from the brainstorm)

| # | Decision | Choice |
|---|---|---|
| L1 | Language / runtime | **Python**, standalone package under `tools/` — reuses the repo's Python + `jsonschema`/pytest stack; no Godot dependency to run a build; not bundled into the `api` proxy image. |
| L2 | Output layout | **Per-category dirs** that fit the existing readers. Minimal, additive loader change; no per-town bundle rewrite. |
| L3 | When it runs | **Manual command + CI `--check`.** Built JSON is committed; CI fails on stale-or-invalid. |
| L4 | `start_disposition` → integer | **Band midpoint.** *Refinement (L4a):* only the client needs band integers at runtime, so the midpoint mapping lives client-side; the build stays **number-free** and emits the rung name. |
| L5 | Promote Duncarrow this cycle | **Yes** — the tool's first real use and acceptance test. |

### Refinements settled in the brainstorm (explicitly approved)

- **L4a — the build needs zero disposition numbers.** The only runtime consumer
  of band *integers* is the client (it owns disposition state and computes
  `available_moves`). The build validates gate **names** only. So band numbers +
  the rung→midpoint mapping live in **one** client GDScript const (the
  `palette.gd` pattern); the build emits `start_disposition` as the **rung name**
  and passes it through. Legal rung names are read from a structured `rungs: [...]`
  field on `rule.disposition-ladder` in the bible — the bible owns names/order
  (D1), code owns numbers, and there is no cross-language table to keep in sync.
- **L5a — the validator rejects `knows → canon-entity`.** Spec category A: canon
  entities never appear in a `knows` list. The specimen's Mayor and Harn carry
  `knows: town.duncarrow`; the validator enforces the rule and would reject them.
  Promotion **drops those links** (the town is public/tier-0, spoken freely, and
  needs no gated knows-link). This is intended — the validator catching real
  authoring drift.

---

## Scope boundary

**In scope (this cycle):**

- The `tools/content_build/` compiler: parse → resolve/validate → emit → split.
- The validation gate (all rules below), abort-on-failure with source-line context.
- A `--check` mode for CI (stale-or-invalid → nonzero exit).
- Promoting Duncarrow: `candidate-town.md` → `content/lore/duncarrow.md`,
  reformat blocks to valid YAML, drop the `knows → canon-entity` links, add
  `rungs:` to the ladder, flip entries `candidate → canon`, add
  `content/lore/canon-roadmap.md`.
- Committing the built `content/world/**` + `content/server/**` artifacts.
- A **minimal, additive** `content_db.gd` change so the client loader tolerates
  the new `canon/` and `topics/` dirs (existing dirs + legacy `fenn.json`
  untouched).

**Out of scope (next cycle / deferred):**

- **Runtime consumption** — the client computing gated `available_moves` from the
  new npc skeletons, the api reading personas from `content/server/`, and voicing
  Mera Fenn end-to-end. Promotion proves the **compiler**, not the runtime.
- **Legacy Fenn migration** — `content/world/npcs/fenn.json`'s old
  persona+knowledge+capabilities shape stays as-is; it is superseded when Fenn
  gets his own bible (flagged deferred in the schema doc).
- **Godot export packaging** — deliberately including `world/` and excluding
  `server/` from the client export. `/content` already sits *outside* `res://`,
  so `server/` is out of the export by default; the packaging step is the
  pre-existing debt, **unblocked** by this clean split but not built here.

---

## Architecture

`tools/content_build/` — a Python package, run as `python -m content_build`.

```
tools/content_build/
  __init__.py
  __main__.py    # CLI entry: build [--check] [--lore DIR] [--world DIR] [--server DIR]
  parse.py       # Markdown → list of raw entries (dict + source line)
  model.py       # Entry / NpcLayer / KnowsLink dataclasses; envelope + category rules
  resolve.py     # id map, cross-ref resolution, the validation gate
  rungs.py       # reads ladder `rungs:`; legal-gate set = rungs + {never}
  emit.py        # route fields per D4; serialize the two JSON trees
  errors.py      # BuildError carrying (entry id, source file:line, message)
  tests/
    fixtures/    # tiny valid + deliberately-broken bibles
    test_parse.py
    test_resolve.py       # one test per gate rule (each broken fixture)
    test_emit.py          # split routing + secrecy safety-check
    test_duncarrow.py     # the real end-to-end round-trip
```

Dependencies: `PyYAML` (block parsing). Added to a **dev** requirements file,
not `api/requirements.txt` — the proxy runtime never imports the build tool.

Each module has one job and a narrow interface:

- **parse** — `parse_bible(path) -> list[RawEntry]`. Knows Markdown fences and
  YAML; knows nothing about the schema's meaning.
- **model** — the dataclasses + envelope/category predicates. Pure data + rules,
  no I/O.
- **resolve** — `resolve(entries) -> World | raises BuildError`. The gate. Pure
  function over parsed entries; no file writes.
- **rungs** — the one place that knows the legal gate vocabulary, sourced from the
  ladder entry.
- **emit** — `emit(world, world_dir, server_dir)`. Routing + serialization only.
- **__main__** — orchestration + `--check` diffing + exit codes.

---

## Pipeline

### 1. Parse

For each `content/lore/*.md`: extract every ` ```yaml `-fenced block, `yaml.safe_load`
each into a dict, attach source `file:line`. Prose between blocks is ignored. One
block = one entry.

The promoted bible **must contain valid YAML** — the specimen's current
`- fact: x   gate: y` one-liners are *not* valid YAML and become
`{fact: x, gate: y}` (or the two-line block form) during promotion (spec D3:
"formalized as real YAML so parsing is trivial").

### 2. Resolve & validate — the gate

Build the id → entry map, then enforce every rule below. **Any failure aborts the
build, emits nothing, and prints the offending id + source `file:line`.**

- **No id collision** — two entries with the same `id` → fail.
- **id/type agreement** — the `id` prefix must be a **legal namespace** from the
  known set `{town, place, region, faction, person, rule, rumor, fact, secret,
  npc}`, and `type` must be **consistent** with it: prefix `npc` → `type` is
  `person` (the interactive layer vs the `person.*` world-record); otherwise the
  prefix equals `type`. A non-namespace prefix (e.g. the specimen's `shrine.`) →
  fail; such ids are renamed to their type namespace at promotion
  (`shrine.the-white-antlers` → `place.the-white-antlers`).
- **`related` resolves** — every id in every `related` list is a real entry
  **defined in the built world** (strict, per schema §Build). The specimen
  references `region.the-tallow-reach` and `faction.elves` without defining them;
  promotion adds minimal canon stubs for both so the graph stays closed (there is
  no forward-ref-to-pending concept — the closed graph is the invariant).
- **NPC-layer shape** — `npc.*` entries carry `knows` **and** `start_disposition`;
  `person.*` records carry **neither**.
- **`knows` targets** — every `knows[].fact` resolves to a **knowledge entry**
  (`rumor`/`fact`/`secret`). A canon-entity id in a `knows` list → fail (L5a).
- **Gate legality** — every `knows[].gate` ∈ `rungs` (read from
  `rule.disposition-ladder`) ∪ `{never}`.
- **Ladder coverage** — exactly one `rule.disposition-ladder` entry exists and
  carries a non-empty `rungs:` list.
- **Secrecy safety-check** — after routing (stage 4), no `secrecy ≥ 3` body may
  appear in any client (`content/world/**`) artifact → fail. Defense-in-depth
  behind the structural routing, not the router itself (schema D4).
- **`start_disposition` rung legality** — the rung is one of `rungs`.

Contradiction-with-existing-canon is **author-asserted**, not auto-checked — it is
resolved by the human at the `candidate → canon` promotion (schema §Build).

### 3. Emit

Only `status: canon` entries build into artifacts. `candidate` entries are
dev-visible (they parse and validate) but are **not emitted** — with an
all-`candidate` bible the artifacts are empty, which is why L5 promotes Duncarrow
to `canon` this cycle.

### 4. Split (routing per schema D4)

**Client — `content/world/`** (bundled in the export; also readable by the server):

| Dir | Contents |
|---|---|
| `npcs/<slug>.json` | npc **skeleton**: `{id, type:"person", start_disposition:"<rung>", related, knows:[{fact_id, gate}]}` — no persona/body |
| `canon/<slug>.json` | each public canon entity (`town/place/region/faction/person/rule`) **whole**, including its public `body` and (for the ladder) `rungs` |
| `topics/<slug>.json` | knowledge-entry **skeleton**: `{id, type, related}` — **no body** |

**Server — `content/server/`** (server-only; excluded from the Godot export):

| Dir | Contents |
|---|---|
| `npcs/<slug>.json` | `{id, persona:<body>, disposition_notes}` |
| `topics/<slug>.json` | `{id, body, secrecy}` — the knowledge bodies `/npc/speak` voices |

- Canon-entity `body` is **public** (category A, `secrecy:0`) → ships in
  `world/canon/`.
- Knowledge-entry `body` and NPC persona/`disposition_notes` → `content/server/`
  only.
- The server process reads **both** trees (it has full filesystem access); only
  the **client export** excludes `content/server/`.

`<slug>` = the id with its `<type>.`/`npc.` prefix stripped
(`npc.mera-fenn` → `mera-fenn`, `town.duncarrow` → `duncarrow`).

---

## Client loader change (minimal, additive)

`client/scripts/content/content_db.gd` gains two dirs:

- `load_from()` also loads `world/canon/` and `world/topics/` into generic
  id-keyed dictionaries (`canon`, `topics`), mirroring the existing
  `_load_dir` pattern.
- Existing `locations/npcs/quests/items` loading is untouched; the legacy
  `fenn.json` still loads (it has an `id`, which is all `_load_dir` requires).
- Add `canon(id)` / `topic(id)` / `has_canon` / `has_topic` accessors.

No api change this cycle — runtime consumption of the new shapes is next cycle
(scope boundary).

---

## CLI & CI

- `python -m content_build` — rebuild `content/world/**` + `content/server/**`
  from `content/lore/*.md`.
- `python -m content_build --check` — build to memory, diff against the committed
  JSON, exit **nonzero** if the tree is **stale** (would-differ) **or** validation
  fails. This is the CI gate.
- Dir overrides (`--lore/--world/--server`) exist for tests to run against
  fixtures in a temp dir.

Built JSON is **committed** (schema/prompt "content is source, reviewed" culture);
`--check` keeps the committed artifacts honest.

---

## Testing

- **Parse** — a fixture bible with prose + multiple `yaml` blocks parses to the
  expected entry list with source lines.
- **Resolve** — **one deliberately-broken fixture per gate rule** (id collision,
  bad id/type, dangling `related`, `person.*` with `knows`, `knows → canon-entity`,
  illegal gate, missing ladder, `secrecy ≥ 3` body routed to client), each
  asserting the specific `BuildError` + that nothing is emitted.
- **Emit** — a valid fixture routes every field to the correct tree; the
  secrecy safety-check passes for a legal body and fails for an illegal one.
- **Duncarrow round-trip** (`test_duncarrow.py`) — the real acceptance test: the
  promoted `content/lore/duncarrow.md` builds green, and the emitted client/server
  JSON matches expectations (npc skeletons carry no persona; the `secret.crell-*`
  body lands only in `content/server/topics/`; `world/canon/` holds the public
  entities incl. the ladder `rungs`).

---

## Promotion checklist (Duncarrow — the acceptance run)

1. `git mv candidate-town.md content/lore/duncarrow.md`.
2. Reformat every fenced block to a ` ```yaml ` fence with valid YAML (fix the
   `knows` one-liners to `{fact, gate}`).
3. Drop the `knows: town.duncarrow` links from Mayor and Harn (L5a).
4. Rename `shrine.the-white-antlers` → `place.the-white-antlers` and update every
   `related` reference to it (id/type-agreement rule).
5. Add canon stubs for `region.the-tallow-reach` and `faction.elves` so all
   `related` ids resolve (closed-graph rule).
6. Drop `secret.crell-runs-slave-trade` from the **public** `person.mayor-oswin-crell`
   record's `related` — a public record must not point at a secret; that graph
   edge lives on the secret entry (server-side). The `npc.*` layer still holds the
   secret at `gate: never`, which is the actual bounded-move test.
7. Add `rungs: [hostile, cold, neutral, warm, trusted]` to
   `rule.disposition-ladder`.
8. Add `content/lore/canon-roadmap.md` (the authored-vs-pending index, schema D5).
9. Run the tool; fix validation errors; flip each entry `candidate → canon` as it
   passes.
10. Commit the promoted bible + built `content/world/**` + `content/server/**`.
11. `--check` is green in CI.

---

## Open / deferred (carried, not decided here)

- **Band numbers** — tunable placeholder in the client const; settle by playtest
  (schema D1). The build never sees them (L4a).
- **Runtime consumption / legacy Fenn migration / export packaging** — the scope
  boundary above; each is its own later cycle.
- **`make`/`just` wrapper** — a convenience target for the build command can be
  added when a task runner lands; not required for this cycle.
- **ids are self-describing (spoiler property, schema-level).** Per D4 the gate
  skeleton ships each NPC's `knows:[{fact_id, gate}]` to the client, so a
  descriptive slug like `secret.crell-runs-slave-trade` reaches the client binary
  even when held at `gate: never` (the body never does). This is a property of the
  **locked** schema, not the build tool — opaque/hashed topic ids are a possible
  future hardening if datamining becomes a concern. Noted, not reopened.