docs(spec): content & canon build tool design
Compiler for content/lore/*.md (MD bible + fenced YAML) -> validated content/world (client) + content/server (server-only) JSON. Python tool under tools/, manual build + CI --check, band-number-free (client owns the midpoint mapping), validation gate aborts on any schema violation. Promotes Duncarrow as the acceptance test; documents the specimen fixes promotion must apply (shrine->place rename, related stubs, dropped knows/related spoiler links). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HuHRPE7VfppUJEaoGBEUqZ
This commit is contained in:
@@ -0,0 +1,299 @@
|
||||
# Content & Canon Build Tool — Design
|
||||
|
||||
- **Date:** 2026-07-11
|
||||
- **Status:** Design approved; not yet planned/implemented.
|
||||
- **Implements:** the build tool whose *contract* was fixed by
|
||||
[`2026-07-11-content-canon-schema-design.md`](2026-07-11-content-canon-schema-design.md)
|
||||
(the locked schema — envelope, two categories, NPC layer, gate logic,
|
||||
role-based split, canon home, promotion/validation). **That schema is locked;
|
||||
this doc does not re-litigate it** — it specifies the compiler that realizes it.
|
||||
- **Specimen:** `candidate-town.md` (Duncarrow) at repo root — the bible the tool
|
||||
parses and, as its acceptance test, promotes.
|
||||
|
||||
---
|
||||
|
||||
## Purpose
|
||||
|
||||
Build the **compiler** that turns the authored Markdown bible
|
||||
(`content/lore/*.md`, prose + fenced `yaml` blocks) into validated runtime JSON,
|
||||
split by role into a client tree and a server-only tree, with validation as a
|
||||
hard gate. Promote Duncarrow as the tool's first real use and end-to-end proof.
|
||||
|
||||
**Charter alignment:** §2 (code owns state, AI owns text — this is a build-time
|
||||
transform, no runtime state), §4 (the server/ split keeps spoilers + personas off
|
||||
the client), §6 (the gate skeleton feeds `available_moves`), §16 (`/content`
|
||||
layout, "prompts/content are source, reviewed and versioned"), §18 (branch off
|
||||
`dev`).
|
||||
|
||||
---
|
||||
|
||||
## Locked decisions (from the brainstorm)
|
||||
|
||||
| # | Decision | Choice |
|
||||
|---|---|---|
|
||||
| L1 | Language / runtime | **Python**, standalone package under `tools/` — reuses the repo's Python + `jsonschema`/pytest stack; no Godot dependency to run a build; not bundled into the `api` proxy image. |
|
||||
| L2 | Output layout | **Per-category dirs** that fit the existing readers. Minimal, additive loader change; no per-town bundle rewrite. |
|
||||
| L3 | When it runs | **Manual command + CI `--check`.** Built JSON is committed; CI fails on stale-or-invalid. |
|
||||
| L4 | `start_disposition` → integer | **Band midpoint.** *Refinement (L4a):* only the client needs band integers at runtime, so the midpoint mapping lives client-side; the build stays **number-free** and emits the rung name. |
|
||||
| L5 | Promote Duncarrow this cycle | **Yes** — the tool's first real use and acceptance test. |
|
||||
|
||||
### Refinements settled in the brainstorm (explicitly approved)
|
||||
|
||||
- **L4a — the build needs zero disposition numbers.** The only runtime consumer
|
||||
of band *integers* is the client (it owns disposition state and computes
|
||||
`available_moves`). The build validates gate **names** only. So band numbers +
|
||||
the rung→midpoint mapping live in **one** client GDScript const (the
|
||||
`palette.gd` pattern); the build emits `start_disposition` as the **rung name**
|
||||
and passes it through. Legal rung names are read from a structured `rungs: [...]`
|
||||
field on `rule.disposition-ladder` in the bible — the bible owns names/order
|
||||
(D1), code owns numbers, and there is no cross-language table to keep in sync.
|
||||
- **L5a — the validator rejects `knows → canon-entity`.** Spec category A: canon
|
||||
entities never appear in a `knows` list. The specimen's Mayor and Harn carry
|
||||
`knows: town.duncarrow`; the validator enforces the rule and would reject them.
|
||||
Promotion **drops those links** (the town is public/tier-0, spoken freely, and
|
||||
needs no gated knows-link). This is intended — the validator catching real
|
||||
authoring drift.
|
||||
|
||||
---
|
||||
|
||||
## Scope boundary
|
||||
|
||||
**In scope (this cycle):**
|
||||
|
||||
- The `tools/content_build/` compiler: parse → resolve/validate → emit → split.
|
||||
- The validation gate (all rules below), abort-on-failure with source-line context.
|
||||
- A `--check` mode for CI (stale-or-invalid → nonzero exit).
|
||||
- Promoting Duncarrow: `candidate-town.md` → `content/lore/duncarrow.md`,
|
||||
reformat blocks to valid YAML, drop the `knows → canon-entity` links, add
|
||||
`rungs:` to the ladder, flip entries `candidate → canon`, add
|
||||
`content/lore/canon-roadmap.md`.
|
||||
- Committing the built `content/world/**` + `content/server/**` artifacts.
|
||||
- A **minimal, additive** `content_db.gd` change so the client loader tolerates
|
||||
the new `canon/` and `topics/` dirs (existing dirs + legacy `fenn.json`
|
||||
untouched).
|
||||
|
||||
**Out of scope (next cycle / deferred):**
|
||||
|
||||
- **Runtime consumption** — the client computing gated `available_moves` from the
|
||||
new npc skeletons, the api reading personas from `content/server/`, and voicing
|
||||
Mera Fenn end-to-end. Promotion proves the **compiler**, not the runtime.
|
||||
- **Legacy Fenn migration** — `content/world/npcs/fenn.json`'s old
|
||||
persona+knowledge+capabilities shape stays as-is; it is superseded when Fenn
|
||||
gets his own bible (flagged deferred in the schema doc).
|
||||
- **Godot export packaging** — deliberately including `world/` and excluding
|
||||
`server/` from the client export. `/content` already sits *outside* `res://`,
|
||||
so `server/` is out of the export by default; the packaging step is the
|
||||
pre-existing debt, **unblocked** by this clean split but not built here.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
`tools/content_build/` — a Python package, run as `python -m content_build`.
|
||||
|
||||
```
|
||||
tools/content_build/
|
||||
__init__.py
|
||||
__main__.py # CLI entry: build [--check] [--lore DIR] [--world DIR] [--server DIR]
|
||||
parse.py # Markdown → list of raw entries (dict + source line)
|
||||
model.py # Entry / NpcLayer / KnowsLink dataclasses; envelope + category rules
|
||||
resolve.py # id map, cross-ref resolution, the validation gate
|
||||
rungs.py # reads ladder `rungs:`; legal-gate set = rungs + {never}
|
||||
emit.py # route fields per D4; serialize the two JSON trees
|
||||
errors.py # BuildError carrying (entry id, source file:line, message)
|
||||
tests/
|
||||
fixtures/ # tiny valid + deliberately-broken bibles
|
||||
test_parse.py
|
||||
test_resolve.py # one test per gate rule (each broken fixture)
|
||||
test_emit.py # split routing + secrecy safety-check
|
||||
test_duncarrow.py # the real end-to-end round-trip
|
||||
```
|
||||
|
||||
Dependencies: `PyYAML` (block parsing). Added to a **dev** requirements file,
|
||||
not `api/requirements.txt` — the proxy runtime never imports the build tool.
|
||||
|
||||
Each module has one job and a narrow interface:
|
||||
|
||||
- **parse** — `parse_bible(path) -> list[RawEntry]`. Knows Markdown fences and
|
||||
YAML; knows nothing about the schema's meaning.
|
||||
- **model** — the dataclasses + envelope/category predicates. Pure data + rules,
|
||||
no I/O.
|
||||
- **resolve** — `resolve(entries) -> World | raises BuildError`. The gate. Pure
|
||||
function over parsed entries; no file writes.
|
||||
- **rungs** — the one place that knows the legal gate vocabulary, sourced from the
|
||||
ladder entry.
|
||||
- **emit** — `emit(world, world_dir, server_dir)`. Routing + serialization only.
|
||||
- **__main__** — orchestration + `--check` diffing + exit codes.
|
||||
|
||||
---
|
||||
|
||||
## Pipeline
|
||||
|
||||
### 1. Parse
|
||||
|
||||
For each `content/lore/*.md`: extract every ` ```yaml `-fenced block, `yaml.safe_load`
|
||||
each into a dict, attach source `file:line`. Prose between blocks is ignored. One
|
||||
block = one entry.
|
||||
|
||||
The promoted bible **must contain valid YAML** — the specimen's current
|
||||
`- fact: x gate: y` one-liners are *not* valid YAML and become
|
||||
`{fact: x, gate: y}` (or the two-line block form) during promotion (spec D3:
|
||||
"formalized as real YAML so parsing is trivial").
|
||||
|
||||
### 2. Resolve & validate — the gate
|
||||
|
||||
Build the id → entry map, then enforce every rule below. **Any failure aborts the
|
||||
build, emits nothing, and prints the offending id + source `file:line`.**
|
||||
|
||||
- **No id collision** — two entries with the same `id` → fail.
|
||||
- **id/type agreement** — the `id` prefix must be a **legal namespace** from the
|
||||
known set `{town, place, region, faction, person, rule, rumor, fact, secret,
|
||||
npc}`, and `type` must be **consistent** with it: prefix `npc` → `type` is
|
||||
`person` (the interactive layer vs the `person.*` world-record); otherwise the
|
||||
prefix equals `type`. A non-namespace prefix (e.g. the specimen's `shrine.`) →
|
||||
fail; such ids are renamed to their type namespace at promotion
|
||||
(`shrine.the-white-antlers` → `place.the-white-antlers`).
|
||||
- **`related` resolves** — every id in every `related` list is a real entry
|
||||
**defined in the built world** (strict, per schema §Build). The specimen
|
||||
references `region.the-tallow-reach` and `faction.elves` without defining them;
|
||||
promotion adds minimal canon stubs for both so the graph stays closed (there is
|
||||
no forward-ref-to-pending concept — the closed graph is the invariant).
|
||||
- **NPC-layer shape** — `npc.*` entries carry `knows` **and** `start_disposition`;
|
||||
`person.*` records carry **neither**.
|
||||
- **`knows` targets** — every `knows[].fact` resolves to a **knowledge entry**
|
||||
(`rumor`/`fact`/`secret`). A canon-entity id in a `knows` list → fail (L5a).
|
||||
- **Gate legality** — every `knows[].gate` ∈ `rungs` (read from
|
||||
`rule.disposition-ladder`) ∪ `{never}`.
|
||||
- **Ladder coverage** — exactly one `rule.disposition-ladder` entry exists and
|
||||
carries a non-empty `rungs:` list.
|
||||
- **Secrecy safety-check** — after routing (stage 4), no `secrecy ≥ 3` body may
|
||||
appear in any client (`content/world/**`) artifact → fail. Defense-in-depth
|
||||
behind the structural routing, not the router itself (schema D4).
|
||||
- **`start_disposition` rung legality** — the rung is one of `rungs`.
|
||||
|
||||
Contradiction-with-existing-canon is **author-asserted**, not auto-checked — it is
|
||||
resolved by the human at the `candidate → canon` promotion (schema §Build).
|
||||
|
||||
### 3. Emit
|
||||
|
||||
Only `status: canon` entries build into artifacts. `candidate` entries are
|
||||
dev-visible (they parse and validate) but are **not emitted** — with an
|
||||
all-`candidate` bible the artifacts are empty, which is why L5 promotes Duncarrow
|
||||
to `canon` this cycle.
|
||||
|
||||
### 4. Split (routing per schema D4)
|
||||
|
||||
**Client — `content/world/`** (bundled in the export; also readable by the server):
|
||||
|
||||
| Dir | Contents |
|
||||
|---|---|
|
||||
| `npcs/<slug>.json` | npc **skeleton**: `{id, type:"person", start_disposition:"<rung>", related, knows:[{fact_id, gate}]}` — no persona/body |
|
||||
| `canon/<slug>.json` | each public canon entity (`town/place/region/faction/person/rule`) **whole**, including its public `body` and (for the ladder) `rungs` |
|
||||
| `topics/<slug>.json` | knowledge-entry **skeleton**: `{id, type, related}` — **no body** |
|
||||
|
||||
**Server — `content/server/`** (server-only; excluded from the Godot export):
|
||||
|
||||
| Dir | Contents |
|
||||
|---|---|
|
||||
| `npcs/<slug>.json` | `{id, persona:<body>, disposition_notes}` |
|
||||
| `topics/<slug>.json` | `{id, body, secrecy}` — the knowledge bodies `/npc/speak` voices |
|
||||
|
||||
- Canon-entity `body` is **public** (category A, `secrecy:0`) → ships in
|
||||
`world/canon/`.
|
||||
- Knowledge-entry `body` and NPC persona/`disposition_notes` → `content/server/`
|
||||
only.
|
||||
- The server process reads **both** trees (it has full filesystem access); only
|
||||
the **client export** excludes `content/server/`.
|
||||
|
||||
`<slug>` = the id with its `<type>.`/`npc.` prefix stripped
|
||||
(`npc.mera-fenn` → `mera-fenn`, `town.duncarrow` → `duncarrow`).
|
||||
|
||||
---
|
||||
|
||||
## Client loader change (minimal, additive)
|
||||
|
||||
`client/scripts/content/content_db.gd` gains two dirs:
|
||||
|
||||
- `load_from()` also loads `world/canon/` and `world/topics/` into generic
|
||||
id-keyed dictionaries (`canon`, `topics`), mirroring the existing
|
||||
`_load_dir` pattern.
|
||||
- Existing `locations/npcs/quests/items` loading is untouched; the legacy
|
||||
`fenn.json` still loads (it has an `id`, which is all `_load_dir` requires).
|
||||
- Add `canon(id)` / `topic(id)` / `has_canon` / `has_topic` accessors.
|
||||
|
||||
No api change this cycle — runtime consumption of the new shapes is next cycle
|
||||
(scope boundary).
|
||||
|
||||
---
|
||||
|
||||
## CLI & CI
|
||||
|
||||
- `python -m content_build` — rebuild `content/world/**` + `content/server/**`
|
||||
from `content/lore/*.md`.
|
||||
- `python -m content_build --check` — build to memory, diff against the committed
|
||||
JSON, exit **nonzero** if the tree is **stale** (would-differ) **or** validation
|
||||
fails. This is the CI gate.
|
||||
- Dir overrides (`--lore/--world/--server`) exist for tests to run against
|
||||
fixtures in a temp dir.
|
||||
|
||||
Built JSON is **committed** (schema/prompt "content is source, reviewed" culture);
|
||||
`--check` keeps the committed artifacts honest.
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
- **Parse** — a fixture bible with prose + multiple `yaml` blocks parses to the
|
||||
expected entry list with source lines.
|
||||
- **Resolve** — **one deliberately-broken fixture per gate rule** (id collision,
|
||||
bad id/type, dangling `related`, `person.*` with `knows`, `knows → canon-entity`,
|
||||
illegal gate, missing ladder, `secrecy ≥ 3` body routed to client), each
|
||||
asserting the specific `BuildError` + that nothing is emitted.
|
||||
- **Emit** — a valid fixture routes every field to the correct tree; the
|
||||
secrecy safety-check passes for a legal body and fails for an illegal one.
|
||||
- **Duncarrow round-trip** (`test_duncarrow.py`) — the real acceptance test: the
|
||||
promoted `content/lore/duncarrow.md` builds green, and the emitted client/server
|
||||
JSON matches expectations (npc skeletons carry no persona; the `secret.crell-*`
|
||||
body lands only in `content/server/topics/`; `world/canon/` holds the public
|
||||
entities incl. the ladder `rungs`).
|
||||
|
||||
---
|
||||
|
||||
## Promotion checklist (Duncarrow — the acceptance run)
|
||||
|
||||
1. `git mv candidate-town.md content/lore/duncarrow.md`.
|
||||
2. Reformat every fenced block to a ` ```yaml ` fence with valid YAML (fix the
|
||||
`knows` one-liners to `{fact, gate}`).
|
||||
3. Drop the `knows: town.duncarrow` links from Mayor and Harn (L5a).
|
||||
4. Rename `shrine.the-white-antlers` → `place.the-white-antlers` and update every
|
||||
`related` reference to it (id/type-agreement rule).
|
||||
5. Add canon stubs for `region.the-tallow-reach` and `faction.elves` so all
|
||||
`related` ids resolve (closed-graph rule).
|
||||
6. Drop `secret.crell-runs-slave-trade` from the **public** `person.mayor-oswin-crell`
|
||||
record's `related` — a public record must not point at a secret; that graph
|
||||
edge lives on the secret entry (server-side). The `npc.*` layer still holds the
|
||||
secret at `gate: never`, which is the actual bounded-move test.
|
||||
7. Add `rungs: [hostile, cold, neutral, warm, trusted]` to
|
||||
`rule.disposition-ladder`.
|
||||
8. Add `content/lore/canon-roadmap.md` (the authored-vs-pending index, schema D5).
|
||||
9. Run the tool; fix validation errors; flip each entry `candidate → canon` as it
|
||||
passes.
|
||||
10. Commit the promoted bible + built `content/world/**` + `content/server/**`.
|
||||
11. `--check` is green in CI.
|
||||
|
||||
---
|
||||
|
||||
## Open / deferred (carried, not decided here)
|
||||
|
||||
- **Band numbers** — tunable placeholder in the client const; settle by playtest
|
||||
(schema D1). The build never sees them (L4a).
|
||||
- **Runtime consumption / legacy Fenn migration / export packaging** — the scope
|
||||
boundary above; each is its own later cycle.
|
||||
- **`make`/`just` wrapper** — a convenience target for the build command can be
|
||||
added when a task runner lands; not required for this cycle.
|
||||
- **ids are self-describing (spoiler property, schema-level).** Per D4 the gate
|
||||
skeleton ships each NPC's `knows:[{fact_id, gate}]` to the client, so a
|
||||
descriptive slug like `secret.crell-runs-slave-trade` reaches the client binary
|
||||
even when held at `gate: never` (the body never does). This is a property of the
|
||||
**locked** schema, not the build tool — opaque/hashed topic ids are a possible
|
||||
future hardening if datamining becomes a concern. Noted, not reopened.
|
||||
Reference in New Issue
Block a user