# Content & Canon Build Tool Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Build a Python compiler that turns the authored Markdown bible (`content/lore/*.md`, prose + fenced `yaml` blocks) into validated, role-split runtime JSON (`content/world` client tree + `content/server` server-only tree), and promote Duncarrow as its acceptance test. **Architecture:** A standalone `tools/content_build/` package with a four-stage pipeline — parse (Markdown → yaml blocks) → resolve/validate (a hard gate that aborts on any schema violation) → emit (route each field per the schema's role split) → write. Run manually (`python -m content_build`) or as a CI gate (`--check`, fails on stale-or-invalid). The build carries **no disposition numbers** — it validates rung *names* only; the client owns the band integers and the rung→midpoint mapping. **Tech Stack:** Python 3.12, PyYAML (block parsing), pytest (TDD). No new runtime deps for the `api` proxy — the tool is dev/CI only. GDScript for the one client-loader change. ## Global Constraints - **Locked schema — do not re-litigate.** The contract is fixed by `docs/superpowers/specs/2026-07-11-content-canon-schema-design.md`; this plan realizes `docs/superpowers/specs/2026-07-11-content-canon-build-tool-design.md`. - **Charter §18 git flow:** this is non-doc code → all work on a branch off `dev` (`feature/content-build-tool`). Never commit to `master`. Doc/plan commits may land on `dev`. The human owns `dev → master`. - **The build tool must not be importable by / bundled into the `api` proxy** (it ships to fly.io). It lives under `tools/`, never under `api/`. - **Bands live in code, not the bible; the build never hardcodes a band integer.** Rung *names/order* come from the bible's `rule.disposition-ladder` `rungs:` field. - **Only `status: canon` entries emit into shipping artifacts.** `candidate` entries still parse + validate (authors get errors early) but are not emitted. - **Built JSON is committed.** Deterministic serialization (`json.dumps(obj, indent=2, sort_keys=True) + "\n"`) so `--check` diffs are stable. - **The build never clears `content/world/npcs/`** — it is shared with legacy hand-authored NPCs (`fenn.json`, `cadwyn_vell.json`, `brannoc_thane.json`). The build overwrites only files it generates. - **All commands run from the repo root**, using the repo venv and `PYTHONPATH=tools`: - Tests: `PYTHONPATH=tools .venv/bin/python -m pytest tools/content_build/tests -q` - Build: `PYTHONPATH=tools .venv/bin/python -m content_build` - Check: `PYTHONPATH=tools .venv/bin/python -m content_build --check` --- ## File Structure ``` tools/ requirements-dev.txt # PyYAML + pytest pins (dev/CI only) content_build/ __init__.py __main__.py # CLI: build / --check, orchestration, exit codes errors.py # BuildError(message, source, entry_id) parse.py # parse_bible(path) -> list[RawEntry] model.py # Entry / KnowsLink dataclasses + entry_from_raw + predicates rungs.py # ladder_rungs(), legal_gates() — the rung vocabulary resolve.py # resolve(entries) -> World (the validation gate) emit.py # build_trees / check_secrecy / write_trees / dump_json tests/ __init__.py fixtures/valid/world.md # a minimal valid bible for emit + cli tests test_parse.py test_model.py test_resolve.py # one test per gate rule test_emit.py test_cli.py test_duncarrow.py # real end-to-end round-trip (after promotion) content/ lore/duncarrow.md # promoted from candidate-town.md (Task 6) lore/canon-roadmap.md # authored-vs-pending index (Task 6) world/{canon,topics}/*.json # BUILT (Task 6 output, committed) world/npcs/*.json # BUILT skeletons alongside legacy files server/{npcs,topics}/*.json # BUILT server-only (Task 6 output, committed) client/scripts/content/content_db.gd # +canon/ +topics/ dirs (Task 7) client/tests/unit/test_content_db.gd # +loader tests (Task 7) .github/workflows/content-build.yml # CI gate (Task 8) ``` --- ### Task 1: Package scaffold, errors, and the Markdown parser **Files:** - Create: `tools/requirements-dev.txt` - Create: `tools/content_build/__init__.py` (empty) - Create: `tools/content_build/errors.py` - Create: `tools/content_build/parse.py` - Create: `tools/content_build/tests/__init__.py` (empty) - Test: `tools/content_build/tests/test_parse.py` **Interfaces:** - Produces: `BuildError(message: str, source: str = "", entry_id: str = "")` with `.message/.source/.entry_id` attrs; `RawEntry` dataclass `{data: dict, source: str}`; `parse_bible(path: Path) -> list[RawEntry]`. - [ ] **Step 1: Create the branch** ```bash git checkout dev && git pull --ff-only 2>/dev/null; git checkout -b feature/content-build-tool ``` - [ ] **Step 2: Write the dev requirements and empty package files** `tools/requirements-dev.txt`: ``` # Build-tool dev/CI deps. NOT installed into the api proxy runtime. PyYAML==6.0.3 pytest==8.3.2 ``` Create empty `tools/content_build/__init__.py` and `tools/content_build/tests/__init__.py`: ```bash mkdir -p tools/content_build/tests/fixtures/valid : > tools/content_build/__init__.py : > tools/content_build/tests/__init__.py ``` - [ ] **Step 3: Write `errors.py`** `tools/content_build/errors.py`: ```python """The single failure type. Every validation/parse problem raises BuildError so the CLI can print a uniform 'BUILD FAILED: [] ()'.""" class BuildError(Exception): def __init__(self, message: str, source: str = "", entry_id: str = ""): self.message = message self.source = source self.entry_id = entry_id loc = f" [{entry_id}]" if entry_id else "" src = f" ({source})" if source else "" super().__init__(f"{message}{loc}{src}") ``` - [ ] **Step 4: Write the failing parser test** `tools/content_build/tests/test_parse.py`: ```python from pathlib import Path import pytest from content_build.errors import BuildError from content_build.parse import parse_bible BIBLE = """# A bible Some prose that must be ignored. ```yaml id: town.a type: town status: canon secrecy: 0 related: [] body: > Town A. ``` More prose. ```yaml id: rumor.x type: rumor status: canon secrecy: 1 related: [town.a] body: > A rumor. ``` """ def _write(tmp_path: Path, text: str) -> Path: p = tmp_path / "bible.md" p.write_text(text) return p def test_extracts_yaml_blocks_and_ignores_prose(tmp_path): entries = parse_bible(_write(tmp_path, BIBLE)) assert [e.data["id"] for e in entries] == ["town.a", "rumor.x"] def test_source_carries_filename_and_line(tmp_path): entries = parse_bible(_write(tmp_path, BIBLE)) assert entries[0].source.startswith("bible.md:") def test_unterminated_block_raises(tmp_path): bad = "```yaml\nid: town.a\n" # no closing fence with pytest.raises(BuildError): parse_bible(_write(tmp_path, bad)) def test_non_mapping_block_raises(tmp_path): bad = "```yaml\n- just\n- a\n- list\n```\n" with pytest.raises(BuildError): parse_bible(_write(tmp_path, bad)) ``` - [ ] **Step 5: Run the test to verify it fails** Run: `PYTHONPATH=tools .venv/bin/python -m pytest tools/content_build/tests/test_parse.py -q` Expected: FAIL — `ModuleNotFoundError: No module named 'content_build.parse'`. - [ ] **Step 6: Write `parse.py`** `tools/content_build/parse.py`: ```python """Markdown bible -> raw yaml-block entries. Knows only fences + YAML; nothing about the schema's meaning. A block is opened by a line that is exactly ```yaml (after strip) and closed by a line that is exactly ```.""" from dataclasses import dataclass from pathlib import Path import yaml from .errors import BuildError @dataclass class RawEntry: data: dict source: str # "duncarrow.md:53" def parse_bible(path: Path) -> list[RawEntry]: lines = path.read_text().splitlines() entries: list[RawEntry] = [] i, n = 0, len(lines) while i < n: if lines[i].strip() == "```yaml": fence_line = i + 1 # 1-based line of the opening fence i += 1 block: list[str] = [] while i < n and lines[i].strip() != "```": block.append(lines[i]) i += 1 if i >= n: raise BuildError("unterminated ```yaml block", source=f"{path.name}:{fence_line}") data = yaml.safe_load("\n".join(block)) if not isinstance(data, dict): raise BuildError("yaml block is not a mapping", source=f"{path.name}:{fence_line}") entries.append(RawEntry(data=data, source=f"{path.name}:{fence_line}")) i += 1 return entries ``` - [ ] **Step 7: Run the test to verify it passes** Run: `PYTHONPATH=tools .venv/bin/python -m pytest tools/content_build/tests/test_parse.py -q` Expected: PASS (4 passed). - [ ] **Step 8: Commit** ```bash git add tools/requirements-dev.txt tools/content_build/__init__.py tools/content_build/errors.py tools/content_build/parse.py tools/content_build/tests/__init__.py tools/content_build/tests/test_parse.py git commit -m "feat(content-build): scaffold package + Markdown yaml-block parser" ``` --- ### Task 2: The entry model **Files:** - Create: `tools/content_build/model.py` - Test: `tools/content_build/tests/test_model.py` **Interfaces:** - Consumes: `RawEntry` (Task 1). - Produces: - Constants `CANON_TYPES`, `KNOWLEDGE_TYPES`, `LEGAL_NAMESPACES`, `STATUSES` (all `set[str]`). - `KnowsLink` dataclass `{fact_id: str, gate: str}`. - `Entry` dataclass with fields `id, type, status, secrecy, related, body, source, start_disposition, knows (list[KnowsLink]), disposition_notes, rungs` and read-only properties `namespace, slug, is_npc_layer, is_knowledge, is_canon_entity`. - `entry_from_raw(raw: RawEntry) -> Entry`. - [ ] **Step 1: Write the failing model test** `tools/content_build/tests/test_model.py`: ```python import pytest from content_build.errors import BuildError from content_build.model import Entry, entry_from_raw from content_build.parse import RawEntry def _raw(**over) -> RawEntry: data = {"id": "town.a", "type": "town", "status": "canon", "secrecy": 0, "related": []} data.update(over) return RawEntry(data=data, source="t:1") def test_namespace_and_slug(): e = entry_from_raw(_raw(id="npc.mera-fenn", type="person", start_disposition="cold", knows=[{"fact": "rumor.x", "gate": "warm"}])) assert e.namespace == "npc" assert e.slug == "mera-fenn" def test_category_predicates(): npc = entry_from_raw(_raw(id="npc.t", type="person", start_disposition="cold", knows=[{"fact": "rumor.x", "gate": "warm"}])) person = entry_from_raw(_raw(id="person.p", type="person")) rumor = entry_from_raw(_raw(id="rumor.x", type="rumor", secrecy=1)) assert npc.is_npc_layer and not npc.is_canon_entity assert person.is_canon_entity and not person.is_npc_layer assert rumor.is_knowledge and not rumor.is_canon_entity def test_knows_parsed_into_links(): e = entry_from_raw(_raw(id="npc.t", type="person", start_disposition="cold", knows=[{"fact": "rumor.x", "gate": "neutral"}])) assert e.knows[0].fact_id == "rumor.x" assert e.knows[0].gate == "neutral" def test_missing_required_field_raises(): bad = RawEntry(data={"id": "town.a", "type": "town"}, source="t:1") with pytest.raises(BuildError): entry_from_raw(bad) def test_knows_missing_gate_raises(): with pytest.raises(BuildError): entry_from_raw(_raw(id="npc.t", type="person", start_disposition="cold", knows=[{"fact": "rumor.x"}])) ``` - [ ] **Step 2: Run to verify it fails** Run: `PYTHONPATH=tools .venv/bin/python -m pytest tools/content_build/tests/test_model.py -q` Expected: FAIL — `No module named 'content_build.model'`. - [ ] **Step 3: Write `model.py`** `tools/content_build/model.py`: ```python """Typed entries + the envelope/category rules. Pure data; no I/O. The npc interactive layer vs the person world-record is distinguished by id NAMESPACE (prefix), not by `type` (both are type 'person').""" from dataclasses import dataclass, field from .errors import BuildError from .parse import RawEntry CANON_TYPES = {"town", "place", "region", "faction", "person", "rule"} KNOWLEDGE_TYPES = {"rumor", "fact", "secret"} LEGAL_NAMESPACES = {"town", "place", "region", "faction", "person", "rule", "rumor", "fact", "secret", "npc"} STATUSES = {"candidate", "canon"} @dataclass class KnowsLink: fact_id: str gate: str @dataclass class Entry: id: str type: str status: str secrecy: int related: list body: str | None source: str start_disposition: str | None = None knows: list = field(default_factory=list) # list[KnowsLink] disposition_notes: str | None = None rungs: list | None = None @property def namespace(self) -> str: return self.id.split(".", 1)[0] @property def slug(self) -> str: parts = self.id.split(".", 1) return parts[1] if len(parts) == 2 else parts[0] @property def is_npc_layer(self) -> bool: return self.namespace == "npc" @property def is_knowledge(self) -> bool: return self.type in KNOWLEDGE_TYPES @property def is_canon_entity(self) -> bool: return (not self.is_npc_layer) and self.type in CANON_TYPES def entry_from_raw(raw: RawEntry) -> Entry: d = raw.data for key in ("id", "type", "status", "secrecy", "related"): if key not in d: raise BuildError(f"missing required field '{key}'", source=raw.source, entry_id=d.get("id", "")) knows: list[KnowsLink] = [] for k in d.get("knows", []): if "fact" not in k or "gate" not in k: raise BuildError("knows entry needs 'fact' and 'gate'", source=raw.source, entry_id=d.get("id", "")) knows.append(KnowsLink(fact_id=k["fact"], gate=k["gate"])) return Entry( id=d["id"], type=d["type"], status=d["status"], secrecy=d["secrecy"], related=list(d["related"]), body=d.get("body"), source=raw.source, start_disposition=d.get("start_disposition"), knows=knows, disposition_notes=d.get("disposition_notes"), rungs=d.get("rungs"), ) ``` - [ ] **Step 4: Run to verify it passes** Run: `PYTHONPATH=tools .venv/bin/python -m pytest tools/content_build/tests/test_model.py -q` Expected: PASS (5 passed). - [ ] **Step 5: Commit** ```bash git add tools/content_build/model.py tools/content_build/tests/test_model.py git commit -m "feat(content-build): typed Entry model + category predicates" ``` --- ### Task 3: The validation gate (`rungs.py` + `resolve.py`) **Files:** - Create: `tools/content_build/rungs.py` - Create: `tools/content_build/resolve.py` - Test: `tools/content_build/tests/test_resolve.py` **Interfaces:** - Consumes: `Entry`, `KnowsLink`, category constants (Task 2). - Produces: - `rungs.py`: `LADDER_ID = "rule.disposition-ladder"`, `NEVER = "never"`, `ladder_rungs(entries) -> list[str]`, `legal_gates(entries) -> set[str]`. - `resolve.py`: `World` dataclass `{entries: list[Entry], by_id: dict[str, Entry]}`; `resolve(entries: list[Entry]) -> World` (raises `BuildError` on the first violation). - [ ] **Step 1: Write the failing resolve tests (one per rule)** `tools/content_build/tests/test_resolve.py`: ```python import pytest from content_build.errors import BuildError from content_build.model import Entry, KnowsLink from content_build.resolve import resolve def ladder(): return Entry(id="rule.disposition-ladder", type="rule", status="canon", secrecy=0, related=[], body="x", source="t:1", rungs=["hostile", "cold", "neutral", "warm", "trusted"]) def town(): return Entry(id="town.a", type="town", status="canon", secrecy=0, related=[], body="A town.", source="t:2") def rumor(): return Entry(id="rumor.x", type="rumor", status="canon", secrecy=1, related=["town.a"], body="A rumor.", source="t:3") def npc(**over): base = dict(id="npc.t", type="person", status="canon", secrecy=0, related=["town.a"], body="Persona.", source="t:4", start_disposition="cold", knows=[KnowsLink("rumor.x", "neutral")]) base.update(over) return Entry(**base) def test_valid_world_resolves(): world = resolve([ladder(), town(), rumor(), npc()]) assert world.by_id["npc.t"].id == "npc.t" def test_duplicate_id(): with pytest.raises(BuildError): resolve([ladder(), town(), town()]) def test_illegal_namespace(): bad = Entry(id="shrine.s", type="place", status="canon", secrecy=0, related=[], body="x", source="t:9") with pytest.raises(BuildError): resolve([ladder(), bad]) def test_id_type_mismatch(): bad = Entry(id="place.s", type="town", status="canon", secrecy=0, related=[], body="x", source="t:9") with pytest.raises(BuildError): resolve([ladder(), bad]) def test_npc_type_must_be_person(): with pytest.raises(BuildError): resolve([ladder(), town(), rumor(), npc(type="town")]) def test_dangling_related(): with pytest.raises(BuildError): resolve([ladder(), town(), rumor(), npc(related=["town.nowhere"])]) def test_person_record_with_knows_rejected(): rec = Entry(id="person.p", type="person", status="canon", secrecy=0, related=[], body="x", source="t:9", knows=[KnowsLink("rumor.x", "warm")]) with pytest.raises(BuildError): resolve([ladder(), town(), rumor(), rec]) def test_npc_missing_start_disposition(): with pytest.raises(BuildError): resolve([ladder(), town(), rumor(), npc(start_disposition=None)]) def test_knows_target_must_be_knowledge_entry(): # points at the town (a canon entity), not a knowledge entry with pytest.raises(BuildError): resolve([ladder(), town(), rumor(), npc(knows=[KnowsLink("town.a", "cold")])]) def test_illegal_gate(): with pytest.raises(BuildError): resolve([ladder(), town(), rumor(), npc(knows=[KnowsLink("rumor.x", "besties")])]) def test_never_is_a_legal_gate(): world = resolve([ladder(), town(), rumor(), npc(knows=[KnowsLink("rumor.x", "never")])]) assert world.by_id["npc.t"].knows[0].gate == "never" def test_missing_ladder(): with pytest.raises(BuildError): resolve([town()]) def test_illegal_start_disposition(): with pytest.raises(BuildError): resolve([ladder(), town(), rumor(), npc(start_disposition="besties")]) ``` - [ ] **Step 2: Run to verify it fails** Run: `PYTHONPATH=tools .venv/bin/python -m pytest tools/content_build/tests/test_resolve.py -q` Expected: FAIL — `No module named 'content_build.resolve'`. - [ ] **Step 3: Write `rungs.py`** `tools/content_build/rungs.py`: ```python """The rung vocabulary, sourced from the bible's disposition-ladder entry. The bible owns rung names/order; band NUMBERS live in client code and never here.""" from .errors import BuildError from .model import Entry LADDER_ID = "rule.disposition-ladder" NEVER = "never" def ladder_rungs(entries: list[Entry]) -> list[str]: ladders = [e for e in entries if e.id == LADDER_ID] if len(ladders) != 1: raise BuildError( f"exactly one '{LADDER_ID}' entry required, found {len(ladders)}") if not ladders[0].rungs: raise BuildError("disposition-ladder needs a non-empty 'rungs:' list", source=ladders[0].source, entry_id=LADDER_ID) return list(ladders[0].rungs) def legal_gates(entries: list[Entry]) -> set[str]: return set(ladder_rungs(entries)) | {NEVER} ``` - [ ] **Step 4: Write `resolve.py`** `tools/content_build/resolve.py`: ```python """The validation gate. Pure function over parsed entries: builds the id map and enforces every schema rule, raising BuildError on the first violation. Emits nothing. Validates ALL entries (candidate + canon) so authors get errors early; emit (Task 4) filters to canon.""" from dataclasses import dataclass from .errors import BuildError from .model import Entry, KNOWLEDGE_TYPES, LEGAL_NAMESPACES, STATUSES from .rungs import ladder_rungs, legal_gates @dataclass class World: entries: list # list[Entry] by_id: dict # id -> Entry def resolve(entries: list[Entry]) -> World: by_id: dict[str, Entry] = {} for e in entries: if e.id in by_id: raise BuildError("duplicate id", source=e.source, entry_id=e.id) by_id[e.id] = e for e in entries: if e.status not in STATUSES: raise BuildError(f"illegal status '{e.status}'", source=e.source, entry_id=e.id) if e.namespace not in LEGAL_NAMESPACES: raise BuildError(f"illegal id namespace '{e.namespace}'", source=e.source, entry_id=e.id) if e.namespace == "npc": if e.type != "person": raise BuildError("npc.* entries must have type 'person'", source=e.source, entry_id=e.id) elif e.type != e.namespace: raise BuildError( f"id namespace '{e.namespace}' must equal type '{e.type}'", source=e.source, entry_id=e.id) for e in entries: for rid in e.related: if rid not in by_id: raise BuildError(f"related id '{rid}' does not resolve", source=e.source, entry_id=e.id) for e in entries: if e.is_npc_layer: if e.start_disposition is None: raise BuildError("npc.* requires start_disposition", source=e.source, entry_id=e.id) if not e.knows: raise BuildError("npc.* requires a non-empty knows list", source=e.source, entry_id=e.id) else: if e.knows: raise BuildError("non-npc entry must not carry a knows list", source=e.source, entry_id=e.id) if e.start_disposition is not None: raise BuildError("non-npc entry must not carry start_disposition", source=e.source, entry_id=e.id) gates = legal_gates(entries) # enforces the single ladder + rungs rungs = set(ladder_rungs(entries)) for e in entries: if not e.is_npc_layer: continue if e.start_disposition not in rungs: raise BuildError( f"start_disposition '{e.start_disposition}' is not a legal rung", source=e.source, entry_id=e.id) for link in e.knows: if link.fact_id not in by_id: raise BuildError(f"knows fact '{link.fact_id}' does not resolve", source=e.source, entry_id=e.id) target = by_id[link.fact_id] if target.type not in KNOWLEDGE_TYPES: raise BuildError( f"knows target '{link.fact_id}' is not a knowledge entry " f"(is '{target.type}')", source=e.source, entry_id=e.id) if link.gate not in gates: raise BuildError(f"illegal gate '{link.gate}'", source=e.source, entry_id=e.id) return World(entries=entries, by_id=by_id) ``` - [ ] **Step 5: Run to verify it passes** Run: `PYTHONPATH=tools .venv/bin/python -m pytest tools/content_build/tests/test_resolve.py -q` Expected: PASS (13 passed). - [ ] **Step 6: Commit** ```bash git add tools/content_build/rungs.py tools/content_build/resolve.py tools/content_build/tests/test_resolve.py git commit -m "feat(content-build): validation gate (rungs + resolve)" ``` **Note (accepted this cycle):** cross-status `related` integrity (a `canon` entry referencing a `candidate` entry, which would emit a dangling reference) is **not** checked — Duncarrow promotes atomically (all entries flip together). Revisit if a bible ever ships mixed-status. --- ### Task 4: Emit — routing, secrecy safety-check, write **Files:** - Create: `tools/content_build/emit.py` - Create: `tools/content_build/tests/fixtures/valid/world.md` - Test: `tools/content_build/tests/test_emit.py` **Interfaces:** - Consumes: `World` (Task 3), `Entry`, `LADDER_ID` (Task 3). - Produces: - `dump_json(obj) -> str` — `json.dumps(obj, indent=2, sort_keys=True) + "\n"`. - `build_trees(world: World) -> tuple[dict, dict]` — `(client_files, server_files)`, each `{relpath: json_obj}` over **canon-status** entries only. - `check_secrecy(client_files: dict, world: World) -> None` — raises `BuildError` if any client record carrying a `body` originates from a `secrecy >= 3` entry. - `write_trees(client_files, server_files, world_dir: Path, server_dir: Path) -> None`. - [ ] **Step 1: Write the valid fixture bible** `tools/content_build/tests/fixtures/valid/world.md`: ```markdown # Test bible ```yaml id: rule.disposition-ladder type: rule status: canon secrecy: 0 related: [] rungs: [hostile, cold, neutral, warm, trusted] body: > The five-rung ladder. ``` ```yaml id: town.testburg type: town status: canon secrecy: 0 related: [] body: > A test town. ``` ```yaml id: rumor.something type: rumor status: canon secrecy: 1 related: [town.testburg] body: > A rumor body. ``` ```yaml id: secret.big-twist type: secret status: canon secrecy: 4 related: [town.testburg] body: > The twist body. Server-only. ``` ```yaml id: npc.tess type: person status: canon secrecy: 0 start_disposition: cold related: [town.testburg] body: > Tess persona. knows: - {fact: rumor.something, gate: neutral} - {fact: secret.big-twist, gate: never} disposition_notes: > Author notes. ``` ``` - [ ] **Step 2: Write the failing emit test** `tools/content_build/tests/test_emit.py`: ```python from pathlib import Path import pytest from content_build.emit import build_trees, check_secrecy, write_trees, dump_json from content_build.model import entry_from_raw from content_build.parse import parse_bible from content_build.resolve import resolve from content_build.errors import BuildError FIXTURE = Path(__file__).parent / "fixtures" / "valid" / "world.md" def _world(): raws = parse_bible(FIXTURE) return resolve([entry_from_raw(r) for r in raws]) def test_npc_skeleton_has_no_persona_client_side(): client, _ = build_trees(_world()) npc = client["npcs/tess.json"] assert "persona" not in npc and "body" not in npc assert npc["start_disposition"] == "cold" assert {"fact_id": "secret.big-twist", "gate": "never"} in npc["knows"] def test_persona_goes_server_side(): _, server = build_trees(_world()) assert server["npcs/tess.json"]["persona"].startswith("Tess") def test_secret_body_only_server_side(): client, server = build_trees(_world()) assert "topics/big-twist.json" in server assert server["topics/big-twist.json"]["body"].startswith("The twist") # client topic skeleton carries no body assert "body" not in client["topics/big-twist.json"] def test_canon_entity_body_ships_client_side(): client, _ = build_trees(_world()) assert client["canon/testburg.json"]["body"].startswith("A test town") assert client["canon/disposition-ladder.json"]["rungs"][0] == "hostile" def test_check_secrecy_passes_on_valid_world(): client, _ = build_trees(_world()) check_secrecy(client, _world()) # no raise def test_check_secrecy_fails_when_secret_body_reaches_client(): world = _world() client, _ = build_trees(world) # simulate a routing regression: a secrecy-4 body in a client record client["canon/leak.json"] = {"id": "secret.big-twist", "body": "leak"} with pytest.raises(BuildError): check_secrecy(client, world) def test_write_trees_serializes_deterministically(tmp_path): client, server = build_trees(_world()) write_trees(client, server, tmp_path / "world", tmp_path / "server") written = (tmp_path / "world" / "npcs" / "tess.json").read_text() assert written == dump_json(client["npcs/tess.json"]) ``` - [ ] **Step 3: Run to verify it fails** Run: `PYTHONPATH=tools .venv/bin/python -m pytest tools/content_build/tests/test_emit.py -q` Expected: FAIL — `No module named 'content_build.emit'`. - [ ] **Step 4: Write `emit.py`** `tools/content_build/emit.py`: ```python """Route each canon entry's fields into the client tree (content/world) and the server-only tree (content/server), per the schema's role split. build_trees is pure (returns dicts); write_trees does the I/O; check_secrecy is the after-routing defense-in-depth lint.""" import json from pathlib import Path from .errors import BuildError from .model import Entry from .resolve import World from .rungs import LADDER_ID def dump_json(obj) -> str: return json.dumps(obj, indent=2, sort_keys=True) + "\n" def _client_relpath(e: Entry) -> str: if e.is_npc_layer: return f"npcs/{e.slug}.json" if e.is_knowledge: return f"topics/{e.slug}.json" return f"canon/{e.slug}.json" def _client_record(e: Entry) -> dict: if e.is_npc_layer: return { "id": e.id, "type": e.type, "start_disposition": e.start_disposition, "related": e.related, "knows": [{"fact_id": k.fact_id, "gate": k.gate} for k in e.knows], } if e.is_knowledge: return {"id": e.id, "type": e.type, "related": e.related} rec = {"id": e.id, "type": e.type, "related": e.related} if e.body is not None: rec["body"] = e.body if e.id == LADDER_ID and e.rungs: rec["rungs"] = e.rungs return rec def _server_entry(e: Entry): """Return (relpath, record) for the server tree, or None for canon entities (their bodies are public and ship client-side).""" if e.is_npc_layer: rec = {"id": e.id, "persona": e.body} if e.disposition_notes is not None: rec["disposition_notes"] = e.disposition_notes return f"npcs/{e.slug}.json", rec if e.is_knowledge: return f"topics/{e.slug}.json", {"id": e.id, "body": e.body, "secrecy": e.secrecy} return None def build_trees(world: World) -> tuple[dict, dict]: client: dict[str, dict] = {} server: dict[str, dict] = {} for e in world.entries: if e.status != "canon": continue client[_client_relpath(e)] = _client_record(e) se = _server_entry(e) if se is not None: relpath, rec = se server[relpath] = rec return client, server def check_secrecy(client_files: dict, world: World) -> None: for relpath, rec in client_files.items(): if "body" in rec and world.by_id[rec["id"]].secrecy >= 3: e = world.by_id[rec["id"]] raise BuildError( f"secrecy {e.secrecy} body routed to client artifact {relpath}", source=e.source, entry_id=e.id) def write_trees(client_files: dict, server_files: dict, world_dir: Path, server_dir: Path) -> None: for relpath, obj in client_files.items(): p = Path(world_dir) / relpath p.parent.mkdir(parents=True, exist_ok=True) p.write_text(dump_json(obj)) for relpath, obj in server_files.items(): p = Path(server_dir) / relpath p.parent.mkdir(parents=True, exist_ok=True) p.write_text(dump_json(obj)) ``` - [ ] **Step 5: Run to verify it passes** Run: `PYTHONPATH=tools .venv/bin/python -m pytest tools/content_build/tests/test_emit.py -q` Expected: PASS (7 passed). - [ ] **Step 6: Commit** ```bash git add tools/content_build/emit.py tools/content_build/tests/fixtures/valid/world.md tools/content_build/tests/test_emit.py git commit -m "feat(content-build): emit routing + secrecy lint + writer" ``` --- ### Task 5: CLI orchestration and `--check` **Files:** - Create: `tools/content_build/__main__.py` - Test: `tools/content_build/tests/test_cli.py` **Interfaces:** - Consumes: `parse_bible`, `entry_from_raw`, `resolve`, `build_trees`, `check_secrecy`, `write_trees`, `dump_json` (Tasks 1–4). - Produces: `main(argv=None) -> int`; helpers `build(lore, world, server) -> None`, `check(lore, world, server) -> int`. CLI: `--check`, `--lore`, `--world`, `--server` (defaults `content/lore`, `content/world`, `content/server`, repo-root-relative). Exit 0 = ok, 1 = build failed or stale. - [ ] **Step 1: Write the failing CLI test** `tools/content_build/tests/test_cli.py`: ```python import shutil from pathlib import Path from content_build.__main__ import main FIXTURE_DIR = Path(__file__).parent / "fixtures" / "valid" def _lore(tmp_path): lore = tmp_path / "lore" lore.mkdir() shutil.copy(FIXTURE_DIR / "world.md", lore / "world.md") return lore def test_build_then_check_is_clean(tmp_path): lore = _lore(tmp_path) world, server = tmp_path / "world", tmp_path / "server" argv = ["--lore", str(lore), "--world", str(world), "--server", str(server)] assert main(argv) == 0 assert (world / "npcs" / "tess.json").exists() assert (server / "topics" / "big-twist.json").exists() # a fresh --check against just-written output is clean assert main(argv + ["--check"]) == 0 def test_check_detects_stale(tmp_path): lore = _lore(tmp_path) world, server = tmp_path / "world", tmp_path / "server" argv = ["--lore", str(lore), "--world", str(world), "--server", str(server)] assert main(argv) == 0 # mutate a committed artifact -> --check must fail (world / "canon" / "testburg.json").write_text('{"id": "town.testburg"}\n') assert main(argv + ["--check"]) == 1 def test_check_detects_missing(tmp_path): lore = _lore(tmp_path) world, server = tmp_path / "world", tmp_path / "server" argv = ["--lore", str(lore), "--world", str(world), "--server", str(server)] assert main(argv) == 0 (world / "npcs" / "tess.json").unlink() assert main(argv + ["--check"]) == 1 def test_invalid_bible_fails_build(tmp_path): lore = tmp_path / "lore" lore.mkdir() (lore / "bad.md").write_text( "```yaml\nid: town.a\ntype: town\nstatus: canon\nsecrecy: 0\n" "related: [does.not-exist]\nbody: x\n```\n") argv = ["--lore", str(lore), "--world", str(tmp_path / "w"), "--server", str(tmp_path / "s")] assert main(argv) == 1 ``` - [ ] **Step 2: Run to verify it fails** Run: `PYTHONPATH=tools .venv/bin/python -m pytest tools/content_build/tests/test_cli.py -q` Expected: FAIL — `ImportError` (no `main` in `content_build.__main__`). - [ ] **Step 3: Write `__main__.py`** `tools/content_build/__main__.py`: ```python """CLI entry: `python -m content_build [--check]`. Orchestrates parse -> model -> resolve -> emit. --check builds in memory and diffs against the on-disk committed JSON (stale-or-invalid -> exit 1).""" import argparse import sys from pathlib import Path from .emit import build_trees, check_secrecy, dump_json, write_trees from .errors import BuildError from .model import entry_from_raw from .parse import parse_bible from .resolve import resolve def _load_world(lore_dir: Path): raws = [] for md in sorted(Path(lore_dir).glob("*.md")): raws.extend(parse_bible(md)) return resolve([entry_from_raw(r) for r in raws]) def build(lore_dir, world_dir, server_dir) -> None: world = _load_world(lore_dir) client_files, server_files = build_trees(world) check_secrecy(client_files, world) write_trees(client_files, server_files, world_dir, server_dir) def _diff(path: Path, obj) -> list[str]: want = dump_json(obj) if not path.exists(): return [f"missing {path}"] if path.read_text() != want: return [f"differs {path}"] return [] def check(lore_dir, world_dir, server_dir) -> int: world = _load_world(lore_dir) client_files, server_files = build_trees(world) check_secrecy(client_files, world) stale: list[str] = [] for relpath, obj in client_files.items(): stale += _diff(Path(world_dir) / relpath, obj) for relpath, obj in server_files.items(): stale += _diff(Path(server_dir) / relpath, obj) for s in stale: print("STALE:", s, file=sys.stderr) return 1 if stale else 0 def main(argv=None) -> int: ap = argparse.ArgumentParser(prog="content_build") ap.add_argument("--check", action="store_true", help="verify committed JSON is fresh + valid; no writes") ap.add_argument("--lore", default="content/lore") ap.add_argument("--world", default="content/world") ap.add_argument("--server", default="content/server") args = ap.parse_args(argv) try: if args.check: return check(args.lore, args.world, args.server) build(args.lore, args.world, args.server) return 0 except BuildError as e: print("BUILD FAILED:", e, file=sys.stderr) return 1 if __name__ == "__main__": raise SystemExit(main()) ``` - [ ] **Step 4: Run to verify it passes** Run: `PYTHONPATH=tools .venv/bin/python -m pytest tools/content_build/tests/test_cli.py -q` Expected: PASS (4 passed). - [ ] **Step 5: Run the whole tool suite** Run: `PYTHONPATH=tools .venv/bin/python -m pytest tools/content_build/tests -q` Expected: PASS (all tests green). - [ ] **Step 6: Commit** ```bash git add tools/content_build/__main__.py tools/content_build/tests/test_cli.py git commit -m "feat(content-build): CLI build + --check staleness gate" ``` --- ### Task 6: Promote Duncarrow (the acceptance run) **Files:** - Rename: `candidate-town.md` → `content/lore/duncarrow.md` - Create: `content/lore/canon-roadmap.md` - Create (BUILT, committed): `content/world/canon/*.json`, `content/world/topics/*.json`, `content/world/npcs/{mera-fenn,mayor-oswin-crell,harn-blackwood}.json`, `content/server/npcs/*.json`, `content/server/topics/*.json` - Test: `tools/content_build/tests/test_duncarrow.py` **Interfaces:** - Consumes: the whole tool (Tasks 1–5). - Produces: the committed Duncarrow artifacts; a green `--check`. - [ ] **Step 1: Move the specimen into the lore tree** ```bash git mv candidate-town.md content/lore/duncarrow.md ``` - [ ] **Step 2: Convert every fenced block to a `yaml` fence** In `content/lore/duncarrow.md`, change every block's opening fence from ` ``` ` to ` ```yaml ` (the parser only reads `yaml`-tagged blocks). Leave prose commentary and closing fences as-is. - [ ] **Step 3: Make each `knows` list valid YAML** Rewrite every `knows` entry from the one-line `- fact: X gate: Y` form to a proper mapping. Example (Mera Fenn's block becomes): ```yaml knows: - {fact: rumor.elves-avoid-the-shrine, gate: neutral} - {fact: rumor.travelers-go-missing, gate: warm} - {fact: fact.militia-never-investigates, gate: warm} - {fact: fact.crells-guard-acts-alone, gate: trusted} ``` Apply the same conversion to Mayor Crell's and Harn Blackwood's `knows` lists. - [ ] **Step 4: Apply the six specimen conformance fixes** 1. **Drop `knows: town.duncarrow`** links from `npc.mayor-oswin-crell` and `npc.harn-blackwood` (a canon entity may not appear in a `knows` list; the town is public and spoken freely). 2. **Rename** `shrine.the-white-antlers` → `place.the-white-antlers` (id namespace must equal type), and update **every** `related` reference to it — in `town.duncarrow`, `rumor.elves-avoid-the-shrine`, `rumor.travelers-go-missing`, `secret.crell-runs-slave-trade`. 3. **Add two canon stubs** so all `related` ids resolve (paste as new `yaml` blocks under the WORLD RULE section): ```yaml id: region.the-tallow-reach type: region status: canon secrecy: 0 related: [] body: > The Tallow Reach — the lowland march of grain towns and wool roads that Duncarrow sits at the edge of, feeding the pass trade. ``` ```yaml id: faction.elves type: faction status: canon secrecy: 0 related: [] body: > The elven travellers of the road — no single polity, but the people whose passage through the White Antlers, and its recent absence, the town notices. ``` 4. **Drop `secret.crell-runs-slave-trade`** from the **public** `person.mayor-oswin-crell` record's `related` (leave it `related: [town.duncarrow]`). The secret still links *to* the mayor, and the `npc.mayor-oswin-crell` layer still holds it at `gate: never` — that is the real bounded-move test. 5. **Add `rungs:`** to `rule.disposition-ladder` (keep the existing prose `body`): ```yaml id: rule.disposition-ladder type: rule status: canon secrecy: 0 related: [] rungs: [hostile, cold, neutral, warm, trusted] body: > NPC trust toward the player is one of five ordered states: hostile < cold < neutral < warm < trusted. ... ``` 6. **Flip every entry** `status: candidate` → `status: canon` (all of them — Duncarrow promotes atomically). - [ ] **Step 5: Write the canon roadmap index** `content/lore/canon-roadmap.md`: ```markdown # Canon Roadmap The world's answer to `docs/roadmap.md`: what is authored vs pending, and the order the Margreave is being written. Source of truth is `content/lore/*.md`; `content/world` + `content/server` are BUILT from it (`python -m content_build`). ## Authored (status: canon) - **duncarrow.md** — Duncarrow (the Tallow Reach market town), the White Antlers shrine, the Crell secret chain, and NPCs Mera Fenn (witness), Mayor Crell (control), Harn Blackwood (texture). The bounded-dialogue / disposition-gate specimen. ## Pending (not yet authored) - The rest of the Tallow Reach region; the elven road-peoples beyond the stub `faction.elves`; the seven worldbuilding topics tracked in the canon-architecture direction. ``` - [ ] **Step 6: Run the build and fix any validation errors** Run: `PYTHONPATH=tools .venv/bin/python -m content_build` Expected: exit 0, no `BUILD FAILED` output. If it reports an error, the message names the entry id + `duncarrow.md:` — fix that block and re-run. - [ ] **Step 7: Write the round-trip acceptance test** `tools/content_build/tests/test_duncarrow.py`: ```python """End-to-end: the real promoted Duncarrow bible builds green and routes the Crell secret correctly (body server-only; the id ships in the gate skeleton).""" import json from pathlib import Path from content_build.__main__ import _load_world, check from content_build.emit import build_trees REPO = Path(__file__).resolve().parents[3] LORE = REPO / "content" / "lore" WORLD = REPO / "content" / "world" SERVER = REPO / "content" / "server" def test_duncarrow_builds_and_check_is_clean(): assert check(LORE, WORLD, SERVER) == 0 def test_secret_body_is_server_only(): client, server = build_trees(_load_world(LORE)) assert "topics/crell-runs-slave-trade.json" in server # the secret's descriptive id ships in the gate skeleton (schema property)... npc = client["npcs/mayor-oswin-crell.json"] assert any(k["fact_id"] == "secret.crell-runs-slave-trade" and k["gate"] == "never" for k in npc["knows"]) # ...but its BODY never reaches any client artifact blob = json.dumps(client) assert "elf-slaving" not in blob and "sells them into slavery" not in blob def test_mera_never_holds_the_secret(): client, _ = build_trees(_load_world(LORE)) fenn = client["npcs/mera-fenn.json"] facts = {k["fact_id"] for k in fenn["knows"]} assert "secret.crell-runs-slave-trade" not in facts assert "fact.crells-guard-acts-alone" in facts ``` - [ ] **Step 8: Run the acceptance test** Run: `PYTHONPATH=tools .venv/bin/python -m pytest tools/content_build/tests/test_duncarrow.py -q` Expected: PASS (3 passed). If `test_secret_body_is_server_only` fails on the `elf-slaving`/`slavery` assertion, the secret body leaked into a client artifact — a real bug; stop and fix routing. - [ ] **Step 9: Commit the promoted bible + built artifacts** ```bash git add content/lore/duncarrow.md content/lore/canon-roadmap.md content/world content/server tools/content_build/tests/test_duncarrow.py git commit -m "feat(content): promote Duncarrow; build client/server artifacts" ``` --- ### Task 7: Client loader tolerates the new `canon/` + `topics/` dirs **Files:** - Modify: `client/scripts/content/content_db.gd` - Test: `client/tests/unit/test_content_db.gd` **Interfaces:** - Consumes: the built `content/world/{canon,topics}/*.json` (Task 6). - Produces: `ContentDB` gains `canon_entities: Dictionary`, `topics: Dictionary`, and `canon(id)/topic(id)/has_canon(id)/has_topic(id)` accessors. The backing var is `canon_entities` (not `canon`) because GDScript forbids a var and a method sharing a name — mirroring the existing `var locations` / `func location` pairing. Existing dirs + legacy `fenn.json` unchanged. - [ ] **Step 1: Write the failing loader tests** Append to `client/tests/unit/test_content_db.gd`: ```gdscript func test_loads_canon_and_topics(): assert_true(db.has_canon("town.duncarrow")) assert_true(db.has_canon("place.the-white-antlers")) assert_true(db.has_topic("rumor.travelers-go-missing")) func test_topic_skeleton_has_no_body(): var t := db.topic("secret.crell-runs-slave-trade") assert_false(t.has("body")) # bodies are server-only func test_legacy_npcs_still_load(): assert_true(db.has_npc("fenn")) assert_true(db.has_npc("mera-fenn")) ``` - [ ] **Step 2: Run to verify it fails** Run: `cd client && ./run_tests.sh -gtest=res://tests/unit/test_content_db.gd` Expected: FAIL — `has_canon`/`has_topic` not found (or the dicts are empty). - [ ] **Step 3: Add the two dirs and accessors to `content_db.gd`** In `client/scripts/content/content_db.gd`, add two vars after the existing `items` var (line ~10): ```gdscript var canon_entities: Dictionary = {} var topics: Dictionary = {} ``` In `load_from()`, after the existing `items = _load_dir(...)` line, add: ```gdscript canon_entities = _load_dir(world.path_join("canon")) topics = _load_dir(world.path_join("topics")) ``` Add accessors next to the existing `location()/npc()/...` accessors (note the backing var `canon_entities` — a `func canon` cannot share a name with its var): ```gdscript func canon(id: String) -> Dictionary: return canon_entities.get(id, {}) func topic(id: String) -> Dictionary: return topics.get(id, {}) func has_canon(id: String) -> bool: return canon_entities.has(id) func has_topic(id: String) -> bool: return topics.has(id) ``` Note: `_load_dir` already `push_error`s and skips a missing dir, so this stays safe if `canon/`/`topics/` are ever absent. - [ ] **Step 4: Run to verify it passes** Run: `cd client && ./run_tests.sh -gtest=res://tests/unit/test_content_db.gd` Expected: PASS — the new tests green, existing `test_content_db` tests still green. - [ ] **Step 5: Run the full client suite to confirm no regressions** Run: `cd client && ./run_tests.sh` Expected: PASS (no new failures). - [ ] **Step 6: Commit** ```bash git add client/scripts/content/content_db.gd client/tests/unit/test_content_db.gd git commit -m "feat(client): ContentDB loads canon/ + topics/ built dirs" ``` --- ### Task 8: CI gate **Files:** - Create: `.github/workflows/content-build.yml` **Interfaces:** - Consumes: the tool + committed artifacts (Tasks 1–6). - Produces: a CI job that runs the tool tests and `--check`. > **Note:** the repo has no CI yet. This adds a minimal GitHub Actions workflow. If the project uses a different CI provider, port these two commands (`pytest tools/content_build/tests` and `python -m content_build --check`) to it and skip this file. - [ ] **Step 1: Write the workflow** `.github/workflows/content-build.yml`: ```yaml name: content-build on: push: paths: ["content/**", "tools/content_build/**", ".github/workflows/content-build.yml"] pull_request: paths: ["content/**", "tools/content_build/**"] jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: "3.12" - run: pip install -r tools/requirements-dev.txt - name: Unit tests run: PYTHONPATH=tools python -m pytest tools/content_build/tests -q - name: Content freshness + validity run: PYTHONPATH=tools python -m content_build --check ``` - [ ] **Step 2: Sanity-run the two CI commands locally** Run: ```bash PYTHONPATH=tools .venv/bin/python -m pytest tools/content_build/tests -q PYTHONPATH=tools .venv/bin/python -m content_build --check ``` Expected: both exit 0. - [ ] **Step 3: Commit** ```bash git add .github/workflows/content-build.yml git commit -m "ci: content-build tests + --check gate" ``` --- ## Done criteria - `PYTHONPATH=tools .venv/bin/python -m pytest tools/content_build/tests -q` → all green (parse, model, resolve, emit, cli, duncarrow). - `PYTHONPATH=tools .venv/bin/python -m content_build --check` → exit 0. - `content/lore/duncarrow.md` + `content/lore/canon-roadmap.md` exist; `candidate-town.md` is gone. - Built `content/world/{canon,topics,npcs}` + `content/server/{npcs,topics}` committed; the Crell secret **body** appears only under `content/server/`. - `cd client && ./run_tests.sh` → green, legacy NPCs still load. - Then: **superpowers:finishing-a-development-branch** to smoke-test and (on the human's confirmation, per §18) merge `feature/content-build-tool` → `dev` with `--no-ff`. ## Out of scope (next cycles — do NOT build here) - Runtime consumption: client computing gated `available_moves` from the new npc skeletons; `api` reading personas from `content/server/`; voicing Mera Fenn end to end. - Legacy `fenn.json` migration to a bible. - Godot export packaging (include `world/`, exclude `server/`). - Opaque topic ids (the descriptive-slug spoiler property is inherent to the locked schema — noted, not fixed). ```