diff --git a/CLAUDE.md b/CLAUDE.md index be8457d..1a3c575 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -49,6 +49,24 @@ When asked to synthesize a topic: `md//-synthesis.md`, unless told otherwise. 4. Syntheses are part of the record — they get committed. +## Structured data (fact-caches) + +A topic may carry a machine-readable **fact-cache** alongside its markdown so you +can answer attribute questions without re-reading the corpus. Convention, per +topic folder `md//`: + +- `.json` — facts, one record per entity, each with `sources` + + `citations` back to the source files. +- `.schema.json` — JSON Schema; validates the sibling `.json`. +- `INDEX.md` — read this **first**: query the `.json` for facts → the + `*-synthesis.md` for narrative → a source `.md` only for verbatim quotes. + +Validate after any edit: `./.venv/bin/python validate.py ` (or no arg for +the whole repo). Edit the **JSON first**, validate, then update prose. + +Full pattern + how to add it to a new topic: **`docs/data-convention.md`**. +Worked example: `md/demonology/`. + ## Git conventions - Treat this repo as code: **everything goes in git** except what `.gitignore` diff --git a/docs/data-convention.md b/docs/data-convention.md new file mode 100644 index 0000000..ed68c50 --- /dev/null +++ b/docs/data-convention.md @@ -0,0 +1,118 @@ +# Structured data convention — fact-caches per topic + +How this research center stores **machine-readable facts** on top of its +markdown, so an AI agent (or a script) can query a topic without re-reading the +whole source corpus. Topic-agnostic: the same pattern is meant to repeat for +every research topic. + +## Why this exists + +The repo's core workflow is **anti-RAG**: Claude reads *whole* markdown files to +synthesize across a topic (see `CLAUDE.md`). That is the right tool for +*synthesis over source text* — but it is expensive, and re-reading megabytes of +markdown every session just to recall "what rank is Asmodeus" is wasteful. + +So each topic may carry a **derived fact-cache**: a structured JSON file that +captures the facts extracted during a thorough read, once, so later sessions load +a small structured file instead of the corpus. The cache does **not** replace the +sources — it sits on top of them and points back to them with citations. + +- **Sources** (`md//*.md`) = ground truth for *text*. Read whole, for + synthesis and verbatim quotes. +- **Fact-cache** (`md//.json`) = ground truth for *facts*. Query + for attributes; never needs the corpus re-read. +- **Synthesis** (`md//*-synthesis.md`) = ground truth for *narrative*. + The human-readable argument, derived from the same facts. + +## The file set (per topic) + +Inside a topic folder `md//`: + +| File | Role | Required? | +|------|------|-----------| +| `.json` | The fact-cache: one object per entity, fields + citations. | The data | +| `.schema.json` | JSON Schema for `.json`. Self-documents the shape; enables validation. | Yes, pairs with the data | +| `INDEX.md` | Topic front-door: tells an agent which artifact to use for what. | Recommended | +| `*-synthesis.md` | Prose synthesis derived from the data. | Optional | + +**Naming convention (the only wiring):** a file named `.schema.json` +validates its sibling `.json` in the same folder. `validate.py` discovers +pairs by this rule alone — no central registry to maintain. + +Worked example: `md/demonology/` contains `demons.json`, `demons.schema.json`, +`INDEX.md`, and `demon-hierarchy-synthesis.md`. + +## How an AI agent should use a topic + +Read the topic's `INDEX.md` first, then follow this order: + +1. **Need a fact / attribute / to filter or sort?** → query the `.json` + (e.g. with `jq`). Do *not* read the sources. +2. **Need the narrative or how sources disagree?** → read the `*-synthesis.md`. +3. **Need a verbatim quote or a detail not in the data?** → open the *one* source + `.md` named in the entity's `citations`, at the cited location. Read it whole + (repo convention), not a snippet. + +This keeps the expensive whole-corpus read for when it is actually needed +(quotes, fresh synthesis) and serves everything else from ~tens of KB of JSON. + +## Record shape + +The schema is the authority, but the house style for an entity record is: + +- a stable `id` (kebab-case) — the target of any `cross_refs` +- a primary `name` plus `aliases` +- the substantive fields for the topic (ranks, domains, signs, origins…) +- **provenance on every record:** `sources` (array of source codes), + `source_count`, and `citations` (source code → location string) +- `cross_refs` — `id`s of related/conflated entities +- `notes` — uncertainty flags, conflation warnings, scope caveats + +Each dataset also carries a top-level `meta` block (scope rule, caveats, +`generated_from`) and a `sources` map (source code → human description) so the +file is self-explanatory in isolation. + +## Validation + +`validate.py` (repo root) checks every topic's datasets: + +```bash +./.venv/bin/python validate.py # every topic under md/ +./.venv/bin/python validate.py # one topic +./.venv/bin/python validate.py md//.json # one file +``` + +It applies two layers: + +1. **JSON Schema** — structure, types, enums, required fields, `minItems`, etc. +2. **House rules** (cross-field invariants the schema can't express, applied to + any record that uses the conventional field names): + - `source_count` must equal `len(sources)` + - every `cross_refs` entry must reference an `id` that exists in the dataset + - `id`s must be unique + +Exit code is non-zero on failure, so it works as a git pre-commit hook or CI +step. Requires `jsonschema` (in `requirements.txt`). + +> Validation lives in a **standalone** script, not in `convert.py`, on purpose: +> conversion is triggered by PDF changes, dataset edits are triggered by +> extraction/AI — different events. The validator is what the *editor* calls. + +## Adding the pattern to a new topic + +1. Convert sources as usual: `./.venv/bin/python convert.py` → `md//`. +2. Extract facts (AI or script) into `md//.json`, giving every + record `sources` + `source_count` + `citations`. +3. Write `md//.schema.json` describing that shape (copy + `md/demonology/demons.schema.json` as a starting point and adapt fields). +4. Add a short `md//INDEX.md` (copy the demonology one and adjust). +5. Validate: `./.venv/bin/python validate.py `. +6. Commit the markdown, the JSON, the schema, and the index. (Sources/PDFs stay + gitignored as always.) + +## Keeping facts and prose in sync + +The `.json` is the source of truth for **facts**; the `*-synthesis.md` is the +source of truth for **narrative**. When something changes, edit the **JSON +first**, re-validate, then reflect it in the prose — not the other way around. A +fact that lives only in prose is a fact the machine can't see. diff --git a/md/demonology/INDEX.md b/md/demonology/INDEX.md new file mode 100644 index 0000000..aa2010b --- /dev/null +++ b/md/demonology/INDEX.md @@ -0,0 +1,73 @@ +# Demonology — Topic Index + +**Read this file first.** It tells an AI agent what exists for this topic and +which artifact to use, so you don't re-read the full source corpus (~3.4 MB) +unless you actually need a direct quote. + +## Decision order for an agent + +1. **Need a fact, attribute, rank, or to filter/sort entities?** + → Query `demons.json`. Do **not** read the sources. It is the fact-cache. +2. **Need the narrative, the argument, or how the sources disagree?** + → Read `demon-hierarchy-synthesis.md` (prose). +3. **Need a verbatim quote or a detail not in the data?** + → Open the one source `.md` named in the entity's `citations`, at the cited + page/line. Read that file whole (repo convention), not a snippet. + +## Artifacts + +| File | Type | Use it for | +|------|------|-----------| +| `demons.json` | structured data (33 entities) | Querying attributes: rank, legions, domain, signs, sin, source_count, cross_refs. The machine source of truth for facts. | +| `demons.schema.json` | JSON Schema | Field definitions, allowed values, validation. Read to learn the data shape. | +| `demon-hierarchy-synthesis.md` | prose synthesis | The ranked hierarchy as narrative + the recurrence map + where sources disagree. | +| *(6 source `.md` files)* | converted sources | Direct quotes / details beyond the dataset. Listed under `sources` in `demons.json`. | + +## What's in `demons.json` + +- **Scope:** only entities attested in **2+ of the 6 sources** (33 entities). + Single-source names (most of the Goetia 72, the world-folklore long tail) are + intentionally excluded — see the synthesis for why. +- **Tiers:** `supreme-prince` (11), `directional-king` (4), `goetia-officer` (2), + `watcher` (7), `other-recurring` (9). +- **Source codes:** `FG` Field Guide · `DM` Dark Mirrors · `PU` Personal/Impersonal + (Barth/Unger) · `SP` Spiritism · `DJ` Daemonologie · `EN` Encyclopedia. Full + descriptions in `demons.json` → `sources`. +- **Aliases are merged** to a primary name; `cross_refs` link conflated/related + entities (e.g. `satan` ↔ `lucifer` ↔ `satanael`; `asmodeus` ↔ `aeshma`/`ahriman`). + +## Example queries (jq) + +```bash +cd md/demonology +# All Kings attested in 2+ sources +jq -r '.demons[] | select(.rank != null and (.rank|test("King"))) | .name' demons.json +# The seven Princes of Hell by deadly sin +jq '.demons[] | select(.deadly_sin) | {name, deadly_sin}' demons.json +# Everything naming the Watcher myth (Dark Mirrors) +jq '.demons[] | select(.sources|index("DM")) | .name' demons.json +# Most-corroborated entities first +jq '[.demons[] | {name, source_count}] | sort_by(-.source_count)' demons.json +``` + +## Keeping it in sync + +`demons.json` is the source of truth for **facts**; the synthesis is the source of +truth for **narrative**. If you add/correct an entity, update `demons.json` first, +re-validate against the schema, then reflect it in the synthesis prose — not the +other way around. + +```bash +# validate after editing (schema + house rules: source_count, cross_refs, unique ids) +./.venv/bin/python validate.py demonology # this topic +./.venv/bin/python validate.py # every topic in the repo +``` + +The validator is generic: any topic that drops a `.schema.json` next to its +`.json` is picked up automatically. Exit code is non-zero on failure, so it +doubles as a git pre-commit hook or CI step. + +To extend scope (e.g. add the full Goetia 72 or the folklore long tail), re-run +the per-source extraction, set each record's `sources`/`source_count` +accordingly, and relax the schema's `minItems: 2` on `sources` if you drop the +2+ rule. diff --git a/md/demonology/demons.json b/md/demonology/demons.json new file mode 100644 index 0000000..18427fd --- /dev/null +++ b/md/demonology/demons.json @@ -0,0 +1,771 @@ +{ + "meta": { + "topic": "demonology", + "scope": "entities attested in 2 or more of the six source documents", + "rank_ladder": ["Emperor", "King", "Prince", "Duke", "Marquis", "Earl", "President", "Knight"], + "generated_from": [ + "demon-hierarchy-synthesis.md", + "the six source documents in md/demonology/" + ], + "caveat": "Rank depth derives almost entirely from the Encyclopedia (EN). Daemonologie (DJ) names these ranks only to reject them as the Devil's lies. This is one tradition's hierarchy, not a consensus of the sources. Aliases are merged to the most common form; see cross_refs for conflations." + }, + "sources": { + "FG": "A Field Guide to Demons, Fairies, Fallen Angels and Other Subversive Spirits (Mack & Mack) — global folklore catalog", + "DM": "Dark Mirrors: Azazel and Satanael in Early Jewish Demonology (Orlov) — Enochic / Second-Temple Watcher myth", + "PU": "Personal or Impersonal: Karl Barth & Merrill Unger on the Demonic — theology (personal vs. 'nothingness')", + "SP": "Spiritism and the Fallen Angels in the Light of the OT and NT — Protestant polemic on mediumship/idolatry", + "DJ": "Daemonologie (King James I) + News from Scotland — witch-hunt theology; rejects the grimoire hierarchy", + "EN": "The Encyclopedia of Demons and Demonology (Guiley) — reference; full Goetia 72, seven princes, Watchers" + }, + "demons": [ + { + "id": "satan", + "name": "Satan", + "aliases": ["the Devil", "the Adversary", "Diabolos", "the Dragon", "the Serpent", "the Deceiver", "the Evil One", "Sathan"], + "type": "individual", + "tier": "supreme-prince", + "rank": "Head of all demons / Prince of Darkness", + "legions": null, + "goetia_number": null, + "deadly_sin": "Anger", + "domain": ["temptation", "accusation", "rule of Hell", "the power of death", "deception"], + "signs": ["the dragon/serpent of Revelation 12", "disguises as an angel of light (2 Cor 11:14)", "shape-shifter: luminous angel, serpent, wild beast", "appears to witches as a man in black; cold to the touch; marks servants with the Devil's mark", "walks 'as a roaring lion'"], + "origins": "Originally an exalted cherub fallen through pride (Isa 14 / Ezek 28); Hebrew 'adversary'. Identified with the Eden serpent, Lucifer, Iblis.", + "sources": ["FG", "DM", "PU", "SP", "DJ", "EN"], + "source_count": 6, + "citations": { + "DM": "Satan and the Visionary, pp. 107-112", + "PU": "§1.3 Satanology", + "SP": "Ch. III 'Satan—Or Spiritism at its Source', pp. 27-38", + "DJ": "Bk III Ch V (~p. 175)", + "EN": "'Satan' ~21423", + "FG": "PSYCHE (ALL NAMES)" + }, + "cross_refs": ["lucifer", "satanael", "samael", "beelzebub", "belial", "iblis", "apollyon"], + "notes": "The only entity, with Lucifer, named in all six sources. Functions as the merge-point for most upper-tier aliases." + }, + { + "id": "lucifer", + "name": "Lucifer", + "aliases": ["Helel", "the Morning Star", "the Day Star", "Satanael"], + "type": "individual", + "tier": "supreme-prince", + "rank": "Emperor of Hell", + "legions": null, + "goetia_number": null, + "deadly_sin": "Pride", + "domain": ["light-bringing", "pride", "rule over Europeans/Asiatics", "temptation"], + "signs": ["appears as a beautiful child when conjured", "associated with Venus / the morning star"], + "origins": "Latin lucem ferre, 'light-bearer', from the Day Star (Isa 14:12). = Satanael in 2 Enoch; treated as Satan's pre-fall name. Ranked above Satan in EN's scheme; DJ calls it merely an allegorical name for Satan before his fall.", + "sources": ["FG", "DM", "PU", "SP", "DJ", "EN"], + "source_count": 6, + "citations": { + "DM": "The Watchers of Satanael, pp. 85-99", + "PU": "Lutzer quote ~line 8030", + "SP": "Satan's names list, p. 28", + "DJ": "Bk I Ch VI (~p. 90)", + "EN": "'Lucifer' ~15055", + "FG": "PSYCHE (ALL NAMES)" + }, + "cross_refs": ["satan", "satanael"], + "notes": "Aliased to Satan in most sources; given distinct top rank only in EN." + }, + { + "id": "beelzebub", + "name": "Beelzebub", + "aliases": ["Baal-zebul", "Beelzeboul", "Lord of the Flies"], + "type": "individual", + "tier": "supreme-prince", + "rank": "Prince of Demons", + "legions": null, + "goetia_number": null, + "deadly_sin": "Gluttony", + "domain": ["command of the decan-demons", "lust in holy men", "war and murder", "rule over the sabbats", "demon worship"], + "signs": ["a gigantic ugly fly", "a giant enthroned: swollen face, horns, bat wings, duck feet, lion's tail", "thwarted by the oath 'Elo-i'"], + "origins": "From Baal-zebul, 'Lord of the Flies'; highest-ranking fallen angel; chief possessing demon at Aix and Loudun.", + "sources": ["FG", "SP", "DJ", "EN"], + "source_count": 4, + "citations": { + "SP": "p. 28 (Satan's names) & Ch. IV p. 42", + "DJ": "Bk III Ch V (~p. 175); Grandier pact note ~p. 97", + "EN": "'Beelzebub' ~2732; Testament of Solomon", + "FG": "PSYCHE (ALL NAMES)" + }, + "cross_refs": ["satan"], + "notes": null + }, + { + "id": "leviathan", + "name": "Leviathan", + "aliases": ["the Tortuous Serpent", "the Slant Serpent"], + "type": "individual", + "tier": "supreme-prince", + "rank": "King of beasts", + "legions": null, + "goetia_number": null, + "deadly_sin": "Envy", + "domain": ["the sea and the deep", "dragging down ships", "primordial chaos"], + "signs": ["whale-like, shield-scaled", "flaming breath, 'eyes like the dawn'", "near-invulnerable"], + "origins": "Primordial sea monster of Job / Jonah. Female aspect ('Tortuous Serpent') linked to Lilith; male the 'Slant Serpent'. A Loudun possessing demon.", + "sources": ["FG", "DM", "DJ", "EN"], + "source_count": 4, + "citations": { + "DM": "The Likeness of Heaven, p. 21", + "DJ": "Bk I Ch VI note 10 (~p. 97), Grandier pact", + "EN": "'Leviathan' ~14260", + "FG": "PSYCHE (ALL NAMES)" + }, + "cross_refs": ["lilith"], + "notes": null + }, + { + "id": "belial", + "name": "Belial", + "aliases": ["Beliar", "Bernael"], + "type": "individual", + "tier": "supreme-prince", + "rank": "King", + "legions": 80, + "goetia_number": 68, + "deadly_sin": null, + "domain": ["worthlessness", "fornication", "lies", "treachery", "leadership of the Sons of Darkness"], + "signs": ["a deceptively beautiful, soft-voiced angel in a fire-dragon chariot", "'visage like a viper' (Testament of Amram)", "requires sacrifices to invoke"], + "origins": "Hebrew 'without worth'; one of the Watchers; titled Prince of Darkness, King of Evil; synonym for Satan/Antichrist. At Qumran, head of 'the men of the lot of Belial'. Via Milton (SP), a chief of the fallen who coupled with the daughters of men.", + "sources": ["DM", "SP", "EN"], + "source_count": 3, + "citations": { + "DM": "Eschatological Yom Kippur, p. 40 (Qumran)", + "SP": "p. 28 & Milton quote p. 52", + "EN": "'Belial' ~2945" + }, + "cross_refs": ["satan", "watchers"], + "notes": "Both a top-tier prince and Goetia #68; rank/legions from the Goetia (EN)." + }, + { + "id": "astaroth", + "name": "Astaroth", + "aliases": ["Ashtaroth", "Astarte (origin)"], + "type": "individual", + "tier": "supreme-prince", + "rank": "Grand Duke and Treasurer", + "legions": 40, + "goetia_number": 29, + "deadly_sin": "Sloth", + "domain": ["all sciences", "secrets of past/present/future", "encourages sloth", "necromantic divination"], + "signs": ["an ugly or beautiful angel astride a dragon, holding a viper", "foul stinking breath (hold a magic ring to the face)", "conjured Wednesday 10-11pm"], + "origins": "A male demon derived from the goddess Astarte/Ashtoreth; fallen seraph, prince of thrones; archdemon of Chesed. One of three supreme demons (with Beelzebub, Lucifer) in the Grand Grimoire. In SP, Milton lists Ashtaroth among the chiefs of the fallen.", + "sources": ["SP", "DJ", "EN"], + "source_count": 3, + "citations": { + "SP": "Ch. VI p. 65 (Milton)", + "DJ": "Bk I Ch VI note 10 (~p. 97), Grandier pact", + "EN": "'Astaroth' ~2279" + }, + "cross_refs": ["beelzebub", "lucifer"], + "notes": "Both a top-tier prince and Goetia #29." + }, + { + "id": "asmodeus", + "name": "Asmodeus", + "aliases": ["Asmoday", "Ashmedai", "Aeshma", "Sydonay"], + "type": "individual", + "tier": "supreme-prince", + "rank": "King", + "legions": 72, + "goetia_number": 32, + "deadly_sin": "Lust", + "domain": ["lust", "jealousy", "anger", "revenge", "wrecks marriages", "gambling", "teaches arithmetic/geometry/astronomy"], + "signs": ["three heads (ogre, ram, bull)", "cock's feet, serpent tail, wings", "rides a fire-breathing dragon", "fears water", "thwarted by smoke of fish liver and gall (Tobit)"], + "origins": "Persian, from the daeva Aeshma 'of the bloody mace'; son of Naamah and Shamdon; fallen seraph; husband of Lilith; first under Amaymon. Ruler of the shedim.", + "sources": ["FG", "DJ", "EN"], + "source_count": 3, + "citations": { + "DJ": "Bk I Ch VI note 8 (~p. 96)", + "EN": "'Asmodeus' ~2170; Zoroastrian 'Aeshma' ~642; Testament of Solomon", + "FG": "PSYCHE (ALL NAMES)" + }, + "cross_refs": ["amaymon", "lilith", "shedim", "ahriman"], + "notes": "Both a top-tier prince (Lust) and Goetia #32. Aeshma ties him to the Zoroastrian Ahriman tradition." + }, + { + "id": "mephistopheles", + "name": "Mephistopheles", + "aliases": ["Mephostophiles", "Mephistophilis"], + "type": "individual", + "tier": "supreme-prince", + "rank": "one of the seven great princes of Hell", + "legions": null, + "goetia_number": null, + "deadly_sin": null, + "domain": ["serving and damning Faust", "pact negotiation", "the Devil's representative"], + "signs": ["a tall man in black", "shape-shifts: grey friar, fiery bear, bald dwarf, invisible ringing bell"], + "origins": "Faust-legend figure, named by Trithemius; from Marlowe's Doctor Faustus and Goethe's Faust. The archetypal pact-serving familiar.", + "sources": ["FG", "DJ", "EN"], + "source_count": 3, + "citations": { + "DJ": "Bk I Ch VI note 4 (~p. 95)", + "EN": "'Mephistopheles' ~16801", + "FG": "PSYCHE (ALL NAMES)" + }, + "cross_refs": [], + "notes": null + }, + { + "id": "moloch", + "name": "Moloch", + "aliases": ["Molech", "Baal (identified)", "Malik"], + "type": "individual", + "tier": "supreme-prince", + "rank": "first of the evil demons of the Tree of Life (Kether, with Satan)", + "legions": null, + "goetia_number": null, + "deadly_sin": null, + "domain": ["child sacrifice", "plague", "'prince of the valley of tears'", "detrimental sun"], + "signs": ["a bull-headed man with long arms on a brass throne", "children burned in his hollow bronze belly (passing 'through the fire')"], + "origins": "Ammonite/Phoenician god; identified with Baal, Malik, and Greek Cronos.", + "sources": ["SP", "EN"], + "source_count": 2, + "citations": { + "SP": "Ch. VII 'Abominations of the Canaanites', p. 71", + "EN": "'Moloch' ~17459" + }, + "cross_refs": ["baal", "satan"], + "notes": null + }, + { + "id": "mammon", + "name": "Mammon", + "aliases": [], + "type": "individual", + "tier": "supreme-prince", + "rank": "prince of tempters / archdemon", + "legions": null, + "goetia_number": null, + "deadly_sin": "Avarice", + "domain": ["greed", "riches"], + "signs": ["symbol: wolf"], + "origins": "Aramaic 'riches'; a fallen angel, often equated with Lucifer/Satan/Beelzebub. Hell's ambassador to England.", + "sources": ["FG", "EN"], + "source_count": 2, + "citations": { + "EN": "'Mammon' ~16356", + "FG": "PSYCHE (ALL NAMES)" + }, + "cross_refs": ["lucifer", "satan", "beelzebub"], + "notes": null + }, + { + "id": "belphegor", + "name": "Belphegor", + "aliases": ["Baal-Peor"], + "type": "individual", + "tier": "supreme-prince", + "rank": "archdemon of the sixth sephirah (Tipareth)", + "legions": null, + "goetia_number": null, + "deadly_sin": "Sloth", + "domain": ["sloth", "riches", "inventions and discoveries", "misogyny", "licentiousness"], + "signs": ["sits on a pierced chair (excrement is his offering)", "appears as a beautiful young girl to tempt men"], + "origins": "The Moabite god Baal-Peor (phallus worship); a fallen Principality. Infernal ambassador to France.", + "sources": ["FG", "EN"], + "source_count": 2, + "citations": { + "EN": "'Belphegor' ~2993", + "FG": "PSYCHE (ALL NAMES)" + }, + "cross_refs": ["baal"], + "notes": "Note: deadly_sin Sloth overlaps with Astaroth; sources assign Sloth to both in different schemes." + }, + { + "id": "amaymon", + "name": "Amaymon", + "aliases": [], + "type": "individual", + "tier": "directional-king", + "rank": "King of the East", + "legions": null, + "goetia_number": null, + "deadly_sin": null, + "domain": ["rule of the East quarter", "command over the 72 Goetic demons"], + "signs": [], + "origins": "One of the four infernal directional kings who command the Goetia 72; Asmodeus, Gaap, and Seere serve under him.", + "sources": ["DJ", "EN"], + "source_count": 2, + "citations": { + "DJ": "Bk I Ch VI note 8 (~p. 96)", + "EN": "Goetia four-kings conjuration; subordinations under Amaymon" + }, + "cross_refs": ["asmodeus", "gaap", "corson", "zimimar"], + "notes": null + }, + { + "id": "corson", + "name": "Corson", + "aliases": ["Curson (?)"], + "type": "individual", + "tier": "directional-king", + "rank": "King of the West", + "legions": null, + "goetia_number": null, + "deadly_sin": null, + "domain": ["rule of the West quarter"], + "signs": [], + "origins": "One of the four infernal directional kings (DJ).", + "sources": ["DJ", "EN"], + "source_count": 2, + "citations": { + "DJ": "Bk I Ch VI note 8 (~p. 96)", + "EN": "Goetia four-kings conjuration; cf. Purson/Curson #20" + }, + "cross_refs": ["amaymon", "zimimar", "gaap"], + "notes": "UNCERTAIN: DJ's directional king Corson is often conflated with the Goetia's Duke Purson/Curson (#20) but may be a distinct figure." + }, + { + "id": "zimimar", + "name": "Zimimar", + "aliases": ["Zimimay", "Ziminiar"], + "type": "individual", + "tier": "directional-king", + "rank": "King of the North", + "legions": null, + "goetia_number": null, + "deadly_sin": null, + "domain": ["rule of the North quarter"], + "signs": [], + "origins": "One of the four infernal directional kings (DJ); named in the Goetia four-kings conjuration.", + "sources": ["DJ", "EN"], + "source_count": 2, + "citations": { + "DJ": "Bk I Ch VI note 8 (~p. 96)", + "EN": "Goetia four-kings conjuration" + }, + "cross_refs": ["amaymon", "corson", "gaap"], + "notes": null + }, + { + "id": "gaap", + "name": "Gaap", + "aliases": ["Goap", "Tap"], + "type": "individual", + "tier": "directional-king", + "rank": "King of the South (also President and Prince, Goetia)", + "legions": 66, + "goetia_number": 33, + "deadly_sin": null, + "domain": ["rule of the South quarter", "teaches sciences/philosophy", "excites love/hate", "transports people", "gives familiars"], + "signs": ["a human with huge bat wings, preceded by four kings", "appears when the Sun is in the southern signs"], + "origins": "A fallen Power; both a Goetia officer (President and Prince, under Amaymon) and the directional King of the South.", + "sources": ["DJ", "EN"], + "source_count": 2, + "citations": { + "DJ": "Bk I Ch VI note 8 (~p. 96)", + "EN": "'Gaap' ~9168" + }, + "cross_refs": ["amaymon"], + "notes": "Spans two ranks: directional King and Goetia #33." + }, + { + "id": "baal", + "name": "Baal", + "aliases": ["Bael", "Baell", "Baalim"], + "type": "individual", + "tier": "goetia-officer", + "rank": "King ruling the East", + "legions": 66, + "goetia_number": 1, + "deadly_sin": null, + "domain": ["imparts invisibility and wisdom", "idolatry / false prophecy (SP)"], + "signs": ["triple-headed: a man's head flanked by cat and toad heads", "a hoarse voice"], + "origins": "Canaanite fertility lord, 'son of El'. In SP, Milton names Baalim among the male chiefs of the fallen, and Ahab's Baal-worship is read as Satanic mediumship.", + "sources": ["SP", "EN"], + "source_count": 2, + "citations": { + "SP": "Ch. VI p. 65 & Ch. VIII p. 86", + "EN": "'Baal' ~2494" + }, + "cross_refs": ["moloch", "belphegor"], + "notes": "Goetia #1; the root deity behind several other entries (Beelzebub/Baal-zebul, Belphegor/Baal-Peor, Berith/Baalberith)." + }, + { + "id": "foras", + "name": "Foras", + "aliases": ["Forcas", "Forras"], + "type": "individual", + "tier": "goetia-officer", + "rank": "President", + "legions": 29, + "goetia_number": 31, + "deadly_sin": null, + "domain": ["logic, ethics, virtues of herbs and gems", "makes invisible", "finds treasure", "grants eloquence and longevity"], + "signs": ["a strong man, or a chevalier"], + "origins": "A Goetia officer; cited in DJ as a specimen of the hierarchy James rejects ('Foras, a mighty President').", + "sources": ["DJ", "EN"], + "source_count": 2, + "citations": { + "DJ": "Bk I Ch VI note 8 (~p. 96)", + "EN": "'Foras' ~9075" + }, + "cross_refs": [], + "notes": "One of the few mid-rank Goetia officers attested outside EN." + }, + { + "id": "azazel", + "name": "Azazel", + "aliases": ["Asael", "Azael", "Iblis (Islam)"], + "type": "individual", + "tier": "watcher", + "rank": "Chief of the fallen / King of the seirim", + "legions": null, + "goetia_number": null, + "deadly_sin": null, + "domain": ["forbidden knowledge", "weapon-making", "cosmetics", "idolatry", "the 'lot of Azazel' (the wicked)"], + "signs": ["the scapegoat of Yom Kippur driven to the wilderness pit Dudael (Lev 16)", "appears as an 'unclean bird' descending on sacrifices", "a serpent with human hands/feet and twelve wings (Apoc. Abraham)", "goat-like, dwells in the desert"], + "origins": "A Watcher who lusted after mortal women (1 Enoch 6-11); bound by Raphael in Dudael. In the Apocalypse of Abraham elevated to THE arch-demon. In Islam = Iblis's pre-fall name; later conflated with Satan.", + "sources": ["FG", "DM", "EN"], + "source_count": 3, + "citations": { + "DM": "The Likeness of Heaven / Eschatological Yom Kippur / The Garment of Azazel (throughout)", + "FG": "DESERT 'Azazel', pp. 144-145", + "EN": "'Azazel' ~2411" + }, + "cross_refs": ["iblis", "satan", "shedim", "semyaza", "watchers"], + "notes": null + }, + { + "id": "semyaza", + "name": "Semyaza", + "aliases": ["Shemihazah", "Shemyaza", "Semjaza", "Shemhazai"], + "type": "individual", + "tier": "watcher", + "rank": "Leader of the Watchers (the 200)", + "legions": null, + "goetia_number": null, + "deadly_sin": null, + "domain": ["led the Watchers' descent", "the binding oath on Mount Hermon", "fathered the giants"], + "signs": ["hung upside-down in Orion as punishment", "father of the giants Ohya and Ahya"], + "origins": "Chief Watcher (1 Enoch 6); in 2 Enoch the leadership is reassigned to Satanael. Paired with Azael as the two arch-Watchers.", + "sources": ["DM", "SP", "EN"], + "source_count": 3, + "citations": { + "DM": "The Watchers of Satanael, pp. 85, 94, 100", + "SP": "Ch. VI (sons of God / Nephilim), pp. 51-57", + "EN": "'Semyaza' ~22117" + }, + "cross_refs": ["azazel", "satanael", "watchers", "nephilim"], + "notes": "SP attests by close cognate (the Genesis 6 'sons of God' narrative) rather than the name Semyaza." + }, + { + "id": "satanael", + "name": "Satanael", + "aliases": ["Satanail"], + "type": "individual", + "tier": "watcher", + "rank": "Prince of the Watchers (200 myriads)", + "legions": null, + "goetia_number": null, + "deadly_sin": null, + "domain": ["leadership of the rebel Watchers", "seduction/corruption of Adam and Eve", "the cursed vine"], + "signs": ["flies ceaselessly above the abyss after his fall", "made dark and ugly, loses creative power", "loses the '-el' to become Satan"], + "origins": "The chief celestial rebel of the Adamic myth (2 Enoch) who refused to venerate Adam. = Lucifer (EN); folds into Satan/Samael.", + "sources": ["DM", "EN"], + "source_count": 2, + "citations": { + "DM": "The Watchers of Satanael, pp. 85-99", + "EN": "'Satanail' ~21556" + }, + "cross_refs": ["satan", "lucifer", "samael", "semyaza"], + "notes": "The bridge figure between the Enochic Watcher myth and the Devil of the supreme tier." + }, + { + "id": "samael", + "name": "Samael", + "aliases": ["Sammael", "the venom of God"], + "type": "individual", + "tier": "watcher", + "rank": "first prince and accuser / chief of the ten evil sephirot", + "legions": null, + "goetia_number": null, + "deadly_sin": null, + "domain": ["jealousy", "executioner of God's death sentences", "death and the desert wind (Samiel/Simoon)"], + "signs": ["the rider on the serpent ('Samael came down riding on this serpent', Zohar)", "the dark spots of the Moon are his excrement"], + "origins": "The serpent who tempted Eve; uncircumcised husband of Lilith; 'prince of Rome' (3 Enoch); equated with Satan.", + "sources": ["FG", "DM", "EN"], + "source_count": 3, + "citations": { + "DM": "The Garment of Azazel, pp. 76-77", + "EN": "'Samael' ~21366", + "FG": "PSYCHE (ALL NAMES, 'Sammael')" + }, + "cross_refs": ["satan", "satanael", "lilith"], + "notes": null + }, + { + "id": "uzza-azza-azael", + "name": "Uzza, Azza & Azael", + "aliases": ["the three Watchers", "Ouza", "Uzzah"], + "type": "collective", + "tier": "watcher", + "rank": "fallen angelic leaders (a trio)", + "legions": null, + "goetia_number": null, + "deadly_sin": null, + "domain": ["opposed the creation of man", "taught humanity sorcery", "led Enosh's generation into idolatry"], + "signs": ["appear as a trio", "bow before / accuse Enoch-Metatron"], + "origins": "Reworked Watcher names echoing Shemihazah/Asael (3 Enoch). Uzza is also the tutelary (fallen) spirit of Egypt.", + "sources": ["DM", "EN"], + "source_count": 2, + "citations": { + "DM": "The Watchers of Satanael, pp. 90-91, 100-101 (3 Enoch)", + "EN": "'Uzza' ~25329" + }, + "cross_refs": ["azazel", "semyaza", "watchers"], + "notes": null + }, + { + "id": "watchers", + "name": "The Watchers", + "aliases": ["Grigori", "Egregoroi", "Sons of God", "Angels of Mastemoth"], + "type": "collective", + "tier": "watcher", + "rank": "200 fallen angels under chiefs of tens", + "legions": null, + "goetia_number": null, + "deadly_sin": null, + "domain": ["cohabited with women", "taught forbidden arts", "fathered the Nephilim"], + "signs": ["human-like but larger than the great giants", "dejected faces, perpetual silence, no liturgy", "imprisoned in darkness / under the earth"], + "origins": "The 200 'watching' angels who descended to Mount Hermon (1/2/3 Enoch, Genesis 6), led by Semyaza/Azael or (in 2 Enoch) Satanael.", + "sources": ["DM", "SP", "EN"], + "source_count": 3, + "citations": { + "DM": "The Watchers of Satanael, pp. 87-97", + "SP": "Ch. VI 'Sons of God Marrying the Daughters of Men', pp. 51-57", + "EN": "'Watchers' ~25683" + }, + "cross_refs": ["semyaza", "azazel", "satanael", "nephilim", "uzza-azza-azael"], + "notes": null + }, + { + "id": "nephilim", + "name": "Nephilim", + "aliases": ["the fallen ones", "the giants", "sons of Anak (post-Flood)"], + "type": "class", + "tier": "watcher", + "rank": "race of giants (offspring, not rulers)", + "legions": null, + "goetia_number": null, + "deadly_sin": null, + "domain": ["corruption", "bloodshed", "devouring", "the antediluvian wickedness that brought the Flood"], + "signs": ["300 cubits tall (1 Enoch)", "named sons of Semyaza: Ohya and Ahya (Hiwa/Hiya)", "ancestors of the Anakim"], + "origins": "The monstrous progeny of the Watchers and the daughters of men; identified with the Genesis 6 'mighty men of renown'; destroyed in the Flood but reappearing as the sons of Anak.", + "sources": ["DM", "SP", "EN"], + "source_count": 3, + "citations": { + "DM": "The Flooded Arboretums, pp. 113-125", + "SP": "Ch. VI pp. 51, 57", + "EN": "'Nephilim' ~17925" + }, + "cross_refs": ["watchers", "semyaza"], + "notes": null + }, + { + "id": "iblis", + "name": "Iblis", + "aliases": ["Shaytan (ruler of)", "Azazel, Lord of the Djinn"], + "type": "individual", + "tier": "other-recurring", + "rank": "Chief of all djinn / King of the Shaitans", + "legions": null, + "goetia_number": null, + "deadly_sin": null, + "domain": ["doubt", "tempting humankind", "command of an army of marid and shaitan"], + "signs": ["a fallen angel 'puffed up with pride'", "refused to bow to Adam ('you created me of fire, him of clay')"], + "origins": "The arch-demon of Islam; closely identified with Azazel (his pre-fall name). The Shaitan are a class of evil djinn ruled by Iblis.", + "sources": ["FG", "EN"], + "source_count": 2, + "citations": { + "FG": "DESERT 'Iblis' p. 146 & 'Shaitan' pp. 151-153", + "EN": "'Iblis' / 'Shaytan'" + }, + "cross_refs": ["azazel", "satan"], + "notes": "The Islamic counterpart/merge of Azazel and Satan." + }, + { + "id": "lilith", + "name": "Lilith", + "aliases": ["Lilitu", "Lil", "the Seducer", "mother of the lilin"], + "type": "individual", + "tier": "other-recurring", + "rank": "bride of Samael / archdemon of Yesod", + "legions": 420, + "goetia_number": null, + "deadly_sin": null, + "domain": ["night demon and succubus", "strangles newborns", "seduces sleeping men to spawn demons", "Lady of the Beasts"], + "signs": ["a beautiful woman with long red hair and a body of fire (or hairy bestial legs)", "a screech owl", "power peaks at the dark Moon", "repelled by iron and the names Sanvi/Sansanvi/Semangelaf"], + "origins": "Adam's first wife (Isa 34:14); Babylonian lilitu; from a kelippah (Zohar). Kabbalistic 'queen of hell', conflated with the Roman strix/stirges owl-demon (DJ). In FG, also mother of Norway's Huldrefolk.", + "sources": ["FG", "DJ", "EN"], + "source_count": 3, + "citations": { + "DJ": "Bk III Ch I note 4 (~p. 154)", + "EN": "'Lilith' ~14415", + "FG": "MOUNTAIN (Huldrefolk) & DOMICILE (ALL NAMES)" + }, + "cross_refs": ["samael", "asmodeus", "leviathan"], + "notes": "Rides with 420 legions on the Day of Atonement (legions field reflects this)." + }, + { + "id": "python", + "name": "Python", + "aliases": ["Pytho", "the Pythoness (host)", "spirit of divination"], + "type": "individual", + "tier": "other-recurring", + "rank": "oracular possessing spirit / familiar", + "legions": null, + "goetia_number": null, + "deadly_sin": null, + "domain": ["divination", "oracular prophecy", "mediumship"], + "signs": ["speaks oracular words THROUGH a possessed woman ('by her tongue', not at her command)", "the host swells, 'full of the god'"], + "origins": "From the serpent-dragon Python slain by Apollo at Delphi; the power behind the Delphic oracle and the Philippian damsel (Acts 16). 'Pythoness' generalized to any oracular woman, incl. the Witch of Endor.", + "sources": ["SP", "DJ"], + "source_count": 2, + "citations": { + "SP": "Ch. III p. 37 & Ch. IX pp. 97-99", + "DJ": "Bk II Ch I (~pp. 106-107)" + }, + "cross_refs": ["diana"], + "notes": null + }, + { + "id": "diana", + "name": "Diana", + "aliases": ["Diana of Ephesus", "leader of the night-court"], + "type": "individual", + "tier": "other-recurring", + "rank": "head of the fourth class of spirits (the fairy/witch host)", + "legions": null, + "goetia_number": null, + "deadly_sin": null, + "domain": ["idolatry and magic (Ephesus)", "the witches' nocturnal flying host", "the fairies / 'Good Neighbours'"], + "signs": ["women 'seduced by the illusions of demons' believe they ride by night with Diana over vast distances"], + "origins": "A pagan goddess reinterpreted as a demonic power: object of demon-worship at Ephesus (SP, Acts 19); leader of the witches' night-court (DJ, quoting Regino of Prum).", + "sources": ["SP", "DJ"], + "source_count": 2, + "citations": { + "SP": "Ch. IX p. 100", + "DJ": "Bk III Ch V (~p. 173) & note 1" + }, + "cross_refs": ["python"], + "notes": null + }, + { + "id": "apollyon", + "name": "Apollyon", + "aliases": ["Abaddon"], + "type": "individual", + "tier": "other-recurring", + "rank": "Prince ruling the seventh hierarchy (the Furies); king of the abyss", + "legions": null, + "goetia_number": null, + "deadly_sin": null, + "domain": ["death", "destruction", "war and devastation", "the bottomless pit"], + "signs": ["king of the bottomless pit (Rev 9:11)"], + "origins": "Hebrew Abaddon, 'place of destruction', personified; Greek Apollyon, 'the destroyer'; equated with Satan/Samael. Listed as a name of Satan in SP.", + "sources": ["SP", "EN"], + "source_count": 2, + "citations": { + "SP": "p. 28 (Satan's names)", + "EN": "'Abaddon' ~437" + }, + "cross_refs": ["satan", "samael"], + "notes": null + }, + { + "id": "legion", + "name": "Legion", + "aliases": ["the Gerasene Demon"], + "type": "collective", + "tier": "other-recurring", + "rank": "a multitude acting as one", + "legions": null, + "goetia_number": null, + "deadly_sin": null, + "domain": ["demonic possession"], + "signs": ["'many demons as one' (Mark 5)", "a persistent negotiator who bargained with Jesus", "transferred into the herd of swine"], + "origins": "The collective of demons possessing the Gerasene demoniac; cited as proof of demonic personhood (Unger).", + "sources": ["FG", "PU"], + "source_count": 2, + "citations": { + "PU": "§1.7 (~lines 2434-2450)", + "FG": "PSYCHE (ALL NAMES)" + }, + "cross_refs": [], + "notes": null + }, + { + "id": "shedim", + "name": "Shedim / Seirim", + "aliases": ["Shedhim", "Seirim", "goat demons", "satyrs"], + "type": "class", + "tier": "other-recurring", + "rank": "a class of wilderness demons, ruled by Asmodeus", + "legions": null, + "goetia_number": null, + "deadly_sin": null, + "domain": ["recipients of idolatrous child-sacrifice", "haunting of ruins and waste places"], + "signs": ["hairy, horned, wide-mouthed, with lolling tongues and wings", "'eat, drink, marry, and die like men'"], + "origins": "Hebrew wilderness demons: the seirim ('goat demons'/'satyrs', Lev 17:7, Isa 34:14) and shedhim (Deut 32:17, Ps 106:37). Azazel is their king.", + "sources": ["FG", "PU", "EN"], + "source_count": 3, + "citations": { + "PU": "§2.5.2 (~lines 3285-3317)", + "FG": "FOREST 'Shedim', pp. 125-127", + "EN": "'Shedim' ~22557" + }, + "cross_refs": ["azazel", "asmodeus"], + "notes": null + }, + { + "id": "ahriman", + "name": "Ahriman", + "aliases": ["Angra Mainyu", "the Destructive Spirit"], + "type": "individual", + "tier": "other-recurring", + "rank": "the demon of all demons (Zoroastrian)", + "legions": null, + "goetia_number": null, + "deadly_sin": null, + "domain": ["drought", "famine", "war", "disease", "source of all evil"], + "signs": ["created 99,999 diseases, six archdemons, and the dragon Azhi Dahaka", "falls unconscious 3,000 years at the Ahunvar chant"], + "origins": "Twin of Ahura Mazda; commands the daeva. His lieutenant Aeshma is the root of the Hebrew Asmodeus.", + "sources": ["FG", "EN"], + "source_count": 2, + "citations": { + "FG": "(ALL NAMES, 'Ahriman / Angra Mainyu')", + "EN": "'Ahriman' ~695" + }, + "cross_refs": ["asmodeus"], + "notes": "Ties the Zoroastrian tradition back to the supreme tier via Aeshma/Asmodeus." + }, + { + "id": "pazuzu", + "name": "Pazuzu", + "aliases": [], + "type": "individual", + "tier": "other-recurring", + "rank": null, + "legions": null, + "goetia_number": null, + "deadly_sin": null, + "domain": ["wind", "plague"], + "signs": [], + "origins": "Mesopotamian wind/plague demon. Little rank detail given in either source.", + "sources": ["FG", "EN"], + "source_count": 2, + "citations": { + "FG": "DOMICILE (ALL NAMES)", + "EN": "'Pazuzu'" + }, + "cross_refs": [], + "notes": "Included for completeness as a 2-source recurrence; sparse attributes." + } + ] +} diff --git a/md/demonology/demons.schema.json b/md/demonology/demons.schema.json new file mode 100644 index 0000000..37a25a2 --- /dev/null +++ b/md/demonology/demons.schema.json @@ -0,0 +1,138 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://demonology.local/demons.schema.json", + "title": "Demonology Entity Dataset", + "description": "Machine-readable roster of demonic entities cross-referenced across the source documents in md/demonology/. Each record is a fact-cache derived from whole-file reads of the sources, so AI agents can query attributes without re-reading the corpus. Scope: entities attested in 2 or more sources.", + "type": "object", + "required": ["meta", "sources", "demons"], + "properties": { + "meta": { + "type": "object", + "required": ["topic", "scope", "rank_ladder", "generated_from"], + "properties": { + "topic": { "type": "string" }, + "scope": { + "type": "string", + "description": "Inclusion rule for this dataset, e.g. 'entities attested in 2+ sources'." + }, + "rank_ladder": { + "type": "array", + "description": "Classical grimoire ranks, highest authority first.", + "items": { "type": "string" } + }, + "generated_from": { + "type": "array", + "description": "Sibling files this data was derived from / kept in sync with.", + "items": { "type": "string" } + }, + "caveat": { "type": "string" } + } + }, + "sources": { + "type": "object", + "description": "Map of source code -> human-readable source description. Codes are used in each demon's 'sources' array.", + "patternProperties": { + "^[A-Z]{2}$": { "type": "string" } + }, + "additionalProperties": false + }, + "demons": { + "type": "array", + "items": { "$ref": "#/$defs/demon" } + } + }, + "$defs": { + "sourceCode": { + "type": "string", + "enum": ["FG", "DM", "PU", "SP", "DJ", "EN"] + }, + "demon": { + "type": "object", + "required": ["id", "name", "type", "tier", "domain", "signs", "origins", "sources", "source_count", "citations"], + "additionalProperties": false, + "properties": { + "id": { + "type": "string", + "pattern": "^[a-z0-9-]+$", + "description": "Stable kebab-case identifier; used as the target of cross_refs." + }, + "name": { "type": "string", "description": "Primary / most common name." }, + "aliases": { + "type": "array", + "items": { "type": "string" } + }, + "type": { + "type": "string", + "enum": ["individual", "collective", "class", "title"], + "description": "individual = a single named entity; collective = a named group acting as one (e.g. Legion); class = a category of spirits (e.g. Shedim); title = an honorific/role applied to others." + }, + "tier": { + "type": "string", + "enum": ["supreme-prince", "directional-king", "goetia-officer", "watcher", "other-recurring"], + "description": "Which sub-hierarchy this entity belongs to in the synthesis." + }, + "rank": { + "type": ["string", "null"], + "description": "Grimoire rank as stated (King, Duke, Prince, Marquis, Earl, President, Knight, Emperor, etc.). Null if none stated." + }, + "legions": { + "type": ["integer", "null"], + "description": "Number of legions commanded, if stated." + }, + "goetia_number": { + "type": ["integer", "null"], + "minimum": 1, + "maximum": 72, + "description": "Position among the 72 Spirits of the Ars Goetia, if applicable." + }, + "deadly_sin": { + "type": ["string", "null"], + "description": "Associated deadly sin, for the seven Princes of Hell scheme." + }, + "domain": { + "type": "array", + "description": "Spheres of influence / what it governs.", + "items": { "type": "string" } + }, + "signs": { + "type": "array", + "description": "Identifying marks: apparition form, associations, sigils, manner of manifesting.", + "items": { "type": "string" } + }, + "origins": { + "type": "string", + "description": "Etymology, backstory, derivation from earlier deities/traditions." + }, + "sources": { + "type": "array", + "minItems": 2, + "uniqueItems": true, + "items": { "$ref": "#/$defs/sourceCode" }, + "description": "Source codes that name this entity. Scope rule requires length >= 2." + }, + "source_count": { + "type": "integer", + "minimum": 2, + "description": "Number of distinct sources; must equal sources.length." + }, + "citations": { + "type": "object", + "description": "Map of source code -> location string within that source file.", + "patternProperties": { + "^[A-Z]{2}$": { "type": "string" } + }, + "additionalProperties": false + }, + "cross_refs": { + "type": "array", + "description": "ids of entities this one is identified/conflated with, or commands/serves.", + "items": { "type": "string", "pattern": "^[a-z0-9-]+$" } + }, + "notes": { + "type": ["string", "null"], + "description": "Uncertainty flags, conflation warnings, or scope caveats." + } + } + } + } +} diff --git a/requirements.txt b/requirements.txt index 2bfba19..dbd1d35 100644 --- a/requirements.txt +++ b/requirements.txt @@ -5,6 +5,10 @@ # text PDFs without a tessdata install and pulls heavy deps we don't want here. pymupdf4llm==0.3.4 +# Validates topic datasets (md//.json) against their JSON Schemas. +# Used by validate.py; lightweight, pure-Python. +jsonschema>=4.0 + # Fallbacks (install only if needed, see README): # markitdown # DOCX/PPTX/XLSX/HTML -> md # marker-pdf # heavier, ML/GPU, OCRs scanned PDFs diff --git a/validate.py b/validate.py new file mode 100644 index 0000000..4ae662a --- /dev/null +++ b/validate.py @@ -0,0 +1,173 @@ +#!/usr/bin/env python3 +"""Validate topic datasets against their JSON Schemas. + +The research center keeps machine-readable fact-caches alongside the markdown in +each ``md//`` folder. This script checks them so an AI or a generation +script can edit data and confirm it is still well-formed in one step. + +Convention (the only wiring needed per topic): + + md//.schema.json validates its sibling + md//.json + +Drop a ``*.schema.json`` next to a ``*.json`` and it is picked up automatically. + +Two layers of checking: + +1. **Schema** — full JSON Schema validation (structure, types, enums, required + fields, ``minItems`` etc.). +2. **House rules** — cross-field invariants JSON Schema can't easily express, + applied generically to any list-of-records that uses the conventional field + names: + * ``source_count`` must equal ``len(sources)`` when both are present. + * every ``cross_refs`` entry must reference an ``id`` that exists in the + same dataset. + * ``id`` values must be unique within a dataset. + +Usage: + ./.venv/bin/python validate.py # every topic under md/ + ./.venv/bin/python validate.py demonology # one topic + ./.venv/bin/python validate.py md/demonology/demons.json # one file + +Exit code is non-zero if anything fails, so it works as a pre-commit hook or CI +step. +""" + +from __future__ import annotations + +import json +import sys +from pathlib import Path + +try: + import jsonschema +except ImportError: + sys.exit( + "jsonschema not installed. Run: ./.venv/bin/pip install -r requirements.txt" + ) + +REPO = Path(__file__).resolve().parent +MD = REPO / "md" +SCHEMA_SUFFIX = ".schema.json" + + +def find_pairs(target: str | None) -> list[tuple[Path, Path]]: + """Return (schema, data) path pairs to validate. + + ``target`` may be a topic name, a path to a topic dir, a path to a data + file, or None (whole repo). + """ + schemas: list[Path] + if target is None: + schemas = sorted(MD.glob(f"**/*{SCHEMA_SUFFIX}")) + else: + p = Path(target) + if p.is_file() and p.name.endswith(SCHEMA_SUFFIX): + schemas = [p] + elif p.is_file() and p.suffix == ".json": + schemas = [p.with_name(p.name[: -len(".json")] + SCHEMA_SUFFIX)] + else: + # treat as topic name or topic dir + topic_dir = p if p.is_dir() else MD / target + if not topic_dir.is_dir(): + sys.exit(f"No such topic or path: {target!r} (looked in {topic_dir})") + schemas = sorted(topic_dir.glob(f"*{SCHEMA_SUFFIX}")) + + pairs = [] + for schema in schemas: + data = schema.with_name(schema.name[: -len(SCHEMA_SUFFIX)] + ".json") + pairs.append((schema, data)) + return pairs + + +def iter_records(obj): + """Yield every dict that looks like a record (has an 'id') anywhere in obj.""" + if isinstance(obj, dict): + if "id" in obj: + yield obj + for v in obj.values(): + yield from iter_records(v) + elif isinstance(obj, list): + for v in obj: + yield from iter_records(v) + + +def house_rules(data) -> list[str]: + """Cross-field invariants JSON Schema can't express. Returns error strings.""" + errors = [] + records = list(iter_records(data)) + ids = [r["id"] for r in records] + id_set = set(ids) + + for rid in ids: + if ids.count(rid) > 1: + errors.append(f"duplicate id: {rid!r}") + # dedupe the duplicate-id messages + errors = sorted(set(errors)) + + for r in records: + rid = r.get("id", "") + if "source_count" in r and "sources" in r: + if r["source_count"] != len(r["sources"]): + errors.append( + f"{rid}: source_count {r['source_count']} != len(sources) {len(r['sources'])}" + ) + for ref in r.get("cross_refs", []): + if ref not in id_set: + errors.append(f"{rid}: cross_ref -> missing id {ref!r}") + return errors + + +def validate_pair(schema_path: Path, data_path: Path) -> list[str]: + errors = [] + if not data_path.exists(): + return [f"data file missing for schema: {data_path}"] + try: + schema = json.loads(schema_path.read_text()) + except json.JSONDecodeError as e: + return [f"schema is not valid JSON: {e}"] + try: + data = json.loads(data_path.read_text()) + except json.JSONDecodeError as e: + return [f"data is not valid JSON: {e}"] + + validator = jsonschema.Draft202012Validator(schema) + for err in sorted(validator.iter_errors(data), key=lambda e: list(e.path)): + loc = "/".join(str(p) for p in err.path) or "" + errors.append(f"schema [{loc}]: {err.message}") + + errors.extend(house_rules(data)) + return errors + + +def main() -> int: + target = sys.argv[1] if len(sys.argv) > 1 else None + pairs = find_pairs(target) + if not pairs: + print("No *.schema.json files found — nothing to validate.") + return 0 + + failed = 0 + for schema_path, data_path in pairs: + rel = data_path.relative_to(REPO) if data_path.is_relative_to(REPO) else data_path + errors = validate_pair(schema_path, data_path) + if errors: + failed += 1 + print(f"FAIL {rel}") + for e in errors: + print(f" - {e}") + else: + n = len(list(iter_records(json.loads(data_path.read_text())))) + print(f"OK {rel} ({n} records)") + + print() + total = len(pairs) + if failed: + print(f"{failed}/{total} dataset(s) failed validation.") + return 1 + print(f"All {total} dataset(s) valid.") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main())