research/docs/data-convention.md

Structured data convention — fact-caches per topic

How this research center stores machine-readable facts on top of its markdown, so an AI agent (or a script) can query a topic without re-reading the whole source corpus. Topic-agnostic: the same pattern is meant to repeat for every research topic.

Why this exists

The repo's core workflow is anti-RAG: Claude reads whole markdown files to synthesize across a topic (see CLAUDE.md). That is the right tool for synthesis over source text — but it is expensive, and re-reading megabytes of markdown every session just to recall "what rank is Asmodeus" is wasteful.

So each topic may carry a derived fact-cache: a structured JSON file that captures the facts extracted during a thorough read, once, so later sessions load a small structured file instead of the corpus. The cache does not replace the sources — it sits on top of them and points back to them with citations.

• Sources (md/<topic>/*.md) = ground truth for text. Read whole, for synthesis and verbatim quotes.
• Fact-cache (md/<topic>/<name>.json) = ground truth for facts. Query for attributes; never needs the corpus re-read.
• Synthesis (md/<topic>/*-synthesis.md) = ground truth for narrative. The human-readable argument, derived from the same facts.

The file set (per topic)

Inside a topic folder md/<topic>/:

File	Role	Required?
<name>.json	The fact-cache: one object per entity, fields + citations.	The data
<name>.schema.json	JSON Schema for <name>.json. Self-documents the shape; enables validation.	Yes, pairs with the data
INDEX.md	Topic front-door: tells an agent which artifact to use for what.	Recommended
*-synthesis.md	Prose synthesis derived from the data.	Optional

Naming convention (the only wiring): a file named <name>.schema.json validates its sibling <name>.json in the same folder. validate.py discovers pairs by this rule alone — no central registry to maintain.

Worked example: md/demonology/ contains demons.json, demons.schema.json, INDEX.md, and demon-hierarchy-synthesis.md.

How an AI agent should use a topic

Read the topic's INDEX.md first, then follow this order:

1. Need a fact / attribute / to filter or sort? → query the .json (e.g. with jq). Do not read the sources.
2. Need the narrative or how sources disagree? → read the *-synthesis.md.
3. Need a verbatim quote or a detail not in the data? → open the one source .md named in the entity's citations, at the cited location. Read it whole (repo convention), not a snippet.

This keeps the expensive whole-corpus read for when it is actually needed (quotes, fresh synthesis) and serves everything else from ~tens of KB of JSON.

Record shape

The schema is the authority, but the house style for an entity record is:

• a stable id (kebab-case) — the target of any cross_refs
• a primary name plus aliases
• the substantive fields for the topic (ranks, domains, signs, origins…)
• provenance on every record: sources (array of source codes), source_count, and citations (source code → location string)
• cross_refs — ids of related/conflated entities
• notes — uncertainty flags, conflation warnings, scope caveats

Each dataset also carries a top-level meta block (scope rule, caveats, generated_from) and a sources map (source code → human description) so the file is self-explanatory in isolation.

Validation

validate.py (repo root) checks every topic's datasets:

./.venv/bin/python validate.py              # every topic under md/
./.venv/bin/python validate.py <topic>      # one topic
./.venv/bin/python validate.py md/<topic>/<name>.json   # one file

It applies two layers:

1. JSON Schema — structure, types, enums, required fields, minItems, etc.
2. House rules (cross-field invariants the schema can't express, applied to any record that uses the conventional field names):
   • source_count must equal len(sources)
   • every cross_refs entry must reference an id that exists in the dataset
   • ids must be unique

Exit code is non-zero on failure, so it works as a git pre-commit hook or CI step. Requires jsonschema (in requirements.txt).

Validation lives in a standalone script, not in convert.py, on purpose: conversion is triggered by PDF changes, dataset edits are triggered by extraction/AI — different events. The validator is what the editor calls.

Adding the pattern to a new topic

1. Convert sources as usual: ./.venv/bin/python convert.py → md/<topic>/.
2. Extract facts (AI or script) into md/<topic>/<name>.json, giving every record sources + source_count + citations.
3. Write md/<topic>/<name>.schema.json describing that shape (copy md/demonology/demons.schema.json as a starting point and adapt fields).
4. Add a short md/<topic>/INDEX.md (copy the demonology one and adjust).
5. Validate: ./.venv/bin/python validate.py <topic>.
6. Commit the markdown, the JSON, the schema, and the index. (Sources/PDFs stay gitignored as always.)

Keeping facts and prose in sync

The .json is the source of truth for facts; the *-synthesis.md is the source of truth for narrative. When something changes, edit the JSON first, re-validate, then reflect it in the prose — not the other way around. A fact that lives only in prose is a fact the machine can't see.