research/docs/data-convention.md

# Structured data convention — fact-caches per topic

How this research center stores **machine-readable facts** on top of its
markdown, so an AI agent (or a script) can query a topic without re-reading the
whole source corpus. Topic-agnostic: the same pattern is meant to repeat for
every research topic.

## Why this exists

The repo's core workflow is **anti-RAG**: Claude reads *whole* markdown files to
synthesize across a topic (see `CLAUDE.md`). That is the right tool for
*synthesis over source text* — but it is expensive, and re-reading megabytes of
markdown every session just to recall "what rank is Asmodeus" is wasteful.

So each topic may carry a **derived fact-cache**: a structured JSON file that
captures the facts extracted during a thorough read, once, so later sessions load
a small structured file instead of the corpus. The cache does **not** replace the
sources — it sits on top of them and points back to them with citations.

- **Sources** (`md/<topic>/*.md`) = ground truth for *text*. Read whole, for
  synthesis and verbatim quotes.
- **Fact-cache** (`md/<topic>/<name>.json`) = ground truth for *facts*. Query
  for attributes; never needs the corpus re-read.
- **Synthesis** (`md/<topic>/*-synthesis.md`) = ground truth for *narrative*.
  The human-readable argument, derived from the same facts.

## The file set (per topic)

Inside a topic folder `md/<topic>/`:

| File | Role | Required? |
|------|------|-----------|
| `<name>.json` | The fact-cache: one object per entity, fields + citations. | The data |
| `<name>.schema.json` | JSON Schema for `<name>.json`. Self-documents the shape; enables validation. | Yes, pairs with the data |
| `INDEX.md` | Topic front-door: tells an agent which artifact to use for what. | Recommended |
| `*-synthesis.md` | Prose synthesis derived from the data. | Optional |

**Naming convention (the only wiring):** a file named `<name>.schema.json`
validates its sibling `<name>.json` in the same folder. `validate.py` discovers
pairs by this rule alone — no central registry to maintain.

Worked example: `md/demonology/` contains `demons.json`, `demons.schema.json`,
`INDEX.md`, and `demon-hierarchy-synthesis.md`.

## How an AI agent should use a topic

Read the topic's `INDEX.md` first, then follow this order:

1. **Need a fact / attribute / to filter or sort?** → query the `.json`
   (e.g. with `jq`). Do *not* read the sources.
2. **Need the narrative or how sources disagree?** → read the `*-synthesis.md`.
3. **Need a verbatim quote or a detail not in the data?** → open the *one* source
   `.md` named in the entity's `citations`, at the cited location. Read it whole
   (repo convention), not a snippet.

This keeps the expensive whole-corpus read for when it is actually needed
(quotes, fresh synthesis) and serves everything else from ~tens of KB of JSON.

## Record shape

The schema is the authority, but the house style for an entity record is:

- a stable `id` (kebab-case) — the target of any `cross_refs`
- a primary `name` plus `aliases`
- the substantive fields for the topic (ranks, domains, signs, origins…)
- **provenance on every record:** `sources` (array of source codes),
  `source_count`, and `citations` (source code → location string)
- `cross_refs` — `id`s of related/conflated entities
- `notes` — uncertainty flags, conflation warnings, scope caveats

Each dataset also carries a top-level `meta` block (scope rule, caveats,
`generated_from`) and a `sources` map (source code → human description) so the
file is self-explanatory in isolation.

## Validation

`validate.py` (repo root) checks every topic's datasets:

```bash
./.venv/bin/python validate.py              # every topic under md/
./.venv/bin/python validate.py <topic>      # one topic
./.venv/bin/python validate.py md/<topic>/<name>.json   # one file
```

It applies two layers:

1. **JSON Schema** — structure, types, enums, required fields, `minItems`, etc.
2. **House rules** (cross-field invariants the schema can't express, applied to
   any record that uses the conventional field names):
   - `source_count` must equal `len(sources)`
   - every `cross_refs` entry must reference an `id` that exists in the dataset
   - `id`s must be unique

Exit code is non-zero on failure, so it works as a git pre-commit hook or CI
step. Requires `jsonschema` (in `requirements.txt`).

> Validation lives in a **standalone** script, not in `convert.py`, on purpose:
> conversion is triggered by PDF changes, dataset edits are triggered by
> extraction/AI — different events. The validator is what the *editor* calls.

## Adding the pattern to a new topic

1. Convert sources as usual: `./.venv/bin/python convert.py` → `md/<topic>/`.
2. Extract facts (AI or script) into `md/<topic>/<name>.json`, giving every
   record `sources` + `source_count` + `citations`.
3. Write `md/<topic>/<name>.schema.json` describing that shape (copy
   `md/demonology/demons.schema.json` as a starting point and adapt fields).
4. Add a short `md/<topic>/INDEX.md` (copy the demonology one and adjust).
5. Validate: `./.venv/bin/python validate.py <topic>`.
6. Commit the markdown, the JSON, the schema, and the index. (Sources/PDFs stay
   gitignored as always.)

## Keeping facts and prose in sync

The `.json` is the source of truth for **facts**; the `*-synthesis.md` is the
source of truth for **narrative**. When something changes, edit the **JSON
first**, re-validate, then reflect it in the prose — not the other way around. A
fact that lives only in prose is a fact the machine can't see.