Add per-topic structured fact-cache pattern, validator, and docs
Introduce a machine-readable layer on top of the markdown corpus so AI/scripts can query a topic's facts without re-reading whole sources (anti-RAG stays for synthesis/quotes). - md/demonology/demons.json: fact-cache, 33 entities attested in 2+ sources, each with rank/domain/signs/origins + provenance (sources, citations). - md/demonology/demons.schema.json: JSON Schema for the dataset. - md/demonology/INDEX.md: topic front-door (query JSON -> synthesis -> source). - validate.py: generic schema + house-rule validator (source_count, cross_refs, unique ids); discovers <name>.schema.json/<name>.json pairs across all topics. - docs/data-convention.md: the reusable, topic-agnostic pattern + how to add it to a new topic. - CLAUDE.md: pointer so the convention is picked up every session. - requirements.txt: add jsonschema (used by validate.py). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
118
docs/data-convention.md
Normal file
118
docs/data-convention.md
Normal file
@@ -0,0 +1,118 @@
|
||||
# Structured data convention — fact-caches per topic
|
||||
|
||||
How this research center stores **machine-readable facts** on top of its
|
||||
markdown, so an AI agent (or a script) can query a topic without re-reading the
|
||||
whole source corpus. Topic-agnostic: the same pattern is meant to repeat for
|
||||
every research topic.
|
||||
|
||||
## Why this exists
|
||||
|
||||
The repo's core workflow is **anti-RAG**: Claude reads *whole* markdown files to
|
||||
synthesize across a topic (see `CLAUDE.md`). That is the right tool for
|
||||
*synthesis over source text* — but it is expensive, and re-reading megabytes of
|
||||
markdown every session just to recall "what rank is Asmodeus" is wasteful.
|
||||
|
||||
So each topic may carry a **derived fact-cache**: a structured JSON file that
|
||||
captures the facts extracted during a thorough read, once, so later sessions load
|
||||
a small structured file instead of the corpus. The cache does **not** replace the
|
||||
sources — it sits on top of them and points back to them with citations.
|
||||
|
||||
- **Sources** (`md/<topic>/*.md`) = ground truth for *text*. Read whole, for
|
||||
synthesis and verbatim quotes.
|
||||
- **Fact-cache** (`md/<topic>/<name>.json`) = ground truth for *facts*. Query
|
||||
for attributes; never needs the corpus re-read.
|
||||
- **Synthesis** (`md/<topic>/*-synthesis.md`) = ground truth for *narrative*.
|
||||
The human-readable argument, derived from the same facts.
|
||||
|
||||
## The file set (per topic)
|
||||
|
||||
Inside a topic folder `md/<topic>/`:
|
||||
|
||||
| File | Role | Required? |
|
||||
|------|------|-----------|
|
||||
| `<name>.json` | The fact-cache: one object per entity, fields + citations. | The data |
|
||||
| `<name>.schema.json` | JSON Schema for `<name>.json`. Self-documents the shape; enables validation. | Yes, pairs with the data |
|
||||
| `INDEX.md` | Topic front-door: tells an agent which artifact to use for what. | Recommended |
|
||||
| `*-synthesis.md` | Prose synthesis derived from the data. | Optional |
|
||||
|
||||
**Naming convention (the only wiring):** a file named `<name>.schema.json`
|
||||
validates its sibling `<name>.json` in the same folder. `validate.py` discovers
|
||||
pairs by this rule alone — no central registry to maintain.
|
||||
|
||||
Worked example: `md/demonology/` contains `demons.json`, `demons.schema.json`,
|
||||
`INDEX.md`, and `demon-hierarchy-synthesis.md`.
|
||||
|
||||
## How an AI agent should use a topic
|
||||
|
||||
Read the topic's `INDEX.md` first, then follow this order:
|
||||
|
||||
1. **Need a fact / attribute / to filter or sort?** → query the `.json`
|
||||
(e.g. with `jq`). Do *not* read the sources.
|
||||
2. **Need the narrative or how sources disagree?** → read the `*-synthesis.md`.
|
||||
3. **Need a verbatim quote or a detail not in the data?** → open the *one* source
|
||||
`.md` named in the entity's `citations`, at the cited location. Read it whole
|
||||
(repo convention), not a snippet.
|
||||
|
||||
This keeps the expensive whole-corpus read for when it is actually needed
|
||||
(quotes, fresh synthesis) and serves everything else from ~tens of KB of JSON.
|
||||
|
||||
## Record shape
|
||||
|
||||
The schema is the authority, but the house style for an entity record is:
|
||||
|
||||
- a stable `id` (kebab-case) — the target of any `cross_refs`
|
||||
- a primary `name` plus `aliases`
|
||||
- the substantive fields for the topic (ranks, domains, signs, origins…)
|
||||
- **provenance on every record:** `sources` (array of source codes),
|
||||
`source_count`, and `citations` (source code → location string)
|
||||
- `cross_refs` — `id`s of related/conflated entities
|
||||
- `notes` — uncertainty flags, conflation warnings, scope caveats
|
||||
|
||||
Each dataset also carries a top-level `meta` block (scope rule, caveats,
|
||||
`generated_from`) and a `sources` map (source code → human description) so the
|
||||
file is self-explanatory in isolation.
|
||||
|
||||
## Validation
|
||||
|
||||
`validate.py` (repo root) checks every topic's datasets:
|
||||
|
||||
```bash
|
||||
./.venv/bin/python validate.py # every topic under md/
|
||||
./.venv/bin/python validate.py <topic> # one topic
|
||||
./.venv/bin/python validate.py md/<topic>/<name>.json # one file
|
||||
```
|
||||
|
||||
It applies two layers:
|
||||
|
||||
1. **JSON Schema** — structure, types, enums, required fields, `minItems`, etc.
|
||||
2. **House rules** (cross-field invariants the schema can't express, applied to
|
||||
any record that uses the conventional field names):
|
||||
- `source_count` must equal `len(sources)`
|
||||
- every `cross_refs` entry must reference an `id` that exists in the dataset
|
||||
- `id`s must be unique
|
||||
|
||||
Exit code is non-zero on failure, so it works as a git pre-commit hook or CI
|
||||
step. Requires `jsonschema` (in `requirements.txt`).
|
||||
|
||||
> Validation lives in a **standalone** script, not in `convert.py`, on purpose:
|
||||
> conversion is triggered by PDF changes, dataset edits are triggered by
|
||||
> extraction/AI — different events. The validator is what the *editor* calls.
|
||||
|
||||
## Adding the pattern to a new topic
|
||||
|
||||
1. Convert sources as usual: `./.venv/bin/python convert.py` → `md/<topic>/`.
|
||||
2. Extract facts (AI or script) into `md/<topic>/<name>.json`, giving every
|
||||
record `sources` + `source_count` + `citations`.
|
||||
3. Write `md/<topic>/<name>.schema.json` describing that shape (copy
|
||||
`md/demonology/demons.schema.json` as a starting point and adapt fields).
|
||||
4. Add a short `md/<topic>/INDEX.md` (copy the demonology one and adjust).
|
||||
5. Validate: `./.venv/bin/python validate.py <topic>`.
|
||||
6. Commit the markdown, the JSON, the schema, and the index. (Sources/PDFs stay
|
||||
gitignored as always.)
|
||||
|
||||
## Keeping facts and prose in sync
|
||||
|
||||
The `.json` is the source of truth for **facts**; the `*-synthesis.md` is the
|
||||
source of truth for **narrative**. When something changes, edit the **JSON
|
||||
first**, re-validate, then reflect it in the prose — not the other way around. A
|
||||
fact that lives only in prose is a fact the machine can't see.
|
||||
Reference in New Issue
Block a user