ptarrant
2ea33cff61
Introduce a machine-readable layer on top of the markdown corpus so AI/scripts can query a topic's facts without re-reading whole sources (anti-RAG stays for synthesis/quotes). - md/demonology/demons.json: fact-cache, 33 entities attested in 2+ sources, each with rank/domain/signs/origins + provenance (sources, citations). - md/demonology/demons.schema.json: JSON Schema for the dataset. - md/demonology/INDEX.md: topic front-door (query JSON -> synthesis -> source). - validate.py: generic schema + house-rule validator (source_count, cross_refs, unique ids); discovers <name>.schema.json/<name>.json pairs across all topics. - docs/data-convention.md: the reusable, topic-agnostic pattern + how to add it to a new topic. - CLAUDE.md: pointer so the convention is picked up every session. - requirements.txt: add jsonschema (used by validate.py). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
91 lines
3.8 KiB
Markdown
91 lines
3.8 KiB
Markdown
# CLAUDE.md
|
|
|
|
Operating guide for Claude Code in this repo. See `README.md` for the full
|
|
project description.
|
|
|
|
## What this is
|
|
|
|
An AI-assisted research center. Source material lives by topic, gets converted to
|
|
markdown, and **you read whole markdown files** to cross-reference and synthesize
|
|
across a topic. This is deliberately *not* chunked RAG — read entire files.
|
|
|
|
## Repo map
|
|
|
|
- `pdfs/<topic>/*.pdf` — source PDFs. **Gitignored** (large, often copyrighted). Local only.
|
|
- `md/<topic>/*.md` — converted markdown **and** syntheses you write. This is what you read.
|
|
- `convert.py` — batch PDF→markdown converter.
|
|
- `requirements.txt` — pins `pymupdf4llm==0.3.4` (lightweight line; see Gotchas).
|
|
- `needs-ocr.txt` — generated list of no-text-layer PDFs. Gitignored.
|
|
- `.venv/` — virtualenv. Gitignored.
|
|
|
|
## Environment
|
|
|
|
Always use the venv. It is already built.
|
|
|
|
```bash
|
|
source .venv/bin/activate # or call ./.venv/bin/python directly
|
|
```
|
|
|
|
If a package is missing, `pip install -r requirements.txt`.
|
|
|
|
## Converting PDFs
|
|
|
|
```bash
|
|
./.venv/bin/python convert.py # pdfs/ -> md/, idempotent (skips current files)
|
|
./.venv/bin/python convert.py --force # reconvert everything
|
|
```
|
|
|
|
- Idempotent on mtime — safe to rerun; only new/changed PDFs convert.
|
|
- Adding a topic: `mkdir pdfs/<topic>`, drop PDFs in, run `convert.py`.
|
|
- After converting, check the summary line and `needs-ocr.txt` for anything flagged.
|
|
|
|
## Doing synthesis (the main job)
|
|
|
|
When asked to synthesize a topic:
|
|
1. Read **every** file under `md/<topic>/` — full files, not snippets.
|
|
2. Cross-reference across documents; cite using page boundaries (`-----`
|
|
separators mark PDF pages) and the source name in each file's header.
|
|
3. Save the result as a new markdown file **in that topic folder**, e.g.
|
|
`md/<topic>/<slug>-synthesis.md`, unless told otherwise.
|
|
4. Syntheses are part of the record — they get committed.
|
|
|
|
## Structured data (fact-caches)
|
|
|
|
A topic may carry a machine-readable **fact-cache** alongside its markdown so you
|
|
can answer attribute questions without re-reading the corpus. Convention, per
|
|
topic folder `md/<topic>/`:
|
|
|
|
- `<name>.json` — facts, one record per entity, each with `sources` +
|
|
`citations` back to the source files.
|
|
- `<name>.schema.json` — JSON Schema; validates the sibling `<name>.json`.
|
|
- `INDEX.md` — read this **first**: query the `.json` for facts → the
|
|
`*-synthesis.md` for narrative → a source `.md` only for verbatim quotes.
|
|
|
|
Validate after any edit: `./.venv/bin/python validate.py <topic>` (or no arg for
|
|
the whole repo). Edit the **JSON first**, validate, then update prose.
|
|
|
|
Full pattern + how to add it to a new topic: **`docs/data-convention.md`**.
|
|
Worked example: `md/demonology/`.
|
|
|
|
## Git conventions
|
|
|
|
- Treat this repo as code: **everything goes in git** except what `.gitignore`
|
|
excludes (PDFs, `.venv/`, `.claude/`, `needs-ocr.txt`).
|
|
- Commit converted markdown and syntheses. Do **not** commit source PDFs.
|
|
- Commit when a unit of work is done (a topic converted, a synthesis written).
|
|
Don't push unless asked.
|
|
|
|
## Gotchas
|
|
|
|
- **pymupdf4llm is pinned to `0.3.4` on purpose.** The `1.27.x` releases bundle
|
|
an ML layout/OCR pipeline (onnxruntime + Tesseract) that fails on plain text
|
|
PDFs without a tessdata install. Don't "upgrade" it without testing a full
|
|
conversion pass.
|
|
- **Plain-text fallback:** on some PDFs pymupdf4llm emits near-empty markdown
|
|
despite a real text layer. `convert.py` detects this and falls back to plain
|
|
per-page text (logged `[plain-text fallback]`). Those files lose
|
|
heading/table structure but keep all text — still fine to read.
|
|
- **For poor conversions or scans:** use `marker-pdf`/`docling` (OCR + layout) or
|
|
`markitdown` (DOCX/PPTX/XLSX/HTML) on the single problem file, then drop the
|
|
result into the matching `md/<topic>/` path. See README → Fallbacks.
|