research/CLAUDE.md

CLAUDE.md

Operating guide for Claude Code in this repo. See README.md for the full project description.

What this is

An AI-assisted research center. Source material lives by topic, gets converted to markdown, and you read whole markdown files to cross-reference and synthesize across a topic. This is deliberately not chunked RAG — read entire files.

Repo map

• pdfs/<topic>/*.pdf — source PDFs. Gitignored (large, often copyrighted). Local only.
• md/<topic>/*.md — converted markdown and syntheses you write. This is what you read.
• convert.py — batch PDF→markdown converter.
• requirements.txt — pins pymupdf4llm==0.3.4 (lightweight line; see Gotchas).
• needs-ocr.txt — generated list of no-text-layer PDFs. Gitignored.
• .venv/ — virtualenv. Gitignored.

Environment

Always use the venv. It is already built.

source .venv/bin/activate          # or call ./.venv/bin/python directly

If a package is missing, pip install -r requirements.txt.

Converting PDFs

./.venv/bin/python convert.py            # pdfs/ -> md/, idempotent (skips current files)
./.venv/bin/python convert.py --force    # reconvert everything

• Idempotent on mtime — safe to rerun; only new/changed PDFs convert.
• Adding a topic: mkdir pdfs/<topic>, drop PDFs in, run convert.py.
• After converting, check the summary line and needs-ocr.txt for anything flagged.

Doing synthesis (the main job)

When asked to synthesize a topic:

1. Read every file under md/<topic>/ — full files, not snippets.
2. Cross-reference across documents; cite using page boundaries (----- separators mark PDF pages) and the source name in each file's header.
3. Save the result as a new markdown file in that topic folder, e.g. md/<topic>/<slug>-synthesis.md, unless told otherwise.
4. Syntheses are part of the record — they get committed.

Structured data (fact-caches)

A topic may carry a machine-readable fact-cache alongside its markdown so you can answer attribute questions without re-reading the corpus. Convention, per topic folder md/<topic>/:

• <name>.json — facts, one record per entity, each with sources + citations back to the source files.
• <name>.schema.json — JSON Schema; validates the sibling <name>.json.
• INDEX.md — read this first: query the .json for facts → the *-synthesis.md for narrative → a source .md only for verbatim quotes.

Validate after any edit: ./.venv/bin/python validate.py <topic> (or no arg for the whole repo). Edit the JSON first, validate, then update prose.

Full pattern + how to add it to a new topic: docs/data-convention.md. Worked example: md/demonology/.

Git conventions

• Treat this repo as code: everything goes in git except what .gitignore excludes (PDFs, .venv/, .claude/, needs-ocr.txt).
• Commit converted markdown and syntheses. Do not commit source PDFs.
• Commit when a unit of work is done (a topic converted, a synthesis written). Don't push unless asked.

Gotchas

• pymupdf4llm is pinned to 0.3.4 on purpose. The 1.27.x releases bundle an ML layout/OCR pipeline (onnxruntime + Tesseract) that fails on plain text PDFs without a tessdata install. Don't "upgrade" it without testing a full conversion pass.
• Plain-text fallback: on some PDFs pymupdf4llm emits near-empty markdown despite a real text layer. convert.py detects this and falls back to plain per-page text (logged [plain-text fallback]). Those files lose heading/table structure but keep all text — still fine to read.
• For poor conversions or scans: use marker-pdf/docling (OCR + layout) or markitdown (DOCX/PPTX/XLSX/HTML) on the single problem file, then drop the result into the matching md/<topic>/ path. See README → Fallbacks.