# CLAUDE.md

Operating guide for Claude Code in this repo. See `README.md` for the full
project description.

## What this is

An AI-assisted research center. Source material lives by topic, gets converted to
markdown, and **you read whole markdown files** to cross-reference and synthesize
across a topic. This is deliberately *not* chunked RAG — read entire files.

## Repo map

- `pdfs/<topic>/*.pdf` — source PDFs. **Gitignored** (large, often copyrighted). Local only.
- `md/<topic>/*.md` — converted markdown **and** syntheses you write. This is what you read.
- `convert.py` — batch PDF→markdown converter.
- `requirements.txt` — pins `pymupdf4llm==0.3.4` (lightweight line; see Gotchas).
- `needs-ocr.txt` — generated list of no-text-layer PDFs. Gitignored.
- `.venv/` — virtualenv. Gitignored.

## Environment

Always use the venv. It is already built.

```bash
source .venv/bin/activate          # or call ./.venv/bin/python directly
```

If a package is missing, `pip install -r requirements.txt`.

## Converting PDFs

```bash
./.venv/bin/python convert.py            # pdfs/ -> md/, idempotent (skips current files)
./.venv/bin/python convert.py --force    # reconvert everything
```

- Idempotent on mtime — safe to rerun; only new/changed PDFs convert.
- Adding a topic: `mkdir pdfs/<topic>`, drop PDFs in, run `convert.py`.
- After converting, check the summary line and `needs-ocr.txt` for anything flagged.

## Doing synthesis (the main job)

When asked to synthesize a topic:
1. Read **every** file under `md/<topic>/` — full files, not snippets.
2. Cross-reference across documents; cite using page boundaries (`-----`
   separators mark PDF pages) and the source name in each file's header.
3. Save the result as a new markdown file **in that topic folder**, e.g.
   `md/<topic>/<slug>-synthesis.md`, unless told otherwise.
4. Syntheses are part of the record — they get committed.

## Structured data (fact-caches)

A topic may carry a machine-readable **fact-cache** alongside its markdown so you
can answer attribute questions without re-reading the corpus. Convention, per
topic folder `md/<topic>/`:

- `<name>.json` — facts, one record per entity, each with `sources` +
  `citations` back to the source files.
- `<name>.schema.json` — JSON Schema; validates the sibling `<name>.json`.
- `INDEX.md` — read this **first**: query the `.json` for facts → the
  `*-synthesis.md` for narrative → a source `.md` only for verbatim quotes.

Validate after any edit: `./.venv/bin/python validate.py <topic>` (or no arg for
the whole repo). Edit the **JSON first**, validate, then update prose.

Full pattern + how to add it to a new topic: **`docs/data-convention.md`**.
Worked example: `md/demonology/`.

## Git conventions

- Treat this repo as code: **everything goes in git** except what `.gitignore`
  excludes (PDFs, `.venv/`, `.claude/`, `needs-ocr.txt`).
- Commit converted markdown and syntheses. Do **not** commit source PDFs.
- Commit when a unit of work is done (a topic converted, a synthesis written).
  Don't push unless asked.

## Gotchas

- **pymupdf4llm is pinned to `0.3.4` on purpose.** The `1.27.x` releases bundle
  an ML layout/OCR pipeline (onnxruntime + Tesseract) that fails on plain text
  PDFs without a tessdata install. Don't "upgrade" it without testing a full
  conversion pass.
- **Plain-text fallback:** on some PDFs pymupdf4llm emits near-empty markdown
  despite a real text layer. `convert.py` detects this and falls back to plain
  per-page text (logged `[plain-text fallback]`). Those files lose
  heading/table structure but keep all text — still fine to read.
- **For poor conversions or scans:** use `marker-pdf`/`docling` (OCR + layout) or
  `markitdown` (DOCX/PPTX/XLSX/HTML) on the single problem file, then drop the
  result into the matching `md/<topic>/` path. See README → Fallbacks.