From 5f0a542ab221b2ede07b3663ec311ad4da6e9407 Mon Sep 17 00:00:00 2001
From: ptarrant <ptarrant@gmail.com>
Date: Fri, 26 Jun 2026 15:36:26 -0500
Subject: [PATCH] Add CLAUDE.md with operating conventions for Claude Code

Session guide: venv usage, convert.py workflow, whole-file synthesis
convention (read all md/<topic>/, save syntheses back in-folder), git
habits, and the pymupdf4llm pin + plain-text fallback gotchas.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
 CLAUDE.md | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 72 insertions(+)
 create mode 100644 CLAUDE.md

diff --git a/CLAUDE.md b/CLAUDE.md
new file mode 100644
index 0000000..be8457d
--- /dev/null
+++ b/CLAUDE.md
@@ -0,0 +1,72 @@
+# CLAUDE.md
+
+Operating guide for Claude Code in this repo. See `README.md` for the full
+project description.
+
+## What this is
+
+An AI-assisted research center. Source material lives by topic, gets converted to
+markdown, and **you read whole markdown files** to cross-reference and synthesize
+across a topic. This is deliberately *not* chunked RAG — read entire files.
+
+## Repo map
+
+- `pdfs/<topic>/*.pdf` — source PDFs. **Gitignored** (large, often copyrighted). Local only.
+- `md/<topic>/*.md` — converted markdown **and** syntheses you write. This is what you read.
+- `convert.py` — batch PDF→markdown converter.
+- `requirements.txt` — pins `pymupdf4llm==0.3.4` (lightweight line; see Gotchas).
+- `needs-ocr.txt` — generated list of no-text-layer PDFs. Gitignored.
+- `.venv/` — virtualenv. Gitignored.
+
+## Environment
+
+Always use the venv. It is already built.
+
+```bash
+source .venv/bin/activate          # or call ./.venv/bin/python directly
+```
+
+If a package is missing, `pip install -r requirements.txt`.
+
+## Converting PDFs
+
+```bash
+./.venv/bin/python convert.py            # pdfs/ -> md/, idempotent (skips current files)
+./.venv/bin/python convert.py --force    # reconvert everything
+```
+
+- Idempotent on mtime — safe to rerun; only new/changed PDFs convert.
+- Adding a topic: `mkdir pdfs/<topic>`, drop PDFs in, run `convert.py`.
+- After converting, check the summary line and `needs-ocr.txt` for anything flagged.
+
+## Doing synthesis (the main job)
+
+When asked to synthesize a topic:
+1. Read **every** file under `md/<topic>/` — full files, not snippets.
+2. Cross-reference across documents; cite using page boundaries (`-----`
+   separators mark PDF pages) and the source name in each file's header.
+3. Save the result as a new markdown file **in that topic folder**, e.g.
+   `md/<topic>/<slug>-synthesis.md`, unless told otherwise.
+4. Syntheses are part of the record — they get committed.
+
+## Git conventions
+
+- Treat this repo as code: **everything goes in git** except what `.gitignore`
+  excludes (PDFs, `.venv/`, `.claude/`, `needs-ocr.txt`).
+- Commit converted markdown and syntheses. Do **not** commit source PDFs.
+- Commit when a unit of work is done (a topic converted, a synthesis written).
+  Don't push unless asked.
+
+## Gotchas
+
+- **pymupdf4llm is pinned to `0.3.4` on purpose.** The `1.27.x` releases bundle
+  an ML layout/OCR pipeline (onnxruntime + Tesseract) that fails on plain text
+  PDFs without a tessdata install. Don't "upgrade" it without testing a full
+  conversion pass.
+- **Plain-text fallback:** on some PDFs pymupdf4llm emits near-empty markdown
+  despite a real text layer. `convert.py` detects this and falls back to plain
+  per-page text (logged `[plain-text fallback]`). Those files lose
+  heading/table structure but keep all text — still fine to read.
+- **For poor conversions or scans:** use `marker-pdf`/`docling` (OCR + layout) or
+  `markitdown` (DOCX/PPTX/XLSX/HTML) on the single problem file, then drop the
+  result into the matching `md/<topic>/` path. See README → Fallbacks.