Research library: PDF → markdown for Claude-assisted synthesis

Local, git-tracked workflow. Drop text PDFs into topic folders, convert them to markdown, then have Claude Code read whole files to cross-reference and synthesize across a topic — instead of chunked RAG, which gave shallow results.

Layout

demonology/
  pdfs/<topic>/*.pdf     # source PDFs (gitignored — kept local, not committed)
  md/<topic>/*.md        # converted markdown — the files Claude reads
  convert.py             # batch converter
  requirements.txt       # pins pymupdf4llm
  needs-ocr.txt          # generated: PDFs with no text layer (gitignored)
  README.md

Group PDFs into topic subfolders under pdfs/ (e.g. pdfs/angelology/). The converter mirrors that structure into md/. A flat pdfs/ (no subfolders) works too — it just produces a flat md/. Current topics: demonology/. Add more by creating a new pdfs/<topic>/ and dropping PDFs in.

PDFs are gitignored. They are large and copyrighted, so only the generated markdown is committed. Keep your PDFs backed up outside git. To version the PDFs too, remove pdfs/ from .gitignore (consider git-lfs first).

Setup

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Convert

source .venv/bin/activate
python convert.py                 # pdfs/ -> md/, idempotent
python convert.py --force         # reconvert everything
python convert.py --src other --out other-md

Behavior:

• Recurses pdfs/ and mirrors the folder structure into md/.
• Idempotent: skips a PDF whose .md exists and is newer than the PDF.
• Scan detection: PDFs with ~no extractable text are logged to needs-ocr.txt and left unconverted (no empty markdown) — see Fallbacks.
• Plain-text fallback: on some PDFs pymupdf4llm's layout pass emits almost nothing despite a real text layer. When its output is implausibly small versus the raw extractable text, convert.py falls back to plain per-page text (same ----- page separators, marked [plain-text fallback] in the log). Structure (headings/tables) is lost but the text is not.
• Prints a summary: converted / skipped / flagged-for-OCR (/ failed).

Using it with Claude Code

Per topic, ask things like:

"Read everything under md/demonology/ and cross-reference the documents to produce , then save the result as a markdown file in that folder."

The markdown keeps headings, lists, tables, and page boundaries (----- separators) so Claude can cite locations while reading entire files.

Fallbacks

convert.py uses pymupdf4llm (fast, no ML deps, best for clean text PDFs). If a PDF lands in needs-ocr.txt, or converts poorly (garbled tables/layout), use a heavier tool on just that file:

• scanned / no text layer → marker-pdf or docling (OCR + layout).
• DOCX/PPTX/XLSX/HTML sources → markitdown.

Install on demand (see commented lines in requirements.txt), convert the problem file, and drop the result into the matching md/<topic>/ path.