My AI-Assisted Research Center

A local, git-backed research workspace where I collect source material by topic, convert it to clean markdown, and have Claude Code read whole files to cross-reference and synthesize across a subject — instead of chunked RAG, which gave shallow results.

Everything that matters is versioned in git: the converted markdown, the conversion tooling, and any syntheses Claude produces. Source PDFs stay local (see below). The repo is the durable record; Claude is the analyst working over it.

Topics: TBD and growing. The primary one today is demonology. The layout is topic-per-folder so the same workflow scales to any subject — add a folder, drop in sources, convert, ask.

How it works

1. Collect — drop text PDFs into pdfs/<topic>/.
2. Convert — python convert.py mirrors them into md/<topic>/ as markdown (headings, lists, tables, and ----- page boundaries preserved).
3. Synthesize — point Claude Code at md/<topic>/ and ask it to read the whole topic and produce something. It saves the result back into the folder.
4. Commit — markdown sources and syntheses go into git.

Layout

research-center/
  pdfs/<topic>/*.pdf     # source PDFs (gitignored — kept local, not committed)
  md/<topic>/*.md        # converted markdown + saved syntheses — what Claude reads
  convert.py             # batch PDF→markdown converter
  requirements.txt       # pins the converter (pymupdf4llm)
  needs-ocr.txt          # generated: PDFs with no text layer (gitignored)
  README.md

Group sources into topic subfolders under pdfs/ (e.g. pdfs/demonology/, pdfs/alchemy/). The converter mirrors that structure into md/. A flat pdfs/ (no subfolders) works too — it just produces a flat md/.

Add a new topic:

mkdir pdfs/<topic>     # drop PDFs in
python convert.py      # converts only the new files into md/<topic>/

PDFs are gitignored. They are large and often copyrighted, so only the generated markdown is committed. Keep your PDFs backed up outside git. To version the PDFs too, remove pdfs/ from .gitignore (consider git-lfs first).

Setup

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Convert

source .venv/bin/activate
python convert.py                 # pdfs/ -> md/, idempotent
python convert.py --force         # reconvert everything
python convert.py --src other --out other-md

Behavior:

• Recurses pdfs/ and mirrors the folder structure into md/.
• Idempotent: skips a PDF whose .md exists and is newer than the PDF.
• Scan detection: PDFs with ~no extractable text are logged to needs-ocr.txt and left unconverted (no empty markdown) — see Fallbacks.
• Plain-text fallback: on some PDFs pymupdf4llm's layout pass emits almost nothing despite a real text layer. When its output is implausibly small versus the raw extractable text, convert.py falls back to plain per-page text (same ----- page separators, marked [plain-text fallback] in the log). Structure (headings/tables) is lost but the text is not.
• Prints a summary: converted / skipped / flagged-for-OCR (/ failed).

Using it with Claude Code

Per topic, ask things like:

"Read everything under md/demonology/ and cross-reference the documents to produce , then save the result as a markdown file in that folder."

The markdown keeps headings, lists, tables, and page boundaries (----- separators) so Claude can cite locations while reading entire files. Syntheses Claude writes land alongside the sources in md/<topic>/ and get committed too — so the research center accumulates both raw material and worked analysis.

Fallbacks

convert.py uses pymupdf4llm (fast, no ML deps, best for clean text PDFs). If a PDF lands in needs-ocr.txt, or converts poorly (garbled tables/layout), use a heavier tool on just that file:

• scanned / no text layer → marker-pdf or docling (OCR + layout).
• DOCX/PPTX/XLSX/HTML sources → markitdown.

Install on demand (see commented lines in requirements.txt), convert the problem file, and drop the result into the matching md/<topic>/ path.