Add PDF→markdown batch converter and research-library workflow

convert.py walks pdfs/ (recursing topic subfolders), mirrors a .md tree
into md/ via pymupdf4llm, idempotent on mtime. Detects no-text-layer PDFs
(needs-ocr.txt) and falls back to plain per-page text when pymupdf4llm's
layout pass returns near-empty despite a real text layer.

Pin pymupdf4llm==0.3.4 (lightweight line; 1.27.x bundles an ML/OCR
pipeline that fails on plain text PDFs). PDFs gitignored (copyrighted,
large) — only generated markdown is committed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

.gitignore

@@ -0,0 +1,13 @@
+# Source PDFs kept local only — copyrighted, large. Commit markdown, not PDFs.
+pdfs/
+
+# Generated OCR worklist (regenerated each run)
+needs-ocr.txt
+
+# Claude Code session/harness state
+.claude/
+
+# Python
+.venv/
+__pycache__/
+*.pyc

README.md

@@ -0,0 +1,77 @@
+# Research library: PDF → markdown for Claude-assisted synthesis
+
+Local, git-tracked workflow. Drop text PDFs into topic folders, convert them to
+markdown, then have Claude Code **read whole files** to cross-reference and
+synthesize across a topic — instead of chunked RAG, which gave shallow results.
+
+## Layout
+
+```
+demonology/
+  pdfs/<topic>/*.pdf     # source PDFs (gitignored — kept local, not committed)
+  md/<topic>/*.md        # converted markdown — the files Claude reads
+  convert.py             # batch converter
+  requirements.txt       # pins pymupdf4llm
+  needs-ocr.txt          # generated: PDFs with no text layer (gitignored)
+  README.md
+```
+
+Group PDFs into topic subfolders under `pdfs/` (e.g. `pdfs/angelology/`). The
+converter mirrors that structure into `md/`. A flat `pdfs/` (no subfolders) works
+too — it just produces a flat `md/`. Currently all PDFs sit directly in `pdfs/`.
+
+> **PDFs are gitignored.** They are large and copyrighted, so only the generated
+> markdown is committed. Keep your PDFs backed up outside git. To version the
+> PDFs too, remove `pdfs/` from `.gitignore` (consider git-lfs first).
+
+## Setup
+
+```bash
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+```
+
+## Convert
+
+```bash
+source .venv/bin/activate
+python convert.py                 # pdfs/ -> md/, idempotent
+python convert.py --force         # reconvert everything
+python convert.py --src other --out other-md
+```
+
+Behavior:
+- **Recurses** `pdfs/` and mirrors the folder structure into `md/`.
+- **Idempotent**: skips a PDF whose `.md` exists and is newer than the PDF.
+- **Scan detection**: PDFs with ~no extractable text are logged to
+  `needs-ocr.txt` and left unconverted (no empty markdown) — see Fallbacks.
+- **Plain-text fallback**: on some PDFs pymupdf4llm's layout pass emits almost
+  nothing despite a real text layer. When its output is implausibly small versus
+  the raw extractable text, `convert.py` falls back to plain per-page text
+  (same `-----` page separators, marked `[plain-text fallback]` in the log).
+  Structure (headings/tables) is lost but the text is not.
+- Prints a summary: converted / skipped / flagged-for-OCR (/ failed).
+
+## Using it with Claude Code
+
+Per topic, ask things like:
+
+> "Read everything under `md/demonology/` and cross-reference the documents to
+> produce <specific outcome>, then save the result as a markdown file in that
+> folder."
+
+The markdown keeps headings, lists, tables, and page boundaries (`-----`
+separators) so Claude can cite locations while reading entire files.
+
+## Fallbacks
+
+`convert.py` uses **pymupdf4llm** (fast, no ML deps, best for clean text PDFs).
+If a PDF lands in `needs-ocr.txt`, or converts poorly (garbled tables/layout),
+use a heavier tool on just that file:
+
+- **scanned / no text layer** → `marker-pdf` or `docling` (OCR + layout).
+- **DOCX/PPTX/XLSX/HTML** sources → `markitdown`.
+
+Install on demand (see commented lines in `requirements.txt`), convert the
+problem file, and drop the result into the matching `md/<topic>/` path.

convert.py

@@ -0,0 +1,159 @@
+#!/usr/bin/env python3
+"""Batch-convert a tree of text PDFs to markdown for whole-file LLM reading.
+
+Walks a source directory (recursing into topic subfolders), converts each PDF to
+markdown with pymupdf4llm, and writes a mirrored .md tree under the output dir.
+
+Design notes:
+- Idempotent: a PDF is skipped when its .md already exists and is newer.
+- No-text-layer PDFs (scans) yield ~no extractable text. Those are detected
+  cheaply *before* the expensive markdown pass, logged to needs-ocr.txt, and
+  left unwritten (no empty markdown) so they can go through an OCR tool later.
+- Output is optimized for a model reading the entire file: a small provenance
+  header, then page-separated markdown that keeps headings / lists / tables.
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+
+import pymupdf  # bundled with pymupdf4llm
+import pymupdf4llm
+
+# A scan (image-only PDF) extracts almost nothing. Real text PDFs comfortably
+# exceed this; the threshold only has to separate "basically empty" from "has a
+# text layer", so it is deliberately low to avoid false OCR flags.
+MIN_CHARS_PER_PAGE = 50
+
+
+def extractable_chars(pdf_path: Path) -> tuple[int, int]:
+    """Return (total_text_chars, page_count) using a cheap text extraction."""
+    with pymupdf.open(pdf_path) as doc:
+        pages = doc.page_count
+        total = sum(len(page.get_text("text")) for page in doc)
+    return total, pages
+
+
+def has_text_layer(total_chars: int, pages: int) -> bool:
+    if pages == 0:
+        return False
+    return (total_chars / pages) >= MIN_CHARS_PER_PAGE
+
+
+def md_is_current(pdf_path: Path, md_path: Path) -> bool:
+    """True when md exists and is at least as new as the source PDF."""
+    return md_path.exists() and md_path.stat().st_mtime >= pdf_path.stat().st_mtime
+
+
+def plain_text_markdown(pdf_path: Path) -> str:
+    """Fallback: raw per-page text with the same '-----' page separators.
+
+    Loses heading/list/table structure but never loses the text itself."""
+    parts = []
+    with pymupdf.open(pdf_path) as doc:
+        for page in doc:
+            parts.append(page.get_text("text").strip())
+    return "\n\n-----\n\n".join(parts)
+
+
+def convert_one(pdf_path: Path, md_path: Path, raw_chars: int) -> str:
+    """Write markdown for one PDF. Returns the method used: 'pymupdf4llm' or 'plain'.
+
+    pymupdf4llm gives structured markdown ("-----" page separators preserve page
+    boundaries for citation). On some PDFs its layout pass emits almost nothing
+    even though a text layer exists; when its output is implausibly small versus
+    the raw extractable text, fall back to plain text extraction.
+    """
+    md_path.parent.mkdir(parents=True, exist_ok=True)
+    body = pymupdf4llm.to_markdown(str(pdf_path), write_images=False, show_progress=False)
+    method = "pymupdf4llm"
+    # Generous floor: only trips when pymupdf4llm essentially gave up.
+    if len(body.strip()) < 0.2 * raw_chars:
+        body = plain_text_markdown(pdf_path)
+        method = "plain"
+    header = f"# {pdf_path.stem}\n\n> Source: `{pdf_path.name}`\n\n---\n\n"
+    md_path.write_text(header + body, encoding="utf-8")
+    return method
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(
+        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
+    )
+    parser.add_argument("--src", type=Path, default=Path("pdfs"), help="source dir of PDFs (default: pdfs)")
+    parser.add_argument("--out", type=Path, default=Path("md"), help="output dir for markdown (default: md)")
+    parser.add_argument(
+        "--ocr-list", type=Path, default=Path("needs-ocr.txt"),
+        help="file listing PDFs with no usable text layer (default: needs-ocr.txt)",
+    )
+    parser.add_argument("--force", action="store_true", help="reconvert even if md is current")
+    args = parser.parse_args(argv)
+
+    if not args.src.is_dir():
+        print(f"error: source dir not found: {args.src}", file=sys.stderr)
+        return 2
+
+    pdfs = sorted(args.src.rglob("*.pdf"))
+    if not pdfs:
+        print(f"No PDFs found under {args.src}")
+        return 0
+
+    converted, skipped, failed, plain = 0, 0, 0, 0
+    needs_ocr: list[Path] = []
+
+    for pdf in pdfs:
+        rel = pdf.relative_to(args.src).with_suffix(".md")
+        md_path = args.out / rel
+
+        if not args.force and md_is_current(pdf, md_path):
+            print(f"skip   {rel}  (up to date)")
+            skipped += 1
+            continue
+
+        try:
+            total, pages = extractable_chars(pdf)
+        except Exception as exc:  # unreadable / corrupt PDF
+            print(f"FAIL   {rel}  ({type(exc).__name__}: {exc})")
+            failed += 1
+            continue
+
+        if not has_text_layer(total, pages):
+            print(f"ocr?   {rel}  ({total} chars / {pages} pages -> no text layer)")
+            needs_ocr.append(pdf)
+            continue
+
+        try:
+            method = convert_one(pdf, md_path, total)
+        except Exception as exc:
+            print(f"FAIL   {rel}  ({type(exc).__name__}: {exc})")
+            failed += 1
+            continue
+
+        tag = "  [plain-text fallback]" if method == "plain" else ""
+        if method == "plain":
+            plain += 1
+        print(f"conv   {rel}  ({pages} pages){tag}")
+        converted += 1
+
+    # Rewritten fresh each run: flagged PDFs are re-checked every time since they
+    # never produce markdown to skip on.
+    if needs_ocr:
+        args.ocr_list.write_text("\n".join(str(p) for p in needs_ocr) + "\n", encoding="utf-8")
+    elif args.ocr_list.exists():
+        args.ocr_list.unlink()
+
+    print("\n" + "=" * 48)
+    print(f"converted:       {converted}" + (f"  ({plain} via plain-text fallback)" if plain else ""))
+    print(f"skipped:         {skipped}")
+    print(f"flagged for OCR: {len(needs_ocr)}")
+    if failed:
+        print(f"failed:          {failed}")
+    if needs_ocr:
+        print(f"\nSee {args.ocr_list} -> run these through marker/docling (OCR).")
+    return 1 if failed else 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

first_prompt.md

@@ -0,0 +1,62 @@
+# Project: PDF research library → markdown for Claude-assisted synthesis
+
+  ## Goal
+  I'm building a local, git-tracked research workflow. I have folders of **text PDFs**
+  (real text layer, not scans) grouped by topic. I want to convert them to markdown so
+  that you (Claude Code) can read the actual files and answer cross-reference / synthesis
+  questions across a topic — instead of using a RAG tool with a small local model, which
+  gave shallow results.
+
+  Treat this repo as code. Everything goes in git.
+
+  ## Tooling decision (already made — don't re-litigate unless something breaks)
+  - **Primary converter: `pymupdf4llm`** — pip install, no ML deps, fast, LLM-oriented
+    markdown (keeps headings/lists/tables, can mark page boundaries). Best fit for clean
+    text PDFs.
+  - Fallbacks if needed:
+    - `markitdown` (Microsoft) — if I also need DOCX/PPTX/XLSX/HTML → md.
+    - `marker` or `docling` — heavier, ML/GPU, **also OCR scanned PDFs** and handle messy
+      layouts/tables better. Use only for PDFs that pymupdf4llm handles poorly or that
+      turn out to be scans (no text layer).
+
+  ## What I want you to build
+  1. A small **batch converter** (`convert.py` or similar):
+     - Input: a source dir of PDFs (recursing into topic subfolders).
+     - Output: mirrored `.md` files under an output dir, preserving the topic folder
+  structure.
+     - **Idempotent**: skip a PDF if its `.md` already exists and is newer than the PDF.
+     - **Detect no-text-layer PDFs** (pymupdf4llm yields little/no text) and log them to a
+       `needs-ocr.txt` list instead of writing empty markdown — those need the marker/OCR
+  path.
+     - Print a summary: converted / skipped / flagged-for-ocr counts.
+  2. A `requirements.txt` pinning `pymupdf4llm` (and a venv setup note in the README).
+  3. A short `README.md` documenting the workflow and folder layout.
+
+  ## Suggested repo layout (adjust to what you find)
+  ```
+  research-library/
+    pdfs/<topic>/*.pdf        # source (consider git-lfs or .gitignore if large)
+    md/<topic>/*.md           # converted markdown — the stuff you'll read
+    convert.py
+    requirements.txt
+    needs-ocr.txt             # generated
+    README.md
+  ```
+  Recommend whether to commit the source PDFs (size?), git-lfs them, or gitignore them and
+  commit only the markdown. Ask me if it's a judgment call on size.
+
+  ## How I'll use it afterward
+  Per topic, I'll ask you things like: "Read everything under `md/<topic>/` and
+  cross-reference the documents to produce <specific outcome>, then save the result as a
+  markdown file in that folder." So optimize the markdown for *you* reading whole files,
+  not for chunked retrieval.
+
+  ## First steps for you
+  1. Look at the current folder/repo state and tell me what's here.
+  2. Confirm/adjust the layout above.
+  3. Write `convert.py`, `requirements.txt`, `README.md`.
+  4. Run a conversion pass and report the summary + anything flagged in `needs-ocr.txt`.
+
+  That'll get the desktop session productive immediately. When you're ready to actually do
+  synthesis, just point it at md/<topic>/ and ask — that's the part jarvis was failing at, and
+  it's exactly what reading-whole-files is good for.

requirements.txt

@@ -0,0 +1,11 @@
+# Primary converter for clean text PDFs -> LLM-oriented markdown.
+# Pulls pymupdf + tabulate transitively; no separate pin needed.
+# NOTE: pinned to the lightweight 0.3.x line on purpose. The 1.27.x releases
+# bundle an ML layout/OCR pipeline (onnxruntime + Tesseract) that fails on plain
+# text PDFs without a tessdata install and pulls heavy deps we don't want here.
+pymupdf4llm==0.3.4
+
+# Fallbacks (install only if needed, see README):
+#   markitdown      # DOCX/PPTX/XLSX/HTML -> md
+#   marker-pdf      # heavier, ML/GPU, OCRs scanned PDFs
+#   docling         # heavier, ML/GPU, messy layouts/tables