# Project: PDF research library → markdown for Claude-assisted synthesis

  ## Goal
  I'm building a local, git-tracked research workflow. I have folders of **text PDFs**
  (real text layer, not scans) grouped by topic. I want to convert them to markdown so
  that you (Claude Code) can read the actual files and answer cross-reference / synthesis
  questions across a topic — instead of using a RAG tool with a small local model, which
  gave shallow results.

  Treat this repo as code. Everything goes in git.

  ## Tooling decision (already made — don't re-litigate unless something breaks)
  - **Primary converter: `pymupdf4llm`** — pip install, no ML deps, fast, LLM-oriented
    markdown (keeps headings/lists/tables, can mark page boundaries). Best fit for clean
    text PDFs.
  - Fallbacks if needed:
    - `markitdown` (Microsoft) — if I also need DOCX/PPTX/XLSX/HTML → md.
    - `marker` or `docling` — heavier, ML/GPU, **also OCR scanned PDFs** and handle messy
      layouts/tables better. Use only for PDFs that pymupdf4llm handles poorly or that
      turn out to be scans (no text layer).

  ## What I want you to build
  1. A small **batch converter** (`convert.py` or similar):
     - Input: a source dir of PDFs (recursing into topic subfolders).
     - Output: mirrored `.md` files under an output dir, preserving the topic folder
  structure.
     - **Idempotent**: skip a PDF if its `.md` already exists and is newer than the PDF.
     - **Detect no-text-layer PDFs** (pymupdf4llm yields little/no text) and log them to a
       `needs-ocr.txt` list instead of writing empty markdown — those need the marker/OCR
  path.
     - Print a summary: converted / skipped / flagged-for-ocr counts.
  2. A `requirements.txt` pinning `pymupdf4llm` (and a venv setup note in the README).
  3. A short `README.md` documenting the workflow and folder layout.

  ## Suggested repo layout (adjust to what you find)
  ```
  research-library/
    pdfs/<topic>/*.pdf        # source (consider git-lfs or .gitignore if large)
    md/<topic>/*.md           # converted markdown — the stuff you'll read
    convert.py
    requirements.txt
    needs-ocr.txt             # generated
    README.md
  ```
  Recommend whether to commit the source PDFs (size?), git-lfs them, or gitignore them and
  commit only the markdown. Ask me if it's a judgment call on size.

  ## How I'll use it afterward
  Per topic, I'll ask you things like: "Read everything under `md/<topic>/` and
  cross-reference the documents to produce <specific outcome>, then save the result as a
  markdown file in that folder." So optimize the markdown for *you* reading whole files,
  not for chunked retrieval.

  ## First steps for you
  1. Look at the current folder/repo state and tell me what's here.
  2. Confirm/adjust the layout above.
  3. Write `convert.py`, `requirements.txt`, `README.md`.
  4. Run a conversion pass and report the summary + anything flagged in `needs-ocr.txt`.

  That'll get the desktop session productive immediately. When you're ready to actually do
  synthesis, just point it at md/<topic>/ and ask — that's the part jarvis was failing at, and
  it's exactly what reading-whole-files is good for.