research/first_prompt.md

Project: PDF research library → markdown for Claude-assisted synthesis

Goal

I'm building a local, git-tracked research workflow. I have folders of text PDFs (real text layer, not scans) grouped by topic. I want to convert them to markdown so that you (Claude Code) can read the actual files and answer cross-reference / synthesis questions across a topic — instead of using a RAG tool with a small local model, which gave shallow results.

Treat this repo as code. Everything goes in git.

Tooling decision (already made — don't re-litigate unless something breaks)

• Primary converter: pymupdf4llm — pip install, no ML deps, fast, LLM-oriented markdown (keeps headings/lists/tables, can mark page boundaries). Best fit for clean text PDFs.
• Fallbacks if needed:
  • markitdown (Microsoft) — if I also need DOCX/PPTX/XLSX/HTML → md.
  • marker or docling — heavier, ML/GPU, also OCR scanned PDFs and handle messy layouts/tables better. Use only for PDFs that pymupdf4llm handles poorly or that turn out to be scans (no text layer).

What I want you to build

1. A small batch converter (convert.py or similar):
   • Input: a source dir of PDFs (recursing into topic subfolders).
   • Output: mirrored .md files under an output dir, preserving the topic folder structure.
   • Idempotent: skip a PDF if its .md already exists and is newer than the PDF.
   • Detect no-text-layer PDFs (pymupdf4llm yields little/no text) and log them to a needs-ocr.txt list instead of writing empty markdown — those need the marker/OCR path.
   • Print a summary: converted / skipped / flagged-for-ocr counts.
2. A requirements.txt pinning pymupdf4llm (and a venv setup note in the README).
3. A short README.md documenting the workflow and folder layout.

Suggested repo layout (adjust to what you find)

research-library/
  pdfs/<topic>/*.pdf        # source (consider git-lfs or .gitignore if large)
  md/<topic>/*.md           # converted markdown — the stuff you'll read
  convert.py
  requirements.txt
  needs-ocr.txt             # generated
  README.md

Recommend whether to commit the source PDFs (size?), git-lfs them, or gitignore them and commit only the markdown. Ask me if it's a judgment call on size.

How I'll use it afterward

Per topic, I'll ask you things like: "Read everything under md/<topic>/ and cross-reference the documents to produce , then save the result as a markdown file in that folder." So optimize the markdown for you reading whole files, not for chunked retrieval.

First steps for you

1. Look at the current folder/repo state and tell me what's here.
2. Confirm/adjust the layout above.
3. Write convert.py, requirements.txt, README.md.
4. Run a conversion pass and report the summary + anything flagged in needs-ocr.txt.

That'll get the desktop session productive immediately. When you're ready to actually do synthesis, just point it at md// and ask — that's the part jarvis was failing at, and it's exactly what reading-whole-files is good for.