ptarrant
057c96e10b
convert.py walks pdfs/ (recursing topic subfolders), mirrors a .md tree into md/ via pymupdf4llm, idempotent on mtime. Detects no-text-layer PDFs (needs-ocr.txt) and falls back to plain per-page text when pymupdf4llm's layout pass returns near-empty despite a real text layer. Pin pymupdf4llm==0.3.4 (lightweight line; 1.27.x bundles an ML/OCR pipeline that fails on plain text PDFs). PDFs gitignored (copyrighted, large) — only generated markdown is committed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
62 lines
3.1 KiB
Markdown
62 lines
3.1 KiB
Markdown
# Project: PDF research library → markdown for Claude-assisted synthesis
|
|
|
|
## Goal
|
|
I'm building a local, git-tracked research workflow. I have folders of **text PDFs**
|
|
(real text layer, not scans) grouped by topic. I want to convert them to markdown so
|
|
that you (Claude Code) can read the actual files and answer cross-reference / synthesis
|
|
questions across a topic — instead of using a RAG tool with a small local model, which
|
|
gave shallow results.
|
|
|
|
Treat this repo as code. Everything goes in git.
|
|
|
|
## Tooling decision (already made — don't re-litigate unless something breaks)
|
|
- **Primary converter: `pymupdf4llm`** — pip install, no ML deps, fast, LLM-oriented
|
|
markdown (keeps headings/lists/tables, can mark page boundaries). Best fit for clean
|
|
text PDFs.
|
|
- Fallbacks if needed:
|
|
- `markitdown` (Microsoft) — if I also need DOCX/PPTX/XLSX/HTML → md.
|
|
- `marker` or `docling` — heavier, ML/GPU, **also OCR scanned PDFs** and handle messy
|
|
layouts/tables better. Use only for PDFs that pymupdf4llm handles poorly or that
|
|
turn out to be scans (no text layer).
|
|
|
|
## What I want you to build
|
|
1. A small **batch converter** (`convert.py` or similar):
|
|
- Input: a source dir of PDFs (recursing into topic subfolders).
|
|
- Output: mirrored `.md` files under an output dir, preserving the topic folder
|
|
structure.
|
|
- **Idempotent**: skip a PDF if its `.md` already exists and is newer than the PDF.
|
|
- **Detect no-text-layer PDFs** (pymupdf4llm yields little/no text) and log them to a
|
|
`needs-ocr.txt` list instead of writing empty markdown — those need the marker/OCR
|
|
path.
|
|
- Print a summary: converted / skipped / flagged-for-ocr counts.
|
|
2. A `requirements.txt` pinning `pymupdf4llm` (and a venv setup note in the README).
|
|
3. A short `README.md` documenting the workflow and folder layout.
|
|
|
|
## Suggested repo layout (adjust to what you find)
|
|
```
|
|
research-library/
|
|
pdfs/<topic>/*.pdf # source (consider git-lfs or .gitignore if large)
|
|
md/<topic>/*.md # converted markdown — the stuff you'll read
|
|
convert.py
|
|
requirements.txt
|
|
needs-ocr.txt # generated
|
|
README.md
|
|
```
|
|
Recommend whether to commit the source PDFs (size?), git-lfs them, or gitignore them and
|
|
commit only the markdown. Ask me if it's a judgment call on size.
|
|
|
|
## How I'll use it afterward
|
|
Per topic, I'll ask you things like: "Read everything under `md/<topic>/` and
|
|
cross-reference the documents to produce <specific outcome>, then save the result as a
|
|
markdown file in that folder." So optimize the markdown for *you* reading whole files,
|
|
not for chunked retrieval.
|
|
|
|
## First steps for you
|
|
1. Look at the current folder/repo state and tell me what's here.
|
|
2. Confirm/adjust the layout above.
|
|
3. Write `convert.py`, `requirements.txt`, `README.md`.
|
|
4. Run a conversion pass and report the summary + anything flagged in `needs-ocr.txt`.
|
|
|
|
That'll get the desktop session productive immediately. When you're ready to actually do
|
|
synthesis, just point it at md/<topic>/ and ask — that's the part jarvis was failing at, and
|
|
it's exactly what reading-whole-files is good for. |