Add PDF→markdown batch converter and research-library workflow

convert.py walks pdfs/ (recursing topic subfolders), mirrors a .md tree
into md/ via pymupdf4llm, idempotent on mtime. Detects no-text-layer PDFs
(needs-ocr.txt) and falls back to plain per-page text when pymupdf4llm's
layout pass returns near-empty despite a real text layer.

Pin pymupdf4llm==0.3.4 (lightweight line; 1.27.x bundles an ML/OCR
pipeline that fails on plain text PDFs). PDFs gitignored (copyrighted,
large) — only generated markdown is committed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-26 15:24:22 -05:00
commit 057c96e10b
11 changed files with 83616 additions and 0 deletions

11
requirements.txt Normal file
View File

@@ -0,0 +1,11 @@
# Primary converter for clean text PDFs -> LLM-oriented markdown.
# Pulls pymupdf + tabulate transitively; no separate pin needed.
# NOTE: pinned to the lightweight 0.3.x line on purpose. The 1.27.x releases
# bundle an ML layout/OCR pipeline (onnxruntime + Tesseract) that fails on plain
# text PDFs without a tessdata install and pulls heavy deps we don't want here.
pymupdf4llm==0.3.4
# Fallbacks (install only if needed, see README):
# markitdown # DOCX/PPTX/XLSX/HTML -> md
# marker-pdf # heavier, ML/GPU, OCRs scanned PDFs
# docling # heavier, ML/GPU, messy layouts/tables