Add PDF→markdown batch converter and research-library workflow
convert.py walks pdfs/ (recursing topic subfolders), mirrors a .md tree into md/ via pymupdf4llm, idempotent on mtime. Detects no-text-layer PDFs (needs-ocr.txt) and falls back to plain per-page text when pymupdf4llm's layout pass returns near-empty despite a real text layer. Pin pymupdf4llm==0.3.4 (lightweight line; 1.27.x bundles an ML/OCR pipeline that fails on plain text PDFs). PDFs gitignored (copyrighted, large) — only generated markdown is committed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
11
requirements.txt
Normal file
11
requirements.txt
Normal file
@@ -0,0 +1,11 @@
|
||||
# Primary converter for clean text PDFs -> LLM-oriented markdown.
|
||||
# Pulls pymupdf + tabulate transitively; no separate pin needed.
|
||||
# NOTE: pinned to the lightweight 0.3.x line on purpose. The 1.27.x releases
|
||||
# bundle an ML layout/OCR pipeline (onnxruntime + Tesseract) that fails on plain
|
||||
# text PDFs without a tessdata install and pulls heavy deps we don't want here.
|
||||
pymupdf4llm==0.3.4
|
||||
|
||||
# Fallbacks (install only if needed, see README):
|
||||
# markitdown # DOCX/PPTX/XLSX/HTML -> md
|
||||
# marker-pdf # heavier, ML/GPU, OCRs scanned PDFs
|
||||
# docling # heavier, ML/GPU, messy layouts/tables
|
||||
Reference in New Issue
Block a user