ptarrant
04010d5ee2
Document the project as a git-backed research workspace: collect sources by topic, convert to markdown, synthesize with Claude over whole files, commit everything. Topics open-ended; demonology is the primary one today. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
106 lines
4.2 KiB
Markdown
106 lines
4.2 KiB
Markdown
# My AI-Assisted Research Center
|
|
|
|
A local, **git-backed** research workspace where I collect source material by
|
|
topic, convert it to clean markdown, and have **Claude Code read whole files** to
|
|
cross-reference and synthesize across a subject — instead of chunked RAG, which
|
|
gave shallow results.
|
|
|
|
Everything that matters is versioned in git: the converted markdown, the
|
|
conversion tooling, and any syntheses Claude produces. Source PDFs stay local
|
|
(see below). The repo is the durable record; Claude is the analyst working over
|
|
it.
|
|
|
|
**Topics:** TBD and growing. The primary one today is **demonology**. The layout
|
|
is topic-per-folder so the same workflow scales to any subject — add a folder,
|
|
drop in sources, convert, ask.
|
|
|
|
## How it works
|
|
|
|
1. **Collect** — drop text PDFs into `pdfs/<topic>/`.
|
|
2. **Convert** — `python convert.py` mirrors them into `md/<topic>/` as markdown
|
|
(headings, lists, tables, and `-----` page boundaries preserved).
|
|
3. **Synthesize** — point Claude Code at `md/<topic>/` and ask it to read the
|
|
whole topic and produce something. It saves the result back into the folder.
|
|
4. **Commit** — markdown sources and syntheses go into git.
|
|
|
|
## Layout
|
|
|
|
```
|
|
research-center/
|
|
pdfs/<topic>/*.pdf # source PDFs (gitignored — kept local, not committed)
|
|
md/<topic>/*.md # converted markdown + saved syntheses — what Claude reads
|
|
convert.py # batch PDF→markdown converter
|
|
requirements.txt # pins the converter (pymupdf4llm)
|
|
needs-ocr.txt # generated: PDFs with no text layer (gitignored)
|
|
README.md
|
|
```
|
|
|
|
Group sources into topic subfolders under `pdfs/` (e.g. `pdfs/demonology/`,
|
|
`pdfs/alchemy/`). The converter mirrors that structure into `md/`. A flat
|
|
`pdfs/` (no subfolders) works too — it just produces a flat `md/`.
|
|
|
|
**Add a new topic:**
|
|
|
|
```bash
|
|
mkdir pdfs/<topic> # drop PDFs in
|
|
python convert.py # converts only the new files into md/<topic>/
|
|
```
|
|
|
|
> **PDFs are gitignored.** They are large and often copyrighted, so only the
|
|
> generated markdown is committed. Keep your PDFs backed up outside git. To
|
|
> version the PDFs too, remove `pdfs/` from `.gitignore` (consider git-lfs first).
|
|
|
|
## Setup
|
|
|
|
```bash
|
|
python3 -m venv .venv
|
|
source .venv/bin/activate
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
## Convert
|
|
|
|
```bash
|
|
source .venv/bin/activate
|
|
python convert.py # pdfs/ -> md/, idempotent
|
|
python convert.py --force # reconvert everything
|
|
python convert.py --src other --out other-md
|
|
```
|
|
|
|
Behavior:
|
|
- **Recurses** `pdfs/` and mirrors the folder structure into `md/`.
|
|
- **Idempotent**: skips a PDF whose `.md` exists and is newer than the PDF.
|
|
- **Scan detection**: PDFs with ~no extractable text are logged to
|
|
`needs-ocr.txt` and left unconverted (no empty markdown) — see Fallbacks.
|
|
- **Plain-text fallback**: on some PDFs pymupdf4llm's layout pass emits almost
|
|
nothing despite a real text layer. When its output is implausibly small versus
|
|
the raw extractable text, `convert.py` falls back to plain per-page text
|
|
(same `-----` page separators, marked `[plain-text fallback]` in the log).
|
|
Structure (headings/tables) is lost but the text is not.
|
|
- Prints a summary: converted / skipped / flagged-for-OCR (/ failed).
|
|
|
|
## Using it with Claude Code
|
|
|
|
Per topic, ask things like:
|
|
|
|
> "Read everything under `md/demonology/` and cross-reference the documents to
|
|
> produce <specific outcome>, then save the result as a markdown file in that
|
|
> folder."
|
|
|
|
The markdown keeps headings, lists, tables, and page boundaries (`-----`
|
|
separators) so Claude can cite locations while reading entire files. Syntheses
|
|
Claude writes land alongside the sources in `md/<topic>/` and get committed too —
|
|
so the research center accumulates both raw material and worked analysis.
|
|
|
|
## Fallbacks
|
|
|
|
`convert.py` uses **pymupdf4llm** (fast, no ML deps, best for clean text PDFs).
|
|
If a PDF lands in `needs-ocr.txt`, or converts poorly (garbled tables/layout),
|
|
use a heavier tool on just that file:
|
|
|
|
- **scanned / no text layer** → `marker-pdf` or `docling` (OCR + layout).
|
|
- **DOCX/PPTX/XLSX/HTML** sources → `markitdown`.
|
|
|
|
Install on demand (see commented lines in `requirements.txt`), convert the
|
|
problem file, and drop the result into the matching `md/<topic>/` path.
|