Add CLAUDE.md with operating conventions for Claude Code
Session guide: venv usage, convert.py workflow, whole-file synthesis convention (read all md/<topic>/, save syntheses back in-folder), git habits, and the pymupdf4llm pin + plain-text fallback gotchas. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
72
CLAUDE.md
Normal file
72
CLAUDE.md
Normal file
@@ -0,0 +1,72 @@
|
||||
# CLAUDE.md
|
||||
|
||||
Operating guide for Claude Code in this repo. See `README.md` for the full
|
||||
project description.
|
||||
|
||||
## What this is
|
||||
|
||||
An AI-assisted research center. Source material lives by topic, gets converted to
|
||||
markdown, and **you read whole markdown files** to cross-reference and synthesize
|
||||
across a topic. This is deliberately *not* chunked RAG — read entire files.
|
||||
|
||||
## Repo map
|
||||
|
||||
- `pdfs/<topic>/*.pdf` — source PDFs. **Gitignored** (large, often copyrighted). Local only.
|
||||
- `md/<topic>/*.md` — converted markdown **and** syntheses you write. This is what you read.
|
||||
- `convert.py` — batch PDF→markdown converter.
|
||||
- `requirements.txt` — pins `pymupdf4llm==0.3.4` (lightweight line; see Gotchas).
|
||||
- `needs-ocr.txt` — generated list of no-text-layer PDFs. Gitignored.
|
||||
- `.venv/` — virtualenv. Gitignored.
|
||||
|
||||
## Environment
|
||||
|
||||
Always use the venv. It is already built.
|
||||
|
||||
```bash
|
||||
source .venv/bin/activate # or call ./.venv/bin/python directly
|
||||
```
|
||||
|
||||
If a package is missing, `pip install -r requirements.txt`.
|
||||
|
||||
## Converting PDFs
|
||||
|
||||
```bash
|
||||
./.venv/bin/python convert.py # pdfs/ -> md/, idempotent (skips current files)
|
||||
./.venv/bin/python convert.py --force # reconvert everything
|
||||
```
|
||||
|
||||
- Idempotent on mtime — safe to rerun; only new/changed PDFs convert.
|
||||
- Adding a topic: `mkdir pdfs/<topic>`, drop PDFs in, run `convert.py`.
|
||||
- After converting, check the summary line and `needs-ocr.txt` for anything flagged.
|
||||
|
||||
## Doing synthesis (the main job)
|
||||
|
||||
When asked to synthesize a topic:
|
||||
1. Read **every** file under `md/<topic>/` — full files, not snippets.
|
||||
2. Cross-reference across documents; cite using page boundaries (`-----`
|
||||
separators mark PDF pages) and the source name in each file's header.
|
||||
3. Save the result as a new markdown file **in that topic folder**, e.g.
|
||||
`md/<topic>/<slug>-synthesis.md`, unless told otherwise.
|
||||
4. Syntheses are part of the record — they get committed.
|
||||
|
||||
## Git conventions
|
||||
|
||||
- Treat this repo as code: **everything goes in git** except what `.gitignore`
|
||||
excludes (PDFs, `.venv/`, `.claude/`, `needs-ocr.txt`).
|
||||
- Commit converted markdown and syntheses. Do **not** commit source PDFs.
|
||||
- Commit when a unit of work is done (a topic converted, a synthesis written).
|
||||
Don't push unless asked.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **pymupdf4llm is pinned to `0.3.4` on purpose.** The `1.27.x` releases bundle
|
||||
an ML layout/OCR pipeline (onnxruntime + Tesseract) that fails on plain text
|
||||
PDFs without a tessdata install. Don't "upgrade" it without testing a full
|
||||
conversion pass.
|
||||
- **Plain-text fallback:** on some PDFs pymupdf4llm emits near-empty markdown
|
||||
despite a real text layer. `convert.py` detects this and falls back to plain
|
||||
per-page text (logged `[plain-text fallback]`). Those files lose
|
||||
heading/table structure but keep all text — still fine to read.
|
||||
- **For poor conversions or scans:** use `marker-pdf`/`docling` (OCR + layout) or
|
||||
`markitdown` (DOCX/PPTX/XLSX/HTML) on the single problem file, then drop the
|
||||
result into the matching `md/<topic>/` path. See README → Fallbacks.
|
||||
Reference in New Issue
Block a user