Claude-driven web research over local SearXNG + Firecrawl. web.py mechanical
fetch layer (stdlib-only, 19 tests), /research skill drives expand→search→
rank→scrape→manifest→synthesize into md/<topic>/web/.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-driven web fan-out over local SearXNG + Firecrawl, saving cited
sources and a synthesis into md/<topic>/. Skill is the brain, web.py is
the mechanical fetch layer. Uses Claude Code subscription, no LLM API.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce a machine-readable layer on top of the markdown corpus so AI/scripts
can query a topic's facts without re-reading whole sources (anti-RAG stays for
synthesis/quotes).
- md/demonology/demons.json: fact-cache, 33 entities attested in 2+ sources,
each with rank/domain/signs/origins + provenance (sources, citations).
- md/demonology/demons.schema.json: JSON Schema for the dataset.
- md/demonology/INDEX.md: topic front-door (query JSON -> synthesis -> source).
- validate.py: generic schema + house-rule validator (source_count, cross_refs,
unique ids); discovers <name>.schema.json/<name>.json pairs across all topics.
- docs/data-convention.md: the reusable, topic-agnostic pattern + how to add it
to a new topic.
- CLAUDE.md: pointer so the convention is picked up every session.
- requirements.txt: add jsonschema (used by validate.py).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Session guide: venv usage, convert.py workflow, whole-file synthesis
convention (read all md/<topic>/, save syntheses back in-folder), git
habits, and the pymupdf4llm pin + plain-text fallback gotchas.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Document the project as a git-backed research workspace: collect sources by
topic, convert to markdown, synthesize with Claude over whole files, commit
everything. Topics open-ended; demonology is the primary one today.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Topic-per-subfolder layout so the same workflow scales to other subjects.
Converter already mirrors structure; rerun is a no-op (idempotent).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
convert.py walks pdfs/ (recursing topic subfolders), mirrors a .md tree
into md/ via pymupdf4llm, idempotent on mtime. Detects no-text-layer PDFs
(needs-ocr.txt) and falls back to plain per-page text when pymupdf4llm's
layout pass returns near-empty despite a real text layer.
Pin pymupdf4llm==0.3.4 (lightweight line; 1.27.x bundles an ML/OCR
pipeline that fails on plain text PDFs). PDFs gitignored (copyrighted,
large) — only generated markdown is committed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
