added deep-dive-research

deep-dive-research/SKILL.md

@@ -0,0 +1,119 @@
+---
+name: deep-dive-research
+description: Use when asked to deeply research a topic toward a specific goal — fan out searches on the homelab SearXNG box, extract the pages with the homelab Firecrawl box, save the raw sources locally, then compile/synthesize an answer that satisfies the goal. Invoke with a topic and a goal, e.g. /deep-dive-research angels — goal: compile a cited list of angels in Christian/Biblical texts with name, rank, role, and supporting quotes. Do NOT use for interactive UI testing (that's browser-testing) or for a single quick web lookup.
+---
+
+# Deep-dive research (SearXNG + Firecrawl)
+
+Research a **topic** in service of a **goal** (the *why* — the shape of the answer you must produce). This skill fans out real searches on the homelab **SearXNG** box, extracts the resulting pages with the homelab **Firecrawl** box, saves every source to disk under `deep_dive_research/<topic>/`, then reads those sources and synthesizes a cited answer built specifically to meet the goal.
+
+Both boxes are on the **webtools** VM (`10.10.20.37`, VLAN 20, no auth — LAN-gated). They must be reachable from this host.
+
+- **SearXNG** — `http://10.10.20.37:8080/search?q=<query>&format=json` (JSON output is enabled server-side)
+- **Firecrawl** — `POST http://10.10.20.37:3002/v1/scrape`, body `{"url": "...", "formats": ["markdown"]}`
+
+## On Invoke
+
+Parse the argument into two parts:
+
+- **topic** — the subject (drives the output folder name).
+- **goal** — *why* you're researching + the exact deliverable shape (fields, format, what "done" looks like). Look for `goal:`, `— goal:`, or an obvious "I want …" clause.
+
+If the **goal is missing or vague**, ask one to three clarifying questions before doing any work — the goal determines the search queries, which sources matter, and the synthesis format. Do not guess it.
+
+**Derive the topic slug**: lowercase, spaces/punctuation → hyphens (e.g. `Biblical Angels` → `angels` or `biblical-angels`). This is `<topic>` for all paths below.
+
+**Output layout** (relative to repo root; create dirs as needed):
+
+```
+deep_dive_research/<topic>/
+  sources/           # one markdown file per extracted page (raw Firecrawl output)
+  manifest.md        # index: query → URL → local file, with fetch status
+  REPORT.md          # the synthesized answer to the goal
+```
+
+## Process
+
+```dot
+digraph {
+  rankdir=LR;
+  queries -> search -> select -> extract -> synthesize -> report;
+  queries [label="1. Frame\n5-7 queries"];
+  search  [label="2. SearXNG\nJSON search"];
+  select  [label="3. Rank +\ndedupe URLs"];
+  extract [label="4. Firecrawl\nscrape → files"];
+  synthesize [label="5. Read\nsources"];
+  report  [label="6. REPORT.md\n+ summary"];
+}
+```
+
+### 1. Frame the search queries
+
+From the topic **and the goal**, write **5-7 distinct search queries** that attack the topic from different angles — don't paraphrase one query five times. Cover, as relevant: the core term, authoritative/primary sources, taxonomies/lists, specific sub-facets the goal calls for, and counter/edge perspectives. Briefly list the queries to the user before searching.
+
+> Example (goal = cited list of Biblical angels): `list of angels in the Bible`, `angelic hierarchy ranks orders Christian theology`, `archangels names scripture references`, `seraphim cherubim thrones Bible verses`, `named angels Michael Gabriel Raphael role`, `angels Book of Enoch Deuterocanonical`, `angel roles duties biblical texts`.
+
+### 2. Search each query on SearXNG
+
+For each query, hit the JSON API and collect the top results (title + url + snippet). Use `curl` + `jq`:
+
+```bash
+Q="angelic hierarchy ranks Christian theology"
+curl -sG "http://10.10.20.37:8080/search" \
+  --data-urlencode "q=$Q" \
+  --data-urlencode "format=json" \
+  | jq -r '.results[:8][] | "\(.url)\t\(.title)"'
+```
+
+Run the queries (parallel `curl`s are fine). If SearXNG returns an empty `results` array for everything, the box may be down — check reachability (`curl -s -o /dev/null -w '%{http_code}' http://10.10.20.37:8080/`) and tell the user rather than proceeding with nothing.
+
+### 3. Rank and dedupe URLs
+
+Pool all results, **dedupe by URL/domain**, and select the most relevant, authoritative pages toward the goal — prefer primary sources, reference works, and substantive articles over listicle/SEO spam. **Cap the set at ~8-15 URLs** so extraction stays bounded; note in `manifest.md` if you dropped others. Record the query→URL mapping.
+
+### 4. Extract pages with Firecrawl → save to disk
+
+Scrape each selected URL to markdown and write it to `sources/<n>-<domain-slug>.md`, prefixed with a small header (source URL, originating query, fetch date). Example per URL:
+
+```bash
+URL="https://en.wikipedia.org/wiki/Hierarchy_of_angels"
+curl -s -X POST "http://10.10.20.37:3002/v1/scrape" \
+  -H "Content-Type: application/json" \
+  -d "{\"url\": \"$URL\", \"formats\": [\"markdown\"]}" \
+  | jq -r '.data.markdown // .data.content // empty'
+```
+
+Guidance:
+- Extract **sequentially or in small batches** (2-4 at a time), not a giant parallel burst — the playwright backend is patched to `waitUntil: 'domcontentloaded'` but ad-heavy pages can still be slow. Give each a reasonable `--max-time` (e.g. 60s).
+- On a failure/empty/timeout for a URL: record it in `manifest.md` as failed and move on — don't let one dead page stall the run. Optionally substitute the next-best URL from step 3.
+- **For a large set (>~8 pages)**, dispatch the extraction across a few parallel `Agent` sub-agents (each handles a slice of URLs, writes its files, and returns a one-line-per-URL status). Keep the raw markdown out of your own context — you'll read the files back in step 5.
+- Write `manifest.md` as you go: a table of `# | query | url | file | status`.
+
+### 5. Synthesize — read the sources, meet the goal
+
+Now read the saved `sources/*.md` files (this is the point of saving them — work from the extracted text, not memory or snippets). Extract exactly what the goal asks for, **cite as you go**: for every claim, capture the supporting quote/passage and which source file + URL it came from. Cross-check across sources; note conflicts or thin evidence rather than papering over them.
+
+Structure the output to match the goal's requested shape. If the goal implies fields (name / rank / role / supporting text / source), build a **table or per-entry section** with exactly those fields. Include a short direct quote from the source document to support each claim where the goal asks for citations.
+
+### 6. Write REPORT.md and summarize
+
+Write `deep_dive_research/<topic>/REPORT.md`:
+- **Goal** restated at the top.
+- The synthesized answer in the goal's shape (table / structured list / narrative), with inline citations `[n]` or `(source: file / URL)`.
+- A **Sources** section listing every source file + URL used.
+- A short **Gaps / caveats** note: failed fetches, unresolved conflicts, claims resting on weak evidence.
+
+Then give the user a **brief** chat summary (headline findings + where the full report lives) — don't paste the whole report into chat.
+
+## Notes & guardrails
+
+- **No auth** on either box — LAN-gated only. If a `curl` connection refuses/times out, the box is down or you're off VLAN 20; report that, don't silently produce an empty report.
+- Firecrawl response shape can vary by version; the markdown is under `.data.markdown`. If `jq` yields empty, inspect the raw JSON once (`| jq '.'`) to find the right key before looping.
+- `<topic>` folder is intentionally under the **repo root** (`deep_dive_research/`), not `docs/`. Create it if absent. These research artifacts are data outputs, not committed source — don't commit them unless the user asks.
+- Keep the raw extracted pages on disk even after writing the report; they're the audit trail for every cited claim.
+
+## When NOT to use this
+
+- **One quick fact / single lookup** — just search once; this is for multi-source deep dives.
+- **Interactive UI / login / click-through testing** — that's the `browser-testing` (Playwright MCP) skill.
+- **Firecrawl or SearXNG box is unreachable** — fix connectivity first; this skill depends on both.