claude_skills/deep-dive-research/SKILL.md

name: deep-dive-research description: Use when asked to deeply research a topic toward a specific goal — fan out searches on the homelab SearXNG box, extract the pages with the homelab Firecrawl box, save the raw sources locally, then compile/synthesize an answer that satisfies the goal. Invoke with a topic and a goal, e.g. /deep-dive-research angels — goal: compile a cited list of angels in Christian/Biblical texts with name, rank, role, and supporting quotes. Do NOT use for interactive UI testing (that's browser-testing) or for a single quick web lookup.

Deep-dive research (SearXNG + Firecrawl)

Research a topic in service of a goal (the why — the shape of the answer you must produce). This skill fans out real searches on the homelab SearXNG box, extracts the resulting pages with the homelab Firecrawl box, saves every source to disk under deep_dive_research/<topic>/, then reads those sources and synthesizes a cited answer built specifically to meet the goal.

Both boxes are on the webtools VM (10.10.20.37, VLAN 20, no auth — LAN-gated). They must be reachable from this host.

• SearXNG — http://10.10.20.37:8080/search?q=<query>&format=json (JSON output is enabled server-side)
• Firecrawl — POST http://10.10.20.37:3002/v1/scrape, body {"url": "...", "formats": ["markdown"]}

On Invoke

Parse the argument into two parts:

• topic — the subject (drives the output folder name).
• goal — why you're researching + the exact deliverable shape (fields, format, what "done" looks like). Look for goal:, — goal:, or an obvious "I want …" clause.

If the goal is missing or vague, ask one to three clarifying questions before doing any work — the goal determines the search queries, which sources matter, and the synthesis format. Do not guess it.

Derive the topic slug: lowercase, spaces/punctuation → hyphens (e.g. Biblical Angels → angels or biblical-angels). This is <topic> for all paths below.

Output layout (relative to repo root; create dirs as needed):

deep_dive_research/<topic>/
  sources/           # one markdown file per extracted page (raw Firecrawl output)
  manifest.md        # index: query → URL → local file, with fetch status
  REPORT.md          # the synthesized answer to the goal

Process

digraph {
  rankdir=LR;
  queries -> search -> select -> extract -> synthesize -> report;
  queries [label="1. Frame\n5-7 queries"];
  search  [label="2. SearXNG\nJSON search"];
  select  [label="3. Rank +\ndedupe URLs"];
  extract [label="4. Firecrawl\nscrape → files"];
  synthesize [label="5. Read\nsources"];
  report  [label="6. REPORT.md\n+ summary"];
}

1. Frame the search queries

From the topic and the goal, write 5-7 distinct search queries that attack the topic from different angles — don't paraphrase one query five times. Cover, as relevant: the core term, authoritative/primary sources, taxonomies/lists, specific sub-facets the goal calls for, and counter/edge perspectives. Briefly list the queries to the user before searching.

Example (goal = cited list of Biblical angels): list of angels in the Bible, angelic hierarchy ranks orders Christian theology, archangels names scripture references, seraphim cherubim thrones Bible verses, named angels Michael Gabriel Raphael role, angels Book of Enoch Deuterocanonical, angel roles duties biblical texts.

2. Search each query on SearXNG

For each query, hit the JSON API and collect the top results (title + url + snippet). Use curl + jq:

Q="angelic hierarchy ranks Christian theology"
curl -sG "http://10.10.20.37:8080/search" \
  --data-urlencode "q=$Q" \
  --data-urlencode "format=json" \
  | jq -r '.results[:8][] | "\(.url)\t\(.title)"'

Run the queries (parallel curls are fine). If SearXNG returns an empty results array for everything, the box may be down — check reachability (curl -s -o /dev/null -w '%{http_code}' http://10.10.20.37:8080/) and tell the user rather than proceeding with nothing.

3. Rank and dedupe URLs

Pool all results, dedupe by URL/domain, and select the most relevant, authoritative pages toward the goal — prefer primary sources, reference works, and substantive articles over listicle/SEO spam. Cap the set at ~8-15 URLs so extraction stays bounded; note in manifest.md if you dropped others. Record the query→URL mapping.

4. Extract pages with Firecrawl → save to disk

Scrape each selected URL to markdown and write it to sources/<n>-<domain-slug>.md, prefixed with a small header (source URL, originating query, fetch date). Example per URL:

URL="https://en.wikipedia.org/wiki/Hierarchy_of_angels"
curl -s -X POST "http://10.10.20.37:3002/v1/scrape" \
  -H "Content-Type: application/json" \
  -d "{\"url\": \"$URL\", \"formats\": [\"markdown\"]}" \
  | jq -r '.data.markdown // .data.content // empty'

Guidance:

• Extract sequentially or in small batches (2-4 at a time), not a giant parallel burst — the playwright backend is patched to waitUntil: 'domcontentloaded' but ad-heavy pages can still be slow. Give each a reasonable --max-time (e.g. 60s).
• On a failure/empty/timeout for a URL: record it in manifest.md as failed and move on — don't let one dead page stall the run. Optionally substitute the next-best URL from step 3.
• For a large set (>~8 pages), dispatch the extraction across a few parallel Agent sub-agents (each handles a slice of URLs, writes its files, and returns a one-line-per-URL status). Keep the raw markdown out of your own context — you'll read the files back in step 5.
• Write manifest.md as you go: a table of # | query | url | file | status.

5. Synthesize — read the sources, meet the goal

Now read the saved sources/*.md files (this is the point of saving them — work from the extracted text, not memory or snippets). Extract exactly what the goal asks for, cite as you go: for every claim, capture the supporting quote/passage and which source file + URL it came from. Cross-check across sources; note conflicts or thin evidence rather than papering over them.

Structure the output to match the goal's requested shape. If the goal implies fields (name / rank / role / supporting text / source), build a table or per-entry section with exactly those fields. Include a short direct quote from the source document to support each claim where the goal asks for citations.

6. Write REPORT.md and summarize

Write deep_dive_research/<topic>/REPORT.md:

• Goal restated at the top.
• The synthesized answer in the goal's shape (table / structured list / narrative), with inline citations [n] or (source: file / URL).
• A Sources section listing every source file + URL used.
• A short Gaps / caveats note: failed fetches, unresolved conflicts, claims resting on weak evidence.

Then give the user a brief chat summary (headline findings + where the full report lives) — don't paste the whole report into chat.

Notes & guardrails

• No auth on either box — LAN-gated only. If a curl connection refuses/times out, the box is down or you're off VLAN 20; report that, don't silently produce an empty report.
• Firecrawl response shape can vary by version; the markdown is under .data.markdown. If jq yields empty, inspect the raw JSON once (| jq '.') to find the right key before looping.
• <topic> folder is intentionally under the repo root (deep_dive_research/), not docs/. Create it if absent. These research artifacts are data outputs, not committed source — don't commit them unless the user asks.
• Keep the raw extracted pages on disk even after writing the report; they're the audit trail for every cited claim.

When NOT to use this

• One quick fact / single lookup — just search once; this is for multi-source deep dives.
• Interactive UI / login / click-through testing — that's the browser-testing (Playwright MCP) skill.
• Firecrawl or SearXNG box is unreachable — fix connectivity first; this skill depends on both.