From 61e9f23c8065f8fdfee779dbb0a7fd977636dac0 Mon Sep 17 00:00:00 2001 From: Phillip Tarrant Date: Wed, 1 Jul 2026 15:48:51 -0500 Subject: [PATCH] added deep-dive-research --- deep-dive-research/SKILL.md | 119 ++++++++++++++++++++++++++++++++++++ 1 file changed, 119 insertions(+) create mode 100644 deep-dive-research/SKILL.md diff --git a/deep-dive-research/SKILL.md b/deep-dive-research/SKILL.md new file mode 100644 index 0000000..58fec2e --- /dev/null +++ b/deep-dive-research/SKILL.md @@ -0,0 +1,119 @@ +--- +name: deep-dive-research +description: Use when asked to deeply research a topic toward a specific goal — fan out searches on the homelab SearXNG box, extract the pages with the homelab Firecrawl box, save the raw sources locally, then compile/synthesize an answer that satisfies the goal. Invoke with a topic and a goal, e.g. /deep-dive-research angels — goal: compile a cited list of angels in Christian/Biblical texts with name, rank, role, and supporting quotes. Do NOT use for interactive UI testing (that's browser-testing) or for a single quick web lookup. +--- + +# Deep-dive research (SearXNG + Firecrawl) + +Research a **topic** in service of a **goal** (the *why* — the shape of the answer you must produce). This skill fans out real searches on the homelab **SearXNG** box, extracts the resulting pages with the homelab **Firecrawl** box, saves every source to disk under `deep_dive_research//`, then reads those sources and synthesizes a cited answer built specifically to meet the goal. + +Both boxes are on the **webtools** VM (`10.10.20.37`, VLAN 20, no auth — LAN-gated). They must be reachable from this host. + +- **SearXNG** — `http://10.10.20.37:8080/search?q=&format=json` (JSON output is enabled server-side) +- **Firecrawl** — `POST http://10.10.20.37:3002/v1/scrape`, body `{"url": "...", "formats": ["markdown"]}` + +## On Invoke + +Parse the argument into two parts: + +- **topic** — the subject (drives the output folder name). +- **goal** — *why* you're researching + the exact deliverable shape (fields, format, what "done" looks like). Look for `goal:`, `— goal:`, or an obvious "I want …" clause. + +If the **goal is missing or vague**, ask one to three clarifying questions before doing any work — the goal determines the search queries, which sources matter, and the synthesis format. Do not guess it. + +**Derive the topic slug**: lowercase, spaces/punctuation → hyphens (e.g. `Biblical Angels` → `angels` or `biblical-angels`). This is `` for all paths below. + +**Output layout** (relative to repo root; create dirs as needed): + +``` +deep_dive_research// + sources/ # one markdown file per extracted page (raw Firecrawl output) + manifest.md # index: query → URL → local file, with fetch status + REPORT.md # the synthesized answer to the goal +``` + +## Process + +```dot +digraph { + rankdir=LR; + queries -> search -> select -> extract -> synthesize -> report; + queries [label="1. Frame\n5-7 queries"]; + search [label="2. SearXNG\nJSON search"]; + select [label="3. Rank +\ndedupe URLs"]; + extract [label="4. Firecrawl\nscrape → files"]; + synthesize [label="5. Read\nsources"]; + report [label="6. REPORT.md\n+ summary"]; +} +``` + +### 1. Frame the search queries + +From the topic **and the goal**, write **5-7 distinct search queries** that attack the topic from different angles — don't paraphrase one query five times. Cover, as relevant: the core term, authoritative/primary sources, taxonomies/lists, specific sub-facets the goal calls for, and counter/edge perspectives. Briefly list the queries to the user before searching. + +> Example (goal = cited list of Biblical angels): `list of angels in the Bible`, `angelic hierarchy ranks orders Christian theology`, `archangels names scripture references`, `seraphim cherubim thrones Bible verses`, `named angels Michael Gabriel Raphael role`, `angels Book of Enoch Deuterocanonical`, `angel roles duties biblical texts`. + +### 2. Search each query on SearXNG + +For each query, hit the JSON API and collect the top results (title + url + snippet). Use `curl` + `jq`: + +```bash +Q="angelic hierarchy ranks Christian theology" +curl -sG "http://10.10.20.37:8080/search" \ + --data-urlencode "q=$Q" \ + --data-urlencode "format=json" \ + | jq -r '.results[:8][] | "\(.url)\t\(.title)"' +``` + +Run the queries (parallel `curl`s are fine). If SearXNG returns an empty `results` array for everything, the box may be down — check reachability (`curl -s -o /dev/null -w '%{http_code}' http://10.10.20.37:8080/`) and tell the user rather than proceeding with nothing. + +### 3. Rank and dedupe URLs + +Pool all results, **dedupe by URL/domain**, and select the most relevant, authoritative pages toward the goal — prefer primary sources, reference works, and substantive articles over listicle/SEO spam. **Cap the set at ~8-15 URLs** so extraction stays bounded; note in `manifest.md` if you dropped others. Record the query→URL mapping. + +### 4. Extract pages with Firecrawl → save to disk + +Scrape each selected URL to markdown and write it to `sources/-.md`, prefixed with a small header (source URL, originating query, fetch date). Example per URL: + +```bash +URL="https://en.wikipedia.org/wiki/Hierarchy_of_angels" +curl -s -X POST "http://10.10.20.37:3002/v1/scrape" \ + -H "Content-Type: application/json" \ + -d "{\"url\": \"$URL\", \"formats\": [\"markdown\"]}" \ + | jq -r '.data.markdown // .data.content // empty' +``` + +Guidance: +- Extract **sequentially or in small batches** (2-4 at a time), not a giant parallel burst — the playwright backend is patched to `waitUntil: 'domcontentloaded'` but ad-heavy pages can still be slow. Give each a reasonable `--max-time` (e.g. 60s). +- On a failure/empty/timeout for a URL: record it in `manifest.md` as failed and move on — don't let one dead page stall the run. Optionally substitute the next-best URL from step 3. +- **For a large set (>~8 pages)**, dispatch the extraction across a few parallel `Agent` sub-agents (each handles a slice of URLs, writes its files, and returns a one-line-per-URL status). Keep the raw markdown out of your own context — you'll read the files back in step 5. +- Write `manifest.md` as you go: a table of `# | query | url | file | status`. + +### 5. Synthesize — read the sources, meet the goal + +Now read the saved `sources/*.md` files (this is the point of saving them — work from the extracted text, not memory or snippets). Extract exactly what the goal asks for, **cite as you go**: for every claim, capture the supporting quote/passage and which source file + URL it came from. Cross-check across sources; note conflicts or thin evidence rather than papering over them. + +Structure the output to match the goal's requested shape. If the goal implies fields (name / rank / role / supporting text / source), build a **table or per-entry section** with exactly those fields. Include a short direct quote from the source document to support each claim where the goal asks for citations. + +### 6. Write REPORT.md and summarize + +Write `deep_dive_research//REPORT.md`: +- **Goal** restated at the top. +- The synthesized answer in the goal's shape (table / structured list / narrative), with inline citations `[n]` or `(source: file / URL)`. +- A **Sources** section listing every source file + URL used. +- A short **Gaps / caveats** note: failed fetches, unresolved conflicts, claims resting on weak evidence. + +Then give the user a **brief** chat summary (headline findings + where the full report lives) — don't paste the whole report into chat. + +## Notes & guardrails + +- **No auth** on either box — LAN-gated only. If a `curl` connection refuses/times out, the box is down or you're off VLAN 20; report that, don't silently produce an empty report. +- Firecrawl response shape can vary by version; the markdown is under `.data.markdown`. If `jq` yields empty, inspect the raw JSON once (`| jq '.'`) to find the right key before looping. +- `` folder is intentionally under the **repo root** (`deep_dive_research/`), not `docs/`. Create it if absent. These research artifacts are data outputs, not committed source — don't commit them unless the user asks. +- Keep the raw extracted pages on disk even after writing the report; they're the audit trail for every cited claim. + +## When NOT to use this + +- **One quick fact / single lookup** — just search once; this is for multi-source deep dives. +- **Interactive UI / login / click-through testing** — that's the `browser-testing` (Playwright MCP) skill. +- **Firecrawl or SearXNG box is unreachable** — fix connectivity first; this skill depends on both.