added deep-dive-research
This commit is contained in:
119
deep-dive-research/SKILL.md
Normal file
119
deep-dive-research/SKILL.md
Normal file
@@ -0,0 +1,119 @@
|
||||
---
|
||||
name: deep-dive-research
|
||||
description: Use when asked to deeply research a topic toward a specific goal — fan out searches on the homelab SearXNG box, extract the pages with the homelab Firecrawl box, save the raw sources locally, then compile/synthesize an answer that satisfies the goal. Invoke with a topic and a goal, e.g. /deep-dive-research angels — goal: compile a cited list of angels in Christian/Biblical texts with name, rank, role, and supporting quotes. Do NOT use for interactive UI testing (that's browser-testing) or for a single quick web lookup.
|
||||
---
|
||||
|
||||
# Deep-dive research (SearXNG + Firecrawl)
|
||||
|
||||
Research a **topic** in service of a **goal** (the *why* — the shape of the answer you must produce). This skill fans out real searches on the homelab **SearXNG** box, extracts the resulting pages with the homelab **Firecrawl** box, saves every source to disk under `deep_dive_research/<topic>/`, then reads those sources and synthesizes a cited answer built specifically to meet the goal.
|
||||
|
||||
Both boxes are on the **webtools** VM (`10.10.20.37`, VLAN 20, no auth — LAN-gated). They must be reachable from this host.
|
||||
|
||||
- **SearXNG** — `http://10.10.20.37:8080/search?q=<query>&format=json` (JSON output is enabled server-side)
|
||||
- **Firecrawl** — `POST http://10.10.20.37:3002/v1/scrape`, body `{"url": "...", "formats": ["markdown"]}`
|
||||
|
||||
## On Invoke
|
||||
|
||||
Parse the argument into two parts:
|
||||
|
||||
- **topic** — the subject (drives the output folder name).
|
||||
- **goal** — *why* you're researching + the exact deliverable shape (fields, format, what "done" looks like). Look for `goal:`, `— goal:`, or an obvious "I want …" clause.
|
||||
|
||||
If the **goal is missing or vague**, ask one to three clarifying questions before doing any work — the goal determines the search queries, which sources matter, and the synthesis format. Do not guess it.
|
||||
|
||||
**Derive the topic slug**: lowercase, spaces/punctuation → hyphens (e.g. `Biblical Angels` → `angels` or `biblical-angels`). This is `<topic>` for all paths below.
|
||||
|
||||
**Output layout** (relative to repo root; create dirs as needed):
|
||||
|
||||
```
|
||||
deep_dive_research/<topic>/
|
||||
sources/ # one markdown file per extracted page (raw Firecrawl output)
|
||||
manifest.md # index: query → URL → local file, with fetch status
|
||||
REPORT.md # the synthesized answer to the goal
|
||||
```
|
||||
|
||||
## Process
|
||||
|
||||
```dot
|
||||
digraph {
|
||||
rankdir=LR;
|
||||
queries -> search -> select -> extract -> synthesize -> report;
|
||||
queries [label="1. Frame\n5-7 queries"];
|
||||
search [label="2. SearXNG\nJSON search"];
|
||||
select [label="3. Rank +\ndedupe URLs"];
|
||||
extract [label="4. Firecrawl\nscrape → files"];
|
||||
synthesize [label="5. Read\nsources"];
|
||||
report [label="6. REPORT.md\n+ summary"];
|
||||
}
|
||||
```
|
||||
|
||||
### 1. Frame the search queries
|
||||
|
||||
From the topic **and the goal**, write **5-7 distinct search queries** that attack the topic from different angles — don't paraphrase one query five times. Cover, as relevant: the core term, authoritative/primary sources, taxonomies/lists, specific sub-facets the goal calls for, and counter/edge perspectives. Briefly list the queries to the user before searching.
|
||||
|
||||
> Example (goal = cited list of Biblical angels): `list of angels in the Bible`, `angelic hierarchy ranks orders Christian theology`, `archangels names scripture references`, `seraphim cherubim thrones Bible verses`, `named angels Michael Gabriel Raphael role`, `angels Book of Enoch Deuterocanonical`, `angel roles duties biblical texts`.
|
||||
|
||||
### 2. Search each query on SearXNG
|
||||
|
||||
For each query, hit the JSON API and collect the top results (title + url + snippet). Use `curl` + `jq`:
|
||||
|
||||
```bash
|
||||
Q="angelic hierarchy ranks Christian theology"
|
||||
curl -sG "http://10.10.20.37:8080/search" \
|
||||
--data-urlencode "q=$Q" \
|
||||
--data-urlencode "format=json" \
|
||||
| jq -r '.results[:8][] | "\(.url)\t\(.title)"'
|
||||
```
|
||||
|
||||
Run the queries (parallel `curl`s are fine). If SearXNG returns an empty `results` array for everything, the box may be down — check reachability (`curl -s -o /dev/null -w '%{http_code}' http://10.10.20.37:8080/`) and tell the user rather than proceeding with nothing.
|
||||
|
||||
### 3. Rank and dedupe URLs
|
||||
|
||||
Pool all results, **dedupe by URL/domain**, and select the most relevant, authoritative pages toward the goal — prefer primary sources, reference works, and substantive articles over listicle/SEO spam. **Cap the set at ~8-15 URLs** so extraction stays bounded; note in `manifest.md` if you dropped others. Record the query→URL mapping.
|
||||
|
||||
### 4. Extract pages with Firecrawl → save to disk
|
||||
|
||||
Scrape each selected URL to markdown and write it to `sources/<n>-<domain-slug>.md`, prefixed with a small header (source URL, originating query, fetch date). Example per URL:
|
||||
|
||||
```bash
|
||||
URL="https://en.wikipedia.org/wiki/Hierarchy_of_angels"
|
||||
curl -s -X POST "http://10.10.20.37:3002/v1/scrape" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"url\": \"$URL\", \"formats\": [\"markdown\"]}" \
|
||||
| jq -r '.data.markdown // .data.content // empty'
|
||||
```
|
||||
|
||||
Guidance:
|
||||
- Extract **sequentially or in small batches** (2-4 at a time), not a giant parallel burst — the playwright backend is patched to `waitUntil: 'domcontentloaded'` but ad-heavy pages can still be slow. Give each a reasonable `--max-time` (e.g. 60s).
|
||||
- On a failure/empty/timeout for a URL: record it in `manifest.md` as failed and move on — don't let one dead page stall the run. Optionally substitute the next-best URL from step 3.
|
||||
- **For a large set (>~8 pages)**, dispatch the extraction across a few parallel `Agent` sub-agents (each handles a slice of URLs, writes its files, and returns a one-line-per-URL status). Keep the raw markdown out of your own context — you'll read the files back in step 5.
|
||||
- Write `manifest.md` as you go: a table of `# | query | url | file | status`.
|
||||
|
||||
### 5. Synthesize — read the sources, meet the goal
|
||||
|
||||
Now read the saved `sources/*.md` files (this is the point of saving them — work from the extracted text, not memory or snippets). Extract exactly what the goal asks for, **cite as you go**: for every claim, capture the supporting quote/passage and which source file + URL it came from. Cross-check across sources; note conflicts or thin evidence rather than papering over them.
|
||||
|
||||
Structure the output to match the goal's requested shape. If the goal implies fields (name / rank / role / supporting text / source), build a **table or per-entry section** with exactly those fields. Include a short direct quote from the source document to support each claim where the goal asks for citations.
|
||||
|
||||
### 6. Write REPORT.md and summarize
|
||||
|
||||
Write `deep_dive_research/<topic>/REPORT.md`:
|
||||
- **Goal** restated at the top.
|
||||
- The synthesized answer in the goal's shape (table / structured list / narrative), with inline citations `[n]` or `(source: file / URL)`.
|
||||
- A **Sources** section listing every source file + URL used.
|
||||
- A short **Gaps / caveats** note: failed fetches, unresolved conflicts, claims resting on weak evidence.
|
||||
|
||||
Then give the user a **brief** chat summary (headline findings + where the full report lives) — don't paste the whole report into chat.
|
||||
|
||||
## Notes & guardrails
|
||||
|
||||
- **No auth** on either box — LAN-gated only. If a `curl` connection refuses/times out, the box is down or you're off VLAN 20; report that, don't silently produce an empty report.
|
||||
- Firecrawl response shape can vary by version; the markdown is under `.data.markdown`. If `jq` yields empty, inspect the raw JSON once (`| jq '.'`) to find the right key before looping.
|
||||
- `<topic>` folder is intentionally under the **repo root** (`deep_dive_research/`), not `docs/`. Create it if absent. These research artifacts are data outputs, not committed source — don't commit them unless the user asks.
|
||||
- Keep the raw extracted pages on disk even after writing the report; they're the audit trail for every cited claim.
|
||||
|
||||
## When NOT to use this
|
||||
|
||||
- **One quick fact / single lookup** — just search once; this is for multi-source deep dives.
|
||||
- **Interactive UI / login / click-through testing** — that's the `browser-testing` (Playwright MCP) skill.
|
||||
- **Firecrawl or SearXNG box is unreachable** — fix connectivity first; this skill depends on both.
|
||||
Reference in New Issue
Block a user