diff --git a/docs/superpowers/specs/2026-07-09-narrator-ollama-pipeline-design.md b/docs/superpowers/specs/2026-07-09-narrator-ollama-pipeline-design.md index 7868b20..a9f4a25 100644 --- a/docs/superpowers/specs/2026-07-09-narrator-ollama-pipeline-design.md +++ b/docs/superpowers/specs/2026-07-09-narrator-ollama-pipeline-design.md @@ -48,8 +48,8 @@ POST /dm/narrate (main.py handler — thin) └─ narrate.run(canon_log): system = prompts.system_prompt("narrator") # narrator.md body (the IP) user = prompts.render_digest(canon_log) # curated labeled digest (§11 context) - cfg = routing.for_role("narrator") # model + options (+ per-call seed) - text = ollama_client.chat(cfg.model, [system, user], cfg.options) # 1 retry, else ModelError + cfg = routing.for_role("narrator") # model + options (+ per-call seed) + think flag + text = ollama_client.chat(cfg.model, [system, user], cfg.options, think=cfg.think) # 1 retry, else ModelError call_log.record(role, model, options, seed, messages, canon_log, text|error, latency_ms, ok) return text └─ 200 {"prose": text} | except ModelError → 502 {"detail": {"model_error": ""}} @@ -102,21 +102,36 @@ Describe the scene as it is now. class RoleConfig: model: str options: dict # Ollama generation options, includes a per-call seed at call time + think: bool | None = None # Ollama top-level `think` flag; None omits it ROLES = { "narrator": RoleConfig( - model=env("OLLAMA_NARRATOR_MODEL", "llama3.1:latest"), # §5 "the good model" — swappable per deploy + model=env("OLLAMA_NARRATOR_MODEL", "qwen3.5:latest"), # §5 "the good model" — swappable per deploy options={"temperature": 0.8, "top_p": 0.9, "num_predict": 300}, # voice; capped length (§14) + think=False, # qwen3.x thinking OFF — no preamble in narrator prose ), } def for_role(role: str) -> RoleConfig: ... ``` - Base URL from `OLLAMA_BASE_URL` (already in `.env.example`). -- Model and options are **config, not law** — swapping the Narrator model is an env change, exactly §4's "a deploy, not a client patch." `llama3.1:latest` is a starting default (local models available: `llama3.1`, `llama3.2`, `qwen3.5`). +- Model and options are **config, not law** — swapping the Narrator model is an env change, exactly §4's "a deploy, not a client patch." `qwen3.5:latest` is the default (local models available: `qwen3.5`, `llama3.1`, `llama3.2`). +- **`think: False` (qwen3.x thinking disabled):** qwen3.5 is a thinking model; without this it emits a `` preamble that would pollute the narrator prose. `ollama_client` passes `think` as a **top-level** `/api/chat` field (not inside `options`) when `RoleConfig.think is not None`. Verified in the 2026-07-09 probe: `think:false` yields an empty `thinking` field and no leakage, and is **gracefully accepted by non-thinking models too** (llama3.1/3.2 return normally) — so it is safe across an env model swap. - **Per-call seed (§10):** the server sets an explicit `options.seed` per call (a random int, captured in the log) so any call is replayable against a candidate model — the "free eval infrastructure" of §4. Without setting it, Ollama's own random seed is unrecoverable. - **Provider abstraction (Ollama → Replicate) is deferred** (YAGNI). `routing.py` is the seam it slots into later; today, Ollama only. +### Model probe (2026-07-09) + +Warm, seeded, narrator-shaped call (system + digest, `num_predict:300`) against the reachable Ollama: + +| model | warm latency | tok/s | notes | +|---|---|---|---| +| **qwen3.5** (`think:false`) | ~2.0s | ~107 | richest prose; only model to emit `[FACT:]`; no `` leak | +| llama3.1 | ~1.2s | ~149 | solid 2nd-person prose | +| llama3.2 | ~0.5s | ~295 | fastest, coherent, shorter | + +Decision: **qwen3.5 is the default** (§5 "the good model") — ~2s non-streaming is acceptable for the POC and streaming hides it later. llama3.1/llama3.2 remain one-env-var swaps if latency needs to drop; the speed data is on record for that tuning call. + ## Failure, retry, error contract (§12/§13) `ollama_client.chat()` owns the retry; callers see success-or-`ModelError`. @@ -165,7 +180,7 @@ On failure: `"ok": false, "error": ""`, no `response`. **Unit — httpx `MockTransport`, deterministic, no new dependency:** -- `ollama_client`: success parses content; retry (fail-once → recovers; fail-twice → `ModelError`); timeout → `ModelError`; non-2xx → retry → `ModelError`; empty content → retry → `ModelError`. Asserts the outgoing payload (model, `messages`, `stream:false`, options incl. `seed`). +- `ollama_client`: success parses content; retry (fail-once → recovers; fail-twice → `ModelError`); timeout → `ModelError`; non-2xx → retry → `ModelError`; empty content → retry → `ModelError`. Asserts the outgoing payload (model, `messages`, `stream:false`, options incl. `seed`, and top-level `think` present only when set). - `prompts`: `render_digest` on `canon_log_valid.json` — correct labeled block, dispositions omitted, luck as descriptor-not-number; `system_prompt("narrator")` loads the file. - `routing`: `for_role` returns the configured model + options; env override respected. - `call_log`: writes a parseable JSON line with every field to the injected sink; failure-record shape.