docs(spec): qwen3.5 narrator default + think:false; add model probe
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -48,8 +48,8 @@ POST /dm/narrate (main.py handler — thin)
|
||||
└─ narrate.run(canon_log):
|
||||
system = prompts.system_prompt("narrator") # narrator.md body (the IP)
|
||||
user = prompts.render_digest(canon_log) # curated labeled digest (§11 context)
|
||||
cfg = routing.for_role("narrator") # model + options (+ per-call seed)
|
||||
text = ollama_client.chat(cfg.model, [system, user], cfg.options) # 1 retry, else ModelError
|
||||
cfg = routing.for_role("narrator") # model + options (+ per-call seed) + think flag
|
||||
text = ollama_client.chat(cfg.model, [system, user], cfg.options, think=cfg.think) # 1 retry, else ModelError
|
||||
call_log.record(role, model, options, seed, messages, canon_log, text|error, latency_ms, ok)
|
||||
return text
|
||||
└─ 200 {"prose": text} | except ModelError → 502 {"detail": {"model_error": "<reason>"}}
|
||||
@@ -102,21 +102,36 @@ Describe the scene as it is now.
|
||||
class RoleConfig:
|
||||
model: str
|
||||
options: dict # Ollama generation options, includes a per-call seed at call time
|
||||
think: bool | None = None # Ollama top-level `think` flag; None omits it
|
||||
|
||||
ROLES = {
|
||||
"narrator": RoleConfig(
|
||||
model=env("OLLAMA_NARRATOR_MODEL", "llama3.1:latest"), # §5 "the good model" — swappable per deploy
|
||||
model=env("OLLAMA_NARRATOR_MODEL", "qwen3.5:latest"), # §5 "the good model" — swappable per deploy
|
||||
options={"temperature": 0.8, "top_p": 0.9, "num_predict": 300}, # voice; capped length (§14)
|
||||
think=False, # qwen3.x thinking OFF — no <think> preamble in narrator prose
|
||||
),
|
||||
}
|
||||
def for_role(role: str) -> RoleConfig: ...
|
||||
```
|
||||
|
||||
- Base URL from `OLLAMA_BASE_URL` (already in `.env.example`).
|
||||
- Model and options are **config, not law** — swapping the Narrator model is an env change, exactly §4's "a deploy, not a client patch." `llama3.1:latest` is a starting default (local models available: `llama3.1`, `llama3.2`, `qwen3.5`).
|
||||
- Model and options are **config, not law** — swapping the Narrator model is an env change, exactly §4's "a deploy, not a client patch." `qwen3.5:latest` is the default (local models available: `qwen3.5`, `llama3.1`, `llama3.2`).
|
||||
- **`think: False` (qwen3.x thinking disabled):** qwen3.5 is a thinking model; without this it emits a `<think>…</think>` preamble that would pollute the narrator prose. `ollama_client` passes `think` as a **top-level** `/api/chat` field (not inside `options`) when `RoleConfig.think is not None`. Verified in the 2026-07-09 probe: `think:false` yields an empty `thinking` field and no leakage, and is **gracefully accepted by non-thinking models too** (llama3.1/3.2 return normally) — so it is safe across an env model swap.
|
||||
- **Per-call seed (§10):** the server sets an explicit `options.seed` per call (a random int, captured in the log) so any call is replayable against a candidate model — the "free eval infrastructure" of §4. Without setting it, Ollama's own random seed is unrecoverable.
|
||||
- **Provider abstraction (Ollama → Replicate) is deferred** (YAGNI). `routing.py` is the seam it slots into later; today, Ollama only.
|
||||
|
||||
### Model probe (2026-07-09)
|
||||
|
||||
Warm, seeded, narrator-shaped call (system + digest, `num_predict:300`) against the reachable Ollama:
|
||||
|
||||
| model | warm latency | tok/s | notes |
|
||||
|---|---|---|---|
|
||||
| **qwen3.5** (`think:false`) | ~2.0s | ~107 | richest prose; only model to emit `[FACT:]`; no `<think>` leak |
|
||||
| llama3.1 | ~1.2s | ~149 | solid 2nd-person prose |
|
||||
| llama3.2 | ~0.5s | ~295 | fastest, coherent, shorter |
|
||||
|
||||
Decision: **qwen3.5 is the default** (§5 "the good model") — ~2s non-streaming is acceptable for the POC and streaming hides it later. llama3.1/llama3.2 remain one-env-var swaps if latency needs to drop; the speed data is on record for that tuning call.
|
||||
|
||||
## Failure, retry, error contract (§12/§13)
|
||||
|
||||
`ollama_client.chat()` owns the retry; callers see success-or-`ModelError`.
|
||||
@@ -165,7 +180,7 @@ On failure: `"ok": false, "error": "<reason>"`, no `response`.
|
||||
|
||||
**Unit — httpx `MockTransport`, deterministic, no new dependency:**
|
||||
|
||||
- `ollama_client`: success parses content; retry (fail-once → recovers; fail-twice → `ModelError`); timeout → `ModelError`; non-2xx → retry → `ModelError`; empty content → retry → `ModelError`. Asserts the outgoing payload (model, `messages`, `stream:false`, options incl. `seed`).
|
||||
- `ollama_client`: success parses content; retry (fail-once → recovers; fail-twice → `ModelError`); timeout → `ModelError`; non-2xx → retry → `ModelError`; empty content → retry → `ModelError`. Asserts the outgoing payload (model, `messages`, `stream:false`, options incl. `seed`, and top-level `think` present only when set).
|
||||
- `prompts`: `render_digest` on `canon_log_valid.json` — correct labeled block, dispositions omitted, luck as descriptor-not-number; `system_prompt("narrator")` loads the file.
|
||||
- `routing`: `for_role` returns the configured model + options; env override respected.
|
||||
- `call_log`: writes a parseable JSON line with every field to the injected sink; failure-record shape.
|
||||
|
||||
Reference in New Issue
Block a user