code_of_conquest_dnd/docs/superpowers/specs/2026-07-10-narrator-streaming-design.md

Narrator streaming (M2) — design

Date: 2026-07-10 Milestone: M2 — prove aliveness (roadmap). The third M2 item; polish, not proof — gated on the aliveness question, which the NPC work answered. Charter side (§2): text — the client streams and displays AI prose; it still never lets prose write state directly (facts harvested at stream end, applied by the caller).

Goal

Make the ~2s model call read as the DM is speaking rather than the game is frozen (§14). The server streams tokens from Ollama as they are produced; the client renders them as they arrive — first words on screen in ~200ms instead of a 2s blank wait.

CanonLog → HTTP → /dm/narrate (200 NDJSON stream) → tokens as they arrive
   → client renders incrementally → at stream end: TagExtractor → fact-harvest

This piece proves the streaming mechanism on the Narrator only — pure prose, no moves. The NPC surface (which must run move validation on the complete text at stream end) is a deliberate fast-follow on the same transport. This is the riskiest transport rework in the POC, so it lands on the simplest surface first.

What already exists (and is proven)

• Server: ollama_client.chat(...) posts {"stream": False} to Ollama's /api/chat with one retry, raising ModelError; its docstring already says it was "built so a streaming mode is an additive path later." narrate.run(canon_log) assembles messages, calls chat, logs via call_log.record(...), returns raw prose. /dm/narrate returns {"prose": …} / 502 / 422. call_log.record takes response=/error= and an injectable write sink.
• Client: DmService.narrate(canon_log) -> NarrateResult posts through an injectable DmTransport (post_json(path, body) -> DmResponse), degrades to an authored fallback on any failure, harvests [FACT] via TagExtractor, and returns facts for the caller to apply (§2). The real HttpTransport wraps a Godot HTTPRequest Node (which buffers the whole body and fires request_completed once — it cannot stream). FakeDmTransport returns a canned DmResponse. narrate_harness.gd drives it on screen.

The server is genuinely one flag from streaming. The client is not: Godot's HTTPRequest has no incremental-read path, so real streaming requires the lower-level HTTPClient (poll + read_response_body_chunk()). That rework is the weight of this piece.

Decisions

Settled during brainstorming; each is load-bearing.

1. Real token streaming, not a cosmetic reveal. A client-side typewriter over already-received text does not hide the initial wait — it just animates it. Meeting §14 requires the server to emit tokens as Ollama produces them and the client to render them live. That mandates the HTTPClient rework; there is no cheaper path that meets the goal.

2. Narrator first; NPC is a fast-follow. Both prose endpoints share the machinery, but NPC must run TagExtractor/MoveValidator/MoveApplier on the complete text at stream end. Proving the transport on the move-free Narrator means a transport bug cannot corrupt move application. The NPC extension reuses this transport unchanged and adds only parse-at-end.

3. Proxy emits its own minimal NDJSON, not Ollama's shape. §4 — the client knows nothing about the model. One JSON object per line: {"delta": "..."} per token, terminated by {"done": true}, or by {"error": "..."} on a model failure. NDJSON (not SSE) because HTTPClient.read_response_body_chunk() returns arbitrary byte boundaries; the client buffers bytes and splits on \n, and line-delimited JSON is trivial to reassemble. The proxy translates Ollama's {"message":{"content":…},"done":…} lines into these frames.

4. Endpoint contract shift on /dm/narrate: 422 stays synchronous; model errors become an in-stream frame. Canon-log validation runs before any streaming — a bad log still returns a plain 422 JSON (never a stream). A valid log returns a 200 NDJSON stream. Because a StreamingResponse's status is fixed at 200 once it starts, a model failure is no longer a 502 — it is an {"error": …} frame inside the 200 stream. Same principle (the client degrades on failure), new shape. The non-streaming JSON contract is retired for this endpoint (the client is its only consumer and moves to streaming).

5. §12 retry moves to before-first-token, server-side. Once bytes flow you cannot cleanly retry. chat_stream retries the Ollama call only if it fails before yielding the first delta; once streaming, an error propagates as the {"error"} frame. The client never retries a stream — it degrades. This keeps the retry in one place (server) and the client purely reactive, as today.

6. Client streaming transport coexists with HttpTransport. The proven non-streaming transport keeps its job (the Adjudicator's strict JSON, later). A new StreamTransport seam is added, not a replacement. DmService gains a streaming method; the old narrate() path may remain for reference but the harness uses the streaming one.

7. Testable seam: post_stream(path, body, on_delta) -> StreamResult. The service passes an on_delta: Callable that buffers text and forwards to the UI. StreamResult carries {ok: bool, error: String} at the end. A FakeStreamTransport invokes on_delta for each canned chunk then returns a canned StreamResult, so the service's accumulate/hold-back/degrade logic is pure and unit-testable. The real HTTPClient poll loop is the one untested shim — exactly the posture HttpTransport holds today.

8. On-screen tags: hold-back buffering. When a [ arrives, the visible reveal holds from that point until the matching ], then the tag is stripped. This prevents [FACT: the we… flashing then vanishing, which reads as a bug and undercuts the "feels alive" goal. Implemented as a small running buffer in the on-delta path — the full raw text is still accumulated separately for the end-of-stream TagExtractor pass.

9. Mid-stream failure: keep the partial prose, mark degraded, harvest no facts. If the stream dies after some tokens, the player keeps the valid prose already shown, the result is flagged degraded, and no facts are harvested — the tag pipeline needs the complete text to be trustworthy. The pre-first-token failure path (retry once server-side, then fail) shows the full authored fallback, as today. No surface ever shows an error (§13).

10. The HttpTransport timeout gap is closed here. The HTTPClient transport gets a connect timeout + a time-to-first-token timeout + an inter-chunk timeout — not a total-duration timeout (a long stream is legitimate). A stall at any of those is a stream failure → degrade. This retires the deferred _http.timeout=0 minor in the code that is reworking the transport anyway.

Contracts

POST /dm/narrate — request unchanged

Body: {"canon_log": { …schema-v1… }}.

Responses

• Invalid log → 422 {"detail": {"canon_log_errors": [ … ]}} (synchronous JSON, before any stream — unchanged).

• Valid log → 200, Content-Type: application/x-ndjson, body is a sequence of newline-terminated JSON objects:

  {"delta": "The chamber "}
  {"delta": "is cold. "}
  {"delta": "[FACT: the well is dry]"}
  {"done": true}
  

  A model failure replaces the terminal frame:

  {"delta": "The chamber "}
  {"error": "connection reset"}
  

  done and error are mutually exclusive and terminal. Frames carry no model identity, seed, or Ollama metadata (§4).

Components

Server:

• api/app/ollama_client.py — add chat_stream(model, messages, options, *, think=None, client=None) -> Iterator[str]. Uses httpx client.stream("POST", "/api/chat", json={... "stream": True ...}), iterates iter_lines(), parses each Ollama line, yields message.content deltas. Retries once only if the request fails before the first delta; a failure after the first delta raises ModelError from within the generator (the service converts it to an error frame). The existing chat() is untouched.
• api/app/narrate.py — add run_stream(canon_log) -> Iterator[str] (or a sibling module narrate_stream.py if narrate.py grows unwieldy): assembles the same messages, calls chat_stream, yields proxy NDJSON frames ({"delta":…} / terminal {"done":true} / {"error":…}), accumulates the full text, and logs via call_log.record(...) at stream end (both success — with the accumulated response — and error paths). Logs time-to-first-token.
• api/app/main.py — /dm/narrate validates the canon log synchronously (unchanged valid_turn), then returns StreamingResponse(narrate_stream_gen(...), media_type="application/x-ndjson"). A ModelError raised before the first frame still surfaces as an in-stream {"error"} frame (the generator catches and emits it), not a 502.

Client:

• client/scripts/net/stream_transport.gd — StreamTransport (class_name), the seam: post_stream(path: String, body: Dictionary, on_delta: Callable) -> StreamResult (async). Base pushes an error like DmTransport does.
• client/scripts/net/http_stream_transport.gd — the real one: wraps an HTTPClient, connects, POSTs, polls poll() + read_response_body_chunk(), buffers bytes, splits on \n, parses each frame, calls on_delta(delta) per {"delta"}, and returns a StreamResult on {"done"} / {"error"} / transport failure / timeout. Untested shim (verified by the harness + gated live smoke), like HttpTransport. Holds the connect / first-token / inter-chunk timeouts (§decision 10).
• client/scripts/net/stream_result.gd — StreamResult{ok: bool, error: String}
  • StreamResult.failed(msg).
• client/scripts/net/dm_service.gd — add narrate_stream(canon_log, on_delta: Callable) -> NarrateResult. It wraps the caller's on_delta in a hold-back buffer (reveals text, holding an open […] span), accumulates the full raw text, awaits post_stream, and at the end: if the StreamResult failed with no text → NarrateResult.fallback(...); if it failed mid-stream → degraded result carrying the partial display_text and no facts; on success → TagExtractor.extract(full_text) and return {clean_text, facts, degraded:false}. Reuses NarrateResult and FallbackLibrary unchanged.
• client/scripts/text/tag_stream_filter.gd — a small pure hold-back filter: feed(delta) -> String returns the safe-to-show text (buffering from an unmatched [), flush() -> String returns any trailing buffered text at end. Pure and unit-tested; DmService.narrate_stream uses it for the on-screen reveal while TagExtractor still runs on the full raw text.
• client/scripts/harness/narrate_harness.gd — switch the button handler to narrate_stream, appending each on_delta chunk to the RichTextLabel live, then applying facts at the end (unchanged §2 apply).

Error handling

• Invalid canon log → synchronous 422; client shows fallback (unchanged).
• Pre-first-token model failure → chat_stream retries once; still failing → the generator's first frame is {"error"} → client shows the full authored fallback line, no facts.
• Mid-stream failure ({"error"} after deltas, dropped connection, or an inter-chunk timeout) → keep the partial prose on screen, mark degraded, harvest no facts.
• Stall (no connect / no first token / gap between chunks past the timeout) → treated as the matching failure above.
• No surface ever shows an error string (§13).

Testing

• TagStreamFilter (pure, GUT): a tag split across two deltas is held then revealed cleanly; a [FACT:…] is stripped from the visible reveal; plain text passes straight through; flush() returns a dangling partial.
• DmService.narrate_stream with FakeStreamTransport: deltas accumulate → display_text is the cleaned full text and facts are harvested; a canned {"error"} after deltas → degraded, partial text kept, no facts; a pre-token failure → full fallback, no facts; the on-screen reveal (via the injected on_delta) never shows raw tag text.
• Server chat_stream with a fake httpx stream: yields deltas in order; retries once before the first delta; a mid-stream raise surfaces as ModelError.
• Server narrate stream service with a fake chat_stream: emits well-formed NDJSON frames terminated by done; a raise emits a terminal error frame; the call is logged once with the accumulated text.
• Endpoint (TestClient): a valid log streams application/x-ndjson frames ending in {"done":true}; an invalid log still returns 422 JSON (no stream); a model failure yields a 200 stream whose last frame is {"error"}.
• Gated live stream smoke (@pytest.mark.live): real Ollama, assert deltas arrive incrementally (more than one frame) and the concatenation is non-empty grounded prose, no Luck number.

Scope

In: proxy NDJSON stream format; ollama_client.chat_stream; the streaming narrate service + /dm/narrate streaming endpoint (422 synchronous, model error as in-stream frame); the client HTTPClient StreamTransport (+ connect / first-token / inter-chunk timeouts) behind a testable seam; StreamResult; DmService.narrate_stream; the pure TagStreamFilter hold-back; keep-partial mid-stream degrade; narrate_harness streaming on screen; full test set + gated live stream smoke.

Out: NPC streaming (the fast-follow — reuses this transport, adds parse-at-end); the Adjudicator; combat flavor (async, later); any real dialogue UI; a total-duration timeout; SSE (NDJSON chosen).