From 1ed863d3fee2c0f229496d8976af64d3cc924bd8 Mon Sep 17 00:00:00 2001 From: Phillip Tarrant Date: Fri, 10 Jul 2026 14:23:32 -0500 Subject: [PATCH] docs(spec): Narrator streaming (M2) design MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Real token streaming to hide the ~2s call (§14). Proxy emits its own minimal NDJSON (delta/done/error); /dm/narrate becomes a 200 stream (422 stays sync, model errors become an in-stream frame). Client reworks to an HTTPClient poll-loop StreamTransport behind a testable seam; hold-back tag filter; keep-partial mid-stream degrade; folds in the HttpTransport timeout fix. Narrator first; NPC fast-follow. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../2026-07-10-narrator-streaming-design.md | 254 ++++++++++++++++++ 1 file changed, 254 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-10-narrator-streaming-design.md diff --git a/docs/superpowers/specs/2026-07-10-narrator-streaming-design.md b/docs/superpowers/specs/2026-07-10-narrator-streaming-design.md new file mode 100644 index 0000000..b6b82c1 --- /dev/null +++ b/docs/superpowers/specs/2026-07-10-narrator-streaming-design.md @@ -0,0 +1,254 @@ +# Narrator streaming (M2) — design + +Date: 2026-07-10 +Milestone: **M2 — prove aliveness** (roadmap). The third M2 item; **polish, not +proof** — gated on the aliveness question, which the NPC work answered. +Charter side (§2): **text** — the client streams and displays AI prose; it still +never lets prose write state directly (facts harvested at stream end, applied by +the caller). + +## Goal + +Make the ~2s model call read as *the DM is speaking* rather than *the game is +frozen* (§14). The server streams tokens from Ollama as they are produced; the +client renders them as they arrive — first words on screen in ~200ms instead of +a 2s blank wait. + +``` +CanonLog → HTTP → /dm/narrate (200 NDJSON stream) → tokens as they arrive + → client renders incrementally → at stream end: TagExtractor → fact-harvest +``` + +This piece proves the streaming mechanism on the **Narrator only** — pure prose, +no moves. The NPC surface (which must run move validation on the complete text +at stream end) is a deliberate fast-follow on the *same* transport. This is the +riskiest transport rework in the POC, so it lands on the simplest surface first. + +## What already exists (and is proven) + +- **Server:** `ollama_client.chat(...)` posts `{"stream": False}` to Ollama's + `/api/chat` with one retry, raising `ModelError`; its docstring already says it + was "built so a streaming mode is an additive path later." `narrate.run(canon_log)` + assembles messages, calls `chat`, logs via `call_log.record(...)`, returns raw + prose. `/dm/narrate` returns `{"prose": …}` / `502` / `422`. `call_log.record` + takes `response=`/`error=` and an injectable `write` sink. +- **Client:** `DmService.narrate(canon_log) -> NarrateResult` posts through an + injectable `DmTransport` (`post_json(path, body) -> DmResponse`), degrades to + an authored fallback on any failure, harvests `[FACT]` via `TagExtractor`, and + returns facts for the caller to apply (§2). The real `HttpTransport` wraps a + Godot `HTTPRequest` **Node** (which buffers the whole body and fires + `request_completed` once — it cannot stream). `FakeDmTransport` returns a + canned `DmResponse`. `narrate_harness.gd` drives it on screen. + +The server is genuinely one flag from streaming. The client is not: Godot's +`HTTPRequest` has no incremental-read path, so real streaming requires the +lower-level `HTTPClient` (poll + `read_response_body_chunk()`). That rework is +the weight of this piece. + +## Decisions + +Settled during brainstorming; each is load-bearing. + +1. **Real token streaming, not a cosmetic reveal.** A client-side typewriter over + already-received text does not hide the initial wait — it just animates it. + Meeting §14 requires the server to emit tokens as Ollama produces them and the + client to render them live. That mandates the `HTTPClient` rework; there is no + cheaper path that meets the goal. + +2. **Narrator first; NPC is a fast-follow.** Both prose endpoints share the + machinery, but NPC must run `TagExtractor`/`MoveValidator`/`MoveApplier` on the + *complete* text at stream end. Proving the transport on the move-free Narrator + means a transport bug cannot corrupt move application. The NPC extension reuses + this transport unchanged and adds only parse-at-end. + +3. **Proxy emits its own minimal NDJSON, not Ollama's shape.** §4 — the client + knows nothing about the model. One JSON object per line: + `{"delta": "..."}` per token, terminated by `{"done": true}`, or by + `{"error": "..."}` on a model failure. NDJSON (not SSE) because + `HTTPClient.read_response_body_chunk()` returns arbitrary byte boundaries; the + client buffers bytes and splits on `\n`, and line-delimited JSON is trivial to + reassemble. The proxy translates Ollama's `{"message":{"content":…},"done":…}` + lines into these frames. + +4. **Endpoint contract shift on `/dm/narrate`: 422 stays synchronous; model + errors become an in-stream frame.** Canon-log validation runs **before** any + streaming — a bad log still returns a plain `422` JSON (never a stream). A + valid log returns a **200 NDJSON stream**. Because a `StreamingResponse`'s + status is fixed at 200 once it starts, a **model failure is no longer a `502`** + — it is an `{"error": …}` frame inside the 200 stream. Same principle (the + client degrades on failure), new shape. The non-streaming JSON contract is + retired for this endpoint (the client is its only consumer and moves to + streaming). + +5. **§12 retry moves to before-first-token, server-side.** Once bytes flow you + cannot cleanly retry. `chat_stream` retries the Ollama call **only if it fails + before yielding the first delta**; once streaming, an error propagates as the + `{"error"}` frame. The client never retries a stream — it degrades. This keeps + the retry in one place (server) and the client purely reactive, as today. + +6. **Client streaming transport coexists with `HttpTransport`.** The proven + non-streaming transport keeps its job (the Adjudicator's strict JSON, later). + A new `StreamTransport` seam is added, not a replacement. `DmService` gains a + streaming method; the old `narrate()` path may remain for reference but the + harness uses the streaming one. + +7. **Testable seam: `post_stream(path, body, on_delta) -> StreamResult`.** The + service passes an `on_delta: Callable` that buffers text and forwards to the + UI. `StreamResult` carries `{ok: bool, error: String}` at the end. A + `FakeStreamTransport` invokes `on_delta` for each canned chunk then returns a + canned `StreamResult`, so the service's accumulate/hold-back/degrade logic is + pure and unit-testable. The real `HTTPClient` poll loop is the one untested + shim — exactly the posture `HttpTransport` holds today. + +8. **On-screen tags: hold-back buffering.** When a `[` arrives, the visible + reveal holds from that point until the matching `]`, then the tag is stripped. + This prevents `[FACT: the we…` flashing then vanishing, which reads as a bug + and undercuts the "feels alive" goal. Implemented as a small running buffer in + the on-delta path — the full raw text is still accumulated separately for the + end-of-stream `TagExtractor` pass. + +9. **Mid-stream failure: keep the partial prose, mark degraded, harvest no + facts.** If the stream dies after some tokens, the player keeps the valid + prose already shown, the result is flagged `degraded`, and **no** facts are + harvested — the tag pipeline needs the complete text to be trustworthy. The + pre-first-token failure path (retry once server-side, then fail) shows the full + authored fallback, as today. No surface ever shows an error (§13). + +10. **The `HttpTransport` timeout gap is closed here.** The `HTTPClient` transport + gets a **connect timeout + a time-to-first-token timeout + an inter-chunk + timeout** — not a total-duration timeout (a long stream is legitimate). A stall + at any of those is a stream failure → degrade. This retires the deferred + `_http.timeout=0` minor in the code that is reworking the transport anyway. + +## Contracts + +### `POST /dm/narrate` — request unchanged + +Body: `{"canon_log": { …schema-v1… }}`. + +### Responses + +- Invalid log → `422 {"detail": {"canon_log_errors": [ … ]}}` (synchronous JSON, + before any stream — unchanged). +- Valid log → `200`, `Content-Type: application/x-ndjson`, body is a sequence of + newline-terminated JSON objects: + + ``` + {"delta": "The chamber "} + {"delta": "is cold. "} + {"delta": "[FACT: the well is dry]"} + {"done": true} + ``` + + A model failure replaces the terminal frame: + + ``` + {"delta": "The chamber "} + {"error": "connection reset"} + ``` + + `done` and `error` are mutually exclusive and terminal. Frames carry no model + identity, seed, or Ollama metadata (§4). + +## Components + +**Server:** + +- `api/app/ollama_client.py` — add `chat_stream(model, messages, options, *, + think=None, client=None) -> Iterator[str]`. Uses `httpx` `client.stream("POST", + "/api/chat", json={... "stream": True ...})`, iterates `iter_lines()`, parses + each Ollama line, yields `message.content` deltas. Retries once **only** if the + request fails before the first delta; a failure after the first delta raises + `ModelError` from within the generator (the service converts it to an error + frame). The existing `chat()` is untouched. +- `api/app/narrate.py` — add `run_stream(canon_log) -> Iterator[str]` (or a + sibling module `narrate_stream.py` if `narrate.py` grows unwieldy): assembles + the same messages, calls `chat_stream`, yields proxy NDJSON frames + (`{"delta":…}` / terminal `{"done":true}` / `{"error":…}`), accumulates the full + text, and logs via `call_log.record(...)` at stream end (both success — with the + accumulated `response` — and error paths). Logs time-to-first-token. +- `api/app/main.py` — `/dm/narrate` validates the canon log synchronously + (unchanged `valid_turn`), then returns + `StreamingResponse(narrate_stream_gen(...), media_type="application/x-ndjson")`. + A `ModelError` raised before the first frame still surfaces as an in-stream + `{"error"}` frame (the generator catches and emits it), not a 502. + +**Client:** + +- `client/scripts/net/stream_transport.gd` — `StreamTransport` (`class_name`), the + seam: `post_stream(path: String, body: Dictionary, on_delta: Callable) -> + StreamResult` (async). Base pushes an error like `DmTransport` does. +- `client/scripts/net/http_stream_transport.gd` — the real one: wraps an + `HTTPClient`, connects, POSTs, polls `poll()` + `read_response_body_chunk()`, + buffers bytes, splits on `\n`, parses each frame, calls `on_delta(delta)` per + `{"delta"}`, and returns a `StreamResult` on `{"done"}` / `{"error"}` / + transport failure / timeout. Untested shim (verified by the harness + gated live + smoke), like `HttpTransport`. Holds the connect / first-token / inter-chunk + timeouts (§decision 10). +- `client/scripts/net/stream_result.gd` — `StreamResult{ok: bool, error: String}` + + `StreamResult.failed(msg)`. +- `client/scripts/net/dm_service.gd` — add `narrate_stream(canon_log, on_delta: + Callable) -> NarrateResult`. It wraps the caller's `on_delta` in a hold-back + buffer (reveals text, holding an open `[`…`]` span), accumulates the full raw + text, awaits `post_stream`, and at the end: if the `StreamResult` failed with no + text → `NarrateResult.fallback(...)`; if it failed mid-stream → degraded result + carrying the partial `display_text` and **no** facts; on success → + `TagExtractor.extract(full_text)` and return `{clean_text, facts, degraded:false}`. + Reuses `NarrateResult` and `FallbackLibrary` unchanged. +- `client/scripts/text/tag_stream_filter.gd` — a small **pure** hold-back filter: + `feed(delta) -> String` returns the safe-to-show text (buffering from an + unmatched `[`), `flush() -> String` returns any trailing buffered text at end. + Pure and unit-tested; `DmService.narrate_stream` uses it for the on-screen + reveal while `TagExtractor` still runs on the full raw text. +- `client/scripts/harness/narrate_harness.gd` — switch the button handler to + `narrate_stream`, appending each `on_delta` chunk to the `RichTextLabel` live, + then applying facts at the end (unchanged §2 apply). + +## Error handling + +- **Invalid canon log** → synchronous `422`; client shows fallback (unchanged). +- **Pre-first-token model failure** → `chat_stream` retries once; still failing → + the generator's first frame is `{"error"}` → client shows the full authored + fallback line, no facts. +- **Mid-stream failure** (`{"error"}` after deltas, dropped connection, or an + inter-chunk timeout) → keep the partial prose on screen, mark degraded, harvest + no facts. +- **Stall** (no connect / no first token / gap between chunks past the timeout) → + treated as the matching failure above. +- No surface ever shows an error string (§13). + +## Testing + +- **`TagStreamFilter` (pure, GUT):** a tag split across two deltas is held then + revealed cleanly; a `[FACT:…]` is stripped from the visible reveal; plain text + passes straight through; `flush()` returns a dangling partial. +- **`DmService.narrate_stream` with `FakeStreamTransport`:** deltas accumulate → + `display_text` is the cleaned full text and facts are harvested; a canned + `{"error"}` after deltas → degraded, partial text kept, no facts; a pre-token + failure → full fallback, no facts; the on-screen reveal (via the injected + `on_delta`) never shows raw tag text. +- **Server `chat_stream` with a fake httpx stream:** yields deltas in order; + retries once before the first delta; a mid-stream raise surfaces as `ModelError`. +- **Server `narrate` stream service with a fake `chat_stream`:** emits well-formed + NDJSON frames terminated by `done`; a raise emits a terminal `error` frame; the + call is logged once with the accumulated text. +- **Endpoint (`TestClient`):** a valid log streams `application/x-ndjson` frames + ending in `{"done":true}`; an invalid log still returns `422` JSON (no stream); + a model failure yields a 200 stream whose last frame is `{"error"}`. +- **Gated live stream smoke** (`@pytest.mark.live`): real Ollama, assert deltas + arrive incrementally (more than one frame) and the concatenation is non-empty + grounded prose, no Luck number. + +## Scope + +**In:** proxy NDJSON stream format; `ollama_client.chat_stream`; the streaming +narrate service + `/dm/narrate` streaming endpoint (422 synchronous, model error +as in-stream frame); the client `HTTPClient` `StreamTransport` (+ connect / +first-token / inter-chunk timeouts) behind a testable seam; `StreamResult`; +`DmService.narrate_stream`; the pure `TagStreamFilter` hold-back; keep-partial +mid-stream degrade; `narrate_harness` streaming on screen; full test set + gated +live stream smoke. + +**Out:** NPC streaming (the fast-follow — reuses this transport, adds +parse-at-end); the Adjudicator; combat flavor (async, later); any real dialogue +UI; a total-duration timeout; SSE (NDJSON chosen).