# Narrator streaming (M2) — design

> **STATUS: SHELVED for the POC (2026-07-10).** The considering-state
> (`2026-07-10-considering-state-design.md`) shipped and is live-confirmed; it
> meets §14's anti-freeze *goal* by reframing the wait in-character rather than
> hiding it, for a fraction of the risk. Real streaming is the POC's largest
> transport rework (client `HTTPRequest` → poll-based `HTTPClient`) and adds
> mid-stream error/tag-boundary failure surface judged not worth it. The roadmap
> item is struck. This design is retained only as a record — revive it solely if
> playtesting proves the wait genuinely hurts immersion (long Narrator responses
> or a slow prod model). Not planned work.

Date: 2026-07-10
Milestone: **M2 — prove aliveness** (roadmap). The third M2 item; **polish, not
proof** — gated on the aliveness question, which the NPC work answered.
Charter side (§2): **text** — the client streams and displays AI prose; it still
never lets prose write state directly (facts harvested at stream end, applied by
the caller).

## Goal

Make the ~2s model call read as *the DM is speaking* rather than *the game is
frozen* (§14). The server streams tokens from Ollama as they are produced; the
client renders them as they arrive — first words on screen in ~200ms instead of
a 2s blank wait.

```
CanonLog → HTTP → /dm/narrate (200 NDJSON stream) → tokens as they arrive
   → client renders incrementally → at stream end: TagExtractor → fact-harvest
```

This piece proves the streaming mechanism on the **Narrator only** — pure prose,
no moves. The NPC surface (which must run move validation on the complete text
at stream end) is a deliberate fast-follow on the *same* transport. This is the
riskiest transport rework in the POC, so it lands on the simplest surface first.

## What already exists (and is proven)

- **Server:** `ollama_client.chat(...)` posts `{"stream": False}` to Ollama's
  `/api/chat` with one retry, raising `ModelError`; its docstring already says it
  was "built so a streaming mode is an additive path later." `narrate.run(canon_log)`
  assembles messages, calls `chat`, logs via `call_log.record(...)`, returns raw
  prose. `/dm/narrate` returns `{"prose": …}` / `502` / `422`. `call_log.record`
  takes `response=`/`error=` and an injectable `write` sink.
- **Client:** `DmService.narrate(canon_log) -> NarrateResult` posts through an
  injectable `DmTransport` (`post_json(path, body) -> DmResponse`), degrades to
  an authored fallback on any failure, harvests `[FACT]` via `TagExtractor`, and
  returns facts for the caller to apply (§2). The real `HttpTransport` wraps a
  Godot `HTTPRequest` **Node** (which buffers the whole body and fires
  `request_completed` once — it cannot stream). `FakeDmTransport` returns a
  canned `DmResponse`. `narrate_harness.gd` drives it on screen.

The server is genuinely one flag from streaming. The client is not: Godot's
`HTTPRequest` has no incremental-read path, so real streaming requires the
lower-level `HTTPClient` (poll + `read_response_body_chunk()`). That rework is
the weight of this piece.

## Decisions

Settled during brainstorming; each is load-bearing.

1. **Real token streaming, not a cosmetic reveal.** A client-side typewriter over
   already-received text does not hide the initial wait — it just animates it.
   Meeting §14 requires the server to emit tokens as Ollama produces them and the
   client to render them live. That mandates the `HTTPClient` rework; there is no
   cheaper path that meets the goal.

2. **Narrator first; NPC is a fast-follow.** Both prose endpoints share the
   machinery, but NPC must run `TagExtractor`/`MoveValidator`/`MoveApplier` on the
   *complete* text at stream end. Proving the transport on the move-free Narrator
   means a transport bug cannot corrupt move application. The NPC extension reuses
   this transport unchanged and adds only parse-at-end.

3. **Proxy emits its own minimal NDJSON, not Ollama's shape.** §4 — the client
   knows nothing about the model. One JSON object per line:
   `{"delta": "..."}` per token, terminated by `{"done": true}`, or by
   `{"error": "..."}` on a model failure. NDJSON (not SSE) because
   `HTTPClient.read_response_body_chunk()` returns arbitrary byte boundaries; the
   client buffers bytes and splits on `\n`, and line-delimited JSON is trivial to
   reassemble. The proxy translates Ollama's `{"message":{"content":…},"done":…}`
   lines into these frames.

4. **Endpoint contract shift on `/dm/narrate`: 422 stays synchronous; model
   errors become an in-stream frame.** Canon-log validation runs **before** any
   streaming — a bad log still returns a plain `422` JSON (never a stream). A
   valid log returns a **200 NDJSON stream**. Because a `StreamingResponse`'s
   status is fixed at 200 once it starts, a **model failure is no longer a `502`**
   — it is an `{"error": …}` frame inside the 200 stream. Same principle (the
   client degrades on failure), new shape. The non-streaming JSON contract is
   retired for this endpoint (the client is its only consumer and moves to
   streaming).

5. **§12 retry moves to before-first-token, server-side.** Once bytes flow you
   cannot cleanly retry. `chat_stream` retries the Ollama call **only if it fails
   before yielding the first delta**; once streaming, an error propagates as the
   `{"error"}` frame. The client never retries a stream — it degrades. This keeps
   the retry in one place (server) and the client purely reactive, as today.

6. **Client streaming transport coexists with `HttpTransport`.** The proven
   non-streaming transport keeps its job (the Adjudicator's strict JSON, later).
   A new `StreamTransport` seam is added, not a replacement. `DmService` gains a
   streaming method; the old `narrate()` path may remain for reference but the
   harness uses the streaming one.

7. **Testable seam: `post_stream(path, body, on_delta) -> StreamResult`.** The
   service passes an `on_delta: Callable` that buffers text and forwards to the
   UI. `StreamResult` carries `{ok: bool, error: String}` at the end. A
   `FakeStreamTransport` invokes `on_delta` for each canned chunk then returns a
   canned `StreamResult`, so the service's accumulate/hold-back/degrade logic is
   pure and unit-testable. The real `HTTPClient` poll loop is the one untested
   shim — exactly the posture `HttpTransport` holds today.

8. **On-screen tags: hold-back buffering.** When a `[` arrives, the visible
   reveal holds from that point until the matching `]`, then the tag is stripped.
   This prevents `[FACT: the we…` flashing then vanishing, which reads as a bug
   and undercuts the "feels alive" goal. Implemented as a small running buffer in
   the on-delta path — the full raw text is still accumulated separately for the
   end-of-stream `TagExtractor` pass.

9. **Mid-stream failure: keep the partial prose, mark degraded, harvest no
   facts.** If the stream dies after some tokens, the player keeps the valid
   prose already shown, the result is flagged `degraded`, and **no** facts are
   harvested — the tag pipeline needs the complete text to be trustworthy. The
   pre-first-token failure path (retry once server-side, then fail) shows the full
   authored fallback, as today. No surface ever shows an error (§13).

10. **The `HttpTransport` timeout gap is closed here.** The `HTTPClient` transport
   gets a **connect timeout + a time-to-first-token timeout + an inter-chunk
   timeout** — not a total-duration timeout (a long stream is legitimate). A stall
   at any of those is a stream failure → degrade. This retires the deferred
   `_http.timeout=0` minor in the code that is reworking the transport anyway.

## Contracts

### `POST /dm/narrate` — request unchanged

Body: `{"canon_log": { …schema-v1… }}`.

### Responses

- Invalid log → `422 {"detail": {"canon_log_errors": [ … ]}}` (synchronous JSON,
  before any stream — unchanged).
- Valid log → `200`, `Content-Type: application/x-ndjson`, body is a sequence of
  newline-terminated JSON objects:

  ```
  {"delta": "The chamber "}
  {"delta": "is cold. "}
  {"delta": "[FACT: the well is dry]"}
  {"done": true}
  ```

  A model failure replaces the terminal frame:

  ```
  {"delta": "The chamber "}
  {"error": "connection reset"}
  ```

  `done` and `error` are mutually exclusive and terminal. Frames carry no model
  identity, seed, or Ollama metadata (§4).

## Components

**Server:**

- `api/app/ollama_client.py` — add `chat_stream(model, messages, options, *,
  think=None, client=None) -> Iterator[str]`. Uses `httpx` `client.stream("POST",
  "/api/chat", json={... "stream": True ...})`, iterates `iter_lines()`, parses
  each Ollama line, yields `message.content` deltas. Retries once **only** if the
  request fails before the first delta; a failure after the first delta raises
  `ModelError` from within the generator (the service converts it to an error
  frame). The existing `chat()` is untouched.
- `api/app/narrate.py` — add `run_stream(canon_log) -> Iterator[str]` (or a
  sibling module `narrate_stream.py` if `narrate.py` grows unwieldy): assembles
  the same messages, calls `chat_stream`, yields proxy NDJSON frames
  (`{"delta":…}` / terminal `{"done":true}` / `{"error":…}`), accumulates the full
  text, and logs via `call_log.record(...)` at stream end (both success — with the
  accumulated `response` — and error paths). Logs time-to-first-token.
- `api/app/main.py` — `/dm/narrate` validates the canon log synchronously
  (unchanged `valid_turn`), then returns
  `StreamingResponse(narrate_stream_gen(...), media_type="application/x-ndjson")`.
  A `ModelError` raised before the first frame still surfaces as an in-stream
  `{"error"}` frame (the generator catches and emits it), not a 502.

**Client:**

- `client/scripts/net/stream_transport.gd` — `StreamTransport` (`class_name`), the
  seam: `post_stream(path: String, body: Dictionary, on_delta: Callable) ->
  StreamResult` (async). Base pushes an error like `DmTransport` does.
- `client/scripts/net/http_stream_transport.gd` — the real one: wraps an
  `HTTPClient`, connects, POSTs, polls `poll()` + `read_response_body_chunk()`,
  buffers bytes, splits on `\n`, parses each frame, calls `on_delta(delta)` per
  `{"delta"}`, and returns a `StreamResult` on `{"done"}` / `{"error"}` /
  transport failure / timeout. Untested shim (verified by the harness + gated live
  smoke), like `HttpTransport`. Holds the connect / first-token / inter-chunk
  timeouts (§decision 10).
- `client/scripts/net/stream_result.gd` — `StreamResult{ok: bool, error: String}`
  + `StreamResult.failed(msg)`.
- `client/scripts/net/dm_service.gd` — add `narrate_stream(canon_log, on_delta:
  Callable) -> NarrateResult`. It wraps the caller's `on_delta` in a hold-back
  buffer (reveals text, holding an open `[`…`]` span), accumulates the full raw
  text, awaits `post_stream`, and at the end: if the `StreamResult` failed with no
  text → `NarrateResult.fallback(...)`; if it failed mid-stream → degraded result
  carrying the partial `display_text` and **no** facts; on success →
  `TagExtractor.extract(full_text)` and return `{clean_text, facts, degraded:false}`.
  Reuses `NarrateResult` and `FallbackLibrary` unchanged.
- `client/scripts/text/tag_stream_filter.gd` — a small **pure** hold-back filter:
  `feed(delta) -> String` returns the safe-to-show text (buffering from an
  unmatched `[`), `flush() -> String` returns any trailing buffered text at end.
  Pure and unit-tested; `DmService.narrate_stream` uses it for the on-screen
  reveal while `TagExtractor` still runs on the full raw text.
- `client/scripts/harness/narrate_harness.gd` — switch the button handler to
  `narrate_stream`, appending each `on_delta` chunk to the `RichTextLabel` live,
  then applying facts at the end (unchanged §2 apply).

## Error handling

- **Invalid canon log** → synchronous `422`; client shows fallback (unchanged).
- **Pre-first-token model failure** → `chat_stream` retries once; still failing →
  the generator's first frame is `{"error"}` → client shows the full authored
  fallback line, no facts.
- **Mid-stream failure** (`{"error"}` after deltas, dropped connection, or an
  inter-chunk timeout) → keep the partial prose on screen, mark degraded, harvest
  no facts.
- **Stall** (no connect / no first token / gap between chunks past the timeout) →
  treated as the matching failure above.
- No surface ever shows an error string (§13).

## Testing

- **`TagStreamFilter` (pure, GUT):** a tag split across two deltas is held then
  revealed cleanly; a `[FACT:…]` is stripped from the visible reveal; plain text
  passes straight through; `flush()` returns a dangling partial.
- **`DmService.narrate_stream` with `FakeStreamTransport`:** deltas accumulate →
  `display_text` is the cleaned full text and facts are harvested; a canned
  `{"error"}` after deltas → degraded, partial text kept, no facts; a pre-token
  failure → full fallback, no facts; the on-screen reveal (via the injected
  `on_delta`) never shows raw tag text.
- **Server `chat_stream` with a fake httpx stream:** yields deltas in order;
  retries once before the first delta; a mid-stream raise surfaces as `ModelError`.
- **Server `narrate` stream service with a fake `chat_stream`:** emits well-formed
  NDJSON frames terminated by `done`; a raise emits a terminal `error` frame; the
  call is logged once with the accumulated text.
- **Endpoint (`TestClient`):** a valid log streams `application/x-ndjson` frames
  ending in `{"done":true}`; an invalid log still returns `422` JSON (no stream);
  a model failure yields a 200 stream whose last frame is `{"error"}`.
- **Gated live stream smoke** (`@pytest.mark.live`): real Ollama, assert deltas
  arrive incrementally (more than one frame) and the concatenation is non-empty
  grounded prose, no Luck number.

## Scope

**In:** proxy NDJSON stream format; `ollama_client.chat_stream`; the streaming
narrate service + `/dm/narrate` streaming endpoint (422 synchronous, model error
as in-stream frame); the client `HTTPClient` `StreamTransport` (+ connect /
first-token / inter-chunk timeouts) behind a testable seam; `StreamResult`;
`DmService.narrate_stream`; the pure `TagStreamFilter` hold-back; keep-partial
mid-stream degrade; `narrate_harness` streaming on screen; full test set + gated
live stream smoke.

**Out:** NPC streaming (the fast-follow — reuses this transport, adds
parse-at-end); the Adjudicator; combat flavor (async, later); any real dialogue
UI; a total-duration timeout; SSE (NDJSON chosen).