docs(spec): Narrator streaming (M2) design
Real token streaming to hide the ~2s call (§14). Proxy emits its own minimal NDJSON (delta/done/error); /dm/narrate becomes a 200 stream (422 stays sync, model errors become an in-stream frame). Client reworks to an HTTPClient poll-loop StreamTransport behind a testable seam; hold-back tag filter; keep-partial mid-stream degrade; folds in the HttpTransport timeout fix. Narrator first; NPC fast-follow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
254
docs/superpowers/specs/2026-07-10-narrator-streaming-design.md
Normal file
254
docs/superpowers/specs/2026-07-10-narrator-streaming-design.md
Normal file
@@ -0,0 +1,254 @@
|
||||
# Narrator streaming (M2) — design
|
||||
|
||||
Date: 2026-07-10
|
||||
Milestone: **M2 — prove aliveness** (roadmap). The third M2 item; **polish, not
|
||||
proof** — gated on the aliveness question, which the NPC work answered.
|
||||
Charter side (§2): **text** — the client streams and displays AI prose; it still
|
||||
never lets prose write state directly (facts harvested at stream end, applied by
|
||||
the caller).
|
||||
|
||||
## Goal
|
||||
|
||||
Make the ~2s model call read as *the DM is speaking* rather than *the game is
|
||||
frozen* (§14). The server streams tokens from Ollama as they are produced; the
|
||||
client renders them as they arrive — first words on screen in ~200ms instead of
|
||||
a 2s blank wait.
|
||||
|
||||
```
|
||||
CanonLog → HTTP → /dm/narrate (200 NDJSON stream) → tokens as they arrive
|
||||
→ client renders incrementally → at stream end: TagExtractor → fact-harvest
|
||||
```
|
||||
|
||||
This piece proves the streaming mechanism on the **Narrator only** — pure prose,
|
||||
no moves. The NPC surface (which must run move validation on the complete text
|
||||
at stream end) is a deliberate fast-follow on the *same* transport. This is the
|
||||
riskiest transport rework in the POC, so it lands on the simplest surface first.
|
||||
|
||||
## What already exists (and is proven)
|
||||
|
||||
- **Server:** `ollama_client.chat(...)` posts `{"stream": False}` to Ollama's
|
||||
`/api/chat` with one retry, raising `ModelError`; its docstring already says it
|
||||
was "built so a streaming mode is an additive path later." `narrate.run(canon_log)`
|
||||
assembles messages, calls `chat`, logs via `call_log.record(...)`, returns raw
|
||||
prose. `/dm/narrate` returns `{"prose": …}` / `502` / `422`. `call_log.record`
|
||||
takes `response=`/`error=` and an injectable `write` sink.
|
||||
- **Client:** `DmService.narrate(canon_log) -> NarrateResult` posts through an
|
||||
injectable `DmTransport` (`post_json(path, body) -> DmResponse`), degrades to
|
||||
an authored fallback on any failure, harvests `[FACT]` via `TagExtractor`, and
|
||||
returns facts for the caller to apply (§2). The real `HttpTransport` wraps a
|
||||
Godot `HTTPRequest` **Node** (which buffers the whole body and fires
|
||||
`request_completed` once — it cannot stream). `FakeDmTransport` returns a
|
||||
canned `DmResponse`. `narrate_harness.gd` drives it on screen.
|
||||
|
||||
The server is genuinely one flag from streaming. The client is not: Godot's
|
||||
`HTTPRequest` has no incremental-read path, so real streaming requires the
|
||||
lower-level `HTTPClient` (poll + `read_response_body_chunk()`). That rework is
|
||||
the weight of this piece.
|
||||
|
||||
## Decisions
|
||||
|
||||
Settled during brainstorming; each is load-bearing.
|
||||
|
||||
1. **Real token streaming, not a cosmetic reveal.** A client-side typewriter over
|
||||
already-received text does not hide the initial wait — it just animates it.
|
||||
Meeting §14 requires the server to emit tokens as Ollama produces them and the
|
||||
client to render them live. That mandates the `HTTPClient` rework; there is no
|
||||
cheaper path that meets the goal.
|
||||
|
||||
2. **Narrator first; NPC is a fast-follow.** Both prose endpoints share the
|
||||
machinery, but NPC must run `TagExtractor`/`MoveValidator`/`MoveApplier` on the
|
||||
*complete* text at stream end. Proving the transport on the move-free Narrator
|
||||
means a transport bug cannot corrupt move application. The NPC extension reuses
|
||||
this transport unchanged and adds only parse-at-end.
|
||||
|
||||
3. **Proxy emits its own minimal NDJSON, not Ollama's shape.** §4 — the client
|
||||
knows nothing about the model. One JSON object per line:
|
||||
`{"delta": "..."}` per token, terminated by `{"done": true}`, or by
|
||||
`{"error": "..."}` on a model failure. NDJSON (not SSE) because
|
||||
`HTTPClient.read_response_body_chunk()` returns arbitrary byte boundaries; the
|
||||
client buffers bytes and splits on `\n`, and line-delimited JSON is trivial to
|
||||
reassemble. The proxy translates Ollama's `{"message":{"content":…},"done":…}`
|
||||
lines into these frames.
|
||||
|
||||
4. **Endpoint contract shift on `/dm/narrate`: 422 stays synchronous; model
|
||||
errors become an in-stream frame.** Canon-log validation runs **before** any
|
||||
streaming — a bad log still returns a plain `422` JSON (never a stream). A
|
||||
valid log returns a **200 NDJSON stream**. Because a `StreamingResponse`'s
|
||||
status is fixed at 200 once it starts, a **model failure is no longer a `502`**
|
||||
— it is an `{"error": …}` frame inside the 200 stream. Same principle (the
|
||||
client degrades on failure), new shape. The non-streaming JSON contract is
|
||||
retired for this endpoint (the client is its only consumer and moves to
|
||||
streaming).
|
||||
|
||||
5. **§12 retry moves to before-first-token, server-side.** Once bytes flow you
|
||||
cannot cleanly retry. `chat_stream` retries the Ollama call **only if it fails
|
||||
before yielding the first delta**; once streaming, an error propagates as the
|
||||
`{"error"}` frame. The client never retries a stream — it degrades. This keeps
|
||||
the retry in one place (server) and the client purely reactive, as today.
|
||||
|
||||
6. **Client streaming transport coexists with `HttpTransport`.** The proven
|
||||
non-streaming transport keeps its job (the Adjudicator's strict JSON, later).
|
||||
A new `StreamTransport` seam is added, not a replacement. `DmService` gains a
|
||||
streaming method; the old `narrate()` path may remain for reference but the
|
||||
harness uses the streaming one.
|
||||
|
||||
7. **Testable seam: `post_stream(path, body, on_delta) -> StreamResult`.** The
|
||||
service passes an `on_delta: Callable` that buffers text and forwards to the
|
||||
UI. `StreamResult` carries `{ok: bool, error: String}` at the end. A
|
||||
`FakeStreamTransport` invokes `on_delta` for each canned chunk then returns a
|
||||
canned `StreamResult`, so the service's accumulate/hold-back/degrade logic is
|
||||
pure and unit-testable. The real `HTTPClient` poll loop is the one untested
|
||||
shim — exactly the posture `HttpTransport` holds today.
|
||||
|
||||
8. **On-screen tags: hold-back buffering.** When a `[` arrives, the visible
|
||||
reveal holds from that point until the matching `]`, then the tag is stripped.
|
||||
This prevents `[FACT: the we…` flashing then vanishing, which reads as a bug
|
||||
and undercuts the "feels alive" goal. Implemented as a small running buffer in
|
||||
the on-delta path — the full raw text is still accumulated separately for the
|
||||
end-of-stream `TagExtractor` pass.
|
||||
|
||||
9. **Mid-stream failure: keep the partial prose, mark degraded, harvest no
|
||||
facts.** If the stream dies after some tokens, the player keeps the valid
|
||||
prose already shown, the result is flagged `degraded`, and **no** facts are
|
||||
harvested — the tag pipeline needs the complete text to be trustworthy. The
|
||||
pre-first-token failure path (retry once server-side, then fail) shows the full
|
||||
authored fallback, as today. No surface ever shows an error (§13).
|
||||
|
||||
10. **The `HttpTransport` timeout gap is closed here.** The `HTTPClient` transport
|
||||
gets a **connect timeout + a time-to-first-token timeout + an inter-chunk
|
||||
timeout** — not a total-duration timeout (a long stream is legitimate). A stall
|
||||
at any of those is a stream failure → degrade. This retires the deferred
|
||||
`_http.timeout=0` minor in the code that is reworking the transport anyway.
|
||||
|
||||
## Contracts
|
||||
|
||||
### `POST /dm/narrate` — request unchanged
|
||||
|
||||
Body: `{"canon_log": { …schema-v1… }}`.
|
||||
|
||||
### Responses
|
||||
|
||||
- Invalid log → `422 {"detail": {"canon_log_errors": [ … ]}}` (synchronous JSON,
|
||||
before any stream — unchanged).
|
||||
- Valid log → `200`, `Content-Type: application/x-ndjson`, body is a sequence of
|
||||
newline-terminated JSON objects:
|
||||
|
||||
```
|
||||
{"delta": "The chamber "}
|
||||
{"delta": "is cold. "}
|
||||
{"delta": "[FACT: the well is dry]"}
|
||||
{"done": true}
|
||||
```
|
||||
|
||||
A model failure replaces the terminal frame:
|
||||
|
||||
```
|
||||
{"delta": "The chamber "}
|
||||
{"error": "connection reset"}
|
||||
```
|
||||
|
||||
`done` and `error` are mutually exclusive and terminal. Frames carry no model
|
||||
identity, seed, or Ollama metadata (§4).
|
||||
|
||||
## Components
|
||||
|
||||
**Server:**
|
||||
|
||||
- `api/app/ollama_client.py` — add `chat_stream(model, messages, options, *,
|
||||
think=None, client=None) -> Iterator[str]`. Uses `httpx` `client.stream("POST",
|
||||
"/api/chat", json={... "stream": True ...})`, iterates `iter_lines()`, parses
|
||||
each Ollama line, yields `message.content` deltas. Retries once **only** if the
|
||||
request fails before the first delta; a failure after the first delta raises
|
||||
`ModelError` from within the generator (the service converts it to an error
|
||||
frame). The existing `chat()` is untouched.
|
||||
- `api/app/narrate.py` — add `run_stream(canon_log) -> Iterator[str]` (or a
|
||||
sibling module `narrate_stream.py` if `narrate.py` grows unwieldy): assembles
|
||||
the same messages, calls `chat_stream`, yields proxy NDJSON frames
|
||||
(`{"delta":…}` / terminal `{"done":true}` / `{"error":…}`), accumulates the full
|
||||
text, and logs via `call_log.record(...)` at stream end (both success — with the
|
||||
accumulated `response` — and error paths). Logs time-to-first-token.
|
||||
- `api/app/main.py` — `/dm/narrate` validates the canon log synchronously
|
||||
(unchanged `valid_turn`), then returns
|
||||
`StreamingResponse(narrate_stream_gen(...), media_type="application/x-ndjson")`.
|
||||
A `ModelError` raised before the first frame still surfaces as an in-stream
|
||||
`{"error"}` frame (the generator catches and emits it), not a 502.
|
||||
|
||||
**Client:**
|
||||
|
||||
- `client/scripts/net/stream_transport.gd` — `StreamTransport` (`class_name`), the
|
||||
seam: `post_stream(path: String, body: Dictionary, on_delta: Callable) ->
|
||||
StreamResult` (async). Base pushes an error like `DmTransport` does.
|
||||
- `client/scripts/net/http_stream_transport.gd` — the real one: wraps an
|
||||
`HTTPClient`, connects, POSTs, polls `poll()` + `read_response_body_chunk()`,
|
||||
buffers bytes, splits on `\n`, parses each frame, calls `on_delta(delta)` per
|
||||
`{"delta"}`, and returns a `StreamResult` on `{"done"}` / `{"error"}` /
|
||||
transport failure / timeout. Untested shim (verified by the harness + gated live
|
||||
smoke), like `HttpTransport`. Holds the connect / first-token / inter-chunk
|
||||
timeouts (§decision 10).
|
||||
- `client/scripts/net/stream_result.gd` — `StreamResult{ok: bool, error: String}`
|
||||
+ `StreamResult.failed(msg)`.
|
||||
- `client/scripts/net/dm_service.gd` — add `narrate_stream(canon_log, on_delta:
|
||||
Callable) -> NarrateResult`. It wraps the caller's `on_delta` in a hold-back
|
||||
buffer (reveals text, holding an open `[`…`]` span), accumulates the full raw
|
||||
text, awaits `post_stream`, and at the end: if the `StreamResult` failed with no
|
||||
text → `NarrateResult.fallback(...)`; if it failed mid-stream → degraded result
|
||||
carrying the partial `display_text` and **no** facts; on success →
|
||||
`TagExtractor.extract(full_text)` and return `{clean_text, facts, degraded:false}`.
|
||||
Reuses `NarrateResult` and `FallbackLibrary` unchanged.
|
||||
- `client/scripts/text/tag_stream_filter.gd` — a small **pure** hold-back filter:
|
||||
`feed(delta) -> String` returns the safe-to-show text (buffering from an
|
||||
unmatched `[`), `flush() -> String` returns any trailing buffered text at end.
|
||||
Pure and unit-tested; `DmService.narrate_stream` uses it for the on-screen
|
||||
reveal while `TagExtractor` still runs on the full raw text.
|
||||
- `client/scripts/harness/narrate_harness.gd` — switch the button handler to
|
||||
`narrate_stream`, appending each `on_delta` chunk to the `RichTextLabel` live,
|
||||
then applying facts at the end (unchanged §2 apply).
|
||||
|
||||
## Error handling
|
||||
|
||||
- **Invalid canon log** → synchronous `422`; client shows fallback (unchanged).
|
||||
- **Pre-first-token model failure** → `chat_stream` retries once; still failing →
|
||||
the generator's first frame is `{"error"}` → client shows the full authored
|
||||
fallback line, no facts.
|
||||
- **Mid-stream failure** (`{"error"}` after deltas, dropped connection, or an
|
||||
inter-chunk timeout) → keep the partial prose on screen, mark degraded, harvest
|
||||
no facts.
|
||||
- **Stall** (no connect / no first token / gap between chunks past the timeout) →
|
||||
treated as the matching failure above.
|
||||
- No surface ever shows an error string (§13).
|
||||
|
||||
## Testing
|
||||
|
||||
- **`TagStreamFilter` (pure, GUT):** a tag split across two deltas is held then
|
||||
revealed cleanly; a `[FACT:…]` is stripped from the visible reveal; plain text
|
||||
passes straight through; `flush()` returns a dangling partial.
|
||||
- **`DmService.narrate_stream` with `FakeStreamTransport`:** deltas accumulate →
|
||||
`display_text` is the cleaned full text and facts are harvested; a canned
|
||||
`{"error"}` after deltas → degraded, partial text kept, no facts; a pre-token
|
||||
failure → full fallback, no facts; the on-screen reveal (via the injected
|
||||
`on_delta`) never shows raw tag text.
|
||||
- **Server `chat_stream` with a fake httpx stream:** yields deltas in order;
|
||||
retries once before the first delta; a mid-stream raise surfaces as `ModelError`.
|
||||
- **Server `narrate` stream service with a fake `chat_stream`:** emits well-formed
|
||||
NDJSON frames terminated by `done`; a raise emits a terminal `error` frame; the
|
||||
call is logged once with the accumulated text.
|
||||
- **Endpoint (`TestClient`):** a valid log streams `application/x-ndjson` frames
|
||||
ending in `{"done":true}`; an invalid log still returns `422` JSON (no stream);
|
||||
a model failure yields a 200 stream whose last frame is `{"error"}`.
|
||||
- **Gated live stream smoke** (`@pytest.mark.live`): real Ollama, assert deltas
|
||||
arrive incrementally (more than one frame) and the concatenation is non-empty
|
||||
grounded prose, no Luck number.
|
||||
|
||||
## Scope
|
||||
|
||||
**In:** proxy NDJSON stream format; `ollama_client.chat_stream`; the streaming
|
||||
narrate service + `/dm/narrate` streaming endpoint (422 synchronous, model error
|
||||
as in-stream frame); the client `HTTPClient` `StreamTransport` (+ connect /
|
||||
first-token / inter-chunk timeouts) behind a testable seam; `StreamResult`;
|
||||
`DmService.narrate_stream`; the pure `TagStreamFilter` hold-back; keep-partial
|
||||
mid-stream degrade; `narrate_harness` streaming on screen; full test set + gated
|
||||
live stream smoke.
|
||||
|
||||
**Out:** NPC streaming (the fast-follow — reuses this transport, adds
|
||||
parse-at-end); the Adjudicator; combat flavor (async, later); any real dialogue
|
||||
UI; a total-duration timeout; SSE (NDJSON chosen).
|
||||
Reference in New Issue
Block a user