# Realtime demarkus: WATCH and live views

An active plan to make demarkus realtime: writes reach watchers, caches, and open views within a latency budget instead of on the next poll. Audience: demarkus maintainers implementing it. Drafted 2026-09-15 as a WATCH-only future plan; widened to realtime levels 1 and 2 on 2026-09-25, with fan-out moved to a QUIC broker the same day. PR 1 and PR 2 implemented 2026-09-27 on branches `fix/fetch-error-classification` and `feat/watch-protocol`, awaiting review; nothing serves WATCH yet.

## Goal and scope

Three levels of realtime were weighed on 2026-09-25:

| Level | What users see | In scope |
|---|---|---|
| 1. Live changes | A write reaches every watcher, cache, and open view within budget | Yes |
| 2. Live views | LOOKUP results, graph neighborhoods, and library panes update themselves | Yes |
| 3. Live collaboration | Concurrent editing of one document, presence, cursors | No. Session state that belongs in a separate room service checkpointing through PUBLISH, never in the store. The event format below leaves room for it. |

The store stays a bookshelf: it emits change hints and nothing more. Everything that interprets or multiplies events (fan-out, caching, live queries, delivery to browsers and agents) lives at the edge: the QUIC broker, shared client packages, the HTTPS adapter, and the library.

## Why QUIC matters here

The [transport review](/plans/quic-first.md) found that QUIC's original reasons mostly hold over TLS on TCP too or are unused, and that the transport is rarely the bottleneck for request and response traffic. Realtime is where QUIC pays: one stream per subscription, many on one connection without head-of-line blocking, cancel by stream reset, migration keeping a laptop's subscriptions alive, client-opened streams so subscribers need no public address, and relay fan-out. If this plan stalls, the fallback is a TLS on TCP binding for direct clients.

The WebSub-style `SUBSCRIBE` with a callback URL in `docs/DESIGN.md` stays removed ([backlog](/roadmap/backlog.md)): agents are not servers. The SPEC section written in PR 2 replaces the Subscriptions entry in SPEC §13.

## Latency budget

Write committed to event delivered, p99. Correctness tests assert delivery with generous bounds; the budget itself is measured by a harness on the `benchmarks` branch, results recorded in the soul.

| Path | Budget |
|---|---|
| Local server to direct watcher | 100 ms |
| Knowledge-server replica A write to watcher on replica B | 500 ms |
| Any write to a watcher on the QUIC broker | 500 ms |
| Any write to a browser | 1 s |

A budget that cannot be met is a design finding recorded in the plan, not a reason to loosen the test silently.

## Gates, resolved

| Gate | Resolution |
|---|---|
| ADR superseding "Verb Set: Complete" | PR 2 wrote [`docs/adr/0025-watch-verb.md`](/adr/0025-watch-verb.md) (proposed 2026-09-27), mirrored to `/adr/`, stating the realtime levels, that level 3 stays out of the store, and that protocol 1.0 freezes after WATCH and ALPN `mark/1` |
| Protocol version on the wire | Waived for WATCH. A server without it answers `bad-request` (`protocol/request.go:102`, `handler.go:148`), never a hang, so WATCH is additive. Pinned by a handler test in PR 2. Fortification 1 stays a separate 1.0 item. |
| A consumer where polling hurts | Agent mailboxes, cache and graph freshness, library live views |
| Connection vs stream error classification | PR 1, done 2026-09-27, before any watch code |
| MCP feature freeze | Not triggered: no new MCP tool. Agents get updates through MCP resource notifications, an existing MCP mechanism. |

## Consumers

1. **Agent mailbox.** Agents publish to `/agents/<id>/inbox/<ulid>.md`; a harness runs `demarkus watch` in the background and wakes the agent on a line. Claim by expected version, reply by APPEND, close by ARCHIVE, discover by LOOKUP.
2. **Cache and graph freshness.** `demarkus-mcp`, the brokers, and the library invalidate cached reads and graph nodes by path; `resync` clears. Complements [ADR 0013](/adr/0013-graph-source-freshness.md).
3. **Library live views.** The open document, LOOKUP panes, and graph neighborhoods update in place.

## Wire shape

Specified in SPEC §6.8 (PR 2). Request: `WATCH /prefix/` (or an exact document path), request metadata `since: <cursor>` optional, the usual token. The path need not exist: a watch on a mailbox directory that is not there yet delivers its creation.

Response: a stream of frontmatter blocks, no bodies.

- First block: `status: ok`, `cursor` (current head). A request refused before subscription is an ordinary response in place of it; the client treats any first block whose status is not `ok` as terminal.
- Event block: `cursor`, `path`, `version`, `hash`, `op` (`publish`, `append`, `archive`), `agent` when the write carried one. Events are hints; clients fetch by hash for content. A block without `status` is an event.
- Heartbeat block: `status: ok`, `cursor`, every 20s while idle, so the client detects a stalled stream (three missed intervals) and keeps its cursor fresh. Fixed in the SPEC as RECOMMENDED; advertising it in the first block was dropped as an open question on 2026-09-27.
- Terminal blocks end the stream: `resync` (cursor unknown or too old), `unauthorized` (token revoked or expired), `closing` (server drain).
- One block is at most 8192 bytes, fences included; block values follow the metadata grammar (no line breaks), which is what lets the reader work line by line.

Clients ignore unknown keys and unknown `op` values in event blocks. That rule lets level 3 add ops later without a new verb, and lets the QUIC broker add inline bodies and coalescing without changing the world.

Cursor: opaque `<epoch>:<seq>`, epoch 1 to 64 characters of `[A-Za-z0-9._-]`, seq a canonical decimal. At least once, ordered within an epoch. Clients never infer a gap from cursor values (read-authorised events are omitted silently); only `resync` signals one. On `resync` the client rebuilds from LIST and VERSIONS and resumes from the returned cursor.

`ParseResponse` reads one response per stream; WATCH needs an incremental block reader (ADR 0014: the module is wire and stored format). PR 2 added it to `protocol/`: `Cursor` and `ParseCursor`, `WatchBlock` with `WriteTo`, `WatchReader.Next`, `WatchEvent` with `Block` and `WatchBlock.Event`, `WatchControl`, `IsKnownOp`, `MaxWatchBlockLength`, `WatchHeartbeatInterval`, statuses `resync` and `closing`. The client side is copied into the [SDK](/plans/demarkus-sdk.md).

## Replica push

The knowledge server runs two replicas over one bucket. Head receipts carry no paths (`bucketstore/model.go:97`), and polling puts a floor under latency equal to the interval, so replicas push instead:

- **Commit hint.** After a commit, a replica sends each peer `head <world> <sequence>` over an internal QUIC ALPN (`mark-peer`) on the Mark port. Peers come from a headless Service in the chart.
- **On a hint,** the peer reads the head, diffs old and new root snapshots per path, and emits in sequence order. Several commits between reads coalesce to the latest version per path; sequences already emitted are dropped.
- **Hints are only triggers.** A lost hint delays delivery; a forged one costs one head read. NetworkPolicy limits `mark-peer` to the replica set, and hints are coalesced per world.
- **Backstop poll** every 5s per world while it has subscribers, bounding delay when hints are lost or a peer is unreachable.
- **Epoch** is the world ID and sequence the head sequence, so cursors survive restarts.

GCS Pub/Sub notifications were rejected: GCP-only, added infrastructure and IAM, and no latency guarantee.

## Delivery paths

Worlds send hints to few subscribers; the QUIC broker multiplies them. Each client type gets one path:

| Client | Path |
|---|---|
| Local tools on a soul | Direct WATCH to the local server |
| SDK apps, CLI, `demarkus-mcp`, the library's Go server | [QUIC broker](/plans/quic-first.md) (workstream 9): one upstream watch per world per replica, filtered and fanned out through a shared ring, with hash-addressed content cache, opt-in inline small bodies, and opt-in coalescing |
| Browsers | Library server pushes htmx fragments; later WebTransport from the QUIC broker, SSE through the HTTPS adapter as fallback |
| Remote MCP hosts over Streamable HTTP | HTTPS adapter exposes documents as MCP resources and sends `notifications/resources/updated`, fed by a watch on the QUIC broker. Gated on checking which MCP hosts act on resource notifications. |

## Live views (level 2)

A shared package `client/live` (one implementation, per the shared-client guideline) consumes a watch and maintains derived state:

- **Cache invalidation:** evict by path from the response cache; clear on `resync`.
- **Live LOOKUP:** run the query, watch its scope, debounce events 50 ms, re-run, emit the row diff.
- **Live graph:** mark nodes stale by path, re-expand only affected neighborhoods.

`demarkus-mcp`, the brokers, and the library call `client/live`; none reimplements it.

## Implementation order

One PR per row. A PR is done when its proof test exists and `bash pre-commit.sh` passes. The [SDK](/plans/demarkus-sdk.md) extraction lands before PR 5, so `Watch` is written once, in the SDK. PR 1 landed in `client/fetch` because extraction had not started; it moves with the file.

| PR | Scope | Proof |
|---|---|---|
| 1 | **Client error classification.** Done 2026-09-27 (`fix/fetch-error-classification`). `isTransientError` split into `isConnectionError` (dial failure, close by either side, idle or handshake timeout, stateless reset, a deadline before the stream was open), which evicts, and `isStreamError` (peer reset, a deadline after the request went out), which retries on a new stream and keeps the connection. A timed out `OpenStreamSync` stays connection-level, the wedge guard from the classify test. After any failure a connection whose context is done is evicted too, so a dead peer is never kept because its failure looked stream-level. | A FETCH whose stream is reset leaves the pooled connection; the next request on it succeeds. A connection close still evicts. Both proven end to end against an in-process QUIC server that counts accepted connections; a table test pins the classification. |
| 2 | **Protocol.** Done 2026-09-27 (`feat/watch-protocol`). ADR 0025, SPEC §6.8 plus §5.2, §7, §8, §11.3, §11.8, §13 and §14, `VerbWatch`, statuses `resync` and `closing`, incremental block codec, cursor type, unknown-key rule. Verb table in `/architecture/protocol.md` updated. `rate-limited` added to SPEC §7, since the server already emits it and WATCH limits answer with it. | Codec round trip and three fuzz targets; oversize block rejected on read and on write; unknown keys and ops kept and skipped; a handler test shows a server without WATCH serving answers `bad-request`. |
| 3 | **Change hub** `server/internal/changefeed`, no store deps. Subscribe by prefix and cursor, ring buffer for resume, bounded queue per subscriber, full queue ends that subscriber with `resync`, write path never blocks. | Contiguous sequences under concurrent publishers; slow subscriber gets `resync` while a fast one keeps receiving; ring overflow on resume gives `resync`. |
| 4 | **Single-server serving.** `backend.Store` decorator (the `writepolicy.Enforce` pattern, `backend.go:94`) emits after committed Publish, Append, SetArchived. `worldruntime.serveStream` (`runtime.go:137`): release the concurrency slot once subscribed, exempt from the request deadline, charge the rate limiter once, per-world and per-connection watch limits below `MaxIncomingStreams`, heartbeats, `closing` on drain. Auth: `read` on the prefix, each event rechecked against the token's paths, token reload rechecks live watches. | Watch then publish delivers. Drain sends `closing` and the wait group finishes. A token scoped to `/a/**` watching `/` sees no `/b/` events. Watches at the limit get `rate-limited` while ordinary requests are still served. |
| 5 | **Client and CLI.** SDK `Watch` returns an event iterator, one reader goroutine per watch so unread data cannot stall the shared flow-control window, reconnect with the last cursor, `resync` surfaced. `demarkus watch mark://host/prefix/` prints one tab-separated line per event. | Mailbox on a local soul: session A publishes, session B's `demarkus watch` prints it. Server restart gives `resync` and the watch continues. A FETCH stream reset during a watch leaves the watch delivering. |
| 6 | **Replica push.** `mark-peer` ALPN, headless Service and NetworkPolicy in the chart, commit hints, snapshot diff, backstop poll. | Two replicas: a write through A reaches a watcher on B; with hints blocked, delivery still arrives within the backstop interval; restart B and the watcher resumes without `resync`; a cursor older than the ring gets `resync`. |
| 7 | **QUIC broker fan-out.** Delivered by [Lean into QUIC](/plans/quic-first.md) workstream 9, after its auth-model ADR. | That workstream's fan-out proof: zero allocations per event per additional watcher, p99 within budget at 1k, 10k, and 100k watchers, resume on any replica. |
| 8 | **`client/live`.** Cache invalidation, live LOOKUP, live graph; wired into `demarkus-mcp`, the brokers, and the library. | A publish through one session is read fresh by another session's MCP without a TTL wait; a live LOOKUP gains a row when a matching document is tagged. |
| 9 | **Library live views** (library repo). The library's Go server watches through the QUIC broker and pushes fragments to browsers. | An edit in one browser updates another's open document and pane. |
| 10 | **MCP resource notifications.** HTTPS adapter maps documents to MCP resources, subscribes through the QUIC broker, sends updates. Starts with a check of host support. | A host that supports resource subscriptions receives an update after a publish; the support table is recorded. |
| 11 | **Mailbox skill.** Inbox layout, claim by expected version, ack by archive, `demarkus watch` under a harness monitor. No core change. | Two agents split a task list with no double claims. |

The latency harness starts on the `benchmarks` branch alongside PR 5 and grows with PRs 6, 7, and 9.

## Risks

- **Dumb-server drift.** The store now holds subscriptions and replica hints. Kept small: events are hints, no durable subscriber state, overload degrades to `resync`, interpretation and fan-out live at the edge.
- **Store as a message bus.** Every mailbox message is a stored version. The skill says documents are for durable coordination; chatter belongs elsewhere.
- **Fan-out cost.** Long-lived watches become the norm for every MCP session, library, and SDK app. The QUIC broker absorbs them so each world sees one watch per broker replica; measure broker memory per watcher before enabling by default.
- **Silent misses.** A dropped event looks like quiet. Cursors make gaps detectable; tests assert contiguous sequences; heartbeats carry the cursor.
- **UDP reachability.** Direct and QUIC broker watchers inherit QUIC egress limits; browsers and blocked networks fall back to SSE through the HTTPS adapter.
- **Slow-peer detection after PR 1.** A read deadline is now stream-level, so a peer that vanished without a close frame costs up to three request timeouts before quic-go's idle timeout (30 s) marks the connection dead and the pool evicts it; before, the first timeout evicted. Acceptable for reads; revisit if the latency harness shows it.

## Open questions

- Should a subscription accept LOOKUP filter predicates (`tag=inbox`) so the server filters instead of the client?
- Should events carry changed section anchors so a live view refetches one section?
- Should the file store persist its sequence so a local restart does not force a cache-wide `resync`?

## Related plans

- [Lean into QUIC](/plans/quic-first.md): the transport review, the QUIC broker and its fan-out design, broker convergence, and the pure-QUIC profile.
- [demarkus-sdk](/plans/demarkus-sdk.md): the client SDK where `Watch` lives.
- [ADR 0025](/adr/0025-watch-verb.md): WATCH is the eighth verb, hints only, protocol freeze after WATCH and ALPN `mark/1`.