Realtime demarkus: WATCH and live views

An active plan to make demarkus realtime: writes reach watchers, caches, and open views within a latency budget instead of on the next poll. Audience: demarkus maintainers implementing it. Drafted 2026-09-15 as a WATCH-only future plan; widened to realtime levels 1 and 2 on 2026-09-25, with fan-out moved to a QUIC edge the same day. No code yet.

Goal and scope

Three levels of realtime were weighed on 2026-09-25:

Level	What users see	In scope
1. Live changes	A write reaches every watcher, cache, and open view within budget	Yes
2. Live views	LOOKUP results, graph neighborhoods, and library panes update themselves	Yes
3. Live collaboration	Concurrent editing of one document, presence, cursors	No. Session state that belongs in a separate room service checkpointing through PUBLISH, never in the store. The event format below leaves room for it.

The store stays a bookshelf: it emits change hints and nothing more. Everything that interprets or multiplies events (fan-out, caching, live queries, delivery to browsers and agents) lives at the edge: the QUIC edge, shared client packages, the HTTPS broker, and the library.

Why QUIC matters here

The transport review found that QUIC's original reasons mostly hold over TLS on TCP too or are unused, and that the transport is rarely the bottleneck for request and response traffic. Realtime is where QUIC pays: one stream per subscription, many on one connection without head-of-line blocking, cancel by stream reset, migration keeping a laptop's subscriptions alive, client-opened streams so subscribers need no public address, and relay fan-out. If this plan stalls, the fallback is a TLS on TCP binding for direct clients.

The WebSub-style SUBSCRIBE with a callback URL in docs/DESIGN.md stays removed (backlog): agents are not servers. The SPEC section written in PR 2 replaces the Subscriptions entry in SPEC §13.

Latency budget

Write committed to event delivered, p99. Correctness tests assert delivery with generous bounds; the budget itself is measured by a harness on the benchmarks branch, results recorded in the soul.

Path	Budget
Local server to direct watcher	100 ms
Knowledge-server replica A write to watcher on replica B	500 ms
Any write to a watcher on the QUIC edge	500 ms
Any write to a browser	1 s

A budget that cannot be met is a design finding recorded in the plan, not a reason to loosen the test silently.

Gates, resolved

Gate	Resolution
ADR superseding "Verb Set: Complete"	PR 2 writes `docs/adr/0025-watch-verb.md`, mirrored to `/adr/`, stating the realtime levels and that level 3 stays out of the store
Protocol version on the wire	Waived for WATCH. A server without it answers `bad-request` (`protocol/request.go:102`, `handler.go:148`), never a hang, so WATCH is additive. Fortification 1 stays a separate 1.0 item.
A consumer where polling hurts	Agent mailboxes, cache and graph freshness, library live views
Connection vs stream error classification	PR 1, before any watch code
MCP feature freeze	Not triggered: no new MCP tool. Agents get updates through MCP resource notifications, an existing MCP mechanism.

Consumers

Agent mailbox. Agents publish to /agents/<id>/inbox/<ulid>.md; a harness runs demarkus watch in the background and wakes the agent on a line. Claim by expected version, reply by APPEND, close by ARCHIVE, discover by LOOKUP.
Cache and graph freshness. demarkus-mcp, the broker world pool, and the library invalidate cached reads and graph nodes by path; resync clears. Complements ADR 0013.
Library live views. The open document, LOOKUP panes, and graph neighborhoods update in place.

Wire shape

Request: WATCH /prefix/ (or an exact document path), request metadata since: <cursor> optional, the usual token.

Response: a stream of frontmatter blocks, no bodies.

First block: status: ok, cursor (current head).
Event block: cursor, path, version, hash, op (publish, append, archive), agent when the write carried one. Events are hints; clients fetch by hash for content.
Heartbeat block: status: ok, cursor, every 20s while idle, so the client detects a stalled stream and keeps its cursor fresh.
Terminal blocks end the stream: resync (cursor unknown or too old), unauthorized (token revoked or expired), closing (server drain).

Clients ignore unknown keys and unknown op values in event blocks. That rule lets level 3 add ops later without a new verb, and lets the QUIC edge add an inline body to events without changing the world.

Cursor: opaque <epoch>:<seq>. At least once, ordered within an epoch. On resync the client rebuilds from LIST and VERSIONS and resumes from the returned cursor.

ParseResponse reads one response per stream; WATCH needs an incremental block reader (ADR 0014: the module is wire and stored format). The client side lives in the SDK.

Replica push

The knowledge server runs two replicas over one bucket. Head receipts carry no paths (bucketstore/model.go:97), and polling puts a floor under latency equal to the interval, so replicas push instead:

Commit hint. After a commit, a replica sends each peer head <world> <sequence> over an internal QUIC ALPN (mark-peer) on the Mark port. Peers come from a headless Service in the chart.
On a hint, the peer reads the head, diffs old and new root snapshots per path, and emits in sequence order. Several commits between reads coalesce to the latest version per path; sequences already emitted are dropped.
Hints are only triggers. A lost hint delays delivery; a forged one costs one head read. NetworkPolicy limits mark-peer to the replica set, and hints are coalesced per world.
Backstop poll every 5s per world while it has subscribers, bounding delay when hints are lost or a peer is unreachable.
Epoch is the world ID and sequence the head sequence, so cursors survive restarts.

GCS Pub/Sub notifications were rejected: GCP-only, added infrastructure and IAM, and no latency guarantee.

Delivery paths

Worlds send hints to few subscribers; the edge multiplies them. Each client type gets one path:

Client	Path
Local tools on a soul	Direct WATCH to the local server
SDK apps, CLI, the library's Go server	QUIC edge (workstream 9): one upstream WATCH per world and prefix, fanned out, with edge ring buffer, hash-addressed content cache, and opt-in inline small bodies
Browsers	Library server pushes htmx fragments; later WebTransport from the edge, SSE as fallback
MCP agents	HTTPS broker exposes documents as MCP resources and sends `notifications/resources/updated`, fed by a watch on the edge. Gated on checking which MCP hosts act on resource notifications.

Live views (level 2)

A shared package client/live (one implementation, per the shared-client guideline) consumes a watch and maintains derived state:

Cache invalidation: evict by path from the response cache; clear on resync.
Live LOOKUP: run the query, watch its scope, debounce events 50 ms, re-run, emit the row diff.
Live graph: mark nodes stale by path, re-expand only affected neighborhoods.

demarkus-mcp, the broker, and the library call client/live; none reimplements it.

Implementation order

One PR per row. A PR is done when its proof test exists and bash pre-commit.sh passes. The SDK extraction lands before PR 5, so PR 1 and Watch are written once, in the SDK.

PR	Scope	Proof
1	Client error classification. Split `isTransientError` (`client/fetch/fetch.go:484`) into connection-level (idle timeout, application close, handshake, dial), which evicts, and stream-level (stream reset, one request's deadline), which retries on a new stream and keeps the connection.	A FETCH whose stream is reset leaves the pooled connection; the next request on it succeeds. A connection close still evicts.
2	Protocol. ADR 0025, SPEC section, `VerbWatch`, statuses `resync` and `closing`, incremental block codec, cursor type, unknown-key rule. Update the verb table in `/architecture/protocol.md`.	Codec round trip and fuzzing; oversize block rejected; unknown keys ignored; an old server answers `bad-request`.
3	Change hub `server/internal/changefeed`, no store deps. Subscribe by prefix and cursor, ring buffer for resume, bounded queue per subscriber, full queue ends that subscriber with `resync`, write path never blocks.	Contiguous sequences under concurrent publishers; slow subscriber gets `resync` while a fast one keeps receiving; ring overflow on resume gives `resync`.
4	Single-server serving. `backend.Store` decorator (the `writepolicy.Enforce` pattern, `backend.go:94`) emits after committed Publish, Append, SetArchived. `worldruntime.serveStream` (`runtime.go:137`): release the concurrency slot once subscribed, exempt from the request deadline, charge the rate limiter once, per-world and per-connection watch limits below `MaxIncomingStreams`, heartbeats, `closing` on drain. Auth: `read` on the prefix, each event rechecked against the token's paths, token reload rechecks live watches.	Watch then publish delivers. Drain sends `closing` and the wait group finishes. A token scoped to `/a/**` watching `/` sees no `/b/` events. Watches at the limit get an error status while ordinary requests are still served.
5	Client and CLI. SDK `Watch` returns an event iterator, one reader goroutine per watch so unread data cannot stall the shared flow-control window, reconnect with the last cursor, `resync` surfaced. `demarkus watch mark://host/prefix/` prints one tab-separated line per event.	Mailbox on a local soul: session A publishes, session B's `demarkus watch` prints it. Server restart gives `resync` and the watch continues. A FETCH stream reset during a watch leaves the watch delivering.
6	Replica push. `mark-peer` ALPN, headless Service and NetworkPolicy in the chart, commit hints, snapshot diff, backstop poll.	Two replicas: a write through A reaches a watcher on B; with hints blocked, delivery still arrives within the backstop interval; restart B and the watcher resumes without `resync`; a cursor older than the ring gets `resync`.
7	QUIC edge fan-out. Delivered by Lean into QUIC workstream 9, after its auth-model ADR.	That workstream's proof: 1,000 watchers on one prefix hold one upstream watch and all receive within budget.
8	`client/live`. Cache invalidation, live LOOKUP, live graph; wired into `demarkus-mcp`, the broker world pool, and the library.	A publish through one session is read fresh by another session's MCP without a TTL wait; a live LOOKUP gains a row when a matching document is tagged.
9	Library live views (library repo). The library's Go server watches through the edge and pushes fragments to browsers.	An edit in one browser updates another's open document and pane.
10	MCP resource notifications. HTTPS broker maps documents to MCP resources, subscribes through the edge, sends updates. Starts with a check of host support.	A host that supports resource subscriptions receives an update after a publish; the support table is recorded.
11	Mailbox skill. Inbox layout, claim by expected version, ack by archive, `demarkus watch` under a harness monitor. No core change.	Two agents split a task list with no double claims.

The latency harness starts on the benchmarks branch alongside PR 5 and grows with PRs 6, 7, and 9.

Risks

Dumb-server drift. The store now holds subscriptions and replica hints. Kept small: events are hints, no durable subscriber state, overload degrades to resync, interpretation and fan-out live at the edge.
Store as a message bus. Every mailbox message is a stored version. The skill says documents are for durable coordination; chatter belongs elsewhere.
Fan-out cost. Long-lived watches become the norm for every MCP session, library, and SDK app. The edge absorbs them so worlds see one watch per prefix; measure edge memory per watcher before enabling by default.
Silent misses. A dropped event looks like quiet. Cursors make gaps detectable; tests assert contiguous sequences; heartbeats carry the cursor.
UDP reachability. Direct and edge watchers inherit QUIC egress limits; browsers and blocked networks fall back to SSE through the library or broker.

Open questions

Should a subscription accept LOOKUP filter predicates (tag=inbox) so the server filters instead of the client?
Should events carry changed section anchors so a live view refetches one section?
Should the file store persist its sequence so a local restart does not force a cache-wide resync?
Heartbeat interval: fixed 20s, or advertised by the server in the first block?

Lean into QUIC: the transport review, and the QUIC edge that carries fan-out.
demarkus-sdk: the client SDK where Watch lives.