Realtime demarkus: WATCH and live views
An active plan to make demarkus realtime: writes reach watchers, caches, and open views within a latency budget instead of on the next poll. Audience: demarkus maintainers implementing it. Drafted 2026-09-15 as a WATCH-only future plan; widened to realtime levels 1 and 2 on 2026-09-25. No code yet.
Goal and scope
Three levels of realtime were weighed on 2026-09-25:
| Level | What users see | In scope |
|---|---|---|
| 1. Live changes | A write reaches every watcher, cache, and open view within budget | Yes |
| 2. Live views | LOOKUP results, graph neighborhoods, and library panes update themselves | Yes |
| 3. Live collaboration | Concurrent editing of one document, presence, cursors | No. Session state that belongs in a separate room service checkpointing through PUBLISH, never in the store. The event format below leaves room for it. |
The store stays a bookshelf: it emits change hints and nothing more. Everything that interprets events (cache invalidation, live queries, fan-out to browsers) lives at the edge, in shared client packages, the broker, and the library.
Why QUIC matters here
A transport review on 2026-09-25 found that QUIC's original reasons (DESIGN.md: TLS 1.3, migration, multiplexing, 0-RTT, no HTTP) mostly hold over TLS on TCP too or are unused. Realtime is where QUIC pays: one stream per subscription, many on one connection without head-of-line blocking, cancel by stream reset, migration keeping a laptop's subscriptions alive, and client-opened streams so subscribers need no public address. If this plan stalls, the fallback is a TLS on TCP binding for direct clients.
The WebSub-style SUBSCRIBE with a callback URL in docs/DESIGN.md stays removed (backlog): agents are not servers. The SPEC section written in PR 2 replaces the Subscriptions entry in SPEC §13.
Latency budget
Write committed to event delivered, p99. Correctness tests assert delivery with generous bounds; the budget itself is measured by a harness on the benchmarks branch, results recorded in the soul.
| Path | Budget |
|---|---|
| Local server to direct watcher | 100 ms |
| Knowledge-server replica A write to watcher on replica B | 500 ms |
| Any write to a browser through broker relay and library | 1 s |
A budget that cannot be met is a design finding recorded in the plan, not a reason to loosen the test silently.
Gates, resolved
| Gate | Resolution |
|---|---|
| ADR superseding "Verb Set: Complete" | PR 2 writes docs/adr/0025-watch-verb.md, mirrored to /adr/, stating the realtime levels and that level 3 stays out of the store |
| Protocol version on the wire | Waived for WATCH. A server without it answers bad-request (protocol/request.go:102, handler.go:148), never a hang, so WATCH is additive. Fortification 1 stays a separate 1.0 item. |
| A consumer where polling hurts | Agent mailboxes, cache and graph freshness, library live views |
| Connection vs stream error classification | PR 1, before any watch code |
| MCP feature freeze | Not triggered: no new MCP tool. Browser delivery is an SSE endpoint on the broker. |
Consumers
- Agent mailbox. Agents publish to
/agents/<id>/inbox/<ulid>.md; a harness runsdemarkus watchin the background and wakes the agent on a line. Claim by expected version, reply by APPEND, close by ARCHIVE, discover by LOOKUP. - Cache and graph freshness.
demarkus-mcp, the broker world pool, and the library invalidate cached reads and graph nodes by path;resyncclears. Complements ADR 0013. - Library live views. The open document, LOOKUP panes, and graph neighborhoods update in place. Browsers cannot speak QUIC, so events arrive through the broker relay.
Wire shape
Request: WATCH /prefix/ (or an exact document path), request metadata since: <cursor> optional, the usual token.
Response: a stream of frontmatter blocks, no bodies.
- First block:
status: ok,cursor(current head). - Event block:
cursor,path,version,hash,op(publish,append,archive),agentwhen the write carried one. Events are hints; clients fetch by hash for content. - Heartbeat block:
status: ok,cursor, every 20s while idle, so the client detects a stalled stream and keeps its cursor fresh. - Terminal blocks end the stream:
resync(cursor unknown or too old),unauthorized(token revoked or expired),closing(server drain).
Clients ignore unknown keys and unknown op values in event blocks. That rule is what lets level 3 add ops later without a new verb.
Cursor: opaque <epoch>:<seq>. At least once, ordered within an epoch. On resync the client rebuilds from LIST and VERSIONS and resumes from the returned cursor.
ParseResponse reads one response per stream; WATCH needs an incremental block reader in protocol/ with per-block size limits (ADR 0014: the module is wire and stored format).
Replica push
The knowledge server runs two replicas over one bucket. Head receipts carry no paths (bucketstore/model.go:97), and polling puts a floor under latency equal to the interval, so replicas push instead:
- Commit hint. After a commit, a replica sends each peer
head <world> <sequence>over an internal QUIC ALPN (mark-peer) on the Mark port. Peers come from a headless Service in the chart. - On a hint, the peer reads the head, diffs old and new root snapshots per path, and emits in sequence order. Several commits between reads coalesce to the latest version per path; sequences already emitted are dropped.
- Hints are only triggers. A lost hint delays delivery; a forged one costs one head read. NetworkPolicy limits
mark-peerto the replica set, and hints are coalesced per world. - Backstop poll every 5s per world while it has subscribers, bounding delay when hints are lost or a peer is unreachable.
- Epoch is the world ID and sequence the head sequence, so cursors survive restarts.
GCS Pub/Sub notifications were rejected: GCP-only, added infrastructure and IAM, and no latency guarantee.
Live views (level 2)
A shared package client/live (one implementation, per the shared-client guideline) consumes a watch and maintains derived state:
- Cache invalidation: evict by path from the fetch client response cache; clear on
resync. - Live LOOKUP: run the query, watch its scope, debounce events 50 ms, re-run, emit the row diff.
- Live graph: mark nodes stale by path, re-expand only affected neighborhoods.
demarkus-mcp, the broker, and the library call client/live; none reimplements it.
Broker relay
The broker exposes an authenticated SSE endpoint per world and prefix. One upstream WATCH per world and prefix per broker replica fans out to every browser session; per-event authorization uses the session's world access. The relay forwards events and does not interpret them. The library's Go server consumes the relay, runs client/live, and pushes htmx fragments to the browser.
Implementation order
One PR per row. A PR is done when its proof test exists and bash pre-commit.sh passes.
| PR | Scope | Proof |
|---|---|---|
| 1 | Client error classification. Split isTransientError (client/fetch/fetch.go:484) into connection-level (idle timeout, application close, handshake, dial), which evicts, and stream-level (stream reset, one request's deadline), which retries on a new stream and keeps the connection. |
A FETCH whose stream is reset leaves the pooled connection; the next request on it succeeds. A connection close still evicts. |
| 2 | Protocol. ADR 0025, SPEC section, VerbWatch, statuses resync and closing, incremental block codec, cursor type, unknown-key rule. Update the verb table in /architecture/protocol.md. |
Codec round trip and fuzzing; oversize block rejected; unknown keys ignored; an old server answers bad-request. |
| 3 | Change hub server/internal/changefeed, no store deps. Subscribe by prefix and cursor, ring buffer for resume, bounded queue per subscriber, full queue ends that subscriber with resync, write path never blocks. |
Contiguous sequences under concurrent publishers; slow subscriber gets resync while a fast one keeps receiving; ring overflow on resume gives resync. |
| 4 | Single-server serving. backend.Store decorator (the writepolicy.Enforce pattern, backend.go:94) emits after committed Publish, Append, SetArchived. worldruntime.serveStream (runtime.go:137): release the concurrency slot once subscribed, exempt from the request deadline, charge the rate limiter once, per-world and per-connection watch limits below MaxIncomingStreams, heartbeats, closing on drain. Auth: read on the prefix, each event rechecked against the token's paths, token reload rechecks live watches. |
Watch then publish delivers. Drain sends closing and the wait group finishes. A token scoped to /a/** watching / sees no /b/ events. Watches at the limit get an error status while ordinary requests are still served. |
| 5 | Client and CLI. fetch.Client.Watch(ctx, WatchRequest) returns an event iterator, one reader goroutine per watch so unread data cannot stall the shared flow-control window, reconnect with the last cursor, resync surfaced. demarkus watch mark://host/prefix/ prints one tab-separated line per event. |
Mailbox on a local soul: session A publishes, session B's demarkus watch prints it. Server restart gives resync and the watch continues. A FETCH stream reset during a watch leaves the watch delivering. |
| 6 | Replica push. mark-peer ALPN, headless Service and NetworkPolicy in the chart, commit hints, snapshot diff, backstop poll. |
Two replicas: a write through A reaches a watcher on B; with hints blocked, delivery still arrives within the backstop interval; restart B and the watcher resumes without resync; a cursor older than the ring gets resync. |
| 7 | Broker relay. SSE endpoint, one upstream per world and prefix per replica, per-session authorization, reconnect with Last-Event-ID as the cursor. |
Two browser sessions on one prefix share one upstream watch; a session without world access gets no events; a broker restart resumes from Last-Event-ID. |
| 8 | client/live. Cache invalidation, live LOOKUP, live graph; wired into demarkus-mcp and the broker world pool. |
A publish through one session is read fresh by another session's MCP without a TTL wait; a live LOOKUP gains a row when a matching document is tagged. |
| 9 | Library live views (library repo). Open document, LOOKUP panes, graph neighborhood update in place through the relay. | An edit in one browser updates another's open document and pane. |
| 10 | Mailbox skill. Inbox layout, claim by expected version, ack by archive, demarkus watch under a harness monitor. No core change. |
Two agents split a task list with no double claims. |
The latency harness starts on the benchmarks branch alongside PR 5 and grows with PRs 6, 7, and 9.
Risks
- Dumb-server drift. The store now holds subscriptions and replica hints. Kept small: events are hints, no durable subscriber state, overload degrades to
resync, interpretation lives at the edge. - Store as a message bus. Every mailbox message is a stored version. The skill says documents are for durable coordination; chatter belongs elsewhere.
- Fan-out cost. PR 8 makes long-lived watches the norm for every MCP session and broker replica. Measure watch counts and backstop poll cost before enabling it by default.
- Silent misses. A dropped event looks like quiet. Cursors make gaps detectable; tests assert contiguous sequences; heartbeats carry the cursor.
- UDP reachability. Direct watchers inherit QUIC egress limits; browsers and blocked networks go through the relay.
Open questions
- Should a subscription accept LOOKUP filter predicates (
tag=inbox) so the server filters instead of the client? - Should events carry changed section anchors so a live view refetches one section?
- Should the file store persist its sequence so a local restart does not force a cache-wide
resync? - Heartbeat interval: fixed 20s, or advertised by the server in the first block?