Realtime demarkus: WATCH and live views

An active plan to make demarkus realtime: writes reach watchers, caches, and open views within a latency budget instead of on the next poll. Audience: demarkus maintainers implementing it. Drafted 2026-09-15 as a WATCH-only future plan; widened to realtime levels 1 and 2 on 2026-09-25, with fan-out moved to a QUIC broker the same day. Level 1 (PRs 1 to 5 and the backstop half of PR 6) implemented 2026-09-27 on branch feat/live-track-front, with ADR 0025 and ADR 0026, awaiting review; the smoke on a real server and CLI passed.

Goal and scope

Three levels of realtime were weighed on 2026-09-25:

Level	What users see	In scope
1. Live changes	A write reaches every watcher, cache, and open view within budget	Yes
2. Live views	LOOKUP results, graph neighborhoods, and library panes update themselves	Yes
3. Live collaboration	Concurrent editing of one document, presence, cursors	No. Session state that belongs in a separate room service checkpointing through PUBLISH, never in the store. The event format below leaves room for it.

The store stays a bookshelf: it emits change hints and nothing more. Everything that interprets or multiplies events (fan-out, caching, live queries, delivery to browsers and agents) lives at the edge: the QUIC broker, shared client packages, the HTTPS adapter, and the library.

Why QUIC matters here

The transport review found that QUIC's original reasons mostly hold over TLS on TCP too or are unused, and that the transport is rarely the bottleneck for request and response traffic. Realtime is where QUIC pays: one stream per subscription, many on one connection without head-of-line blocking, cancel by stream reset, migration keeping a laptop's subscriptions alive, client-opened streams so subscribers need no public address, and relay fan-out. If this plan stalls, the fallback is a TLS on TCP binding for direct clients.

The WebSub-style SUBSCRIBE with a callback URL in docs/DESIGN.md stays removed (backlog): agents are not servers. The SPEC section written in PR 2 replaces the Subscriptions entry in SPEC §13.

Latency budget

Write committed to event delivered, p99. Correctness tests assert delivery with generous bounds; the budget itself is measured by a harness on the benchmarks branch, results recorded in the soul.

Path	Budget
Local server to direct watcher	100 ms
Knowledge-server replica A write to watcher on replica B	500 ms
Any write to a watcher on the QUIC broker	500 ms
Any write to a browser	1 s

A budget that cannot be met is a design finding recorded in the plan, not a reason to loosen the test silently. Finding 2026-09-27: without commit hints, replica A to a watcher on B is bounded by the 5 s backstop poll, not 500 ms; the hint push in PR 6 closes it.

Gates, resolved

Gate	Resolution
ADR superseding "Verb Set: Complete"	PR 2 wrote `docs/adr/0025-watch-verb.md` (proposed 2026-09-27), mirrored to `/adr/`, stating the realtime levels, that level 3 stays out of the store, and that protocol 1.0 freezes after WATCH and ALPN `mark/1`
Protocol version on the wire	Waived for WATCH. A server without it answers `bad-request` (`protocol/request.go:102`, `handler.go:148`), never a hang, so WATCH is additive. Pinned by a handler test in PR 2. Fortification 1 stays a separate 1.0 item.
A consumer where polling hurts	Agent mailboxes, cache and graph freshness, library live views
Connection vs stream error classification	PR 1, done 2026-09-27, before any watch code
MCP feature freeze	Not triggered: no new MCP tool. Agents get updates through MCP resource notifications, an existing MCP mechanism.

Consumers

Agent mailbox. Agents publish to /agents/<id>/inbox/<ulid>.md; a harness runs demarkus watch in the background and wakes the agent on a line. Claim by expected version, reply by APPEND, close by ARCHIVE, discover by LOOKUP.
Cache and graph freshness. demarkus-mcp, the brokers, and the library invalidate cached reads and graph nodes by path; resync clears. Complements ADR 0013.
Library live views. The open document, LOOKUP panes, and graph neighborhoods update in place.

Wire shape

Specified in SPEC §6.8 (PR 2). Request: WATCH /prefix/ (or an exact document path), request metadata since: <cursor> optional, the usual token. The path need not exist: a watch on a mailbox directory that is not there yet delivers its creation.

Response: a stream of frontmatter blocks, no bodies.

First block: status: ok, cursor (the position the stream continues from: the head for a new watch, the client's since for a resumed one). A request refused before subscription is an ordinary response in place of it; the client treats any first block whose status is not ok as terminal.
Event block: cursor, path, version, hash, op (publish, append, archive), agent when the write carried one. Events are hints; clients fetch by hash for content. A block without status is an event.
Heartbeat block: status: ok, cursor, every 20s while idle, so the client detects a stalled stream (three missed intervals) and keeps its cursor fresh. Fixed in the SPEC as RECOMMENDED; advertising it in the first block was dropped as an open question on 2026-09-27.
Terminal blocks end the stream: resync (cursor unknown or too old), unauthorized or not-permitted (token revoked, expired, or no longer covering the path), closing (server drain).
One block is at most 8192 bytes, fences included; block values follow the metadata grammar (no line breaks), which is what lets the reader work line by line.

Clients ignore unknown keys and unknown op values in event blocks. That rule lets level 3 add ops later without a new verb, and lets the QUIC broker add inline bodies and coalescing without changing the world.

Cursor: opaque <epoch>:<seq>, epoch 1 to 64 characters of [A-Za-z0-9._-], seq a canonical decimal. At least once, ordered within an epoch. Clients never infer a gap from cursor values (read-authorised events are omitted silently); only resync signals one. On resync the client rebuilds from LIST and VERSIONS and resumes from the returned cursor.

ParseResponse reads one response per stream; WATCH needs an incremental block reader (ADR 0014: the module is wire and stored format). PR 2 added it to protocol/: Cursor and ParseCursor, WatchBlock with WriteTo, WatchReader.Next, WatchEvent with Block and WatchBlock.Event, WatchControl, IsKnownOp, MaxWatchBlockLength, WatchHeartbeatInterval, statuses resync and closing. The client side is copied into the SDK.

Replica push

The knowledge server runs two replicas over one bucket. Head receipts carry no paths (bucketstore/model.go:97), and polling puts a floor under latency equal to the interval, so replicas push instead:

Commit hint. After a commit, a replica sends each peer head <world> <sequence> over an internal QUIC ALPN (mark-peer) on the Mark port. Peers come from a headless Service in the chart.
On a hint, the peer reads the head, diffs old and new root snapshots per path, and emits in sequence order. Several commits between reads coalesce to the latest version per path; sequences already emitted are dropped.
Hints are only triggers. A lost hint delays delivery; a forged one costs one head read. NetworkPolicy limits mark-peer to the replica set, and hints are coalesced per world.
Backstop poll every 5s per world while it has subscribers, bounding delay when hints are lost or a peer is unreachable.
Epoch is the world ID and sequence the head sequence, so cursors survive restarts.

GCS Pub/Sub notifications were rejected: GCP-only, added infrastructure and IAM, and no latency guarantee.

Built 2026-09-27: the snapshot diff and the backstop poll. bucketstore reports its own commits with the exact op and agent, and Poll refreshes the head and reports every path whose (version, archived, body hash) changed, with the op inferred (an append seen only in the snapshot is a publish; the version and hash are exact); the first report is the baseline, and a snapshot older than the last reported is ignored so a refresh racing a local commit cannot announce a superseded version. The knowledge server polls every world with open watches every 5 s. Not built: the commit hint over mark-peer, the chart's headless Service and NetworkPolicy, and the durable epoch, so cursors are per process and a restart or a move to the other replica costs one resync.

Delivery paths

Worlds send hints to few subscribers; the QUIC broker multiplies them. Each client type gets one path:

Client	Path
Local tools on a soul	Direct WATCH to the local server
SDK apps, CLI, `demarkus-mcp`, the library's Go server	QUIC broker (workstream 9): one upstream watch per world per replica, filtered and fanned out through a shared ring, with hash-addressed content cache, opt-in inline small bodies, and opt-in coalescing
Browsers	Library server pushes htmx fragments; later WebTransport from the QUIC broker, SSE through the HTTPS adapter as fallback
Remote MCP hosts over Streamable HTTP	HTTPS adapter exposes documents as MCP resources and sends `notifications/resources/updated`, fed by a watch on the QUIC broker. Gated on checking which MCP hosts act on resource notifications.

Live views (level 2)

A shared package client/live (one implementation, per the shared-client guideline) consumes a watch and maintains derived state:

Cache invalidation: evict by path from the response cache; clear on resync.
Live LOOKUP: run the query, watch its scope, debounce events 50 ms, re-run, emit the row diff.
Live graph: mark nodes stale by path, re-expand only affected neighborhoods.

demarkus-mcp, the brokers, and the library call client/live; none reimplements it.

Implementation order

One PR per row. A PR is done when its proof test exists and bash pre-commit.sh passes. The SDK extraction lands before PR 5, so Watch is written once, in the SDK. PR 1 landed in client/fetch because extraction had not started; it moves with the file. PRs 1 to 6 shipped together with ADR 0026 on one branch at Fritz's request.

PR	Scope	Proof
1	Client error classification. Done 2026-09-27 (`feat/live-track-front`). `isTransientError` split into `isConnectionError` (dial failure, close by either side, idle or handshake timeout, stateless reset, a deadline before the stream was open), which evicts, and `isStreamError` (peer reset, a deadline after the request went out), which retries on a new stream and keeps the connection. A timed out `OpenStreamSync` stays connection-level, the wedge guard from the classify test. After any failure a connection whose context is done is evicted too, so a dead peer is never kept because its failure looked stream-level.	A FETCH whose stream is reset leaves the pooled connection; the next request on it succeeds. A connection close still evicts. Both proven end to end against an in-process QUIC server that counts accepted connections; a table test pins the classification.
2	Protocol. Done 2026-09-27 (`feat/live-track-front`). ADR 0025, SPEC §6.8 plus §5.2, §7, §8, §11.3, §11.8, §13 and §14, `VerbWatch`, statuses `resync` and `closing`, incremental block codec, cursor type, unknown-key rule. Verb table in `/architecture/protocol.md` updated. `rate-limited` added to SPEC §7, since the server already emits it and WATCH limits answer with it.	Codec round trip and three fuzz targets; oversize block rejected on read and on write; unknown keys and ops kept and skipped; a handler test shows a server without WATCH serving answers `bad-request`.
3	Change hub `server/internal/changefeed`. Done 2026-09-27. A ring of events with hub-assigned sequences (`PublishAt` for a caller's own), a broadcast channel per publish, and per-subscriber read positions: no per-subscriber queue, the ring is the queue, so the write path never blocks and a lapped reader gets `resync`. `Subscribe(scope, since)` resumes after a cursor when the epoch matches and nothing after it was overwritten. `Subscription.Cursor` is the last consumed event, matching or not, so a heartbeat's resume point stays current. `Emit` decorates a `backend.Store` for the file store.	Contiguous sequences under 8 concurrent publishers; a slow subscriber gets `resync` while a fast one keeps receiving and publishing never waited; resume at every ring boundary; scope filter; Close wakes waiters.
4	Single-server serving. Done 2026-09-27. `handler.ReadRequest` and `handler.Serve` split `HandleStream` so the runtime sees the verb; `serveWatch` subscribes, acknowledges, then pumps events (each rechecked against the current token store, omitted when not readable), heartbeats (which recheck the scope and end the watch with `unauthorized` or `not-permitted` after a reload), and the terminal block. `worldruntime` admits a watch against per-world and per-connection caps (`quicserve.ConnState` in the stream context), releases the concurrency slot once subscribed, exempts the watch from the request deadline while bounding each block write, and `Drain` closes the hub so a shutdown ends watches with `closing` before the listener drains (found by the smoke: without it the old process held the port for the full 10 s). `DEMARKUS_MAX_WATCHES` and `limits.maxWatches` configure the caps; the per-connection cap is half the stream limit.	Watch then publish delivers, including on a prefix that does not exist yet; fetch by the event's hash returns the body; a token scoped to `/a/**` watching `/` sees `/a/` and public events and no `/b/`; a revoked token ends the watch within one heartbeat; resume replays after the cursor, another epoch gets `resync` in place of the ack; a lapped watch gets `resync`; Drain and Close send `closing` and the wait group finishes; watches at the world or connection cap get `rate-limited` while a FETCH still passes; a subscribed watch holds no concurrency slot.
5	Client and CLI. Done 2026-09-27. `fetch.Client.Watch` returns a `Watch` with `Next`, `Cursor`, `Err`, `Close`: one goroutine reads the stream into a bounded queue; a full queue cancels the stream (freeing the connection's window) and reopens from the cursor once there is room, losing nothing; `closing`, a cut stream, a stall of three heartbeats or a lost connection reopen from the cursor with backoff; `resync` is a `Notice`; a refusal is a `StatusError`. The reader that read the acknowledgement stays with the stream, since its buffer may already hold the events that followed (a bug the slow-consumer test caught). `demarkus watch [-since] [-auth] mark://host/prefix/` prints `cursor op path version hash agent` per line, `resync` as an op.	Against an in-process feed: delivery and cursor tracking, reopen after `closing` with `since`, resync surfaced and continued, refusal and `unauthorized` as errors, redial after a dropped connection, a consumer 3× the queue behind loses nothing. Smoke against the real server: publish and append lines with agent and hash; restart of the server gives one `resync` line and the watch continues; Ctrl-C exits 0; the server drains in 13 ms.
6	Replica push. Partly done 2026-09-27: snapshot diff and backstop poll (see Replica push). Open: `mark-peer` ALPN, headless Service and NetworkPolicy in the chart, commit hints, durable epoch.	Done: a write through A reaches B's hub after a poll, with the op exact on A and inferred on B; A's own writes are never re-reported; a replica opened later takes the world as baseline. Open: delivery within 500 ms with hints; restart B and resume without `resync`.
7	QUIC broker fan-out. Delivered by Lean into QUIC workstream 9, after its auth-model ADR.	That workstream's fan-out proof: zero allocations per event per additional watcher, p99 within budget at 1k, 10k, and 100k watchers, resume on any replica.
8	`client/live`. Cache invalidation, live LOOKUP, live graph; wired into `demarkus-mcp`, the brokers, and the library. Note 2026-09-27: `demarkus-mcp`'s response cache already revalidates every read with `if-none-match`, so invalidation buys round trips, not freshness; the broker caches are where a TTL wait exists.	A publish through one session is read fresh by another session's MCP without a TTL wait; a live LOOKUP gains a row when a matching document is tagged.
9	Library live views (library repo). The library's Go server watches through the QUIC broker and pushes fragments to browsers.	An edit in one browser updates another's open document and pane.
10	MCP resource notifications. HTTPS adapter maps documents to MCP resources, subscribes through the QUIC broker, sends updates. Starts with a check of host support.	A host that supports resource subscriptions receives an update after a publish; the support table is recorded.
11	Mailbox skill. Inbox layout, claim by expected version, ack by archive, `demarkus watch` under a harness monitor. No core change.	Two agents split a task list with no double claims.

The latency harness starts on the benchmarks branch alongside PR 5 and grows with PRs 6, 7, and 9.

Risks

Dumb-server drift. The store now holds subscriptions and replica hints. Kept small: events are hints, no durable subscriber state, overload degrades to resync, interpretation and fan-out live at the edge.
Store as a message bus. Every mailbox message is a stored version. The skill says documents are for durable coordination; chatter belongs elsewhere.
Fan-out cost. Long-lived watches become the norm for every MCP session, library, and SDK app. The QUIC broker absorbs them so each world sees one watch per broker replica; measure broker memory per watcher before enabling by default.
Silent misses. A dropped event looks like quiet. Cursors make gaps detectable; tests assert contiguous sequences; heartbeats carry the cursor.
UDP reachability. Direct and QUIC broker watchers inherit QUIC egress limits; browsers and blocked networks fall back to SSE through the HTTPS adapter.
Slow-peer detection after PR 1. A read deadline is now stream-level, so a peer that vanished without a close frame costs up to three request timeouts before quic-go's idle timeout (30 s) marks the connection dead and the pool evicts it; before, the first timeout evicted. Acceptable for reads; revisit if the latency harness shows it.
Local watch caps are small. DEMARKUS_MAX_STREAMS defaults to 10, so a soul allows 5 watches per connection; enough for one MCP session and a CLI, not for a fleet of harness monitors on one connection. Raise the stream limit or share one watch per process.

Open questions

Should a subscription accept LOOKUP filter predicates (tag=inbox) so the server filters instead of the client?
Should events carry changed section anchors so a live view refetches one section?
Should the file store persist its sequence so a local restart does not force a cache-wide resync?

Lean into QUIC: the transport review, the QUIC broker and its fan-out design, broker convergence, and the pure-QUIC profile.
demarkus-sdk: the client SDK where Watch lives.
ADR 0025: WATCH is the eighth verb, hints only, protocol freeze after WATCH and ALPN mark/1.