# Realtime demarkus: WATCH and live views

An active plan to make demarkus realtime: writes reach watchers, caches, and open views within a latency budget instead of on the next poll. Audience: demarkus maintainers implementing it. Drafted 2026-09-15 as a WATCH-only future plan; widened to realtime levels 1 and 2 on 2026-09-25, with fan-out moved to a QUIC broker the same day. Level 1 (PRs 1 to 6) implemented 2026-09-27 on branch `feat/live-track-front`, with ADR 0025 and ADR 0026, under review as PR 516; the smoke on real servers, a two-replica knowledge pair included, passes.

## Goal and scope

Three levels of realtime were weighed on 2026-09-25:

| Level | What users see | In scope |
|---|---|---|
| 1. Live changes | A write reaches every watcher, cache, and open view within budget | Yes |
| 2. Live views | LOOKUP results, graph neighborhoods, and library panes update themselves | Yes |
| 3. Live collaboration | Concurrent editing of one document, presence, cursors | No. Session state that belongs in a separate room service checkpointing through PUBLISH, never in the store. The event format below leaves room for it. |

The store stays a bookshelf: it emits change hints and nothing more. Everything that interprets or multiplies events (fan-out, caching, live queries, delivery to browsers and agents) lives at the edge: the QUIC broker, shared client packages, the HTTPS adapter, and the library.

## Why QUIC matters here

The [transport review](/plans/quic-first.md) found that QUIC's original reasons mostly hold over TLS on TCP too or are unused, and that the transport is rarely the bottleneck for request and response traffic. Realtime is where QUIC pays: one stream per subscription, many on one connection without head-of-line blocking, cancel by stream reset, migration keeping a laptop's subscriptions alive, client-opened streams so subscribers need no public address, and relay fan-out. If this plan stalls, the fallback is a TLS on TCP binding for direct clients.

The WebSub-style `SUBSCRIBE` with a callback URL in `docs/DESIGN.md` stays removed ([backlog](/roadmap/backlog.md)): agents are not servers. The SPEC section written in PR 2 replaces the Subscriptions entry in SPEC §13.

## Latency budget

Write committed to event delivered, p99. Correctness tests assert delivery with generous bounds; the budget itself is measured by a harness on the `benchmarks` branch, results recorded in the soul.

| Path | Budget |
|---|---|
| Local server to direct watcher | 100 ms |
| Knowledge-server replica A write to watcher on replica B | 500 ms |
| Any write to a watcher on the QUIC broker | 500 ms |
| Any write to a browser | 1 s |

A budget that cannot be met is a design finding recorded in the plan, not a reason to loosen the test silently. Finding 2026-09-27, closed the same day: without commit hints, replica A to a watcher on B was bounded by the 5 s backstop poll; the hint push in PR 6 brings it to one hint round trip plus one head read (the smoke sees it within a second on one machine). Unmeasured against the budget until the harness exists.

## Gates, resolved

| Gate | Resolution |
|---|---|
| ADR superseding "Verb Set: Complete" | PR 2 wrote [`docs/adr/0025-watch-verb.md`](/adr/0025-watch-verb.md) (proposed 2026-09-27), mirrored to `/adr/`, stating the realtime levels, that level 3 stays out of the store, and that protocol 1.0 freezes after WATCH and ALPN `mark/1` |
| Protocol version on the wire | Waived for WATCH. A server without it answers `bad-request` (`protocol/request.go:102`, `handler.go:148`), never a hang, so WATCH is additive. Pinned by a handler test in PR 2. Fortification 1 stays a separate 1.0 item. |
| A consumer where polling hurts | Agent mailboxes, cache and graph freshness, library live views |
| Connection vs stream error classification | PR 1, done 2026-09-27, before any watch code |
| MCP feature freeze | Not triggered: no new MCP tool. Agents get updates through MCP resource notifications, an existing MCP mechanism. |

## Consumers

1. **Agent mailbox.** Agents publish to `/agents/<id>/inbox/<ulid>.md`; a harness runs `demarkus watch` in the background and wakes the agent on a line. Claim by expected version, reply by APPEND, close by ARCHIVE, discover by LOOKUP.
2. **Cache and graph freshness.** `demarkus-mcp`, the brokers, and the library invalidate cached reads and graph nodes by path; `resync` clears. Complements [ADR 0013](/adr/0013-graph-source-freshness.md).
3. **Library live views.** The open document, LOOKUP panes, and graph neighborhoods update in place.

## Wire shape

Specified in SPEC §6.8 (PR 2). Request: `WATCH /prefix/` (or an exact document path), request metadata `since: <cursor>` optional, the usual token. The path need not exist: a watch on a mailbox directory that is not there yet delivers its creation.

Response: a stream of frontmatter blocks, no bodies.

- First block: `status: ok`, `cursor` (the position the stream continues from: the head for a new watch, the client's `since` for a resumed one). A request refused before subscription is an ordinary response in place of it; the client treats any first block whose status is not `ok` as terminal.
- Event block: `cursor`, `path`, `version`, `hash`, `op` (`publish`, `append`, `archive`), `agent` when the write carried one. Events are hints; clients fetch by hash for content. A block without `status` is an event.
- Heartbeat block: `status: ok`, `cursor`, every 20s while idle, so the client detects a stalled stream (three missed intervals) and keeps its cursor fresh. Fixed in the SPEC as RECOMMENDED; advertising it in the first block was dropped as an open question on 2026-09-27.
- Terminal blocks end the stream: `resync` (cursor unknown or too old), `unauthorized` or `not-permitted` (token revoked, expired, or no longer covering the path), `closing` (server drain).
- One block is at most 8192 bytes, fences included; block values follow the metadata grammar (no line breaks), which is what lets the reader work line by line.

Clients ignore unknown keys and unknown `op` values in event blocks. That rule lets level 3 add ops later without a new verb, and lets the QUIC broker add inline bodies and coalescing without changing the world.

Cursor: opaque `<epoch>:<seq>`, epoch 1 to 64 characters of `[A-Za-z0-9._-]`, seq a canonical decimal. At least once, ordered within an epoch. Clients never infer a gap from cursor values (read-authorised events are omitted silently); only `resync` signals one. On `resync` the client rebuilds from LIST and VERSIONS and resumes from the returned cursor.

`ParseResponse` reads one response per stream; WATCH needs an incremental block reader (ADR 0014: the module is wire and stored format). PR 2 added it to `protocol/`: `Cursor` and `ParseCursor`, `WatchBlock` with `WriteTo`, `WatchReader.Next`, `WatchEvent` with `Block` and `WatchBlock.Event`, `WatchControl`, `IsKnownOp`, `MaxWatchBlockLength`, `WatchHeartbeatInterval`, statuses `resync` and `closing`. The client side is copied into the [SDK](/plans/demarkus-sdk.md).

## Replica push

The knowledge server runs replicas over one bucket. Built 2026-09-27 as designed here, with one change from the first draft: the hint listener has its own port rather than sharing the Mark port under a second ALPN, because a NetworkPolicy can only keep a port inside the replica set, not an ALPN.

- **Receipts name the change.** The head object's receipts (`bucketstore/model.go`) carry `path`, `op` and `agent` beside the operation ID and sequence. That is what makes a peer's commit reportable: a replica that installs a new snapshot, whether from its own commit, a read's refresh, a hint or the poll, publishes every receipt the hub has not seen, in sequence order, with the version and hash from the snapshot.
- **Durable epoch.** The hub's epoch is the world ID and each event's sequence is the head sequence of its commit (`Hub.PublishAt`), so cursors are the same on every replica and across a restart. A replica that opens replays the receipt window, so a watcher that moves to it resumes from its cursor.
- **The window bounds the guess.** Receipts cover the last 16 commits. A replica that installs a snapshot more than 16 commits ahead of its hub (`Hub.Skip`) has no names for the gap and tells its watchers to resync from where the window starts; with hints arriving per commit this needs a burst of over 16 commits between one replica's commit and the other's refresh.
- **Commit hint.** After a commit, the store's `Committed` hook hands the world ID and sequence to `peerhint.Sender`, which coalesces per world and sends `head <world id> <sequence>` on one QUIC stream per hint to every peer (ALPN `mark-peer`). Peers come from the headless `<release>-peers` Service (this host's own addresses left out) or a fixed `peers.addresses` list. A peer proves itself by presenting exactly this replica's certificate: replicas share the TLS Secret, so no CA is involved.
- **On a hint,** `worldManager.Hint` finds the world by ID and kicks its poll unless the replica already serves that sequence; the poll refreshes the snapshot, which reports the receipts. Hints may arrive out of order and are only triggers.
- **Backstop poll** every 5 s per world while it has subscribers, for lost hints and unreachable peers.

GCS Pub/Sub notifications were rejected: GCP-only, added infrastructure and IAM, and no latency guarantee.

## Delivery paths

Worlds send hints to few subscribers; the QUIC broker multiplies them. Each client type gets one path:

| Client | Path |
|---|---|
| Local tools on a soul | Direct WATCH to the local server |
| SDK apps, CLI, `demarkus-mcp`, the library's Go server | [QUIC broker](/plans/quic-first.md) (workstream 9): one upstream watch per world per replica, filtered and fanned out through a shared ring, with hash-addressed content cache, opt-in inline small bodies, and opt-in coalescing |
| Browsers | Library server pushes htmx fragments; later WebTransport from the QUIC broker, SSE through the HTTPS adapter as fallback |
| Remote MCP hosts over Streamable HTTP | HTTPS adapter exposes documents as MCP resources and sends `notifications/resources/updated`, fed by a watch on the QUIC broker. Gated on checking which MCP hosts act on resource notifications. |

## Live views (level 2)

A shared package `client/live` (one implementation, per the shared-client guideline) consumes a watch and maintains derived state:

- **Cache invalidation:** evict by path from the response cache; clear on `resync`.
- **Live LOOKUP:** run the query, watch its scope, debounce events 50 ms, re-run, emit the row diff.
- **Live graph:** mark nodes stale by path, re-expand only affected neighborhoods.

`demarkus-mcp`, the brokers, and the library call `client/live`; none reimplements it.

## Implementation order

One PR per row. A PR is done when its proof test exists and `bash pre-commit.sh` passes. The [SDK](/plans/demarkus-sdk.md) extraction lands before PR 5, so `Watch` is written once, in the SDK. PR 1 landed in `client/fetch` because extraction had not started; it moves with the file. PRs 1 to 6 shipped together with ADR 0026 on one branch at Fritz's request.

| PR | Scope | Proof |
|---|---|---|
| 1 | **Client error classification.** Done 2026-09-27 (`feat/live-track-front`). `isTransientError` split into `isConnectionError` (dial failure, close by either side, idle or handshake timeout, stateless reset, a deadline before the stream was open), which evicts, and `isStreamError` (peer reset, a deadline after the request went out), which retries on a new stream and keeps the connection. A timed out `OpenStreamSync` stays connection-level, the wedge guard from the classify test. After any failure a connection whose context is done is evicted too, so a dead peer is never kept because its failure looked stream-level. | A FETCH whose stream is reset leaves the pooled connection; the next request on it succeeds. A connection close still evicts. Both proven end to end against an in-process QUIC server that counts accepted connections; a table test pins the classification. |
| 2 | **Protocol.** Done 2026-09-27 (`feat/live-track-front`). ADR 0025, SPEC §6.8 plus §5.2, §7, §8, §11.3, §11.8, §13 and §14, `VerbWatch`, statuses `resync` and `closing`, incremental block codec, cursor type, unknown-key rule. Verb table in `/architecture/protocol.md` updated. `rate-limited` added to SPEC §7, since the server already emits it and WATCH limits answer with it. | Codec round trip and three fuzz targets; oversize block rejected on read and on write; unknown keys and ops kept and skipped; a handler test shows a server without WATCH serving answers `bad-request`. |
| 3 | **Change hub** `server/internal/changefeed`. Done 2026-09-27. A ring of events where the sequence is the ring position, a broadcast channel per publish, and per-subscriber read positions: no per-subscriber queue, the ring is the queue, so the write path never blocks and a lapped reader gets `resync`. `Publish` assigns the next sequence (file store); `PublishAt` takes the caller's (knowledge store, head sequence), leaving stale slots readers step over; `Skip` moves the head over a gap nobody can name. `Subscribe(scope, since)` resumes after a cursor when the epoch matches and nothing after it was overwritten or skipped. `Subscription.Cursor` is the last consumed event, matching or not. Each store publishes under its own commit lock so hints follow commit order. | Contiguous sequences under 8 concurrent publishers; a slow subscriber gets `resync` while a fast one keeps receiving and publishing never waited; resume at every ring boundary; caller sequences with gaps and a skip; scope filter; Close wakes waiters. |
| 4 | **Single-server serving.** Done 2026-09-27. `handler.ReadRequest` and `handler.Serve` split `HandleStream` so the runtime sees the verb; `serveWatch` subscribes, acknowledges, then pumps events (each rechecked against the current token store, omitted when not readable), heartbeats due a fixed time after the last block written (which recheck the scope and end the watch with `unauthorized` or `not-permitted` after a reload), and the terminal block. A block the codec refuses is skipped, never a reason to end the stream. `worldruntime` admits a watch against per-world and per-connection caps (half the stream limit, from `Config.MaxStreams`), hands back the request concurrency slot before serving it, exempts it from the request deadline while bounding each block write, and `Drain` closes the hub so a shutdown ends watches with `closing` before the listener drains. | Watch then publish delivers, including on a prefix that does not exist yet; fetch by the event's hash returns the body; a token scoped to `/a/**` watching `/` sees `/a/` and public events and no `/b/`; a revoked token ends the watch within one heartbeat, even under a flood of events it may not read; resume replays after the cursor, another epoch gets `resync` in place of the ack; a lapped watch gets `resync`; an unencodable event is skipped; Drain and Close send `closing`; watches at the world or connection cap get `rate-limited` while a FETCH still passes; a subscribed watch holds no concurrency slot. |
| 5 | **Client and CLI.** Done 2026-09-27. `fetch.Client.Watch` returns a `Watch` with `Next`, `Cursor`, `Err`, `Close`: one goroutine reads the stream into a bounded queue; a full queue cancels the stream (freeing the connection's window) and reopens from the stream cursor once there is room, losing nothing; `closing`, a cut stream, a stall of three heartbeats or a lost connection reopen with backoff, and a `rate-limited` or `server-error` refusal on reopen retries rather than ending the watch; `resync` is a `Notice`; a final refusal is a `StatusError`. `Cursor()` is the last notice the caller consumed, so a persisted cursor never skips what was still queued. `demarkus watch [-since] [-auth] mark://host/prefix/` prints `cursor op path version hash agent` per line, `resync` as an op. | Against an in-process feed: delivery and cursor tracking, reopen after `closing` with `since`, resync surfaced and continued, refusal and `unauthorized` as errors, a rate-limited reopen retried, redial after a dropped connection, a consumer 3× the queue behind loses nothing, Close returns with a full queue and a resync on reopen, a restart from the consumed cursor sees what was queued. Smoke against the real server: publish and append lines with agent and hash; restart of the file server gives one `resync` line and the watch continues; Ctrl-C exits 0. |
| 6 | **Replica push.** Done 2026-09-27 (see Replica push): receipts naming the change, the durable epoch with `PublishAt` and `Skip`, `server/internal/peerhint` (sender, endpoint, certificate-pinned client TLS, Service and static resolution), the `peers` section of the knowledge config, the world manager's hint hook and kick, the chart's `server.peerPort`, headless `-peers` Service and NetworkPolicy rules, and the backstop poll. | Two stores over one bucket report a commit under the same sequence with the exact op and agent on both; a read's refresh reports without a poll; a replica opened later replays the window and a cursor from the other replica resumes on it; a gap beyond the window resyncs from the covered sequence; hints reach peers and coalesce, a peer with another certificate gets nothing, a peer hint makes the world poll. Smoke: a watch on replica B prints a publish made through A after A's hint, B restarts under the watch and the watch resumes without `resync`. |
| 7 | **QUIC broker fan-out.** Delivered by [Lean into QUIC](/plans/quic-first.md) workstream 9, after its auth-model ADR. | That workstream's fan-out proof: zero allocations per event per additional watcher, p99 within budget at 1k, 10k, and 100k watchers, resume on any replica. |
| 8 | **`client/live`.** Cache invalidation, live LOOKUP, live graph; wired into `demarkus-mcp`, the brokers, and the library. Note 2026-09-27: `demarkus-mcp`'s response cache already revalidates every read with `if-none-match`, so invalidation buys round trips, not freshness; the broker caches are where a TTL wait exists. | A publish through one session is read fresh by another session's MCP without a TTL wait; a live LOOKUP gains a row when a matching document is tagged. |
| 9 | **Library live views** (library repo). The library's Go server watches through the QUIC broker and pushes fragments to browsers. | An edit in one browser updates another's open document and pane. |
| 10 | **MCP resource notifications.** HTTPS adapter maps documents to MCP resources, subscribes through the QUIC broker, sends updates. Starts with a check of host support. | A host that supports resource subscriptions receives an update after a publish; the support table is recorded. |
| 11 | **Mailbox skill.** Inbox layout, claim by expected version, ack by archive, `demarkus watch` under a harness monitor. No core change. | Two agents split a task list with no double claims. |

The latency harness starts on the `benchmarks` branch alongside PR 5 and grows with PRs 6, 7, and 9.

## Risks

- **Dumb-server drift.** The store now holds subscriptions and replica hints. Kept small: events are hints, no durable subscriber state, overload degrades to `resync`, interpretation and fan-out live at the edge.
- **Store as a message bus.** Every mailbox message is a stored version. The skill says documents are for durable coordination; chatter belongs elsewhere.
- **Fan-out cost.** Long-lived watches become the norm for every MCP session, library, and SDK app. The QUIC broker absorbs them so each world sees one watch per broker replica; measure broker memory per watcher before enabling by default.
- **Silent misses.** A dropped event looks like quiet. Cursors make gaps detectable; tests assert contiguous sequences; heartbeats carry the cursor.
- **Receipt window bursts.** More than 16 commits landing between one replica's commit and another's refresh cost that replica's watchers a resync. Hints make the refresh follow each commit, so this needs hints lost under a burst; widen `maximumReceipts` if it shows.
- **Unauthenticated hints.** A peer hint is accepted from anyone who reaches the peer port and presents the replicas' certificate; the port is kept inside the replica set by the NetworkPolicy, and a hint costs at most one head read per world, coalesced.
- **UDP reachability.** Direct and QUIC broker watchers inherit QUIC egress limits; browsers and blocked networks fall back to SSE through the HTTPS adapter.
- **Slow-peer detection after PR 1.** A read deadline is now stream-level, so a peer that vanished without a close frame costs up to three request timeouts before quic-go's idle timeout (30 s) marks the connection dead and the pool evicts it; before, the first timeout evicted. Acceptable for reads; revisit if the latency harness shows it.
- **Local watch caps are small.** `DEMARKUS_MAX_STREAMS` defaults to 10, so a soul allows 5 watches per connection; enough for one MCP session and a CLI, not for a fleet of harness monitors on one connection. Raise the stream limit or share one watch per process.

## Open questions

- Should a subscription accept LOOKUP filter predicates (`tag=inbox`) so the server filters instead of the client?
- Should events carry changed section anchors so a live view refetches one section?
- Should the file store persist its sequence so a local restart does not force a cache-wide `resync`? The knowledge store's answer (receipts as the durable sequence) does not transfer: the file store has no head object.

## Related plans

- [Lean into QUIC](/plans/quic-first.md): the transport review, the QUIC broker and its fan-out design, broker convergence, and the pure-QUIC profile.
- [demarkus-sdk](/plans/demarkus-sdk.md): the client SDK where `Watch` lives.
- [ADR 0025](/adr/0025-watch-verb.md): WATCH is the eighth verb, hints only, protocol freeze after WATCH and ALPN `mark/1`.

## Addendum 2026-09-28: file store change journal

Resolves the third open question above (persist the file store sequence): yes, as the first PR after 516 merges, tracked as [File store change journal](/roadmap/realtime.md#file-store-change-journal) on the roadmap. A journal `<root>/.changes` in the WATCH block codec holds the epoch and the last ring's worth of events; open replays it with `PublishAt`, commit appends and fsyncs under the existing write lock, and a `HashIndexSize()` mismatch at open (the tree changed out of band while stopped) starts a new epoch. The knowledge server is unaffected; its bucket head already is the durable sequence. Design and the performance reasoning: [WATCH and the change hub](/architecture/realtime.md).