# Plan: The engine, not the wire

Audience: demarkus maintainers. Repositions demarkus as a memory and knowledge engine for agents, demotes QUIC from "the protocol" to one binding of the store contract, settles the identity door as MCP over HTTPS, says what becomes of the soul server, freezes transport work that has no shipping consumer, and names the fortification work the engine needs. Drafted 2026-10-08 in an idea session with Fritz; proposed, no code. Revises [On the Protocol as the Permanent Layer](/thoughts.md#on-the-protocol-as-the-permanent-layer) and extends [Fortify the durable core](/plans/durable-core.md).

## Premise

The word "protocol" has been covering three layers that now pull in different directions.

| Layer | What it is | Who uses it | Verdict |
|---|---|---|---|
| Product | memory for agents (soul), shared knowledge for orgs, promotion edge, write-time gates, recall | every user, through MCP plugins | this is what the project became |
| Contract | eight verbs, frontmatter metadata, immutable versions with a hash chain, LOOKUP catalog, WATCH cursors, grants | every binding | load-bearing; it is the memory model |
| Transport | QUIC on 6309, bearer listener on UDP 443, SNI routing, planned h3, WebTransport, 0-RTT, HTTPS adapter | server to server, replicas, CLI, souls | thin consumers, high tax |

Since [ADR 0032](/adr/0032-broker-in-the-knowledge-server-process.md) the MCP gateway dispatches to local worlds in process with no stream and no bytes. Every human and every LLM host reaches the knowledge system through MCP. The wire is left with machine edges: cross-cluster reads through the pool, replica hints, the CLI, souls on `demarkus-server`, and the bearer listener of [ADR 0033](/adr/0033-identity-bearers-on-the-mark-front-door.md), built for CLI, SDK and live views that do not ship against it yet.

What the engine offers that a plain MCP memory server does not is the contract, not the socket: versioned immutable markdown with declared metadata, two tiers with a promotion edge and gates, federation across worlds and servers, a catalog that never reads bodies, and change hints for sync without a search engine.

## Two doors, by credential kind

Settled 2026-10-08. One credential per protocol, not one per port.

| Door | Protocol | Credential | Who | Port |
|---|---|---|---|---|
| Machines | raw `mark://` over QUIC | capability token | server to server, replicas, souls, operators, the plugin binary | 6309, inside the cluster, on loopback, or over a private network |
| People and agents | MCP over HTTPS | OAuth identity (Google login, broker token) | every plugin host, browsers, SDK apps that want identity | 443 at the load balancer |

Consequences:

- HTTP/3 is a load balancer setting, not server work. The GKE load balancer negotiates h3 with clients and speaks h1 or h2 to the pod. MCP on UDP 443 costs no code.
- WATCH reaches browsers and hosts through MCP's streamable HTTP as server-sent events. Hints are small; WebTransport is not needed. This is the roadmap entry [Reaching MCP clients](/roadmap/realtime.md#reaching-mcp-clients).
- Identity on the raw `mark` wire is unnecessary. The bearer listener of ADR 0033 is removed from the branch before merge, not shipped dark: dark code with no path to live is debt. ADR 0033 moves to superseded by ADR 0037.
- The CLI as yourself: if ever needed, the CLI speaks MCP to the gateway with the device flow, like every other identity client. Not scheduled; the CLI is an operator tool with capability tokens. A broker-issued short-lived capability token (the old [Remote token issuance](/plans/remote-token-issuance.md) idea) is the fallback if MCP in the CLI proves too odd; it costs re-adding a mint path ADR 0032 deleted.
- Owning the wire matters inside the cluster. Outside, HTTP wins, and that is fine.

## Souls

The soul server (`demarkus-server`: file store, raw `mark` on 6309, capability tokens, no broker, no bucket) is the engine in its simplest form. For a soul the MCP door is client side: the plugin binary is the MCP server and dials raw `mark` behind it, so the engine only ever sees a machine with a token. That fits the two doors. One shape does not:

| Soul shape | Who dials the engine | Door | Verdict |
|---|---|---|---|
| Local soul | plugin binary on the same machine, `localhost:6309` | machine door on loopback | stays as is |
| Hosted soul (`soul.demarkus.io`) | plugin binary over the internet with a token file | machine door across the internet | the exception; moves |
| Library and demo worlds | library binary | machine door | stays as is |

Decisions:

- Hosted souls become memory-broker tenants: a tenant world with Google login on 443. That is what the memory broker exists for. `soul.demarkus.io` stays a personal instance or retires; `demarkus-server` never faces the public internet as a product.
- The local soul stays `demarkus-server` today. Nothing changes for the plugin.
- Long term the two binaries merge: `demarkus-knowledge` gains a file-store backend (the self-host without GCS item in workstream 4) and a `local` profile (file store, loopback, no OAuth, capability token or none). `demarkus-server` becomes that profile. One codepath for LOOKUP, WATCH fan-out, retention and the memtest suite. Not urgent; the two-binary split is a blast radius choice recorded 2026-08-29 and still defensible until the file backend exists.
- Constraint for the merge: a soul on a laptop outlives the hosted product. The local profile runs with no broker, no bucket, no account and no network. Goes in ADR 0037.

Plugin compatibility is the gate for the merge. The plugin contract stays byte for byte:

- The MCP tool names and schemas in the plugin binary do not change.
- `mark://localhost:6309` stays the local soul address; the default port stays 6309; ALPN stays `mark`.
- Token discovery stays `~/.mark/tokens.d`; an existing token file keeps working.
- The managed-binary launcher keeps launching a binary named `demarkus-server`; a rename ships as a symlink or wrapper first and the launcher learns the new name one release later.
- The soul's files on disk are read by the merged binary without migration; the file store format is the stored format of [ADR 0014](/adr/0014-protocol-module-is-wire-and-stored-format.md).
- Proof: the plugin's eval suite and `/soul-status` pass against the old binary and the merged binary with the same config and the same soul directory.

## Decision rules

- The durable-core rule stays: would this still matter if every product on top were thrown away? It applies to the contract.
- New rule for transport: no transport work without a named consumer that ships in the same quarter. "Live views will want it" is not a consumer.
- The existing revisit clause in [Verb set and wire format](/architecture/protocol.md) stands: if the live layer stalls or QUIC shows no gain on most hops, the fallback is a TLS on TCP binding, by ADR.

## Workstreams

### 1. Reposition in the soul and the repo

- ADR 0037: the contract is the protocol; QUIC, MCP and the in-process exchange are bindings of it; two doors by credential kind; the local profile constraint. Supersedes the "Transport is QUIC by design" framing and ADR 0033; keeps [ADR 0014](/adr/0014-protocol-module-is-wire-and-stored-format.md) as the contract's home.
- Rewrite the core principle in [Architecture](/architecture.md): the librarian stays, the bookshelf is an engine with three bindings and two doors.
- Rewrite the transport section of [Verb set and wire format](/architecture/protocol.md) as "Bindings": in-process, QUIC, MCP, with what each owns.
- README and the pages branch lead with memory and knowledge, not `mark://`. Positioning line from 2026-08-29 holds: storage engine for agent memory, brokers are the access tier, MCP is the driver.
- Proof: a new reader of the README can say what demarkus is without the word QUIC.

### 2. Freeze transport

- [Lean into QUIC](/plans/quic-first.md): workstreams 2 (UDP 443 for raw mark), 7a (h3 on the broker) and 7b (WebTransport) are closed by the two-doors decision; 3 to 6, 8, 10 and 11 move to parked; 1 (measure) stays only if the baseline is cheap. Workstream 9 is done by ADR 0032.
- Bearer listener (ADR 0033): removed from `feat/broker-in-process` before merge. The SNI routing and verifier reuse it added stay where the gateways already use them.
- [demarkus-sdk](/plans/demarkus-sdk.md): keep, narrowed to the license fix for demarkus-library and a Go driver for capability-token clients. Identity clients use MCP. Other languages stay unscheduled.
- [Realtime demarkus](/plans/watch-agent-messaging.md): level 1 and the fan-out are shipped; the next level is WATCH hints over MCP streamable HTTP, gated by the live-views decision in workstream 5.
- Protocol freeze after WATCH and ALPN `mark/1` (declared 2026-09-27) stands. No ninth verb.
- Proof: the active plans hub carries no transport plan without a consumer line.

### 3. Fortify the memory tier (soul)

- Recall quality: measure before improving. Reuse the leaderboard adapter shape as a local harness: a fixed set of recall questions against a production-shaped soul, hit rate on `mark_lookup` top 3, tokens spent to a correct fetch. Lives on `benchmarks`; results in `/research/`.
- Promotion edge: the `promote` cascade's dedup and refresh legs get the same failure-line treatment as the skills convention demands; measure promoted versus staged ratio.
- Retention and purge: [Version retention](/plans/version-retention.md) and [Purge](/plans/purge.md) are engine work and stay first class.
- Plugin parity: Claude Code, Cursor, OpenCode, Cowork expose the same recall and record behaviour; one prompt source, one gate set.
- Hosted souls onto the memory broker: a migration path from a `soul.demarkus.io` token to a tenant world, documents and versions intact.
- Proof: a recall harness number that moves release to release; one hosted soul migrated with its history.

### 4. Fortify the knowledge tier

- Store correctness under load: [Bucket store commit log](/plans/bucket-commit-log.md), its [GCS gate](/plans/bucket-commit-log-gcs-gate.md) and [production rollout](/plans/bucket-commit-log-rollout.md), [the compactor off the writers](/plans/compactor-off-writers.md), [the memory wall](/plans/memory-wall.md). These are the engine; they keep their order.
- Provenance and supersession: durable-core fortifications 2, 3 and 4 (client-verifiable history, `superseded-by` on read, `valid-from` and `valid-until`) are the properties that make the curated layer worth more than raw sources. Pull them forward.
- Ingestion: [Knowledge ingestion](/plans/knowledge-ingestion.md) through one curation gate, with dedup across worlds.
- Self-host without GCS: a file-store backend for `demarkus-knowledge` with the in-process broker. This is also the first step of the soul-server merge in the Souls section.
- Proof: durable-core items 2 to 4 shipped; one self-hosted knowledge system on a single VPS with SSO.

### 5. Decide live views

Live views (reading room, realtime library) are the one product feature that needs change hints in a browser. With the two-doors decision they ride MCP streamable HTTP, so the question is priority, not transport. Decide: is it a 2026 goal or not? If yes, WATCH over MCP is the next realtime level. If no, the library reads through MCP without hints.

## Sequencing

1. Workstream 1 (reposition) and the live-views decision first; both are writing and one conversation.
2. Workstream 2 (freeze) follows the ADR: remove the bearer listener from the branch, status lines on the parked plans.
3. Workstreams 3 and 4 run as the ordinary roadmap, with the recall harness as the first new item.
4. The soul-server merge waits for the file-store backend and ships behind the plugin compatibility gate.

## Non-goals

- Ripping QUIC out or adding an HTTP verb mapping. The rejected bindings stay rejected; MCP is the HTTP-shaped door and a TLS on TCP binding for raw mark is the recorded fallback only.
- A ninth verb or a wire change.
- Dropping the SDK; the library license conflict is real.
- Changing the plugin's tool surface, address, token discovery or on-disk soul format for the merge.

## Risks

- Live views lose momentum if the decision in workstream 5 slips; name a date.
- Removing the bearer listener discards a built step of ADR 0032's branch. Accepted: a verifier call per request on an internet-facing UDP port with no dialer is worse than deleted code in history.
- The CLI has no identity path. Accepted while it is an operator tool; the MCP device flow is the answer if that changes.
- The merged binary carries OAuth and bucket dependencies into a laptop process. Mitigation: build tags or a thin `local` main over the same packages; the managed binary size is the metric to watch.
- "Engine" positioning can read as a database pitch. Keep the agent-memory framing in front; the engine is the how.

## Open questions

- Is demarkus-library a product or a demo? It decides the SDK and live-views priority.
- Public word: engine, store, or memory system? The 1.0 concept model settled soul, memory and knowledge; the thing underneath still has no one name.
- Hosted soul migration: does a tenant world accept a bulk import with versions intact, or does the move flatten history to one version per document?