Plan: Broker MCP Gateway (Knowledge-System Layer)

Plan rewritten 2026-05-20. v1 described a REST HTTP API. After alignment with Fritz, the architectural target shifted: the gateway is an MCP server exposed at a single /mcp endpoint, authenticated against the company SSO (id_token bearer), with world access-tokens cached broker-side per session. Agent-as-client is the only first-pass consumer. A browser/REST surface is a later, additive plan if a real consumer asks.

The enterprise-deployment piece. Adds a remote MCP server to the broker that exposes the demarkus tool surface (mark_fetch, mark_publish, etc.) to plugin-style agents over the company's existing SSO. Worlds stay cluster-internal; the broker becomes the single MCP endpoint the plugin connects to; corporate networks that block QUIC over UDP reach the universe over standard HTTPS. The plugin's local vault holds only the id_token (the company SSO bearer); world access-tokens are minted lazily broker-side per session, scoped to the authenticated user, never persisted to disk.

Architectural framing in /thoughts.md § "On the Protocol as the Permanent Layer": demarkus core is the permanent layer; this gateway is a knowledge-system overlay that the protocol doesn't know exists. See also feedback_core_vs_knowledge_layer.md in agent auto-memory.

Why this is the priority

Stated 2026-05-20: "this has to be the next task we do, it is the missing piece that the company needs." Two enterprise blockers dissolve together:

• "20 dev teams × N worlds plugin-config grind." Without the gateway, every developer at a 20-team org installs N local MCP servers per workstation, one per world they need. With the gateway, every developer adds ONE MCP server entry (the broker URL) and the broker exposes the universe's worlds as a single tool surface.
• Corporate-network UDP blocking. QUIC over UDP gets blocked by enterprise firewalls, transparent proxies, and DPI appliances. HTTPS over 443 works everywhere. The gateway speaks HTTPS publicly and translates to QUIC internally.

Universe-onboarding's PR6/7/8 are reshaped by this work: PR6 (originally tools/demarkus-join shelling out to a local MCP binary) is no longer necessary in its planned form — the plugin just configures the broker URL as a remote MCP server. PR6's scope shrinks to a config-writer or disappears entirely.

Non-Negotiables

• No protocol changes. The demarkus message format (FETCH/PUBLISH/APPEND/VERSIONS/LIST/ARCHIVE/MERGE/etc.), content addressing, versioning, and capability-token mechanics are unchanged.
• No demarkus-server changes. Worlds keep their current QUIC + mark ALPN listener. They see a bearer token in the demarkus protocol header (the world access-token the broker is holding on the user's behalf) and serve it. They do not know the broker is the caller.
• Capabilities stay on worlds. The world is the trust boundary for what a token can do. The broker holds world access-tokens in trust per-authenticated-user; the world validates them on every request. The broker does NOT re-implement capability checks.
• Tool surface parity with the local MCP server. All 13 tools the existing client/cmd/demarkus-mcp exposes (mark_fetch, mark_list, mark_versions, mark_publish, mark_append, mark_archive, mark_discover, mark_resolve, mark_index, mark_backlinks, mark_graph, mark_graph_export, mark_graph_publish) are exposed by the broker MCP from day one. Anything else is a plugin-side change, not a gateway concern.
• Opt-in deployment. The MCP listener is off by default. Operators enable it explicitly; the existing direct-QUIC universe deployment stays the default.
• No MVPs, no shortcuts. OAuth correctness, per-session world-token cache lifecycle (lazy mint + eviction on id_token expiry + cap on token churn), structured error envelopes, content-hash propagation through the tool surface, conflict-aware merge — all from day one within each slice (not later).
• QUIC stays first-class for direct-world access. This gateway is an enterprise overlay, not a transport replacement. The CLI, Obsidian plugin, and in-cluster server-to-server traffic all keep direct-QUIC access. The broker's existing /auth, /me/install, /tokens surfaces all stay as they are.

Out of Scope (explicit)

• Browsable REST surface. Deferred until a real non-agent consumer asks. The plugin is the only first-pass consumer; the agent is always the client.
• tools/demarkus-join binary in its original PR6 shape. Replaced by "plugin configures broker MCP URL."
• mTLS broker → world. Confirmed deferred to a hardening pass. TLS terminates at the broker; broker→world QUIC traffic stays inside the cluster, bounded by NetworkPolicy + RBAC.
• Persistent world-token storage. Raw world tokens are NEVER persisted to disk by the broker. They live in process memory per session; broker restart drops them; next tool call re-mints. This matches the broker's existing posture (raw tokens are never recoverable from the issuances Secret).
• MCP resources and prompts features. Only the tools capability is exposed. Resources (file-style refs) and prompts (templates) are not part of the demarkus model.
• Long-lived subscriptions / server-initiated events. Polling is the model (/thoughts.md § "On Subscriptions and Polling").
• Multi-broker / multi-universe. Single broker per universe stays the model.
• Content negotiation beyond text/markdown. Same scope contract as the protocol (project_scope_markdown_only.md).

Architecture

┌──────────────┐  HTTPS (MCP)   ┌──────────────────────────────────┐  QUIC  ┌─────────┐
│ Claude Code  │ ─────────────► │  broker (/mcp endpoint)           │ ─────► │ world A │
│ plugin       │                │                                    │        │ (cluster│
│ (or any MCP  │ ◄───────────── │  JSON-RPC over Streamable HTTP    │ ◄───── │ internal)
│  client)     │   tool result  │                                    │        └─────────┘
└──────────────┘                │  Authorization: Bearer <id_token> │
                                │  - requireAuth (PR4 compositeV)   │        ┌─────────┐
                                │  - per-subject rate limit         │ ─────► │ world B │
                                │                                    │ ◄───── │         │
                                │  Per-session world-token cache:    │        └─────────┘
                                │  - key: (subject_hash, world_name) │
                                │  - mint lazily via Issuer.Mint-    │
                                │    Filtered on cache miss          │
                                │  - evict on id_token / token expiry│
                                │                                    │
                                │  13 tools (parity with local       │
                                │  demarkus-mcp): mark_fetch, ...    │
                                │                                    │
                                │  OAuth metadata:                   │
                                │  /.well-known/oauth-protected-     │
                                │    resource (RFC 9728)             │
                                │  /.well-known/oauth-authorization- │
                                │    server (RFC 8414)               │
                                └──────────────────────────────────┘

Layer responsibilities

Pre-Flight Tasks (one PR ahead of Slice 1)

Pre-Flight 0 — Hoist client/internal/fetch to public

• client/internal/fetch/fetch.go → client/fetch/fetch.go (or client/pkg/fetch/fetch.go). Mirror for client/internal/merge since MERGE support lands in MVP via the conflict-aware mark_publish tool.
• Update import paths in client/cmd/demarkus-mcp, client/cmd/demarkus, client/cmd/demarkus-tui (and any other consumers within the client module).
• Same content; just relocated. Surface unchanged.
• Why: Go's internal-package rule blocks any module outside client/ from importing client/internal/.... The broker is in tools/demarkus-broker (different module under monorepo replace-directives), so it can't import the existing client library while it's internal.
• This is a CLIENT-module API change, not a protocol-core change. Not gated by feedback_plugin_scope.md (broker isn't a plugin), but worth flagging explicitly so reviewers see the architectural shift.

Pre-Flight 1 — Verify mark3labs/mcp-go's Streamable HTTP support

• The existing client/cmd/demarkus-mcp uses mcp-go's stdio transport (mcpserver.ServeStdio()). Confirm mcp-go v0.44+ supports Streamable HTTP (the current MCP spec's preferred remote transport).
• If yes: lifecycle the new gateway around mcp-go's HTTP server constructor.
• If no: upgrade to a version that does, swap to a different MCP Go library, or implement Streamable HTTP transport in-house (worst case).
• Output: one-paragraph journal note on transport choice before Slice 1 begins.

Sub-Tasks (sequenced; one slice per PR)

Slice 1 — Foundation: listener + initialize + tools/list (no tool implementations)

• tools/demarkus-broker/internal/broker/mcp_gateway.go (new) — mcpGateway listener, JSON-RPC dispatcher, Streamable HTTP transport. Separate listener from the existing management API.
• tools/demarkus-broker/internal/broker/mcp_oauth.go (new) — OAuth metadata endpoints: /.well-known/oauth-protected-resource (RFC 9728), /.well-known/oauth-authorization-server (RFC 8414). Reuses broker's existing Discovery machinery.
• tools/demarkus-broker/internal/broker/mcp_auth.go (new) — gatewayAuth middleware. id_token bearer extraction, validation via existing compositeVerifier, subject claim extraction, per-subject rate limiting (shared budget with /tokens routes — same identity, one bucket).
• tools/demarkus-broker/internal/broker/config.go — ServerConfig.MCP substruct: Enabled, Addr, TLS{...}, SessionMaxIdle, WorldTokenTTL.
• tools/demarkus-broker/main.go — start the MCP listener when enabled, alongside the management API.
• tools/demarkus-broker/internal/broker/mcp_initialize.go (new) — handles MCP initialize JSON-RPC method. Advertises server capabilities (tools only).
• tools/demarkus-broker/internal/broker/mcp_tools_list.go (new) — handles tools/list. Returns the 13 tool definitions (names + JSON schemas) without implementations yet. Definitions mirror client/cmd/demarkus-mcp exactly.
• Tests: OAuth metadata round-trip, initialize handshake, tools/list returns the 13 expected tool names, unauth requests rejected with 401 + WWW-Authenticate header pointing at auth-server metadata URL, expired/revoked bearer rejected, per-subject rate limit triggers 429.

Slice 2 — Read tools + session cache foundation: mark_fetch, mark_list, mark_versions

• tools/demarkus-broker/internal/broker/mcp_session.go (new) — sessionCache keyed by subject_hash. Holds map[worldName]cachedWorldToken{raw, expiresAt}. Lazy initialization on first cache miss. LRU cap (default 10000 subjects) + idle eviction (SessionMaxIdle default 1h after last use).
• tools/demarkus-broker/internal/broker/world_pool.go (new) — worldPool for fetch.Client reuse per world.
• tools/demarkus-broker/internal/broker/mcp_tools_read.go (new) — handlers for mark_fetch, mark_list, mark_versions. URL parameter shape: mark://{worldName}/{path} (see Open Question 1). Resolves world by name; lazy-mints world access-token via Issuer.MintFiltered on cache miss; calls fetch.Client.Fetch/List/Versions with the token; maps the demarkus Result to an MCP tool response (matching local MCP server output format for parity).
• Tests: each tool happy path, world-not-found tool error, document-not-found maps to MCP error, expired-bearer 401 mid-call, cache hit reuses token, cache miss mints, two concurrent calls for the same (subject, world) coalesce to one mint (singleflight), session eviction on id_token expiry causes re-mint.

Slice 3 — Write tools: mark_publish, mark_append, mark_archive

• tools/demarkus-broker/internal/broker/mcp_tools_write.go (new) — handlers for the three write ops.
• mark_publish reuses the existing expected_version + on_conflict shape from the local MCP server. Conflict-aware merge candidate flow lands in Slice 6; for Slice 3, on-conflict defaults to "fail" with the conflict envelope, NOT the merge-candidate path.
• mark_append reuses the auto-resolved-version pattern (omit expected_version, broker calls VERSIONS internally) — matches local MCP server behavior.
• mark_archive deletes by archiving (per the demarkus protocol's actual DELETE shape; the protocol op is ARCHIVE, not DELETE).
• Tests: happy paths, version-mismatch conflict, missing-expected_version on PUBLISH 400, auto-resolve path for APPEND, world-side RBAC failure surfaces as MCP tool error.

Slice 4 — Federation read tools: mark_discover, mark_resolve, mark_backlinks, mark_graph

• tools/demarkus-broker/internal/broker/mcp_tools_federation.go (new) — handlers that delegate to whichever demarkus client library functions back the local MCP server's federation tools (mapped during a second-pass spike at start of Slice 4, deferred until Pre-Flight 0 hoist completes and the public API is visible).
• These tools are read-only and don't change the session-cache shape.
• Tests: happy paths + edge cases per tool.

Slice 5 — Federation write tools: mark_index, mark_graph_export, mark_graph_publish

• tools/demarkus-broker/internal/broker/mcp_tools_federation_write.go (new).
• Same shape as Slice 3 for writes (expected_version, conflict handling).
• Tests: happy paths + conflict cases.

Slice 6 — Conflict-aware merge in mark_publish

• Reuses client/internal/merge (hoisted to client/merge in Pre-Flight 0).
• Adds the on_conflict: "merge" branch to mark_publish, matching local MCP server behavior.
• Tests: clean merge, structural merge, conflict-markers returned.

Slice 7 — Chart, RBAC, docs

• deploy/helm/demarkus-broker/values.yaml — server.mcp.enabled, server.mcp.addr, server.mcp.tls (existingSecretRef recommended), server.mcp.sessionMaxIdle, server.mcp.worldTokenTTL, server.mcp.worldPool.
• Templates: deployment.yaml (new containerPort + TLS mount), service.yaml (gateway port exposed), ingress.yaml (route mcp host to new port), networkpolicy.yaml (allow ingress on mcp port).
• No new RBAC needed — broker SA already has perms for issuances Secret + world Secrets that lazy-mint touches.
• deploy/helm/demarkus-broker/README.md — new "MCP gateway" section. When to enable, TLS setup, the plugin-side claude mcp add invocation, OAuth flow, rate-limit behavior.
• tools/demarkus-broker/main.go package doc — bump Current scope to mention /mcp.
• New tools/demarkus-broker/MCP-API.md — operator/developer-facing spec for the MCP tool surface mirrored from client/cmd/demarkus-mcp. Lives in the broker package; not a protocol document.

Scope estimate

Seven slices (plus Pre-Flight) across ~2-3 working weeks. Larger than the original REST plan because MCP brings real protocol-handling work (initialize handshake, tools/list, JSON-RPC dispatch, OAuth metadata) that REST didn't, AND tool-parity with the local MCP server means 13 tools instead of 7 ops. Each slice still individually mergeable.

Open Questions To Resolve Before/During Implementation

1. URL shape inside MCP tool args. Lean: mark://{worldName}/{path} — world by NAME, broker resolves to internal address. Today's local MCP server takes mark://{host}:{port}/{path} (full network address). Switching to world-name routing means tool calls become broker-aware; an open-proxy SSRF surface is eliminated (broker only routes to configured worlds); URL strings get shorter. Backward-compat: accept full host:port form only if it matches a known world's PublicURL field. Confirm before Slice 2.
2. MCP transport: Streamable HTTP vs SSE. Lean: Streamable HTTP per current MCP spec (single endpoint, POST for requests, optional SSE for server-initiated messages). Locked in by Pre-Flight 1.
3. OAuth authorization-server identity. The broker is BOTH the MCP server (resource) AND the authorization server (the existing OIDC + device-flow surface). Lean: broker advertises itself as the authorization server via /.well-known/oauth-authorization-server. The actual IdP (Google, Okta, etc.) is one hop further — handled by existing broker OIDC machinery. Plugin doesn't talk to Google directly; talks to broker, which talks to Google. Matches PR3 device-flow architecture.
4. Session keying. Lean: subject claim hash from id_token. Multiple devices for the same user share one session (which is fine — same identity, same world access).
5. World-token TTL and minting cadence. Lean: cache world-tokens with their natural expiry from MintFiltered (configurable via worlds[].defaultToken.expiresAfter, default 24h). On id_token rotation (refresh-grant), session_cache survives because subject claim is stable across refreshes. World-token churn is bounded by world_count × user_count × (1 / world_token_ttl) — well under the issuances Secret's 5000-record cap for realistic deployments.
6. Federation tool implementations. Slice 4/5 reuse the client library functions backing the existing federation tools in client/cmd/demarkus-mcp. Second-pass spike at start of Slice 4.
7. MCP error envelope for partial mint failures. When the broker mints a world token on first call, partial-mint surfaces as ONE successful tool response + server log entry, NOT a partial-mint envelope. Reason: MCP is single-tool-call-per-request; the gateway only mints the world the current tool needs.
8. MCP session lifecycle on broker restart. Lean: in-memory only; broker restart drops all sessions; plugin re-authenticates via the existing OAuth refresh path (PR4 broker-signed refresh tokens). Persistent session storage deferred to Phase 7+ if customer demand surfaces.

Risks Specific To The MCP Gateway

• MCP protocol surface gains an internet-reachable port. Same risk shape as the original REST plan: hardened auth middleware (existing compositeVerifier), strict per-subject rate limit, structured MCP error envelope, TLS-only in production.
• World-token cache memory growth. A broker serving 10k subjects × 5 worlds × ~200 bytes per cached token entry = ~10MB. Bounded but not zero. Mitigation: LRU cap, idle eviction (default 1h after last tool call), per-process memory metric for ops dashboards.
• Mint storm on broker restart. Every active plugin reconnects after a restart and re-mints tokens for every world it touches. Same shape as PR4-flagged refresh-storm risk but slightly worse (no client-side world-token persistence to skip the mint). Mitigation: mutateSecret's optimistic-concurrency retry handles contention; sharded issuances Secret if a real customer hits the wall (existing Phase 7 path).
• Issuance bloat under high session churn. Plugin sessions that come and go faster than the world-token TTL leak issuance records into the Secret. Sweeper retires them on expiry. For high-throughput deployments, operators tune worlds[].defaultToken.expiresAfter shorter (e.g., 4h).
• Plugin holds N world identities through one broker connection. Single MCP-server-for-the-universe architecture means a buggy broker handler could expose one user's session to another's bearer. Mitigation: strict subject-claim keying on session cache; every tool call re-validates the bearer against the cache key; integration test that proves cross-subject isolation under concurrent load.
• mcp-go library maturity. Third-party Go MCP library; if it lacks Streamable HTTP, the choice in Pre-Flight 1 cascades. Mitigation: Pre-Flight 1 is fact-finding; the next decision (upgrade / swap library / implement in-house) lands as a one-paragraph journal entry before Slice 1.
• OAuth metadata vs OIDC metadata overlap. Broker already serves /.well-known/openid-configuration. RFC 8414 (OAuth) and OIDC discovery have largely-overlapping shapes; serving both means two endpoints that mostly mirror each other. Mitigation: one shared rendering helper, two route registrations.

Touch Points With Adjacent Work

• Universe-onboarding PR5 (shipped #141, 2026-05-20). No code coupling. /me/install stays as-is — it remains the right way to introspect "what worlds am I authorized for?" outside an MCP session. The MCP gateway and /me/install answer two related-but-distinct questions: MCP is the operational data plane; /me/install is the identity-introspection surface.
• Universe-onboarding PR6 (tools/demarkus-join): substantially reduced. With the MCP gateway live, the plugin just runs claude mcp add demarkus https://broker.example.com/mcp (OAuth flow handles the rest). The "join" command may shrink to a config helper or disappear entirely.
• Universe-onboarding PR7 (plugin slash commands): shape depends on PR6's final form. /soul-join may end up writing ~/.claude/mcp.json directly.
• /thoughts.md § "On the Protocol as the Permanent Layer": principle this plan is built on.
• Future browsable REST surface plan: additive. The MCP gateway is on /mcp; a REST surface would live on /v1/... paths on the same listener, sharing OAuth auth and session cache.
• Future mTLS broker → world plan: anchors here. The worldPool can grow mTLS in place without changing the tool surface.

Resume Steps (when starting work)

1. PR5 (#141) confirmed on main (6c1c354).
2. mark_fetch /plans/broker-https-gateway.md (this doc).
3. Re-confirm Open Question 1 (URL shape) if needed — the others have stated leans with documented trade-offs.
4. Execute Pre-Flight 0 (hoist) as its own small PR. Existing tests pass after the relocation.
5. Execute Pre-Flight 1 (mcp-go transport spike). Output: one-paragraph note in the journal.
6. Cut branch feat-tools-broker-mcp-gateway-foundation. Start at Slice 1.
7. Each slice is its own PR. go test -race + bash pre-commit.sh green before moving on.
8. Journal at session end. Update Implementation Status (below) as slices land.

Done When

• All seven slices merged (plus Pre-Flight 0).
• go test -race ./... green inside tools/demarkus-broker/ and client/.
• helm unittest . green for the broker chart.
• pre-commit.sh green.
• Manual end-to-end: claude mcp add demarkus https://broker.example.com/mcp → OAuth device flow completes → plugin sees all 13 demarkus tools → mark_fetch mark://team-a/foo.md returns the world's content; cached world-token reused on the next call; broker logs show only one mint per (subject, world).
• Operator-facing README documents the deployment.
• Journal entry covering any design decisions that landed differently from this plan.

Implementation Status

Not started. Plan v2 published 2026-05-20 after architectural pivot from REST → MCP. Awaiting:

• Confirmation on Open Question 1 (URL shape: mark://{worldName}/{path} vs full host:port form).
• Decision on whether tools/demarkus-join (universe-onboarding PR6) is canceled outright or kept as a thin config-writer.

Then Pre-Flight 0 begins.