Plan: Broker MCP Gateway (Knowledge-System Layer)

STATUS: COMPLETE (2026-05-23). All eight slices + Pre-Flight 0/1 merged. Plan archived. See /completed-plans.md § "Broker MCP Gateway" for the shipping summary; this document stays in place as the architectural reference + decision trail.

Plan rewritten 2026-05-20. v1 described a REST HTTP API. After alignment with Fritz, the architectural target shifted: the gateway is an MCP server exposed at a single /mcp endpoint, authenticated against the company SSO (id_token bearer), with world access-tokens cached broker-side per session. Agent-as-client is the only first-pass consumer. A browser/REST surface is a later, additive plan if a real consumer asks.

v3 (2026-05-20 same session): Open Question 1 resolved (mark://{worldName}/{path} with cluster-internal DNS resolution). PR6 (tools/demarkus-join binary) canceled outright; the join flow becomes the /knowledge-join slash command in plugins/claude-code/, distinct from a future /soul-join (personal direct-QUIC). Vocabulary pin: knowledge system = organizational brokered universe; soul = personal, direct-QUIC. Different commands, different deployment shapes, shared protocol underneath.

v4 (2026-05-20 same session, post-Slice-1): "Opt-in deployment" Non-Negotiable removed — every broker is a knowledge-system gateway; MCP is part of the binary, not a feature flag. MCPConfig.Addr defaults to :8081 so pre-gateway YAMLs upgrade silently. New § "Architectural framing (enterprise shape)" pins the HTTP-gateway-with-OIDC/session-state framing and the byte-for-byte proxy contract: the broker transports markdown unchanged from the world's stdio/QUIC surface — only the wire shape (HTTP/JSON-RPC) is new. Slice 1 shipped (foundation: listener + initialize + tools/list).

v5 (2026-05-21, pre-Slice-2): Session cache is email-keyed, not subject-hash-keyed. The broker's existing identity primitive (PR4 Issuer, /me/install, /tokens list/revoke, AllowConfig, audit log) is canonical verified email. Slice 2's sessionCache must use the same key so the broker has ONE identity dimension, not two. Concretely: the cache key is canonicalEmail (trimmed, lowercased) and MintFiltered is called with Claims{Email, EmailVerified, Groups} — same shape every other broker surface already uses. Unverified-email id_tokens are rejected at the gatewayAuth boundary (matches ErrEmailUnverified semantics in Issuer.Mint). Subject claim is still available from the id_token for log correlation if useful, but is NOT the cache key. Lean answer on Open Question 4 updated accordingly.

v6 (2026-05-22, post-Slice-7): Slices 2–7 all shipped. Only Slice 8 (/knowledge-join plugin slash command) remains. Plan vs code drift discovered + documented during Slice 7: WorldTokenTTL

• WorldPool{} fields the plan listed under MCPConfig were never implemented (Slice 2 went with natural expiry from MintFiltered per OQ#5's lean answer; worldPool lifecycle is internal to Server.MCPGateway(), not chart-configurable). Slice 7 dropped both knobs from values.yaml rather than ship chart fields the broker silently ignores. Ingress topology for the MCP gateway pinned to a SEPARATE hostname (ingress.mcp.host) — not path-routing on the management host — to avoid future .well-known/* collisions between OIDC discovery and OAuth resource metadata. The --with-mcp-smoke kind-harness stage ships three lightweight checks (RFC 9728 + RFC 8414 metadata fetch, POST /mcp 401 + WWW-Authenticate); the full id_token / initialize / tools/call dance is deferred to Slice 8 where /knowledge-join will exercise it naturally.

v7 (2026-05-23, plan complete): Slice 8 (/knowledge-join plugin slash command) shipped as PR #152 (bc0d5cc). All eight slices merged; plan archived. Slug heuristic landed as "first DNS label" (not "strip-broker.-and-.com → org") to match typical enterprise broker URL shapes. Test-only DEMARKUS_KNOWLEDGE_JOIN_ALLOW_HTTP=1 escape hatch lets the shell tests drive a local http:// mock without TLS round-tripping; documented in script comments as namespaced/long deliberately so no user accidentally enables it. CodeRabbit round-1 fixes: case-insensitive URL scheme match (RFC 3986), scheme normalization to lowercase in emitted url=/mcp-url=, bash language tag on doc fences, test rewrite to actually exercise upper→lower (the original test claimed to but didn't). CI workflow needed a permissions block for dorny/paths-filter@v3 after a Settings → Actions tightening — shipped as part of the same PR after Bad credentials from the API broke the PR's own checks.

The enterprise-deployment piece. Adds a remote MCP server to the broker that exposes the demarkus tool surface (mark_fetch, mark_publish, etc.) to plugin-style agents over the company's existing SSO. Worlds stay cluster-internal; the broker becomes the single MCP endpoint the plugin connects to; corporate networks that block QUIC over UDP reach the universe over standard HTTPS. The plugin's local vault holds only the id_token (the company SSO bearer); world access-tokens are minted lazily broker-side per session, scoped to the authenticated user, never persisted to disk.

Architectural framing in /thoughts.md § "On the Protocol as the Permanent Layer": demarkus core is the permanent layer; this gateway is a knowledge-system overlay that the protocol doesn't know exists. See also feedback_core_vs_knowledge_layer.md in agent auto-memory.

Why this is the priority

Stated 2026-05-20: "this has to be the next task we do, it is the missing piece that the company needs." Two enterprise blockers dissolve together:

• "20 dev teams × N worlds plugin-config grind." Without the gateway, every developer at a 20-team org installs N local MCP servers per workstation, one per world they need. With the gateway, every developer adds ONE MCP server entry (the broker URL) and the broker exposes the universe's worlds as a single tool surface.
• Corporate-network UDP blocking. QUIC over UDP gets blocked by enterprise firewalls, transparent proxies, and DPI appliances. HTTPS over 443 works everywhere. The gateway speaks HTTPS publicly and translates to QUIC internally.

Universe-onboarding's PR6/7/8 are reshaped by this work: PR6 (tools/demarkus-join binary) is canceled. The join flow folds into /knowledge-join in plugins/claude-code/. PR7 becomes "plugin slash command implementation" and PR8 is documentation.

Architectural framing (enterprise shape)

The broker is the HTTP gateway that fronts the knowledge system. From the enterprise side it slots into the same plumbing every other internal SaaS app uses; from the demarkus side it's a thin proxy that hands markdown back to the agent verbatim. Three responsibilities, in this order:

1. Termination at the org boundary. The gateway speaks HTTPS + JSON-RPC over Streamable HTTP at /mcp. Corporate proxies, Ingress, DPI appliances, TLS-terminating LBs — all the standard enterprise plumbing — pass it through unchanged. QUIC stays inside the cluster, where firewalls don't see it.

2. Identity + session state. The agent presents Authorization: Bearer <id_token> (the company SSO bearer). The gateway validates via the existing PR4 compositeVerifier, extracts the email and email_verified claims (rejecting unverified emails, matching Issuer.ErrEmailUnverified), canonicalizes the email (trim + lowercase) and keys a per-session in-memory map of world access-tokens off that canonical email. World tokens are minted lazily on first cache miss via Issuer.MintFiltered(ctx, Claims{Email, EmailVerified, Groups}, keep); never persisted; broker restart drops them and the next tool call re-mints. The same canonical email across multiple plugin installs or devices shares one session — familiar mental model from SAML/OIDC apps. Subject claim remains in the id_token and is available for log correlation but is NOT the cache key — the broker has exactly one identity dimension (email) across MCP, /me/install, /tokens, audit logs, and AllowConfig.

3. Fan-out to worlds over mark:// (QUIC). Tool URLs carry the worldName as host: mark://{worldName}/{path}. The gateway resolves the name to a cluster-internal Service DNS address (<name>.<namespace>.svc.cluster.local:6309 by default, overridable via worlds[].internalAddress) and dispatches via client/fetch.Client with the cached world token. The world server is unchanged — it sees a demarkus protocol request carrying a valid token and serves it. The world does not know the broker is the caller.

The proxy contract: byte-for-byte markdown

The gateway transports markdown unchanged. A mark_fetch call through the broker returns the same Result struct — same body bytes, same version, same modified, same etag, same content-hash, same metadata keys — that the local client/cmd/demarkus-mcp (stdio) returns from a direct-QUIC connection to the same world. The gateway is not a transformation layer; it's a wire-shape adapter. Implications:

• A document fetched through the broker and the same document fetched direct-QUIC produce identical content hashes. Federation, mark_resolve, and any downstream content-addressing assumptions stay intact.
• The 13 tool responses retain whatever format the existing demarkus-mcp emits. Slice 2-5 handlers map fetch.Result to the MCP CallToolResult in the same shape the local server uses; no new framing, no enrichment, no field renaming.
• Future world-side features (new metadata, new operations) propagate through the gateway without gateway-side code changes — the gateway forwards what the world emits.

This is the difference between a transport bridge and a protocol adapter. The gateway is the former.

Enterprise affordances that fall out for free

• Single audit point. Every tool call passes through one handler with the email claim already extracted. The broker's existing slog setup picks up email=alice@acme.com correlation without per-route plumbing (same field every other broker surface logs).
• One MCP server per knowledge system, not N per world. The plugin's claude mcp add only ever holds one entry per org the user joined; the broker exposes the org's worlds collectively.
• Rate-limiting at the gate. Per-email buckets shared across /tokens and /mcp (same identity = same bucket). Abuse from one identity can't fan out across surfaces.
• NetworkPolicy + RBAC bound the broker→world surface. Worlds stay reachable only from broker pods inside the cluster; mTLS broker→world is a hardening pass for later but the trust model already works without it because the network plane is itself bounded.

What this is NOT

The gateway is an additive overlay for the agent-shaped consumer, not a replacement for the direct-QUIC surface. The CLI, Obsidian plugin, kind-harness, and in-cluster server-to-server traffic all keep talking direct-QUIC mark://. Two consumers, two transports, same protocol underneath.

Vocabulary

A single Claude Code installation can have BOTH a knowledge system (the org's) and one or more souls (personal). They don't conflict — different commands, different MCP server entries, different auth modes.

Non-Negotiables

• No protocol changes. The demarkus message format (FETCH/PUBLISH/APPEND/VERSIONS/LIST/ARCHIVE/MERGE/etc.), content addressing, versioning, and capability-token mechanics are unchanged.
• No demarkus-server changes. Worlds keep their current QUIC + mark ALPN listener. They see a bearer token in the demarkus protocol header (the world access-token the broker is holding on the user's behalf) and serve it. They do not know the broker is the caller.
• Byte-for-byte markdown proxy. The gateway transports document content unchanged. mark_fetch through the broker returns the same body, version, etag, content-hash, and metadata keys as mark_fetch against the same world via stdio/direct-QUIC. The gateway adapts the wire shape (HTTP/JSON-RPC ↔ QUIC/demarkus), not the content. See § "Architectural framing" above.
• Capabilities stay on worlds. The world is the trust boundary for what a token can do. The broker holds world access-tokens in trust per-authenticated-user; the world validates them on every request. The broker does NOT re-implement capability checks.
• Single identity dimension: canonical verified email. The MCP session cache, /me/install, /tokens list/revoke, AllowConfig, and audit logs all key on the same canonical email (trimmed + lowercased). Unverified-email id_tokens are rejected at the gatewayAuth boundary (matches Issuer.ErrEmailUnverified). Subject claim is available for log correlation but is not a separate keying dimension. (Added in v5 — see top-of-doc changelog.)
• Tool surface parity with the local MCP server. All 13 tools the existing client/cmd/demarkus-mcp exposes (mark_fetch, mark_list, mark_versions, mark_publish, mark_append, mark_archive, mark_discover, mark_resolve, mark_index, mark_backlinks, mark_graph, mark_graph_export, mark_graph_publish) are exposed by the broker MCP from day one. Anything else is a plugin-side change, not a gateway concern.
• Gateway is always on. Every broker is a knowledge-system gateway; MCP is part of the binary, not a feature flag. MCPConfig.Addr defaults to :8081 when omitted so pre-gateway YAMLs upgrade silently. (Replaces v1-v3's "Opt-in deployment" Non-Negotiable, reverted post-Slice-1 — the universe-onboarding-only broker shape is not a product we ship.)
• No MVPs, no shortcuts. OAuth correctness, per-session world-token cache lifecycle (lazy mint + eviction on id_token expiry + cap on token churn), structured error envelopes, content-hash propagation through the tool surface, conflict-aware merge — all from day one within each slice (not later).
• QUIC stays first-class for direct-world access. This gateway is an enterprise overlay, not a transport replacement. The CLI, Obsidian plugin, and in-cluster server-to-server traffic all keep direct-QUIC access. The broker's existing /auth, /me/install, /tokens surfaces all stay as they are.

Out of Scope (explicit)

• Browsable REST surface. Deferred until a real non-agent consumer asks. The plugin is the only first-pass consumer; the agent is always the client.
• tools/demarkus-join binary. Canceled (see v3 changelog above). The join flow is a slash command, not a binary.
• mTLS broker → world. Confirmed deferred to a hardening pass. TLS terminates at the broker; broker→world QUIC traffic stays inside the cluster, bounded by NetworkPolicy + RBAC.
• Persistent world-token storage. Raw world tokens are NEVER persisted to disk by the broker. They live in process memory per session; broker restart drops them; next tool call re-mints. This matches the broker's existing posture (raw tokens are never recoverable from the issuances Secret).
• MCP resources and prompts features. Only the tools capability is exposed. Resources (file-style refs) and prompts (templates) are not part of the demarkus model.
• Long-lived subscriptions / server-initiated events. Polling is the model (/thoughts.md § "On Subscriptions and Polling").
• Multi-broker / multi-universe. Single broker per universe stays the model.
• Content negotiation beyond text/markdown. Same scope contract as the protocol (project_scope_markdown_only.md).
• Content transformation at the gateway. The byte-for-byte proxy contract excludes any kind of body rewriting, frontmatter normalization, link adjustment, or hash-stripping at the broker. The gateway forwards what the world emits.
• Subject-claim-based session keying. The session cache keys on canonical email, not the OIDC subject claim. (Pinned in v5 — see Non-Negotiables.)
• Persistent broker-side graph store. Slice 4b+5 shipped the graph-store tools backed by an in-memory, pod-scoped store. Re-crawl after broker restart is the documented operator expectation. Bucket-store-backed persistence is parked for the post-broker design window (/thoughts.md § "On Bucket Stores"). (Pinned post-Slice-4b — graph-store ephemerality is a deliberate trade-off, not a debt item.)

Architecture

┌──────────────┐  HTTPS (MCP)   ┌──────────────────────────────────┐  QUIC  ┌─────────┐
│ Claude Code  │ ─────────────► │  broker (/mcp endpoint)           │ ─────► │ world A │
│ plugin       │                │                                    │        │ (cluster│
│ (or any MCP  │ ◄───────────── │  JSON-RPC over Streamable HTTP    │ ◄───── │ internal)
│  client)     │   tool result  │                                    │        └─────────┘
└──────────────┘                │  Authorization: Bearer <id_token> │
                                │  - requireAuth (PR4 compositeV)   │        ┌─────────┐
                                │  - reject unverified-email tokens │ ─────► │ world B │
                                │  - per-email rate limit           │ ◄───── │         │
                                │                                    │        └─────────┘
                                │  Per-session world-token cache:    │
                                │  - key: (canonicalEmail,           │
                                │          world_name)               │
                                │  - mint lazily via Issuer.Mint-    │
                                │    Filtered on cache miss          │
                                │  - evict on id_token / token expiry│
                                │                                    │
                                │  Tool URLs: mark://{worldName}/    │
                                │  {path} — broker resolves worldName│
                                │  to <ns>.svc.cluster.local:6309    │
                                │  via cfg.Worlds[].Namespace        │
                                │                                    │
                                │  13 tools (parity with local       │
                                │  demarkus-mcp): mark_fetch, ...    │
                                │                                    │
                                │  OAuth metadata:                   │
                                │  /.well-known/oauth-protected-     │
                                │    resource (RFC 9728)             │
                                │  /.well-known/oauth-authorization- │
                                │    server (RFC 8414)               │
                                └──────────────────────────────────┘

Layer responsibilities

World name → internal address resolution

Tool URLs have the form mark://{worldName}/{path}. The broker resolves worldName to a cluster-internal address using the existing cfg.Worlds[].Namespace field. Default resolution:

mark://team-a/foo.md  →  mark://team-a.team-a.svc.cluster.local:6309/foo.md
                                ^^^^^^ ^^^^^^
                                name   namespace (from cfg.Worlds[].Namespace)

This is the standard Kubernetes Service DNS pattern. The world's Service must be named the same as the world (default for the demarkus-server chart). If an operator deploys a world with a non-default Service name, they configure cfg.Worlds[].InternalAddress explicitly — overrides the default DNS pattern. Default empty means "use the convention."

Pre-Flight Tasks (one PR ahead of Slice 1)

Pre-Flight 0 — Hoist client/internal/fetch to public

• client/internal/fetch/fetch.go → client/fetch/fetch.go (or client/pkg/fetch/fetch.go). Mirror for client/internal/merge since MERGE support lands in MVP via the conflict-aware mark_publish tool.
• Update import paths in client/cmd/demarkus-mcp, client/cmd/demarkus, client/cmd/demarkus-tui (and any other consumers within the client module).
• Same content; just relocated. Surface unchanged.
• Why: Go's internal-package rule blocks any module outside client/ from importing client/internal/.... The broker is in tools/demarkus-broker (different module under monorepo replace-directives), so it can't import the existing client library while it's internal.
• This is a CLIENT-module API change, not a protocol-core change. Not gated by feedback_plugin_scope.md (broker isn't a plugin), but worth flagging explicitly so reviewers see the architectural shift.

Pre-Flight 1 — Verify mark3labs/mcp-go's Streamable HTTP support

• The existing client/cmd/demarkus-mcp uses mcp-go's stdio transport (mcpserver.ServeStdio()). Confirm mcp-go v0.44+ supports Streamable HTTP (the current MCP spec's preferred remote transport).
• If yes: lifecycle the new gateway around mcp-go's HTTP server constructor.
• If no: upgrade to a version that does, swap to a different MCP Go library, or implement Streamable HTTP transport in-house (worst case).
• Output: one-paragraph journal note on transport choice before Slice 1 begins.

Sub-Tasks (sequenced; one slice per PR)

Slice 1 — Foundation: listener + initialize + tools/list (no tool implementations)

• tools/demarkus-broker/internal/broker/mcp_gateway.go (new) — mcpGateway listener, JSON-RPC dispatcher, Streamable HTTP transport. Separate listener from the existing management API.
• tools/demarkus-broker/internal/broker/mcp_oauth.go (new) — OAuth metadata endpoints: /.well-known/oauth-protected-resource (RFC 9728), /.well-known/oauth-authorization-server (RFC 8414). Reuses broker's existing Discovery machinery.
• tools/demarkus-broker/internal/broker/mcp_auth.go (new) — gatewayAuth middleware. id_token bearer extraction, validation via existing compositeVerifier, claim extraction for session keying, per-identity rate limiting (shared budget with /tokens routes — same identity, one bucket).
• tools/demarkus-broker/internal/broker/config.go — ServerConfig.MCP substruct: Addr (defaulted to :8081), TLS{...}. Slice 2+ adds SessionMaxIdle and WorldTokenTTL. Plus WorldConfig.InternalAddress (optional override for the default Service-DNS resolution).
• tools/demarkus-broker/main.go — start the MCP listener alongside the management API. Always on.
• tools/demarkus-broker/internal/broker/mcp_initialize.go (new) — handles MCP initialize JSON-RPC method. Advertises server capabilities (tools only).
• tools/demarkus-broker/internal/broker/mcp_tools_list.go (new) — handles tools/list. Returns the 13 tool definitions (names + JSON schemas) without implementations yet. Definitions mirror client/cmd/demarkus-mcp exactly (URL hint adapted to mark://{worldName}/{path}).
• Tests: OAuth metadata round-trip, initialize handshake, tools/list returns the 13 expected tool names, unauth requests rejected with 401 + WWW-Authenticate header pointing at auth-server metadata URL, expired/revoked bearer rejected, per-identity rate limit triggers 429.
• Note (v5 retro): Slice 1 landed before v5 pinned the email-keying decision. Any subject-keyed test or middleware label that shipped will be rewritten against canonical email in Slice 2. The middleware additionally rejects unverified-email tokens starting in Slice 2.

Slice 2 — Read tools + session cache foundation: mark_fetch, mark_list, mark_versions

• tools/demarkus-broker/internal/broker/mcp_session.go (new) — sessionCache keyed by canonical verified email. Holds map[worldName]cachedWorldToken{raw, expiresAt}. Lazy initialization on first cache miss. LRU cap (default 10000 emails) + idle eviction (SessionMaxIdle default 1h after last use). Singleflight on concurrent (email, world) mints so a burst of tool calls for the same identity+world coalesces to one MintFiltered call.
• tools/demarkus-broker/internal/broker/world_pool.go (new) — worldPool for fetch.Client reuse per world. Resolves worldName → internal address via the rule documented above.
• tools/demarkus-broker/internal/broker/mcp_tools_read.go (new) — handlers for mark_fetch, mark_list, mark_versions. URL parameter shape: mark://{worldName}/{path}. Resolves world by name; lazy-mints world access-token via Issuer.MintFiltered(ctx, Claims{Email, EmailVerified, Groups}, keep) on cache miss; calls fetch.Client.Fetch/List/Versions with the token; maps the demarkus Result to an MCP tool response byte-for-byte from the world's payload (matching local MCP server output format for parity per the Non-Negotiable).
• Extend gatewayAuth (from Slice 1) to reject id_tokens whose email_verified claim is false or missing — match Issuer.ErrEmailUnverified semantics so the broker has one identity gate, not two.
• Tests: each tool happy path, world-not-found tool error, document-not-found maps to MCP error, expired-bearer 401 mid-call, cache hit reuses token, cache miss mints, two concurrent calls for the same (email, world) coalesce to one mint (singleflight), session eviction on id_token expiry causes re-mint, unverified-email id_token rejected at gateway with the same shape as ErrEmailUnverified, proxy-fidelity test: same document fetched through broker MCP vs direct-QUIC produces identical body + content-hash + metadata.

Slice 3 — Write tools: mark_publish, mark_append, mark_archive

• tools/demarkus-broker/internal/broker/mcp_tools_write.go (new) — handlers for the three write ops.
• mark_publish reuses the existing expected_version + on_conflict shape from the local MCP server. Conflict-aware merge candidate flow lands in Slice 6; for Slice 3, on-conflict defaults to "fail" with the conflict envelope, NOT the merge-candidate path.
• mark_append reuses the auto-resolved-version pattern (omit expected_version, broker calls VERSIONS internally) — matches local MCP server behavior.
• mark_archive deletes by archiving (per the demarkus protocol's actual DELETE shape; the protocol op is ARCHIVE, not DELETE).
• Tests: happy paths, version-mismatch conflict, missing-expected_version on PUBLISH 400, auto-resolve path for APPEND, world-side RBAC failure surfaces as MCP tool error.

Slice 4 — Federation read tools: mark_discover, mark_resolve, mark_backlinks, mark_graph

• tools/demarkus-broker/internal/broker/mcp_tools_federation.go (new) — handlers that delegate to whichever demarkus client library functions back the local MCP server's federation tools (mapped during a second-pass spike at start of Slice 4, deferred until Pre-Flight 0 hoist completes and the public API is visible).
• These tools are read-only and don't change the session-cache shape.
• Tests: happy paths + edge cases per tool.

Slice 5 — Federation write tools: mark_index, mark_graph_export, mark_graph_publish

• tools/demarkus-broker/internal/broker/mcp_tools_federation_write.go (new).
• Same shape as Slice 3 for writes (expected_version, conflict handling).
• Tests: happy paths + conflict cases.

Slice 6 — Conflict-aware merge in mark_publish

• Reuses client/merge (hoisted in Pre-Flight 0).
• Adds the on_conflict: "merge" branch to mark_publish, matching local MCP server behavior.
• Tests: clean merge, structural merge, conflict-markers returned.

Slice 7 — Chart, RBAC, docs

• deploy/helm/demarkus-broker/values.yaml — server.mcp.addr (defaults to :8081), server.mcp.tls (existingSecretRef recommended), server.mcp.sessionMaxIdle, server.mcp.maxSessions, server.mcp.firstMint{MaxAttempts,InitialBackoff,MaxBackoff}. Plus worlds[].internalAddress for non-default Service DNS overrides. No enabled knob — the gateway is part of the binary. worldTokenTTL + worldPool dropped from earlier plan drafts (binary never had the fields — see v6 changelog).
• Templates: deployment.yaml (new containerPort + optional TLS mount), service.yaml (gateway port exposed), ingress.yaml (route mcp host to new port via SEPARATE hostname ingress.mcp.host, not path-routing), networkpolicy.yaml (allow ingress on mcp port), certificate.yaml (parallel cert-manager Certificate for the MCP host).
• No new RBAC needed — broker SA already has perms for issuances Secret + world Secrets that lazy-mint touches.
• deploy/helm/demarkus-broker/README.md — new "MCP gateway" section. TLS setup, the plugin-side flow (/knowledge-join from Claude Code), OAuth flow, rate-limit behavior, ephemeral graph-store callout (operators expect re-crawl after restart). Upgrade note: pre-gateway deployments get the listener on :8081 after upgrade; chart values let operators retune.
• tools/demarkus-broker/main.go package doc — bump Current scope to mention /mcp.
• New tools/demarkus-broker/MCP-API.md — operator/developer-facing spec for the MCP tool surface mirrored from client/cmd/demarkus-mcp. Lives in the broker package; not a protocol document.
• Optional --with-mcp-smoke stage in deploy/kind/up.sh — builds broker locally, sideloads into kind, installs the LOCAL chart with the locally-built image, runs three lightweight checks (RFC 9728 + 8414 metadata + 401 challenge). Full id_token / initialize / tools/call E2E deferred to Slice 8 (/knowledge-join exercises it naturally).

Slice 8 — Plugin /knowledge-join slash command (replaces canceled PR6)

• plugins/claude-code/commands/knowledge-join.md (new, per Claude Code plugin slash-command shape) — prompt-shaped command that takes a broker URL argument.
• Behavior:
  1. Validates the broker URL by issuing HEAD <url>/.well-known/oauth-protected-resource.
  2. Derives a slug from the broker hostname (first DNS label, lowercased + sanitized to [a-z0-9-]+).
  3. Runs claude mcp add --transport http {slug} {url}/mcp. Claude Code handles the OAuth device flow when the plugin first talks to the MCP server.
  4. Reports success + slug to the user; on failure shows the validation error.
• plugins/claude-code/scripts/lib.sh — bump SERVER/CLIENT/TOOLS_VERSION pins per feedback_plugin_version_pins.md.
• Tests: shell-script tests in plugins/claude-code/tests/ covering happy path (mock broker URL), validation-fail path, slug-derivation cases.

Scope estimate

Eight slices (plus Pre-Flight) across ~2-3 working weeks. Slice 8 is small but ships separately so the broker side can land + bake before plugin work touches it.

Open Questions To Resolve Before/During Implementation

1. URL shape inside MCP tool args. Resolved 2026-05-20: mark://{worldName}/{path} — broker resolves worldName to cluster-internal Service DNS (<name>.<namespace>.svc.cluster.local:6309); worlds[].internalAddress overrides the default for non-conforming deployments.
2. MCP transport: Streamable HTTP vs SSE. Resolved 2026-05-20 (Pre-Flight 1): Streamable HTTP via mcp-go.NewStreamableHTTPServer. Stream Resumability not supported in mcp-go v0.44 — irrelevant since the plan's Out-of-Scope already excludes long-lived subscriptions.
3. OAuth authorization-server identity. The broker is BOTH the MCP server (resource) AND the authorization server (the existing OIDC + device-flow surface). Lean: broker advertises itself as the authorization server via /.well-known/oauth-authorization-server. The actual IdP (Google, Okta, etc.) is one hop further — handled by existing broker OIDC machinery. Matches PR3 device-flow architecture. Slice 1 lands this as an alias of the OIDC discovery handler — RFC 8414 §3 tolerates the OIDC body's extra fields, and the iss-override Discovery already applies makes it usable as auth-server metadata.
4. Session keying. Resolved 2026-05-21 (v5): canonical verified email (trim + lowercase, email_verified=true required). Same key the broker already uses for /me/install, /tokens list/revoke, AllowConfig, and audit logs. Multiple devices for the same canonical email share one session. Subject claim remains in the id_token for log correlation but is not the cache key.
5. World-token TTL and minting cadence. Resolved (Slice 2): cache world-tokens with their natural expiry from MintFiltered — broker respects the world's own TTL (worlds[].defaultToken.expiresAfter) rather than overriding broker-side. On id_token rotation (refresh-grant), session_cache survives because canonical email is stable across refreshes. No chart-side worldTokenTTL knob (would silently no-op).
6. Federation tool implementations. Resolved (Slices 4a + 4b+5): client/index, client/graphstore, client/graph, client/links hoisted (Pre-Flight 0-style hoists) to make them importable from the broker. Federation tools delegate to those packages.
7. MCP error envelope for partial mint failures. When the broker mints a world token on first call, partial-mint surfaces as ONE successful tool response + server log entry, NOT a partial-mint envelope.
8. MCP session lifecycle on broker restart. Lean: in-memory only; broker restart drops all sessions; plugin re-authenticates via the existing OAuth refresh path (PR4 broker-signed refresh tokens). Persistent session storage deferred.
9. First-mint dispatch latency / Secret-propagation race. Resolved 2026-05-21 (Slice 2): retry-on-401-after-mint with exponential backoff inside the broker handler. Defaults: 6 attempts, 250ms → 8s exponential (~16s total worst-case waiting). Configurable via cfg.Server.MCP.FirstMintMax*. Distinguishes fresh-mint 401s (retry) from cache-hit 401s (invalidate + immediate re-mint). Sync-wait + prewarm options rejected — see journal 2026-05-21.

Risks Specific To The MCP Gateway

• MCP protocol surface gains an internet-reachable port. Same risk shape as the original REST plan: hardened auth middleware (existing compositeVerifier), strict per-identity rate limit, structured MCP error envelope, TLS-only in production.
• World-token cache memory growth. A broker serving 10k emails × 5 worlds × ~200 bytes per cached token entry = ~10MB. Bounded but not zero. Mitigation: LRU cap, idle eviction (default 1h after last tool call), per-process memory metric for ops dashboards.
• Mint storm on broker restart. Every active plugin reconnects after a restart and re-mints tokens for every world it touches. Mitigation: mutateSecret's optimistic-concurrency retry handles contention; sharded issuances Secret if a real customer hits the wall (existing Phase 7 path).
• SIGHUP cadence on worlds. Each first-use mint per (email, world) triggers a SIGHUP on the addressed world. The reload itself is cheap (atomic pointer swap behind sync.RWMutex; no connection drop), but the upstream Secret-propagation lag (Open Question 9) shapes the perceived latency. Mitigation: singleflight on (email, world) coalesces concurrent first-call bursts to one mint; the retry-on-401 loop (OQ#9 resolution) absorbs the residual.
• Issuance bloat under high session churn. Plugin sessions that come and go faster than the world-token TTL leak issuance records into the Secret. Sweeper retires them on expiry.
• Plugin holds N world identities through one broker connection. Single MCP-server-for-the-universe architecture means a buggy broker handler could expose one user's session to another's bearer. Mitigation: strict canonical-email keying on session cache; every tool call re-validates the bearer against the cache key; integration test that proves cross-email isolation under concurrent load.
• mcp-go library maturity. Third-party Go MCP library; if it lacks Streamable HTTP, the choice in Pre-Flight 1 cascades. Resolved — v0.44 ships NewStreamableHTTPServer as a production http.Handler.
• OAuth metadata vs OIDC metadata overlap. Broker already serves /.well-known/openid-configuration. RFC 8414 (OAuth) and OIDC discovery have largely-overlapping shapes; serving both means two endpoints that mostly mirror each other. Mitigation: Slice 1 aliases auth-server to the existing Discovery handler — one renderer, two routes. Slice 7 follow-on: splitting the management and MCP surfaces onto separate hostnames (ingress.host vs ingress.mcp.host) also eliminates any future .well-known/* collision risk by giving each surface its own host's namespace.
• Plugin slash-command surface name collision. /knowledge-join is new; verify no other plugin in the marketplace uses it. Mitigation: the slash command is namespaced to the demarkus-memory plugin; slot the command into the plugin's commands/ directory and Claude Code's resolution handles namespacing.
• Always-on listener forces a port assignment on every deployment. With the v4 "gateway is always on" revision, every broker binds the second listener whether the operator configured it or not. Mitigation: defaultMCPAddr = :8081 picked to avoid the management API's typical :8080; chart's values.yaml surfaces the override; kind-harness and dev configs that don't want the listener route the port to nowhere (no service, no ingress) rather than disabling it via config.
• Graph-store ephemerality is a user-visible behavior. Bucket-store-backed persistence is parked, but until it ships, operators MUST document the re-crawl-after-restart expectation. The chart README's "MCP gateway" section + MCP-API.md's per-tool warning on mark_backlinks and friends both call it out prominently. Mitigation deferred to bucket-store work; current mitigation is documentation, not code.

Touch Points With Adjacent Work

• Universe-onboarding PR5 (shipped #141, 2026-05-20). No code coupling. /me/install stays as-is — identity-introspection surface alongside the MCP gateway's operational data plane. Same identity dimension (canonical email) underneath, so audit and rate-limit semantics line up across the two surfaces (v5).
• Universe-onboarding PR6 (tools/demarkus-join): canceled. Replaced by Slice 8 (/knowledge-join plugin slash command).
• Universe-onboarding PR7/PR8: substantially reshaped. PR7 (originally "plugin slash commands") may merge into Slice 8; PR8 (originally docs) merges into Slice 7 + Slice 8's docs.
• Future /soul-join plugin slash command (separate plan): direct-QUIC join for personal souls. Distinct from /knowledge-join (organizational, broker-fronted). Same vocabulary axis as the project's own demarkus-soul vs the eventual enterprise knowledge systems.
• /thoughts.md § "On the Protocol as the Permanent Layer": principle this plan is built on.
• /thoughts.md § "On Bucket Stores": post-broker direction for the graph-store persistence question Slice 4b deliberately punted on.
• Future browsable REST surface plan: additive. The MCP gateway is on /mcp; a REST surface would live on /v1/... paths on the same listener, sharing OAuth auth and session cache.
• Future mTLS broker → world plan: anchors here. The worldPool can grow mTLS in place without changing the tool surface.

Resume Steps

Plan complete — no resume needed. Shipping summary lives in /completed-plans.md § "Broker MCP Gateway". This document stays in place as the architectural reference + decision trail; the changelog at the top traces every load-bearing pivot from v1 (REST) to v7 (plan complete). For follow-on work the plan punted on, see Out of Scope (bucket-store persistence + browsable REST surface + mTLS broker→world).

Done When

• All eight slices merged (plus Pre-Flight 0). Met 2026-05-23.
• go test -race ./... green inside tools/demarkus-broker/ and client/. Met per slice merges.
• helm unittest . green for the broker chart. Met (99/99 at Slice 7 close).
• pre-commit.sh green. Met per slice merges.
• Manual end-to-end: user runs /knowledge-join https://broker.example.com in Claude Code → OAuth device flow completes → plugin sees all 13 demarkus tools → mark_fetch mark://team-a/foo.md returns the world's content; cached world-token reused on the next call; broker logs show only one mint per (email, world). Surface shipped; field validation against a real org broker is the natural next operator action.
• Operator-facing README documents the deployment. Met (Slice 7 README + MCP-API.md).
• Journal entry covering any design decisions that landed differently from this plan. Met per slice merges (see /journal/2026-05-21.md, /2026-05-22.md, /2026-05-23.md).

Implementation Status

• Pre-Flight 0 (hoist client/internal/fetch + merge) — shipped 2026-05-20 (#142). 5 files relocated, 9 consumers re-imported. Cleared the path for tools/demarkus-broker to import client/fetch and client/merge across module boundaries.
• Pre-Flight 1 (mcp-go Streamable HTTP spike) — resolved 2026-05-20. mcp-go.NewStreamableHTTPServer is a production http.Handler at v0.44; Stream Resumability is the only missing feature and it's already out-of-scope.
• Slice 1 (foundation: listener + initialize + tools/list) — shipped 2026-05-20 (#143, b90cda6). ~660 LOC production, ~798 LOC tests. Always-on listener + 13-tool registration (placeholders) + OAuth metadata + gatewayAuth with RFC 6750+9728 challenge.
• Slice 2 (read tools + session cache) — shipped 2026-05-21 (#146, 7c529d4). mark_fetch / mark_list / mark_versions backed by canonical-email-keyed session cache, worldPool, lazy mint via Issuer.MintFiltered, singleflight on concurrent first-call bursts, retry-on-401-after-mint with exponential backoff (OQ#9 resolution). Proxy-fidelity test pins byte-for-byte parity with local demarkus-mcp.
• Slice 3 (write tools) — shipped 2026-05-21 (#147, 5387701). mark_publish / mark_append / mark_archive. dispatchWithAuth refactor (closure-based, shared retry loop across read + write). Conflict + not-permitted forward verbatim (not as tool errors); on_conflict=merge rejected with explicit Slice 6 pointer.
• Slice 4a (federation reads: discover + resolve) — shipped 2026-05-22 (#148, 115a09b). mark_discover + mark_resolve. Cross-org candidate skip semantics; client/internal/index hoisted to client/index. Split off from original Slice 4 scope to keep the graph-store design call (4b) cleanly separated.
• Slice 4b+5 (graph-store federation tools) — shipped 2026-05-22 (#149, fa9f86d). mark_backlinks + mark_graph + mark_index + mark_graph_export + mark_graph_publish backed by an ephemeral in-memory graph store (per-pod lifetime, re-crawl after restart). Fritz signed off on the persistence trade-off after the gap framing. client/graphstore + client/graph + client/links hoisted. Brought all 13 tools to real handlers — notImplementedHandler remains as a defensive fallback but no advertised tool falls through to it. Bucket-store-backed persistence parked for the post-broker window (/thoughts.md § "On Bucket Stores").
• Slice 6 (conflict-aware merge in mark_publish) — shipped 2026-05-22 (#150, 80a4008). on_conflict="merge" candidate flow lit up, default flipped from "fail" to "merge" to match local demarkus-mcp. brokerMergeAdapter captures ctx in the struct (only viable shape against merge.Client's ctx-free interface). All 13 tools now have full semantic parity with local demarkus-mcp.
• Slice 7 (chart + RBAC + docs) — shipped 2026-05-22 (#151, 277f83f). Chart wiring: server.mcp.* block, worlds[].internalAddress, deployment second containerPort + optional TLS volume, service mcp port, networkpolicy ingress port, ingress.mcp.host (separate-hostname topology), parallel cert-manager Certificate. New README "MCP gateway" section + new tools/demarkus-broker/MCP-API.md 13-tool spec. Kind harness --with-mcp-smoke stage (builds broker locally, sideloads, installs local chart, runs RFC 9728 + 8414 metadata + 401-challenge checks). 99 helm-unittest cases green. CodeRabbit round-1 fixes: strict-digit mcpPort (closes silent int "abc"→0 footgun), ingress.mcp.host ≠ ingress.host guard, numeric port-collision compare (catches 0.0.0.0:8080), corrected TLS validation comment, MCP-API.md fence + ephemeral-graph wording. Plan-vs-code drift discovered: WorldTokenTTL + WorldPool{} knobs dropped from values.yaml (binary has no fields).
• Slice 8 (plugin /knowledge-join slash command) — shipped 2026-05-23 (#152, bc0d5cc). New plugins/claude-code/commands/knowledge-join.md prompt-shaped command + scripts/knowledge-join.sh URL-validation/slug-derivation helper + first shell-test suite under plugins/claude-code/tests/ (11 cases, python3-mocked broker). Plugin bumped to v0.2.0; CLIENT_VERSION caught up 0.12.33 → 0.12.36 and TOOLS_VERSION 0.1.10 → 0.1.16 (drift catch-up per feedback_plugin_version_pins.md). CodeRabbit round-1 fixes: case-insensitive URL scheme match (RFC 3986) + scheme normalization to lowercase in emitted URLs, bash language tag on doc fences, test rewrite to actually exercise upper→lower (http://LOCALHOST → slug=localhost). CI workflow needed permissions: contents:read pull-requests:read block on detect-changes job for dorny/paths-filter@v3 after Settings → Actions tightening — shipped same PR.

Plan complete. All eight slices + Pre-Flight 0/1 on main. See /completed-plans.md § "Broker MCP Gateway".

History of plan revisions:

• v1: REST HTTP API
• v2: pivoted to MCP-server architecture
• v3: URL shape resolved (mark://{worldName}/{path}), PR6 canceled, /knowledge-join plugin slash command becomes Slice 8
• v4: gateway is always on (no enabled flag); enterprise framing + byte-for-byte proxy semantics pinned as Non-Negotiables
• v5: session cache pinned to canonical verified email (single identity dimension across broker); unverified-email id_tokens rejected at gatewayAuth; Open Question 9 (Secret-propagation race) added as Slice-2-blocking decision; SIGHUP-cadence risk + singleflight mitigation documented
• v6: Slices 2–7 shipped, only Slice 8 remains. Plan-vs-code drift documented (WorldTokenTTL + WorldPool{} never implemented; dropped from chart). Ingress topology pinned to separate-hostname split. Graph-store ephemerality added to Out-of-Scope as a deliberate trade-off (not debt). OQ#5, #6, #9 marked resolved.
• v7: plan complete. Slice 8 (/knowledge-join plugin slash command + first plugin shell-test suite) shipped as #152. Slug heuristic clarified (first DNS label, lowercased, sanitized). Test-only HTTPS escape hatch documented. CI workflow permissions fix shipped same PR. All Done-When criteria met.

Related documents

• Universe onboarding: parent sequence whose PR6 to PR8 this gateway reshaped
• Universe onboarding PR5: /me/install identity surface alongside the gateway
• Soul join: the personal direct-QUIC counterpart to /knowledge-join
• Bucket store: graph-store persistence the gateway punted on
• Completed plans: shipping summary for the Broker MCP Gateway
• Thoughts: protocol as permanent layer principle this plan builds on