Plan: Broker MCP Gateway (Knowledge-System Layer)

Plan rewritten 2026-05-20. v1 described a REST HTTP API. After alignment with Fritz, the architectural target shifted: the gateway is an MCP server exposed at a single /mcp endpoint, authenticated against the company SSO (id_token bearer), with world access-tokens cached broker-side per session. Agent-as-client is the only first-pass consumer. A browser/REST surface is a later, additive plan if a real consumer asks.

v3 (2026-05-20 same session): Open Question 1 resolved (mark://{worldName}/{path} with cluster-internal DNS resolution). PR6 (tools/demarkus-join binary) canceled outright; the join flow becomes the /knowledge-join slash command in plugins/claude-code/, distinct from a future /soul-join (personal direct-QUIC). Vocabulary pin: knowledge system = organizational brokered universe; soul = personal, direct-QUIC. Different commands, different deployment shapes, shared protocol underneath.

The enterprise-deployment piece. Adds a remote MCP server to the broker that exposes the demarkus tool surface (mark_fetch, mark_publish, etc.) to plugin-style agents over the company's existing SSO. Worlds stay cluster-internal; the broker becomes the single MCP endpoint the plugin connects to; corporate networks that block QUIC over UDP reach the universe over standard HTTPS. The plugin's local vault holds only the id_token (the company SSO bearer); world access-tokens are minted lazily broker-side per session, scoped to the authenticated user, never persisted to disk.

Architectural framing in /thoughts.md § "On the Protocol as the Permanent Layer": demarkus core is the permanent layer; this gateway is a knowledge-system overlay that the protocol doesn't know exists. See also feedback_core_vs_knowledge_layer.md in agent auto-memory.

Why this is the priority

Stated 2026-05-20: "this has to be the next task we do, it is the missing piece that the company needs." Two enterprise blockers dissolve together:

• "20 dev teams × N worlds plugin-config grind." Without the gateway, every developer at a 20-team org installs N local MCP servers per workstation, one per world they need. With the gateway, every developer adds ONE MCP server entry (the broker URL) and the broker exposes the universe's worlds as a single tool surface.
• Corporate-network UDP blocking. QUIC over UDP gets blocked by enterprise firewalls, transparent proxies, and DPI appliances. HTTPS over 443 works everywhere. The gateway speaks HTTPS publicly and translates to QUIC internally.

Universe-onboarding's PR6/7/8 are reshaped by this work: PR6 (tools/demarkus-join binary) is canceled. The join flow folds into /knowledge-join in plugins/claude-code/. PR7 becomes "plugin slash command implementation" and PR8 is documentation.

Vocabulary

A single Claude Code installation can have BOTH a knowledge system (the org's) and one or more souls (personal). They don't conflict — different commands, different MCP server entries, different auth modes.

Non-Negotiables

• No protocol changes. The demarkus message format (FETCH/PUBLISH/APPEND/VERSIONS/LIST/ARCHIVE/MERGE/etc.), content addressing, versioning, and capability-token mechanics are unchanged.
• No demarkus-server changes. Worlds keep their current QUIC + mark ALPN listener. They see a bearer token in the demarkus protocol header (the world access-token the broker is holding on the user's behalf) and serve it. They do not know the broker is the caller.
• Capabilities stay on worlds. The world is the trust boundary for what a token can do. The broker holds world access-tokens in trust per-authenticated-user; the world validates them on every request. The broker does NOT re-implement capability checks.
• Tool surface parity with the local MCP server. All 13 tools the existing client/cmd/demarkus-mcp exposes (mark_fetch, mark_list, mark_versions, mark_publish, mark_append, mark_archive, mark_discover, mark_resolve, mark_index, mark_backlinks, mark_graph, mark_graph_export, mark_graph_publish) are exposed by the broker MCP from day one. Anything else is a plugin-side change, not a gateway concern.
• Opt-in deployment. The MCP listener is off by default. Operators enable it explicitly; the existing direct-QUIC universe deployment stays the default.
• No MVPs, no shortcuts. OAuth correctness, per-session world-token cache lifecycle (lazy mint + eviction on id_token expiry + cap on token churn), structured error envelopes, content-hash propagation through the tool surface, conflict-aware merge — all from day one within each slice (not later).
• QUIC stays first-class for direct-world access. This gateway is an enterprise overlay, not a transport replacement. The CLI, Obsidian plugin, and in-cluster server-to-server traffic all keep direct-QUIC access. The broker's existing /auth, /me/install, /tokens surfaces all stay as they are.

Out of Scope (explicit)

• Browsable REST surface. Deferred until a real non-agent consumer asks. The plugin is the only first-pass consumer; the agent is always the client.
• tools/demarkus-join binary. Canceled (see v3 changelog above). The join flow is a slash command, not a binary.
• mTLS broker → world. Confirmed deferred to a hardening pass. TLS terminates at the broker; broker→world QUIC traffic stays inside the cluster, bounded by NetworkPolicy + RBAC.
• Persistent world-token storage. Raw world tokens are NEVER persisted to disk by the broker. They live in process memory per session; broker restart drops them; next tool call re-mints. This matches the broker's existing posture (raw tokens are never recoverable from the issuances Secret).
• MCP resources and prompts features. Only the tools capability is exposed. Resources (file-style refs) and prompts (templates) are not part of the demarkus model.
• Long-lived subscriptions / server-initiated events. Polling is the model (/thoughts.md § "On Subscriptions and Polling").
• Multi-broker / multi-universe. Single broker per universe stays the model.
• Content negotiation beyond text/markdown. Same scope contract as the protocol (project_scope_markdown_only.md).

Architecture

┌──────────────┐  HTTPS (MCP)   ┌──────────────────────────────────┐  QUIC  ┌─────────┐
│ Claude Code  │ ─────────────► │  broker (/mcp endpoint)           │ ─────► │ world A │
│ plugin       │                │                                    │        │ (cluster│
│ (or any MCP  │ ◄───────────── │  JSON-RPC over Streamable HTTP    │ ◄───── │ internal)
│  client)     │   tool result  │                                    │        └─────────┘
└──────────────┘                │  Authorization: Bearer <id_token> │
                                │  - requireAuth (PR4 compositeV)   │        ┌─────────┐
                                │  - per-subject rate limit         │ ─────► │ world B │
                                │                                    │ ◄───── │         │
                                │  Per-session world-token cache:    │        └─────────┘
                                │  - key: (subject_hash, world_name) │
                                │  - mint lazily via Issuer.Mint-    │
                                │    Filtered on cache miss          │
                                │  - evict on id_token / token expiry│
                                │                                    │
                                │  Tool URLs: mark://{worldName}/    │
                                │  {path} — broker resolves worldName│
                                │  to <ns>.svc.cluster.local:6309    │
                                │  via cfg.Worlds[].Namespace        │
                                │                                    │
                                │  13 tools (parity with local       │
                                │  demarkus-mcp): mark_fetch, ...    │
                                │                                    │
                                │  OAuth metadata:                   │
                                │  /.well-known/oauth-protected-     │
                                │    resource (RFC 9728)             │
                                │  /.well-known/oauth-authorization- │
                                │    server (RFC 8414)               │
                                └──────────────────────────────────┘

Layer responsibilities

World name → internal address resolution

Tool URLs have the form mark://{worldName}/{path}. The broker resolves worldName to a cluster-internal address using the existing cfg.Worlds[].Namespace field. Default resolution:

mark://team-a/foo.md  →  mark://team-a.team-a.svc.cluster.local:6309/foo.md
                                ^^^^^^ ^^^^^^
                                name   namespace (from cfg.Worlds[].Namespace)

This is the standard Kubernetes Service DNS pattern. The world's Service must be named the same as the world (default for the demarkus-server chart). If an operator deploys a world with a non-default Service name, they configure cfg.Worlds[].InternalAddress explicitly — overrides the default DNS pattern. Default empty means "use the convention."

Pre-Flight Tasks (one PR ahead of Slice 1)

Pre-Flight 0 — Hoist client/internal/fetch to public

• client/internal/fetch/fetch.go → client/fetch/fetch.go (or client/pkg/fetch/fetch.go). Mirror for client/internal/merge since MERGE support lands in MVP via the conflict-aware mark_publish tool.
• Update import paths in client/cmd/demarkus-mcp, client/cmd/demarkus, client/cmd/demarkus-tui (and any other consumers within the client module).
• Same content; just relocated. Surface unchanged.
• Why: Go's internal-package rule blocks any module outside client/ from importing client/internal/.... The broker is in tools/demarkus-broker (different module under monorepo replace-directives), so it can't import the existing client library while it's internal.
• This is a CLIENT-module API change, not a protocol-core change. Not gated by feedback_plugin_scope.md (broker isn't a plugin), but worth flagging explicitly so reviewers see the architectural shift.

Pre-Flight 1 — Verify mark3labs/mcp-go's Streamable HTTP support

• The existing client/cmd/demarkus-mcp uses mcp-go's stdio transport (mcpserver.ServeStdio()). Confirm mcp-go v0.44+ supports Streamable HTTP (the current MCP spec's preferred remote transport).
• If yes: lifecycle the new gateway around mcp-go's HTTP server constructor.
• If no: upgrade to a version that does, swap to a different MCP Go library, or implement Streamable HTTP transport in-house (worst case).
• Output: one-paragraph journal note on transport choice before Slice 1 begins.

Sub-Tasks (sequenced; one slice per PR)

Slice 1 — Foundation: listener + initialize + tools/list (no tool implementations)

• tools/demarkus-broker/internal/broker/mcp_gateway.go (new) — mcpGateway listener, JSON-RPC dispatcher, Streamable HTTP transport. Separate listener from the existing management API.
• tools/demarkus-broker/internal/broker/mcp_oauth.go (new) — OAuth metadata endpoints: /.well-known/oauth-protected-resource (RFC 9728), /.well-known/oauth-authorization-server (RFC 8414). Reuses broker's existing Discovery machinery.
• tools/demarkus-broker/internal/broker/mcp_auth.go (new) — gatewayAuth middleware. id_token bearer extraction, validation via existing compositeVerifier, subject claim extraction, per-subject rate limiting (shared budget with /tokens routes — same identity, one bucket).
• tools/demarkus-broker/internal/broker/config.go — ServerConfig.MCP substruct: Enabled, Addr, TLS{...}, SessionMaxIdle, WorldTokenTTL. Plus WorldConfig.InternalAddress (optional override for the default Service-DNS resolution).
• tools/demarkus-broker/main.go — start the MCP listener when enabled, alongside the management API.
• tools/demarkus-broker/internal/broker/mcp_initialize.go (new) — handles MCP initialize JSON-RPC method. Advertises server capabilities (tools only).
• tools/demarkus-broker/internal/broker/mcp_tools_list.go (new) — handles tools/list. Returns the 13 tool definitions (names + JSON schemas) without implementations yet. Definitions mirror client/cmd/demarkus-mcp exactly.
• Tests: OAuth metadata round-trip, initialize handshake, tools/list returns the 13 expected tool names, unauth requests rejected with 401 + WWW-Authenticate header pointing at auth-server metadata URL, expired/revoked bearer rejected, per-subject rate limit triggers 429.

Slice 2 — Read tools + session cache foundation: mark_fetch, mark_list, mark_versions

• tools/demarkus-broker/internal/broker/mcp_session.go (new) — sessionCache keyed by subject_hash. Holds map[worldName]cachedWorldToken{raw, expiresAt}. Lazy initialization on first cache miss. LRU cap (default 10000 subjects) + idle eviction (SessionMaxIdle default 1h after last use).
• tools/demarkus-broker/internal/broker/world_pool.go (new) — worldPool for fetch.Client reuse per world. Resolves worldName → internal address via the rule documented above.
• tools/demarkus-broker/internal/broker/mcp_tools_read.go (new) — handlers for mark_fetch, mark_list, mark_versions. URL parameter shape: mark://{worldName}/{path}. Resolves world by name; lazy-mints world access-token via Issuer.MintFiltered on cache miss; calls fetch.Client.Fetch/List/Versions with the token; maps the demarkus Result to an MCP tool response (matching local MCP server output format for parity).
• Tests: each tool happy path, world-not-found tool error, document-not-found maps to MCP error, expired-bearer 401 mid-call, cache hit reuses token, cache miss mints, two concurrent calls for the same (subject, world) coalesce to one mint (singleflight), session eviction on id_token expiry causes re-mint.

Slice 3 — Write tools: mark_publish, mark_append, mark_archive

• tools/demarkus-broker/internal/broker/mcp_tools_write.go (new) — handlers for the three write ops.
• mark_publish reuses the existing expected_version + on_conflict shape from the local MCP server. Conflict-aware merge candidate flow lands in Slice 6; for Slice 3, on-conflict defaults to "fail" with the conflict envelope, NOT the merge-candidate path.
• mark_append reuses the auto-resolved-version pattern (omit expected_version, broker calls VERSIONS internally) — matches local MCP server behavior.
• mark_archive deletes by archiving (per the demarkus protocol's actual DELETE shape; the protocol op is ARCHIVE, not DELETE).
• Tests: happy paths, version-mismatch conflict, missing-expected_version on PUBLISH 400, auto-resolve path for APPEND, world-side RBAC failure surfaces as MCP tool error.

Slice 4 — Federation read tools: mark_discover, mark_resolve, mark_backlinks, mark_graph

• tools/demarkus-broker/internal/broker/mcp_tools_federation.go (new) — handlers that delegate to whichever demarkus client library functions back the local MCP server's federation tools (mapped during a second-pass spike at start of Slice 4, deferred until Pre-Flight 0 hoist completes and the public API is visible).
• These tools are read-only and don't change the session-cache shape.
• Tests: happy paths + edge cases per tool.

Slice 5 — Federation write tools: mark_index, mark_graph_export, mark_graph_publish

• tools/demarkus-broker/internal/broker/mcp_tools_federation_write.go (new).
• Same shape as Slice 3 for writes (expected_version, conflict handling).
• Tests: happy paths + conflict cases.

Slice 6 — Conflict-aware merge in mark_publish

• Reuses client/internal/merge (hoisted to client/merge in Pre-Flight 0).
• Adds the on_conflict: "merge" branch to mark_publish, matching local MCP server behavior.
• Tests: clean merge, structural merge, conflict-markers returned.

Slice 7 — Chart, RBAC, docs

• deploy/helm/demarkus-broker/values.yaml — server.mcp.enabled, server.mcp.addr, server.mcp.tls (existingSecretRef recommended), server.mcp.sessionMaxIdle, server.mcp.worldTokenTTL, server.mcp.worldPool. Plus worlds[].internalAddress for non-default Service DNS overrides.
• Templates: deployment.yaml (new containerPort + TLS mount), service.yaml (gateway port exposed), ingress.yaml (route mcp host to new port), networkpolicy.yaml (allow ingress on mcp port).
• No new RBAC needed — broker SA already has perms for issuances Secret + world Secrets that lazy-mint touches.
• deploy/helm/demarkus-broker/README.md — new "MCP gateway" section. When to enable, TLS setup, the plugin-side flow (/knowledge-join from Claude Code), OAuth flow, rate-limit behavior.
• tools/demarkus-broker/main.go package doc — bump Current scope to mention /mcp.
• New tools/demarkus-broker/MCP-API.md — operator/developer-facing spec for the MCP tool surface mirrored from client/cmd/demarkus-mcp. Lives in the broker package; not a protocol document.

Slice 8 — Plugin /knowledge-join slash command (replaces canceled PR6)

• plugins/claude-code/commands/knowledge-join.md (new, per Claude Code plugin slash-command shape) — prompt-shaped command that takes a broker URL argument.
• Behavior:
  1. Validates the broker URL by issuing HEAD <url>/.well-known/oauth-protected-resource.
  2. Derives a slug from the broker hostname (e.g., acme from broker.acme.com).
  3. Runs claude mcp add --transport http {slug} {url}/mcp. Claude Code handles the OAuth device flow when the plugin first talks to the MCP server.
  4. Reports success + slug to the user; on failure shows the validation error.
• plugins/claude-code/scripts/lib.sh — bump SERVER/CLIENT/TOOLS_VERSION pins per feedback_plugin_version_pins.md.
• Tests: shell-script tests in plugins/claude-code/tests/ covering happy path (mock broker URL), validation-fail path, slug-derivation cases.

Scope estimate

Eight slices (plus Pre-Flight) across ~2-3 working weeks. Slice 8 is small but ships separately so the broker side can land + bake before plugin work touches it.

Open Questions To Resolve Before/During Implementation

1. URL shape inside MCP tool args. Resolved 2026-05-20: mark://{worldName}/{path} — broker resolves worldName to cluster-internal Service DNS (<name>.<namespace>.svc.cluster.local:6309); worlds[].internalAddress overrides the default for non-conforming deployments.
2. MCP transport: Streamable HTTP vs SSE. Lean: Streamable HTTP per current MCP spec. Locked in by Pre-Flight 1.
3. OAuth authorization-server identity. The broker is BOTH the MCP server (resource) AND the authorization server (the existing OIDC + device-flow surface). Lean: broker advertises itself as the authorization server via /.well-known/oauth-authorization-server. The actual IdP (Google, Okta, etc.) is one hop further — handled by existing broker OIDC machinery. Matches PR3 device-flow architecture.
4. Session keying. Lean: subject claim hash from id_token. Multiple devices for the same user share one session.
5. World-token TTL and minting cadence. Lean: cache world-tokens with their natural expiry from MintFiltered. On id_token rotation (refresh-grant), session_cache survives because subject claim is stable across refreshes.
6. Federation tool implementations. Slice 4/5 reuse the client library functions backing the existing federation tools. Second-pass spike at start of Slice 4.
7. MCP error envelope for partial mint failures. When the broker mints a world token on first call, partial-mint surfaces as ONE successful tool response + server log entry, NOT a partial-mint envelope.
8. MCP session lifecycle on broker restart. Lean: in-memory only; broker restart drops all sessions; plugin re-authenticates via the existing OAuth refresh path (PR4 broker-signed refresh tokens). Persistent session storage deferred.

Risks Specific To The MCP Gateway

• MCP protocol surface gains an internet-reachable port. Same risk shape as the original REST plan: hardened auth middleware (existing compositeVerifier), strict per-subject rate limit, structured MCP error envelope, TLS-only in production.
• World-token cache memory growth. A broker serving 10k subjects × 5 worlds × ~200 bytes per cached token entry = ~10MB. Bounded but not zero. Mitigation: LRU cap, idle eviction (default 1h after last tool call), per-process memory metric for ops dashboards.
• Mint storm on broker restart. Every active plugin reconnects after a restart and re-mints tokens for every world it touches. Mitigation: mutateSecret's optimistic-concurrency retry handles contention; sharded issuances Secret if a real customer hits the wall (existing Phase 7 path).
• Issuance bloat under high session churn. Plugin sessions that come and go faster than the world-token TTL leak issuance records into the Secret. Sweeper retires them on expiry.
• Plugin holds N world identities through one broker connection. Single MCP-server-for-the-universe architecture means a buggy broker handler could expose one user's session to another's bearer. Mitigation: strict subject-claim keying on session cache; every tool call re-validates the bearer against the cache key; integration test that proves cross-subject isolation under concurrent load.
• mcp-go library maturity. Third-party Go MCP library; if it lacks Streamable HTTP, the choice in Pre-Flight 1 cascades. Mitigation: Pre-Flight 1 is fact-finding; the next decision lands as a one-paragraph journal entry before Slice 1.
• OAuth metadata vs OIDC metadata overlap. Broker already serves /.well-known/openid-configuration. RFC 8414 (OAuth) and OIDC discovery have largely-overlapping shapes; serving both means two endpoints that mostly mirror each other. Mitigation: one shared rendering helper, two route registrations.
• Plugin slash-command surface name collision. /knowledge-join is new; verify no other plugin in the marketplace uses it. Mitigation: the slash command is namespaced to the demarkus-memory plugin; slot the command into the plugin's commands/ directory and Claude Code's resolution handles namespacing.

Touch Points With Adjacent Work

• Universe-onboarding PR5 (shipped #141, 2026-05-20). No code coupling. /me/install stays as-is — identity-introspection surface alongside the MCP gateway's operational data plane.
• Universe-onboarding PR6 (tools/demarkus-join): canceled. Replaced by Slice 8 (/knowledge-join plugin slash command).
• Universe-onboarding PR7/PR8: substantially reshaped. PR7 (originally "plugin slash commands") may merge into Slice 8; PR8 (originally docs) merges into Slice 7 + Slice 8's docs.
• Future /soul-join plugin slash command (separate plan): direct-QUIC join for personal souls. Distinct from /knowledge-join (organizational, broker-fronted). Same vocabulary axis as the project's own demarkus-soul vs the eventual enterprise knowledge systems.
• /thoughts.md § "On the Protocol as the Permanent Layer": principle this plan is built on.
• Future browsable REST surface plan: additive. The MCP gateway is on /mcp; a REST surface would live on /v1/... paths on the same listener, sharing OAuth auth and session cache.
• Future mTLS broker → world plan: anchors here. The worldPool can grow mTLS in place without changing the tool surface.

Resume Steps (when starting work)

1. PR5 (#141) confirmed on main (6c1c354).
2. mark_fetch /plans/broker-https-gateway.md (this doc).
3. Open Questions: only #2-#8 remain (lean answers documented).
4. Execute Pre-Flight 0 (hoist) as its own small PR. Existing tests pass after the relocation.
5. Execute Pre-Flight 1 (mcp-go transport spike). Output: one-paragraph note in the journal.
6. Cut branch feat-tools-broker-mcp-gateway-foundation. Start at Slice 1.
7. Each slice is its own PR. go test -race + bash pre-commit.sh green before moving on.
8. Journal at session end. Update Implementation Status (below) as slices land.

Done When

• All eight slices merged (plus Pre-Flight 0).
• go test -race ./... green inside tools/demarkus-broker/ and client/.
• helm unittest . green for the broker chart.
• pre-commit.sh green.
• Manual end-to-end: user runs /knowledge-join https://broker.example.com in Claude Code → OAuth device flow completes → plugin sees all 13 demarkus tools → mark_fetch mark://team-a/foo.md returns the world's content; cached world-token reused on the next call; broker logs show only one mint per (subject, world).
• Operator-facing README documents the deployment.
• Journal entry covering any design decisions that landed differently from this plan.

Implementation Status

Not started. Plan v3 published 2026-05-20:

• v1: REST HTTP API
• v2: pivoted to MCP-server architecture
• v3: URL shape resolved (mark://{worldName}/{path}), PR6 canceled, /knowledge-join plugin slash command becomes Slice 8

Awaiting kick-off. Pre-Flight 0 (hoist client/internal/fetch) begins next, as its own PR.