# Plan: Broker MCP Gateway (Knowledge-System Layer) > **Plan rewritten 2026-05-20.** v1 described a REST HTTP API. After > alignment with Fritz, the architectural target shifted: the gateway > is an **MCP server** exposed at a single `/mcp` endpoint, authenticated > against the company SSO (id_token bearer), with world access-tokens > cached broker-side per session. Agent-as-client is the only first-pass > consumer. A browser/REST surface is a later, additive plan if a real > consumer asks. > > **v3 (2026-05-20 same session):** Open Question 1 resolved > (`mark://{worldName}/{path}` with cluster-internal DNS resolution). > PR6 (`tools/demarkus-join` binary) canceled outright; the join flow > becomes the `/knowledge-join` slash command in `plugins/claude-code/`, > distinct from a future `/soul-join` (personal direct-QUIC). Vocabulary > pin: **knowledge system** = organizational brokered universe; > **soul** = personal, direct-QUIC. Different commands, different > deployment shapes, shared protocol underneath. > > **v4 (2026-05-20 same session, post-Slice-1):** "Opt-in deployment" > Non-Negotiable removed — every broker is a knowledge-system gateway; > MCP is part of the binary, not a feature flag. `MCPConfig.Addr` > defaults to `:8081` so pre-gateway YAMLs upgrade silently. New > § "Architectural framing (enterprise shape)" pins the > HTTP-gateway-with-OIDC/session-state framing and the byte-for-byte > proxy contract: the broker transports markdown unchanged from the > world's stdio/QUIC surface — only the wire shape (HTTP/JSON-RPC) is > new. Slice 1 shipped (foundation: listener + initialize + tools/list). > > **v5 (2026-05-21, pre-Slice-2):** Session cache is **email-keyed**, > not subject-hash-keyed. The broker's existing identity primitive > (PR4 `Issuer`, `/me/install`, `/tokens` list/revoke, `AllowConfig`, > audit log) is canonical verified email. Slice 2's `sessionCache` > must use the same key so the broker has ONE identity dimension, not > two. Concretely: the cache key is `canonicalEmail` (trimmed, > lowercased) and `MintFiltered` is called with `Claims{Email, > EmailVerified, Groups}` — same shape every other broker surface > already uses. Unverified-email id_tokens are rejected at the > gatewayAuth boundary (matches `ErrEmailUnverified` semantics in > `Issuer.Mint`). Subject claim is still available from the id_token > for log correlation if useful, but is NOT the cache key. Lean > answer on Open Question 4 updated accordingly. > > **v6 (2026-05-22, post-Slice-7):** Slices 2–7 all shipped. Only > Slice 8 (`/knowledge-join` plugin slash command) remains. Plan vs > code drift discovered + documented during Slice 7: `WorldTokenTTL` > + `WorldPool{}` fields the plan listed under `MCPConfig` were > never implemented (Slice 2 went with natural expiry from > `MintFiltered` per OQ#5's lean answer; `worldPool` lifecycle is > internal to `Server.MCPGateway()`, not chart-configurable). Slice 7 > dropped both knobs from `values.yaml` rather than ship chart fields > the broker silently ignores. Ingress topology for the MCP gateway > pinned to a SEPARATE hostname (`ingress.mcp.host`) — not > path-routing on the management host — to avoid future `.well-known/*` > collisions between OIDC discovery and OAuth resource metadata. The > `--with-mcp-smoke` kind-harness stage ships three lightweight > checks (RFC 9728 + RFC 8414 metadata fetch, POST /mcp 401 + > WWW-Authenticate); the full id_token / initialize / tools/call > dance is deferred to Slice 8 where `/knowledge-join` will exercise > it naturally. The enterprise-deployment piece. Adds a remote MCP server to the broker that exposes the demarkus tool surface (mark_fetch, mark_publish, etc.) to plugin-style agents over the company's existing SSO. Worlds stay cluster-internal; the broker becomes the single MCP endpoint the plugin connects to; corporate networks that block QUIC over UDP reach the universe over standard HTTPS. The plugin's local vault holds only the id_token (the company SSO bearer); world access-tokens are minted lazily broker-side per session, scoped to the authenticated user, never persisted to disk. Architectural framing in `/thoughts.md` § "On the Protocol as the Permanent Layer": demarkus core is the permanent layer; this gateway is a knowledge-system overlay that the protocol doesn't know exists. See also `feedback_core_vs_knowledge_layer.md` in agent auto-memory. ## Why this is the priority Stated 2026-05-20: "this has to be the next task we do, it is the missing piece that the company needs." Two enterprise blockers dissolve together: - **"20 dev teams × N worlds plugin-config grind."** Without the gateway, every developer at a 20-team org installs N local MCP servers per workstation, one per world they need. With the gateway, every developer adds ONE MCP server entry (the broker URL) and the broker exposes the universe's worlds as a single tool surface. - **Corporate-network UDP blocking.** QUIC over UDP gets blocked by enterprise firewalls, transparent proxies, and DPI appliances. HTTPS over 443 works everywhere. The gateway speaks HTTPS publicly and translates to QUIC internally. Universe-onboarding's PR6/7/8 are reshaped by this work: PR6 (`tools/demarkus-join` binary) is canceled. The join flow folds into `/knowledge-join` in `plugins/claude-code/`. PR7 becomes "plugin slash command implementation" and PR8 is documentation. ## Architectural framing (enterprise shape) The broker is the HTTP gateway that fronts the knowledge system. From the enterprise side it slots into the same plumbing every other internal SaaS app uses; from the demarkus side it's a thin proxy that hands markdown back to the agent verbatim. Three responsibilities, in this order: 1. **Termination at the org boundary.** The gateway speaks HTTPS + JSON-RPC over Streamable HTTP at `/mcp`. Corporate proxies, Ingress, DPI appliances, TLS-terminating LBs — all the standard enterprise plumbing — pass it through unchanged. QUIC stays *inside* the cluster, where firewalls don't see it. 2. **Identity + session state.** The agent presents `Authorization: Bearer ` (the company SSO bearer). The gateway validates via the existing PR4 `compositeVerifier`, extracts the `email` and `email_verified` claims (rejecting unverified emails, matching `Issuer.ErrEmailUnverified`), canonicalizes the email (trim + lowercase) and keys a per-session in-memory map of world access-tokens off that canonical email. World tokens are minted lazily on first cache miss via `Issuer.MintFiltered(ctx, Claims{Email, EmailVerified, Groups}, keep)`; never persisted; broker restart drops them and the next tool call re-mints. The same canonical email across multiple plugin installs or devices shares one session — familiar mental model from SAML/OIDC apps. Subject claim remains in the id_token and is available for log correlation but is NOT the cache key — the broker has exactly one identity dimension (email) across MCP, `/me/install`, `/tokens`, audit logs, and `AllowConfig`. 3. **Fan-out to worlds over `mark://` (QUIC).** Tool URLs carry the worldName as host: `mark://{worldName}/{path}`. The gateway resolves the name to a cluster-internal Service DNS address (`..svc.cluster.local:6309` by default, overridable via `worlds[].internalAddress`) and dispatches via `client/fetch.Client` with the cached world token. The world server is unchanged — it sees a demarkus protocol request carrying a valid token and serves it. The world does not know the broker is the caller. ### The proxy contract: byte-for-byte markdown **The gateway transports markdown unchanged.** A `mark_fetch` call through the broker returns the *same* `Result` struct — same body bytes, same `version`, same `modified`, same `etag`, same `content-hash`, same metadata keys — that the local `client/cmd/demarkus-mcp` (stdio) returns from a direct-QUIC connection to the same world. The gateway is not a transformation layer; it's a wire-shape adapter. Implications: - A document fetched through the broker and the same document fetched direct-QUIC produce identical content hashes. Federation, mark_resolve, and any downstream content-addressing assumptions stay intact. - The 13 tool responses retain whatever format the existing demarkus-mcp emits. Slice 2-5 handlers map `fetch.Result` to the MCP `CallToolResult` in the same shape the local server uses; no new framing, no enrichment, no field renaming. - Future world-side features (new metadata, new operations) propagate through the gateway without gateway-side code changes — the gateway forwards what the world emits. This is the difference between a *transport bridge* and a *protocol adapter*. The gateway is the former. ### Enterprise affordances that fall out for free - **Single audit point.** Every tool call passes through one handler with the email claim already extracted. The broker's existing `slog` setup picks up `email=alice@acme.com` correlation without per-route plumbing (same field every other broker surface logs). - **One MCP server per knowledge system, not N per world.** The plugin's `claude mcp add` only ever holds one entry per org the user joined; the broker exposes the org's worlds collectively. - **Rate-limiting at the gate.** Per-email buckets shared across `/tokens` and `/mcp` (same identity = same bucket). Abuse from one identity can't fan out across surfaces. - **NetworkPolicy + RBAC bound the broker→world surface.** Worlds stay reachable only from broker pods inside the cluster; mTLS broker→world is a hardening pass for later but the trust model already works without it because the network plane is itself bounded. ### What this is NOT The gateway is an *additive overlay* for the agent-shaped consumer, not a replacement for the direct-QUIC surface. The CLI, Obsidian plugin, kind-harness, and in-cluster server-to-server traffic all keep talking direct-QUIC `mark://`. Two consumers, two transports, same protocol underneath. ## Vocabulary | Term | Meaning | | --- | --- | | **knowledge system** | An organizational, broker-fronted universe. Joined via `/knowledge-join`. MCP traffic terminates at the broker over HTTPS; broker translates to QUIC and routes to internal worlds. The deployment shape this plan ships. | | **soul** | A personal demarkus knowledge base — direct-QUIC, no broker required. Joined via a future `/soul-join` if needed. The original demarkus deployment shape. | | **world** | A demarkus server (QUIC). Both knowledge systems and souls compose one or more worlds. | | **universe** | An operator-managed bundle of worlds + (for knowledge systems) a broker. | A single Claude Code installation can have BOTH a knowledge system (the org's) and one or more souls (personal). They don't conflict — different commands, different MCP server entries, different auth modes. ## Non-Negotiables - **No protocol changes.** The demarkus message format (FETCH/PUBLISH/APPEND/VERSIONS/LIST/ARCHIVE/MERGE/etc.), content addressing, versioning, and capability-token mechanics are unchanged. - **No demarkus-server changes.** Worlds keep their current QUIC + `mark` ALPN listener. They see a bearer token in the demarkus protocol header (the world access-token the broker is holding on the user's behalf) and serve it. They do not know the broker is the caller. - **Byte-for-byte markdown proxy.** The gateway transports document content unchanged. `mark_fetch` through the broker returns the same body, version, etag, content-hash, and metadata keys as `mark_fetch` against the same world via stdio/direct-QUIC. The gateway adapts the *wire shape* (HTTP/JSON-RPC ↔ QUIC/demarkus), not the *content*. See § "Architectural framing" above. - **Capabilities stay on worlds.** The world is the trust boundary for what a token can do. The broker holds world access-tokens in trust per-authenticated-user; the world validates them on every request. The broker does NOT re-implement capability checks. - **Single identity dimension: canonical verified email.** The MCP session cache, `/me/install`, `/tokens` list/revoke, `AllowConfig`, and audit logs all key on the same canonical email (trimmed + lowercased). Unverified-email id_tokens are rejected at the gatewayAuth boundary (matches `Issuer.ErrEmailUnverified`). Subject claim is available for log correlation but is not a separate keying dimension. *(Added in v5 — see top-of-doc changelog.)* - **Tool surface parity with the local MCP server.** All 13 tools the existing `client/cmd/demarkus-mcp` exposes (mark_fetch, mark_list, mark_versions, mark_publish, mark_append, mark_archive, mark_discover, mark_resolve, mark_index, mark_backlinks, mark_graph, mark_graph_export, mark_graph_publish) are exposed by the broker MCP from day one. Anything else is a plugin-side change, not a gateway concern. - **Gateway is always on.** Every broker is a knowledge-system gateway; MCP is part of the binary, not a feature flag. `MCPConfig.Addr` defaults to `:8081` when omitted so pre-gateway YAMLs upgrade silently. *(Replaces v1-v3's "Opt-in deployment" Non-Negotiable, reverted post-Slice-1 — the universe-onboarding-only broker shape is not a product we ship.)* - **No MVPs, no shortcuts.** OAuth correctness, per-session world-token cache lifecycle (lazy mint + eviction on id_token expiry + cap on token churn), structured error envelopes, content-hash propagation through the tool surface, conflict-aware merge — all from day one within each slice (not later). - **QUIC stays first-class for direct-world access.** This gateway is an enterprise overlay, not a transport replacement. The CLI, Obsidian plugin, and in-cluster server-to-server traffic all keep direct-QUIC access. The broker's existing /auth, /me/install, /tokens surfaces all stay as they are. ## Out of Scope (explicit) - **Browsable REST surface.** Deferred until a real non-agent consumer asks. The plugin is the only first-pass consumer; the agent is always the client. - **`tools/demarkus-join` binary.** Canceled (see v3 changelog above). The join flow is a slash command, not a binary. - **mTLS broker → world.** Confirmed deferred to a hardening pass. TLS terminates at the broker; broker→world QUIC traffic stays inside the cluster, bounded by NetworkPolicy + RBAC. - **Persistent world-token storage.** Raw world tokens are NEVER persisted to disk by the broker. They live in process memory per session; broker restart drops them; next tool call re-mints. This matches the broker's existing posture (raw tokens are never recoverable from the issuances Secret). - **MCP `resources` and `prompts` features.** Only the `tools` capability is exposed. Resources (file-style refs) and prompts (templates) are not part of the demarkus model. - **Long-lived subscriptions / server-initiated events.** Polling is the model (`/thoughts.md` § "On Subscriptions and Polling"). - **Multi-broker / multi-universe.** Single broker per universe stays the model. - **Content negotiation beyond `text/markdown`.** Same scope contract as the protocol (`project_scope_markdown_only.md`). - **Content transformation at the gateway.** The byte-for-byte proxy contract excludes any kind of body rewriting, frontmatter normalization, link adjustment, or hash-stripping at the broker. The gateway forwards what the world emits. - **Subject-claim-based session keying.** The session cache keys on canonical email, not the OIDC subject claim. *(Pinned in v5 — see Non-Negotiables.)* - **Persistent broker-side graph store.** Slice 4b+5 shipped the graph-store tools backed by an in-memory, pod-scoped store. Re-crawl after broker restart is the documented operator expectation. Bucket-store-backed persistence is parked for the post-broker design window (`/thoughts.md` § "On Bucket Stores"). *(Pinned post-Slice-4b — graph-store ephemerality is a deliberate trade-off, not a debt item.)* ## Architecture ``` ┌──────────────┐ HTTPS (MCP) ┌──────────────────────────────────┐ QUIC ┌─────────┐ │ Claude Code │ ─────────────► │ broker (/mcp endpoint) │ ─────► │ world A │ │ plugin │ │ │ │ (cluster│ │ (or any MCP │ ◄───────────── │ JSON-RPC over Streamable HTTP │ ◄───── │ internal) │ client) │ tool result │ │ └─────────┘ └──────────────┘ │ Authorization: Bearer │ │ - requireAuth (PR4 compositeV) │ ┌─────────┐ │ - reject unverified-email tokens │ ─────► │ world B │ │ - per-email rate limit │ ◄───── │ │ │ │ └─────────┘ │ Per-session world-token cache: │ │ - key: (canonicalEmail, │ │ world_name) │ │ - mint lazily via Issuer.Mint- │ │ Filtered on cache miss │ │ - evict on id_token / token expiry│ │ │ │ Tool URLs: mark://{worldName}/ │ │ {path} — broker resolves worldName│ │ to .svc.cluster.local:6309 │ │ via cfg.Worlds[].Namespace │ │ │ │ 13 tools (parity with local │ │ demarkus-mcp): mark_fetch, ... │ │ │ │ OAuth metadata: │ │ /.well-known/oauth-protected- │ │ resource (RFC 9728) │ │ /.well-known/oauth-authorization- │ │ server (RFC 8414) │ └──────────────────────────────────┘ ``` ### Layer responsibilities | Component | Owns | Does NOT own | | --- | --- | --- | | `mcpGateway` HTTP listener | Listener / TLS / single `/mcp` endpoint dispatch. Streamable HTTP transport per current MCP spec. Separate from existing management API listener. | Tool semantics, auth, session state. | | MCP protocol layer | Handling `initialize`, `tools/list`, `tools/call` JSON-RPC methods. Capability negotiation. MCP error envelope. | demarkus protocol semantics. | | `gatewayAuth` middleware | Validating the id_token bearer via existing PR4 `compositeVerifier` (broker-signed OR IdP-signed). Email + email_verified extraction; rejecting unverified-email tokens; canonicalizing the email (trim + lowercase) for session keying. Per-email rate limiting. RFC 6750 + RFC 9728 WWW-Authenticate challenge on 401. | World-level capability enforcement. | | `sessionCache` | Per-session in-memory map keyed by canonical verified email. Holds cached world access-tokens (raw, never persisted). Lazy-mint via `Issuer.MintFiltered(ctx, Claims{Email, EmailVerified, Groups}, keep)`. Eviction on id_token expiry, world-token expiry, or LRU cap. Singleflight on concurrent (email, world) mints. | Auth, tool dispatch. | | Per-tool handlers | One per MCP tool. Parses tool args (URL form: `mark://{worldName}/{path}`), resolves the addressed world via `cfg.Worlds` by name, fetches/mints the world token from `sessionCache`, dispatches via `fetch.Client` / `merge.Candidate`. Surfaces errors via MCP error envelope. **Forwards world response content unchanged** per the byte-for-byte proxy contract. | Transport, connection pool, session state. | | `worldPool` | Per-world `fetch.Client` reuse keyed by worldName. Resolves the internal address from `cfg.Worlds[].Namespace` (e.g. `..svc.cluster.local:6309`). Connection lifecycle. Reconnect on transport failure. Bounded parallelism per world. | Auth, tool semantics. | | `cfg.Server.MCP` | MCP-gateway-specific config: `Addr` (defaulted to `:8081`), `TLS{CertFile, KeyFile}`, `SessionMaxIdle`, `MaxSessions`, `FirstMintMaxAttempts`, `FirstMintInitialBackoff`, `FirstMintMaxBackoff`. Validated at LoadConfig. *(Note: `WorldTokenTTL` and `WorldPool{}` listed in earlier plan drafts never landed — Slice 2 went with the natural token expiry from `MintFiltered` per OQ#5; `worldPool` lifecycle is internal to `Server.MCPGateway()`, not chart-configurable. Both knobs dropped from `values.yaml` during Slice 7 — see v6 changelog.)* | Wire format. | ### World name → internal address resolution Tool URLs have the form `mark://{worldName}/{path}`. The broker resolves `worldName` to a cluster-internal address using the existing `cfg.Worlds[].Namespace` field. Default resolution: ``` mark://team-a/foo.md → mark://team-a.team-a.svc.cluster.local:6309/foo.md ^^^^^^ ^^^^^^ name namespace (from cfg.Worlds[].Namespace) ``` This is the standard Kubernetes Service DNS pattern. The world's Service must be named the same as the world (default for the demarkus-server chart). If an operator deploys a world with a non-default Service name, they configure `cfg.Worlds[].InternalAddress` explicitly — overrides the default DNS pattern. Default empty means "use the convention." ## Pre-Flight Tasks (one PR ahead of Slice 1) ### Pre-Flight 0 — Hoist `client/internal/fetch` to public - `client/internal/fetch/fetch.go` → `client/fetch/fetch.go` (or `client/pkg/fetch/fetch.go`). Mirror for `client/internal/merge` since MERGE support lands in MVP via the conflict-aware `mark_publish` tool. - Update import paths in `client/cmd/demarkus-mcp`, `client/cmd/demarkus`, `client/cmd/demarkus-tui` (and any other consumers within the client module). - Same content; just relocated. Surface unchanged. - Why: Go's internal-package rule blocks any module outside `client/` from importing `client/internal/...`. The broker is in `tools/demarkus-broker` (different module under monorepo replace-directives), so it can't import the existing client library while it's internal. - This is a CLIENT-module API change, not a protocol-core change. Not gated by `feedback_plugin_scope.md` (broker isn't a plugin), but worth flagging explicitly so reviewers see the architectural shift. ### Pre-Flight 1 — Verify mark3labs/mcp-go's Streamable HTTP support - The existing `client/cmd/demarkus-mcp` uses mcp-go's stdio transport (`mcpserver.ServeStdio()`). Confirm mcp-go v0.44+ supports Streamable HTTP (the current MCP spec's preferred remote transport). - If yes: lifecycle the new gateway around mcp-go's HTTP server constructor. - If no: upgrade to a version that does, swap to a different MCP Go library, or implement Streamable HTTP transport in-house (worst case). - Output: one-paragraph journal note on transport choice before Slice 1 begins. ## Sub-Tasks (sequenced; one slice per PR) ### Slice 1 — Foundation: listener + initialize + tools/list (no tool implementations) - `tools/demarkus-broker/internal/broker/mcp_gateway.go` (new) — `mcpGateway` listener, JSON-RPC dispatcher, Streamable HTTP transport. Separate listener from the existing management API. - `tools/demarkus-broker/internal/broker/mcp_oauth.go` (new) — OAuth metadata endpoints: `/.well-known/oauth-protected-resource` (RFC 9728), `/.well-known/oauth-authorization-server` (RFC 8414). Reuses broker's existing `Discovery` machinery. - `tools/demarkus-broker/internal/broker/mcp_auth.go` (new) — `gatewayAuth` middleware. id_token bearer extraction, validation via existing `compositeVerifier`, claim extraction for session keying, per-identity rate limiting (shared budget with /tokens routes — same identity, one bucket). - `tools/demarkus-broker/internal/broker/config.go` — `ServerConfig.MCP` substruct: `Addr` (defaulted to `:8081`), `TLS{...}`. Slice 2+ adds `SessionMaxIdle` and `WorldTokenTTL`. Plus `WorldConfig.InternalAddress` (optional override for the default Service-DNS resolution). - `tools/demarkus-broker/main.go` — start the MCP listener alongside the management API. Always on. - `tools/demarkus-broker/internal/broker/mcp_initialize.go` (new) — handles MCP `initialize` JSON-RPC method. Advertises server capabilities (tools only). - `tools/demarkus-broker/internal/broker/mcp_tools_list.go` (new) — handles `tools/list`. Returns the 13 tool definitions (names + JSON schemas) without implementations yet. Definitions mirror `client/cmd/demarkus-mcp` exactly (URL hint adapted to `mark://{worldName}/{path}`). - Tests: OAuth metadata round-trip, initialize handshake, tools/list returns the 13 expected tool names, unauth requests rejected with 401 + WWW-Authenticate header pointing at auth-server metadata URL, expired/revoked bearer rejected, per-identity rate limit triggers 429. - **Note (v5 retro):** Slice 1 landed before v5 pinned the email-keying decision. Any subject-keyed test or middleware label that shipped will be rewritten against canonical email in Slice 2. The middleware additionally rejects unverified-email tokens starting in Slice 2. ### Slice 2 — Read tools + session cache foundation: mark_fetch, mark_list, mark_versions - `tools/demarkus-broker/internal/broker/mcp_session.go` (new) — `sessionCache` keyed by canonical verified email. Holds `map[worldName]cachedWorldToken{raw, expiresAt}`. Lazy initialization on first cache miss. LRU cap (default 10000 emails) + idle eviction (`SessionMaxIdle` default 1h after last use). Singleflight on concurrent (email, world) mints so a burst of tool calls for the same identity+world coalesces to one `MintFiltered` call. - `tools/demarkus-broker/internal/broker/world_pool.go` (new) — `worldPool` for `fetch.Client` reuse per world. Resolves worldName → internal address via the rule documented above. - `tools/demarkus-broker/internal/broker/mcp_tools_read.go` (new) — handlers for `mark_fetch`, `mark_list`, `mark_versions`. URL parameter shape: `mark://{worldName}/{path}`. Resolves world by name; lazy-mints world access-token via `Issuer.MintFiltered(ctx, Claims{Email, EmailVerified, Groups}, keep)` on cache miss; calls `fetch.Client.Fetch/List/Versions` with the token; maps the demarkus Result to an MCP tool response **byte-for-byte from the world's payload** (matching local MCP server output format for parity per the Non-Negotiable). - Extend `gatewayAuth` (from Slice 1) to reject id_tokens whose `email_verified` claim is false or missing — match `Issuer.ErrEmailUnverified` semantics so the broker has one identity gate, not two. - Tests: each tool happy path, world-not-found tool error, document-not-found maps to MCP error, expired-bearer 401 mid-call, cache hit reuses token, cache miss mints, two concurrent calls for the same (email, world) coalesce to one mint (singleflight), session eviction on id_token expiry causes re-mint, unverified-email id_token rejected at gateway with the same shape as `ErrEmailUnverified`, **proxy-fidelity test: same document fetched through broker MCP vs direct-QUIC produces identical body + content-hash + metadata**. ### Slice 3 — Write tools: mark_publish, mark_append, mark_archive - `tools/demarkus-broker/internal/broker/mcp_tools_write.go` (new) — handlers for the three write ops. - `mark_publish` reuses the existing `expected_version` + `on_conflict` shape from the local MCP server. Conflict-aware merge candidate flow lands in Slice 6; for Slice 3, on-conflict defaults to "fail" with the conflict envelope, NOT the merge-candidate path. - `mark_append` reuses the auto-resolved-version pattern (omit `expected_version`, broker calls VERSIONS internally) — matches local MCP server behavior. - `mark_archive` deletes by archiving (per the demarkus protocol's actual DELETE shape; the protocol op is ARCHIVE, not DELETE). - Tests: happy paths, version-mismatch conflict, missing-expected_version on PUBLISH 400, auto-resolve path for APPEND, world-side RBAC failure surfaces as MCP tool error. ### Slice 4 — Federation read tools: mark_discover, mark_resolve, mark_backlinks, mark_graph - `tools/demarkus-broker/internal/broker/mcp_tools_federation.go` (new) — handlers that delegate to whichever demarkus client library functions back the local MCP server's federation tools (mapped during a second-pass spike at start of Slice 4, deferred until Pre-Flight 0 hoist completes and the public API is visible). - These tools are read-only and don't change the session-cache shape. - Tests: happy paths + edge cases per tool. ### Slice 5 — Federation write tools: mark_index, mark_graph_export, mark_graph_publish - `tools/demarkus-broker/internal/broker/mcp_tools_federation_write.go` (new). - Same shape as Slice 3 for writes (expected_version, conflict handling). - Tests: happy paths + conflict cases. ### Slice 6 — Conflict-aware merge in `mark_publish` - Reuses `client/merge` (hoisted in Pre-Flight 0). - Adds the `on_conflict: "merge"` branch to `mark_publish`, matching local MCP server behavior. - Tests: clean merge, structural merge, conflict-markers returned. ### Slice 7 — Chart, RBAC, docs - `deploy/helm/demarkus-broker/values.yaml` — `server.mcp.addr` (defaults to `:8081`), `server.mcp.tls` (`existingSecretRef` recommended), `server.mcp.sessionMaxIdle`, `server.mcp.maxSessions`, `server.mcp.firstMint{MaxAttempts,InitialBackoff,MaxBackoff}`. Plus `worlds[].internalAddress` for non-default Service DNS overrides. **No `enabled` knob** — the gateway is part of the binary. `worldTokenTTL` + `worldPool` dropped from earlier plan drafts (binary never had the fields — see v6 changelog). - Templates: deployment.yaml (new containerPort + optional TLS mount), service.yaml (gateway port exposed), ingress.yaml (route mcp host to new port via SEPARATE hostname `ingress.mcp.host`, not path-routing), networkpolicy.yaml (allow ingress on mcp port), certificate.yaml (parallel cert-manager Certificate for the MCP host). - No new RBAC needed — broker SA already has perms for issuances Secret + world Secrets that lazy-mint touches. - `deploy/helm/demarkus-broker/README.md` — new "MCP gateway" section. TLS setup, the plugin-side flow (`/knowledge-join` from Claude Code), OAuth flow, rate-limit behavior, **ephemeral graph-store callout (operators expect re-crawl after restart)**. Upgrade note: pre-gateway deployments get the listener on `:8081` after upgrade; chart values let operators retune. - `tools/demarkus-broker/main.go` package doc — bump Current scope to mention `/mcp`. - New `tools/demarkus-broker/MCP-API.md` — operator/developer-facing spec for the MCP tool surface mirrored from `client/cmd/demarkus-mcp`. Lives in the broker package; not a protocol document. - Optional `--with-mcp-smoke` stage in `deploy/kind/up.sh` — builds broker locally, sideloads into kind, installs the LOCAL chart with the locally-built image, runs three lightweight checks (RFC 9728 + 8414 metadata + 401 challenge). Full id_token / initialize / tools/call E2E deferred to Slice 8 (`/knowledge-join` exercises it naturally). ### Slice 8 — Plugin `/knowledge-join` slash command (replaces canceled PR6) - `plugins/claude-code/commands/knowledge-join.md` (new, per Claude Code plugin slash-command shape) — prompt-shaped command that takes a broker URL argument. - Behavior: 1. Validates the broker URL by issuing `HEAD /.well-known/oauth-protected-resource`. 2. Derives a slug from the broker hostname (e.g., `acme` from `broker.acme.com`). 3. Runs `claude mcp add --transport http {slug} {url}/mcp`. Claude Code handles the OAuth device flow when the plugin first talks to the MCP server. 4. Reports success + slug to the user; on failure shows the validation error. - `plugins/claude-code/scripts/lib.sh` — bump SERVER/CLIENT/TOOLS_VERSION pins per `feedback_plugin_version_pins.md`. - Tests: shell-script tests in `plugins/claude-code/tests/` covering happy path (mock broker URL), validation-fail path, slug-derivation cases. ## Scope estimate | Slice | Production code | Tests | Chart / docs | | --- | ---: | ---: | ---: | | Pre-Flight 0 (hoist) | ~50 (file moves + import updates) | — | — | | Pre-Flight 1 (mcp-go spike) | 0 | 0 | — | | 1. Foundation + initialize + tools/list | ~500 | ~600 | — | | 2. Read tools + session cache | ~400 | ~500 | — | | 3. Write tools | ~250 | ~400 | — | | 4. Federation read tools | ~200 | ~300 | — | | 5. Federation write tools | ~150 | ~250 | — | | 6. Conflict-aware merge | ~150 | ~250 | — | | 7. Chart + RBAC + docs | ~50 | ~80 (helm-unittest) | ~300 | | 8. Plugin `/knowledge-join` | ~50 | ~80 | — | | **Total** | **~1800** | **~2460** | **~300** | Eight slices (plus Pre-Flight) across ~2-3 working weeks. Slice 8 is small but ships separately so the broker side can land + bake before plugin work touches it. ## Open Questions To Resolve Before/During Implementation 1. ~~**URL shape inside MCP tool args.**~~ **Resolved 2026-05-20**: `mark://{worldName}/{path}` — broker resolves worldName to cluster-internal Service DNS (`..svc.cluster.local:6309`); `worlds[].internalAddress` overrides the default for non-conforming deployments. 2. **MCP transport: Streamable HTTP vs SSE.** **Resolved 2026-05-20 (Pre-Flight 1)**: Streamable HTTP via `mcp-go.NewStreamableHTTPServer`. Stream Resumability not supported in mcp-go v0.44 — irrelevant since the plan's Out-of-Scope already excludes long-lived subscriptions. 3. **OAuth authorization-server identity.** The broker is BOTH the MCP server (resource) AND the authorization server (the existing OIDC + device-flow surface). Lean: broker advertises itself as the authorization server via `/.well-known/oauth-authorization-server`. The actual IdP (Google, Okta, etc.) is one hop further — handled by existing broker OIDC machinery. Matches PR3 device-flow architecture. **Slice 1 lands this as an alias of the OIDC discovery handler** — RFC 8414 §3 tolerates the OIDC body's extra fields, and the iss-override Discovery already applies makes it usable as auth-server metadata. 4. ~~**Session keying.**~~ **Resolved 2026-05-21 (v5)**: canonical verified email (trim + lowercase, `email_verified=true` required). Same key the broker already uses for `/me/install`, `/tokens` list/revoke, `AllowConfig`, and audit logs. Multiple devices for the same canonical email share one session. Subject claim remains in the id_token for log correlation but is not the cache key. 5. ~~**World-token TTL and minting cadence.**~~ **Resolved (Slice 2)**: cache world-tokens with their natural expiry from `MintFiltered` — broker respects the world's own TTL (`worlds[].defaultToken.expiresAfter`) rather than overriding broker-side. On id_token rotation (refresh-grant), session_cache survives because canonical email is stable across refreshes. No chart-side `worldTokenTTL` knob (would silently no-op). 6. ~~**Federation tool implementations.**~~ **Resolved (Slices 4a + 4b+5)**: `client/index`, `client/graphstore`, `client/graph`, `client/links` hoisted (Pre-Flight 0-style hoists) to make them importable from the broker. Federation tools delegate to those packages. 7. **MCP error envelope for partial mint failures.** When the broker mints a world token on first call, partial-mint surfaces as ONE successful tool response + server log entry, NOT a partial-mint envelope. 8. **MCP session lifecycle on broker restart.** Lean: in-memory only; broker restart drops all sessions; plugin re-authenticates via the existing OAuth refresh path (PR4 broker-signed refresh tokens). Persistent session storage deferred. 9. ~~**First-mint dispatch latency / Secret-propagation race.**~~ **Resolved 2026-05-21 (Slice 2)**: retry-on-401-after-mint with exponential backoff inside the broker handler. Defaults: 6 attempts, 250ms → 8s exponential (~16s total worst-case waiting). Configurable via `cfg.Server.MCP.FirstMintMax*`. Distinguishes fresh-mint 401s (retry) from cache-hit 401s (invalidate + immediate re-mint). Sync-wait + prewarm options rejected — see journal 2026-05-21. ## Risks Specific To The MCP Gateway - **MCP protocol surface gains an internet-reachable port.** Same risk shape as the original REST plan: hardened auth middleware (existing compositeVerifier), strict per-identity rate limit, structured MCP error envelope, TLS-only in production. - **World-token cache memory growth.** A broker serving 10k emails × 5 worlds × ~200 bytes per cached token entry = ~10MB. Bounded but not zero. Mitigation: LRU cap, idle eviction (default 1h after last tool call), per-process memory metric for ops dashboards. - **Mint storm on broker restart.** Every active plugin reconnects after a restart and re-mints tokens for every world it touches. Mitigation: `mutateSecret`'s optimistic-concurrency retry handles contention; sharded issuances Secret if a real customer hits the wall (existing Phase 7 path). - **SIGHUP cadence on worlds.** Each first-use mint per (email, world) triggers a SIGHUP on the addressed world. The reload itself is cheap (atomic pointer swap behind sync.RWMutex; no connection drop), but the upstream Secret-propagation lag (Open Question 9) shapes the perceived latency. Mitigation: singleflight on (email, world) coalesces concurrent first-call bursts to one mint; the retry-on-401 loop (OQ#9 resolution) absorbs the residual. - **Issuance bloat under high session churn.** Plugin sessions that come and go faster than the world-token TTL leak issuance records into the Secret. Sweeper retires them on expiry. - **Plugin holds N world identities through one broker connection.** Single MCP-server-for-the-universe architecture means a buggy broker handler could expose one user's session to another's bearer. Mitigation: strict canonical-email keying on session cache; every tool call re-validates the bearer against the cache key; integration test that proves cross-email isolation under concurrent load. - **mcp-go library maturity.** Third-party Go MCP library; if it lacks Streamable HTTP, the choice in Pre-Flight 1 cascades. **Resolved** — v0.44 ships `NewStreamableHTTPServer` as a production `http.Handler`. - **OAuth metadata vs OIDC metadata overlap.** Broker already serves `/.well-known/openid-configuration`. RFC 8414 (OAuth) and OIDC discovery have largely-overlapping shapes; serving both means two endpoints that mostly mirror each other. Mitigation: Slice 1 aliases auth-server to the existing Discovery handler — one renderer, two routes. **Slice 7 follow-on**: splitting the management and MCP surfaces onto separate hostnames (`ingress.host` vs `ingress.mcp.host`) also eliminates any future `.well-known/*` collision risk by giving each surface its own host's namespace. - **Plugin slash-command surface name collision.** `/knowledge-join` is new; verify no other plugin in the marketplace uses it. Mitigation: the slash command is namespaced to the demarkus-memory plugin; slot the command into the plugin's `commands/` directory and Claude Code's resolution handles namespacing. - **Always-on listener forces a port assignment on every deployment.** With the v4 "gateway is always on" revision, every broker binds the second listener whether the operator configured it or not. Mitigation: `defaultMCPAddr` = `:8081` picked to avoid the management API's typical `:8080`; chart's values.yaml surfaces the override; kind-harness and dev configs that don't want the listener route the port to nowhere (no service, no ingress) rather than disabling it via config. - **Graph-store ephemerality is a user-visible behavior.** Bucket-store-backed persistence is parked, but until it ships, operators MUST document the re-crawl-after-restart expectation. The chart README's "MCP gateway" section + MCP-API.md's per-tool warning on `mark_backlinks` and friends both call it out prominently. Mitigation deferred to bucket-store work; current mitigation is documentation, not code. ## Touch Points With Adjacent Work - **Universe-onboarding PR5** (shipped #141, 2026-05-20). No code coupling. `/me/install` stays as-is — identity-introspection surface alongside the MCP gateway's operational data plane. **Same identity dimension (canonical email) underneath**, so audit and rate-limit semantics line up across the two surfaces (v5). - **Universe-onboarding PR6 (`tools/demarkus-join`)**: **canceled.** Replaced by Slice 8 (`/knowledge-join` plugin slash command). - **Universe-onboarding PR7/PR8**: substantially reshaped. PR7 (originally "plugin slash commands") may merge into Slice 8; PR8 (originally docs) merges into Slice 7 + Slice 8's docs. - **Future `/soul-join` plugin slash command** (separate plan): direct-QUIC join for personal souls. Distinct from `/knowledge-join` (organizational, broker-fronted). Same vocabulary axis as the project's own `demarkus-soul` vs the eventual enterprise knowledge systems. - **`/thoughts.md` § "On the Protocol as the Permanent Layer"**: principle this plan is built on. - **`/thoughts.md` § "On Bucket Stores"**: post-broker direction for the graph-store persistence question Slice 4b deliberately punted on. - **Future browsable REST surface plan**: additive. The MCP gateway is on `/mcp`; a REST surface would live on `/v1/...` paths on the same listener, sharing OAuth auth and session cache. - **Future mTLS broker → world plan**: anchors here. The worldPool can grow mTLS in place without changing the tool surface. ## Resume Steps (when starting work) 1. Confirm all merged slices on `main` (latest as of 2026-05-22: Slice 7 = `277f83f`). 2. `mark_fetch /plans/broker-https-gateway.md` (this doc). 3. Implementation Status (below) shows what's shipped; only Slice 8 remains. 4. Cut branch for Slice 8 work (`feat-plugin-claude-code-knowledge-join` or similar). The slash command exercises the full id_token + initialize + tools/call dance that Slice 7's smoke deliberately deferred. 5. Bump `plugins/claude-code/scripts/lib.sh` SERVER/CLIENT/TOOLS_VERSION pins per `feedback_plugin_version_pins.md` (Slice 7 was chart-only so no plugin bump was warranted; Slice 8 ships plugin-visible behavior). 6. `go test -race` + `bash pre-commit.sh` green before merge. Plugin shell-script tests added under `plugins/claude-code/tests/`. 7. Journal at session end. Mark Slice 8 as shipped + the whole plan as complete in Implementation Status. ## Done When - All eight slices merged (plus Pre-Flight 0). - `go test -race ./...` green inside `tools/demarkus-broker/` and `client/`. - `helm unittest .` green for the broker chart. - `pre-commit.sh` green. - Manual end-to-end: user runs `/knowledge-join https://broker.example.com` in Claude Code → OAuth device flow completes → plugin sees all 13 demarkus tools → `mark_fetch mark://team-a/foo.md` returns the world's content; cached world-token reused on the next call; broker logs show only one mint per (email, world). **Proxy-fidelity check**: same document fetched via broker MCP and direct-QUIC produces identical body + content-hash. - Operator-facing README documents the deployment. - Journal entry covering any design decisions that landed differently from this plan. ## Implementation Status - **Pre-Flight 0 (hoist `client/internal/fetch` + `merge`)** — shipped 2026-05-20 (#142). 5 files relocated, 9 consumers re-imported. Cleared the path for `tools/demarkus-broker` to import `client/fetch` and `client/merge` across module boundaries. - **Pre-Flight 1 (mcp-go Streamable HTTP spike)** — resolved 2026-05-20. `mcp-go.NewStreamableHTTPServer` is a production `http.Handler` at v0.44; Stream Resumability is the only missing feature and it's already out-of-scope. - **Slice 1 (foundation: listener + initialize + tools/list)** — shipped 2026-05-20 (#143, `b90cda6`). ~660 LOC production, ~798 LOC tests. Always-on listener + 13-tool registration (placeholders) + OAuth metadata + gatewayAuth with RFC 6750+9728 challenge. - **Slice 2 (read tools + session cache)** — shipped 2026-05-21 (#146, `7c529d4`). mark_fetch / mark_list / mark_versions backed by canonical-email-keyed session cache, worldPool, lazy mint via `Issuer.MintFiltered`, singleflight on concurrent first-call bursts, retry-on-401-after-mint with exponential backoff (OQ#9 resolution). Proxy-fidelity test pins byte-for-byte parity with local demarkus-mcp. - **Slice 3 (write tools)** — shipped 2026-05-21 (#147, `5387701`). mark_publish / mark_append / mark_archive. `dispatchWithAuth` refactor (closure-based, shared retry loop across read + write). Conflict + not-permitted forward verbatim (not as tool errors); on_conflict=merge rejected with explicit Slice 6 pointer. - **Slice 4a (federation reads: discover + resolve)** — shipped 2026-05-22 (#148, `115a09b`). mark_discover + mark_resolve. Cross-org candidate skip semantics; `client/internal/index` hoisted to `client/index`. Split off from original Slice 4 scope to keep the graph-store design call (4b) cleanly separated. - **Slice 4b+5 (graph-store federation tools)** — shipped 2026-05-22 (#149, `fa9f86d`). mark_backlinks + mark_graph + mark_index + mark_graph_export + mark_graph_publish backed by an **ephemeral in-memory graph store** (per-pod lifetime, re-crawl after restart). Fritz signed off on the persistence trade-off after the gap framing. `client/graphstore` + `client/graph` + `client/links` hoisted. Brought all 13 tools to real handlers — `notImplementedHandler` remains as a defensive fallback but no advertised tool falls through to it. Bucket-store-backed persistence parked for the post-broker window (`/thoughts.md` § "On Bucket Stores"). - **Slice 6 (conflict-aware merge in mark_publish)** — shipped 2026-05-22 (#150, `80a4008`). `on_conflict="merge"` candidate flow lit up, default flipped from `"fail"` to `"merge"` to match local demarkus-mcp. `brokerMergeAdapter` captures ctx in the struct (only viable shape against `merge.Client`'s ctx-free interface). All 13 tools now have full semantic parity with local demarkus-mcp. - **Slice 7 (chart + RBAC + docs)** — shipped 2026-05-22 (#151, `277f83f`). Chart wiring: `server.mcp.*` block, `worlds[].internalAddress`, deployment second containerPort + optional TLS volume, service `mcp` port, networkpolicy ingress port, ingress.mcp.host (separate-hostname topology), parallel cert-manager Certificate. New README "MCP gateway" section + new `tools/demarkus-broker/MCP-API.md` 13-tool spec. Kind harness `--with-mcp-smoke` stage (builds broker locally, sideloads, installs local chart, runs RFC 9728 + 8414 metadata + 401-challenge checks). 99 helm-unittest cases green. CodeRabbit round-1 fixes: strict-digit `mcpPort` (closes silent `int "abc"`→0 footgun), `ingress.mcp.host` ≠ `ingress.host` guard, numeric port-collision compare (catches `0.0.0.0:8080`), corrected TLS validation comment, MCP-API.md fence + ephemeral-graph wording. Plan-vs-code drift discovered: `WorldTokenTTL` + `WorldPool{}` knobs dropped from values.yaml (binary has no fields). - **Slice 8 (plugin `/knowledge-join` slash command)** — **next.** Small (~50 LOC + tests). The plugin slash command is the user-facing onboarding closer + natural test surface for the full id_token / initialize / tools/call E2E that Slice 7's smoke deferred. History of plan revisions: - v1: REST HTTP API - v2: pivoted to MCP-server architecture - v3: URL shape resolved (`mark://{worldName}/{path}`), PR6 canceled, `/knowledge-join` plugin slash command becomes Slice 8 - v4: gateway is always on (no `enabled` flag); enterprise framing + byte-for-byte proxy semantics pinned as Non-Negotiables - v5: session cache pinned to canonical verified email (single identity dimension across broker); unverified-email id_tokens rejected at gatewayAuth; Open Question 9 (Secret-propagation race) added as Slice-2-blocking decision; SIGHUP-cadence risk + singleflight mitigation documented - v6: Slices 2–7 shipped, only Slice 8 remains. Plan-vs-code drift documented (`WorldTokenTTL` + `WorldPool{}` never implemented; dropped from chart). Ingress topology pinned to separate-hostname split. Graph-store ephemerality added to Out-of-Scope as a deliberate trade-off (not debt). OQ#5, #6, #9 marked resolved.