Plan: Broker HTTPS Gateway (Knowledge-System Layer)

The enterprise-deployment piece. Adds an HTTPS frontend to the broker that translates incoming HTTP/2 requests into demarkus QUIC calls to internal worlds, without touching the demarkus protocol core. Worlds stay cluster-internal; the broker becomes the single public surface; corporate networks that block QUIC over UDP can reach the universe over standard HTTPS. Tokens stay on the plugin; capabilities stay on the worlds; the broker is a transport bridge with identity-aware rate limiting.

Articulated 2026-05-20 after the universe-onboarding PR5 implementation. See /thoughts.md § "On the Protocol as the Permanent Layer" for the architectural framing: demarkus core is the permanent layer; the gateway is a knowledge-system overlay built on top, throwable-away without affecting the protocol. See also feedback_core_vs_knowledge_layer.md in agent auto-memory.

Why this is the priority

Stated 2026-05-20: "this has to be the next task we do, it is the missing piece that the company needs." The "20 dev teams × N worlds" plugin-config grind blocks enterprise adoption; the corporate-network UDP-blocking blocks enterprise reachability. Both are dissolved by the gateway. Universe-onboarding's PR6/7/8 are reshaped by this work (PR6 becomes transport-agnostic from the start instead of QUIC-only with a future retrofit), so it makes sense to land the gateway between PR5 and PR6.

Non-Negotiables

• No protocol changes. The demarkus message format (FETCH/PUBLISH/APPEND/VERSIONS/LIST/DELETE/MERGE), content addressing, versioning, and capability-token mechanics are unchanged. The gateway maps HTTP onto these ops 1:1.
• No demarkus-server changes. Worlds keep their current QUIC + mark ALPN listener. They don't know the broker is the caller; they see a bearer token and serve it.
• Tokens stay on the plugin. The plugin holds its own per-world access tokens. The broker forwards Authorization: Bearer … headers verbatim. The broker does NOT mint tokens on behalf of users mid-request and does NOT accumulate world privilege.
• Capabilities stay on the worlds. The world is the trust boundary for what a token can do. The broker may validate the token against its own issuances Secret for rate-limiting and revocation enforcement, but it does NOT re-implement the world's path/operation gates.
• Opt-in deployment. The gateway listener is off by default. Operators enable it explicitly; the existing direct-QUIC universe deployment stays the default.
• No MVPs, no shortcuts. Auth correctness, rate limiting per identity, structured error envelopes, content-hash propagation, conflict-aware merge — all from day one within the MVP slice (not later).
• QUIC stays first-class. The gateway is an enterprise overlay, not a transport replacement. Direct-QUIC access for CLI, Obsidian plugin, and in-cluster server-to-server traffic is preserved.

Out of Scope (explicit)

• Federation ops in MVP. INDEX, RESOLVE, GRAPH, GRAPH-EXPORT, GRAPH-PUBLISH are deferred to a follow-up slice. Knowledge-system enterprise consumers want read/write first; federation is nice-to-have.
• mTLS broker → world. Defer to a hardening pass. Initially the broker reaches worlds over the cluster network; NetworkPolicy + Service-token RBAC are the auth boundary. mTLS is additive and can land without breaking the gateway's wire shape.
• WebSocket / SSE long-lived streams. No streaming-subscription surface in MVP — matches the protocol's polling-with-If-Modified-Since posture (see /thoughts.md § "On Subscriptions and Polling").
• Multi-broker / multi-universe. Single broker per universe stays the model. A future "broker discovery" layer is its own plan.
• Content negotiation beyond text/markdown. Same scope contract as the protocol — markdown only (project_scope_markdown_only.md).
• Caching / CDN integration. The broker may emit Cache-Control headers honestly (immutable hashed reads → immutable; mutable reads → no-cache), but does not front-cache responses itself in MVP. CDN deployment is an operator concern.

Architecture

┌──────────────┐  HTTPS  ┌──────────────────────────────┐  QUIC  ┌─────────┐
│  Plugin /    │ ──────► │  broker (gateway listener)    │ ─────► │ world A │
│  CLI /       │         │                                │        │ (cluster│
│  curl        │ ◄────── │  /v1/worlds/{world}/docs/{path}│ ◄───── │ internal)
└──────────────┘         │    GET    → demarkus FETCH     │        └─────────┘
                         │    POST   → PUBLISH            │
                         │    PATCH  → APPEND             │
                         │    DELETE → DELETE             │        ┌─────────┐
                         │  /v1/worlds/{world}/versions/  │ ─────► │ world B │
                         │    GET    → VERSIONS           │ ◄───── │         │
                         │  /v1/worlds/{world}/list/      │        └─────────┘
                         │    GET    → LIST               │
                         │  /v1/worlds/{world}/merge/     │
                         │    POST   → MERGE              │
                         │                                │
                         │  Authorization: Bearer <wt>    │
                         │  - validate against issuances  │
                         │    Secret (broker minted it)   │
                         │  - rate-limit per subject hash │
                         │  - forward token to world      │
                         │  - world does capability check │
                         └──────────────────────────────┘

Layer responsibilities

Wire mapping (the HTTP↔demarkus contract)

The mapping uses path-based addressing that mirrors the mark:// URL shape so the operator's mental model is unchanged: mark://world/path ↔ https://broker/v1/worlds/world/docs/path.

Status code mapping:

Error envelope (4xx/5xx bodies, application/json):

{
  "error": "stable_code",
  "message": "human-readable detail",
  "world": "team-a",
  "path": "/docs/setup.md"
}

Stable codes match the protocol's existing error vocabulary so consumers can switch on them: not_found, unauthorized, forbidden, conflict, payload_too_large, rate_limited, world_unreachable, internal.

Pre-Flight Tasks

1. Confirm PR5 (#TBD) merged. The gateway plan assumes /me/install is in production so the plugin install bundle's publicURL field is the gateway's URL (operator-controlled scheme). If PR5 is still in review when work starts, fine — the gateway doesn't depend on it at code level, only at deployment-pattern level.
2. Inventory the demarkus client library surface. Check that client/internal/... exposes a Go-callable client for FETCH, PUBLISH, APPEND, VERSIONS, LIST, DELETE, MERGE that the broker can import. If any op is only exposed via the CLI command layer, hoist it to the library first as its own commit. Not expected to be a blocker — Mark protocol's library design has always favored embeddable clients.

Sub-Tasks (sequenced; one slice per PR)

Slice 1 — Foundation + FETCH (read-only gateway, ~PR-sized)

• tools/demarkus-broker/internal/broker/gateway.go (new) — HTTP/2 listener scaffolding, route mux, request-context propagation. Separate from the existing management-API listener so operators can run one without the other.
• tools/demarkus-broker/internal/broker/gateway_auth.go (new) — gatewayAuth middleware. Validates the world-access-token against the broker's issuances Secret (it minted the token; it knows the hash + scope + identity). Rejects expired/revoked/unknown tokens with 401. Extracts identity for downstream rate limiting + logging.
• tools/demarkus-broker/internal/broker/world_pool.go (new) — worldPool type. Lazy-initialized demarkus QUIC client per world name. Reuse on subsequent requests; reconnect on transport failure. Bounded parallelism via a per-world semaphore. Lifecycle bound to broker shutdown.
• tools/demarkus-broker/internal/broker/gateway_fetch.go (new) — handleGatewayFetch translates GET /v1/worlds/{world}/docs/{path} into a demarkus FETCH against the routed world. Maps response to HTTP including ETag, X-Demarkus-Content-Hash, Content-Type.
• tools/demarkus-broker/internal/broker/config.go — ServerConfig.Gateway substruct: Enabled, Addr, TLS{...}, RequestSizeLimit, WorldPool{...}. Validation: when Enabled, Addr + TLS.CertFile + TLS.KeyFile (or TLS.AutoCert{}) required.
• tools/demarkus-broker/main.go — start the gateway listener when enabled, alongside the management-API listener. Single shutdown signal tears both down.
• gateway_test.go + gateway_auth_test.go + gateway_fetch_test.go — full HTTP-driven tests against a fake demarkus server backend. Auth happy path, token-unknown 401, token-expired 401, token-revoked 401, world-unknown 404, world-unreachable 502, FETCH-200, FETCH-not-found-404, content-hash + ETag round trip.

Slice 2 — Write ops (PUBLISH / APPEND / DELETE)

• tools/demarkus-broker/internal/broker/gateway_write.go (new) — three handlers. Each requires If-Match (precondition required); a missing or empty If-Match returns 428 Precondition Required. Body size enforced per cfg.Server.Gateway.RequestSizeLimit (default mirrors protocol's 1MB).
• Conflict handling: maps demarkus optimistic-concurrency rejection to 409 Conflict with the conflict envelope. PUBLISH on a stale version returns a current_version field so the client can re-fetch and retry.
• If-Match parsing: strict quoted-string form ("<version>") per RFC 9110.
• Tests: happy paths for each verb, missing-If-Match 428, version-mismatch 409 with envelope, oversize-body 413, world-permission-denied 403 (token-vs-path mismatch enforced by world), idempotency-check for PUBLISH (two writes with same version → second is 409).

Slice 3 — Metadata ops (VERSIONS / LIST)

• tools/demarkus-broker/internal/broker/gateway_meta.go (new) — VERSIONS returns JSON-encoded version metadata (matches the protocol's existing VERSIONS frontmatter shape). LIST returns JSON listing with frontmatter pagination shape. Both GET, no body, idempotent.
• Tests: VERSIONS happy + not-found, LIST happy + empty prefix + paginated case.

Slice 4 — Conflict-aware merge (MERGE)

• tools/demarkus-broker/internal/broker/gateway_merge.go (new) — POST /v1/worlds/{world}/merge/{path}. Body: candidate markdown. If-Match indicates the version the candidate was edited against. Response: 200 with merged body, OR 409 with three-way conflict envelope (base / ours / theirs / merged-with-markers).
• Reuses the protocol's existing merge-candidate logic shipped in mark_publish's conflict-aware mode.
• Tests: clean-merge happy path, structural-merge 200, conflict-marker 409 with envelope, malformed-base 400.

Slice 5 — Chart, RBAC, docs

• deploy/helm/demarkus-broker/values.yaml — server.gateway.enabled, server.gateway.addr, server.gateway.tls (existingSecretRef recommended for production; cleartext path for dev), server.gateway.requestSizeLimit, server.gateway.worldPool knobs.
• deploy/helm/demarkus-broker/templates/deployment.yaml — when enabled, expose an additional containerPort + volume mount for the gateway TLS material. Same Pod, separate listener.
• deploy/helm/demarkus-broker/templates/service.yaml — when enabled, expose the gateway port on the Service.
• deploy/helm/demarkus-broker/templates/ingress.yaml — when enabled AND ingress is enabled, route the gateway-prefix host onto the new port.
• deploy/helm/demarkus-broker/templates/networkpolicy.yaml — when enabled, allow ingress on the gateway port from the same source set as the management API.
• deploy/helm/demarkus-broker/templates/rbac-broker-ns.yaml — broker SA already has get/update on the issuances Secret (used by gatewayAuth); no new RBAC needed in MVP.
• deploy/helm/demarkus-broker/README.md — new "HTTPS gateway" section: when to enable, TLS setup, the path shape, the If-Match contract, the rate-limit behavior, the corporate-network rationale.
• tools/demarkus-broker/main.go package doc — bump Current scope.
• New tools/demarkus-broker/HTTP-API.md — operator/developer-facing spec for the wire mapping. Lives in the broker package so it's clear the spec is a knowledge-system artifact, not part of the protocol contract.

Scope estimate

Five PRs across ~1.5–2 working weeks. Slice 1 is the biggest because it stands up the entire surface (listener, auth, pool, error envelope, one op end-to-end); subsequent slices add ops on top of the scaffold and are smaller.

Open Questions To Resolve Before/During Implementation

1. URL shape: path-based vs verb-based. Lean: path-based with op-distinct subpaths (/docs/, /versions/, /list/, /merge/), HTTP method drives content-op semantics. Mirrors the mark://world/path mental model. Alternative is gRPC-style explicit verb paths (/fetch/, /publish/, /append/); rejected because PUBLISH/APPEND/MERGE are all POST-shaped and verb-in-path is redundant with method.
2. Auth posture. Lean: world-access-token alone (no id_token layer). The broker validates the token against its own issuances Secret — it has the data to do this without an extra round trip. Identity comes from the issuance record; rate-limit and audit log key on that identity. World still does capability enforcement independently. Alternative (id_token + world-token) adds the broker as a second auth gate but doubles the bearer the plugin has to manage per request.
3. mTLS broker → world. Lean: trust the in-cluster network for MVP. NetworkPolicy already restricts ingress to worlds; the broker SA is bound by RBAC. mTLS is a hardening pass — useful, but not blocker-level. Add as a follow-up plan after the gateway lands.
4. Federation ops in MVP. Lean: defer (INDEX, RESOLVE, GRAPH, GRAPH-EXPORT, GRAPH-PUBLISH). Knowledge-system enterprise consumers want read/write/list/merge first. Federation can land as Slice 6 if there's customer demand.
5. worldPool parallelism. Lean: configurable per-world parallelism, default 8 concurrent QUIC streams per world (matches QUIC's stream multiplexing sweet spot). Single-connection-per-world is too tight under load; one-connection-per-request defeats the point of the pool.
6. Streaming response bodies. Lean: buffer for MVP. Demarkus's 1MB body limit (see /thoughts.md) makes streaming unnecessary. LIST responses are JSON metadata; also bounded. Revisit if a customer hits a real streaming need.
7. TLS cert source. Lean: same shape as the existing broker (cert-manager via existingSecretRef for production; cleartext file path for dev). Operators familiar with the management-API TLS story can reuse the pattern.
8. API versioning. Lean: /v1/ prefix in the path. Cheap insurance against a future incompatible reshape — and the path shape is the only thing v2 could conceivably reshape (the demarkus message semantics underneath are stable).
9. CORS posture. Lean: deny by default; emit Vary: Origin but no Access-Control-Allow-* headers in MVP. The plugin and tools/demarkus-join are first-party; cross-origin browser callers aren't a target audience yet. Add allowlist config when a real consumer asks.
10. Request-ID propagation. Lean: accept incoming X-Request-Id if present; generate one otherwise; forward via demarkus's correlation-id mechanism (if present) or as a request header. Log on both legs (HTTP request + QUIC request) with the same ID for cross-leg debugging.
11. Conflict envelope shape. Lean: same shape as mark_publish's existing conflict response (publish-at-version, base body, candidate body, merge candidate with markers). Reuse the protocol-level type; don't invent a gateway-specific one.

Risks Specific To The Gateway

• Public attack surface gains an HTTP listener. Previously the broker exposed only OIDC + management APIs. The gateway is now another internet-reachable port talking to the most sensitive part of the universe. Mitigation: hardened auth middleware (validate against issuances Secret on every request), per-subject rate limiting (shared bucket with /tokens per identity, not a separate quota that doubles the available budget), strict TLS (no downgrade, no plaintext mode in production), structured error envelopes that don't leak internal world names beyond what the user already knows.
• Connection pool starvation under load. A burst of concurrent requests to one world exhausts the pool and subsequent requests queue or fail. Mitigation: per-world semaphore with a configurable cap + a request-side timeout that fails closed rather than queues unbounded. The 429 / 502 surfaces are honest about the broker's state.
• QUIC connection breakage cascades. A world's QUIC listener restarts; every request mid-flight fails. Mitigation: client-side reconnect with backoff; broker handlers translate transport errors into 502 with a Retry-After hint so the consumer retries cleanly.
• Token-validation latency. Validating against the issuances Secret every request adds a k8s API round trip per call. Mitigation: in-memory cache (keyed by token hash, short TTL ~30s, invalidate on revoke). Same cache pattern the existing token surfaces could benefit from; consider extracting if it lands well here.
• Plugin holds N tokens in one process. The shift from "one MCP server per world" to "one MCP server for the universe" consolidates N world tokens into one process. A buggy world handler leaks tokens across worlds. Mitigation: design the plugin-side client with per-world goroutine isolation + token-zeroize-on-eviction from the start (PR6 concern, not gateway concern, but flagging here so PR6 inherits the design intent).
• Operator confusion: two listeners, two URLs. The management API stays on its existing address; the gateway is a new address. Operators have to wire two DNS entries, two TLS certs, two Ingress rules. Mitigation: chart docs are explicit; the default gateway config sample pairs with the management default; consider a single-listener mode where the gateway routes are mounted on the management API listener under a path prefix (deferred — not MVP; chart can offer both shapes once the pattern settles).
• The gateway is the only new place we re-implement parts of the protocol's wire semantics in HTTP terms. This violates DRY across the protocol↔HTTP boundary by design (the HTTP shape is the gateway's invention, not the protocol's). Mitigation: keep the wire mapping as a single document (tools/demarkus-broker/HTTP-API.md), and write the handler tests against the demarkus client library's existing behavior rather than re-asserting protocol semantics. The handlers should be thin.

Touch Points With Adjacent Work

• Universe-onboarding PR5 (just shipped). No code coupling. PR5's worlds[].publicURL field is operator-controlled and can carry a gateway URL (https://broker.acme.com/v1/worlds/team-a) without code changes. Operator config decides which transport the plugin uses.
• Universe-onboarding PR6 (tools/demarkus-join). Reshaped by the gateway. PR6 should be transport-agnostic from the start: read whatever scheme the publicURL field carries (mark:// direct or https:// via gateway) and pick the matching client transport. Sequencing: gateway lands → PR6 designed against the gateway-aware client surface → PR7 → PR8.
• Universe-onboarding PR7 (plugin slash commands). Inherits the multi-world-in-one-MCP-server shape implicit in the gateway architecture. /soul-join adds a world to the running MCP server's world list rather than installing a new MCP server.
• Future mTLS broker→world plan. This gateway plan is the natural anchor for that work; the world pool can grow mTLS in place without API changes.
• Future federation-over-gateway plan. When INDEX/RESOLVE/GRAPH land, the gateway adds them as additional URL prefixes (/v1/worlds/{world}/index/, /v1/worlds/{world}/graph/). No core change.
• /thoughts.md § "On the Protocol as the Permanent Layer". The principle this plan is built on. If a future feature request would require changing the demarkus protocol to support a gateway feature, the right answer is to push it back into the gateway layer or refuse it — not to extend the core.

Resume Steps (when starting work)

1. Confirm PR5 (the feat-tools-broker-me-install branch) is merged. If still under review, work on Slice 1 can proceed on a branch off main without conflict; the gateway code doesn't touch PR5's surfaces.
2. mark_fetch /plans/broker-https-gateway.md (this doc).
3. Re-confirm Open Questions 1-3 (URL shape, auth posture, mTLS) before writing code — they shape the foundation of Slice 1. Lean answers above are the defaults; deviations require an explicit decision.
4. Cut branch feat-tools-broker-gateway-foundation (or similar). Start at Slice 1.
5. Each slice is its own PR. go test -race + bash pre-commit.sh green before moving on.
6. Journal at session end. Update this plan's Implementation Status section as slices land.

Done When

• All five slices merged.
• go test -race ./... green inside tools/demarkus-broker/.
• helm unittest . green for the broker chart.
• pre-commit.sh green.
• Manual end-to-end via curl: POST /auth/device/... (device flow) → GET /me/install (bundle) → curl https://broker/v1/worlds/team-a/docs/foo.md -H "Authorization: Bearer <token>" returns the world's response.
• Operator-facing README documents the deployment.
• Journal entry covering any design decisions that landed differently from this plan.

Implementation Status

Not started. Plan published 2026-05-20; awaiting PR5 merge + Open Questions 1-3 confirmation before Slice 1 begins.