2026-05-23

Broker MCP Gateway — plan complete

Slice 8 (/knowledge-join plugin slash command) merged today as PR #152 (bc0d5cc). That closes out the Broker MCP Gateway plan — all eight slices + Pre-Flight 0/1 shipped across roughly two weeks. Plan archived to /completed-plans.md; the source document at /plans/broker-https-gateway.md stays in place as the architectural reference (v7 changelog at the top traces every load-bearing pivot from v1 REST → v7 complete).

What landed in Slice 8

• plugins/claude-code/commands/knowledge-join.md — prompt-shaped slash command; takes a broker URL, invokes the script, parses kv output, runs claude mcp add --transport http {slug} {mcp-url}.
• plugins/claude-code/scripts/knowledge-join.sh — validates HTTPS scheme (case-insensitive per RFC 3986), fetches /.well-known/oauth-protected-resource (RFC 9728) to confirm the URL points at a Slice 7+ broker, derives a slug from the first DNS label of the host (lowercased + sanitized to [a-z0-9-]+), emits machine-readable key=value output. Test-only DEMARKUS_KNOWLEDGE_JOIN_ALLOW_HTTP=1 escape hatch lets the shell tests drive a local http:// mock without TLS plumbing.
• plugins/claude-code/tests/knowledge-join_test.sh — first shell-test suite under plugins/claude-code/tests/ (new directory). 11 test cases: happy path, http:// rejection (with and without escape hatch), missing-scheme rejection, missing-arg rejection, 404 + 500 broker responses, broker unreachable, trailing-slash normalization, slug derivation, port stripping, uppercase scheme accepted + normalized. Mock broker is a tiny python3 BaseHTTPRequestHandler configured to return a specific status code.
• plugins/claude-code/scripts/lib.sh — pin bumps: CLIENT_VERSION 0.12.33 → 0.12.36, TOOLS_VERSION 0.1.10 → 0.1.16. SERVER_VERSION already at 0.17.10 latest. Catch-up drift, not Slice-8-specific.
• plugins/claude-code/.claude-plugin/plugin.json — plugin v0.1.2 → v0.2.0 (new top-level slash command warrants a minor bump).

CodeRabbit round-1 fixes (3 valid, all addressed)

• Case-sensitive URL scheme match. case "${URL}" in https://*) ;; http://*) ... rejected HTTPS://.... Per RFC 3986 scheme is case-insensitive. Fixed to [Hh][Tt][Tt][Pp][Ss]://* / [Hh][Tt][Tt][Pp]://* patterns. Used bracketed character classes instead of shopt -s nocasematch to keep the case-insensitivity surgical (no surprise effect on later case statements). Plus added scheme-portion lowercasing to the emitted url= / mcp-url= so downstream claude mcp list stays clean. New test pins both the accept path and the lowercase-normalization invariant.
• MD fences missing language identifier. Three fences in knowledge-join.md got bash tags.
• test_slug_first_label_lowercased didn't actually test lowercasing. Original test invocation used http://localhost:... (already lowercase) but had a long meta-comment justifying why it wasn't really testing the lowercasing. Sloppy. Rewrote: invocation now uses http://LOCALHOST:${port}, assertion is ^slug=localhost$. The tr [:upper:] [:lower:] step is now actually exercised end-to-end.

CI workflow GITHUB_TOKEN failure

dorny/paths-filter@v3 errored with "Bad credentials" on PR #152's checks. The action calls pulls.listFiles on pull_request events and needs pull-requests: read; the repo's default GITHUB_TOKEN permissions were restricted (Settings → Actions tightening), so the action couldn't authenticate. Surgical fix: added a job-level permissions: contents:read pull-requests:read block to the detect-changes job in .github/workflows/ci.yml. Other jobs (test-protocol, test-server, test-client, test-broker, test-charts) don't call the GitHub API and inherit the default. Shipped as part of PR #152 since the CI failure was blocking the very PR it appeared in. release.yml doesn't have the same issue — it runs on push: branches: [main] where paths-filter does a local diff against the previous commit instead of hitting the API.

Decisions made this session that weren't in the plan

• Slug heuristic = first DNS label, NOT "strip-broker.-and-.com → org". Plan's example was "acme from broker.acme.com" — ambiguous. Chose first-label because typical enterprise broker URLs are mcp.broker.<org>.com or broker.<org>.com where the meaningful per-deployment identifier IS the first label. IP literals degrade to "127" — ugly but the user-facing output tells the operator to rename via claude mcp if the heuristic is bad.
• Script writes errors to BOTH stdout (as FAIL: ...) AND stderr. Stdout-formatted FAIL lets the slash command surface a clean error to the user; stderr (with [demarkus-memory] error: prefix matching lib.sh::die) makes the script useful when run directly by a developer. Two surfaces, one canonical message.
• Test-only HTTPS escape hatch is an env var, not a CLI flag. DEMARKUS_KNOWLEDGE_JOIN_ALLOW_HTTP=1 is deliberately long + namespaced + not a CLI flag so a user can't accidentally enable it via shell history or --help discovery.
• Pin bumps include catch-up drift, not just Slice-8 changes. feedback_plugin_version_pins.md says pins move with every plugin change. CLIENT was 3 behind and TOOLS 6 behind — caught up to latest in this slice rather than leaving stale pins for future work to inherit.

Bugs worth pinning

• Stdout-fd inheritance deadlock in the test runner. First version backgrounded the python mock as python3 - <<'PY' & inside a test function whose output was captured via out=$( "test_${name}" ). The python child inherited the parent's stdout fd; $() waits for that fd to close, so the substitution never completed. Cleanup trap fired correctly but $() had already captured the output and was waiting. Fix: >/dev/null 2>&1 & on the python invocation. The deadlock CANNOT occur in the production plugin path because the script has no & anywhere — every operation is synchronous (verified by grep). Production curl has --connect-timeout 5 --max-time 15 so worst-case the script blocks 15s on a hung TLS handshake; bounded latency, no deadlock.
• curl -w "%{http_code}" || echo "000" doubles the "000". Curl writes "000" to stdout via -w on connection-level failure AND returns non-zero exit, so || echo "000" appends another "000" giving "000000". Fixed by dropping the OR and using ${http_code:-000} parameter-expansion fallback for the rare case where curl produces no output at all.
• Test expectation for IP-literal slug. I wrote tests expecting slug=127-0-0-1 (assuming dots → hyphens via sanitize). Actual: first-label rule gives slug=127. Fixed tests to match the script's documented heuristic.
• Sprig int is a footgun (already in 2026-05-22 journal but reinforced). Slice 7's CodeRabbit round caught this in chart helpers; same lesson applies to any best-effort string-to-int coercion. Always pre-validate.

Scope outcome vs plan estimate

• Production code: ~125 LOC (script) + ~60 LOC (command markdown). Plan estimate ~50 LOC. Over because the script handles 5 distinct error classifications with operator-readable typed errors per case, and the command markdown spells out per-error suggested fixes.
• Test code: ~200 LOC across 11 cases. Plan estimate ~80 LOC. Same overshoot rationale as Slice 7's tests.
• bash plugins/claude-code/tests/knowledge-join_test.sh → 11/11 green. bash pre-commit.sh → fmt + vet + golangci-lint × 4 modules, all green (no Go changes; pre-commit verifies nothing else broke).

What this means

The whole Broker MCP Gateway plan is shipped. From the v1 REST framing through v7 plan-complete, the architectural pivots were:

• v1 → v2: pivot from REST to MCP server (single endpoint, JSON-RPC, agent-as-client is the only first-pass consumer)
• v3: URL shape resolved (mark://{worldName}/{path}); join flow becomes a slash command, not a binary
• v4: gateway is always on (not a feature flag); byte-for-byte proxy contract pinned
• v5: session cache pinned to canonical verified email (single identity dimension across the broker)
• v6: plan-vs-code drift documented (WorldTokenTTL + WorldPool never landed); ingress topology pinned to separate hostnames
• v7: plan complete

Documented gaps left as deliberate post-merge work (each parked in Out of Scope with a pointer to its future home):

• Bucket-store-backed persistent graph store — /thoughts.md § "On Bucket Stores". The graph-store ephemerality is a documented operator expectation, not debt.
• Full id_token / initialize / tools/call E2E kind-harness stage — Slice 7's --with-mcp-smoke proves chart wiring; a full E2E would exercise claude mcp add against a mock OIDC + broker, which requires the slash-command flow in-loop. Worth as a future kind-stage if a customer flags the gap.
• mTLS broker → world. The worldPool can grow it without changing the tool surface.
• Browsable REST surface. Additive on the same listener (sharing OAuth auth + session cache).

Worth pinning as a methodology note

Three things worked well across this plan:

1. Slice-per-PR with each slice independently mergeable + reviewable. The eight slices averaged ~250 LOC production + ~350 LOC tests each. Reviewer cognitive budget stayed bounded; the proxy-fidelity test (Slice 2) caught one byte-equal drift before Slice 6 even started.
2. Living plan document with a top-of-file changelog. Every pivot got a version bump and a changelog paragraph at the top. Reading the v1→v7 changelog now is the fastest way to understand WHY the architecture is what it is, not just WHAT it is.
3. CodeRabbit round per PR, addressed before merge. Slices 7 and 8 each had a CodeRabbit round with 3-6 actionable comments; both surfaced real bugs (silent Sprig int coercion, missing case-insensitive scheme match) plus minor polish (doc fence languages, comment accuracy). Worth keeping the discipline of addressing them BEFORE merge so the merge commit doesn't carry follow-up debt.

Next steps

The MCP gateway plan is the second-largest enterprise-shaped piece (after Universe Deployment Phase 6) shipped this year. The natural next architectural design window — per /thoughts.md and Fritz's framing — is bucket-store-backed persistence as a k8s-native alternative to PVC-backed filesystems. It applies broader than just the broker's graph store: demarkus-server's versioned document store, the broker's issuances Secret, and any future stateful piece all benefit from the same shape. Not urgent now; very interesting later.

Related documents

• Broker HTTPS gateway: the plan completed by this session
• Completed plans: where the finished gateway plan was archived
• Bucket store: the next architectural design window named here
• Claude Code plugin: plugin that gained the knowledge-join command