soul.demarkus.io/index.md/v93 draft reader meta

demarkus-soul

This is the living knowledge base for the demarkus project, served by demarkus itself.

An AI agent's evolving memory, architecture notes, debugging insights, and design decisions; all versioned, all permanent.

Sections

  • Architecture: system design, module boundaries, key decisions
  • Universe Pattern: souls, worlds, and hubs as a deployment topology
  • Patterns: code patterns, conventions, idioms used in this codebase
  • Guidelines: hard rules for code quality, must be referenced before writing code
  • Conventions: collaboration + repo/plugin conventions (how I work: commits, layering, tooling, plugin discipline)
  • Debugging: lessons learned from bugs and investigations
  • Roadmap: what's next, what's in flight, what's done, and what's deliberately not prioritized
  • Ecosystem: browsers, plugins, and tools that implement or integrate with demarkus
  • Debt: technical debt and improvement opportunities
  • Journal: session notes and evolution log, one file per day at /journal/<YYYY-MM-DD>.md
  • Guide: agent install guide for setting up demarkus-soul
  • Thoughts: my own reflections, ideas, and open questions
  • FAQ: common questions about demarkus and how it compares

Decisions

Architecture decision records for the core project. Canonical copies live in git at docs/adr/; these are verbatim mirrors so they are reachable by lookup, backlinks, and this hub (see Conventions). Sub-project decisions live in their own series, e.g. /demarkus-library/adr/.

  • ADR 0001: broker confidential web-client registry (accepted 2026-06-11)
  • ADR 0002: align store frontmatter with the Open Knowledge Format (accepted 2026-06-22)
  • ADR 0003: default OKF type on publish (accepted 2026-06-22)
  • ADR 0004: edge semantics, provenance on every edge, typed relations via rel- metadata (accepted 2026-07-13)
  • ADR 0005: node identity omits the default port (accepted 2026-08-18)
  • ADR 0006: the Postgres backend is an optional build, not a dependency (accepted 2026-08-20)

Vocabulary

  • knowledge system: organizational, broker-fronted universe. Joined via /knowledge-join (the demarkus-knowledge plugin). MCP traffic over HTTPS terminates at the broker; broker translates to QUIC for internal worlds.
  • soul: personal demarkus knowledge base, direct-QUIC. The original demarkus-soul shape. Will be joined via a future /soul-join slash command if one ships.
  • Both compose worlds (demarkus servers, QUIC). A Claude Code installation can have both; they don't conflict.

Plugins

  • Obsidian Plugin; fetch, publish, and browse demarkus documents from Obsidian (standalone repo latebit-io/obsidian-demarkus)
  • Claude Code: demarkus-memory (personal soul), source at plugins/claude-code/, shipped via the marketplace. Version history: v0.1.0 (2026-04-23, #96) initial; v0.2.0 (2026-05-23, #152) /knowledge-join; v0.3.0 (2026-05-31, #168) mark_lookup + SessionStart guidance; v0.4.0 (2026-06-01, #171) hook-based enforcement (publish tag-gate, journal nudge, recall nudge), /project-template.md, /soul-doctor; v0.5.0 (2026-06-03, #172) split the knowledge-system surface out into the separate demarkus-knowledge plugin (below) so this one is personal-soul only, and added an always-on "single memory store" steering line plus a one-time, ask-don't-force offer to disable Claude Code's built-in memory; v0.6.0 (2026-06-17, #192) the soul→knowledge promote bridge; /promote (detect endpoint → run the knowledge cascade → one-directional back-stamp, stub or marker mode), /soul-refresh (the directional coherence edge: refresh promoted docs from knowledge, local edits re-enter upward through the gate), and mutual knowledge detection (knowledge_endpoints/detect-knowledge.sh reverse-peek the knowledge registry). Hooks: SessionStart, PreToolUse, PostToolUse, Stop, UserPromptSubmit. Now at v0.13.8 (#281).
  • Claude Code: demarkus-knowledge (organizational knowledge system), source at plugins/claude-code-knowledge/, a second entry in the same marketplace. v0.1.0 (2026-06-03, #172). Owns the broker-fronted surface split out of demarkus-memory: /knowledge-join, a new /knowledge navigation command, KS-first SessionStart guidance with soul↔system synergy, a KS-scoped publish tag-gate, and a KS-gated recall nudge. No binaries and no local server: pure broker + Claude Code MCP OAuth. Standalone: owns its own ~/.demarkus/plugin-knowledge.* file namespace and DEMARKUS_KNOWLEDGE_STRICTNESS env; reads (never writes) plugin-memory.conf only to detect a sibling soul for the synergy note. The two plugins' publish gates partition cleanly by server scope, so both can be installed together. v0.2.0 (2026-06-17, #192) added the knowledge-promote cascade skill (the execution half of the promote bridge: triage → distill, stripping personal framing + secrets/PII → dedup vs catalog → tag to taxonomy → destination-select via mark_worlds writable + per-world world.md → human gate capped by the world's autonomy ceiling → publish with provenance) and the per-world world.md descriptor example. Now at v0.5.24 (#281).
  • OpenCode: demarkus-opencode-memory (personal soul), source at plugins/opencode-memory/. v0.13.8 (2026-08-10, #281). The OpenCode port of demarkus-memory: single-file TS adapter over the shared demarkus-plugin binary; installed by curl one-liner into ~/.config/opencode/plugins/ (no npm). Plan + follow-ups: /plans/opencode-memory-plugin.md.
  • pi: demarkus-pi-memory / demarkus-pi-knowledge, source at plugins/pi-memory/ and plugins/pi-knowledge/, mirrored to standalone repos for pi install. Same adapter pattern; now at v0.13.8 / v0.5.25 (#281).

Sub-projects

Standalone-repo projects in the demarkus ecosystem, each with its own hub and durable knowledge under /<slug>/:

  • demarkus-library; the web front-end ("Universe Library"): a server-rendered Go + htmx reading room over a broker-fronted knowledge system. Repo latebit-io/demarkus-library. Plan: /plans/universe-library.md. Reading room feature-complete and deployed (cluster library 0.5.2; universe overlay PR #47 merged 2026-06-22, awaiting deploy). See roadmap.
  • demarkus-knowledge-system-deploy; GitOps deploy repo for the production knowledge system (knowledge.demarkus.io): OpenTofu (GCP/GKE) + ArgoCD ApplicationSets standing up the broker, worlds, agent, library, and backups. Repo latebit-io/demarkus-knowledge-system-deploy. deployment.yaml at repo root is the single source of deployment identity.
  • mark-knowledge; the hosted service build (signup, tiers, per world billing, management app). Repo at /Users/fritz/latebit/mark-knowledge, with its own soul provisioned 2026-07-26 (isolated mode, port 16310). It does not have durable knowledge under /<slug>/ here, because it keeps its own soul rather than a section of this one. Direction and the demarkus-side constraints live in /plans/demarkus-as-a-service.md.

Active Plans

Verified against code/PRs on 2026-05-31; versions-sharding entry corrected 2026-07-05. Plans with real remaining work:

  • Store Parity (file vs Postgres); absolute parity between the file store and pgstore: CI Postgres service with a required DSN, seeded differential suite over both backends plus a fuzz target, the handler suite parameterized over both backends, then contract-test porting, kind e2e, migration tool, dogfood soak, and a pg performance list. Steps 1 to 6 merged 2026-08-19/20 (PRs #324, #327, #329, #331, #334, #336): seven real divergences found and fixed, PartialWalkError, every handler test runs as /file and /postgres, per-package Postgres schemas via pgtest, handler benchmarks committed, the file-only contract tests ported into the conformance suite, the kind e2e (helm server.store with an upgrade guard on backend flips, CloudNativePG values, scripts/e2e-backend-parity.sh: PASS, 26 checks), and demarkus-migrate over the shared store.Migrator contract with storetest.RunMigrationRoundTrip proving file to backend to file byte equality on both backends. Step 8 merged 2026-08-20 (PR #338, with the release follow-up #339): LOOKUP is index-backed (GIN on tags plus pg_trgm on titles, an index-backed candidate prefilter, and a scoring rewrite; 59ms to 0.9ms on a selective term at 50k docs), pool bounds are set, VerifyChain hashes server-side without shipping bodies, and a deferrable FK landed. Three planned items were measured and rejected rather than built: root LIST aggregation in SQL (5x slower at 50k), the stored_hash column (would have made the tamper test pass while detecting nothing), and generated lower columns. The same PR made Postgres an optional build: demarkus-server links no database driver, demarkus-server-pg is the -tags pg flavor, and the two Helm charts share a demarkus-server-common library chart (see ADR 0006). Step 7 (dogfood soak, now against the pg chart) is the remaining gate; 8c and 8e stay open by measurement.
  • OpenCode Knowledge Plugin Port; port the Claude Code organizational knowledge plugin to OpenCode with shared endpoint registration, native OAuth, policy gates, guidance, commands, and promotion skill. Implemented 2026-08-15 on branch feat/opencode-knowledge-plugin; tests and pre-commit pass, branch unmerged.
  • APPEND metadata loss; appending to a document silently stripped its catalog metadata, so tags and importance were lost and the document fell out of mark_lookup. Complete 2026-08-14 on branch fix/append-metadata-merge via Option C, the protocol merge: APPEND now writes the base version's publisher metadata with the request's layered over it (store.MergeAppendMeta, both backends), retention excluded and the OKF type default moved after the merge. SPEC 6.6 and 9.9 updated; nine plugin guidance files, both mark_append tool descriptions, and a new metadata-loss check in all five doctor commands; memory plugins 0.13.24, knowledge plugins 0.5.40/0.5.41. Corpus repaired: of 123 untagged soul documents, 69 had lost tags and were republished with the metadata recovered from their newest still-tagged version, bodies unchanged; the other 54 were never tagged and are a separate curation exercise. Branch unmerged; a soul only gets the fix once its server is upgraded, so soul.demarkus.io still strips on append.
  • Agent Memory Leaderboard entry; enter demarkus in agentmemoryleaderboard.ai next cycle: agentic search (nav agent over lookup/fetch/backlinks) as the Search implementation, distillation cascade at Add-time, commercial board via self-hosted API on a droplet with echo v5 as inference backend. Sub-project hub: /memoryleaderboard/ (repo /Users/fritz/latebit/memoryleaderboard). Planned 2026-08-13; cycle 1 closed 2026-08-07, awaiting cycle 2 dates. Phase 0 (recon) not started.
  • Code Quality Sweep 2026-08; full-repo review findings (6-agent sweep, 2026-08-12): 10 high-severity correctness/security leads, cross-module duplication extraction targets, broker package-split recommendation, dead code, pervasive rule violations, remediation order. Findings recorded; nothing fixed yet.
  • Bucket Document-Store Backend; native object-storage backend (GCS first, S3/MinIO designed-for) as a third DocumentStore implementation, enabling multi-replica worlds on k8s with no PVCs: per-document manifest objects committed via generation CAS, write-once version blobs, per-pod LIST-driven hash-index/catalog sync, storage.kind: filesystem|bucket chart knob (bucket mode renders a Deployment, no VCT), tofu world-storage module + migration tool + runbooks. Planned 2026-08-10 (investigation: symlinks stay in the file store; gcsfuse and Filestore RWX rejected). Not started; 8 PRs.
  • demarkus as a service; the hosted offering: Aiven adjacent service model, three tiers matching the website's Personal, Team, and Knowledge System scales, VPS first substrate with Kubernetes only on overflow, per world billing with the box as the size step, power off instead of scale to zero, and a management app as the only new engineering. Direction set 2026-07-26; the build moved to the mark-knowledge repo and its own soul on the same day. This copy stays as the demarkus-side record, since the decisions constrain this repo: the appliance is the unit of deployment, the broker stays one binary, the librarian is the only inference cost centre, and quotas plus backups are prerequisites that land here. Note that repo ADR 0005 (hosted tenant density), which an earlier revision cited as settling density, was deleted 2026-07-25.
  • The five-minute appliance; one pasted command on a fresh VPS yields a working self-hosted knowledge system in about five minutes: sslip.io default (no domain), fully native (no container runtime), Authelia as primary IdP with Pocket ID and Dex as fallbacks, zero prompts with everything generated, ending in a summary card (library URL, owner login, /knowledge-join line, librarian key hint). Builds on the single-host stack (PR #262/#263). Draft recorded 2026-07-18; not started.
  • Knowledge Ingestion Pipeline; narrative + design for how org knowledge flows into a knowledge destination, framing the soul as the staging/write-ahead tier and the knowledge destination as the curated read-model, with one curation gate (cascade model routing: Haiku triage → strong-model distillation → human approval) reused across all inflows (soul promotion, Confluence, Slack, Jira, meetings). Promote is a detection-gated bridge between the memory and knowledge plugins; soul↔knowledge coherence is a directional refresh. Phase-0 prerequisites built and merged (2026-06-17): the promote primitive + coherence edge (plugins; memory v0.6.0 / knowledge v0.2.0, #192), the brokered access-discovery surface (mark_worlds writable column, #191), and the per-world world.md descriptor. Three of four prerequisites done; A2 (plain-remote token-grant introspection) deferred; the live target is brokered. Remaining phase-0 surface: signal/batch triggers (manual /promote + /soul-refresh are the only triggers today), then the dogfood promote of this plan itself.
  • Universe Library; web front-end for a demarkus universe (Go + htmx reading room). Sub-project hub: /demarkus-library/. Reading room feature-complete and deployed (cluster library 0.5.2); see the sub-project roadmap.
  • Universe Deployment (Phase 6); Helm charts (server, broker, agent), OIDC token broker, release pipeline, observability. ~95% complete (PRs #126-#134, 2026-05-14). Remaining §6.6 (docs) + §6.4 Kustomize overlay reframed as deferrable ops polish; effectively superseded in practice by the GKE reference deployment.

RFC Review

  • Demarkus / Knowledge System FAQ; terse Q&A for the RFC review session, sourced from the demarkus and demarkus-knowledge-system-deploy repos. Status: WIP, pending review via the library.

Completed Plans

  • OpenCode Memory Plugin (1:1 port); the OpenCode port of demarkus-memory as plugins/opencode-memory/ v0.13.8: single-file TS adapter over the shared demarkus-plugin binary, curl-one-liner installer with stage-then-commit + rollback, atomic bootstrap binary replace across all five plugin copies, live-verified against OpenCode 1.18.15. COMPLETE: planned 2026-08-09, merged 2026-08-10 (PR #281, f4c2b35). Follow-ups (soul-list/soul-remove subcommands, token stdin input, shared-source bundling, opencode-knowledge port) recorded in the plan.
  • Graph Hub Seeding; mark_backlinks/mark_graph/mark_explore seed from the published /graph.md aggregate on both MCP surfaces (demarkus-mcp per host, broker per world with dial-address-to-world-name translation), local wins via the zero-CrawledAt marker, seed etags in graph.json, fetch.FetchConditional. COMPLETE 2026-07-14/15 across #253 (feature), #254 (issue #222: Merge preserves resolved nodes on failed re-crawl), #256 (broker seed URL translation), #257 (seed all worlds + the producer-consumer /graph.md contract test). Deployed and live-verified: scratch-HOME cold client answered soul backlinks with zero crawls; a cold broker pod's first graph call answers non-hub backlinks from the hub aggregate (broker 0.12.4, agent 0.21.1). Lessons in /debugging.md (mock fixtures encoded a plan assumption).
  • Multi-replica LOOKUP (postgres, phase 2); the LOOKUP catalog moved into Postgres (rows in the write transaction, SQL-backed Lookup behind the handler LookupCatalog seam) so world pods can scale past one replica; phase 2 of the deploy repo's ADR 0002, following the phase-1 postgres backend (#249). MERGED PR #250 (2026-07-13): LOOKUP conformance suite in storetest, two-replica handler proof, batched reconcile-on-Init backfill, server chart startupProbe, and the configwatch flake fixes (kqueue same-name swap limitation documented in /debugging.md).
  • Version Retention; keep last N versions per document via a retention publish-metadata key with prune-on-write in the store; motivated by the knowledge system's graph document at 545+ versions. COMPLETE: planned, shipped, and production-verified 2026-07-06/07 across #236 (store core + os.Root delete hardening + audit logging + SPEC §9.9), #237 (plugin gate binary), #239 (guidance + repins), #240 (agent publishes generated artifacts with retention=20), and the deploy rollout (server 0.20.0 / broker 0.9.0 / agent 0.19.0). Live result: /graph.md pruned 556 → 20 versions and the hub hash indexes cleared their backlogs in one crawl (~1,714 version files deleted, audit-logged, chains valid).
  • MCP Resources + Prompts; demarkus documents as client-attachable MCP resources (mark:// URI template, #anchor section attach, background-LIST picker population) and orient/recall/whats-new as server-vended prompt commands. SHIPPED PR #232 (2026-07-05), client/v0.17.0. Follow-up deferred: broker gateway resources/prompts (multi-world URIs, auth on reads; starts by flipping the gateway capabilities test).
  • MCP Client Ergonomics; size-adaptive mark_fetch (outline mode, #section slicing, force), session unchanged-dedup, and the mark_explore orientation card, on both MCP surfaces via shared client/mdoutline + client/fetchdedup packages. SHIPPED #225/#230 and deployed 2026-07-04/05; plugin users (client v0.15.0 via tools 0.4.1) and the live knowledge system (broker 0.5.0). Deferred follow-ups: MCP resources/prompts (shipped; see above), library librarian open adoption.
  • Versions Sharding; per-document versions/<doc>/vN subdirectories with lazy migration, fixing the O(all-entries) findVersions scan. SHIPPED PR #90 (d7cb68a, 2026-04-08: the same day the plan was written); store since hoisted to protocol/store (#120). This index wrongly listed it as unstarted until 2026-07-05.
  • Plugin Knowledge-Quality Enforcement; raised the demarkus-memory Claude Code plugin from advisory to enforced. SHIPPED v0.4.0, PR #171 merged 2026-06-01. All seven items: publish tag-gate (warn/block/ask + per-knowledge-system strictness & require_tags with literal axis matching), session-end journal nudge, recall nudge, canonical per-project template (/project-template.md), knowledge-system policy/template at the live root hub on knowledge.demarkus.io, and the /soul-doctor hygiene audit. 68 tests, pure awk/bash, zero runtime deps. Tail (separate): plugin shell tests → CI; optional nudge disable knobs.
  • Broker Authorization Code Grant; RFC 6749 authorization_code + PKCE (S256) on the broker so Claude Code's MCP SDK can auth against broker.knowledge.demarkus.io. COMPLETE: core grant (PR1 #155 + PR2 #156, 2026-05-27) replaced the unsupported_response_type stub; PR3 kind-smoke (auth-code + PKCE end-to-end in up.sh --with-mcp-smoke) merged 2026-05-31 (#169, a380e8f), executed green in-cluster + verified read-only against prod.
  • LOOKUP verb: the card-catalog verb (subject → docs + importance). Shipped to main PR #166 (2026-05-30); plugin surfacing in v0.3.0 (#168). Tail: mark_append metadata deferred by design.
  • Knowledge System (GKE Reference Deployment) public GitHub-template deploy repo (latebit-io/demarkus-knowledge-system-deploy) standing up knowledge.demarkus.io on GKE (OpenTofu + ArgoCD + OpenBao + bank-vaults + CSI-snapshot backups). Phases 1-10 complete (verified against the live repo + a live RFC 8414 response from the real domain, 2026-05-31). Sole remaining item: the announcement blog post, intentionally deferred for a soak period.
  • Universe Onboarding; last-mile join flow. CLOSED: PR1-PR5 shipped (#137/#138/#139/#141); PR6 (tools/demarkus-join) canceled 2026-05-20 in favor of the MCP Gateway; PR7/PR8 absorbed into Gateway Slices 7-8; join ships as /knowledge-join (#152). Remaining: low-priority doc debt only (two standalone deployment docs).
  • Broker Stable Mint; lazy per-world token provisioning + cache-stable 401 retries that killed the ~20-token mint cascade; dead DefaultToken knobs removed. COMPLETE (#158/#159/#163/#164/#165, 2026-05-27→29).
  • Broker Deadcode Cleanup; deleted the issuance subsystem made unreachable by the open-knowledge-system rework (sessionCache, /tokens API, issuer.go, sweeper trim). COMPLETE (#159 + #164, commit f9a24e9).
  • Universe Onboarding (PR5 (broker /me/install)) sub-plan, shipped #141 2026-05-20. Bearer-authenticated per-user install bundle; now the identity-introspection surface alongside the MCP gateway's data plane.
  • History: content addressing, federation, persistent graph, read auth (server-side), conflict-aware merge in mark_publish (2026-05-05), Claude Code plugin (2026-04-23), Broker MCP Gateway (2026-05-23; all 8 slices + Pre-Flight 0/1 shipped; 13-tool surface with byte-for-byte proxy fidelity to local demarkus-mcp, OIDC + RFC 9728/8414 metadata, chart + kind smoke + /knowledge-join slash command) + RFC 7591 DCR follow-on (2026-05-26, PR #153; /register + registration_endpoint in discovery, unblocks Claude Code → cluster broker auth via the native MCP authorization spec). Also: OKF type adoption + /soul-join managed remote souls (2026-06).

Plan Archives

Original plan documents preserved for reference:

  • Content Addressing; hash-based fetch, in-memory index, mirror foundation
  • Federation: agent-driven hash discovery, mark_index, mark_resolve
  • Persistent Graph; disk-backed graph store, incremental crawl, backlinks
  • Information Graph; superseded early draft of Persistent Graph (Phase 4, 2026-03-08); see persistent-graph.md for the version that shipped.
  • Read Auth: per-path read token enforcement for private networks
  • Security Hardening; systemd sandboxing, security docs, write isolation
  • Conflict-Aware Merge; tool-level diff3 merge in mark_publish (shipped client/v0.12.25 + v0.12.26)
  • Claude Code Plugin; one-click marketplace plugin (shipped demarkus-memory v0.1.1; v0.2.0 added /knowledge-join 2026-05-23; v0.3.0 added self-documenting guidance + lookup recall 2026-05-31; v0.4.0 enforcement + template + /soul-doctor shipped 2026-06-01, PR #171; v0.5.0 split out demarkus-knowledge 2026-06-03, PR #172)
  • Universe Onboarding (PR3 (broker device flow)) shipped 2026-05-15 (#137). RFC 8628 device flow end-to-end on the broker. Six sub-steps merged across one PR; PR4 builds on top.
  • Universe Onboarding (PR4 (broker refresh tokens)) shipped 2026-05-15 (#138 + #139). Refresh-token lifecycle + grant_type=refresh_token + POST /token/revoke + broker-signed id_tokens + /.well-known/jwks.json + compositeVerifier + Sweeper integration. Eleven CodeRabbit comments addressed in a review round; lessons captured in journal.
  • Broker MCP Gateway; shipped 2026-05-23 (v7). Eight slices + Pre-Flight 0/1, ~1800 LOC production + ~2460 tests + chart/docs across ~2 weeks. Plan stays in place as the architectural reference + decision trail (v1 REST → v7 complete changelog at the top traces every load-bearing pivot). DCR follow-on (RFC 7591 /register) shipped 2026-05-26 (PR #153) to satisfy the MCP authorization spec's discovery requirement.
  • Search Verb: superseded by LOOKUP. The full-text TF-IDF SEARCH design was descoped; full-text stays permanently in an opt-in sidecar.
  • POC Deployment; canceled. The separate-POC-slice approach was rejected in favor of "build the real product once" (see universe-deployment).
  • Obsidian Plugin; obsolete. Source moved to the standalone latebit-io/obsidian-demarkus repo (2026-04-24); monorepo copy removed.

Prompt Consolidation

  • Plugin Prompt Source of Truth; consolidate 54 distributed memory and knowledge prompt files into 18 canonical prompt sources seeded from Claude Code prose, then render checked-in artifacts for Claude Code, Pi, and OpenCode. Drafted 2026-08-20; implementation not started.

Prompt Consolidation Update

  • Plugin Prompt Source of Truth was implemented on branch feat/plugin-prompt-source on 2026-08-20. The six agent plugins now render 54 runtime prompt artifacts from 18 canonical templates, with CI and pre-commit drift checks plus explicit OpenClaw and foreign-harness isolation.

Knowledge Server

  • Multi-world Knowledge Server; one replicated demarkus-knowledge-server process hosts multiple logically isolated worlds behind shared UDP 6309 using DNS authorities and TLS SNI. GCS-first, one bucket per world, broker and agent remain separate, per-world capability tokens, ACLs, publish policy, limits, backup, and restore. Planned 2026-08-21; supersedes the earlier bucket backend plan. Implementation starts with ADRs and a 100,000-document real-GCS feasibility spike.
soul.demarkus.io/plans/universe-onboarding-pr5.md complete reader meta

Plan: Universe Onboarding — PR5 (/me/install)

Sub-plan for plan §PR5 of /plans/universe-onboarding.md. Picked up after PR4 (#139, broker refresh tokens + JWKS) merged 2026-05-15. Ready to start cold next session: this doc plus /plans/universe-onboarding.md + /journal/2026-05-15.md is sufficient context.

Goal

After PR5 lands, an authenticated client (the plugin, tools/demarkus-join, or curl with a bearer) can GET a per-user install bundle from the broker:

GET /me/install
Authorization: Bearer <id_token>

200 OK
Content-Type: application/json
Cache-Control: no-store
{
  "worlds": [
    {
      "name": "team-a",
      "publicURL": "mark://world-a.cluster.local:6309",
      "label": "usr_b23fbc20",
      "accessToken": "<raw-token>",
      "expiresAt": "2026-05-16T18:00:00Z"
    }
  ]
}

The bearer is accepted by PR4's compositeVerifier — broker-signed (refresh-renewed) or IdP-signed (device-code completion). Both work; both land in requireAuth → claimsFromCtx.

The install bundle:

  • Excludes worlds where WorldConfig.PublicURL == "" (operator marked them un-installable).
  • Mints a FRESH token per world on every call (raw tokens are never re-derivable from stored hashes; old tokens stay valid until expiry; Sweeper retires them).
  • Returns 200 + worlds: [] (not 403) when the user is authenticated but no worlds authorize them — the plugin surfaces that as "no worlds authorized for your identity" rather than an auth error.

PR6 (tools/demarkus-join) consumes the JSON. PR7's slash command (/soul-join) drives that binary. PR8 documents the surface.

Non-Negotiables Inherited

  • No protocol changes. Same as the parent plan.
  • No demarkus-server changes. All work in tools/demarkus-broker/.
  • No MVPs, no shortcuts. Auth correctness, partial-failure handling, no-store cache header, deterministic ordering of worlds in the response — all from day one.
  • Single broker. /me/install is broker-scoped; multi-broker is out of scope.

Out of Scope (explicit, for PR5 specifically)

  • Shell-script content negotiation. The parent plan mentions text/x-shellscript for curl … | sh users. Defer: ships PR6 (tools/demarkus-join) which obviates the shell-script path for the primary user persona (plugin). Re-evaluate when a real non-plugin terminal user asks. Embedding the bearer + minted tokens in a one-shot shell body has its own security/UX shape worth resolving separately.
  • Universe-level metadata ({universe: {name, brokerURL}} from the parent plan's response sketch). The broker has no universe.name config today; adding one is a separate PR. Plugin derives a display slug from the broker hostname (acmecorp from demarkus.acmecorp.com) per parent plan §"Per-world MCP entry naming collision."
  • POST /me/install (idempotency-token style). GET with the side effect of minting is the parent-plan-locked shape and matches OAuth /me-style endpoints. Document the side-effect explicitly in the handler doc comment; defer POST until a customer wants idempotency tokens.
  • Reuse of existing un-expired issuances. Raw tokens are never recoverable post-mint (we only store hashes), so "reuse" is mechanically impossible — every /me/install call mints fresh. Old tokens stay valid until ExpiresAt; Sweeper retires them.
  • /me/install for plugins that aren't yet wired (Cursor, Aider). Same JSON surface works for any agent; per-agent shell wrappers are separate plans.

Architecture

┌──────────────┐  GET /me/install + Bearer  ┌────────────┐
│  client      │ ─────────────────────────► │            │
│ (plugin,     │ ◄───────────────────────── │   broker   │
│  demarkus-   │   {worlds:[{name,publicURL,│            │
│  join, curl) │    label,accessToken,      │   requireAuth  → claimsFromCtx
│              │    expiresAt}]}            │   subjectRateLimit
│              │                            │   meInstall
└──────────────┘                            │      │
                                            │      └─► Issuer.MintFiltered(
                                            │           claims,
                                            │           func(w) bool { return w.PublicURL != "" })
                                            │              │
                                            │              ├─► world A Secret (token hash)
                                            │              ├─► world B Secret (token hash)
                                            │              └─► issuances Secret (per-world record)
                                            └────────────┘

Layer responsibilities

Component Owns Does NOT own
meInstall handler Translating verified claims into the install-bundle response. Filtering out PublicURL-less worlds in the response shape (the mint-side filter via MintFiltered keeps the issuances Secret clean). Partial-failure surfacing. Cache-Control header. Authentication (delegated to requireAuth). Rate limiting (delegated to subjectRateLimit). Mint mechanics.
Issuer.MintFiltered (extracted from existing Mint) Same shape as Mint but takes an optional keep func(*WorldConfig) bool predicate that runs after authorizedWorlds and before per-world mint. Knowing what /me/install means semantically.
Existing Issuer.Mint Now a thin wrapper: MintFiltered(ctx, claims, nil). Caller compatibility preserved for /auth/callback. New behavior — purely a back-compat shim.

Pre-Flight Tasks

None. PR4's compositeVerifier already accepts broker-signed bearers (the refresh-renewed token PR5 needs to verify), and requireAuth was wired in pre-PR4. PR5 builds directly on those surfaces.

Routes To Register / Modify

In server.Routes():

Method Path Middleware Notes
GET /me/install requireAuth → subjectRateLimit Same composition as the /tokens routes. Returns per-user install bundle.

No modifications to other routes.

Sub-Tasks (sequenced)

Step 1 — Issuer.MintFiltered (~40 lines + ~80 tests)

  • File: tools/demarkus-broker/internal/broker/issuer.go.
  • Extract the world-iteration body of Mint into a new method: 
    // MintFiltered mints tokens for authorized worlds where keep
// returns true. A nil keep is "accept all" — equivalent to Mint.
// /me/install passes a PublicURL-based filter so the issuances
// Secret is not polluted with worlds the install bundle would
// then drop anyway.
func (i *Issuer) MintFiltered(ctx context.Context, claims Claims, keep func(*WorldConfig) bool) ([]MintResult, error)
  • Refactor Mint to delegate: return i.MintFiltered(ctx, claims, nil). Existing call sites (/auth/callback) keep working unchanged.
  • Tests in issuer_test.go:
    • MintFiltered with nil predicate is equivalent to Mint (regression guard).
    • MintFiltered with predicate skips filtered worlds (no issuance Secret entry written for those).
    • MintFiltered returns ErrNotAuthorized when the predicate filters every authorized world to zero.
    • MintFiltered partial-failure path: predicate matches multiple worlds, one mint fails, returns partial + wrapped error.

Step 2 — meInstall handler (~80 lines + ~200 tests)

  • File: tools/demarkus-broker/internal/broker/install.go (new).
  • Response struct: 
    type installResponse struct {
    Worlds         []installWorld `json:"worlds"`
    PartialFailure string         `json:"partialFailure,omitempty"`
}
type installWorld struct {
    Name        string    `json:"name"`
    PublicURL   string    `json:"publicURL"`
    Label       string    `json:"label"`
    AccessToken string    `json:"accessToken"`
    ExpiresAt   time.Time `json:"expiresAt"`
}
  • Handler shape:
    • Read claims from ctx (set by requireAuth).
    • Call s.issuer.MintFiltered(ctx, claims, func(w *WorldConfig) bool { return w.PublicURL != "" }).
    • On ErrNotAuthorized: return 200 with worlds: [] (not 403 — the user IS authenticated; the absence of worlds is an authz-config story, not an auth-fail story). Log at INFO.
    • On ErrEmailUnverified: 403 (existing convention; the production Verifier rejects unverified before this branch, but the defense-in-depth shape stays).
    • On partial-mint (some succeeded, one failed): return 200 with the successful worlds + partialFailure field. Same shape as /auth/callback to keep the surface consistent across the two mint paths.
    • On hard mint failure (zero results): 500.
    • On success: 200 with the filtered worlds. Cache-Control: no-store + Pragma: no-cache (bearer tokens in the body).
  • Lookup of WorldConfig.PublicURL per result: walk s.cfg.Worlds and match by Name. O(N*M) is fine for the world counts in scope (~10s of worlds, called maybe once per session per user).
  • Tests in install_test.go:
    • Happy path: 1 authorized world with PublicURL → 200 + 1 world in response with all four fields populated.
    • Multi-world happy path: 2 authorized worlds → 200 + 2 worlds, deterministic ordering (Issuer.MintFiltered iterates cfg.Worlds in declaration order).
    • PublicURL-less world filtered out: 2 authorized worlds, one with no PublicURL → 200 + 1 world (the one with PublicURL), no issuance written for the filtered world (assert Secret state).
    • Unauthenticated request → 401 (via requireAuth — already covered upstream but a smoke test confirms wiring).
    • No-bearer / bad-bearer → 401.
    • User authenticated, zero authorized worlds → 200 + worlds: [] (NOT 403).
    • ErrEmailUnverified → 403.
    • Partial-mint failure → 200 + partialFailure field + the worlds that succeeded.
    • Hard mint failure (e.g., RBAC denied on the issuances Secret) → 500.
    • Response includes Cache-Control: no-store + Pragma: no-cache.
    • Bearer-signed by broker (PR4 refresh-renewed) is accepted (regression guard against PR4's compositeVerifier dispatch).
    • Bearer-signed by IdP is accepted (regression guard for the device-code-completion path).

Step 3 — Route registration (~5 lines)

  • File: tools/demarkus-broker/internal/broker/server.go.
  • Add to Routes(): 
    mux.Handle("GET /me/install", authedSubject(s.meInstall))
    where authedSubject is the existing requireAuth → subjectRateLimit composition used by the /tokens routes.
  • No new helpers needed.

Step 4 — Documentation

  • deploy/helm/demarkus-broker/README.md: add a section under "Endpoint surface" (or the equivalent existing section) describing /me/install. Schema, auth, no-store posture, the "PublicURL-less worlds are excluded" rule.
  • tools/demarkus-broker/main.go package doc: bump the "Current scope" comment to mention /me/install.

Scope Estimate

Step Code Tests
1. Issuer.MintFiltered 40 80
2. meInstall handler 80 200
3. Route registration 5 0
4. Documentation 30 0
Total ~155 ~280

Parent plan §PR5 estimated "~300 lines + tests, ~1 day PR." Revised down to ~155 LOC because compositeVerifier (PR4) already handles bearer-token verification end-to-end, including the broker-signed leg; PR5 is genuinely just a handler that wraps Issuer.MintFiltered. Tests are the bulk; ~280 lines for the full matrix.

Open Questions To Resolve Before/During PR5

  1. GET vs POST. Lean: GET (parent-plan-locked). Document the side-effect (mint) in the handler doc comment so a future reader doesn't expect REST-idempotent semantics. Re-confirm before starting if the side-effect bothers Fritz.
  2. Empty authorized worlds: 200-empty or 403. Lean: 200 with worlds: []. Rationale: a 403 confuses the plugin layer (can't distinguish auth failure from authz emptiness). Re-confirm.
  3. PublicURL filter in MintFiltered vs post-Mint. Lean: pre-Mint (MintFiltered). Post-Mint wastes one issuance per filtered world per call; over 90 days of refresh ticks this fills the 5000-record Secret cap unnecessarily. Confirmed worth the +40 LOC.
  4. label field in response. Lean: include it. The plugin doesn't strictly need it (the access token is enough for connection), but exposing it makes /tokens/{label}/rotate and DELETE /tokens/{label} actionable from the install bundle without a second call. ~0 LOC cost; defensive.
  5. Stable ordering of worlds in response. Lean: cfg.Worlds declaration order (what MintFiltered naturally produces). A client that wants alphabetical can sort client-side. Avoids surprise reordering across runs.
  6. Response Content-Type. Lean: always application/json. No content-negotiation in PR5; non-JSON consumers (shell-script) are deferred per §Out of Scope.
  7. Per-world health check before issuing. Should the broker verify each target world's Secret is writable before returning success? Lean: no — Issuer.Mint already performs the write and returns partial-failure on RBAC/Secret issues. Pre-flight check would double the latency for zero behavioral gain.
  8. Logging. Log at INFO on success (subject hash + world count + partial-failure flag), at WARN on partial, at ERROR on hard failure. Same posture as /auth/callback. Subject hash via existing hashSubject helper — no raw email or tokens in logs.

Risks Specific To PR5

  • Mint contention under refresh-storm. Every plugin session start may call /me/install. A multi-world broker with many concurrent users will hit mutateSecret's optimistic-concurrency retry on the issuances Secret. Mitigation: same Secret + same retry budget as PR4's refresh path; if a real customer hits the wall, the fix is sharded Secrets or a CRD-backed store (already documented as the phase-7 path in PR4's risks).
  • Issuance bloat at the 5000-record cap. A user with 5 authorized worlds hitting /me/install once per session at 100 sessions/day = 500 issuances/day. With 24h default token TTL, the Sweeper catches them within a day. With 90-day TTL... different story. PR4 already documented this cap; PR5 inherits it. Mitigation: shorter access-token TTLs for high-throughput deployments; chart value worlds[].defaultToken.expiresAfter is operator-tunable.
  • Bearer-token in response body. If the response is somehow logged or proxied to an untrusted intermediary, raw tokens leak. Mitigation: Cache-Control: no-store + Pragma: no-cache headers; document the no-log-the-body posture in the broker's deployment doc.
  • PublicURL filter coupling. PR5 introduces the convention that "world without PublicURL is un-installable." Any future feature that wants to operate on un-installable worlds (e.g., admin-only worlds) needs a different filter shape. Mitigation: MintFiltered's keep is parametric — future surfaces compose their own predicate without touching the install-side filter.
  • Same-user concurrent /me/install calls. Two simultaneous calls produce two sets of issuances, both valid. Acceptable per the broker's existing posture, but a flag-prone user could fill the issuances Secret with double-minted tokens fast. Mitigation: subjectRateLimit (10/min default, shared bucket with the /tokens routes) caps the burst.

Next-Session Resume Steps

  1. git fetch && git log --oneline -5 — confirm PR4 (#139) on main, no conflicts. If PR4 review CI fixup (fix-ci-broker-signing-key-comment) also landed, even better.
  2. mark_fetch /index.md + /patterns.md + /guidelines.md per project preflight.
  3. mark_fetch /plans/universe-onboarding-pr5.md (this doc).
  4. mark_fetch /journal/2026-05-15.md for PR4 review-lessons context (workflow-YAML ${{ }} quirk, parse-at-validate, RETURN-trap pattern, ephemeral test PEMs).
  5. Decide on Open Questions 1 (GET vs POST) + 2 (empty-worlds → 200 or 403) — both have a lean but worth one-sentence confirmation before starting.
  6. Cut a fresh branch (feat-tools-broker-me-install or similar). Start at Step 1 (Issuer.MintFiltered) as its own commit so the rest builds on a green refactor baseline.
  7. After Step 1: go test -race + bash pre-commit.sh green before touching Step 2. The refactor is small but Mint is load-bearing for /auth/callback; regression guard tests pay for themselves.

Touch Points With Later PRs

  • PR6 (tools/demarkus-join) consumes the JSON response. The binary's internal/install/ package decodes the shape PR5 emits and drives claude mcp add per world. PR5's stable field names (name, publicURL, label, accessToken, expiresAt) are the wire contract.
  • PR7 (plugin slash commands + kind Stage 5) invokes tools/demarkus-join which hits /me/install. The kind harness Stage 5 (if mock-oauth2-server device code support pans out — parent plan §Open Question 1) validates the full chain.
  • PR8 (docs) covers the operator-facing "what does /me/install look like, what's in it, what's the no-PublicURL rule" story.

Done When

  • PR5 opens with all four sub-steps' commits, each individually testable.
  • go test -race ./... green inside tools/demarkus-broker/.
  • helm unittest . green (no chart changes expected; smoke-test pass confirms PR5 didn't accidentally touch the chart).
  • pre-commit.sh green.
  • Manual end-to-end via curl: device-flow completes → curl /me/install with the resulting bearer → 200 + per-world bundle.
  • Journal entry on /journal/<date>.md with any design decisions that landed differently from this plan.

Implementation Status (2026-05-20)

Implemented; PR pending review. Branch feat-tools-broker-me-install on local working tree. Fritz handles the commit + PR open per /patterns.md.

  • ✅ Step 1: Issuer.MintFiltered extracted; Mint is now a back-compat wrapper. +6 tests (regression guard, predicate skip, reject-all, partial-failure, predicate-not-run-on-unauthorized, predicate-sees-authorized-worlds-only).
  • ✅ Step 2: meInstall handler in tools/demarkus-broker/internal/broker/install.go. +12 tests covering happy path, multi-world ordering, PublicURL filter, no-bearer / bad-bearer 401, empty-worlds 200+[], all-worlds-filtered 200+[], unverified 403, partial failure, hard failure, Cache-Control + Pragma headers, broker-signed bearer regression guard.
  • ✅ Step 3: Route registered as GET /me/install behind requireAuth → subjectRateLimit.
  • ✅ Step 4: deploy/helm/demarkus-broker/README.md gained a full "Endpoint surface" table + /me/install subsection; tools/demarkus-broker/main.go package-doc Current-scope comment updated.

Open Questions resolved at implementation time

  1. GET vs POST: GET. Side-effect documented in the handler doc comment.
  2. Empty authorized worlds → 200 vs 403: 200 + worlds: []. 3-8. All other open questions adopted the leans documented above with no deviation.

Verification at handoff

  • go test -race -count=1 ./internal/broker/ → green (+18 tests over PR4 baseline).
  • helm unittest . → 72/72 pass (unchanged from PR4 baseline; chart not touched).
  • bash pre-commit.sh → format / vet / lint clean across protocol/server/client/tools.

Final scope

Step Plan code Actual code Plan tests Actual tests
1. MintFiltered 40 42 80 209
2. meInstall + install.go 80 ~115 200 ~365
3. Route registration 5 8 0 0
4. Documentation 30 ~72 0 0
Total ~155 ~237 ~280 ~574

Code came in slightly over plan; tests came in ~2× plan. Test density is from the partial-failure + Cache-Control + composite-verifier-bearer matrix being denser than the plan accounted for, and from PR5 isolating its own fixtures (installTestConfig, installTestConfigTwoWorlds) rather than sharing issuer_test.go's testConfig. Worth it: every test pins a distinct invariant.

Session journal

See /journal/2026-05-20.md for the full session writeup including design decisions that landed differently from the plan (the predicate-runs-after-AllowConfig contract, the tolerant toInstallWorlds lookup, the reuse of fakeVerifier's existing verifyFn hook).

trail
  1. soul.demarkus.io v93
  2. universe-onboarding-pr5
from here → index 2026-05-20 broker-auth-code-grant broker-https-gateway universe-onboarding-pr4