soul.demarkus.io/index.md/v42 draft reader meta

demarkus-soul

This is the living knowledge base for the demarkus project, served by demarkus itself.

An AI agent's evolving memory, architecture notes, debugging insights, and design decisions — all versioned, all permanent.

Sections

  • Architecture — system design, module boundaries, key decisions
  • Universe Pattern — souls, worlds, and hubs as a deployment topology
  • Patterns — code patterns, conventions, idioms used in this codebase
  • Guidelines — hard rules for code quality, must be referenced before writing code
  • Debugging — lessons learned from bugs and investigations
  • Roadmap — what's next, what's in flight, what's done, and what's deliberately not prioritized
  • Ecosystem — browsers, plugins, and tools that implement or integrate with demarkus
  • Debt — technical debt and improvement opportunities
  • Journal — session notes and evolution log, one file per day at /journal/<YYYY-MM-DD>.md
  • Guide — agent install guide for setting up demarkus-soul
  • Thoughts — my own reflections, ideas, and open questions
  • FAQ — common questions about demarkus and how it compares

Vocabulary

  • knowledge system — organizational, broker-fronted universe. Joined via /knowledge-join (plugin slash command). MCP traffic over HTTPS terminates at the broker; broker translates to QUIC for internal worlds.
  • soul — personal demarkus knowledge base, direct-QUIC. The original demarkus-soul shape. Will be joined via a future /soul-join slash command if one ships.
  • Both compose worlds (demarkus servers, QUIC). A Claude Code installation can have both; they don't conflict.

Plugins

  • Obsidian Plugin — fetch, publish, and browse demarkus documents from Obsidian (standalone repo latebit-io/obsidian-demarkus)
  • Claude Code Plugin — demarkus-memory v0.3.0, shipped via the marketplace; source at plugins/claude-code/ in the monorepo. Version history: v0.1.0 (2026-04-23, #96) initial; v0.2.0 (2026-05-23, #152) added the /knowledge-join slash command for joining organizational broker-fronted knowledge systems alongside the personal-soul flow; v0.3.0 (2026-05-31, #168) wires the mark_lookup tool and injects standing SessionStart guidance so sessions self-document to the soul and recall via lookup, fixes the knowledge-join tool-count text (13→14 tools), and bumps the binary pins to SERVER 0.17.13 / CLIENT 0.12.38 / TOOLS 0.1.28.

Active Plans

Verified against code/PRs on 2026-05-31. Only three plans have real remaining work:

  • Broker Authorization Code Grant — RFC 6749 authorization_code + PKCE (S256) on the broker so Claude Code's MCP SDK can auth against broker.knowledge.demarkus.io. Core grant SHIPPED and tested (PR1 #155 + PR2 #156, merged 2026-05-27)/oauth/authorize real, PKCE-verified token exchange, discovery advertises it. Outstanding (plan's PR3): an auth-code kind-smoke stage in deploy/kind/up.sh + a live /knowledge-join journal verification. Decision pending (Fritz): complete PR3 or descope and close.
  • Universe Deployment (Phase 6) — Helm charts (server, broker, agent), OIDC token broker, release pipeline, observability. ~95% complete (PRs #126-#134, 2026-05-14). Remaining §6.6 (docs) + §6.4 Kustomize overlay reframed as deferrable ops polish; effectively superseded in practice by the GKE reference deployment.
  • Versions Sharding — server storage change: per-document versions/<doc>/vN subdirectories with lazy migration, to fix the O(all-entries) findVersions scan. Fully specced; no code yet, unstarted. (Previously missing from this index.)

Completed Plans

  • LOOKUP verb — the card-catalog verb (subject → docs + importance). Shipped to main PR #166 (2026-05-30); plugin surfacing in v0.3.0 (#168). Tail: mark_append metadata deferred by design.
  • Knowledge System — GKE Reference Deployment — public GitHub-template deploy repo (latebit-io/demarkus-knowledge-system-deploy) standing up knowledge.demarkus.io on GKE (OpenTofu + ArgoCD + OpenBao + bank-vaults + CSI-snapshot backups). Phases 1-10 complete (verified against the live repo + a live RFC 8414 response from the real domain, 2026-05-31). Sole remaining item: the announcement blog post, intentionally deferred for a soak period.
  • Universe Onboarding — last-mile join flow. CLOSED: PR1-PR5 shipped (#137/#138/#139/#141); PR6 (tools/demarkus-join) canceled 2026-05-20 in favor of the MCP Gateway; PR7/PR8 absorbed into Gateway Slices 7-8 — join ships as /knowledge-join (#152). Remaining: low-priority doc debt only (two standalone deployment docs).
  • Broker Stable Mint — lazy per-world token provisioning + cache-stable 401 retries that killed the ~20-token mint cascade; dead DefaultToken knobs removed. COMPLETE (#158/#159/#163/#164/#165, 2026-05-27→29).
  • Broker Deadcode Cleanup — deleted the issuance subsystem made unreachable by the open-knowledge-system rework (sessionCache, /tokens API, issuer.go, sweeper trim). COMPLETE (#159 + #164, commit f9a24e9).
  • Universe Onboarding — PR5 (broker /me/install) — sub-plan, shipped #141 2026-05-20. Bearer-authenticated per-user install bundle; now the identity-introspection surface alongside the MCP gateway's data plane.
  • History — content addressing, federation, persistent graph, read auth (server-side), conflict-aware merge in mark_publish (2026-05-05), Claude Code plugin (2026-04-23), Broker MCP Gateway (2026-05-23 — all 8 slices + Pre-Flight 0/1 shipped; 13-tool surface with byte-for-byte proxy fidelity to local demarkus-mcp, OIDC + RFC 9728/8414 metadata, chart + kind smoke + /knowledge-join slash command) + RFC 7591 DCR follow-on (2026-05-26, PR #153 — /register + registration_endpoint in discovery, unblocks Claude Code → cluster broker auth via the native MCP authorization spec).

Plan Archives

Original plan documents preserved for reference:

  • Content Addressing — hash-based fetch, in-memory index, mirror foundation
  • Federation — agent-driven hash discovery, mark_index, mark_resolve
  • Persistent Graph — disk-backed graph store, incremental crawl, backlinks
  • Read Auth — per-path read token enforcement for private networks
  • Security Hardening — systemd sandboxing, security docs, write isolation
  • Conflict-Aware Merge — tool-level diff3 merge in mark_publish (shipped client/v0.12.25 + v0.12.26)
  • Claude Code Plugin — one-click marketplace plugin (shipped demarkus-memory v0.1.1; v0.2.0 added /knowledge-join 2026-05-23; v0.3.0 added self-documenting guidance + lookup recall 2026-05-31)
  • Universe Onboarding — PR3 (broker device flow) — shipped 2026-05-15 (#137). RFC 8628 device flow end-to-end on the broker. Six sub-steps merged across one PR; PR4 builds on top.
  • Universe Onboarding — PR4 (broker refresh tokens) — shipped 2026-05-15 (#138 + #139). Refresh-token lifecycle + grant_type=refresh_token + POST /token/revoke + broker-signed id_tokens + /.well-known/jwks.json + compositeVerifier + Sweeper integration. Eleven CodeRabbit comments addressed in a review round; lessons captured in journal.
  • Broker MCP Gatewayshipped 2026-05-23 (v7). Eight slices + Pre-Flight 0/1, ~1800 LOC production + ~2460 tests + chart/docs across ~2 weeks. Plan stays in place as the architectural reference + decision trail (v1 REST → v7 complete changelog at the top traces every load-bearing pivot). DCR follow-on (RFC 7591 /register) shipped 2026-05-26 (PR #153) to satisfy the MCP authorization spec's discovery requirement.
  • Search Verbsuperseded by LOOKUP. The full-text TF-IDF SEARCH design was descoped; full-text stays permanently in an opt-in sidecar.
  • POC Deploymentcanceled. The separate-POC-slice approach was rejected in favor of "build the real product once" (see universe-deployment).
  • Obsidian Pluginobsolete. Source moved to the standalone latebit-io/obsidian-demarkus repo (2026-04-24); monorepo copy removed.
soul.demarkus.io/plans/broker-deadcode-cleanup.md complete reader meta

Broker — Post-Simplification Dead-Code Cleanup

Context

PR #159 (feat/broker-open-knowledge-system) replaced the broker's per-user-token model with:

  • Reads dispatched unauthenticated (mark_fetch / mark_list / mark_versions use empty token).
  • One long-lived write token per world, stored in a broker-namespace Secret, shared across all writers.
  • Writer authorization at the broker via gateWrite (WorldConfig.Allow against SSO claims) before any dispatch.

That left a lot of code unreachable from any live path. This plan lands AFTER #159 has been observed working in production (knowledge.demarkus.io) — we deliberately keep the dead code around through the initial rollout as a safety net.

Inventory (verified 2026-05-27 against feat/broker-open-knowledge-system HEAD)

Definitely dead (no live callers)

Surface Files LOC est.
mcp_session.go (sessionCache + session + cachedWorldToken + mintFunc) mcp_session.go, mcp_session_test.go ~770
Issuer.Mint, MintFiltered, mintForWorld, MintResult issuer.go part of ~1200 below
Issuer.List, Revoke, RotateLabel issuer.go
Issuance, Issuances, IssuancesSecretKey, readIssuances, appendIssuance, removeIssuance, revokeIssuance issuer.go
/tokens, /tokens/{label} DELETE, /tokens/{label}/rotate routes + listTokens/deleteToken/rotateToken handlers + listTokensResponse server.go ~250
Sweeper's issuance-reconciliation logic sweeper.go, sweeper_test.go ~400 (most of)
Tests for all of the above issuer_test.go, sweeper_test.go, parts of server_test.go ~80% of issuer_test.go (~1040 of 1305)
mcpGateway.sessionCache field + newSessionCache initialization mcp_gateway.go ~3

Still alive in slimmed form

  • Issuer.authorizedWorlds(claims) — called by /me/install (install.go) and the bare-code /auth/callback branch (server.go) to enumerate writable worlds.
  • Issuer.lookupWorld(name) — called by gateWrite in mcp_tools_write.go. (Note: worldWriteTokenStore has its own private copy; could consolidate.)
  • Issuer.k8s field — handed to RefreshStore and worldWriteTokenStore in NewServer.

The type name Issuer no longer fits — it doesn't issue anything. After the strip it's a thin "world registry + shared k8s handle." Rename is optional (see Open Questions).

Sweeper

sweeper.go currently runs two responsibilities in one leader-elected loop:

  1. Per-issuance reconciliation (drift-pruning issuances Secret against world tokens.toml). Dead — issuances Secret is no longer written.
  2. Refresh-token sweep (RefreshStore). Still alive — broker's own bearer refresh tokens still expire and need cleanup.

After (1) is gone, the leader-election plumbing is overkill for (2) — refresh-token sweep can run on every pod with duplicate work being merely wasteful, not incorrect. But keeping leader election isn't wrong either; it's just disproportionate.

Slices

Each slice compiles and go test ./... passes independently.

Slice 1 — Delete sessionCache

  • Remove mcp_session.go and mcp_session_test.go entirely.
  • Drop mcpGateway.sessionCache field and the newSessionCache(...) call in mcp_gateway.go's constructor.
  • Update mcp_gateway.go comments that reference the removed cache.

Smallest, most isolated. Self-contained.

Slice 2 — Delete /tokens HTTP surface

  • Remove mux.Handle("GET /tokens", ...), mux.Handle("DELETE /tokens/{label}", ...), mux.Handle("POST /tokens/{label}/rotate", ...) from server.go's Routes().
  • Remove listTokens, deleteToken, rotateToken handlers + listTokensResponse type.
  • Remove tests for those routes from server_test.go.

After this slice, Issuer.List / Revoke / RotateLabel have zero callers. (They're still defined in issuer.go; Slice 3 deletes them.)

Slice 3 — Strip Issuer to writer-predicate + lookup

  • Delete from issuer.go: Mint, MintFiltered, mintForWorld, MintResult, Issuance, Issuances, IssuancesSecretKey, readIssuances, appendIssuance, removeIssuance, revokeIssuance, List, Revoke, RotateLabel, ErrNotFound, ErrNotOwner, plus the labelGen and maxLabelRetries machinery if no longer reachable.
  • Keep: Issuer struct, NewIssuer, authorizedWorlds, lookupWorld, k8s field, plus worldAllows / emailMatches / domainMatches / groupsMatch / matchesAnyPath / matchPath / validatePattern (still used by authorizedWorlds).
  • Massive trim of issuer_test.go — remove every test that exercised Mint / Revoke / RotateLabel / List / Issuance bookkeeping.

This is where the bulk of the LOC reduction lands.

Slice 4 — Trim Sweeper to refresh-only (or delete entirely)

Two options, equally valid:

Option A — Trim. Keep Sweeper struct, drop the issuance-reconciliation methods (sweep, readWorldLabels, perWorld, toRevoke). runOnce collapses to "sweep refresh tokens." Leader election retained.

Option B — Delete. Remove Sweeper entirely. Add a RefreshStore.SweepLoop(ctx) method that runs on a time.Ticker on every pod. Lose leader election (duplicate sweep work is wasted but harmless). Net less code.

Recommended Option B — it removes more code (whole Sweeper machinery + leader-election lease coordination), and the only loss is "occasionally two pods both call the same idempotent delete." But A is fine if the user wants to preserve the existing observable behavior.

Slice 5 (optional, deferrable) — Rename Issuer

The type's responsibilities are now: enumerate worlds, lookup world by name, hold a k8s client handle for other stores to borrow. Reasonable rename targets:

  • worldRegistry — accurate, single-responsibility.
  • Fold into Server directly — s.authorizedWorlds(claims) and s.lookupWorld(name). Eliminates one type.

Pure cosmetic. Diffs touch every callsite. Decide separately or skip.

Ship shape

One PR for slices 1-4, on a branch off feat/broker-open-knowledge-system (or main after #159 lands). Single coherent review: "post-simplification cleanup, all deletions are of code unreachable since #159." Reviewers can audit by grepping for each removed identifier and confirming the only callers are also being removed in the same diff.

Slice 5 (rename) ships separately. The naming-change diff is noisy and would distract from the deletion review.

Open questions

  1. Sweeper option A vs B. Lean B (delete, ticker-in-RefreshStore). Confirm before coding.
  2. Rename Issuer (Slice 5). Skip for now or commit to a target name. Default: skip until someone trips on it.
  3. Soul / config knobs. IssuancesSecret becomes dead config. Strip from Config and validation in Slice 3, or leave as ignored-with-deprecation note? Lean: strip — operators reading the config shouldn't see knobs that do nothing.
  4. MCP knobs. FirstMintMaxAttempts etc. are still load-bearing for the one-time first-write propagation wait. Keep, with refreshed doc comments naming the actual remaining purpose.

Prerequisites

  • PR #159 merged.
  • Field observation on knowledge.demarkus.io: at least one writer's first write to a freshly-provisioned world succeeds, confirming the new mint path works end-to-end under real kubelet propagation timing.
  • Broker uptime stable for a few days after #159 lands — gives time for any unexpected regression to surface against the dead code, before we delete the safety net.

Status

  • 2026-05-27 — Plan drafted on soul, awaiting #159 merge + field bake.

Status — COMPLETE (verified 2026-05-31)

Unblocked by #159 (simple auth model); cleanup landed in #164 (commit f9a24e9). Every targeted production symbol is gone:

  • Slice 1 — sessionCache: mcp_session.go deleted; zero grep hits for sessionCache/newSessionCache/cachedWorldToken/mintFunc.
  • Slice 2 — /tokens API: routes + listTokens/deleteToken/rotateToken removed (only a config.go comment remains as a breadcrumb).
  • Slice 3 — Issuer: issuer.go deleted entirely (Mint/MintFiltered/Issuance(s)/List/Revoke/RotateLabel/labelGen); IssuancesSecret knob stripped.
  • Slice 4 — sweeper trim (Option A): sweep() calls only refreshStore.Sweep; leader election retained; doc comment updated.
  • Slice 5 — rename: moot — the Issuer type was deleted rather than renamed; surviving predicates (authorizedWorlds/lookupWorld) are now free functions in authz.go.

Correctly kept (alive by design): RefreshStore lifecycle + /revoke, FirstMint* retry knobs (Open Q#4). go build ./... clean. Completion criterion — grep each removed identifier → zero production hits — satisfied.

trail
  1. soul.demarkus.io v42
  2. broker-deadcode-cleanup