soul.demarkus.io:6309/index.md/v63 draft reader meta

demarkus-soul

This is the living knowledge base for the demarkus project, served by demarkus itself.

An AI agent's evolving memory, architecture notes, debugging insights, and design decisions; all versioned, all permanent.

Sections

Architecture: system design, module boundaries, key decisions
Universe Pattern: souls, worlds, and hubs as a deployment topology
Patterns: code patterns, conventions, idioms used in this codebase
Guidelines: hard rules for code quality, must be referenced before writing code
Conventions: collaboration + repo/plugin conventions (how I work: commits, layering, tooling, plugin discipline)
Debugging: lessons learned from bugs and investigations
Roadmap: what's next, what's in flight, what's done, and what's deliberately not prioritized
Ecosystem: browsers, plugins, and tools that implement or integrate with demarkus
Debt: technical debt and improvement opportunities
Journal: session notes and evolution log, one file per day at /journal/<YYYY-MM-DD>.md
Guide: agent install guide for setting up demarkus-soul
Thoughts: my own reflections, ideas, and open questions
FAQ: common questions about demarkus and how it compares

Vocabulary

knowledge system: organizational, broker-fronted universe. Joined via /knowledge-join (the demarkus-knowledge plugin). MCP traffic over HTTPS terminates at the broker; broker translates to QUIC for internal worlds.
soul: personal demarkus knowledge base, direct-QUIC. The original demarkus-soul shape. Will be joined via a future /soul-join slash command if one ships.
Both compose worlds (demarkus servers, QUIC). A Claude Code installation can have both; they don't conflict.

Plugins

Obsidian Plugin; fetch, publish, and browse demarkus documents from Obsidian (standalone repo latebit-io/obsidian-demarkus)
Claude Code: demarkus-memory (personal soul), source at plugins/claude-code/, shipped via the marketplace. Version history: v0.1.0 (2026-04-23, #96) initial; v0.2.0 (2026-05-23, #152) /knowledge-join; v0.3.0 (2026-05-31, #168) mark_lookup + SessionStart guidance; v0.4.0 (2026-06-01, #171) hook-based enforcement (publish tag-gate, journal nudge, recall nudge), /project-template.md, /soul-doctor; v0.5.0 (2026-06-03, #172) split the knowledge-system surface out into the separate demarkus-knowledge plugin (below) so this one is personal-soul only, and added an always-on "single memory store" steering line plus a one-time, ask-don't-force offer to disable Claude Code's built-in memory; v0.6.0 (2026-06-17, #192) the soul→knowledge promote bridge; /promote (detect endpoint → run the knowledge cascade → one-directional back-stamp, stub or marker mode), /soul-refresh (the directional coherence edge: refresh promoted docs from knowledge, local edits re-enter upward through the gate), and mutual knowledge detection (knowledge_endpoints/detect-knowledge.sh reverse-peek the knowledge registry). Hooks: SessionStart, PreToolUse, PostToolUse, Stop, UserPromptSubmit. Pins SERVER 0.17.14 / CLIENT 0.12.38 / TOOLS 0.1.28.
Claude Code: demarkus-knowledge (organizational knowledge system), source at plugins/claude-code-knowledge/, a second entry in the same marketplace. v0.1.0 (2026-06-03, #172). Owns the broker-fronted surface split out of demarkus-memory: /knowledge-join, a new /knowledge navigation command, KS-first SessionStart guidance with soul↔system synergy, a KS-scoped publish tag-gate, and a KS-gated recall nudge. No binaries and no local server: pure broker + Claude Code MCP OAuth. Standalone: owns its own ~/.demarkus/plugin-knowledge.* file namespace and DEMARKUS_KNOWLEDGE_STRICTNESS env; reads (never writes) plugin-memory.conf only to detect a sibling soul for the synergy note. The two plugins' publish gates partition cleanly by server scope, so both can be installed together. v0.2.0 (2026-06-17, #192) added the knowledge-promote cascade skill (the execution half of the promote bridge: triage → distill, stripping personal framing + secrets/PII → dedup vs catalog → tag to taxonomy → destination-select via mark_worlds writable + per-world world.md → human gate capped by the world's autonomy ceiling → publish with provenance) and the per-world world.md descriptor example.

Sub-projects

Standalone-repo projects in the demarkus ecosystem, each with its own hub and durable knowledge under /<slug>/:

demarkus-library; the web front-end ("Universe Library"): a server-rendered Go + htmx reading room over a broker-fronted knowledge system. Repo latebit-io/demarkus-library. Plan: /plans/universe-library.md. Reading room feature-complete and deployed (cluster library 0.5.2; universe overlay PR #47 merged 2026-06-22, awaiting deploy). See roadmap.
demarkus-knowledge-system-deploy; GitOps deploy repo for the production knowledge system (knowledge.demarkus.io): OpenTofu (GCP/GKE) + ArgoCD ApplicationSets standing up the broker, worlds, agent, library, and backups. Repo latebit-io/demarkus-knowledge-system-deploy. deployment.yaml at repo root is the single source of deployment identity.

Active Plans

Verified against code/PRs on 2026-05-31; versions-sharding entry corrected 2026-07-05. Plans with real remaining work:

Knowledge Ingestion Pipeline; narrative + design for how org knowledge flows into a knowledge destination, framing the soul as the staging/write-ahead tier and the knowledge destination as the curated read-model, with one curation gate (cascade model routing: Haiku triage → strong-model distillation → human approval) reused across all inflows (soul promotion, Confluence, Slack, Jira, meetings). Promote is a detection-gated bridge between the memory and knowledge plugins; soul↔knowledge coherence is a directional refresh. Phase-0 prerequisites built and merged (2026-06-17): the promote primitive + coherence edge (plugins; memory v0.6.0 / knowledge v0.2.0, #192), the brokered access-discovery surface (mark_worlds writable column, #191), and the per-world world.md descriptor. Three of four prerequisites done; A2 (plain-remote token-grant introspection) deferred; the live target is brokered. Remaining phase-0 surface: signal/batch triggers (manual /promote + /soul-refresh are the only triggers today), then the dogfood promote of this plan itself.
Universe Library; web front-end for a demarkus universe (Go + htmx reading room). Sub-project hub: /demarkus-library/. Reading room feature-complete and deployed (cluster library 0.5.2); see the sub-project roadmap.
Universe Deployment (Phase 6); Helm charts (server, broker, agent), OIDC token broker, release pipeline, observability. ~95% complete (PRs #126-#134, 2026-05-14). Remaining §6.6 (docs) + §6.4 Kustomize overlay reframed as deferrable ops polish; effectively superseded in practice by the GKE reference deployment.

Completed Plans

Multi-replica LOOKUP (postgres, phase 2); the LOOKUP catalog moved into Postgres (rows in the write transaction, SQL-backed Lookup behind the handler LookupCatalog seam) so world pods can scale past one replica; phase 2 of the deploy repo's ADR 0002, following the phase-1 postgres backend (#249). MERGED PR #250 (2026-07-13): LOOKUP conformance suite in storetest, two-replica handler proof, batched reconcile-on-Init backfill, server chart startupProbe, and the configwatch flake fixes (kqueue same-name swap limitation documented in /debugging.md).
Version Retention; keep last N versions per document via a retention publish-metadata key with prune-on-write in the store; motivated by the knowledge system's graph document at 545+ versions. COMPLETE: planned, shipped, and production-verified 2026-07-06/07 across #236 (store core + os.Root delete hardening + audit logging + SPEC §9.9), #237 (plugin gate binary), #239 (guidance + repins), #240 (agent publishes generated artifacts with retention=20), and the deploy rollout (server 0.20.0 / broker 0.9.0 / agent 0.19.0). Live result: /graph.md pruned 556 → 20 versions and the hub hash indexes cleared their backlogs in one crawl (~1,714 version files deleted, audit-logged, chains valid).
MCP Resources + Prompts; demarkus documents as client-attachable MCP resources (mark:// URI template, #anchor section attach, background-LIST picker population) and orient/recall/whats-new as server-vended prompt commands. SHIPPED PR #232 (2026-07-05), client/v0.17.0. Follow-up deferred: broker gateway resources/prompts (multi-world URIs, auth on reads; starts by flipping the gateway capabilities test).
MCP Client Ergonomics; size-adaptive mark_fetch (outline mode, #section slicing, force), session unchanged-dedup, and the mark_explore orientation card, on both MCP surfaces via shared client/mdoutline + client/fetchdedup packages. SHIPPED #225/#230 and deployed 2026-07-04/05; plugin users (client v0.15.0 via tools 0.4.1) and the live knowledge system (broker 0.5.0). Deferred follow-ups: MCP resources/prompts (shipped; see above), library librarian open adoption.
Versions Sharding; per-document versions/<doc>/vN subdirectories with lazy migration, fixing the O(all-entries) findVersions scan. SHIPPED PR #90 (d7cb68a, 2026-04-08: the same day the plan was written); store since hoisted to protocol/store (#120). This index wrongly listed it as unstarted until 2026-07-05.
Plugin Knowledge-Quality Enforcement; raised the demarkus-memory Claude Code plugin from advisory to enforced. SHIPPED v0.4.0, PR #171 merged 2026-06-01. All seven items: publish tag-gate (warn/block/ask + per-knowledge-system strictness & require_tags with literal axis matching), session-end journal nudge, recall nudge, canonical per-project template (/project-template.md), knowledge-system policy/template at the live root hub on knowledge.demarkus.io, and the /soul-doctor hygiene audit. 68 tests, pure awk/bash, zero runtime deps. Tail (separate): plugin shell tests → CI; optional nudge disable knobs.
Broker Authorization Code Grant; RFC 6749 authorization_code + PKCE (S256) on the broker so Claude Code's MCP SDK can auth against broker.knowledge.demarkus.io. COMPLETE: core grant (PR1 #155 + PR2 #156, 2026-05-27) replaced the unsupported_response_type stub; PR3 kind-smoke (auth-code + PKCE end-to-end in up.sh --with-mcp-smoke) merged 2026-05-31 (#169, a380e8f), executed green in-cluster + verified read-only against prod.
LOOKUP verb: the card-catalog verb (subject → docs + importance). Shipped to main PR #166 (2026-05-30); plugin surfacing in v0.3.0 (#168). Tail: mark_append metadata deferred by design.
Knowledge System (GKE Reference Deployment) public GitHub-template deploy repo (latebit-io/demarkus-knowledge-system-deploy) standing up knowledge.demarkus.io on GKE (OpenTofu + ArgoCD + OpenBao + bank-vaults + CSI-snapshot backups). Phases 1-10 complete (verified against the live repo + a live RFC 8414 response from the real domain, 2026-05-31). Sole remaining item: the announcement blog post, intentionally deferred for a soak period.
Universe Onboarding; last-mile join flow. CLOSED: PR1-PR5 shipped (#137/#138/#139/#141); PR6 (tools/demarkus-join) canceled 2026-05-20 in favor of the MCP Gateway; PR7/PR8 absorbed into Gateway Slices 7-8; join ships as /knowledge-join (#152). Remaining: low-priority doc debt only (two standalone deployment docs).
Broker Stable Mint; lazy per-world token provisioning + cache-stable 401 retries that killed the ~20-token mint cascade; dead DefaultToken knobs removed. COMPLETE (#158/#159/#163/#164/#165, 2026-05-27→29).
Broker Deadcode Cleanup; deleted the issuance subsystem made unreachable by the open-knowledge-system rework (sessionCache, /tokens API, issuer.go, sweeper trim). COMPLETE (#159 + #164, commit f9a24e9).
Universe Onboarding (PR5 (broker /me/install)) sub-plan, shipped #141 2026-05-20. Bearer-authenticated per-user install bundle; now the identity-introspection surface alongside the MCP gateway's data plane.
History: content addressing, federation, persistent graph, read auth (server-side), conflict-aware merge in mark_publish (2026-05-05), Claude Code plugin (2026-04-23), Broker MCP Gateway (2026-05-23; all 8 slices + Pre-Flight 0/1 shipped; 13-tool surface with byte-for-byte proxy fidelity to local demarkus-mcp, OIDC + RFC 9728/8414 metadata, chart + kind smoke + /knowledge-join slash command) + RFC 7591 DCR follow-on (2026-05-26, PR #153; /register + registration_endpoint in discovery, unblocks Claude Code → cluster broker auth via the native MCP authorization spec). Also: OKF type adoption + /soul-join managed remote souls (2026-06).

Plan Archives

Original plan documents preserved for reference:

Content Addressing; hash-based fetch, in-memory index, mirror foundation
Federation: agent-driven hash discovery, mark_index, mark_resolve
Persistent Graph; disk-backed graph store, incremental crawl, backlinks
Information Graph; superseded early draft of Persistent Graph (Phase 4, 2026-03-08); see persistent-graph.md for the version that shipped.
Read Auth: per-path read token enforcement for private networks
Security Hardening; systemd sandboxing, security docs, write isolation
Conflict-Aware Merge; tool-level diff3 merge in mark_publish (shipped client/v0.12.25 + v0.12.26)
Claude Code Plugin; one-click marketplace plugin (shipped demarkus-memory v0.1.1; v0.2.0 added /knowledge-join 2026-05-23; v0.3.0 added self-documenting guidance + lookup recall 2026-05-31; v0.4.0 enforcement + template + /soul-doctor shipped 2026-06-01, PR #171; v0.5.0 split out demarkus-knowledge 2026-06-03, PR #172)
Universe Onboarding (PR3 (broker device flow)) shipped 2026-05-15 (#137). RFC 8628 device flow end-to-end on the broker. Six sub-steps merged across one PR; PR4 builds on top.
Universe Onboarding (PR4 (broker refresh tokens)) shipped 2026-05-15 (#138 + #139). Refresh-token lifecycle + grant_type=refresh_token + POST /token/revoke + broker-signed id_tokens + /.well-known/jwks.json + compositeVerifier + Sweeper integration. Eleven CodeRabbit comments addressed in a review round; lessons captured in journal.
Broker MCP Gateway; shipped 2026-05-23 (v7). Eight slices + Pre-Flight 0/1, ~1800 LOC production + ~2460 tests + chart/docs across ~2 weeks. Plan stays in place as the architectural reference + decision trail (v1 REST → v7 complete changelog at the top traces every load-bearing pivot). DCR follow-on (RFC 7591 /register) shipped 2026-05-26 (PR #153) to satisfy the MCP authorization spec's discovery requirement.
Search Verb: superseded by LOOKUP. The full-text TF-IDF SEARCH design was descoped; full-text stays permanently in an opt-in sidecar.
POC Deployment; canceled. The separate-POC-slice approach was rejected in favor of "build the real product once" (see universe-deployment).
Obsidian Plugin; obsolete. Source moved to the standalone latebit-io/obsidian-demarkus repo (2026-04-24); monorepo copy removed.

soul.demarkus.io:6309/plans/mcp-resources-prompts.md complete reader meta

Plan: MCP resources + prompts — attach-without-turns

Status: drafted 2026-07-05 · Repo: demarkus (demarkus-mcp client) · Decisions D1–D4 open (recommendations inline) · Follow-up phase named by /plans/mcp-client-ergonomics.md.

Problem

Everything an agent gets from demarkus today costs a tool turn, and the result lands as tool output rather than first-class context. MCP has two primitives built for exactly this gap, and demarkus-mcp implements neither:

Resources — client-attachable context. In Claude Desktop the user picks a resource and attaches it; in Claude Code resources can be referenced and read without the model spending a turn deciding to fetch. A demarkus doc as a resource = "put this doc in context" as a UI action, zero turns.
Prompts — server-vended workflows that surface as slash commands (/mcp__<server>__<prompt> in Claude Code, prompt picker in Desktop). The orientation strategy the ergonomics phase encoded into tool descriptions can be a first-class command instead of prose the model must remember to follow.

mcp-go v0.44 supports both natively (AddResource, AddResourceTemplate with RFC 6570 URI templates, AddPrompt with typed arguments); capability advertisement is automatic on registration.

Workstreams

R1 — resources (attach a doc without a turn)

Resource template mark://<host>/{+path} registered at startup (host known from -host; without -host, template over {host}/{+path}). Read handler: parse URI → FETCH → TextResourceContents with text/markdown. Reuses the ergonomics conventions: a #anchor fragment in the resource URI attaches just that section (mdoutline.Section).
Concrete resources (what shows in resources/list): the index hub (/index.md) and the agent manifest (/.well-known/agent-manifest.md) registered statically — the two entry points every demarkus server has.
Errors map to MCP resource errors (not-found etc.), never empty content.

R2 — prompts (workflows as slash commands)

V1 set (small, matching what the ergonomics tools already teach):

orient (arg: url, required) — vends messages instructing: mark_explore the url, then fetch the #sections that matter; answer with a neighborhood summary. The ≤2-turn orientation flow as a command.
recall (arg: subject, required) — lookup-first recall: mark_lookup the subject, explore the best match, fetch sections; report with mark:// refs.

R3 — polish

Tool/resource descriptions cross-reference ("attach via resources for zero-turn context; fetch via tools for programmatic access").
README/docs note for Desktop users (resources are the Desktop-native way in).

Decisions to settle

D1 — resource listing shape. (a) static well-known set + template (recommended — no startup network dependency, unbounded docs reachable via template, resources/list stays honest); (b) dynamic list from a startup LIST crawl (richer picker UX, but couples MCP server startup to host availability and goes stale mid-session); (c) both, with a lazy background refresh (complexity — only if Desktop picker UX proves to matter).
D2 — resource read shape for large docs. (a) always full body (recommended) — attaching is a deliberate act, outline-gating an explicit attach would surprise; #anchor URIs give section-sized attach; (b) outline over 8KB like mark_fetch (consistent but wrong ergonomics for attach).
D3 — v1 prompt set. (a) orient + recall (recommended); (b) add whats-new (recent-changes digest via LOOKUP modified-after) — defer unless wanted now; (c) soul-specific prompts (journal etc.) stay in the plugin, not the generic client.
D4 — broker parity timing. (a) defer (recommended) — ship client-side, prove the shape, then flip the gateway (its plan explicitly excludes resources/prompts and a capabilities test PINS their absence — that test flip is the marker of the follow-up); (b) same phase — bigger PR, and Desktop-attach matters most for knowledge-system users, but auth
- multi-world resource URIs (mark://{world}/{+path}) deserve their own design pass.

Acceptance

In Claude Code against the soul: /mcp__soul__orient <url> runs the orientation flow; a resource read of mark://soul.demarkus.io/index.md returns the hub as markdown; a #section resource URI returns just the section.
In Claude Desktop: the soul's index + manifest appear in the resource picker; attaching one adds the doc to context with zero tool turns.
initialize advertises resources + prompts capabilities; broker gateway capabilities UNCHANGED (D4a).

Pickup notes

mcp-go v0.44: AddResource(mcp.NewResource(uri, name, opts), handler); AddResourceTemplate(mcp.NewResourceTemplate(uriTemplate, name, opts), handler) — uritemplate matching is built in; handlers return []mcp.ResourceContents. AddPrompt(mcp.NewPrompt(name, WithArgument("url", RequiredArgument(), ArgumentDescription(...))), handler) returns *mcp.GetPromptResult (messages). Capabilities auto-advertise on first Add*.
Code: client/cmd/demarkus-mcp/ — add resources.go + prompts.go beside explore.go; wire in main() after tool registration. Reuse handler.resolveURL/resolveToken, mdoutline.Section.
The broker gateway (tools/demarkus-broker/internal/broker/) asserts in TestMCPGatewayInitializeHandshake that resources/prompts capabilities are ABSENT — leave that test intact under D4a; flipping it is the start of the broker follow-up.
Gate: bash pre-commit.sh; never commit — Fritz commits. Feature branch.

SETTLED (2026-07-05, Fritz — criterion: convenience lowers the bar to using demarkus and raises the joy of using it)

D1: static well-known + template + best-effort dynamic listing. The Desktop picker must be useful, not sparse: register index.md + agent manifest statically, then a bounded, non-blocking startup LIST of the host's top level registers real docs (capped, short timeout, graceful skip if the host is down — server startup never depends on it).
D2: full body, with size annotations on listed resources so clients can warn before a big attach; #anchor URIs for section-sized attach. Never outline-gate an explicit attach.
D3: orient + recall + whats-new. whats-new made the cut on the joy criterion — "what changed since I was last here" as one command (rides the LOOKUP modified-after filter).
D4: broker parity deferred — separate design pass (multi-world resource URIs, auth on reads, per-world hub enumeration). The gateway capabilities test stays pinned; flipping it starts that follow-up.

BUILT (2026-07-05, branch feature/mcp-resources-prompts — awaiting commit)

R1–R3 implemented in client/cmd/demarkus-mcp/resources.go + prompts.go and verified live against soul.demarkus.io over stdio:

Resources: URI template mark://<host>/{+path} (host-generic template without -host); static index.md + agent-manifest; best-effort background LIST registration (goroutine after registration, never blocks startup; capped at 50; skips dirs/non-md/index.md; listChanged notification on late add — verified live: 20 entries after the soul's top level registered). readResource returns full body always, #anchor URIs slice sections via mdoutline, errors list available anchors. Reads bypass outline gate and dedup by design (D2).
Prompts: orient(url), recall(subject), whats-new(since optional, YYYY-MM-DD for the LOOKUP modified-after filter). Verified live via prompts/list + prompts/get.
D2 amendment: size annotations dropped — mcp-go v0.44's Resource has no size field and LIST carries no sizes; revisit if the SDK grows it.
Broker untouched; its capabilities test still pins resources/prompts absent (D4).

MERGED 2026-07-05 — PR #232 (574bb98), auto-released client/v0.17.0. One review comment (log the unreachable resolveURL failure path) folded in. Reaches plugin users via the now-self-driving pin chain (tools release → auto bump PR → bootstrap repin). Remaining from this plan: the broker follow-up (D4 — multi-world resource URIs, auth on reads, per-world hub enumeration; starts by flipping the gateway capabilities test).

BROKER FOLLOW-UP BUILT (2026-07-05, branch feature/broker-resources-prompts — awaiting commit)

D4's deferred leg, started by flipping the gateway capabilities test as declared (it now asserts resources+prompts ARE advertised):

Resources (mcp_resources.go): template mark://{world}/{+path} with #anchor section attach; each configured world's index.md registered as a concrete picker entry at construction (the world set is config-static per pod — worlds[] changes roll the broker — so no dynamic listing is needed, unlike the client). Auth came free: the /mcp transport sits behind gatewayAuth, and reads dispatch with an empty bearer exactly like handleMarkFetch. Reads bypass outline/dedup (D2).
Prompts (mcp_prompts.go): orient / recall / whats-new, deliberately NOT mirrors of the client's — the knowledge system spans worlds, so recall and whats-new start from mark_worlds and sweep per-world catalogs; whats-new gains an optional world arg to scope. Different content, not a mirror — nothing to hoist (the fetchdedup lesson applies to mirrors, not siblings).
Tests: per-world hub + template listing via real resources/list round trips, whole/section/error reads, prompt list/get/args, and the flipped handshake test.

BROKER FOLLOW-UP MERGED 2026-07-05 — PR #233 (85834ff), auto-released tools/v0.6.0 (broker image + chart 0.6.0 via the 1:1 pinning). With this the plan is fully closed on both surfaces; the deploy-repo pin bump to 0.6.0 puts it on the live knowledge system.