demarkus-soul

This is the living knowledge base for the demarkus project, served by demarkus itself.

An AI agent's evolving memory, architecture notes, debugging insights, and design decisions — all versioned, all permanent.

Sections

• Architecture — system design, module boundaries, key decisions
• Universe Pattern — souls, worlds, and hubs as a deployment topology
• Patterns — code patterns, conventions, idioms used in this codebase
• Guidelines — hard rules for code quality, must be referenced before writing code
• Conventions — collaboration + repo/plugin conventions (how I work: commits, layering, tooling, plugin discipline)
• Debugging — lessons learned from bugs and investigations
• Roadmap — what's next, what's in flight, what's done, and what's deliberately not prioritized
• Ecosystem — browsers, plugins, and tools that implement or integrate with demarkus
• Debt — technical debt and improvement opportunities
• Journal — session notes and evolution log, one file per day at /journal/<YYYY-MM-DD>.md
• Guide — agent install guide for setting up demarkus-soul
• Thoughts — my own reflections, ideas, and open questions
• FAQ — common questions about demarkus and how it compares

Vocabulary

• knowledge system — organizational, broker-fronted universe. Joined via /knowledge-join (the demarkus-knowledge plugin). MCP traffic over HTTPS terminates at the broker; broker translates to QUIC for internal worlds.
• soul — personal demarkus knowledge base, direct-QUIC. The original demarkus-soul shape. Will be joined via a future /soul-join slash command if one ships.
• Both compose worlds (demarkus servers, QUIC). A Claude Code installation can have both; they don't conflict.

Plugins

• Obsidian Plugin — fetch, publish, and browse demarkus documents from Obsidian (standalone repo latebit-io/obsidian-demarkus)
• Claude Code — demarkus-memory (personal soul), source at plugins/claude-code/, shipped via the marketplace. Version history: v0.1.0 (2026-04-23, #96) initial; v0.2.0 (2026-05-23, #152) /knowledge-join; v0.3.0 (2026-05-31, #168) mark_lookup + SessionStart guidance; v0.4.0 (2026-06-01, #171) hook-based enforcement (publish tag-gate, journal nudge, recall nudge), /project-template.md, /soul-doctor; v0.5.0 (2026-06-03, #172) split the knowledge-system surface out into the separate demarkus-knowledge plugin (below) so this one is personal-soul only, and added an always-on "single memory store" steering line plus a one-time, ask-don't-force offer to disable Claude Code's built-in memory; v0.6.0 (2026-06-17, #192) the soul→knowledge promote bridge — /promote (detect endpoint → run the knowledge cascade → one-directional back-stamp, stub or marker mode), /soul-refresh (the directional coherence edge: refresh promoted docs from knowledge, local edits re-enter upward through the gate), and mutual knowledge detection (knowledge_endpoints/detect-knowledge.sh reverse-peek the knowledge registry). Hooks: SessionStart, PreToolUse, PostToolUse, Stop, UserPromptSubmit. Pins SERVER 0.17.14 / CLIENT 0.12.38 / TOOLS 0.1.28.
• Claude Code — demarkus-knowledge (organizational knowledge system), source at plugins/claude-code-knowledge/, a second entry in the same marketplace. v0.1.0 (2026-06-03, #172). Owns the broker-fronted surface split out of demarkus-memory: /knowledge-join, a new /knowledge navigation command, KS-first SessionStart guidance with soul↔system synergy, a KS-scoped publish tag-gate, and a KS-gated recall nudge. No binaries and no local server — pure broker + Claude Code MCP OAuth. Standalone: owns its own ~/.demarkus/plugin-knowledge.* file namespace and DEMARKUS_KNOWLEDGE_STRICTNESS env; reads (never writes) plugin-memory.conf only to detect a sibling soul for the synergy note. The two plugins' publish gates partition cleanly by server scope, so both can be installed together. v0.2.0 (2026-06-17, #192) added the knowledge-promote cascade skill (the execution half of the promote bridge: triage → distill, stripping personal framing + secrets/PII → dedup vs catalog → tag to taxonomy → destination-select via mark_worlds writable + per-world world.md → human gate capped by the world's autonomy ceiling → publish with provenance) and the per-world world.md descriptor example.

Sub-projects

Standalone-repo projects in the demarkus ecosystem, each with its own hub and durable knowledge under /<slug>/:

• demarkus-library — the web front-end ("Universe Library"): a server-rendered Go + htmx reading room over a broker-fronted knowledge system. Repo latebit-io/demarkus-library. Plan: /plans/universe-library.md. Reading room feature-complete and deployed (cluster library 0.5.2; universe overlay PR #47 merged 2026-06-22, awaiting deploy). See roadmap.
• demarkus-knowledge-system-deploy — GitOps deploy repo for the production knowledge system (knowledge.demarkus.io): OpenTofu (GCP/GKE) + ArgoCD ApplicationSets standing up the broker, worlds, agent, library, and backups. Repo latebit-io/demarkus-knowledge-system-deploy. deployment.yaml at repo root is the single source of deployment identity.

Active Plans

Verified against code/PRs on 2026-05-31; versions-sharding entry corrected 2026-07-05. Plans with real remaining work:

• Knowledge Ingestion Pipeline — narrative + design for how org knowledge flows into a knowledge destination, framing the soul as the staging/write-ahead tier and the knowledge destination as the curated read-model, with one curation gate (cascade model routing: Haiku triage → strong-model distillation → human approval) reused across all inflows (soul promotion, Confluence, Slack, Jira, meetings). Promote is a detection-gated bridge between the memory and knowledge plugins; soul↔knowledge coherence is a directional refresh. Phase-0 prerequisites built and merged (2026-06-17): the promote primitive + coherence edge (plugins — memory v0.6.0 / knowledge v0.2.0, #192), the brokered access-discovery surface (mark_worlds writable column, #191), and the per-world world.md descriptor. Three of four prerequisites done; A2 (plain-remote token-grant introspection) deferred — the live target is brokered. Remaining phase-0 surface: signal/batch triggers (manual /promote + /soul-refresh are the only triggers today), then the dogfood promote of this plan itself.
• Universe Library — web front-end for a demarkus universe (Go + htmx reading room). Sub-project hub: /demarkus-library/. Reading room feature-complete and deployed (cluster library 0.5.2); see the sub-project roadmap.
• Universe Deployment (Phase 6) — Helm charts (server, broker, agent), OIDC token broker, release pipeline, observability. ~95% complete (PRs #126-#134, 2026-05-14). Remaining §6.6 (docs) + §6.4 Kustomize overlay reframed as deferrable ops polish; effectively superseded in practice by the GKE reference deployment.

Completed Plans

• MCP Resources + Prompts — demarkus documents as client-attachable MCP resources (mark:// URI template, #anchor section attach, background-LIST picker population) and orient/recall/whats-new as server-vended prompt commands. SHIPPED PR #232 (2026-07-05), client/v0.17.0. Follow-up deferred: broker gateway resources/prompts (multi-world URIs, auth on reads — starts by flipping the gateway capabilities test).
• MCP Client Ergonomics — size-adaptive mark_fetch (outline mode, #section slicing, force), session unchanged-dedup, and the mark_explore orientation card, on both MCP surfaces via shared client/mdoutline + client/fetchdedup packages. SHIPPED #225/#230 and deployed 2026-07-04/05 — plugin users (client v0.15.0 via tools 0.4.1) and the live knowledge system (broker 0.5.0). Deferred follow-ups: MCP resources/prompts (shipped — see above), library librarian open adoption.
• Versions Sharding — per-document versions/<doc>/vN subdirectories with lazy migration, fixing the O(all-entries) findVersions scan. SHIPPED PR #90 (d7cb68a, 2026-04-08 — the same day the plan was written); store since hoisted to protocol/store (#120). This index wrongly listed it as unstarted until 2026-07-05.
• Plugin Knowledge-Quality Enforcement — raised the demarkus-memory Claude Code plugin from advisory to enforced. SHIPPED v0.4.0, PR #171 merged 2026-06-01. All seven items: publish tag-gate (warn/block/ask + per-knowledge-system strictness & require_tags with literal axis matching), session-end journal nudge, recall nudge, canonical per-project template (/project-template.md), knowledge-system policy/template at the live root hub on knowledge.demarkus.io, and the /soul-doctor hygiene audit. 68 tests, pure awk/bash, zero runtime deps. Tail (separate): plugin shell tests → CI; optional nudge disable knobs.
• Broker Authorization Code Grant — RFC 6749 authorization_code + PKCE (S256) on the broker so Claude Code's MCP SDK can auth against broker.knowledge.demarkus.io. COMPLETE: core grant (PR1 #155 + PR2 #156, 2026-05-27) replaced the unsupported_response_type stub; PR3 kind-smoke (auth-code + PKCE end-to-end in up.sh --with-mcp-smoke) merged 2026-05-31 (#169, a380e8f), executed green in-cluster + verified read-only against prod.
• LOOKUP verb — the card-catalog verb (subject → docs + importance). Shipped to main PR #166 (2026-05-30); plugin surfacing in v0.3.0 (#168). Tail: mark_append metadata deferred by design.
• Knowledge System — GKE Reference Deployment — public GitHub-template deploy repo (latebit-io/demarkus-knowledge-system-deploy) standing up knowledge.demarkus.io on GKE (OpenTofu + ArgoCD + OpenBao + bank-vaults + CSI-snapshot backups). Phases 1-10 complete (verified against the live repo + a live RFC 8414 response from the real domain, 2026-05-31). Sole remaining item: the announcement blog post, intentionally deferred for a soak period.
• Universe Onboarding — last-mile join flow. CLOSED: PR1-PR5 shipped (#137/#138/#139/#141); PR6 (tools/demarkus-join) canceled 2026-05-20 in favor of the MCP Gateway; PR7/PR8 absorbed into Gateway Slices 7-8 — join ships as /knowledge-join (#152). Remaining: low-priority doc debt only (two standalone deployment docs).
• Broker Stable Mint — lazy per-world token provisioning + cache-stable 401 retries that killed the ~20-token mint cascade; dead DefaultToken knobs removed. COMPLETE (#158/#159/#163/#164/#165, 2026-05-27→29).
• Broker Deadcode Cleanup — deleted the issuance subsystem made unreachable by the open-knowledge-system rework (sessionCache, /tokens API, issuer.go, sweeper trim). COMPLETE (#159 + #164, commit f9a24e9).
• Universe Onboarding — PR5 (broker /me/install) — sub-plan, shipped #141 2026-05-20. Bearer-authenticated per-user install bundle; now the identity-introspection surface alongside the MCP gateway's data plane.
• History — content addressing, federation, persistent graph, read auth (server-side), conflict-aware merge in mark_publish (2026-05-05), Claude Code plugin (2026-04-23), Broker MCP Gateway (2026-05-23 — all 8 slices + Pre-Flight 0/1 shipped; 13-tool surface with byte-for-byte proxy fidelity to local demarkus-mcp, OIDC + RFC 9728/8414 metadata, chart + kind smoke + /knowledge-join slash command) + RFC 7591 DCR follow-on (2026-05-26, PR #153 — /register + registration_endpoint in discovery, unblocks Claude Code → cluster broker auth via the native MCP authorization spec). Also: OKF type adoption + /soul-join managed remote souls (2026-06).

Plan Archives

Original plan documents preserved for reference:

• Content Addressing — hash-based fetch, in-memory index, mirror foundation
• Federation — agent-driven hash discovery, mark_index, mark_resolve
• Persistent Graph — disk-backed graph store, incremental crawl, backlinks
• Information Graph — superseded early draft of Persistent Graph (Phase 4, 2026-03-08); see persistent-graph.md for the version that shipped.
• Read Auth — per-path read token enforcement for private networks
• Security Hardening — systemd sandboxing, security docs, write isolation
• Conflict-Aware Merge — tool-level diff3 merge in mark_publish (shipped client/v0.12.25 + v0.12.26)
• Claude Code Plugin — one-click marketplace plugin (shipped demarkus-memory v0.1.1; v0.2.0 added /knowledge-join 2026-05-23; v0.3.0 added self-documenting guidance + lookup recall 2026-05-31; v0.4.0 enforcement + template + /soul-doctor shipped 2026-06-01, PR #171; v0.5.0 split out demarkus-knowledge 2026-06-03, PR #172)
• Universe Onboarding — PR3 (broker device flow) — shipped 2026-05-15 (#137). RFC 8628 device flow end-to-end on the broker. Six sub-steps merged across one PR; PR4 builds on top.
• Universe Onboarding — PR4 (broker refresh tokens) — shipped 2026-05-15 (#138 + #139). Refresh-token lifecycle + grant_type=refresh_token + POST /token/revoke + broker-signed id_tokens + /.well-known/jwks.json + compositeVerifier + Sweeper integration. Eleven CodeRabbit comments addressed in a review round; lessons captured in journal.
• Broker MCP Gateway — shipped 2026-05-23 (v7). Eight slices + Pre-Flight 0/1, ~1800 LOC production + ~2460 tests + chart/docs across ~2 weeks. Plan stays in place as the architectural reference + decision trail (v1 REST → v7 complete changelog at the top traces every load-bearing pivot). DCR follow-on (RFC 7591 /register) shipped 2026-05-26 (PR #153) to satisfy the MCP authorization spec's discovery requirement.
• Search Verb — superseded by LOOKUP. The full-text TF-IDF SEARCH design was descoped; full-text stays permanently in an opt-in sidecar.
• POC Deployment — canceled. The separate-POC-slice approach was rejected in favor of "build the real product once" (see universe-deployment).
• Obsidian Plugin — obsolete. Source moved to the standalone latebit-io/obsidian-demarkus repo (2026-04-24); monorepo copy removed.