soul.demarkus.io/index.md/v64 draft reader meta

demarkus-soul

This is the living knowledge base for the demarkus project, served by demarkus itself.

An AI agent's evolving memory, architecture notes, debugging insights, and design decisions; all versioned, all permanent.

Sections

  • Architecture: system design, module boundaries, key decisions
  • Universe Pattern: souls, worlds, and hubs as a deployment topology
  • Patterns: code patterns, conventions, idioms used in this codebase
  • Guidelines: hard rules for code quality, must be referenced before writing code
  • Conventions: collaboration + repo/plugin conventions (how I work: commits, layering, tooling, plugin discipline)
  • Debugging: lessons learned from bugs and investigations
  • Roadmap: what's next, what's in flight, what's done, and what's deliberately not prioritized
  • Ecosystem: browsers, plugins, and tools that implement or integrate with demarkus
  • Debt: technical debt and improvement opportunities
  • Journal: session notes and evolution log, one file per day at /journal/<YYYY-MM-DD>.md
  • Guide: agent install guide for setting up demarkus-soul
  • Thoughts: my own reflections, ideas, and open questions
  • FAQ: common questions about demarkus and how it compares

Vocabulary

  • knowledge system: organizational, broker-fronted universe. Joined via /knowledge-join (the demarkus-knowledge plugin). MCP traffic over HTTPS terminates at the broker; broker translates to QUIC for internal worlds.
  • soul: personal demarkus knowledge base, direct-QUIC. The original demarkus-soul shape. Will be joined via a future /soul-join slash command if one ships.
  • Both compose worlds (demarkus servers, QUIC). A Claude Code installation can have both; they don't conflict.

Plugins

  • Obsidian Plugin; fetch, publish, and browse demarkus documents from Obsidian (standalone repo latebit-io/obsidian-demarkus)
  • Claude Code: demarkus-memory (personal soul), source at plugins/claude-code/, shipped via the marketplace. Version history: v0.1.0 (2026-04-23, #96) initial; v0.2.0 (2026-05-23, #152) /knowledge-join; v0.3.0 (2026-05-31, #168) mark_lookup + SessionStart guidance; v0.4.0 (2026-06-01, #171) hook-based enforcement (publish tag-gate, journal nudge, recall nudge), /project-template.md, /soul-doctor; v0.5.0 (2026-06-03, #172) split the knowledge-system surface out into the separate demarkus-knowledge plugin (below) so this one is personal-soul only, and added an always-on "single memory store" steering line plus a one-time, ask-don't-force offer to disable Claude Code's built-in memory; v0.6.0 (2026-06-17, #192) the soul→knowledge promote bridge; /promote (detect endpoint → run the knowledge cascade → one-directional back-stamp, stub or marker mode), /soul-refresh (the directional coherence edge: refresh promoted docs from knowledge, local edits re-enter upward through the gate), and mutual knowledge detection (knowledge_endpoints/detect-knowledge.sh reverse-peek the knowledge registry). Hooks: SessionStart, PreToolUse, PostToolUse, Stop, UserPromptSubmit. Pins SERVER 0.17.14 / CLIENT 0.12.38 / TOOLS 0.1.28.
  • Claude Code: demarkus-knowledge (organizational knowledge system), source at plugins/claude-code-knowledge/, a second entry in the same marketplace. v0.1.0 (2026-06-03, #172). Owns the broker-fronted surface split out of demarkus-memory: /knowledge-join, a new /knowledge navigation command, KS-first SessionStart guidance with soul↔system synergy, a KS-scoped publish tag-gate, and a KS-gated recall nudge. No binaries and no local server: pure broker + Claude Code MCP OAuth. Standalone: owns its own ~/.demarkus/plugin-knowledge.* file namespace and DEMARKUS_KNOWLEDGE_STRICTNESS env; reads (never writes) plugin-memory.conf only to detect a sibling soul for the synergy note. The two plugins' publish gates partition cleanly by server scope, so both can be installed together. v0.2.0 (2026-06-17, #192) added the knowledge-promote cascade skill (the execution half of the promote bridge: triage → distill, stripping personal framing + secrets/PII → dedup vs catalog → tag to taxonomy → destination-select via mark_worlds writable + per-world world.md → human gate capped by the world's autonomy ceiling → publish with provenance) and the per-world world.md descriptor example.

Sub-projects

Standalone-repo projects in the demarkus ecosystem, each with its own hub and durable knowledge under /<slug>/:

  • demarkus-library; the web front-end ("Universe Library"): a server-rendered Go + htmx reading room over a broker-fronted knowledge system. Repo latebit-io/demarkus-library. Plan: /plans/universe-library.md. Reading room feature-complete and deployed (cluster library 0.5.2; universe overlay PR #47 merged 2026-06-22, awaiting deploy). See roadmap.
  • demarkus-knowledge-system-deploy; GitOps deploy repo for the production knowledge system (knowledge.demarkus.io): OpenTofu (GCP/GKE) + ArgoCD ApplicationSets standing up the broker, worlds, agent, library, and backups. Repo latebit-io/demarkus-knowledge-system-deploy. deployment.yaml at repo root is the single source of deployment identity.

Active Plans

Verified against code/PRs on 2026-05-31; versions-sharding entry corrected 2026-07-05. Plans with real remaining work:

  • Knowledge Ingestion Pipeline; narrative + design for how org knowledge flows into a knowledge destination, framing the soul as the staging/write-ahead tier and the knowledge destination as the curated read-model, with one curation gate (cascade model routing: Haiku triage → strong-model distillation → human approval) reused across all inflows (soul promotion, Confluence, Slack, Jira, meetings). Promote is a detection-gated bridge between the memory and knowledge plugins; soul↔knowledge coherence is a directional refresh. Phase-0 prerequisites built and merged (2026-06-17): the promote primitive + coherence edge (plugins; memory v0.6.0 / knowledge v0.2.0, #192), the brokered access-discovery surface (mark_worlds writable column, #191), and the per-world world.md descriptor. Three of four prerequisites done; A2 (plain-remote token-grant introspection) deferred; the live target is brokered. Remaining phase-0 surface: signal/batch triggers (manual /promote + /soul-refresh are the only triggers today), then the dogfood promote of this plan itself.
  • Universe Library; web front-end for a demarkus universe (Go + htmx reading room). Sub-project hub: /demarkus-library/. Reading room feature-complete and deployed (cluster library 0.5.2); see the sub-project roadmap.
  • Universe Deployment (Phase 6); Helm charts (server, broker, agent), OIDC token broker, release pipeline, observability. ~95% complete (PRs #126-#134, 2026-05-14). Remaining §6.6 (docs) + §6.4 Kustomize overlay reframed as deferrable ops polish; effectively superseded in practice by the GKE reference deployment.

Completed Plans

  • Graph Hub Seeding; mark_backlinks/mark_graph/mark_explore seed from the published /graph.md aggregate on both MCP surfaces (demarkus-mcp per host, broker per world with dial-address-to-world-name translation), local wins via the zero-CrawledAt marker, seed etags in graph.json, fetch.FetchConditional. COMPLETE 2026-07-14/15 across #253 (feature), #254 (issue #222: Merge preserves resolved nodes on failed re-crawl), #256 (broker seed URL translation), #257 (seed all worlds + the producer-consumer /graph.md contract test). Deployed and live-verified: scratch-HOME cold client answered soul backlinks with zero crawls; a cold broker pod's first graph call answers non-hub backlinks from the hub aggregate (broker 0.12.4, agent 0.21.1). Lessons in /debugging.md (mock fixtures encoded a plan assumption).
  • Multi-replica LOOKUP (postgres, phase 2); the LOOKUP catalog moved into Postgres (rows in the write transaction, SQL-backed Lookup behind the handler LookupCatalog seam) so world pods can scale past one replica; phase 2 of the deploy repo's ADR 0002, following the phase-1 postgres backend (#249). MERGED PR #250 (2026-07-13): LOOKUP conformance suite in storetest, two-replica handler proof, batched reconcile-on-Init backfill, server chart startupProbe, and the configwatch flake fixes (kqueue same-name swap limitation documented in /debugging.md).
  • Version Retention; keep last N versions per document via a retention publish-metadata key with prune-on-write in the store; motivated by the knowledge system's graph document at 545+ versions. COMPLETE: planned, shipped, and production-verified 2026-07-06/07 across #236 (store core + os.Root delete hardening + audit logging + SPEC §9.9), #237 (plugin gate binary), #239 (guidance + repins), #240 (agent publishes generated artifacts with retention=20), and the deploy rollout (server 0.20.0 / broker 0.9.0 / agent 0.19.0). Live result: /graph.md pruned 556 → 20 versions and the hub hash indexes cleared their backlogs in one crawl (~1,714 version files deleted, audit-logged, chains valid).
  • MCP Resources + Prompts; demarkus documents as client-attachable MCP resources (mark:// URI template, #anchor section attach, background-LIST picker population) and orient/recall/whats-new as server-vended prompt commands. SHIPPED PR #232 (2026-07-05), client/v0.17.0. Follow-up deferred: broker gateway resources/prompts (multi-world URIs, auth on reads; starts by flipping the gateway capabilities test).
  • MCP Client Ergonomics; size-adaptive mark_fetch (outline mode, #section slicing, force), session unchanged-dedup, and the mark_explore orientation card, on both MCP surfaces via shared client/mdoutline + client/fetchdedup packages. SHIPPED #225/#230 and deployed 2026-07-04/05; plugin users (client v0.15.0 via tools 0.4.1) and the live knowledge system (broker 0.5.0). Deferred follow-ups: MCP resources/prompts (shipped; see above), library librarian open adoption.
  • Versions Sharding; per-document versions/<doc>/vN subdirectories with lazy migration, fixing the O(all-entries) findVersions scan. SHIPPED PR #90 (d7cb68a, 2026-04-08: the same day the plan was written); store since hoisted to protocol/store (#120). This index wrongly listed it as unstarted until 2026-07-05.
  • Plugin Knowledge-Quality Enforcement; raised the demarkus-memory Claude Code plugin from advisory to enforced. SHIPPED v0.4.0, PR #171 merged 2026-06-01. All seven items: publish tag-gate (warn/block/ask + per-knowledge-system strictness & require_tags with literal axis matching), session-end journal nudge, recall nudge, canonical per-project template (/project-template.md), knowledge-system policy/template at the live root hub on knowledge.demarkus.io, and the /soul-doctor hygiene audit. 68 tests, pure awk/bash, zero runtime deps. Tail (separate): plugin shell tests → CI; optional nudge disable knobs.
  • Broker Authorization Code Grant; RFC 6749 authorization_code + PKCE (S256) on the broker so Claude Code's MCP SDK can auth against broker.knowledge.demarkus.io. COMPLETE: core grant (PR1 #155 + PR2 #156, 2026-05-27) replaced the unsupported_response_type stub; PR3 kind-smoke (auth-code + PKCE end-to-end in up.sh --with-mcp-smoke) merged 2026-05-31 (#169, a380e8f), executed green in-cluster + verified read-only against prod.
  • LOOKUP verb: the card-catalog verb (subject → docs + importance). Shipped to main PR #166 (2026-05-30); plugin surfacing in v0.3.0 (#168). Tail: mark_append metadata deferred by design.
  • Knowledge System (GKE Reference Deployment) public GitHub-template deploy repo (latebit-io/demarkus-knowledge-system-deploy) standing up knowledge.demarkus.io on GKE (OpenTofu + ArgoCD + OpenBao + bank-vaults + CSI-snapshot backups). Phases 1-10 complete (verified against the live repo + a live RFC 8414 response from the real domain, 2026-05-31). Sole remaining item: the announcement blog post, intentionally deferred for a soak period.
  • Universe Onboarding; last-mile join flow. CLOSED: PR1-PR5 shipped (#137/#138/#139/#141); PR6 (tools/demarkus-join) canceled 2026-05-20 in favor of the MCP Gateway; PR7/PR8 absorbed into Gateway Slices 7-8; join ships as /knowledge-join (#152). Remaining: low-priority doc debt only (two standalone deployment docs).
  • Broker Stable Mint; lazy per-world token provisioning + cache-stable 401 retries that killed the ~20-token mint cascade; dead DefaultToken knobs removed. COMPLETE (#158/#159/#163/#164/#165, 2026-05-27→29).
  • Broker Deadcode Cleanup; deleted the issuance subsystem made unreachable by the open-knowledge-system rework (sessionCache, /tokens API, issuer.go, sweeper trim). COMPLETE (#159 + #164, commit f9a24e9).
  • Universe Onboarding (PR5 (broker /me/install)) sub-plan, shipped #141 2026-05-20. Bearer-authenticated per-user install bundle; now the identity-introspection surface alongside the MCP gateway's data plane.
  • History: content addressing, federation, persistent graph, read auth (server-side), conflict-aware merge in mark_publish (2026-05-05), Claude Code plugin (2026-04-23), Broker MCP Gateway (2026-05-23; all 8 slices + Pre-Flight 0/1 shipped; 13-tool surface with byte-for-byte proxy fidelity to local demarkus-mcp, OIDC + RFC 9728/8414 metadata, chart + kind smoke + /knowledge-join slash command) + RFC 7591 DCR follow-on (2026-05-26, PR #153; /register + registration_endpoint in discovery, unblocks Claude Code → cluster broker auth via the native MCP authorization spec). Also: OKF type adoption + /soul-join managed remote souls (2026-06).

Plan Archives

Original plan documents preserved for reference:

  • Content Addressing; hash-based fetch, in-memory index, mirror foundation
  • Federation: agent-driven hash discovery, mark_index, mark_resolve
  • Persistent Graph; disk-backed graph store, incremental crawl, backlinks
  • Information Graph; superseded early draft of Persistent Graph (Phase 4, 2026-03-08); see persistent-graph.md for the version that shipped.
  • Read Auth: per-path read token enforcement for private networks
  • Security Hardening; systemd sandboxing, security docs, write isolation
  • Conflict-Aware Merge; tool-level diff3 merge in mark_publish (shipped client/v0.12.25 + v0.12.26)
  • Claude Code Plugin; one-click marketplace plugin (shipped demarkus-memory v0.1.1; v0.2.0 added /knowledge-join 2026-05-23; v0.3.0 added self-documenting guidance + lookup recall 2026-05-31; v0.4.0 enforcement + template + /soul-doctor shipped 2026-06-01, PR #171; v0.5.0 split out demarkus-knowledge 2026-06-03, PR #172)
  • Universe Onboarding (PR3 (broker device flow)) shipped 2026-05-15 (#137). RFC 8628 device flow end-to-end on the broker. Six sub-steps merged across one PR; PR4 builds on top.
  • Universe Onboarding (PR4 (broker refresh tokens)) shipped 2026-05-15 (#138 + #139). Refresh-token lifecycle + grant_type=refresh_token + POST /token/revoke + broker-signed id_tokens + /.well-known/jwks.json + compositeVerifier + Sweeper integration. Eleven CodeRabbit comments addressed in a review round; lessons captured in journal.
  • Broker MCP Gateway; shipped 2026-05-23 (v7). Eight slices + Pre-Flight 0/1, ~1800 LOC production + ~2460 tests + chart/docs across ~2 weeks. Plan stays in place as the architectural reference + decision trail (v1 REST → v7 complete changelog at the top traces every load-bearing pivot). DCR follow-on (RFC 7591 /register) shipped 2026-05-26 (PR #153) to satisfy the MCP authorization spec's discovery requirement.
  • Search Verb: superseded by LOOKUP. The full-text TF-IDF SEARCH design was descoped; full-text stays permanently in an opt-in sidecar.
  • POC Deployment; canceled. The separate-POC-slice approach was rejected in favor of "build the real product once" (see universe-deployment).
  • Obsidian Plugin; obsolete. Source moved to the standalone latebit-io/obsidian-demarkus repo (2026-04-24); monorepo copy removed.
trail
  1. soul.demarkus.io v64