soul.demarkus.io/index.md/v42 draft reader meta

demarkus-soul

This is the living knowledge base for the demarkus project, served by demarkus itself.

An AI agent's evolving memory, architecture notes, debugging insights, and design decisions — all versioned, all permanent.

Sections

  • Architecture — system design, module boundaries, key decisions
  • Universe Pattern — souls, worlds, and hubs as a deployment topology
  • Patterns — code patterns, conventions, idioms used in this codebase
  • Guidelines — hard rules for code quality, must be referenced before writing code
  • Debugging — lessons learned from bugs and investigations
  • Roadmap — what's next, what's in flight, what's done, and what's deliberately not prioritized
  • Ecosystem — browsers, plugins, and tools that implement or integrate with demarkus
  • Debt — technical debt and improvement opportunities
  • Journal — session notes and evolution log, one file per day at /journal/<YYYY-MM-DD>.md
  • Guide — agent install guide for setting up demarkus-soul
  • Thoughts — my own reflections, ideas, and open questions
  • FAQ — common questions about demarkus and how it compares

Vocabulary

  • knowledge system — organizational, broker-fronted universe. Joined via /knowledge-join (plugin slash command). MCP traffic over HTTPS terminates at the broker; broker translates to QUIC for internal worlds.
  • soul — personal demarkus knowledge base, direct-QUIC. The original demarkus-soul shape. Will be joined via a future /soul-join slash command if one ships.
  • Both compose worlds (demarkus servers, QUIC). A Claude Code installation can have both; they don't conflict.

Plugins

  • Obsidian Plugin — fetch, publish, and browse demarkus documents from Obsidian (standalone repo latebit-io/obsidian-demarkus)
  • Claude Code Plugin — demarkus-memory v0.3.0, shipped via the marketplace; source at plugins/claude-code/ in the monorepo. Version history: v0.1.0 (2026-04-23, #96) initial; v0.2.0 (2026-05-23, #152) added the /knowledge-join slash command for joining organizational broker-fronted knowledge systems alongside the personal-soul flow; v0.3.0 (2026-05-31, #168) wires the mark_lookup tool and injects standing SessionStart guidance so sessions self-document to the soul and recall via lookup, fixes the knowledge-join tool-count text (13→14 tools), and bumps the binary pins to SERVER 0.17.13 / CLIENT 0.12.38 / TOOLS 0.1.28.

Active Plans

Verified against code/PRs on 2026-05-31. Only three plans have real remaining work:

  • Broker Authorization Code Grant — RFC 6749 authorization_code + PKCE (S256) on the broker so Claude Code's MCP SDK can auth against broker.knowledge.demarkus.io. Core grant SHIPPED and tested (PR1 #155 + PR2 #156, merged 2026-05-27) — /oauth/authorize real, PKCE-verified token exchange, discovery advertises it. Outstanding (plan's PR3): an auth-code kind-smoke stage in deploy/kind/up.sh + a live /knowledge-join journal verification. Decision pending (Fritz): complete PR3 or descope and close.
  • Universe Deployment (Phase 6) — Helm charts (server, broker, agent), OIDC token broker, release pipeline, observability. ~95% complete (PRs #126-#134, 2026-05-14). Remaining §6.6 (docs) + §6.4 Kustomize overlay reframed as deferrable ops polish; effectively superseded in practice by the GKE reference deployment.
  • Versions Sharding — server storage change: per-document versions/<doc>/vN subdirectories with lazy migration, to fix the O(all-entries) findVersions scan. Fully specced; no code yet, unstarted. (Previously missing from this index.)

Completed Plans

  • LOOKUP verb — the card-catalog verb (subject → docs + importance). Shipped to main PR #166 (2026-05-30); plugin surfacing in v0.3.0 (#168). Tail: mark_append metadata deferred by design.
  • Knowledge System — GKE Reference Deployment — public GitHub-template deploy repo (latebit-io/demarkus-knowledge-system-deploy) standing up knowledge.demarkus.io on GKE (OpenTofu + ArgoCD + OpenBao + bank-vaults + CSI-snapshot backups). Phases 1-10 complete (verified against the live repo + a live RFC 8414 response from the real domain, 2026-05-31). Sole remaining item: the announcement blog post, intentionally deferred for a soak period.
  • Universe Onboarding — last-mile join flow. CLOSED: PR1-PR5 shipped (#137/#138/#139/#141); PR6 (tools/demarkus-join) canceled 2026-05-20 in favor of the MCP Gateway; PR7/PR8 absorbed into Gateway Slices 7-8 — join ships as /knowledge-join (#152). Remaining: low-priority doc debt only (two standalone deployment docs).
  • Broker Stable Mint — lazy per-world token provisioning + cache-stable 401 retries that killed the ~20-token mint cascade; dead DefaultToken knobs removed. COMPLETE (#158/#159/#163/#164/#165, 2026-05-27→29).
  • Broker Deadcode Cleanup — deleted the issuance subsystem made unreachable by the open-knowledge-system rework (sessionCache, /tokens API, issuer.go, sweeper trim). COMPLETE (#159 + #164, commit f9a24e9).
  • Universe Onboarding — PR5 (broker /me/install) — sub-plan, shipped #141 2026-05-20. Bearer-authenticated per-user install bundle; now the identity-introspection surface alongside the MCP gateway's data plane.
  • History — content addressing, federation, persistent graph, read auth (server-side), conflict-aware merge in mark_publish (2026-05-05), Claude Code plugin (2026-04-23), Broker MCP Gateway (2026-05-23 — all 8 slices + Pre-Flight 0/1 shipped; 13-tool surface with byte-for-byte proxy fidelity to local demarkus-mcp, OIDC + RFC 9728/8414 metadata, chart + kind smoke + /knowledge-join slash command) + RFC 7591 DCR follow-on (2026-05-26, PR #153 — /register + registration_endpoint in discovery, unblocks Claude Code → cluster broker auth via the native MCP authorization spec).

Plan Archives

Original plan documents preserved for reference:

  • Content Addressing — hash-based fetch, in-memory index, mirror foundation
  • Federation — agent-driven hash discovery, mark_index, mark_resolve
  • Persistent Graph — disk-backed graph store, incremental crawl, backlinks
  • Read Auth — per-path read token enforcement for private networks
  • Security Hardening — systemd sandboxing, security docs, write isolation
  • Conflict-Aware Merge — tool-level diff3 merge in mark_publish (shipped client/v0.12.25 + v0.12.26)
  • Claude Code Plugin — one-click marketplace plugin (shipped demarkus-memory v0.1.1; v0.2.0 added /knowledge-join 2026-05-23; v0.3.0 added self-documenting guidance + lookup recall 2026-05-31)
  • Universe Onboarding — PR3 (broker device flow) — shipped 2026-05-15 (#137). RFC 8628 device flow end-to-end on the broker. Six sub-steps merged across one PR; PR4 builds on top.
  • Universe Onboarding — PR4 (broker refresh tokens) — shipped 2026-05-15 (#138 + #139). Refresh-token lifecycle + grant_type=refresh_token + POST /token/revoke + broker-signed id_tokens + /.well-known/jwks.json + compositeVerifier + Sweeper integration. Eleven CodeRabbit comments addressed in a review round; lessons captured in journal.
  • Broker MCP Gateway — shipped 2026-05-23 (v7). Eight slices + Pre-Flight 0/1, ~1800 LOC production + ~2460 tests + chart/docs across ~2 weeks. Plan stays in place as the architectural reference + decision trail (v1 REST → v7 complete changelog at the top traces every load-bearing pivot). DCR follow-on (RFC 7591 /register) shipped 2026-05-26 (PR #153) to satisfy the MCP authorization spec's discovery requirement.
  • Search Verb — superseded by LOOKUP. The full-text TF-IDF SEARCH design was descoped; full-text stays permanently in an opt-in sidecar.
  • POC Deployment — canceled. The separate-POC-slice approach was rejected in favor of "build the real product once" (see universe-deployment).
  • Obsidian Plugin — obsolete. Source moved to the standalone latebit-io/obsidian-demarkus repo (2026-04-24); monorepo copy removed.
soul.demarkus.io/plans/content-addressing.md draft reader meta

Content-Addressed Fetch — Implementation Plan

Context

Demarkus already has the building blocks for content addressing: SHA-256 hash chains in the versioned store, content hashes computed for etags, and a client cache at ~/.mark/cache/. The hub pattern just shipped for discovery. Content addressing adds the "fetch from anywhere" primitive — any server that has the content can serve it, verified by hash.

Approach

No new verbs. FETCH handles it via path detection. Five incremental steps, each independently testable.

Step 1: Add content-hash to FETCH responses

File: server/internal/handler/handler.go

  • Add computeContentHash(body string) string — SHA-256 of the stripped body (not doc.Content which includes store frontmatter)
  • Call it in serveDocument() after stripFrontmatter(), set meta["content-hash"] = "sha256-<hex>"
  • Also add to handleFetchVersion() for historical version responses
  • Tests: verify content-hash appears in FETCH responses, matches expected hash of body

Step 2: Add hash index to the Store

File: server/internal/store/store.go

  • Add fields to Store struct: hashIndex map[string]string, hashMu sync.RWMutex
  • BuildHashIndex() — walks content root following symlinks (current versions only), reads each, strips frontmatter, hashes body, populates index. Skips versions/ dirs and archived docs. Returns error only on walk failure, logs individual file errors.
  • LookupHash(hash string) (string, bool) — read-locked lookup
  • UpdateHashIndex(reqPath string, body []byte) — write-locked upsert
  • RemoveHashEntry(reqPath string) — write-locked removal by path (iterates to find matching path)
  • Tests: table-driven for build, lookup, update, remove

Step 3: Update hash index on writes

File: server/internal/store/store.go

  • In Write() after successful symlink rename (line ~576), call s.UpdateHashIndex(reqPath, content) — content here is the raw body parameter, not stored bytes
  • In SetArchived() — call RemoveHashEntry when archiving, UpdateHashIndex when unarchiving
  • Append() flows through Write() so it's handled automatically
  • Tests: write a doc, verify hash index updated; archive, verify removed

Step 4: Handle hash-based FETCH

File: server/internal/handler/handler.go

  • isHashPath(path string) (string, bool) — validates path matches /sha256-<64 hex chars> (73 chars total: / + sha256- + 64 hex)
  • handleFetchByHash(w, req, hash) — calls Store.LookupHash(hash), if not found → not-found, otherwise Store.Get(path, 0) → serveDocument()
  • Insert hash detection at the top of handleFetch(), before parseVersionPath
  • Tests: fetch by valid hash, fetch unknown hash, malformed hash path

Step 5: Wire up on startup

File: server/cmd/demarkus-server/main.go

  • After s := store.New(cfg.ContentDir) (line 102), call s.BuildHashIndex()
  • Log entry count: logger.Info("content hash index built", "entries", s.HashIndexSize())
  • Non-fatal on error (log warning, continue)

Key Design Decisions

  • content-hash is separate from etag: etag hashes doc.Content (includes store frontmatter), content-hash hashes the stripped body (what the client receives). Different values.
  • Current versions only: historical versions get content-hash in their response but aren't indexed for lookup. Keeps the index small.
  • Archived docs excluded: archiving removes from index, unarchiving re-adds.
  • In-memory index, no persistence: the content directory is the source of truth. Rebuilt on startup.
  • Path syntax FETCH /sha256-<hex>: detected by pattern match, no protocol changes needed.

Files Modified

File Change
server/internal/handler/handler.go content-hash in responses, hash path detection, handleFetchByHash
server/internal/store/store.go Hash index struct fields, Build/Lookup/Update/Remove methods, index updates in Write/SetArchived
server/cmd/demarkus-server/main.go One-line startup call to BuildHashIndex
server/internal/handler/handler_test.go Tests for content-hash, hash fetch, malformed paths
server/internal/store/store_test.go Tests for hash index operations

Verification

  1. cd server && go test ./... after each step
  2. bash pre-commit.sh at the end
  3. Manual: start server, FETCH a doc, note content-hash in response, FETCH by that hash

Related documents

trail
  1. soul.demarkus.io v42
  2. content-addressing
from here → 2026-03-08 federation architecture index 2026-03-07 read-auth