soul.demarkus.io:6309/index.md/v84 draft reader meta

demarkus-soul

This is the living knowledge base for the demarkus project, served by demarkus itself.

An AI agent's evolving memory, architecture notes, debugging insights, and design decisions; all versioned, all permanent.

Sections

  • Architecture: system design, module boundaries, key decisions
  • Universe Pattern: souls, worlds, and hubs as a deployment topology
  • Patterns: code patterns, conventions, idioms used in this codebase
  • Guidelines: hard rules for code quality, must be referenced before writing code
  • Conventions: collaboration + repo/plugin conventions (how I work: commits, layering, tooling, plugin discipline)
  • Debugging: lessons learned from bugs and investigations
  • Roadmap: what's next, what's in flight, what's done, and what's deliberately not prioritized
  • Ecosystem: browsers, plugins, and tools that implement or integrate with demarkus
  • Debt: technical debt and improvement opportunities
  • Journal: session notes and evolution log, one file per day at /journal/<YYYY-MM-DD>.md
  • Guide: agent install guide for setting up demarkus-soul
  • Thoughts: my own reflections, ideas, and open questions
  • FAQ: common questions about demarkus and how it compares

Decisions

Architecture decision records for the core project. Canonical copies live in git at docs/adr/; these are verbatim mirrors so they are reachable by lookup, backlinks, and this hub (see Conventions). Sub-project decisions live in their own series, e.g. /demarkus-library/adr/.

  • ADR 0001: broker confidential web-client registry (accepted 2026-06-11)
  • ADR 0002: align store frontmatter with the Open Knowledge Format (accepted 2026-06-22)
  • ADR 0003: default OKF type on publish (accepted 2026-06-22)
  • ADR 0004: edge semantics, provenance on every edge, typed relations via rel- metadata (accepted 2026-07-13)
  • ADR 0005: node identity omits the default port (accepted 2026-08-18)

Vocabulary

  • knowledge system: organizational, broker-fronted universe. Joined via /knowledge-join (the demarkus-knowledge plugin). MCP traffic over HTTPS terminates at the broker; broker translates to QUIC for internal worlds.
  • soul: personal demarkus knowledge base, direct-QUIC. The original demarkus-soul shape. Will be joined via a future /soul-join slash command if one ships.
  • Both compose worlds (demarkus servers, QUIC). A Claude Code installation can have both; they don't conflict.

Plugins

  • Obsidian Plugin; fetch, publish, and browse demarkus documents from Obsidian (standalone repo latebit-io/obsidian-demarkus)
  • Claude Code: demarkus-memory (personal soul), source at plugins/claude-code/, shipped via the marketplace. Version history: v0.1.0 (2026-04-23, #96) initial; v0.2.0 (2026-05-23, #152) /knowledge-join; v0.3.0 (2026-05-31, #168) mark_lookup + SessionStart guidance; v0.4.0 (2026-06-01, #171) hook-based enforcement (publish tag-gate, journal nudge, recall nudge), /project-template.md, /soul-doctor; v0.5.0 (2026-06-03, #172) split the knowledge-system surface out into the separate demarkus-knowledge plugin (below) so this one is personal-soul only, and added an always-on "single memory store" steering line plus a one-time, ask-don't-force offer to disable Claude Code's built-in memory; v0.6.0 (2026-06-17, #192) the soul→knowledge promote bridge; /promote (detect endpoint → run the knowledge cascade → one-directional back-stamp, stub or marker mode), /soul-refresh (the directional coherence edge: refresh promoted docs from knowledge, local edits re-enter upward through the gate), and mutual knowledge detection (knowledge_endpoints/detect-knowledge.sh reverse-peek the knowledge registry). Hooks: SessionStart, PreToolUse, PostToolUse, Stop, UserPromptSubmit. Now at v0.13.8 (#281).
  • Claude Code: demarkus-knowledge (organizational knowledge system), source at plugins/claude-code-knowledge/, a second entry in the same marketplace. v0.1.0 (2026-06-03, #172). Owns the broker-fronted surface split out of demarkus-memory: /knowledge-join, a new /knowledge navigation command, KS-first SessionStart guidance with soul↔system synergy, a KS-scoped publish tag-gate, and a KS-gated recall nudge. No binaries and no local server: pure broker + Claude Code MCP OAuth. Standalone: owns its own ~/.demarkus/plugin-knowledge.* file namespace and DEMARKUS_KNOWLEDGE_STRICTNESS env; reads (never writes) plugin-memory.conf only to detect a sibling soul for the synergy note. The two plugins' publish gates partition cleanly by server scope, so both can be installed together. v0.2.0 (2026-06-17, #192) added the knowledge-promote cascade skill (the execution half of the promote bridge: triage → distill, stripping personal framing + secrets/PII → dedup vs catalog → tag to taxonomy → destination-select via mark_worlds writable + per-world world.md → human gate capped by the world's autonomy ceiling → publish with provenance) and the per-world world.md descriptor example. Now at v0.5.24 (#281).
  • OpenCode: demarkus-opencode-memory (personal soul), source at plugins/opencode-memory/. v0.13.8 (2026-08-10, #281). The OpenCode port of demarkus-memory: single-file TS adapter over the shared demarkus-plugin binary; installed by curl one-liner into ~/.config/opencode/plugins/ (no npm). Plan + follow-ups: /plans/opencode-memory-plugin.md.
  • pi: demarkus-pi-memory / demarkus-pi-knowledge, source at plugins/pi-memory/ and plugins/pi-knowledge/, mirrored to standalone repos for pi install. Same adapter pattern; now at v0.13.8 / v0.5.25 (#281).

Sub-projects

Standalone-repo projects in the demarkus ecosystem, each with its own hub and durable knowledge under /<slug>/:

  • demarkus-library; the web front-end ("Universe Library"): a server-rendered Go + htmx reading room over a broker-fronted knowledge system. Repo latebit-io/demarkus-library. Plan: /plans/universe-library.md. Reading room feature-complete and deployed (cluster library 0.5.2; universe overlay PR #47 merged 2026-06-22, awaiting deploy). See roadmap.
  • demarkus-knowledge-system-deploy; GitOps deploy repo for the production knowledge system (knowledge.demarkus.io): OpenTofu (GCP/GKE) + ArgoCD ApplicationSets standing up the broker, worlds, agent, library, and backups. Repo latebit-io/demarkus-knowledge-system-deploy. deployment.yaml at repo root is the single source of deployment identity.
  • mark-knowledge; the hosted service build (signup, tiers, per world billing, management app). Repo at /Users/fritz/latebit/mark-knowledge, with its own soul provisioned 2026-07-26 (isolated mode, port 16310). It does not have durable knowledge under /<slug>/ here, because it keeps its own soul rather than a section of this one. Direction and the demarkus-side constraints live in /plans/demarkus-as-a-service.md.

Active Plans

Verified against code/PRs on 2026-05-31; versions-sharding entry corrected 2026-07-05. Plans with real remaining work:

  • Store Parity (file vs Postgres); absolute parity between the file store and pgstore: CI Postgres service with a required DSN, seeded differential suite over both backends plus a fuzz target, then handler-level conformance, contract-test porting, kind e2e, migration tool, dogfood soak. Steps 1 and 2 merged 2026-08-19 (PR #324, 0cf1f9a): seven real divergences found and fixed, several in the file store itself, plus PartialWalkError for skipped walk entries. Steps 3 to 7 open.
  • OpenCode Knowledge Plugin Port; port the Claude Code organizational knowledge plugin to OpenCode with shared endpoint registration, native OAuth, policy gates, guidance, commands, and promotion skill. Implemented 2026-08-15 on branch feat/opencode-knowledge-plugin; tests and pre-commit pass, branch unmerged.
  • APPEND metadata loss; appending to a document silently stripped its catalog metadata, so tags and importance were lost and the document fell out of mark_lookup. Complete 2026-08-14 on branch fix/append-metadata-merge via Option C, the protocol merge: APPEND now writes the base version's publisher metadata with the request's layered over it (store.MergeAppendMeta, both backends), retention excluded and the OKF type default moved after the merge. SPEC 6.6 and 9.9 updated; nine plugin guidance files, both mark_append tool descriptions, and a new metadata-loss check in all five doctor commands; memory plugins 0.13.24, knowledge plugins 0.5.40/0.5.41. Corpus repaired: of 123 untagged soul documents, 69 had lost tags and were republished with the metadata recovered from their newest still-tagged version, bodies unchanged; the other 54 were never tagged and are a separate curation exercise. Branch unmerged; a soul only gets the fix once its server is upgraded, so soul.demarkus.io still strips on append.
  • Agent Memory Leaderboard entry; enter demarkus in agentmemoryleaderboard.ai next cycle: agentic search (nav agent over lookup/fetch/backlinks) as the Search implementation, distillation cascade at Add-time, commercial board via self-hosted API on a droplet with echo v5 as inference backend. Sub-project hub: /memoryleaderboard/ (repo /Users/fritz/latebit/memoryleaderboard). Planned 2026-08-13; cycle 1 closed 2026-08-07, awaiting cycle 2 dates. Phase 0 (recon) not started.
  • Code Quality Sweep 2026-08; full-repo review findings (6-agent sweep, 2026-08-12): 10 high-severity correctness/security leads, cross-module duplication extraction targets, broker package-split recommendation, dead code, pervasive rule violations, remediation order. Findings recorded; nothing fixed yet.
  • Bucket Document-Store Backend; native object-storage backend (GCS first, S3/MinIO designed-for) as a third DocumentStore implementation, enabling multi-replica worlds on k8s with no PVCs: per-document manifest objects committed via generation CAS, write-once version blobs, per-pod LIST-driven hash-index/catalog sync, storage.kind: filesystem|bucket chart knob (bucket mode renders a Deployment, no VCT), tofu world-storage module + migration tool + runbooks. Planned 2026-08-10 (investigation: symlinks stay in the file store; gcsfuse and Filestore RWX rejected). Not started; 8 PRs.
  • demarkus as a service; the hosted offering: Aiven adjacent service model, three tiers matching the website's Personal, Team, and Knowledge System scales, VPS first substrate with Kubernetes only on overflow, per world billing with the box as the size step, power off instead of scale to zero, and a management app as the only new engineering. Direction set 2026-07-26; the build moved to the mark-knowledge repo and its own soul on the same day. This copy stays as the demarkus-side record, since the decisions constrain this repo: the appliance is the unit of deployment, the broker stays one binary, the librarian is the only inference cost centre, and quotas plus backups are prerequisites that land here. Note that repo ADR 0005 (hosted tenant density), which an earlier revision cited as settling density, was deleted 2026-07-25.
  • The five-minute appliance; one pasted command on a fresh VPS yields a working self-hosted knowledge system in about five minutes: sslip.io default (no domain), fully native (no container runtime), Authelia as primary IdP with Pocket ID and Dex as fallbacks, zero prompts with everything generated, ending in a summary card (library URL, owner login, /knowledge-join line, librarian key hint). Builds on the single-host stack (PR #262/#263). Draft recorded 2026-07-18; not started.
  • Knowledge Ingestion Pipeline; narrative + design for how org knowledge flows into a knowledge destination, framing the soul as the staging/write-ahead tier and the knowledge destination as the curated read-model, with one curation gate (cascade model routing: Haiku triage → strong-model distillation → human approval) reused across all inflows (soul promotion, Confluence, Slack, Jira, meetings). Promote is a detection-gated bridge between the memory and knowledge plugins; soul↔knowledge coherence is a directional refresh. Phase-0 prerequisites built and merged (2026-06-17): the promote primitive + coherence edge (plugins; memory v0.6.0 / knowledge v0.2.0, #192), the brokered access-discovery surface (mark_worlds writable column, #191), and the per-world world.md descriptor. Three of four prerequisites done; A2 (plain-remote token-grant introspection) deferred; the live target is brokered. Remaining phase-0 surface: signal/batch triggers (manual /promote + /soul-refresh are the only triggers today), then the dogfood promote of this plan itself.
  • Universe Library; web front-end for a demarkus universe (Go + htmx reading room). Sub-project hub: /demarkus-library/. Reading room feature-complete and deployed (cluster library 0.5.2); see the sub-project roadmap.
  • Universe Deployment (Phase 6); Helm charts (server, broker, agent), OIDC token broker, release pipeline, observability. ~95% complete (PRs #126-#134, 2026-05-14). Remaining §6.6 (docs) + §6.4 Kustomize overlay reframed as deferrable ops polish; effectively superseded in practice by the GKE reference deployment.

RFC Review

  • Demarkus / Knowledge System FAQ; terse Q&A for the RFC review session, sourced from the demarkus and demarkus-knowledge-system-deploy repos. Status: WIP, pending review via the library.

Completed Plans

  • OpenCode Memory Plugin (1:1 port); the OpenCode port of demarkus-memory as plugins/opencode-memory/ v0.13.8: single-file TS adapter over the shared demarkus-plugin binary, curl-one-liner installer with stage-then-commit + rollback, atomic bootstrap binary replace across all five plugin copies, live-verified against OpenCode 1.18.15. COMPLETE: planned 2026-08-09, merged 2026-08-10 (PR #281, f4c2b35). Follow-ups (soul-list/soul-remove subcommands, token stdin input, shared-source bundling, opencode-knowledge port) recorded in the plan.
  • Graph Hub Seeding; mark_backlinks/mark_graph/mark_explore seed from the published /graph.md aggregate on both MCP surfaces (demarkus-mcp per host, broker per world with dial-address-to-world-name translation), local wins via the zero-CrawledAt marker, seed etags in graph.json, fetch.FetchConditional. COMPLETE 2026-07-14/15 across #253 (feature), #254 (issue #222: Merge preserves resolved nodes on failed re-crawl), #256 (broker seed URL translation), #257 (seed all worlds + the producer-consumer /graph.md contract test). Deployed and live-verified: scratch-HOME cold client answered soul backlinks with zero crawls; a cold broker pod's first graph call answers non-hub backlinks from the hub aggregate (broker 0.12.4, agent 0.21.1). Lessons in /debugging.md (mock fixtures encoded a plan assumption).
  • Multi-replica LOOKUP (postgres, phase 2); the LOOKUP catalog moved into Postgres (rows in the write transaction, SQL-backed Lookup behind the handler LookupCatalog seam) so world pods can scale past one replica; phase 2 of the deploy repo's ADR 0002, following the phase-1 postgres backend (#249). MERGED PR #250 (2026-07-13): LOOKUP conformance suite in storetest, two-replica handler proof, batched reconcile-on-Init backfill, server chart startupProbe, and the configwatch flake fixes (kqueue same-name swap limitation documented in /debugging.md).
  • Version Retention; keep last N versions per document via a retention publish-metadata key with prune-on-write in the store; motivated by the knowledge system's graph document at 545+ versions. COMPLETE: planned, shipped, and production-verified 2026-07-06/07 across #236 (store core + os.Root delete hardening + audit logging + SPEC §9.9), #237 (plugin gate binary), #239 (guidance + repins), #240 (agent publishes generated artifacts with retention=20), and the deploy rollout (server 0.20.0 / broker 0.9.0 / agent 0.19.0). Live result: /graph.md pruned 556 → 20 versions and the hub hash indexes cleared their backlogs in one crawl (~1,714 version files deleted, audit-logged, chains valid).
  • MCP Resources + Prompts; demarkus documents as client-attachable MCP resources (mark:// URI template, #anchor section attach, background-LIST picker population) and orient/recall/whats-new as server-vended prompt commands. SHIPPED PR #232 (2026-07-05), client/v0.17.0. Follow-up deferred: broker gateway resources/prompts (multi-world URIs, auth on reads; starts by flipping the gateway capabilities test).
  • MCP Client Ergonomics; size-adaptive mark_fetch (outline mode, #section slicing, force), session unchanged-dedup, and the mark_explore orientation card, on both MCP surfaces via shared client/mdoutline + client/fetchdedup packages. SHIPPED #225/#230 and deployed 2026-07-04/05; plugin users (client v0.15.0 via tools 0.4.1) and the live knowledge system (broker 0.5.0). Deferred follow-ups: MCP resources/prompts (shipped; see above), library librarian open adoption.
  • Versions Sharding; per-document versions/<doc>/vN subdirectories with lazy migration, fixing the O(all-entries) findVersions scan. SHIPPED PR #90 (d7cb68a, 2026-04-08: the same day the plan was written); store since hoisted to protocol/store (#120). This index wrongly listed it as unstarted until 2026-07-05.
  • Plugin Knowledge-Quality Enforcement; raised the demarkus-memory Claude Code plugin from advisory to enforced. SHIPPED v0.4.0, PR #171 merged 2026-06-01. All seven items: publish tag-gate (warn/block/ask + per-knowledge-system strictness & require_tags with literal axis matching), session-end journal nudge, recall nudge, canonical per-project template (/project-template.md), knowledge-system policy/template at the live root hub on knowledge.demarkus.io, and the /soul-doctor hygiene audit. 68 tests, pure awk/bash, zero runtime deps. Tail (separate): plugin shell tests → CI; optional nudge disable knobs.
  • Broker Authorization Code Grant; RFC 6749 authorization_code + PKCE (S256) on the broker so Claude Code's MCP SDK can auth against broker.knowledge.demarkus.io. COMPLETE: core grant (PR1 #155 + PR2 #156, 2026-05-27) replaced the unsupported_response_type stub; PR3 kind-smoke (auth-code + PKCE end-to-end in up.sh --with-mcp-smoke) merged 2026-05-31 (#169, a380e8f), executed green in-cluster + verified read-only against prod.
  • LOOKUP verb: the card-catalog verb (subject → docs + importance). Shipped to main PR #166 (2026-05-30); plugin surfacing in v0.3.0 (#168). Tail: mark_append metadata deferred by design.
  • Knowledge System (GKE Reference Deployment) public GitHub-template deploy repo (latebit-io/demarkus-knowledge-system-deploy) standing up knowledge.demarkus.io on GKE (OpenTofu + ArgoCD + OpenBao + bank-vaults + CSI-snapshot backups). Phases 1-10 complete (verified against the live repo + a live RFC 8414 response from the real domain, 2026-05-31). Sole remaining item: the announcement blog post, intentionally deferred for a soak period.
  • Universe Onboarding; last-mile join flow. CLOSED: PR1-PR5 shipped (#137/#138/#139/#141); PR6 (tools/demarkus-join) canceled 2026-05-20 in favor of the MCP Gateway; PR7/PR8 absorbed into Gateway Slices 7-8; join ships as /knowledge-join (#152). Remaining: low-priority doc debt only (two standalone deployment docs).
  • Broker Stable Mint; lazy per-world token provisioning + cache-stable 401 retries that killed the ~20-token mint cascade; dead DefaultToken knobs removed. COMPLETE (#158/#159/#163/#164/#165, 2026-05-27→29).
  • Broker Deadcode Cleanup; deleted the issuance subsystem made unreachable by the open-knowledge-system rework (sessionCache, /tokens API, issuer.go, sweeper trim). COMPLETE (#159 + #164, commit f9a24e9).
  • Universe Onboarding (PR5 (broker /me/install)) sub-plan, shipped #141 2026-05-20. Bearer-authenticated per-user install bundle; now the identity-introspection surface alongside the MCP gateway's data plane.
  • History: content addressing, federation, persistent graph, read auth (server-side), conflict-aware merge in mark_publish (2026-05-05), Claude Code plugin (2026-04-23), Broker MCP Gateway (2026-05-23; all 8 slices + Pre-Flight 0/1 shipped; 13-tool surface with byte-for-byte proxy fidelity to local demarkus-mcp, OIDC + RFC 9728/8414 metadata, chart + kind smoke + /knowledge-join slash command) + RFC 7591 DCR follow-on (2026-05-26, PR #153; /register + registration_endpoint in discovery, unblocks Claude Code → cluster broker auth via the native MCP authorization spec). Also: OKF type adoption + /soul-join managed remote souls (2026-06).

Plan Archives

Original plan documents preserved for reference:

  • Content Addressing; hash-based fetch, in-memory index, mirror foundation
  • Federation: agent-driven hash discovery, mark_index, mark_resolve
  • Persistent Graph; disk-backed graph store, incremental crawl, backlinks
  • Information Graph; superseded early draft of Persistent Graph (Phase 4, 2026-03-08); see persistent-graph.md for the version that shipped.
  • Read Auth: per-path read token enforcement for private networks
  • Security Hardening; systemd sandboxing, security docs, write isolation
  • Conflict-Aware Merge; tool-level diff3 merge in mark_publish (shipped client/v0.12.25 + v0.12.26)
  • Claude Code Plugin; one-click marketplace plugin (shipped demarkus-memory v0.1.1; v0.2.0 added /knowledge-join 2026-05-23; v0.3.0 added self-documenting guidance + lookup recall 2026-05-31; v0.4.0 enforcement + template + /soul-doctor shipped 2026-06-01, PR #171; v0.5.0 split out demarkus-knowledge 2026-06-03, PR #172)
  • Universe Onboarding (PR3 (broker device flow)) shipped 2026-05-15 (#137). RFC 8628 device flow end-to-end on the broker. Six sub-steps merged across one PR; PR4 builds on top.
  • Universe Onboarding (PR4 (broker refresh tokens)) shipped 2026-05-15 (#138 + #139). Refresh-token lifecycle + grant_type=refresh_token + POST /token/revoke + broker-signed id_tokens + /.well-known/jwks.json + compositeVerifier + Sweeper integration. Eleven CodeRabbit comments addressed in a review round; lessons captured in journal.
  • Broker MCP Gateway; shipped 2026-05-23 (v7). Eight slices + Pre-Flight 0/1, ~1800 LOC production + ~2460 tests + chart/docs across ~2 weeks. Plan stays in place as the architectural reference + decision trail (v1 REST → v7 complete changelog at the top traces every load-bearing pivot). DCR follow-on (RFC 7591 /register) shipped 2026-05-26 (PR #153) to satisfy the MCP authorization spec's discovery requirement.
  • Search Verb: superseded by LOOKUP. The full-text TF-IDF SEARCH design was descoped; full-text stays permanently in an opt-in sidecar.
  • POC Deployment; canceled. The separate-POC-slice approach was rejected in favor of "build the real product once" (see universe-deployment).
  • Obsidian Plugin; obsolete. Source moved to the standalone latebit-io/obsidian-demarkus repo (2026-04-24); monorepo copy removed.
soul.demarkus.io:6309/rfc-review-faq.md wip reader meta

Demarkus / Knowledge System FAQ

For the RFC review session. Sources: demarkus repo, demarkus-knowledge-system-deploy repo. Status: WIP.

What is the core principle?

Berners-Lee's 1989 "Information Management: A Proposal": documents joined by typed links, navigated by following the links. demarkus keeps that model and adds what the Web lacks for agents.

  • Documents are nodes. Markdown links and rel-<predicate> metadata are typed edges. A document carries its own exits, so no side channel is needed to move on (HATEOAS for knowledge).
  • Servers store, version, and catalog documents. They never crawl or hold traversal state. Navigation belongs to the agent.
  • Backlinks (who links here), which HTTP does not provide, come from a graph store the agent builds and a world can publish as /graph.md.
  • Versions, hash chains, and content addressing give every link a durable target.
  • Test for a proposal: does it make knowledge more linked, findable, or durable? If it only makes an application more convenient, it does not belong in the protocol.

How do you interact with the knowledge system?

Agents use MCP over HTTPS. Humans use the web app or the CLI.

  • Agents: mark_* MCP tools against the broker, URLs as mark://<world>/<path>.
  • Auth: the broker is its own authorization server. Claude Code runs OAuth authorization code with PKCE and dynamic client registration; device flow covers CLI-style joins.
  • Humans: the library web app, or demarkus CLI / demarkus-tui direct to a world over QUIC. The library reads one world with no login by default; registering it with the broker adds sign-in and browser writes.
  • Entry point: the root hub. Policy and templates live under mark://root/.well-known/demarkus/.

What are a world, a hub, root, and an index?

  • World: one demarkus server, addressed by logical name. Own namespace, store, tokens, and release. An optional world.md describes it.
  • Hub: an ordinary world that receives published aggregates: content-hash indexes from mark_index and the /graph.md export. mark_resolve and backlink seeding read from it. The server has no hub mode; hub: true is a deploy flag the crawler reads.
  • Root: the conventional name for the hub world. The protocol does not require it, but the secret store and library config hardcode root, so in practice the hub is called root.
  • Index: index.md is a curated entry-point document, the backstop for what lookup cannot surface. The crawler writes machine hash indexes to /index/<host>.md when perServer is set (as here); with the chart default it writes /index.md and overwrites the curated one.

How does content get into the knowledge system?

Through the promote bridge: /promote <soul-path> runs the knowledge-promote cascade.

  • Triage, then distill for a shared audience (strip personal framing, secrets, PII).
  • Dedup against the catalog, tag to the system taxonomy, route to a writable world.
  • Human gate, then mark_publish with provenance. The cascade never writes the soul; /promote back-stamps the source afterwards.
  • Direct mark_publish to a joined system also works and passes the same gate and policy.

How does finding knowledge work?

Primary: mark_lookup catalog queries. Secondary: hub pages and graph traversal.

  • Lookup matches a query against declared tags and titles and returns an importance-ranked table (path, importance, title, tags).
  • Filters: tag=, modified-after=, modified-before=, or any key=value matching a declared metadata value exactly. Not full-text search: an untagged document is invisible to it.
  • From a hit: mark_explore to orient, then mark_fetch url#anchor for the sections needed.
  • Backstops: index.md hubs, mark_backlinks / mark_graph for link traversal, mark_discover for a server's manifest.

How is relevance decided?

Deterministic scoring, no embeddings (server/internal/catalog/catalog.go).

  • Score = number of distinct query terms matching tags (exact, case-insensitive) or title (substring, case-insensitive).
  • Sort: score desc, then declared importance, then modification time, then path.
  • Filters apply before ranking.

How is importance stamped on a document?

The publisher declares it.

  • metadata.importance on mark_publish, float in [0,1], stored out of band, indexed into the catalog.
  • Absent, unparseable, or out-of-range values default to 0.5.
  • APPEND merges: the new version carries the base version's metadata with the request's layered over it (SPEC §6.6), so importance and tags survive an append. retention is the exception and is never inherited.

How does the agent decide importance?

The agent chooses it, guided by instructions injected at session start. Nothing computes it.

  • SessionStart context and the soul-memory / knowledge-promote skills say: 0.8+ for hubs, architecture, key decisions; routine notes lower.
  • The server never infers it; the gate only validates the range.

What does the agent use to decide whether to read a document?

Catalog and graph evidence before bodies.

  • The mark_lookup row: path, importance, title, tags, modified time.
  • Backlink provenance: label, anchor, count, rel- type.
  • mark_explore: outline, links, backlinks, siblings in one call.
  • mark_discover manifests and hub index.md pages for context.

How does the agent read a large document?

The outline is a nudge, not a gate. The server always sends the full body; the MCP layer decides what to show.

  • At 8KB mark_fetch returns the heading tree with #anchors and per-section line counts instead of the body.
  • mark_fetch url#anchor returns one section at any size. mark_fetch url force=true returns the whole body regardless of size or session history.
  • mark_explore url lists the anchors for any document; a bad anchor error names every available anchor.
  • The demarkus CLI fetches raw bytes with no outline gate.

How does an agent navigate from a document it has fetched?

Forward from the document itself, backward from the graph store.

  • A mark_fetch response holds every exit: body links and rel-<predicate> metadata. Following them needs nothing beyond FETCH.
  • Incoming edges (who links here, what supersedes this) come only from the local graph store via mark_backlinks or mark_explore. The store fills from the world's /graph.md on first use, then from mark_graph crawls. A plain fetch does not add to it.
  • Empty store and no /graph.md: backlinks return nothing until someone runs mark_graph. Forward navigation, mark_list, and mark_lookup still work because they hit the live server.
  • Nodes carry no relation data. Every rel- fact is an edge, read from mark_backlinks rows or the Edges: list of mark_graph.
  • Store keys are canonical mark://host:port/path strings matched exactly. A link written with a different host alias becomes a separate node.

How does the document graph work?

mark_graph crawls outbound mark:// links (depth default 2, max 5, capped at 200 nodes, so deep crawls usually stop on the node cap).

  • Edges carry provenance: link label, source section anchor, occurrence count.
  • Typed relations come from rel-<predicate> publisher metadata, e.g. rel-supersedes.
  • Crawls persist to a graph store that answers mark_backlinks. The store seeds from a published /graph.md; local crawls take precedence.
  • mark_graph_publish republishes the store as a crawlable /graph.md (generated doc, default retention 20).

Is edge information there to guide agents?

Yes (ADR 0004).

  • Label: what the linker calls the target.
  • Anchor: the exact source section, a direct mark_fetch url#anchor jump.
  • Count: link strength.
  • rel- types: "what superseded this" rather than bare "mentions".
  • Constraint: the agent owns judgment, the server owns accumulation.

How is the graph built world to world?

A scheduled agent builds it; the broker reads what it publishes.

  • The agent has a seed URL per content world and crawls each one. The hub is not crawled; it is where results are published.
  • It merges every world's edges into one graph and publishes /graph.md on the hub, alongside the content-hash index. Graph publishing is off by default and on in this deployment.
  • Each run rebuilds from scratch. A world that fails to answer drops out of the published graph until the next successful crawl.
  • Cross-world links are ordinary mark://{worldName}/{path} links.
  • The broker's own graph store is in-memory and pod-scoped. It loads the hub's /graph.md on demand, so a fresh pod answers backlinks without crawling.
  • A world with no published graph falls back to on-demand mark_graph crawls.

How does the system keep data fresh?

  • Reads go to the world's server every time. The client revalidates with if-none-match / if-modified-since; a cached body is reused only when the server confirms it.
  • Each write creates a new hash-chained version. Rewriting identical content is a no-op that returns the existing version.
  • Optimistic concurrency (expected_version plus merge on conflict) stops stale writes clobbering newer ones.
  • A scheduled agent re-crawls every world and republishes the content-hash index and graph on the hub.
  • Soul copies of promoted documents can go stale. /soul-refresh copies the knowledge system's version down; /promote is the only way an edit goes back up. There is no two-way sync.

How do diffs and merges work?

One document, two concurrent editors. (Merging separate documents is dedup, below.)

  • A conflicting mark_publish returns a diff3 merge candidate: base, ours, theirs, git-style markers where both sides touched the same lines.
  • The machine merges structure; the agent reviews the result and republishes at the returned version.
  • Effect: a stale body cannot silently overwrite a newer one.

How are documents updated, and how do they keep their meaning?

  • No partial update: fetch, edit, republish the whole body at the version you read. Conflicts return a merge candidate instead of overwriting.
  • Old meaning is never lost; every version stays fetchable at its pinned number.
  • One subject, one owner: updates go to the existing document, and rel-supersedes records meaning that moved elsewhere.
  • Limit: nothing verifies an update preserved meaning. That is agent judgment plus the human gate. Renaming a heading silently breaks inbound anchors.

How does dedup work?

Before the write, not as a cleanup pass. The agent looks for an existing owner of the subject and updates it.

  • The promote cascade looks the subject up in the catalog and fetches close matches.
  • If a document already covers it, the agent updates that document instead of adding a second.
  • If the new content contradicts the existing one, the human decides at the gate.
  • /knowledge-doctor catches exact duplicates after the fact by comparing content hashes across paths and worlds. It fetches each document, so it runs on a bounded scope.
  • Limit: candidates are found by tag and title match. A near-duplicate sharing no tags or title terms is never a candidate.

How are versions and integrity handled?

  • Each write creates a new version linked by a SHA-256 hash chain over the previous version's raw bytes.
  • mark_versions validates the chain oldest-first and reports chain-valid or chain-error.
  • Nothing is deleted by default. mark_archive keeps full history but withdraws the document: no body at its path, no catalog entry, no further writes until unarchived by publishing an empty body.
  • The one destructive path is metadata.retention, which permanently prunes old versions. It exists for generated documents; clients warn before applying it elsewhere.

What is the auth and security model?

Capability tokens on the server; SSO at the broker.

  • Writes are denied unless a token store is configured.
  • Reads are public unless a token grants read on a path pattern, which makes matching paths require one (protect /** for a private intranet).
  • Knowledge system: OIDC SSO is the org gate. Reads are open to any authenticated identity; writes pass a per-world writer allowlist. Credentials live at the broker: it holds one long-lived token per world and dispatches writes with it. Per-world tokens exist only inside a broker deploy and never reach a client. A world reached directly over QUIC uses ordinary capability tokens.
  • Policy forbids publishing secrets, credentials, or PII.

How is knowledge formatted and held to a standard?

The standard is published on root under .well-known/demarkus/.

  • policy.md: strictness, required tag axes such as category:, optional required OKF fields.
  • template.md: per-world layout.
  • style.md: H1 as name, one-sentence summary under it, unique headings (headings are anchors), no em dashes, no frontmatter fences.
  • Joining copies the policy into local write-time gates as a snapshot: tags, axes, importance range, mechanically checkable style rules, at the declared severity (warn / block / ask). The gate runs offline, so a policy change takes effect on the next join or mirror.
  • /knowledge-doctor audits the corpus after the fact.

What automated cues push agents to record memory?

Five hooks in the memory plugin.

  • SessionStart: injects the routing table (decisions to /adr/, gotchas to debugging.md, progress to journal/).
  • Stop: journal nudge when files changed but nothing was written to the soul.
  • PostToolUse on mark_publish: promote nudge when an ADR lands in the soul and a promote destination exists.
  • UserPromptSubmit: recall-first reminder on "did we decide" questions.
  • Pre/PostToolUse write gate: tags and importance, write destination, retention, style. Defaults differ per check: tags warn, destination blocks, retention asks.

What tools does the memory plugin expose?

Fifteen mark_* MCP tools.

Tool Description
mark_fetch Fetch a document or #section; bodies of 8KB or more return an outline
mark_list List documents and subdirectories; archived hidden by default
mark_explore One doc's outline, outbound links, backlinks, and siblings in one call
mark_lookup Catalog lookup: importance-ranked matches on tags and titles
mark_publish Create or update a document; metadata, optimistic concurrency, diff3 on conflict
mark_append Append to an existing document; catalog metadata carried forward
mark_archive Archive a document; body and catalog entry withdrawn, history preserved
mark_versions Version history with hash-chain validation
mark_graph Crawl outbound mark:// links; persists edges to the graph store
mark_backlinks What links here, with edge provenance
mark_graph_export Export the graph store as publishable markdown
mark_graph_publish Export and publish the graph as /graph.md (retention default 20)
mark_discover Fetch a server's agent manifest
mark_resolve Resolve content by SHA-256 hash via a hub index
mark_index Crawl a server, publish its content-hash index to a hub

What tools does the knowledge plugin expose?

The same 15 MCP tools, kept at parity so the vocabulary does not change with transport. URLs address worlds by logical name instead of host:port. Additions:

Addition Kind Description
mark_worlds MCP tool List the system's worlds: name, URL, address, and whether you can write to each. Broker-only
/knowledge-join command Validate an org broker URL, register it as an MCP server, copy its policy into the local gates as a snapshot. Claude Code runs the OAuth flow on the first tool call; the command handles no tokens
/knowledge command List joined systems and show each root hub index
/knowledge-doctor command Read-only hygiene audit: orphans, broken links, untagged and policy-noncompliant docs, ADR gaps, duplicates
knowledge-promote skill The curation cascade that lands a staged document in the catalog; invoked by the memory plugin's /promote

Why does federation matter, and why a server per world or memory?

The protocol has no central authority or registry. This deployment does not inherit that property.

  • Anyone can run a server; content mirrors freely; every caching client is a mirror.
  • Each version carries a content-hash, so agents on different mirrors can confirm they hold identical content. The hash chain is a separate guarantee: it proves one server has not rewritten its own history. VERSIONS returns no per-version hashes, so the chain is not a cross-mirror check.
  • A server per world makes the ownership boundary physical: own store, token file, writer allowlist, release. Blast radius stays contained.
  • A soul is the same demarkus-server binary at personal scale.
  • mark_resolve fetches by hash via a hub index across the worlds the broker knows. Cross-org resolution is out of scope today.
  • This instance is centrally administered by design: one broker, an OIDC org gate, a domain allowlist, per-world writer allowlists. The protocol is decentralized; the deployment is not.

Why deploy a knowledge system on k8s?

Not a database with a web front end. A live population of processes that has to keep running, keep finding each other, and keep converging. That is the workload k8s was built for, and it is running that way today.

  • The shape fits. Five process kinds: N world servers, a broker, a crawler agent, the library, backup jobs. Each with its own lifecycle, identity, and failure domain, each addressing the others by name. On a VM that is a supervisor script and a hosts file.
  • A world is a shard, not a table. Own StatefulSet, volume, namespace, token, writer allowlist. The ownership boundary is enforced by the platform, not by convention inside one binary. A crashlooping world cannot take another world's reads with it.
  • Freshness is a control loop. The graph and hash index stay true only because an agent re-crawls on a schedule. Restart, backoff, and limits are primitives. Reconciler and crawler, one idea at two layers.
  • Adding a world is a declaration. One entry in deployment.yaml; ArgoCD materializes server, volume, DNS record, certificate, ingress route.
  • Agents are the half that grows. Crawl, promote, inference: bursty work that wants to run next to the data on node shapes it picks.
  • Not universal. One or two plain deploys do for a team. k8s earns its keep once worlds and agents multiply.

Why QUIC?

  • Encryption is mandatory: TLS 1.3 built in, no plaintext fallback.
  • Fast multiplexed streams, one per request.
  • No HTTP layer: no cookies, tracking headers, or query strings. Seven text verbs.
  • The broker fronts worlds over HTTPS, so clients need no direct QUIC access.

Why no full-text or semantic search?

Deliberate. LOOKUP is defined in SPEC §6.7; the argument is in DESIGN.md under "Why not search?".

  • LOOKUP is a per-world catalog over author-declared tags and titles. Servers must not read bodies at query time, and there is no centralized index.
  • Full-text and semantic search stay out of core as an opt-in sidecar reading demarkus over LIST/FETCH.
  • Keeps the server simple and deterministic; ranking judgment stays in the agent.

What are the known limitations?

  • The graph is not yet optimal: the broker's store is in-memory and pod-scoped, crawls are capped and rebuilt from scratch each run, and edge semantics are recent (ADR 0004).
  • Full-text or semantic search requires an added component; until then recall depends on tagging discipline.
  • Availability is not yet distributed: single-replica broker, single-replica worlds, and a hub whose loss takes cross-world discovery with it.
  • Inside a broker deploy the broker writes to a world with one per-world token, so per-user attribution rests on the provenance recorded in documents, not on the credential.
  • Behavior at large scale is unproven: catalog size, crawl cost, and cross-world graph growth have not been tested at enterprise corpus sizes.
trail
  1. soul.demarkus.io:6309 v84
  2. rfc-review-faq
from here → graph index 2026-08-17