soul.demarkus.io/roadmap.md/v48 draft reader meta

Roadmap

Where demarkus has been and where it's going.

Scope: Markdown-Only Server

The demarkus server serves markdown only. Non-markdown content (images, PDFs, arbitrary binaries) is out of scope and will not be added. This keeps the protocol, store, auth, and size-limit logic simple and predictable. Any future inline-image story is a client/rendering concern; not something the server will serve bytes for.

Phase 1: MVP (Read-Only): COMPLETE

Everything shipped:

  • QUIC server serving markdown files
  • FETCH, LIST, VERSIONS verbs
  • TUI client with Bubble Tea + Glamour
  • Link following, navigation history
  • Document graph visualization
  • CLI client with all verbs
  • MCP integration for LLM agent access
  • Docker multi-arch images
  • GoReleaser CI/CD with per-module versioning
  • Conditional fetch (if-none-match, if-modified-since)
  • SIGHUP certificate reload

Phase 2: Publish Operations: COMPLETE

Done:

  • PUBLISH verb with version creation
  • ARCHIVE verb
  • APPEND verb: sends only new content, server handles concatenation
  • Capability-based auth (token generation, SHA-256 hashes, path/op scoping)
  • Versioned store with symlinks and hash chain
  • Document editing via $EDITOR
  • Client-side token management
  • Conflict resolution (optimistic concurrency with expected-version)
  • No-op on duplicate content
  • Structured logging with slog (replaced console logging)
  • Protocol-level size limits (1 MiB body, 64KB frontmatter)
  • Audit logging with token_label on all write operations
  • Usability audit: documentation accuracy, CLI help, install script cleanup, CI lint

Phase 2 is fully complete. No remaining items.

Phase 3: Agent-Native & Advanced Features: COMPLETE

Done:

  • Agent manifest discovery (/.well-known/agent-manifest.md)
  • Bookmarks/favorites
  • MCP mark_append auto-resolve expected_version
  • Content-addressed fetch
  • Federation via MCP tools

Phase 4: The Information Graph: COMPLETE

Done:

  • The Demarkus Hub pattern
  • Persistent graph store
  • Backlinks
  • Graph as content (export/import)
  • Graph-aware navigation (TUI)
  • Agent discovery

Phase 5: The Demarkus Agent & Private Networks: IN PROGRESS

Read Auth for Private Networks: DONE

Per-path read token enforcement. Server-side and client-side shipped.

Security Hardening: DONE

See plan for full details.

  • Systemd hardening: install script generates units with ProtectSystem=strict, ReadWritePaths, NoNewPrivileges, etc. Conditional ProtectHome. Update path detects insecure config and prompts to harden with rollback.
  • -read-only mode: DEMARKUS_READ_ONLY env var / -read-only flag. Handler rejects PUBLISH/APPEND/ARCHIVE with not-permitted. Zero write access needed.
  • demarkus-publish: CLI tool that writes directly to the versioned store on disk. Enables local publishing when server runs read-only. Shares store.Write() with the server.
  • Read-only chroot install: install-readonly.sh. Chrooted server with ReadOnlyPaths=/, BindReadOnlyPaths=/dev/urandom. Maximum lockdown: Gemini-level security with full versioning.
  • Security documentation: attack surface analysis, threat model, comparison table, hardening guide.

Core Loop: Crawl & Index

  1. Seed: start from configured servers (hub, known peers)
  2. Crawl: follow mark:// links, discover new servers and documents
  3. Hash: collect content hashes from every document
  4. Index: publish updated hash indexes to configured hubs
  5. Repeat: on a configurable schedule, with conditional fetch (if-none-match) to be polite

Server-to-Server Sync

Rsync for the Mark Protocol. Replicate content between servers using content hashes as the diff mechanism.

How it works:

  1. LIST both source and destination servers
  2. Compare content hashes: skip documents that match
  3. FETCH changed/new documents from source
  4. PUBLISH to destination with the fetched content
  5. Optionally handle deletions (ARCHIVE on destination for docs removed from source)

Sync modes:

  • Mirror: destination becomes an exact copy of source (one-way)
  • Selective: sync specific paths or glob patterns (e.g., /docs/* only)
  • Multi-source: aggregate content from multiple servers into one destination

Key Design Points

  • Go binary, not an LLM agent: mechanical work, not reasoning
  • Polite crawling: conditional fetch, configurable rate limits
  • Hub-aware: reads and updates hub index documents
  • Daemon or cron: continuous or triggered
  • Auth-aware: passes read and write tokens for private servers

Implementation Sketch

  • New module: tools/demarkus-agent (the tools/ directory is the home for utility binaries; demarkus-token, demarkus-publish, and now sync/agent tooling)
  • Reuses: fetch.Client, graphstore, client/internal/index, graph.Crawl
  • CLI: demarkus-agent crawl, demarkus-agent sync source dest, demarkus-agent daemon

External Tool Sync: PLANNED

The shape that lets any markdown-emitting tool (OpenSPDD, ADR generators, doc generators, custom workflows) land its output on demarkus without integrating the protocol itself. The integration burden lives on demarkus's side; upstream tools change nothing.

demarkus-sync: single binary in tools/. Watches a local directory and PUBLISHes (or writes via demarkus-publish against a local store) on every changed .md. Versioning happens automatically through the existing store. Cross-team sharing emerges naturally from publishing to a team server.

Core flags:

  • --watch <dir>: directory to watch
  • --target mark://host/path/: remote demarkus server (uses fetch.Client PUBLISH)
  • --store <local-path>: alternative: write directly to a local versioned store via store.Write
  • --include / --exclude glob filters (e.g. **/*.canvas.md)
  • --token / DEMARKUS_AUTH for write auth

Project-flow integration is the killer use: drop into a Makefile target, pre-commit hook, CI step, or long-running dev daemon. Design contracts, ADRs, RFCs, runbooks, agent canvases all flow into the team world automatically; no per-tool integration work.

OpenSPDD as first concrete validation: REASONS Canvas docs are pure markdown contracts written to a local directory. demarkus-sync makes them a versioned, federated demarkus surface without OpenSPDD knowing. Same pattern works for any tool that emits markdown.

The deeper framing: demarkus shouldn't ask other tools to integrate it. It should make any markdown-emitting tool's output queryable as a versioned protocol surface, transparently. demarkus-sync is the generic shape of that.

Reuses: fetch.Client for remote PUBLISH, store.Write for local-store writes, fsnotify for the watch loop.

Phase 6: Universe Deployment: PLANNED

See plan for full details.

Make it cheap to stand up N worlds and onboard real users without hand-distributing tokens. Phase 5 gave us a single hardened server; Phase 6 makes a universe of them deployable with one Helm chart, one ApplicationSet, and one OIDC-fronted token broker.

Sub-phases:

  • 6.1: Helm chart (deploy/helm/demarkus-server/): StatefulSet + volumeClaimTemplates so each world owns its own PVC, Service, bootstrap Job that runs demarkus-token generate and writes to a Secret, optional read-only mode, SIGHUP reload wiring.
  • 6.2: Universe topology (deploy/k8s/examples/): reference Argo CD ApplicationSet over a worlds: list, plus a Kustomize overlay for clusters without Argo. No new code.
  • 6.3: Token broker (prototype in tools/demarkus-broker/, splits to latebit-io/demarkus-broker after 6.4): OIDC-fronted HTTP service that mints scoped tokens, writes hashes to world Secrets, SIGHUPs on revocation, audit log to stdout.
  • 6.4: User install flow (lives with broker): GET /me/install returns a one-shot shell script that writes ~/.config/demarkus/auth and idempotently patches ~/.claude.json with a demarkus-mcp server entry per world.
  • 6.5: Docs: /deployment.md for chart values + broker setup; plan doc maintained at /plans/universe-deployment.md.

Constraint: zero changes to protocol or core server. Phase 6 is packaging and lifecycle. Any need for a new core primitive must be discussed and justified before landing, same rule that applied to the Claude Code plugin.

Explicitly deferred to Phase 7+: multi-replica worlds with shared storage, cross-cluster universe federation, an operator with a World CRD, a web UI on top of the broker.

Verb Set: Complete

7 verbs: FETCH, LIST, VERSIONS, PUBLISH, APPEND, ARCHIVE, LOOKUP.

LOOKUP (issue #113) is the catalog verb: subject lookup over an in-memory, importance-ranked catalog of declared tags + title. It is a token-efficient supplement to the index, not full-text search (which stays in an opt-in sidecar). See the LOOKUP Catalog Verb section below and /plans/lookup-verb.md.

LOOKUP Catalog Verb: DONE (merged to main 2026-05-30, PR #166; plugin tail closed by #168)

See plan. Tracks issue #113.

Given a subject, return which documents are about it and how important they are; a card-catalog supplement to the index hub so an agent finds the right doc without fetching and grepping the whole index. Catalog scan only: in-memory path → {tags, importance, title}, built on the startup walk and maintained inline on writes, ranked by match-count then importance. No body reads at query time, no relevance modeling. Full-text / semantic search is permanently out of core, in an opt-in sidecar.

Shipped:

  • Protocol verb + spec §6.7.
  • server/internal/catalog package + handler with read-auth result filtering.
  • Client surfaces: fetch.Client.Lookup, demarkus lookup CLI, MCP mark_lookup.
  • Broker parity: mark_lookup is the gateway's 14th tool.
  • Publisher metadata: tags/importance settable at publish; CLI -meta key=value (repeatable), and a metadata object on MCP + broker mark_publish. This is what makes the tag/importance ranking actually usable (before it, LOOKUP was title-only with uniform 0.5 importance).
  • End-to-end smoke test passed (publish with tags → lookup by tag → archive, over QUIC).
  • Plugin surfacing (v0.3.0, #168, 2026-05-31): Claude Code plugin wires mark_lookup, fixes the knowledge-join tool-count text (13→14), and injects SessionStart guidance so sessions recall via lookup.

Remaining tail (small): mark_append metadata; deferred by design (you tag on publish, not append). The plugin knowledge-join/version-pin items are now shipped in v0.3.0.

Open Knowledge Format (OKF) Compatibility: DONE (merged 2026-06-22, PR #195)

demarkus interoperates with Google's Open Knowledge Format v0.1 (a bundle of markdown files with YAML frontmatter). The document content model is OKF-compatible; demarkus layers versioning, hash chain, QUIC transport, capability auth, and LOOKUP around it.

Shipped:

  • Store metadata alignment: recognized OKF fields (type, title, description, resource, tags, timestamp) serialize as bare frontmatter; tags as a YAML flow list; non-spec keys keep the meta. prefix; reserved operational fields (version, previous-hash, archived) enforced by name. Back-compatible read of legacy meta.*. Caps raised to 50 keys / 1024 bytes (frontmatter budget 2048), tags counted at serialized length.
  • demarkus okf codec (client/internal/okf); validate (v0.1 conformance), import (bundle → world, sanitize/cap with warnings, link rewrite, upsert), export (world subtree → conformant bundle, frontmatter reattach, type synthesis, RFC 3339 timestamp normalization). Verified live: import → export → validate round-trip, byte-identical bodies.
  • Opinionated default: server assigns type: Document on PUBLISH and APPEND when none declared (reserved index.md/log.md exempt), so every served document is a typed OKF concept by construction. Non-breaking for existing typeless writers (ADR 0003).
  • Spec §8.1/§9.4/§13/§14; ADR 0002 (metadata alignment), ADR 0003 (type default).
  • Review hardening (PR #195 CodeRabbit): export path-traversal guard, import symlink rejection, LIST decode/cycle hardening, reserved-key read guard, cap re-validation, link query strip.

MCP Agent Ergonomics & Attach Surface: DONE (2026-07-04/05)

Two same-week phases that made MCP agents cheap to serve and demarkus pleasant to drive from Claude Code / Desktop. Plans: /plans/mcp-client-ergonomics.md and /plans/mcp-resources-prompts.md. Both fully shipped AND deployed on both MCP surfaces (local demarkus-mcp and the broker gateway).

Shipped:

  • Size-adaptive mark_fetch: full body under 8KB; above it an outline (heading tree with GitHub-slug #anchors + per-section line counts + opening paragraph + hint). url#anchor slices one section at any size; force overrides. Killed the five-figure-token fetches of append-heavy docs (measured: a 52KB roadmap → 5.9KB outline).
  • Session unchanged-dedup: re-fetch of an unchanged doc returns status: unchanged since vN (identity = version+etag; identity-flip edge cases spelled out). Process-scoped in the local client; MCP-session-keyed at the broker (multi-tenant safe: no session → no dedup; LRU eviction at cap, since mcp-go only unregisters sessions on explicit DELETE).
  • mark_explore: one-call orientation card (outline head, opening paragraph, outbound links, backlinks, siblings; 10/section + "+N more").
  • MCP resources: documents as client-attachable context, zero tool turns: mark:// URI templates, #anchor section attach, well-known + background-listed picker entries (client) / per-world index hubs (broker).
  • MCP prompts: orient / recall / whats-new as server-vended slash commands; the broker variants sweep worlds via mark_worlds.
  • Shared packages client/mdoutline (outline/slice + render helpers) and client/fetchdedup (dedup identity + notice text); one implementation for both surfaces (the changedNote mirror drifted twice in review before hoisting; lesson in /debugging.md).
  • One #section grammar everywhere: mark_fetch, mark_explore, MCP resource URIs, and the demarkus-library librarian's open tool (adopted 2026-07-05) all share the same anchors from the same implementation.
  • Deployed: plugin users via client v0.15.0→v0.17.0 (tools 0.4.1+, pin chain now self-driving after the #228 pin-bump automation fix); knowledge.demarkus.io via broker 0.5.0 (ergonomics) and 0.6.0 (resources+prompts).

Remaining: none. Follow-ups all landed (broker parity, resources/prompts, librarian adoption).

Distribution & Package Management: PLANNED

Homebrew tap for easier installation on macOS and Linux.

Build Targets

Supported platforms: macOS, Linux, Windows (WSL only).

Plugins: IN PROGRESS

Obsidian Plugin: v0.1.0 RELEASED

Standalone repo: latebit-io/obsidian-demarkus. Source in plugins/obsidian/.

Claude Code Plugin: v0.12.5 (current)

Merged into main via PR #96 (commit af8e210) on 2026-04-23 as v0.1.0. Source at plugins/claude-code/ in the monorepo; marketplace manifest at .claude-plugin/marketplace.json.

See plan for the full design. One-click marketplace install that lazy-spawns a local demarkus-server, auto-generates a token, and wires demarkus-mcp into the agent. Zero config on install. Includes /soul, /soul-journal, /soul-init, /soul-join, /soul-default, /knowledge-join slash commands and a soul-memory skill.

Shipped with a project-centric soul schema: /index.md is a project list, each project under /<slug>/ holds plan/tasks.md, architecture.md, patterns.md, roadmap.md, adr/, journal/<YYYY-MM-DD>.md.

Version history:

  • v0.1.0 (2026-04-23, #96): initial marketplace plugin.
  • v0.2.0 (2026-05-23, #152): /knowledge-join command for organizational broker-fronted knowledge systems.
  • v0.3.0 (2026-05-31, #168): wires mark_lookup; injects standing SessionStart guidance (context/session-guidance.md) so sessions self-document to the soul and recall via lookup; fixes knowledge-join tool count (13→14); bumps binary pins to SERVER 0.17.13 / CLIENT 0.12.38 / TOOLS 0.1.28.
  • v0.4.0–v0.9.x: not enumerated here (promote bridge, soul-refresh, binary-pin automation, OKF release pin, soul catalog groundwork). See the journals for detail.
  • v0.10.0 / v0.10.1 (#200, #201): /soul-join: managed remote souls. Joins a direct-QUIC remote soul (token in a 0600 file via a launch wrapper, never inline in MCP config), records it in a catalog (~/.demarkus/souls) with a per-project binding (~/.demarkus/project-souls), and enforces the binding with a destination gate (PreToolUse on mark_publish/mark_append; a write to a non-bound soul is denied/asked/warned). Publish tag-gate extended to joined remote souls. 0.10.1 closed a reuse-server restart bug (never kill a healthy externally-managed server on binary upgrade).
  • v0.11.0 (#202, 2026-06-22): /soul-default: set a project's default write target. Standalone command to re-point an already-joined repo at a different default without re-running /soul-join (until now the binding was only ever written as a --bind side-effect of join). Lists the joined-soul catalog (local managed soul + remotes), picks one, saves the binding the dest-gate enforces. Routing stays model-driven: the catalog is the discovery surface and each soul is its own MCP server, so no router or dispatch tool; reads and one-off writes to another joined soul go direct to its mcp__<slug>__mark_* tools (the gate fires only on publish/append). New lib.sh helpers (local_soul_present, soul_catalog, is_catalog_soul); no core/protocol change.
  • v0.12.x (late June → 2026-07-05): pi-agent port fixes (#217), the ergonomics release-chain repins (tools 0.3.5 → 0.4.1, #221/#227), and the automated pin-bump sweep (#229) after the pin-bump workflow's closed-PR upsert bug was fixed (#228: the pin chain is self-driving again: release → auto bump PR → merge). Current: demarkus-memory 0.12.5, demarkus-knowledge 0.5.5.

No core code changes landed. The plugin is built entirely on existing primitives: DEMARKUS_AUTH env var for token injection, /health endpoint plus ALPN "mark" negotiation for port probing, and shell redirection of demarkus-token generate stdout for writing the token to a file. Any future need for a new core primitive must be discussed and justified before landing.

TUI Polish

External Links: PLANNED (implementation ready to commit)

See plan. Closes issue #75 in full. Opens http/https/gemini/mailto links in the user's default handler via open/xdg-open/rundll32. Scheme allowlist, URL passed as argv only, no shell.

The original plan had a Phase 2 for same-server non-markdown files (images); dropped because the server is markdown-only and will not serve binaries. Inline image rendering, if ever pursued, is a separate client-rendering initiative (terminal graphics protocols like Kitty/Sixel), not a continuation of this work.

Features Not Prioritized: Backlog

  • WebSub-style Subscriptions (removed from spec)
  • Offline Mode (deferred)
  • Full-Text / Semantic Search (out of core: opt-in sidecar reading via LIST/FETCH; in-core LOOKUP covers catalog-level subject lookup, see above)
  • Diff / Changelog Between Versions (noted, not implemented)
  • Blind Append with Content Deduplication (rejected)
  • Non-markdown content types on the server (out of scope by design)

Operational Polish: Backlog

  • Plain-directory root returns silent not-found; the server only serves the versioned-symlink store layout. Pointing -root at a directory of plain .md files (e.g. docs/site) silently returns not-found for every path and builds a 0-entry hash index and catalog, with no startup warning. Bit setup during the 2026-06-02 load-test work. Flagged to revisit: at minimum a startup warning when the root has markdown files but no versions/ layout; ideally a clear hint.

  • tools/demarkus-loadtest: concurrent read-only load generator (FETCH/LIST/ VERSIONS/LOOKUP) for capacity measurement. Reuses fetch.Client; warm (reused-conn) vs -fresh (handshake-per-request) regimes. Report and first measured numbers in tools/demarkus-loadtest/REPORT.md. See journal 2026-06-02. Key finding: server self-throttles per IP (default 50 req/s, burst 100, DEMARKUS_RATE_LIMIT=0 disables); warm ceiling ~7.7k req/s, cold ~1.4k req/s on one core.

Docs Hygiene

  • SPEC.md markdownlint cleanup: DONE (PR #196, merged 2026-06-22). Added repo-wide .markdownlint.json disabling MD013 (line-length) and MD060 (table-column-style): both pervasive, intentional spec style. Fixed the substantive rules in docs/SPEC.md: fence languages (MD040, 29 fences → text), blank lines around fences/lists (MD031/MD032), bare URLs (MD034), and the emphasis footer (MD036). markdownlint-cli2 docs/SPEC.md → 0 errors; formatting only, no content changes.

Knowledge Layer & Storage Backends: NEXT (added 2026-07-13)

Wire backlinks/graph tools to the published /graph.md

The federation agent already publishes the aggregated link graph to each hub at /graph.md (hourly, cross-server edges, same format as mark_graph_export), but mark_backlinks and mark_graph answer only from the client's local ~/.mark/graph.json crawl cache. A fresh client answers from an empty graph while an authoritative aggregate sits on the hub unread.

  • Seed/refresh the MCP graphstore from the hub's /graph.md (etag-cached fetch, merge into local store) before falling back to local crawl.
  • Fixes cold start and cross-agent consistency with a client-only change; days of work, no server change.
  • Later upgrade, not the fix: publish-time edge extraction in the store (instant, transactional backlinks). Subject to the backend-parity principle below.

Principle: backend parity is system-level, not implementation-level

With two store backends (file, postgres), behavioral parity is a property of the system, not a detail of either implementation. The storetest conformance suite is the contract; both backends run it unchanged and divergence is a protocol bug.

Consequences for future work:

  • Any new store-observable behavior lands in the conformance suite first, then in every backend, or it does not ship.
  • Knowledge-layer features should prefer layers above the store (published documents like /graph.md, broker aggregation, client merge) which are backend-independent by construction.
  • A feature that only one backend can provide efficiently (e.g. a transactional edges table in postgres) must still be defined system-level: either every backend implements the same observable surface, or the capability is exposed as an optional, discoverable feature rather than silently backend-dependent.

Remaining gaps from the 2026-07-13 knowledge-layer analysis

The /graph.md wiring item above came out of a gap analysis (knowledge-system-deploy session, 2026-07-13): demarkus is strong at the document layer (versioning, integrity, provenance, gates) and weak at the knowledge layer. Only gap 1 got recorded at the time; the rest are recorded here so they are not lost. Framing correction from that session: the plugins are the knowledge layer (recall-first, tag-at-publish, promote, doctors), so the governing principle is that the agent owns judgment while the server owns the accumulation of that judgment (edges, indexes, derived rank live in shared storage so the layer compounds across agents instead of being recomputed per client).

Edge semantics: give edges more information

Every link in the graph is an untyped "mentions". Knowledge graphs live on predicates: supersedes, implements, depends-on, derived-from. OKF types the nodes; nothing types the edges (ADR 0002 superseding its v1 draft was expressible only as prose). Edges also carry no provenance: graph.Edge/StoredEdge are bare from/to pairs and /graph.md exports a two-column table, so backlinks cannot say why or where a doc links (no link text, no source anchor, no occurrence count).

Shape: a rel: metadata axis (e.g. rel:supersedes=/adr/0002-....md) surfaced in mark_graph/mark_backlinks, plus richer edge records in the export (label, source anchor, count). This is a plugin and export-format convention, zero server change, so it can land before publish-time edge extraction; when the store later learns edge extraction it must capture the same fields (backend-parity principle above).

Semantic recall beyond tags

Lookup recall depends entirely on declared tags and titles; recall bounded by metadata hygiene loses to grep, and the agent compensates by exploring and fetching more every session (judgment tokens spent on enumeration). Path: Postgres tsvector full-text over bodies (cheap, phase-2 adjacent), then pgvector embeddings, hybrid-ranked with declared importance. Indexes amortize the agent's search loop into infrastructure; they do not replace its judgment.

Universe-wide LOOKUP

Lookup is scoped to one world; the broker routes but does not aggregate, so "search the whole system" is client-side fan-out. A broker-level aggregated LOOKUP (and eventually graph) makes a multi-world system feel like one system. Broker feature, no server change.

Derived ranking signals

importance is entirely self-declared, so a well-tagged orphan outranks the hub everything cites. Blend link-graph signals (in-degree, centrality) with declared importance. Link structure is not usage telemetry, so this respects the no-tracking principle.

Staleness at lookup time

modified exists but nothing decays or flags aging content at query time; the doctors catch it offline only.

Edge semantics: IMPLEMENTED (branch edge-semantics, 2026-07-13, pending PR)

The "Edge semantics: give edges more information" gap above is implemented; see /plans/edge-semantics.md and repo ADR 0004. Edges now carry {From, To, Rel, Label, Anchor, Count} with identity {From, To, Rel}; the rel-<predicate> publisher-metadata convention (hyphen form, CSV values) is ingested as typed edges by both crawlers; the /graph.md export is six columns with legacy 2-column parse kept; mark_graph/mark_backlinks/mark_explore render provenance via the shared graph.EdgeAnnotation. Zero server changes. The fetch-callback debt item went with it. Update this entry to DONE when the PR merges.

trail
  1. soul.demarkus.io v48