soul.demarkus.io/roadmap.md/v44 draft reader meta

Roadmap

Where demarkus has been and where it's going.

Scope: Markdown-Only Server

The demarkus server serves markdown only. Non-markdown content (images, PDFs, arbitrary binaries) is out of scope and will not be added. This keeps the protocol, store, auth, and size-limit logic simple and predictable. Any future inline-image story is a client/rendering concern; not something the server will serve bytes for.

Phase 1: MVP (Read-Only): COMPLETE

Everything shipped:

  • QUIC server serving markdown files
  • FETCH, LIST, VERSIONS verbs
  • TUI client with Bubble Tea + Glamour
  • Link following, navigation history
  • Document graph visualization
  • CLI client with all verbs
  • MCP integration for LLM agent access
  • Docker multi-arch images
  • GoReleaser CI/CD with per-module versioning
  • Conditional fetch (if-none-match, if-modified-since)
  • SIGHUP certificate reload

Phase 2: Publish Operations: COMPLETE

Done:

  • PUBLISH verb with version creation
  • ARCHIVE verb
  • APPEND verb: sends only new content, server handles concatenation
  • Capability-based auth (token generation, SHA-256 hashes, path/op scoping)
  • Versioned store with symlinks and hash chain
  • Document editing via $EDITOR
  • Client-side token management
  • Conflict resolution (optimistic concurrency with expected-version)
  • No-op on duplicate content
  • Structured logging with slog (replaced console logging)
  • Protocol-level size limits (1 MiB body, 64KB frontmatter)
  • Audit logging with token_label on all write operations
  • Usability audit: documentation accuracy, CLI help, install script cleanup, CI lint

Phase 2 is fully complete. No remaining items.

Phase 3: Agent-Native & Advanced Features: COMPLETE

Done:

  • Agent manifest discovery (/.well-known/agent-manifest.md)
  • Bookmarks/favorites
  • MCP mark_append auto-resolve expected_version
  • Content-addressed fetch
  • Federation via MCP tools

Phase 4: The Information Graph: COMPLETE

Done:

  • The Demarkus Hub pattern
  • Persistent graph store
  • Backlinks
  • Graph as content (export/import)
  • Graph-aware navigation (TUI)
  • Agent discovery

Phase 5: The Demarkus Agent & Private Networks: IN PROGRESS

Read Auth for Private Networks: DONE

Per-path read token enforcement. Server-side and client-side shipped.

Security Hardening: DONE

See plan for full details.

  • Systemd hardening: install script generates units with ProtectSystem=strict, ReadWritePaths, NoNewPrivileges, etc. Conditional ProtectHome. Update path detects insecure config and prompts to harden with rollback.
  • -read-only mode: DEMARKUS_READ_ONLY env var / -read-only flag. Handler rejects PUBLISH/APPEND/ARCHIVE with not-permitted. Zero write access needed.
  • demarkus-publish: CLI tool that writes directly to the versioned store on disk. Enables local publishing when server runs read-only. Shares store.Write() with the server.
  • Read-only chroot install: install-readonly.sh. Chrooted server with ReadOnlyPaths=/, BindReadOnlyPaths=/dev/urandom. Maximum lockdown: Gemini-level security with full versioning.
  • Security documentation: attack surface analysis, threat model, comparison table, hardening guide.

Core Loop: Crawl & Index

  1. Seed: start from configured servers (hub, known peers)
  2. Crawl: follow mark:// links, discover new servers and documents
  3. Hash: collect content hashes from every document
  4. Index: publish updated hash indexes to configured hubs
  5. Repeat: on a configurable schedule, with conditional fetch (if-none-match) to be polite

Server-to-Server Sync

Rsync for the Mark Protocol. Replicate content between servers using content hashes as the diff mechanism.

How it works:

  1. LIST both source and destination servers
  2. Compare content hashes: skip documents that match
  3. FETCH changed/new documents from source
  4. PUBLISH to destination with the fetched content
  5. Optionally handle deletions (ARCHIVE on destination for docs removed from source)

Sync modes:

  • Mirror: destination becomes an exact copy of source (one-way)
  • Selective: sync specific paths or glob patterns (e.g., /docs/* only)
  • Multi-source: aggregate content from multiple servers into one destination

Key Design Points

  • Go binary, not an LLM agent: mechanical work, not reasoning
  • Polite crawling: conditional fetch, configurable rate limits
  • Hub-aware: reads and updates hub index documents
  • Daemon or cron: continuous or triggered
  • Auth-aware: passes read and write tokens for private servers

Implementation Sketch

  • New module: tools/demarkus-agent (the tools/ directory is the home for utility binaries; demarkus-token, demarkus-publish, and now sync/agent tooling)
  • Reuses: fetch.Client, graphstore, client/internal/index, graph.Crawl
  • CLI: demarkus-agent crawl, demarkus-agent sync source dest, demarkus-agent daemon

External Tool Sync: PLANNED

The shape that lets any markdown-emitting tool (OpenSPDD, ADR generators, doc generators, custom workflows) land its output on demarkus without integrating the protocol itself. The integration burden lives on demarkus's side; upstream tools change nothing.

demarkus-sync: single binary in tools/. Watches a local directory and PUBLISHes (or writes via demarkus-publish against a local store) on every changed .md. Versioning happens automatically through the existing store. Cross-team sharing emerges naturally from publishing to a team server.

Core flags:

  • --watch <dir>: directory to watch
  • --target mark://host/path/: remote demarkus server (uses fetch.Client PUBLISH)
  • --store <local-path>: alternative: write directly to a local versioned store via store.Write
  • --include / --exclude glob filters (e.g. **/*.canvas.md)
  • --token / DEMARKUS_AUTH for write auth

Project-flow integration is the killer use: drop into a Makefile target, pre-commit hook, CI step, or long-running dev daemon. Design contracts, ADRs, RFCs, runbooks, agent canvases all flow into the team world automatically; no per-tool integration work.

OpenSPDD as first concrete validation: REASONS Canvas docs are pure markdown contracts written to a local directory. demarkus-sync makes them a versioned, federated demarkus surface without OpenSPDD knowing. Same pattern works for any tool that emits markdown.

The deeper framing: demarkus shouldn't ask other tools to integrate it. It should make any markdown-emitting tool's output queryable as a versioned protocol surface, transparently. demarkus-sync is the generic shape of that.

Reuses: fetch.Client for remote PUBLISH, store.Write for local-store writes, fsnotify for the watch loop.

Phase 6: Universe Deployment: PLANNED

See plan for full details.

Make it cheap to stand up N worlds and onboard real users without hand-distributing tokens. Phase 5 gave us a single hardened server; Phase 6 makes a universe of them deployable with one Helm chart, one ApplicationSet, and one OIDC-fronted token broker.

Sub-phases:

  • 6.1: Helm chart (deploy/helm/demarkus-server/): StatefulSet + volumeClaimTemplates so each world owns its own PVC, Service, bootstrap Job that runs demarkus-token generate and writes to a Secret, optional read-only mode, SIGHUP reload wiring.
  • 6.2: Universe topology (deploy/k8s/examples/): reference Argo CD ApplicationSet over a worlds: list, plus a Kustomize overlay for clusters without Argo. No new code.
  • 6.3: Token broker (prototype in tools/demarkus-broker/, splits to latebit-io/demarkus-broker after 6.4): OIDC-fronted HTTP service that mints scoped tokens, writes hashes to world Secrets, SIGHUPs on revocation, audit log to stdout.
  • 6.4: User install flow (lives with broker): GET /me/install returns a one-shot shell script that writes ~/.config/demarkus/auth and idempotently patches ~/.claude.json with a demarkus-mcp server entry per world.
  • 6.5: Docs: /deployment.md for chart values + broker setup; plan doc maintained at /plans/universe-deployment.md.

Constraint: zero changes to protocol or core server. Phase 6 is packaging and lifecycle. Any need for a new core primitive must be discussed and justified before landing, same rule that applied to the Claude Code plugin.

Explicitly deferred to Phase 7+: multi-replica worlds with shared storage, cross-cluster universe federation, an operator with a World CRD, a web UI on top of the broker.

Verb Set: Complete

7 verbs: FETCH, LIST, VERSIONS, PUBLISH, APPEND, ARCHIVE, LOOKUP.

LOOKUP (issue #113) is the catalog verb: subject lookup over an in-memory, importance-ranked catalog of declared tags + title. It is a token-efficient supplement to the index, not full-text search (which stays in an opt-in sidecar). See the LOOKUP Catalog Verb section below and /plans/lookup-verb.md.

LOOKUP Catalog Verb: DONE (merged to main 2026-05-30, PR #166; plugin tail closed by #168)

See plan. Tracks issue #113.

Given a subject, return which documents are about it and how important they are; a card-catalog supplement to the index hub so an agent finds the right doc without fetching and grepping the whole index. Catalog scan only: in-memory path → {tags, importance, title}, built on the startup walk and maintained inline on writes, ranked by match-count then importance. No body reads at query time, no relevance modeling. Full-text / semantic search is permanently out of core, in an opt-in sidecar.

Shipped:

  • Protocol verb + spec §6.7.
  • server/internal/catalog package + handler with read-auth result filtering.
  • Client surfaces: fetch.Client.Lookup, demarkus lookup CLI, MCP mark_lookup.
  • Broker parity: mark_lookup is the gateway's 14th tool.
  • Publisher metadata: tags/importance settable at publish; CLI -meta key=value (repeatable), and a metadata object on MCP + broker mark_publish. This is what makes the tag/importance ranking actually usable (before it, LOOKUP was title-only with uniform 0.5 importance).
  • End-to-end smoke test passed (publish with tags → lookup by tag → archive, over QUIC).
  • Plugin surfacing (v0.3.0, #168, 2026-05-31): Claude Code plugin wires mark_lookup, fixes the knowledge-join tool-count text (13→14), and injects SessionStart guidance so sessions recall via lookup.

Remaining tail (small): mark_append metadata; deferred by design (you tag on publish, not append). The plugin knowledge-join/version-pin items are now shipped in v0.3.0.

Open Knowledge Format (OKF) Compatibility: DONE (merged 2026-06-22, PR #195)

demarkus interoperates with Google's Open Knowledge Format v0.1 (a bundle of markdown files with YAML frontmatter). The document content model is OKF-compatible; demarkus layers versioning, hash chain, QUIC transport, capability auth, and LOOKUP around it.

Shipped:

  • Store metadata alignment: recognized OKF fields (type, title, description, resource, tags, timestamp) serialize as bare frontmatter; tags as a YAML flow list; non-spec keys keep the meta. prefix; reserved operational fields (version, previous-hash, archived) enforced by name. Back-compatible read of legacy meta.*. Caps raised to 50 keys / 1024 bytes (frontmatter budget 2048), tags counted at serialized length.
  • demarkus okf codec (client/internal/okf); validate (v0.1 conformance), import (bundle → world, sanitize/cap with warnings, link rewrite, upsert), export (world subtree → conformant bundle, frontmatter reattach, type synthesis, RFC 3339 timestamp normalization). Verified live: import → export → validate round-trip, byte-identical bodies.
  • Opinionated default: server assigns type: Document on PUBLISH and APPEND when none declared (reserved index.md/log.md exempt), so every served document is a typed OKF concept by construction. Non-breaking for existing typeless writers (ADR 0003).
  • Spec §8.1/§9.4/§13/§14; ADR 0002 (metadata alignment), ADR 0003 (type default).
  • Review hardening (PR #195 CodeRabbit): export path-traversal guard, import symlink rejection, LIST decode/cycle hardening, reserved-key read guard, cap re-validation, link query strip.

MCP Agent Ergonomics & Attach Surface: DONE (2026-07-04/05)

Two same-week phases that made MCP agents cheap to serve and demarkus pleasant to drive from Claude Code / Desktop. Plans: /plans/mcp-client-ergonomics.md and /plans/mcp-resources-prompts.md. Both fully shipped AND deployed on both MCP surfaces (local demarkus-mcp and the broker gateway).

Shipped:

  • Size-adaptive mark_fetch: full body under 8KB; above it an outline (heading tree with GitHub-slug #anchors + per-section line counts + opening paragraph + hint). url#anchor slices one section at any size; force overrides. Killed the five-figure-token fetches of append-heavy docs (measured: a 52KB roadmap → 5.9KB outline).
  • Session unchanged-dedup: re-fetch of an unchanged doc returns status: unchanged since vN (identity = version+etag; identity-flip edge cases spelled out). Process-scoped in the local client; MCP-session-keyed at the broker (multi-tenant safe: no session → no dedup; LRU eviction at cap, since mcp-go only unregisters sessions on explicit DELETE).
  • mark_explore: one-call orientation card (outline head, opening paragraph, outbound links, backlinks, siblings; 10/section + "+N more").
  • MCP resources: documents as client-attachable context, zero tool turns: mark:// URI templates, #anchor section attach, well-known + background-listed picker entries (client) / per-world index hubs (broker).
  • MCP prompts: orient / recall / whats-new as server-vended slash commands; the broker variants sweep worlds via mark_worlds.
  • Shared packages client/mdoutline (outline/slice + render helpers) and client/fetchdedup (dedup identity + notice text); one implementation for both surfaces (the changedNote mirror drifted twice in review before hoisting; lesson in /debugging.md).
  • One #section grammar everywhere: mark_fetch, mark_explore, MCP resource URIs, and the demarkus-library librarian's open tool (adopted 2026-07-05) all share the same anchors from the same implementation.
  • Deployed: plugin users via client v0.15.0→v0.17.0 (tools 0.4.1+, pin chain now self-driving after the #228 pin-bump automation fix); knowledge.demarkus.io via broker 0.5.0 (ergonomics) and 0.6.0 (resources+prompts).

Remaining: none. Follow-ups all landed (broker parity, resources/prompts, librarian adoption).

Distribution & Package Management: PLANNED

Homebrew tap for easier installation on macOS and Linux.

Build Targets

Supported platforms: macOS, Linux, Windows (WSL only).

Plugins: IN PROGRESS

Obsidian Plugin: v0.1.0 RELEASED

Standalone repo: latebit-io/obsidian-demarkus. Source in plugins/obsidian/.

Claude Code Plugin: v0.12.5 (current)

Merged into main via PR #96 (commit af8e210) on 2026-04-23 as v0.1.0. Source at plugins/claude-code/ in the monorepo; marketplace manifest at .claude-plugin/marketplace.json.

See plan for the full design. One-click marketplace install that lazy-spawns a local demarkus-server, auto-generates a token, and wires demarkus-mcp into the agent. Zero config on install. Includes /soul, /soul-journal, /soul-init, /soul-join, /soul-default, /knowledge-join slash commands and a soul-memory skill.

Shipped with a project-centric soul schema: /index.md is a project list, each project under /<slug>/ holds plan/tasks.md, architecture.md, patterns.md, roadmap.md, adr/, journal/<YYYY-MM-DD>.md.

Version history:

  • v0.1.0 (2026-04-23, #96): initial marketplace plugin.
  • v0.2.0 (2026-05-23, #152): /knowledge-join command for organizational broker-fronted knowledge systems.
  • v0.3.0 (2026-05-31, #168): wires mark_lookup; injects standing SessionStart guidance (context/session-guidance.md) so sessions self-document to the soul and recall via lookup; fixes knowledge-join tool count (13→14); bumps binary pins to SERVER 0.17.13 / CLIENT 0.12.38 / TOOLS 0.1.28.
  • v0.4.0–v0.9.x: not enumerated here (promote bridge, soul-refresh, binary-pin automation, OKF release pin, soul catalog groundwork). See the journals for detail.
  • v0.10.0 / v0.10.1 (#200, #201): /soul-join: managed remote souls. Joins a direct-QUIC remote soul (token in a 0600 file via a launch wrapper, never inline in MCP config), records it in a catalog (~/.demarkus/souls) with a per-project binding (~/.demarkus/project-souls), and enforces the binding with a destination gate (PreToolUse on mark_publish/mark_append; a write to a non-bound soul is denied/asked/warned). Publish tag-gate extended to joined remote souls. 0.10.1 closed a reuse-server restart bug (never kill a healthy externally-managed server on binary upgrade).
  • v0.11.0 (#202, 2026-06-22): /soul-default: set a project's default write target. Standalone command to re-point an already-joined repo at a different default without re-running /soul-join (until now the binding was only ever written as a --bind side-effect of join). Lists the joined-soul catalog (local managed soul + remotes), picks one, saves the binding the dest-gate enforces. Routing stays model-driven: the catalog is the discovery surface and each soul is its own MCP server, so no router or dispatch tool; reads and one-off writes to another joined soul go direct to its mcp__<slug>__mark_* tools (the gate fires only on publish/append). New lib.sh helpers (local_soul_present, soul_catalog, is_catalog_soul); no core/protocol change.
  • v0.12.x (late June → 2026-07-05): pi-agent port fixes (#217), the ergonomics release-chain repins (tools 0.3.5 → 0.4.1, #221/#227), and the automated pin-bump sweep (#229) after the pin-bump workflow's closed-PR upsert bug was fixed (#228: the pin chain is self-driving again: release → auto bump PR → merge). Current: demarkus-memory 0.12.5, demarkus-knowledge 0.5.5.

No core code changes landed. The plugin is built entirely on existing primitives: DEMARKUS_AUTH env var for token injection, /health endpoint plus ALPN "mark" negotiation for port probing, and shell redirection of demarkus-token generate stdout for writing the token to a file. Any future need for a new core primitive must be discussed and justified before landing.

TUI Polish

External Links: PLANNED (implementation ready to commit)

See plan. Closes issue #75 in full. Opens http/https/gemini/mailto links in the user's default handler via open/xdg-open/rundll32. Scheme allowlist, URL passed as argv only, no shell.

The original plan had a Phase 2 for same-server non-markdown files (images); dropped because the server is markdown-only and will not serve binaries. Inline image rendering, if ever pursued, is a separate client-rendering initiative (terminal graphics protocols like Kitty/Sixel), not a continuation of this work.

Features Not Prioritized: Backlog

  • WebSub-style Subscriptions (removed from spec)
  • Offline Mode (deferred)
  • Full-Text / Semantic Search (out of core: opt-in sidecar reading via LIST/FETCH; in-core LOOKUP covers catalog-level subject lookup, see above)
  • Diff / Changelog Between Versions (noted, not implemented)
  • Blind Append with Content Deduplication (rejected)
  • Non-markdown content types on the server (out of scope by design)

Operational Polish: Backlog

  • Plain-directory root returns silent not-found; the server only serves the versioned-symlink store layout. Pointing -root at a directory of plain .md files (e.g. docs/site) silently returns not-found for every path and builds a 0-entry hash index and catalog, with no startup warning. Bit setup during the 2026-06-02 load-test work. Flagged to revisit: at minimum a startup warning when the root has markdown files but no versions/ layout; ideally a clear hint.

  • tools/demarkus-loadtest: concurrent read-only load generator (FETCH/LIST/ VERSIONS/LOOKUP) for capacity measurement. Reuses fetch.Client; warm (reused-conn) vs -fresh (handshake-per-request) regimes. Report and first measured numbers in tools/demarkus-loadtest/REPORT.md. See journal 2026-06-02. Key finding: server self-throttles per IP (default 50 req/s, burst 100, DEMARKUS_RATE_LIMIT=0 disables); warm ceiling ~7.7k req/s, cold ~1.4k req/s on one core.

Docs Hygiene

  • SPEC.md markdownlint cleanup: DONE (PR #196, merged 2026-06-22). Added repo-wide .markdownlint.json disabling MD013 (line-length) and MD060 (table-column-style): both pervasive, intentional spec style. Fixed the substantive rules in docs/SPEC.md: fence languages (MD040, 29 fences → text), blank lines around fences/lists (MD031/MD032), bare URLs (MD034), and the emphasis footer (MD036). markdownlint-cli2 docs/SPEC.md → 0 errors; formatting only, no content changes.
trail
  1. soul.demarkus.io v44