Roadmap

Where demarkus has been and where it's going.

Phase 1: MVP (Read-Only) — COMPLETE

Everything shipped:

• QUIC server serving markdown files
• FETCH, LIST, VERSIONS verbs
• TUI client with Bubble Tea + Glamour
• Link following, navigation history
• Document graph visualization
• CLI client with all verbs
• MCP integration for LLM agent access
• Docker multi-arch images
• GoReleaser CI/CD with per-module versioning
• Conditional fetch (if-none-match, if-modified-since)
• SIGHUP certificate reload

Phase 2: Publish Operations — COMPLETE

Done:

• PUBLISH verb with version creation
• ARCHIVE verb
• APPEND verb — sends only new content, server handles concatenation
• Capability-based auth (token generation, SHA-256 hashes, path/op scoping)
• Versioned store with symlinks and hash chain
• Document editing via $EDITOR
• Client-side token management
• Conflict resolution (optimistic concurrency with expected-version)
• No-op on duplicate content
• Structured logging with slog (replaced console logging)
• Protocol-level size limits (1 MiB body, 64KB frontmatter)
• Audit logging with token_label on all write operations
• Usability audit: documentation accuracy, CLI help, install script cleanup, CI lint

Phase 2 is fully complete. No remaining items.

Phase 3: Agent-Native & Advanced Features — COMPLETE

Done:

• Agent manifest discovery (/.well-known/agent-manifest.md)
  • Convention spec: markdown document at well-known path, no server changes needed
  • WellKnownManifestPath constant in protocol module
  • mark_discover MCP tool — fetches manifest from connected server
  • demarkus info CLI subcommand — fetches and displays manifest
  • Spec doc at docs/site/reference/agent-manifest.md
  • First live manifest published on demarkus-soul
  • Directory listing for servers (auto-generated when no index.md)
• Bookmarks/favorites
  • Markdown-backed store at ~/.mark/bookmarks.md
  • TUI: b toggle, B view, Esc exit, star indicator in status bar
  • CLI: bookmark add/remove/list subcommands
  • MCP: not exposed (local-only feature)
• MCP mark_append auto-resolve expected_version
  • expected_version is now optional in mark_append — when omitted or 0, the tool calls VERSIONS to get the current version automatically
  • Optimistic concurrency still enforced — the server always receives an expected_version
  • Introduced markClient interface for handler testability
  • Tests cover auto-resolve (happy path), not-found, explicit version, negative version, no token
  • Does NOT apply to mark_publish (publish requires the agent to have read the document first)
• Content-addressed fetch
  • FETCH responses include content-hash: sha256-<64hex> — SHA-256 of the stripped body
  • FETCH /sha256-<64hex> retrieves documents by content hash from any server that has them
  • In-memory hash index (map[string]string) built on startup, updated on writes/archives
  • Current versions only, archived docs excluded
  • No new verbs — FETCH handles it via path pattern detection
  • Foundation for distributed mirroring and caching
• Federation via MCP tools
  • mark_index — crawls a source server, collects content hashes, publishes index to a hub
  • mark_resolve — resolves content by SHA-256 hash using a hub index document
  • client/internal/index package — Parse, Build, Merge for markdown hash index documents
  • Manifest check enforced by tool, force override, dry_run mode, 1000 doc cap

Phase 4: The Information Graph — COMPLETE

Discovery through linking, not searching. Information is findable because it's connected — servers link to each other through documents, clients and agents traverse the graph.

Done

• The Demarkus Hub pattern — live at mark://hub.demarkus.io. A server whose sole purpose is linking to content on other demarkus servers. No original content, just curated mark:// links organized by topic. CI auto-publishes from Git repo. See github.com/latebit-io/demarkus-hub.
• Persistent graph store — client/internal/graphstore package. Graph stored as JSON at ~/.mark/graph.json with nodes (documents), edges (links), etags, and crawl timestamps. Atomic writes, schema versioning, incremental merge.
• Shared CrawlAndPersist — unified crawl + merge + save method on *Store, nil-safe. All three clients (CLI, TUI, MCP) share the same code path. EtagFetcher adapter collects etags during crawl.
• Backlinks — mark_backlinks MCP tool and Store.BacklinksEnriched() method. Reverse edge lookup from the persistent graph — "what links here?" Returns sorted list with document titles and status.
• Graph seeding from store — TUI graph view loads instantly from stored graph while crawl runs in background.
• Graph as content — export the persistent graph as a publishable markdown document. Store.Export() renders nodes as markdown table rows with mark:// links, edges as a separate table. ParseExport() parses it back. CLI: demarkus graph export [-o file.md]. MCP: mark_graph_export tool. Crawling the exported doc naturally discovers the topology — no special import needed. Cell escaping handles pipes and backslashes in titles.
• Graph-aware navigation — TUI graph view with three sub-views: Links (BFS tree from current doc with [N←] density indicators), Backlinks (reverse links to current doc), Topology (all explored nodes sorted by importance). Sub-views toggled with d/r/t keys. Shared graphstore.BacklinksEnriched() eliminates duplication between TUI and MCP. graph.InDegrees() for O(E) backlink counting. Rune-safe truncation for multi-byte UTF-8.
• Agent discovery — agents crawl the graph to build knowledge maps using mark_graph (crawl + persist), mark_graph_export (publish as markdown), and mark_resolve (hash-based lookup via hub index). Graph documents are themselves crawlable, so sharing and merging happens naturally through the protocol. No special agent-specific code needed — the existing primitives compose.

No new verbs needed. FETCH reads content, LIST enumerates, links connect. The protocol already has the primitives.

See DESIGN.md Phase 4 section for full sketch.

Phase 5: The Demarkus Agent & Private Networks — PLANNED

An autonomous crawler daemon that keeps the demarkus network healthy, a sync engine for server-to-server replication, and read authentication for private networks.

Vision

The network should be self-maintaining. When a new server appears and links to the hub, the agent finds it. When content changes, hash indexes update. When the graph grows, the topology stays fresh. No human intervention needed.

Content should be resilient. If a server goes down, its documents survive on mirrors. Syncing between servers should be as easy as rsync — point at a source, point at a destination, let content hashes handle the diff.

Servers should support both public and private use cases. The same protocol, the same tools, with an opt-in gate for read access when privacy is needed.

Read Auth for Private Networks

The auth system already supports "read" as a token operation — tokens can be scoped to operations = ["read"] with path patterns. But the handler only enforces auth on writes (PUBLISH, APPEND, ARCHIVE). FETCH, LIST, and VERSIONS are currently open to anyone.

What needs to happen:

• Server config option to require read auth (e.g., require-read-auth = true or per-path rules)
• Handler checks Authorize(token, path, "read") on FETCH, LIST, and VERSIONS when enabled
• Default remains open — backwards compatible, public servers stay public
• Read tokens work with existing demarkus-token tool (-ops "read" already works)
• Client, TUI, and MCP tools pass auth token on read operations when configured

Design considerations:

• Per-path granularity: some paths public, some private (e.g., /public/** open, /internal/** gated)
• Agent manifest (/.well-known/agent-manifest.md) should remain unauthenticated so agents can discover capabilities before authenticating
• The agent/sync features need to pass read tokens when crawling private servers
• Content-addressed fetch by hash should respect the same read auth — knowing a hash shouldn't bypass access control

Why this doesn't conflict with core values:

• Privacy is principle #1 in the design doc
• Capability-based auth already exists — this extends it to reads, not a new mechanism
• "Anyone can run a server" includes running a private server
• The protocol stays the same — auth is a server-side policy, not a protocol change
• Public and private servers coexist on the same network

Use cases:

• Team knowledge bases and internal documentation
• Private agent memory (soul servers that shouldn't be world-readable)
• Staging/preview servers before content goes public
• Personal note servers with selective sharing

Core Loop: Crawl & Index

1. Seed — start from configured servers (hub, known peers)
2. Crawl — follow mark:// links, discover new servers and documents
3. Hash — collect content hashes from every document
4. Index — publish updated hash indexes to configured hubs
5. Repeat — on a configurable schedule, with conditional fetch (if-none-match) to be polite

Server-to-Server Sync

Rsync for the Mark Protocol. Replicate content between servers using content hashes as the diff mechanism.

How it works:

1. LIST both source and destination servers
2. Compare content hashes — skip documents that match
3. FETCH changed/new documents from source
4. PUBLISH to destination with the fetched content
5. Optionally handle deletions (ARCHIVE on destination for docs removed from source)

Sync modes:

• Mirror — destination becomes an exact copy of source (one-way)
• Selective — sync specific paths or glob patterns (e.g., /docs/* only)
• Multi-source — aggregate content from multiple servers into one destination

What makes this work:

• Content hashes are already in every FETCH response — the diff is free
• LIST gives the full document tree — no need to crawl links
• PUBLISH handles versioning — optimistic concurrency prevents clobbering
• No new protocol features needed — just FETCH + LIST + PUBLISH in a loop

Use cases:

• Backup — mirror a server to a second host for redundancy
• Edge caching — replicate a hub's content to a server closer to users
• Content aggregation — pull from multiple sources into a single server
• Migration — move content between servers without downtime

What It Uses

Only the existing 6 verbs — no protocol changes needed:

• FETCH — read documents, follow links, conditional fetch for polling
• LIST — enumerate server contents
• VERSIONS — check for changes cheaply
• PUBLISH — update hub index documents and sync content to destinations

Key Design Points

• Go binary, not an LLM agent — this is mechanical work, not reasoning. No API keys, no token costs, no inference latency.
• Polite crawling — conditional fetch, configurable rate limits, respect for server load
• Configurable seeds — start from any set of known servers
• Hub-aware — knows how to read and update hub index documents using the existing client/internal/index package
• Daemon or cron — can run continuously with a sleep interval, or be triggered by cron/systemd timer
• Graph integration — uses the existing graphstore package to persist what it discovers
• Docker-friendly — should run as a lightweight container alongside a hub server
• Auth-aware — can pass read and write tokens when interacting with private servers

Implementation Sketch

• New module: cmd/demarkus-agent (or standalone repo)
• Reuses: fetch.Client, graphstore, client/internal/index, graph.Crawl
• Config: YAML or flags for seed servers, hub target, crawl interval, rate limits, auth tokens
• Sync config: source/destination pairs, sync mode, path filters
• Logging: structured slog, same patterns as server
• CLI: demarkus-agent crawl, demarkus-agent sync source dest, demarkus-agent daemon

What This Enables

• Self-healing network — hub indexes stay current without manual mark_index calls
• Server discovery — new servers that link into the network get found automatically
• Content availability — hash indexes grow as more content is crawled, making mark_resolve more reliable
• Network health monitoring — the agent's graph is a live map of what's up and what's down
• Resilient content — mirroring means no single point of failure for any document
• Easy backup — demarkus-agent sync mark://source mark://backup and you're done
• Private networks — teams and individuals can run authenticated servers that participate in the broader network selectively

Verb Set — Complete

The protocol verb set is finalized at 6 verbs: FETCH, LIST, VERSIONS, PUBLISH, APPEND, ARCHIVE.

Removed from spec:

• SEARCH — Full-text search is better handled as an external tool built on top of the existing primitives (LIST + FETCH) rather than as a protocol-level concern.
• SUBSCRIBE — WebSub-style subscriptions for change notifications were considered but deferred. Polling with conditional FETCH (if-none-match) is simpler and works for most use cases. Real-time push adds complexity and requires maintaining subscriber state. Better addressed by higher-level tools if needed.

Distribution & Package Management — PLANNED

Homebrew tap for easier installation, especially on macOS and Linux developers.

What it takes:

1. Create public latebit/homebrew-demarkus tap repo
2. Add formula for demarkus-client (CLI + TUI) — pull pre-built binaries from GitHub releases
3. Formula needs SHA256 checksums for each architecture
4. Update README with Homebrew install instructions
5. Test installation on macOS and Linux

Notes:

• Server is better deployed via Docker, keep Homebrew focused on the client
• Start with tap (owned by us), not Homebrew Core (requires upstream submission)
• User installs: brew tap latebit/homebrew-demarkus && brew install demarkus-client
• Already have GoReleaser CI/CD and multi-arch builds — just need packaging

Build Targets

Supported platforms: macOS, Linux, Windows (WSL only — runs Linux binaries). Native Windows builds removed from GoReleaser configs.

Plugins — IN PROGRESS

Obsidian Plugin — v0.1.0 RELEASED

Standalone repo: latebit-io/obsidian-demarkus on GitHub. Source lives in plugins/obsidian/ in the monorepo, copied to standalone repo for releases.

Done:

• Fetch, publish, and list documents from Obsidian
• Shells out to demarkus CLI binary
• Token passed via DEMARKUS_AUTH env var (never CLI args)
• Preserves existing Obsidian frontmatter (tags, aliases, etc.) on fetch and publish
• Path traversal prevention, YAML injection prevention, filesystem-safe sanitization
• Multi-line stderr parsing, concurrent folder creation handling
• Installable via BRAT: latebit-io/obsidian-demarkus
• GitHub release 0.1.0 with manifest.json + main.js assets

Next:

• Test in real Obsidian usage
• Submit to Obsidian Community Plugins directory
• Auto-sync from monorepo to standalone repo (CI)

Features Not Prioritized — Backlog

These are features that were discussed, proposed, or noted as missing but deliberately not prioritized. They don't block core functionality.

WebSub-style Subscriptions (SUBSCRIBE verb)

Status: Removed from protocol spec

WebSub-style push subscriptions for change notifications. Servers would notify registered subscribers when documents change. Deferred because:

• Adds complexity: requires maintaining subscriber state and callback reliability
• Polling works well enough: clients use conditional FETCH (if-none-match/if-modified-since) to check for changes efficiently
• Adds a new verb and protocol concern (push delivery, retry logic, subscriber management)
• Most use cases are adequately served by periodic polling or push at a higher level (e.g., webhook integrations built on top of demarkus)
• Simpler protocols are more resilient — no dependency chains for delivery

If real-time updates become critical, webhooks can be built as an application layer on top of PUBLISH without modifying the protocol.

Offline Mode (Client)

Status: Deferred to backlog

Cache and read-only access when network is unavailable. Low priority because:

• Most use cases have intermittent network, not prolonged offline periods
• Graph store already caches documents locally
• Not a blocking use case for current users

Full-Text Search

Status: Removed from protocol spec, available as external tool

Full-text search across all documents was considered as a protocol verb but removed because:

• Search is a policy concern (what to index, how to rank, which fields matter)
• The Mark Protocol handles data, not policy
• Better handled by building external search tools on top of LIST + FETCH
• Clients can build their own indexes (e.g., SQLite FTS, Bleve)

Diff / Changelog Between Versions

Status: Noted as useful, not implemented

Display what changed between two versions of a document. Would be useful for:

• Journals and thoughts that evolve over time
• Understanding revision history at a glance
• Change summaries in versioned stores

Not prioritized because:

• Clients can fetch v1 and v2 and do their own diff
• Could be a TUI/CLI feature rather than protocol-level
• Low-priority UX improvement

Blind Append with Content Deduplication

Status: Proposed, not pursued

Allow APPEND without expected_version and have the server silently de-dupe based on content hash. Rejected because:

• Hides the non-idempotent nature of append operations
• Retries become unsafe — can't distinguish "legitimate second append" from "accidental duplicate"
• Making version-checking mandatory forces callers to understand current state first
• One extra fetch-for-version is the correct cost of safe operations