# Roadmap

Where demarkus has been and where it's going.

## Phase 1: MVP (Read-Only) — COMPLETE

Everything shipped:
- QUIC server serving markdown files
- FETCH, LIST, VERSIONS verbs
- TUI client with Bubble Tea + Glamour
- Link following, navigation history
- Document graph visualization
- CLI client with all verbs
- MCP integration for LLM agent access
- Docker multi-arch images
- GoReleaser CI/CD with per-module versioning
- Conditional fetch (if-none-match, if-modified-since)
- SIGHUP certificate reload

## Phase 2: Publish Operations — COMPLETE

Done:
- PUBLISH verb with version creation
- ARCHIVE verb
- APPEND verb — sends only new content, server handles concatenation
- Capability-based auth (token generation, SHA-256 hashes, path/op scoping)
- Versioned store with symlinks and hash chain
- Document editing via $EDITOR
- Client-side token management
- Conflict resolution (optimistic concurrency with expected-version)
- No-op on duplicate content
- Structured logging with slog (replaced console logging)
- Protocol-level size limits (1 MiB body, 64KB frontmatter)
- Audit logging with token_label on all write operations
- Usability audit: documentation accuracy, CLI help, install script cleanup, CI lint

Phase 2 is fully complete. No remaining items.

## Phase 3: Agent-Native & Advanced Features — COMPLETE

Done:
- Agent manifest discovery (`/.well-known/agent-manifest.md`)
  - Convention spec: markdown document at well-known path, no server changes needed
  - `WellKnownManifestPath` constant in protocol module
  - `mark_discover` MCP tool — fetches manifest from connected server
  - `demarkus info` CLI subcommand — fetches and displays manifest
  - Spec doc at `docs/site/reference/agent-manifest.md`
  - First live manifest published on demarkus-soul
  - Directory listing for servers (auto-generated when no index.md)
- Bookmarks/favorites
  - Markdown-backed store at `~/.mark/bookmarks.md`
  - TUI: `b` toggle, `B` view, `Esc` exit, star indicator in status bar
  - CLI: `bookmark add/remove/list` subcommands
  - MCP: not exposed (local-only feature)
- MCP `mark_append` auto-resolve `expected_version`
  - `expected_version` is now optional in `mark_append` — when omitted or 0, the tool calls VERSIONS to get the current version automatically
  - Optimistic concurrency still enforced — the server always receives an `expected_version`
  - Introduced `markClient` interface for handler testability
  - Tests cover auto-resolve (happy path), not-found, explicit version, negative version, no token
  - Does NOT apply to `mark_publish` (publish requires the agent to have read the document first)
- Content-addressed fetch
  - FETCH responses include `content-hash: sha256-<64hex>` — SHA-256 of the stripped body
  - `FETCH /sha256-<64hex>` retrieves documents by content hash from any server that has them
  - In-memory hash index (`map[string]string`) built on startup, updated on writes/archives
  - Current versions only, archived docs excluded
  - No new verbs — FETCH handles it via path pattern detection
  - Foundation for distributed mirroring and caching
- Federation via MCP tools
  - `mark_index` — crawls a source server, collects content hashes, publishes index to a hub
  - `mark_resolve` — resolves content by SHA-256 hash using a hub index document
  - `client/internal/index` package — Parse, Build, Merge for markdown hash index documents
  - Manifest check enforced by tool, `force` override, `dry_run` mode, 1000 doc cap

## Phase 4: The Information Graph — COMPLETE

Discovery through linking, not searching. Information is findable because it's connected — servers link to each other through documents, clients and agents traverse the graph.

### Done
- **The Demarkus Hub pattern** — live at `mark://hub.demarkus.io`. A server whose sole purpose is linking to content on other demarkus servers. No original content, just curated `mark://` links organized by topic. CI auto-publishes from Git repo. See [github.com/latebit-io/demarkus-hub](https://github.com/latebit-io/demarkus-hub).
- **Persistent graph store** — `client/internal/graphstore` package. Graph stored as JSON at `~/.mark/graph.json` with nodes (documents), edges (links), etags, and crawl timestamps. Atomic writes, schema versioning, incremental merge.
- **Shared `CrawlAndPersist`** — unified crawl + merge + save method on `*Store`, nil-safe. All three clients (CLI, TUI, MCP) share the same code path. `EtagFetcher` adapter collects etags during crawl.
- **Backlinks** — `mark_backlinks` MCP tool and `Store.BacklinksEnriched()` method. Reverse edge lookup from the persistent graph — "what links here?" Returns sorted list with document titles and status.
- **Graph seeding from store** — TUI graph view loads instantly from stored graph while crawl runs in background.
- **Graph as content** — export the persistent graph as a publishable markdown document. `Store.Export()` renders nodes as markdown table rows with `mark://` links, edges as a separate table. `ParseExport()` parses it back. CLI: `demarkus graph export [-o file.md]`. MCP: `mark_graph_export` tool. Crawling the exported doc naturally discovers the topology — no special import needed. Cell escaping handles pipes and backslashes in titles.
- **Graph-aware navigation** — TUI graph view with three sub-views: Links (BFS tree from current doc with `[N←]` density indicators), Backlinks (reverse links to current doc), Topology (all explored nodes sorted by importance). Sub-views toggled with `d`/`r`/`t` keys. Shared `graphstore.BacklinksEnriched()` eliminates duplication between TUI and MCP. `graph.InDegrees()` for O(E) backlink counting. Rune-safe truncation for multi-byte UTF-8.
- **Agent discovery** — agents crawl the graph to build knowledge maps using `mark_graph` (crawl + persist), `mark_graph_export` (publish as markdown), and `mark_resolve` (hash-based lookup via hub index). Graph documents are themselves crawlable, so sharing and merging happens naturally through the protocol. No special agent-specific code needed — the existing primitives compose.

No new verbs needed. FETCH reads content, LIST enumerates, links connect. The protocol already has the primitives.

See DESIGN.md Phase 4 section for full sketch.

## Phase 5: The Demarkus Agent & Private Networks — IN PROGRESS

An autonomous crawler daemon that keeps the demarkus network healthy, a sync engine for server-to-server replication, and read authentication for private networks.

### Vision

The network should be self-maintaining. When a new server appears and links to the hub, the agent finds it. When content changes, hash indexes update. When the graph grows, the topology stays fresh. No human intervention needed.

Content should be resilient. If a server goes down, its documents survive on mirrors. Syncing between servers should be as easy as rsync — point at a source, point at a destination, let content hashes handle the diff.

Servers should support both public and private use cases. The same protocol, the same tools, with an opt-in gate for read access when privacy is needed.

### Read Auth for Private Networks — DONE

Per-path read token enforcement for private networks. See [plan](/plans/read-auth.md) for full design.

**Server-side (shipped earlier):**
- `RequiresReadAuth(path)` on `TokenStore` — pre-computed `readPaths []string` at load time from tokens with `"read"` in operations
- `authorizeRead` handler helper — checks token store, exempts `/.well-known/agent-manifest.md`, gates FETCH/LIST/VERSIONS
- Content-addressed fetch respects read auth — hash lookups resolve to a real path first, then check auth on that path
- Versioned path auth — `/doc.md/v2` checks auth on the base path `/doc.md`
- Directory path normalization — `/private` and `/private/` both match `/private/**` patterns
- Fully backwards compatible — no read tokens configured = everything public
- No new config, no protocol changes — uses existing tokens TOML file and `"read"` operation

**Client-side (shipped 2026-03-16):**
- `tokens.Resolve(explicit, host, store)` — shared token resolution: flag > env > stored token. Used by all three clients.
- `tokens.LoadDefault()` — loads `~/.mark/tokens.toml` with graceful fallback to empty store
- `fetch.Client` read methods (`Fetch`, `List`, `Versions`) now accept a `token` parameter
- CLI: all commands and subcommands (`edit`, `graph`, `info`, `bookmark`) pass tokens to read operations
- TUI: loads tokens fresh on each fetch (picks up changes without restart), loads once per crawl
- MCP: `resolveToken(host)` method on handler, per-host token resolution with `-token` flag as override
- `-auth` flag description updated to reflect it applies to all operations, not just writes
- All duplicate token resolution code consolidated into `tokens.Resolve`

### Core Loop: Crawl & Index

1. **Seed** — start from configured servers (hub, known peers)
2. **Crawl** — follow `mark://` links, discover new servers and documents
3. **Hash** — collect content hashes from every document
4. **Index** — publish updated hash indexes to configured hubs
5. **Repeat** — on a configurable schedule, with conditional fetch (if-none-match) to be polite

### Server-to-Server Sync

Rsync for the Mark Protocol. Replicate content between servers using content hashes as the diff mechanism.

**How it works:**
1. **LIST** both source and destination servers
2. **Compare** content hashes — skip documents that match
3. **FETCH** changed/new documents from source
4. **PUBLISH** to destination with the fetched content
5. Optionally handle deletions (ARCHIVE on destination for docs removed from source)

**Sync modes:**
- **Mirror** — destination becomes an exact copy of source (one-way)
- **Selective** — sync specific paths or glob patterns (e.g., `/docs/*` only)
- **Multi-source** — aggregate content from multiple servers into one destination

**What makes this work:**
- Content hashes are already in every FETCH response — the diff is free
- LIST gives the full document tree — no need to crawl links
- PUBLISH handles versioning — optimistic concurrency prevents clobbering
- No new protocol features needed — just FETCH + LIST + PUBLISH in a loop

**Use cases:**
- **Backup** — mirror a server to a second host for redundancy
- **Edge caching** — replicate a hub's content to a server closer to users
- **Content aggregation** — pull from multiple sources into a single server
- **Migration** — move content between servers without downtime

### What It Uses

Only the existing 6 verbs — no protocol changes needed:
- **FETCH** — read documents, follow links, conditional fetch for polling
- **LIST** — enumerate server contents
- **VERSIONS** — check for changes cheaply
- **PUBLISH** — update hub index documents and sync content to destinations

### Key Design Points

- **Go binary, not an LLM agent** — this is mechanical work, not reasoning. No API keys, no token costs, no inference latency.
- **Polite crawling** — conditional fetch, configurable rate limits, respect for server load
- **Configurable seeds** — start from any set of known servers
- **Hub-aware** — knows how to read and update hub index documents using the existing `client/internal/index` package
- **Daemon or cron** — can run continuously with a sleep interval, or be triggered by cron/systemd timer
- **Graph integration** — uses the existing `graphstore` package to persist what it discovers
- **Docker-friendly** — should run as a lightweight container alongside a hub server
- **Auth-aware** — can pass read and write tokens when interacting with private servers

### Implementation Sketch

- New module: `cmd/demarkus-agent` (or standalone repo)
- Reuses: `fetch.Client`, `graphstore`, `client/internal/index`, `graph.Crawl`
- Config: YAML or flags for seed servers, hub target, crawl interval, rate limits, auth tokens
- Sync config: source/destination pairs, sync mode, path filters
- Logging: structured slog, same patterns as server
- CLI: `demarkus-agent crawl`, `demarkus-agent sync source dest`, `demarkus-agent daemon`

### What This Enables

- **Self-healing network** — hub indexes stay current without manual `mark_index` calls
- **Server discovery** — new servers that link into the network get found automatically
- **Content availability** — hash indexes grow as more content is crawled, making `mark_resolve` more reliable
- **Network health monitoring** — the agent's graph is a live map of what's up and what's down
- **Resilient content** — mirroring means no single point of failure for any document
- **Easy backup** — `demarkus-agent sync mark://source mark://backup` and you're done
- **Private networks** — teams and individuals can run authenticated servers that participate in the broader network selectively

## Verb Set — Complete

The protocol verb set is finalized at 6 verbs: FETCH, LIST, VERSIONS, PUBLISH, APPEND, ARCHIVE. 

**Removed from spec:**
- **SEARCH** — Full-text search is better handled as an external tool built on top of the existing primitives (LIST + FETCH) rather than as a protocol-level concern.
- **SUBSCRIBE** — WebSub-style subscriptions for change notifications were considered but deferred. Polling with conditional FETCH (if-none-match) is simpler and works for most use cases. Real-time push adds complexity and requires maintaining subscriber state. Better addressed by higher-level tools if needed.

## Distribution & Package Management — PLANNED

Homebrew tap for easier installation, especially on macOS and Linux developers.

**What it takes:**
1. Create public `latebit/homebrew-demarkus` tap repo
2. Add formula for `demarkus-client` (CLI + TUI) — pull pre-built binaries from GitHub releases
3. Formula needs SHA256 checksums for each architecture
4. Update README with Homebrew install instructions
5. Test installation on macOS and Linux

**Notes:**
- Server is better deployed via Docker, keep Homebrew focused on the client
- Start with tap (owned by us), not Homebrew Core (requires upstream submission)
- User installs: `brew tap latebit/homebrew-demarkus && brew install demarkus-client`
- Already have GoReleaser CI/CD and multi-arch builds — just need packaging

## Build Targets

Supported platforms: macOS, Linux, Windows (WSL only — runs Linux binaries). Native Windows builds removed from GoReleaser configs.

## Plugins — IN PROGRESS

### Obsidian Plugin — v0.1.0 RELEASED

Standalone repo: `latebit-io/obsidian-demarkus` on GitHub. Source lives in `plugins/obsidian/` in the monorepo, copied to standalone repo for releases.

**Done:**
- Fetch, publish, and list documents from Obsidian
- Shells out to `demarkus` CLI binary
- Token passed via `DEMARKUS_AUTH` env var (never CLI args)
- Preserves existing Obsidian frontmatter (tags, aliases, etc.) on fetch and publish
- Path traversal prevention, YAML injection prevention, filesystem-safe sanitization
- Multi-line stderr parsing, concurrent folder creation handling
- Installable via BRAT: `latebit-io/obsidian-demarkus`
- GitHub release 0.1.0 with manifest.json + main.js assets

**Next:**
- Test in real Obsidian usage
- Submit to Obsidian Community Plugins directory
- Auto-sync from monorepo to standalone repo (CI)

## Features Not Prioritized — Backlog

These are features that were discussed, proposed, or noted as missing but deliberately not prioritized. They don't block core functionality.

### WebSub-style Subscriptions (SUBSCRIBE verb)

**Status:** Removed from protocol spec

WebSub-style push subscriptions for change notifications. Servers would notify registered subscribers when documents change. Deferred because:
- Adds complexity: requires maintaining subscriber state and callback reliability
- Polling works well enough: clients use conditional FETCH (if-none-match/if-modified-since) to check for changes efficiently
- Adds a new verb and protocol concern (push delivery, retry logic, subscriber management)
- Most use cases are adequately served by periodic polling or push at a higher level (e.g., webhook integrations built on top of demarkus)
- Simpler protocols are more resilient — no dependency chains for delivery

If real-time updates become critical, webhooks can be built as an application layer on top of PUBLISH without modifying the protocol.

### Offline Mode (Client)

**Status:** Deferred to backlog

Cache and read-only access when network is unavailable. Low priority because:
- Most use cases have intermittent network, not prolonged offline periods
- Graph store already caches documents locally
- Not a blocking use case for current users

### Full-Text Search

**Status:** Removed from protocol spec, available as external tool

Full-text search across all documents was considered as a protocol verb but removed because:
- Search is a policy concern (what to index, how to rank, which fields matter)
- The Mark Protocol handles data, not policy
- Better handled by building external search tools on top of LIST + FETCH
- Clients can build their own indexes (e.g., SQLite FTS, Bleve)

### Diff / Changelog Between Versions

**Status:** Noted as useful, not implemented

Display what changed between two versions of a document. Would be useful for:
- Journals and thoughts that evolve over time
- Understanding revision history at a glance
- Change summaries in versioned stores

Not prioritized because:
- Clients can fetch v1 and v2 and do their own diff
- Could be a TUI/CLI feature rather than protocol-level
- Low-priority UX improvement

### Blind Append with Content Deduplication

**Status:** Proposed, not pursued

Allow APPEND without `expected_version` and have the server silently de-dupe based on content hash. Rejected because:
- Hides the non-idempotent nature of append operations
- Retries become unsafe — can't distinguish "legitimate second append" from "accidental duplicate"
- Making version-checking mandatory forces callers to understand current state first
- One extra fetch-for-version is the correct cost of safe operations