# Roadmap

Where demarkus has been and where it's going.

## Scope: Markdown-Only Server

The demarkus server serves markdown only. Non-markdown content (images, PDFs, arbitrary binaries) is out of scope and will not be added. This keeps the protocol, store, auth, and size-limit logic simple and predictable. Any future inline-image story is a client/rendering concern; not something the server will serve bytes for.

## Phase 1: MVP (Read-Only): COMPLETE

Everything shipped:
- QUIC server serving markdown files
- FETCH, LIST, VERSIONS verbs
- TUI client with Bubble Tea + Glamour
- Link following, navigation history
- Document graph visualization
- CLI client with all verbs
- MCP integration for LLM agent access
- Docker multi-arch images
- GoReleaser CI/CD with per-module versioning
- Conditional fetch (if-none-match, if-modified-since)
- SIGHUP certificate reload

## Phase 2: Publish Operations: COMPLETE

Done:
- PUBLISH verb with version creation
- ARCHIVE verb
- APPEND verb: sends only new content, server handles concatenation
- Capability-based auth (token generation, SHA-256 hashes, path/op scoping)
- Versioned store with symlinks and hash chain
- Document editing via $EDITOR
- Client-side token management
- Conflict resolution (optimistic concurrency with expected-version)
- No-op on duplicate content
- Structured logging with slog (replaced console logging)
- Protocol-level size limits (1 MiB body, 64KB frontmatter)
- Audit logging with token_label on all write operations
- Usability audit: documentation accuracy, CLI help, install script cleanup, CI lint

Phase 2 is fully complete. No remaining items.

## Phase 3: Agent-Native & Advanced Features: COMPLETE

Done:
- Agent manifest discovery (`/.well-known/agent-manifest.md`)
- Bookmarks/favorites
- MCP `mark_append` auto-resolve `expected_version`
- Content-addressed fetch
- Federation via MCP tools

## Phase 4: The Information Graph: COMPLETE

Done:
- The Demarkus Hub pattern
- Persistent graph store
- Backlinks
- Graph as content (export/import)
- Graph-aware navigation (TUI)
- Agent discovery

## Phase 5: The Demarkus Agent & Private Networks: IN PROGRESS

### Read Auth for Private Networks: DONE

Per-path read token enforcement. Server-side and client-side shipped.

### Security Hardening: DONE

See [plan](/plans/security-hardening.md) for full details.

- **Systemd hardening**: install script generates units with ProtectSystem=strict, ReadWritePaths, NoNewPrivileges, etc. Conditional ProtectHome. Update path detects insecure config and prompts to harden with rollback.
- **`-read-only` mode**: `DEMARKUS_READ_ONLY` env var / `-read-only` flag. Handler rejects PUBLISH/APPEND/ARCHIVE with `not-permitted`. Zero write access needed.
- **`demarkus-publish`**: CLI tool that writes directly to the versioned store on disk. Enables local publishing when server runs read-only. Shares `store.Write()` with the server.
- **Read-only chroot install**: `install-readonly.sh`. Chrooted server with ReadOnlyPaths=/, BindReadOnlyPaths=/dev/urandom. Maximum lockdown: Gemini-level security with full versioning.
- **Security documentation**: attack surface analysis, threat model, comparison table, hardening guide.

### Core Loop: Crawl & Index

1. **Seed**: start from configured servers (hub, known peers)
2. **Crawl**: follow `mark://` links, discover new servers and documents
3. **Hash**: collect content hashes from every document
4. **Index**: publish updated hash indexes to configured hubs
5. **Repeat**: on a configurable schedule, with conditional fetch (if-none-match) to be polite

### Server-to-Server Sync

Rsync for the Mark Protocol. Replicate content between servers using content hashes as the diff mechanism.

**How it works:**
1. **LIST** both source and destination servers
2. **Compare** content hashes: skip documents that match
3. **FETCH** changed/new documents from source
4. **PUBLISH** to destination with the fetched content
5. Optionally handle deletions (ARCHIVE on destination for docs removed from source)

**Sync modes:**
- **Mirror**: destination becomes an exact copy of source (one-way)
- **Selective**: sync specific paths or glob patterns (e.g., `/docs/*` only)
- **Multi-source**: aggregate content from multiple servers into one destination

### Key Design Points

- **Go binary, not an LLM agent**: mechanical work, not reasoning
- **Polite crawling**: conditional fetch, configurable rate limits
- **Hub-aware**: reads and updates hub index documents
- **Daemon or cron**: continuous or triggered
- **Auth-aware**: passes read and write tokens for private servers

### Implementation Sketch

- New module: `tools/demarkus-agent` (the `tools/` directory is the home for utility binaries; `demarkus-token`, `demarkus-publish`, and now sync/agent tooling)
- Reuses: `fetch.Client`, `graphstore`, `client/internal/index`, `graph.Crawl`
- CLI: `demarkus-agent crawl`, `demarkus-agent sync source dest`, `demarkus-agent daemon`

### External Tool Sync: PLANNED

The shape that lets any markdown-emitting tool (OpenSPDD, ADR generators, doc generators, custom workflows) land its output on demarkus without integrating the protocol itself. The integration burden lives on demarkus's side; upstream tools change nothing.

**`demarkus-sync`**: single binary in `tools/`. Watches a local directory and PUBLISHes (or writes via `demarkus-publish` against a local store) on every changed `.md`. Versioning happens automatically through the existing store. Cross-team sharing emerges naturally from publishing to a team server.

**Core flags:**
- `--watch <dir>`: directory to watch
- `--target mark://host/path/`: remote demarkus server (uses `fetch.Client` PUBLISH)
- `--store <local-path>`: alternative: write directly to a local versioned store via `store.Write`
- `--include` / `--exclude` glob filters (e.g. `**/*.canvas.md`)
- `--token` / `DEMARKUS_AUTH` for write auth

**Project-flow integration is the killer use:** drop into a Makefile target, pre-commit hook, CI step, or long-running dev daemon. Design contracts, ADRs, RFCs, runbooks, agent canvases all flow into the team world automatically; no per-tool integration work.

**OpenSPDD as first concrete validation:** REASONS Canvas docs are pure markdown contracts written to a local directory. demarkus-sync makes them a versioned, federated demarkus surface without OpenSPDD knowing. Same pattern works for any tool that emits markdown.

**The deeper framing:** demarkus shouldn't ask other tools to integrate it. It should make any markdown-emitting tool's output queryable as a versioned protocol surface, transparently. demarkus-sync is the generic shape of that.

Reuses: `fetch.Client` for remote PUBLISH, `store.Write` for local-store writes, fsnotify for the watch loop.

## Phase 6: Universe Deployment: PLANNED

See [plan](/plans/universe-deployment.md) for full details.

Make it cheap to stand up *N* worlds and onboard real users without hand-distributing tokens. Phase 5 gave us a single hardened server; Phase 6 makes a universe of them deployable with one Helm chart, one `ApplicationSet`, and one OIDC-fronted token broker.

**Sub-phases:**

- **6.1: Helm chart** (`deploy/helm/demarkus-server/`): StatefulSet + `volumeClaimTemplates` so each world owns its own PVC, Service, bootstrap Job that runs `demarkus-token generate` and writes to a Secret, optional read-only mode, SIGHUP reload wiring.
- **6.2: Universe topology** (`deploy/k8s/examples/`): reference Argo CD `ApplicationSet` over a `worlds:` list, plus a Kustomize overlay for clusters without Argo. No new code.
- **6.3: Token broker** (prototype in `tools/demarkus-broker/`, splits to `latebit-io/demarkus-broker` after 6.4): OIDC-fronted HTTP service that mints scoped tokens, writes hashes to world Secrets, SIGHUPs on revocation, audit log to stdout.
- **6.4: User install flow** (lives with broker): `GET /me/install` returns a one-shot shell script that writes `~/.config/demarkus/auth` and idempotently patches `~/.claude.json` with a `demarkus-mcp` server entry per world.
- **6.5: Docs**: `/deployment.md` for chart values + broker setup; plan doc maintained at `/plans/universe-deployment.md`.

**Constraint:** zero changes to protocol or core server. Phase 6 is packaging and lifecycle. Any need for a new core primitive must be discussed and justified before landing, same rule that applied to the Claude Code plugin.

**Explicitly deferred to Phase 7+:** multi-replica worlds with shared storage, cross-cluster universe federation, an operator with a `World` CRD, a web UI on top of the broker.

## Verb Set: Complete

7 verbs: FETCH, LIST, VERSIONS, PUBLISH, APPEND, ARCHIVE, LOOKUP.

LOOKUP (issue #113) is the catalog verb: subject lookup over an in-memory, importance-ranked catalog of declared tags + title. It is a token-efficient supplement to the index, not full-text search (which stays in an opt-in sidecar). See the LOOKUP Catalog Verb section below and [/plans/lookup-verb.md](/plans/lookup-verb.md).

## LOOKUP Catalog Verb: DONE (merged to main 2026-05-30, PR #166; plugin tail closed by #168)

See [plan](/plans/lookup-verb.md). Tracks [issue #113](https://github.com/latebit-io/demarkus/issues/113).

Given a subject, return which documents are about it and how important they are; a card-catalog supplement to the index hub so an agent finds the right doc without fetching and grepping the whole index. Catalog scan only: in-memory `path → {tags, importance, title}`, built on the startup walk and maintained inline on writes, ranked by match-count then importance. No body reads at query time, no relevance modeling. Full-text / semantic search is permanently out of core, in an opt-in sidecar.

Shipped:
- Protocol verb + spec §6.7.
- `server/internal/catalog` package + handler with read-auth result filtering.
- Client surfaces: `fetch.Client.Lookup`, `demarkus lookup` CLI, MCP `mark_lookup`.
- Broker parity: `mark_lookup` is the gateway's 14th tool.
- **Publisher metadata:** `tags`/`importance` settable at publish; CLI `-meta key=value` (repeatable), and a `metadata` object on MCP + broker `mark_publish`. This is what makes the tag/importance ranking actually usable (before it, LOOKUP was title-only with uniform 0.5 importance).
- End-to-end smoke test passed (publish with tags → lookup by tag → archive, over QUIC).
- **Plugin surfacing (v0.3.0, #168, 2026-05-31):** Claude Code plugin wires `mark_lookup`, fixes the `knowledge-join` tool-count text (13→14), and injects SessionStart guidance so sessions recall via lookup.

Remaining tail (small): `mark_append` metadata; deferred by design (you tag on publish, not append). The plugin `knowledge-join`/version-pin items are now shipped in v0.3.0.

## Open Knowledge Format (OKF) Compatibility: DONE (merged 2026-06-22, PR #195)

demarkus interoperates with Google's Open Knowledge Format v0.1 (a bundle of markdown files with YAML frontmatter). The document content model is OKF-compatible; demarkus layers versioning, hash chain, QUIC transport, capability auth, and LOOKUP around it.

Shipped:
- **Store metadata alignment**: recognized OKF fields (`type`, `title`, `description`, `resource`, `tags`, `timestamp`) serialize as bare frontmatter; `tags` as a YAML flow list; non-spec keys keep the `meta.` prefix; reserved operational fields (`version`, `previous-hash`, `archived`) enforced by name. Back-compatible read of legacy `meta.*`. Caps raised to 50 keys / 1024 bytes (frontmatter budget 2048), `tags` counted at serialized length.
- **`demarkus okf` codec** (`client/internal/okf`); `validate` (v0.1 conformance), `import` (bundle → world, sanitize/cap with warnings, link rewrite, upsert), `export` (world subtree → conformant bundle, frontmatter reattach, type synthesis, RFC 3339 timestamp normalization). Verified live: import → export → validate round-trip, byte-identical bodies.
- **Opinionated default**: server assigns `type: Document` on PUBLISH and APPEND when none declared (reserved `index.md`/`log.md` exempt), so every served document is a typed OKF concept by construction. Non-breaking for existing typeless writers (ADR 0003).
- Spec §8.1/§9.4/§13/§14; ADR 0002 (metadata alignment), ADR 0003 (type default).
- Review hardening (PR #195 CodeRabbit): export path-traversal guard, import symlink rejection, LIST decode/cycle hardening, reserved-key read guard, cap re-validation, link query strip.

## MCP Agent Ergonomics & Attach Surface: DONE (2026-07-04/05)

Two same-week phases that made MCP agents cheap to serve and demarkus
pleasant to drive from Claude Code / Desktop. Plans:
[/plans/mcp-client-ergonomics.md](/plans/mcp-client-ergonomics.md) and
[/plans/mcp-resources-prompts.md](/plans/mcp-resources-prompts.md). Both
fully shipped AND deployed on both MCP surfaces (local `demarkus-mcp` and
the broker gateway).

Shipped:
- **Size-adaptive `mark_fetch`**: full body under 8KB; above it an outline
  (heading tree with GitHub-slug `#anchors` + per-section line counts +
  opening paragraph + hint). `url#anchor` slices one section at any size;
  `force` overrides. Killed the five-figure-token fetches of append-heavy
  docs (measured: a 52KB roadmap → 5.9KB outline).
- **Session unchanged-dedup**: re-fetch of an unchanged doc returns
  `status: unchanged since vN` (identity = version+etag; identity-flip
  edge cases spelled out). Process-scoped in the local client;
  MCP-session-keyed at the broker (multi-tenant safe: no session → no
  dedup; LRU eviction at cap, since mcp-go only unregisters sessions on
  explicit DELETE).
- **`mark_explore`**: one-call orientation card (outline head, opening
  paragraph, outbound links, backlinks, siblings; 10/section + "+N more").
- **MCP resources**: documents as client-attachable context, zero tool
  turns: `mark://` URI templates, `#anchor` section attach, well-known +
  background-listed picker entries (client) / per-world index hubs
  (broker).
- **MCP prompts**: `orient` / `recall` / `whats-new` as server-vended
  slash commands; the broker variants sweep worlds via `mark_worlds`.
- **Shared packages** `client/mdoutline` (outline/slice + render helpers)
  and `client/fetchdedup` (dedup identity + notice text); one
  implementation for both surfaces (the changedNote mirror drifted twice
  in review before hoisting; lesson in /debugging.md).
- **One `#section` grammar everywhere:** mark_fetch, mark_explore, MCP
  resource URIs, and the demarkus-library librarian's `open` tool (adopted
  2026-07-05) all share the same anchors from the same implementation.
- Deployed: plugin users via client v0.15.0→v0.17.0 (tools 0.4.1+, pin
  chain now self-driving after the #228 pin-bump automation fix);
  knowledge.demarkus.io via broker 0.5.0 (ergonomics) and 0.6.0
  (resources+prompts).

Remaining: none. Follow-ups all landed (broker parity, resources/prompts,
librarian adoption).

## Distribution & Package Management: PLANNED

Homebrew tap for easier installation on macOS and Linux.

## Build Targets

Supported platforms: macOS, Linux, Windows (WSL only).

## Plugins: IN PROGRESS

### Obsidian Plugin: v0.1.0 RELEASED

Standalone repo: `latebit-io/obsidian-demarkus`. Source in `plugins/obsidian/`.

### Claude Code Plugin: v0.12.5 (current)

Merged into `main` via PR #96 (commit af8e210) on 2026-04-23 as v0.1.0. Source at `plugins/claude-code/` in the monorepo; marketplace manifest at `.claude-plugin/marketplace.json`.

See [plan](/plans/claude-code-plugin.md) for the full design. One-click marketplace install that lazy-spawns a local `demarkus-server`, auto-generates a token, and wires `demarkus-mcp` into the agent. Zero config on install. Includes `/soul`, `/soul-journal`, `/soul-init`, `/soul-join`, `/soul-default`, `/knowledge-join` slash commands and a `soul-memory` skill.

Shipped with a project-centric soul schema: `/index.md` is a project list, each project under `/<slug>/` holds `plan/tasks.md`, `architecture.md`, `patterns.md`, `roadmap.md`, `adr/`, `journal/<YYYY-MM-DD>.md`.

**Version history:**
- **v0.1.0** (2026-04-23, #96): initial marketplace plugin.
- **v0.2.0** (2026-05-23, #152): `/knowledge-join` command for organizational broker-fronted knowledge systems.
- **v0.3.0** (2026-05-31, #168): wires `mark_lookup`; injects standing SessionStart guidance (`context/session-guidance.md`) so sessions self-document to the soul and recall via lookup; fixes `knowledge-join` tool count (13→14); bumps binary pins to SERVER 0.17.13 / CLIENT 0.12.38 / TOOLS 0.1.28.
- **v0.4.0–v0.9.x**: not enumerated here (promote bridge, soul-refresh, binary-pin automation, OKF release pin, soul catalog groundwork). See the journals for detail.
- **v0.10.0 / v0.10.1** (#200, #201): **`/soul-join`: managed remote souls.** Joins a direct-QUIC remote soul (token in a 0600 file via a launch wrapper, never inline in MCP config), records it in a **catalog** (`~/.demarkus/souls`) with a per-project **binding** (`~/.demarkus/project-souls`), and enforces the binding with a **destination gate** (PreToolUse on `mark_publish`/`mark_append`; a write to a non-bound soul is denied/asked/warned). Publish tag-gate extended to joined remote souls. 0.10.1 closed a reuse-server restart bug (never kill a healthy externally-managed server on binary upgrade).
- **v0.11.0** (#202, 2026-06-22): **`/soul-default`: set a project's default write target.** Standalone command to re-point an already-joined repo at a different default without re-running `/soul-join` (until now the binding was only ever written as a `--bind` side-effect of join). Lists the joined-soul catalog (local managed soul + remotes), picks one, saves the binding the dest-gate enforces. Routing stays model-driven: the catalog is the discovery surface and each soul is its own MCP server, so no router or dispatch tool; reads and one-off writes to another joined soul go direct to its `mcp__<slug>__mark_*` tools (the gate fires only on publish/append). New `lib.sh` helpers (`local_soul_present`, `soul_catalog`, `is_catalog_soul`); no core/protocol change.
- **v0.12.x** (late June → 2026-07-05): pi-agent port fixes (#217), the
  ergonomics release-chain repins (tools 0.3.5 → 0.4.1, #221/#227), and the
  automated pin-bump sweep (#229) after the pin-bump workflow's closed-PR
  upsert bug was fixed (#228: the pin chain is self-driving again:
  release → auto bump PR → merge). Current: demarkus-memory 0.12.5,
  demarkus-knowledge 0.5.5.

**No core code changes landed.** The plugin is built entirely on existing primitives: `DEMARKUS_AUTH` env var for token injection, `/health` endpoint plus ALPN `"mark"` negotiation for port probing, and shell redirection of `demarkus-token generate` stdout for writing the token to a file. Any future need for a new core primitive must be discussed and justified before landing.

## TUI Polish

### External Links: PLANNED (implementation ready to commit)

See [plan](/plans/external-links.md). Closes [issue #75](https://github.com/latebit-io/demarkus/issues/75) in full. Opens http/https/gemini/mailto links in the user's default handler via `open`/`xdg-open`/`rundll32`. Scheme allowlist, URL passed as argv only, no shell.

The original plan had a Phase 2 for same-server non-markdown files (images); **dropped** because the server is markdown-only and will not serve binaries. Inline image rendering, if ever pursued, is a separate client-rendering initiative (terminal graphics protocols like Kitty/Sixel), not a continuation of this work.

## Features Not Prioritized: Backlog

- WebSub-style Subscriptions (removed from spec)
- Offline Mode (deferred)
- Full-Text / Semantic Search (out of core: opt-in sidecar reading via LIST/FETCH; in-core LOOKUP covers catalog-level subject lookup, see above)
- Diff / Changelog Between Versions (noted, not implemented)
- Blind Append with Content Deduplication (rejected)
- Non-markdown content types on the server (out of scope by design)


## Operational Polish: Backlog

- **Plain-directory root returns silent `not-found`**; the server only serves the
  versioned-symlink store layout. Pointing `-root` at a directory of plain `.md`
  files (e.g. `docs/site`) silently returns `not-found` for every path and builds
  a 0-entry hash index and catalog, with no startup warning. Bit setup during the
  2026-06-02 load-test work. Flagged to revisit: at minimum a startup warning when
  the root has markdown files but no `versions/` layout; ideally a clear hint.

- **`tools/demarkus-loadtest`**: concurrent read-only load generator (FETCH/LIST/
  VERSIONS/LOOKUP) for capacity measurement. Reuses `fetch.Client`; warm
  (reused-conn) vs `-fresh` (handshake-per-request) regimes. Report and first
  measured numbers in `tools/demarkus-loadtest/REPORT.md`. See journal 2026-06-02.
  Key finding: server self-throttles per IP (default 50 req/s, burst 100,
  `DEMARKUS_RATE_LIMIT=0` disables); warm ceiling ~7.7k req/s, cold ~1.4k req/s on
  one core.


## Docs Hygiene

- **SPEC.md markdownlint cleanup: DONE (PR #196, merged 2026-06-22).** Added
  repo-wide `.markdownlint.json` disabling MD013 (line-length) and MD060
  (table-column-style): both pervasive, intentional spec style. Fixed the
  substantive rules in `docs/SPEC.md`: fence languages (MD040, 29 fences →
  `text`), blank lines around fences/lists (MD031/MD032), bare URLs (MD034), and
  the emphasis footer (MD036). `markdownlint-cli2 docs/SPEC.md` → 0 errors;
  formatting only, no content changes.