# Roadmap

Where demarkus has been and where it's going.

## Scope: Markdown-Only Server

The demarkus server serves markdown only. Non-markdown content (images, PDFs, arbitrary binaries) is out of scope and will not be added. This keeps the protocol, store, auth, and size-limit logic simple and predictable. Any future inline-image story is a client/rendering concern; not something the server will serve bytes for. Since 2026-07-23 (PR #273) this is enforced at the protocol surface, not just asserted; see the Markdown-Only Content Contract section below.

## Phase 1: MVP (Read-Only): COMPLETE

Everything shipped:
- QUIC server serving markdown files
- FETCH, LIST, VERSIONS verbs
- TUI client with Bubble Tea + Glamour
- Link following, navigation history
- Document graph visualization
- CLI client with all verbs
- MCP integration for LLM agent access
- Docker multi-arch images
- GoReleaser CI/CD with per-module versioning
- Conditional fetch (if-none-match, if-modified-since)
- SIGHUP certificate reload

## Phase 2: Publish Operations: COMPLETE

Done:
- PUBLISH verb with version creation
- ARCHIVE verb
- APPEND verb: sends only new content, server handles concatenation
- Capability-based auth (token generation, SHA-256 hashes, path/op scoping)
- Versioned store with symlinks and hash chain
- Document editing via $EDITOR
- Client-side token management
- Conflict resolution (optimistic concurrency with expected-version)
- No-op on duplicate content
- Structured logging with slog (replaced console logging)
- Protocol-level size limits (1 MiB body, 64KB frontmatter)
- Audit logging with token_label on all write operations
- Usability audit: documentation accuracy, CLI help, install script cleanup, CI lint

Phase 2 is fully complete. No remaining items.

## Phase 3: Agent-Native & Advanced Features: COMPLETE

Done:
- Agent manifest discovery (`/.well-known/agent-manifest.md`)
- Bookmarks/favorites
- MCP `mark_append` auto-resolve `expected_version`
- Content-addressed fetch
- Federation via MCP tools

## Phase 4: The Information Graph: COMPLETE

Done:
- The Demarkus Hub pattern
- Persistent graph store
- Backlinks
- Graph as content (export/import)
- Graph-aware navigation (TUI)
- Agent discovery

## Phase 5: The Demarkus Agent & Private Networks: IN PROGRESS

### Read Auth for Private Networks: DONE

Per-path read token enforcement. Server-side and client-side shipped.

### Security Hardening: DONE

See [plan](/plans/security-hardening.md) for full details.

- **Systemd hardening**: install script generates units with ProtectSystem=strict, ReadWritePaths, NoNewPrivileges, etc. Conditional ProtectHome. Update path detects insecure config and prompts to harden with rollback.
- **`-read-only` mode**: `DEMARKUS_READ_ONLY` env var / `-read-only` flag. Handler rejects PUBLISH/APPEND/ARCHIVE with `not-permitted`. Zero write access needed.
- **`demarkus-publish`**: CLI tool that writes directly to the versioned store on disk. Enables local publishing when server runs read-only. Shares `store.Write()` with the server.
- **Read-only chroot install**: `install-readonly.sh`. Chrooted server with ReadOnlyPaths=/, BindReadOnlyPaths=/dev/urandom. Maximum lockdown: Gemini-level security with full versioning.
- **Security documentation**: attack surface analysis, threat model, comparison table, hardening guide.

### Core Loop: Crawl & Index

1. **Seed**: start from configured servers (hub, known peers)
2. **Crawl**: follow `mark://` links, discover new servers and documents
3. **Hash**: collect content hashes from every document
4. **Index**: publish updated hash indexes to configured hubs
5. **Repeat**: on a configurable schedule, with conditional fetch (if-none-match) to be polite

### Server-to-Server Sync

Rsync for the Mark Protocol. Replicate content between servers using content hashes as the diff mechanism.

**How it works:**
1. **LIST** both source and destination servers
2. **Compare** content hashes: skip documents that match
3. **FETCH** changed/new documents from source
4. **PUBLISH** to destination with the fetched content
5. Optionally handle deletions (ARCHIVE on destination for docs removed from source)

**Sync modes:**
- **Mirror**: destination becomes an exact copy of source (one-way)
- **Selective**: sync specific paths or glob patterns (e.g., `/docs/*` only)
- **Multi-source**: aggregate content from multiple servers into one destination

### Key Design Points

- **Go binary, not an LLM agent**: mechanical work, not reasoning
- **Polite crawling**: conditional fetch, configurable rate limits
- **Hub-aware**: reads and updates hub index documents
- **Daemon or cron**: continuous or triggered
- **Auth-aware**: passes read and write tokens for private servers

### Implementation Sketch

- New module: `tools/demarkus-agent` (the `tools/` directory is the home for utility binaries; `demarkus-token`, `demarkus-publish`, and now sync/agent tooling)
- Reuses: `fetch.Client`, `graphstore`, `client/internal/index`, `graph.Crawl`
- CLI: `demarkus-agent crawl`, `demarkus-agent sync source dest`, `demarkus-agent daemon`

### External Tool Sync: PLANNED

The shape that lets any markdown-emitting tool (OpenSPDD, ADR generators, doc generators, custom workflows) land its output on demarkus without integrating the protocol itself. The integration burden lives on demarkus's side; upstream tools change nothing.

**`demarkus-sync`**: single binary in `tools/`. Watches a local directory and PUBLISHes (or writes via `demarkus-publish` against a local store) on every changed `.md`. Versioning happens automatically through the existing store. Cross-team sharing emerges naturally from publishing to a team server.

**Core flags:**
- `--watch <dir>`: directory to watch
- `--target mark://host/path/`: remote demarkus server (uses `fetch.Client` PUBLISH)
- `--store <local-path>`: alternative: write directly to a local versioned store via `store.Write`
- `--include` / `--exclude` glob filters (e.g. `**/*.canvas.md`)
- `--token` / `DEMARKUS_AUTH` for write auth

**Project-flow integration is the killer use:** drop into a Makefile target, pre-commit hook, CI step, or long-running dev daemon. Design contracts, ADRs, RFCs, runbooks, agent canvases all flow into the team world automatically; no per-tool integration work.

**OpenSPDD as first concrete validation:** REASONS Canvas docs are pure markdown contracts written to a local directory. demarkus-sync makes them a versioned, federated demarkus surface without OpenSPDD knowing. Same pattern works for any tool that emits markdown.

**The deeper framing:** demarkus shouldn't ask other tools to integrate it. It should make any markdown-emitting tool's output queryable as a versioned protocol surface, transparently. demarkus-sync is the generic shape of that.

Reuses: `fetch.Client` for remote PUBLISH, `store.Write` for local-store writes, fsnotify for the watch loop.

## Phase 6: Universe Deployment: PLANNED

See [plan](/plans/universe-deployment.md) for full details.

Make it cheap to stand up *N* worlds and onboard real users without hand-distributing tokens. Phase 5 gave us a single hardened server; Phase 6 makes a universe of them deployable with one Helm chart, one `ApplicationSet`, and one OIDC-fronted token broker.

**Sub-phases:**

- **6.1: Helm chart** (`deploy/helm/demarkus-server/`): StatefulSet + `volumeClaimTemplates` so each world owns its own PVC, Service, bootstrap Job that runs `demarkus-token generate` and writes to a Secret, optional read-only mode, SIGHUP reload wiring.
- **6.2: Universe topology** (`deploy/k8s/examples/`): reference Argo CD `ApplicationSet` over a `worlds:` list, plus a Kustomize overlay for clusters without Argo. No new code.
- **6.3: Token broker** (prototype in `tools/demarkus-broker/`, splits to `latebit-io/demarkus-broker` after 6.4): OIDC-fronted HTTP service that mints scoped tokens, writes hashes to world Secrets, SIGHUPs on revocation, audit log to stdout.
- **6.4: User install flow** (lives with broker): `GET /me/install` returns a one-shot shell script that writes `~/.config/demarkus/auth` and idempotently patches `~/.claude.json` with a `demarkus-mcp` server entry per world.
- **6.5: Docs**: `/deployment.md` for chart values + broker setup; plan doc maintained at `/plans/universe-deployment.md`.

**Constraint:** zero changes to protocol or core server. Phase 6 is packaging and lifecycle. Any need for a new core primitive must be discussed and justified before landing, same rule that applied to the Claude Code plugin.

**Explicitly deferred to Phase 7+:** multi-replica worlds with shared storage, cross-cluster universe federation, an operator with a `World` CRD, a web UI on top of the broker.

**Single-host mode (shipped 2026-07-17, PR #262 + #263):** the broker also runs on a plain VPS beside the server in file-backend mode, no Kubernetes; see the Remote Token Issuance section below and `/plans/remote-token-issuance.md`. This is the non-k8s delivery of the same broker; the k8s sub-phases above are unchanged.

## Verb Set: Complete

7 verbs: FETCH, LIST, VERSIONS, PUBLISH, APPEND, ARCHIVE, LOOKUP.

LOOKUP (issue #113) is the catalog verb: subject lookup over an in-memory, importance-ranked catalog of declared tags + title. It is a token-efficient supplement to the index, not full-text search (which stays in an opt-in sidecar). See the LOOKUP Catalog Verb section below and [/plans/lookup-verb.md](/plans/lookup-verb.md).

## LOOKUP Catalog Verb: DONE (merged to main 2026-05-30, PR #166; plugin tail closed by #168)

See [plan](/plans/lookup-verb.md). Tracks [issue #113](https://github.com/latebit-io/demarkus/issues/113).

Given a subject, return which documents are about it and how important they are; a card-catalog supplement to the index hub so an agent finds the right doc without fetching and grepping the whole index. Catalog scan only: in-memory `path → {tags, importance, title}`, built on the startup walk and maintained inline on writes, ranked by match-count then importance. No body reads at query time, no relevance modeling. Full-text / semantic search is permanently out of core, in an opt-in sidecar.

Shipped:
- Protocol verb + spec §6.7.
- `server/internal/catalog` package + handler with read-auth result filtering.
- Client surfaces: `fetch.Client.Lookup`, `demarkus lookup` CLI, MCP `mark_lookup`.
- Broker parity: `mark_lookup` is the gateway's 14th tool.
- **Publisher metadata:** `tags`/`importance` settable at publish; CLI `-meta key=value` (repeatable), and a `metadata` object on MCP + broker `mark_publish`. This is what makes the tag/importance ranking actually usable (before it, LOOKUP was title-only with uniform 0.5 importance).
- End-to-end smoke test passed (publish with tags → lookup by tag → archive, over QUIC).
- **Plugin surfacing (v0.3.0, #168, 2026-05-31):** Claude Code plugin wires `mark_lookup`, fixes the `knowledge-join` tool-count text (13→14), and injects SessionStart guidance so sessions recall via lookup.

Remaining tail (small): `mark_append` metadata; deferred by design (you tag on publish, not append). The plugin `knowledge-join`/version-pin items are now shipped in v0.3.0.

## Open Knowledge Format (OKF) Compatibility: DONE (merged 2026-06-22, PR #195)

demarkus interoperates with Google's Open Knowledge Format v0.1 (a bundle of markdown files with YAML frontmatter). The document content model is OKF-compatible; demarkus layers versioning, hash chain, QUIC transport, capability auth, and LOOKUP around it.

Shipped:
- **Store metadata alignment**: recognized OKF fields (`type`, `title`, `description`, `resource`, `tags`, `timestamp`) serialize as bare frontmatter; `tags` as a YAML flow list; non-spec keys keep the `meta.` prefix; reserved operational fields (`version`, `previous-hash`, `archived`) enforced by name. Back-compatible read of legacy `meta.*`. Caps raised to 50 keys / 1024 bytes (frontmatter budget 2048), `tags` counted at serialized length.
- **`demarkus okf` codec** (`client/internal/okf`); `validate` (v0.1 conformance), `import` (bundle → world, sanitize/cap with warnings, link rewrite, upsert), `export` (world subtree → conformant bundle, frontmatter reattach, type synthesis, RFC 3339 timestamp normalization). Verified live: import → export → validate round-trip, byte-identical bodies.
- **Opinionated default**: server assigns `type: Document` on PUBLISH and APPEND when none declared (reserved `index.md`/`log.md` exempt), so every served document is a typed OKF concept by construction. Non-breaking for existing typeless writers (ADR 0003).
- Spec §8.1/§9.4/§13/§14; ADR 0002 (metadata alignment), ADR 0003 (type default).
- Review hardening (PR #195 CodeRabbit): export path-traversal guard, import symlink rejection, LIST decode/cycle hardening, reserved-key read guard, cap re-validation, link query strip.

## MCP Agent Ergonomics & Attach Surface: DONE (2026-07-04/05)

Two same-week phases that made MCP agents cheap to serve and demarkus
pleasant to drive from Claude Code / Desktop. Plans:
[/plans/mcp-client-ergonomics.md](/plans/mcp-client-ergonomics.md) and
[/plans/mcp-resources-prompts.md](/plans/mcp-resources-prompts.md). Both
fully shipped AND deployed on both MCP surfaces (local `demarkus-mcp` and
the broker gateway).

Shipped:
- **Size-adaptive `mark_fetch`**: full body under 8KB; above it an outline
  (heading tree with GitHub-slug `#anchors` + per-section line counts +
  opening paragraph + hint). `url#anchor` slices one section at any size;
  `force` overrides. Killed the five-figure-token fetches of append-heavy
  docs (measured: a 52KB roadmap → 5.9KB outline).
- **Session unchanged-dedup**: re-fetch of an unchanged doc returns
  `status: unchanged since vN` (identity = version+etag; identity-flip
  edge cases spelled out). Process-scoped in the local client;
  MCP-session-keyed at the broker (multi-tenant safe: no session → no
  dedup; LRU eviction at cap, since mcp-go only unregisters sessions on
  explicit DELETE).
- **`mark_explore`**: one-call orientation card (outline head, opening
  paragraph, outbound links, backlinks, siblings; 10/section + "+N more").
- **MCP resources**: documents as client-attachable context, zero tool
  turns: `mark://` URI templates, `#anchor` section attach, well-known +
  background-listed picker entries (client) / per-world index hubs
  (broker).
- **MCP prompts**: `orient` / `recall` / `whats-new` as server-vended
  slash commands; the broker variants sweep worlds via `mark_worlds`.
- **Shared packages** `client/mdoutline` (outline/slice + render helpers)
  and `client/fetchdedup` (dedup identity + notice text); one
  implementation for both surfaces (the changedNote mirror drifted twice
  in review before hoisting; lesson in /debugging.md).
- **One `#section` grammar everywhere:** mark_fetch, mark_explore, MCP
  resource URIs, and the demarkus-library librarian's `open` tool (adopted
  2026-07-05) all share the same anchors from the same implementation.
- Deployed: plugin users via client v0.15.0→v0.17.0 (tools 0.4.1+, pin
  chain now self-driving after the #228 pin-bump automation fix);
  knowledge.demarkus.io via broker 0.5.0 (ergonomics) and 0.6.0
  (resources+prompts).

Remaining: none. Follow-ups all landed (broker parity, resources/prompts,
librarian adoption).

## Distribution & Package Management: PLANNED

Homebrew tap for easier installation on macOS and Linux.

## Build Targets

Supported platforms: macOS, Linux, Windows (WSL only).

## Plugins: IN PROGRESS

### Obsidian Plugin: v0.1.0 RELEASED

Standalone repo: `latebit-io/obsidian-demarkus`. Source in `plugins/obsidian/`.

### Claude Code Plugin: v0.13.5 (current)

Merged into `main` via PR #96 (commit af8e210) on 2026-04-23 as v0.1.0. Source at `plugins/claude-code/` in the monorepo; marketplace manifest at `.claude-plugin/marketplace.json`.

See [plan](/plans/claude-code-plugin.md) for the full design. One-click marketplace install that lazy-spawns a local `demarkus-server`, auto-generates a token, and wires `demarkus-mcp` into the agent. Zero config on install. Includes `/soul`, `/soul-journal`, `/soul-init`, `/soul-join`, `/soul-default`, `/knowledge-join` slash commands and a `soul-memory` skill.

Shipped with a project-centric soul schema: `/index.md` is a project list, each project under `/<slug>/` holds `plan/tasks.md`, `architecture.md`, `patterns.md`, `roadmap.md`, `adr/`, `journal/<YYYY-MM-DD>.md`.

**Version history:**
- **v0.1.0** (2026-04-23, #96): initial marketplace plugin.
- **v0.2.0** (2026-05-23, #152): `/knowledge-join` command for organizational broker-fronted knowledge systems.
- **v0.3.0** (2026-05-31, #168): wires `mark_lookup`; injects standing SessionStart guidance (`context/session-guidance.md`) so sessions self-document to the soul and recall via lookup; fixes `knowledge-join` tool count (13→14); bumps binary pins to SERVER 0.17.13 / CLIENT 0.12.38 / TOOLS 0.1.28.
- **v0.4.0–v0.9.x**: not enumerated here (promote bridge, soul-refresh, binary-pin automation, OKF release pin, soul catalog groundwork). See the journals for detail.
- **v0.10.0 / v0.10.1** (#200, #201): **`/soul-join`: managed remote souls.** Joins a direct-QUIC remote soul (token in a 0600 file via a launch wrapper, never inline in MCP config), records it in a **catalog** (`~/.demarkus/souls`) with a per-project **binding** (`~/.demarkus/project-souls`), and enforces the binding with a **destination gate** (PreToolUse on `mark_publish`/`mark_append`; a write to a non-bound soul is denied/asked/warned). Publish tag-gate extended to joined remote souls. 0.10.1 closed a reuse-server restart bug (never kill a healthy externally-managed server on binary upgrade).
- **v0.11.0** (#202, 2026-06-22): **`/soul-default`: set a project's default write target.** Standalone command to re-point an already-joined repo at a different default without re-running `/soul-join` (until now the binding was only ever written as a `--bind` side-effect of join). Lists the joined-soul catalog (local managed soul + remotes), picks one, saves the binding the dest-gate enforces. Routing stays model-driven: the catalog is the discovery surface and each soul is its own MCP server, so no router or dispatch tool; reads and one-off writes to another joined soul go direct to its `mcp__<slug>__mark_*` tools (the gate fires only on publish/append). New `lib.sh` helpers (`local_soul_present`, `soul_catalog`, `is_catalog_soul`); no core/protocol change.
- **v0.12.x** (late June → 2026-07-05): pi-agent port fixes (#217), the
  ergonomics release-chain repins (tools 0.3.5 → 0.4.1, #221/#227), and the
  automated pin-bump sweep (#229) after the pin-bump workflow's closed-PR
  upsert bug was fixed (#228: the pin chain is self-driving again:
  release → auto bump PR → merge).
- **v0.13.0** (2026-07-17, #261): **join URLs.** `/soul-join` accepts
  `mark://host#token=...` (one paste-able string from install.sh or
  `demarkus-token join`); new `demarkus join` CLI subcommand; pi-memory
  0.13.0 ships the same command update.
- **v0.13.x pins** (2026-07-18 → 07-24, #264/#266/#268/#274/#275): automated
  pin-bump chain keeping managed binaries current (the release.yml dispatch fix
  in #265 revived the dead release trigger). Current: demarkus-memory 0.13.5,
  demarkus-knowledge 0.5.21, tools 0.14.3.

**No core code changes landed.** The plugin is built entirely on existing primitives: `DEMARKUS_AUTH` env var for token injection, `/health` endpoint plus ALPN `"mark"` negotiation for port probing, and shell redirection of `demarkus-token generate` stdout for writing the token to a file. Any future need for a new core primitive must be discussed and justified before landing.

## TUI Polish

### External Links: PLANNED (implementation ready to commit)

See [plan](/plans/external-links.md). Closes [issue #75](https://github.com/latebit-io/demarkus/issues/75) in full. Opens http/https/gemini/mailto links in the user's default handler via `open`/`xdg-open`/`rundll32`. Scheme allowlist, URL passed as argv only, no shell.

The original plan had a Phase 2 for same-server non-markdown files (images); **dropped** because the server is markdown-only and will not serve binaries. Inline image rendering, if ever pursued, is a separate client-rendering initiative (terminal graphics protocols like Kitty/Sixel), not a continuation of this work.

## Features Not Prioritized: Backlog

- WebSub-style Subscriptions (removed from spec)
- Offline Mode (deferred)
- Full-Text / Semantic Search (out of core: opt-in sidecar reading via LIST/FETCH; in-core LOOKUP covers catalog-level subject lookup, see above)
- Diff / Changelog Between Versions (noted, not implemented)
- Blind Append with Content Deduplication (rejected)
- Non-markdown content types on the server (out of scope by design)
- TOFU cert-fingerprint pinning in join URLs (built on branch join-string, cut before merge; see the pinning decision in /plans/remote-token-issuance.md; any revival rides a redeem handshake)


## Operational Polish: Backlog

- **Plain-directory root returns silent `not-found`**; the server only serves the
  versioned-symlink store layout. Pointing `-root` at a directory of plain `.md`
  files (e.g. `docs/site`) silently returns `not-found` for every path and builds
  a 0-entry hash index and catalog, with no startup warning. Bit setup during the
  2026-06-02 load-test work. Flagged to revisit: at minimum a startup warning when
  the root has markdown files but no `versions/` layout; ideally a clear hint.

- **`tools/demarkus-loadtest`**: concurrent read-only load generator (FETCH/LIST/
  VERSIONS/LOOKUP) for capacity measurement. Reuses `fetch.Client`; warm
  (reused-conn) vs `-fresh` (handshake-per-request) regimes. Report and first
  measured numbers in `tools/demarkus-loadtest/REPORT.md`. See journal 2026-06-02.
  Key finding: server self-throttles per IP (default 50 req/s, burst 100,
  `DEMARKUS_RATE_LIMIT=0` disables); warm ceiling ~7.7k req/s, cold ~1.4k req/s on
  one core.


## Docs Hygiene

- **SPEC.md markdownlint cleanup: DONE (PR #196, merged 2026-06-22).** Added
  repo-wide `.markdownlint.json` disabling MD013 (line-length) and MD060
  (table-column-style): both pervasive, intentional spec style. Fixed the
  substantive rules in `docs/SPEC.md`: fence languages (MD040, 29 fences →
  `text`), blank lines around fences/lists (MD031/MD032), bare URLs (MD034), and
  the emphasis footer (MD036). `markdownlint-cli2 docs/SPEC.md` → 0 errors;
  formatting only, no content changes.

- **Single-host deployment page: DONE (2026-07-17).** `docs/site/deployment/single-host.md` covers the server + broker + library stack on one VPS, the auth-tier table (read and tool-write need no IdP; browser editing needs OIDC, self-hosted Dex/Keycloak/Authentik included), OIDC app setup, library SSO wiring, and uninstall.

- **Website brought current: DONE (2026-07-23, pages branch).** The site had not
  moved since 2026-06-22; everything from the last month was missing. Added the
  five-minute appliance, the library and its librarian, the indexing agent,
  promote/soul-join, retention, MCP resources and prompts. Framing decision and
  the library-screenshots maintenance note are in journal 2026-07-23.

## Knowledge Layer & Storage Backends: NEXT (added 2026-07-13)

### Wire backlinks/graph tools to the published /graph.md: DONE (see Graph hub seeding below)

The federation agent already publishes the aggregated link graph to each hub at /graph.md (hourly, cross-server edges, same format as mark_graph_export), but mark_backlinks and mark_graph answered only from the client's local ~/.mark/graph.json crawl cache. A fresh client answered from an empty graph while an authoritative aggregate sat on the hub unread.

- Seed/refresh the MCP graphstore from the hub's /graph.md (etag-cached fetch, merge into local store) before falling back to local crawl.
- Fixes cold start and cross-agent consistency with a client-only change; days of work, no server change.
- Later upgrade, not the fix: publish-time edge extraction in the store (instant, transactional backlinks). Subject to the backend-parity principle below.

### Principle: backend parity is system-level, not implementation-level

With two store backends (file, postgres), behavioral parity is a property of the system, not a detail of either implementation. The storetest conformance suite is the contract; both backends run it unchanged and divergence is a protocol bug.

Consequences for future work:

- Any new store-observable behavior lands in the conformance suite first, then in every backend, or it does not ship.
- Knowledge-layer features should prefer layers above the store (published documents like /graph.md, broker aggregation, client merge) which are backend-independent by construction.
- A feature that only one backend can provide efficiently (e.g. a transactional edges table in postgres) must still be defined system-level: either every backend implements the same observable surface, or the capability is exposed as an optional, discoverable feature rather than silently backend-dependent.

### Remaining gaps from the 2026-07-13 knowledge-layer analysis

The /graph.md wiring item above came out of a gap analysis (knowledge-system-deploy session, 2026-07-13): demarkus is strong at the document layer (versioning, integrity, provenance, gates) and weak at the knowledge layer. Only gap 1 got recorded at the time; the rest are recorded here so they are not lost. Framing correction from that session: the plugins are the knowledge layer (recall-first, tag-at-publish, promote, doctors), so the governing principle is that the agent owns judgment while the server owns the accumulation of that judgment (edges, indexes, derived rank live in shared storage so the layer compounds across agents instead of being recomputed per client).

#### Edge semantics: give edges more information

Every link in the graph is an untyped "mentions". Knowledge graphs live on predicates: supersedes, implements, depends-on, derived-from. OKF types the nodes; nothing types the edges (ADR 0002 superseding its v1 draft was expressible only as prose). Edges also carry no provenance: `graph.Edge`/`StoredEdge` are bare from/to pairs and /graph.md exports a two-column table, so backlinks cannot say why or where a doc links (no link text, no source anchor, no occurrence count).

Shape: a `rel:` metadata axis (e.g. `rel:supersedes=/adr/0002-....md`) surfaced in mark_graph/mark_backlinks, plus richer edge records in the export (label, source anchor, count). This is a plugin and export-format convention, zero server change, so it can land before publish-time edge extraction; when the store later learns edge extraction it must capture the same fields (backend-parity principle above).

#### Semantic recall beyond tags

Lookup recall depends entirely on declared tags and titles; recall bounded by metadata hygiene loses to grep, and the agent compensates by exploring and fetching more every session (judgment tokens spent on enumeration). Path: Postgres tsvector full-text over bodies (cheap, phase-2 adjacent), then pgvector embeddings, hybrid-ranked with declared importance. Indexes amortize the agent's search loop into infrastructure; they do not replace its judgment.

#### Universe-wide LOOKUP

Lookup is scoped to one world; the broker routes but does not aggregate, so "search the whole system" is client-side fan-out. A broker-level aggregated LOOKUP (and eventually graph) makes a multi-world system feel like one system. Broker feature, no server change.

#### Derived ranking signals

`importance` is entirely self-declared, so a well-tagged orphan outranks the hub everything cites. Blend link-graph signals (in-degree, centrality) with declared importance. Link structure is not usage telemetry, so this respects the no-tracking principle.

#### Staleness at lookup time

`modified` exists but nothing decays or flags aging content at query time; the doctors catch it offline only.

### Edge semantics: DONE (merged 2026-07-14, PR #251, commit 8798d0a)

The "Edge semantics: give edges more information" gap above is shipped; see [/plans/edge-semantics.md](/plans/edge-semantics.md) and repo ADR 0004. Edges carry {From, To, Rel, Label, Anchor, Count} with identity {From, To, Rel}; the `rel-<predicate>` publisher-metadata convention (hyphen form, CSV values) is ingested as typed edges by both crawlers; the /graph.md export is six columns with legacy 2-column parse kept; mark_graph/mark_backlinks/mark_explore render provenance via the shared `graph.EdgeAnnotation`. Zero server changes. The fetch-callback debt item went with it (removed from /debt.md on merge). Review hardening from three CodeRabbit rounds: whitespace-strict rel refs, Merge replaces refreshed sources' outgoing edges so stale backlinks drop, BaseInline.Lines() panic guard for links inside inline formatting (lesson in /debugging.md), BlockStart provenance for empty-label links. Hubs pick up the enriched /graph.md once the federation agent redeploys with the new binary.

### Graph hub seeding: DONE (merged 2026-07-14, PR #253 commit 8ab14aa; follow-ups in PR #254 commit 8fe9b98)

The "Wire backlinks/graph tools to the published /graph.md" item is shipped per [/plans/graph-hub-seed.md](/plans/graph-hub-seed.md). What shipped:

- graphstore `SeedFromExport` with the local-wins rule: authoritative = real fetched status AND non-zero CrawledAt; seeded nodes carry zero CrawledAt as the durable marker; the same rule arbitrates against Merge's replace-refreshed-sources logic (shared `dropEdgesFromLocked`/`upsertEdgeLocked` helpers plus the `observedStatus` predicate, tested from both directions).
- Seed etags persist in graph.json as an additive `seed_etags` map (host to etag); schema stays v1, legacy files load with a nil map.
- `fetch.FetchConditional(host, path, token, etag)` for explicit if-none-match (token'd souls skip the disk cache, so the seeder tracks freshness itself). First wire-level test of fetch.Client via an in-process QUIC server.
- demarkus-mcp: `seedGraph(host)` in mark_backlinks, mark_graph, and explore's backlinks section, seeding the resolved host from the tool URL (review round: originally defaultHost only); 5-minute per-host process-scoped throttle; never fatal (warn-log and degrade).
- Broker: `seedWorldGraph` through dispatchWithAuth in the same three handlers, per world, so cold pods answer backlinks; `FetchConditional` added to worldDispatcher/worldPool (trivial passthrough); MCP-API.md updated.
- Bonus fix required by the live check: the MCP backlinks/graph URL keys are now canonicalized through resolveURL (default port applied). Without it the seeded aggregate's mark://host:6309 rows never matched queries built from a portless -host flag; regression test pins it.
- ParseExport got its first production caller and the live soul /graph.md (still two-column legacy, v6) seeded correctly.
- PR 253 review also closed a SeedFromExport stale-edge gap: the drop set includes observed seed nodes, so a source whose outgoing set went to zero sheds its stale seeded edges.

Verified live: scratch-HOME cold start against the soul answered 4 backlinks for /patterns.md with zero crawls (133 nodes / 190 edges seeded, etag persisted); a depth-1 crawl of /conventions.md then flipped it authoritative and its enriched edges replaced the seeded rows while 130 nodes stayed seeded. TUI seeding and Exported-timestamp arbitration remain out of scope.

Follow-up shipped same day (PR #254, closes issue #222): Merge no longer clobbers a stored node when a re-crawl fails to read it (unobserved status), so titles converge across flaky crawls instead of regressing, and a stalled federation-agent crawl can no longer publish blanked titles into hub /graph.md for seeding to amplify. Two review findings declined with reasoning on the PR: preserving titles on ok-with-blank fetches (blank is affirmative "no H1") and zeroing CrawledAt on never-seen error nodes (observedStatus already gates authoritative). Released as client v0.21.1.


Same-day broker follow-ups, both found by post-deploy verification and closed (PR #256, PR #257): seeded rows are translated from world dial addresses to mark://{worldName}/... form (the hub aggregate keys rows by cluster-internal DNS, which no legal tool URL can address, so seeding was inert in the real topology), and graph calls seed every configured world instead of only the queried one (the aggregate lives only on the hub). A producer-consumer contract test now pins the agent's real export as a golden that the broker seed suite consumes, so either side drifting fails a build; lesson recorded in /debugging.md. Deployed and verified live on knowledge.demarkus.io (broker 0.12.4): the first graph call on a cold pod answers non-hub backlinks from the hub aggregate with zero crawls.


Two more consumers closed on 2026-07-15. demarkus-library PR 62 (repo latebit-io/demarkus-library, released 0.21.2 and deployed): the floor's hand-rolled /graph.md parser classified tables by column count and got zero edges plus phantom nodes from the enriched format; it now consumes graphstore.ParseExport behind a hexagonal GraphExportParser port returning domain types, with the export-format fixtures living only in the adapter tests. And monorepo #259 (client 0.21.3, agent deployed): fedcrawl falls back to the H1 for node titles when publisher metadata declares none, so soul-style docs stop publishing nameless into the hub graph; precedence pinned by test and silently by the contract golden. Live-verified: /graph.md v766 carries titles on all 147 soul rows.

### Graph completeness analysis (2026-07-15)

Post-enhancement code survey recorded at [/plans/graph-completeness.md](/plans/graph-completeness.md). New Tier 1 findings beyond the gap list above: the 1 MiB /graph.md export wall with silent publish failure (no sharding), tombstone accumulation with no graphstore eviction path, Exported-timestamp arbitration confirmed unimplemented (header written, never parsed), and inconsistent URL canonicalization between crawlers. Suggested order in the plan doc: sharding + loud failure, eviction, timestamp arbitration, shared canonicalization, rel query filter + predicate vocabulary, in-degree into LOOKUP.

## Remote Token Issuance: DONE (added 2026-07-15, closed 2026-07-17)

See [plan](/plans/remote-token-issuance.md). Two onboarding tiers shipped; invites and the capability front door both dropped as unnecessary.

**Step 1 DONE (join URLs, PR #261):** join URL `mark://host#token=...` as the one paste-able onboarding string; `demarkus-token join`; `demarkus join` CLI subcommand; install.sh ready-to-paste `/soul-join` line; `/soul-join` in claude-code 0.13.0 + pi-memory 0.13.0. TOFU cert pinning built then cut (see plan pinning decision).

**Step 2 DONE (broker single-host mode, PR #262 + #263):** the broker runs on a plain VPS in file-backend mode (SecretStore interface, tokens.toml writer that the server hot-reloads), so `/knowledge-join`, library SSO, and per-world authorization work without Kubernetes. `install.sh --with-broker --with-library` with hardened systemd units; `docs/site/deployment/single-host.md`. Verified live end to end (broker-signed bearer drives the gateway; `mark_publish` provisions a write token into the local tokens.toml and the doc lands; the library renders broker-written docs).

**Invites DROPPED, capability front door DECLINED (2026-07-17):** with the broker in single-host mode, invites are redundant, because the broker is OIDC-fronted by construction, so any broker deployment onboards via `/knowledge-join` plus an existing IdP login, which beats an anonymous invite code on identity, revocation, and secret-in-transit. The one remaining case (multi-user browser editing without any IdP) is declined: the library's cataloging desk (browser editor, shipped in library v0.22.0) writes only through broker mode, the broker needs OIDC, and OIDC means any provider including a self-hosted one (Dex/Keycloak/Authentik), so the ecosystem already covers the no-commercial-IdP case and a bespoke broker auth surface would be a worse Dex. Final tier map: read anywhere (no IdP); write via CLI/agent/Obsidian anywhere (no IdP, join URL); browser editing needs an OIDC provider (self-hosted fine). The earlier in-server options (A, B) and the sidecar framing are recorded in the plan as superseded.

## Five-Minute Appliance & Install Hardening: DONE (2026-07-18 → 07-22, PRs #267, #269–#272)

The single-host stack became a one-command appliance, then a week of hardening driven by the soul.demarkus.io droplet migration (journal 2026-07-22). Journals 2026-07-18 through 07-22 carry the detail.

- **Agent release asset (PR #267):** `demarkus-agent` ships as a release tarball so the appliance can install the indexing agent without building from source.
- **`install-stack.sh` appliance (PR #269):** orchestrator that runs `install.sh --with-broker --with-library` then layers Authelia (all secrets generated by the authelia binary itself, mandatory claims_policy), Caddy (path-split reverse proxy owning 80/443, `caddy validate` gate), broker OIDC wiring with an abort-if-any-placeholder-survives guard, sslip.io default hostname, and a final card with library URL, owner login, and the `/knowledge-join` line. Split out of install.sh by decision (precedent: install-readonly.sh); install.sh stays standalone, its uninstall extended to tear down all flavors. Verified live in a debian bookworm container.
- **Library on HTTPS :443 (PR #270):** on TLS installs the library defaults to 443 sharing the world's cert (one hostname, one identity); `--library-port` override; kept `/etc/demarkus-library/env` is authoritative and validated (`validate_port` on kept values too); fail-closed `ss` port preflight keyed on the unit's MainPID, not process names; conditional `CAP_NET_BIND_SERVICE`; cert-group access and renewal hook follow the effective values. Three CodeRabbit rounds; 41-assertion harness.
- **Self-dial probe (PR #271):** after install with `--with-broker`/`--with-library` on a domain, `ensure_self_dial` probes `mark://<domain>/health`; on hairpin-NAT failure it pins the domain to loopback in /etc/hosts (exact-token idempotent) and re-probes; fail-closed diagnostics otherwise. Long-term fix stays open in demarkus-library: decouple display name from dial address.
- **Cert renewal on re-run (PR #272):** a re-run of install.sh silently skipped the renewal-hook setup (gotcha found during the droplet migration); now reachable on re-run and timer-safe, with a CI-run renewal test harness.
- **Ops:** soul.demarkus.io migrated off the home Orange Pi onto a DigitalOcean droplet with full version history intact (2026-07-22); migration gotchas and droplet hardening recorded in journal 2026-07-22 and /debugging.md.

## Markdown-Only Content Contract: DONE (merged 2026-07-23, PR #273)

Implementation caught up with SPEC 2.3 ("content is markdown"): before this, a PNG published to a server round-tripped byte-perfect and every markdown-derived feature degraded silently, worst of all MCP outline mode injecting raw binary as mojibake into an agent's context. Full analysis and layer-split reasoning in journal 2026-07-23; architecture decision recorded in /architecture.md.

- **Handler gate (protocol surface):** PUBLISH/APPEND path must end in lowercase `.md` and body must be valid UTF-8, else `bad-request` with no version created; pure `store.ValidateDocumentContent` called from both write handlers (same one-choke pattern as `ValidateMeta`).
- **Store invariant (defense in depth):** `store.ValidateBody` (UTF-8 only) at both backends' write choke plus a `RejectsBinaryBody` conformance case; the `.md` rule deliberately stays out of the store, which is path-agnostic and whose traversal-safety tests rely on non-.md paths.
- **MCP render guard:** `mark_fetch`, `mark_explore`, and resource reads return a one-line notice for non-UTF-8 bodies instead of mojibake; `force=true` still pulls raw bytes. Raw protocol FETCH stays byte-faithful (refusing to serve would make any binary that reached disk permanently uninspectable).
- SPEC updates (4.4, 6.4, 6.6) and the §7 status table corrected: `bad-request` and `conflict` moved from "future" to active.


**Broker follow-up DONE (merged 2026-07-25, PR #277, closes issue #276).** #273 guarded only the local `demarkus-mcp`, leaving the broker gateway's render paths still able to put mojibake into agent context. The guard now lives once in `client/mdoutline` (`BinaryBody`, `NonMarkdownNotice`), shared by both MCP surfaces the same way `fetchdedup` and the outline helpers already were; the broker calls it at `handleMarkFetch` (before the anchor slice, `mode: binary`), `handleMarkExplore` (before the outline builder), and `readResource` (text/plain notice). As in #273, `force=true` bypasses the size gate but never the binary gate. Deployed to knowledge.demarkus.io as broker 0.14.5 and verified live (gateway fetch and explore render normally; the guard itself is now an unexercisable backstop, since worlds at server 0.22.0 refuse to accept binary in the first place).
## OpenCode Memory Plugin: DONE (merged 2026-08-10, PR #281)

`plugins/opencode-memory/` v0.13.8: the OpenCode port of demarkus-memory. Thin single-file TS adapter over the shared `demarkus-plugin` binary (pi pattern); config hook wires the MCP server + joined souls + slash commands, gates via `tool.execute.before/after`, guidance/recall via `chat.message`, journal nudge via `session.idle`. Installed by curl one-liner (no npm). Plan + follow-ups: [/plans/opencode-memory-plugin.md](/plans/opencode-memory-plugin.md).

## Store: make LIST and FETCH agree on flat files: NEXT (added 2026-08-11)

Root cause behind issue #288. The store's `ListDir` deliberately shows flat files (legacy compat, `store.go` dirHasVisibleEntry path) while `Get` refuses anything without version history, so a flat file LISTs but 404s on FETCH forever. Worse, `Write`'s `next==1` branch skips `migrateFlatFile` (migrations only run under `next > 1` via `prepareExistingDoc`), so publishing over a flat file silently clobbers it instead of promoting it to v1.

The fix: `Write` migrates a versionless flat current file to v1 before writing (extend the existing `migrateFlatFile` machinery to the `next==1` path). A create-only publish over a flat file then conflicts with the migrated v1 instead of destroying it, and any write to a flat path heals it into a served document. Decide separately whether LIST should keep showing never-written flat files or hide them.

Sequencing with the plugin fix (branch `fix/seed-via-publish`): the plugin's seedDoc adoption branch for `index.md` becomes removable once the server pin bumps past this fix. The `project-template.md` adoption stays regardless, because store migration only triggers on a write and nothing else ever writes that path.

Update (same day): the plugin branch dropped its adoption paths entirely rather than carrying transitional code. `cleanupLegacyTemplate` is local-only (pristine hash match → delete; customized flat template left on disk for store migration), and `seedDoc` publishes the embedded seed create-only. Consequence: the store fix must ship first and the plugin's `serverVersion` pin must bump past it before this plugin release, because on an unfixed server the create-only seed publish over a legacy flat `index.md` clobbers it (the `next==1` migrateFlatFile skip) instead of conflicting.

Resolution (2026-08-11, branch `fix/store-flat-file-migration`): Fritz ruled flat files are not documents, and the spec was amended instead of implementing adoption. SPEC 9.8 rewritten from "Flat File Migration" (publish MUST migrate flat to v1) to "Non-Document Files" (publish MUST NOT incorporate flat content; it creates v1 and replaces the file), SPEC 6.2/6.5/11.9 and DESIGN updated to match. Store change is LIST-side only: ListDir and the directory-visibility walks exclude versionless flat files in both views, and a directory whose subtree holds no documents is excluded even from the include-archived audit view. Legacy flat-layout docs (protocol-written, `versions/doc.md.v{N}`) remain visible and are unaffected. Write path needed no change; its replace behavior is now the specified behavior, which also removes the plugin PR's release-ordering constraint entirely.

Merged 2026-08-12 as PR #291 (9f1f062 on main): spec amendment, LIST exclusion of non-documents, eager legacy-layout migration in `store.Open`, legacy flat-layout code removed. The plugin PR #290 has no release-ordering dependency on it and remains open.

PR #290 merged 2026-08-12 (8571554); issue #288 resolved at both layers.


## Related documents

- [Five-minute appliance](/plans/five-minute-appliance.md): the appliance work this roadmap section records
- [OKF type adoption](/plans/okf-type-adoption.md): the type policy behind the OKF section
- [Postgres backend](/plans/postgres-backend.md): the second store backend in the knowledge layer
- [Version retention](/plans/version-retention.md): the retention work named in the backlog
- [Obsidian plugin plan](/plugins/obsidian/plan.md): the plugin released in the plugins section