Roadmap

Where demarkus has been and where it's going.

Scope: Markdown-Only Server

The demarkus server serves markdown only. Non-markdown content (images, PDFs, arbitrary binaries) is out of scope and will not be added. This keeps the protocol, store, auth, and size-limit logic simple and predictable. Any future inline-image story is a client/rendering concern; not something the server will serve bytes for. Since 2026-07-23 (PR #273) this is enforced at the protocol surface, not just asserted; see the Markdown-Only Content Contract section below.

Phase 1: MVP (Read-Only): COMPLETE

Everything shipped:

• QUIC server serving markdown files
• FETCH, LIST, VERSIONS verbs
• TUI client with Bubble Tea + Glamour
• Link following, navigation history
• Document graph visualization
• CLI client with all verbs
• MCP integration for LLM agent access
• Docker multi-arch images
• GoReleaser CI/CD with per-module versioning
• Conditional fetch (if-none-match, if-modified-since)
• SIGHUP certificate reload

Phase 2: Publish Operations: COMPLETE

Done:

• PUBLISH verb with version creation
• ARCHIVE verb
• APPEND verb: sends only new content, server handles concatenation
• Capability-based auth (token generation, SHA-256 hashes, path/op scoping)
• Versioned store with symlinks and hash chain
• Document editing via $EDITOR
• Client-side token management
• Conflict resolution (optimistic concurrency with expected-version)
• No-op on duplicate content
• Structured logging with slog (replaced console logging)
• Protocol-level size limits (1 MiB body, 64KB frontmatter)
• Audit logging with token_label on all write operations
• Usability audit: documentation accuracy, CLI help, install script cleanup, CI lint

Phase 2 is fully complete. No remaining items.

Phase 3: Agent-Native & Advanced Features: COMPLETE

Done:

• Agent manifest discovery (/.well-known/agent-manifest.md)
• Bookmarks/favorites
• MCP mark_append auto-resolve expected_version
• Content-addressed fetch
• Federation via MCP tools

Phase 4: The Information Graph: COMPLETE

Done:

• The Demarkus Hub pattern
• Persistent graph store
• Backlinks
• Graph as content (export/import)
• Graph-aware navigation (TUI)
• Agent discovery

Phase 5: The Demarkus Agent & Private Networks: IN PROGRESS

Read Auth for Private Networks: DONE

Per-path read token enforcement. Server-side and client-side shipped.

Security Hardening: DONE

See plan for full details.

• Systemd hardening: install script generates units with ProtectSystem=strict, ReadWritePaths, NoNewPrivileges, etc. Conditional ProtectHome. Update path detects insecure config and prompts to harden with rollback.
• -read-only mode: DEMARKUS_READ_ONLY env var / -read-only flag. Handler rejects PUBLISH/APPEND/ARCHIVE with not-permitted. Zero write access needed.
• demarkus-publish: CLI tool that writes directly to the versioned store on disk. Enables local publishing when server runs read-only. Shares store.Write() with the server.
• Read-only chroot install: install-readonly.sh. Chrooted server with ReadOnlyPaths=/, BindReadOnlyPaths=/dev/urandom. Maximum lockdown: Gemini-level security with full versioning.
• Security documentation: attack surface analysis, threat model, comparison table, hardening guide.

Core Loop: Crawl & Index

1. Seed: start from configured servers (hub, known peers)
2. Crawl: follow mark:// links, discover new servers and documents
3. Hash: collect content hashes from every document
4. Index: publish updated hash indexes to configured hubs
5. Repeat: on a configurable schedule, with conditional fetch (if-none-match) to be polite

Server-to-Server Sync

Rsync for the Mark Protocol. Replicate content between servers using content hashes as the diff mechanism.

How it works:

1. LIST both source and destination servers
2. Compare content hashes: skip documents that match
3. FETCH changed/new documents from source
4. PUBLISH to destination with the fetched content
5. Optionally handle deletions (ARCHIVE on destination for docs removed from source)

Sync modes:

• Mirror: destination becomes an exact copy of source (one-way)
• Selective: sync specific paths or glob patterns (e.g., /docs/* only)
• Multi-source: aggregate content from multiple servers into one destination

Key Design Points

• Go binary, not an LLM agent: mechanical work, not reasoning
• Polite crawling: conditional fetch, configurable rate limits
• Hub-aware: reads and updates hub index documents
• Daemon or cron: continuous or triggered
• Auth-aware: passes read and write tokens for private servers

Implementation Sketch

• New module: tools/demarkus-agent (the tools/ directory is the home for utility binaries; demarkus-token, demarkus-publish, and now sync/agent tooling)
• Reuses: fetch.Client, graphstore, client/internal/index, graph.Crawl
• CLI: demarkus-agent crawl, demarkus-agent sync source dest, demarkus-agent daemon

External Tool Sync: PLANNED

The shape that lets any markdown-emitting tool (OpenSPDD, ADR generators, doc generators, custom workflows) land its output on demarkus without integrating the protocol itself. The integration burden lives on demarkus's side; upstream tools change nothing.

demarkus-sync: single binary in tools/. Watches a local directory and PUBLISHes (or writes via demarkus-publish against a local store) on every changed .md. Versioning happens automatically through the existing store. Cross-team sharing emerges naturally from publishing to a team server.

Core flags:

• --watch <dir>: directory to watch
• --target mark://host/path/: remote demarkus server (uses fetch.Client PUBLISH)
• --store <local-path>: alternative: write directly to a local versioned store via store.Write
• --include / --exclude glob filters (e.g. **/*.canvas.md)
• --token / DEMARKUS_AUTH for write auth

Project-flow integration is the killer use: drop into a Makefile target, pre-commit hook, CI step, or long-running dev daemon. Design contracts, ADRs, RFCs, runbooks, agent canvases all flow into the team world automatically; no per-tool integration work.

OpenSPDD as first concrete validation: REASONS Canvas docs are pure markdown contracts written to a local directory. demarkus-sync makes them a versioned, federated demarkus surface without OpenSPDD knowing. Same pattern works for any tool that emits markdown.

The deeper framing: demarkus shouldn't ask other tools to integrate it. It should make any markdown-emitting tool's output queryable as a versioned protocol surface, transparently. demarkus-sync is the generic shape of that.

Reuses: fetch.Client for remote PUBLISH, store.Write for local-store writes, fsnotify for the watch loop.

Phase 6: Universe Deployment: PLANNED

See plan for full details.

Make it cheap to stand up N worlds and onboard real users without hand-distributing tokens. Phase 5 gave us a single hardened server; Phase 6 makes a universe of them deployable with one Helm chart, one ApplicationSet, and one OIDC-fronted token broker.

Sub-phases:

• 6.1: Helm chart (deploy/helm/demarkus-server/): StatefulSet + volumeClaimTemplates so each world owns its own PVC, Service, bootstrap Job that runs demarkus-token generate and writes to a Secret, optional read-only mode, SIGHUP reload wiring.
• 6.2: Universe topology (deploy/k8s/examples/): reference Argo CD ApplicationSet over a worlds: list, plus a Kustomize overlay for clusters without Argo. No new code.
• 6.3: Token broker (prototype in tools/demarkus-broker/, splits to latebit-io/demarkus-broker after 6.4): OIDC-fronted HTTP service that mints scoped tokens, writes hashes to world Secrets, SIGHUPs on revocation, audit log to stdout.
• 6.4: User install flow (lives with broker): GET /me/install returns a one-shot shell script that writes ~/.config/demarkus/auth and idempotently patches ~/.claude.json with a demarkus-mcp server entry per world.
• 6.5: Docs: /deployment.md for chart values + broker setup; plan doc maintained at /plans/universe-deployment.md.

Constraint: zero changes to protocol or core server. Phase 6 is packaging and lifecycle. Any need for a new core primitive must be discussed and justified before landing, same rule that applied to the Claude Code plugin.

Explicitly deferred to Phase 7+: multi-replica worlds with shared storage, cross-cluster universe federation, an operator with a World CRD, a web UI on top of the broker.

Single-host mode (shipped 2026-07-17, PR #262 + #263): the broker also runs on a plain VPS beside the server in file-backend mode, no Kubernetes; see the Remote Token Issuance section below and /plans/remote-token-issuance.md. This is the non-k8s delivery of the same broker; the k8s sub-phases above are unchanged.

Verb Set: Complete

7 verbs: FETCH, LIST, VERSIONS, PUBLISH, APPEND, ARCHIVE, LOOKUP.

LOOKUP (issue #113) is the catalog verb: subject lookup over an in-memory, importance-ranked catalog of declared tags + title. It is a token-efficient supplement to the index, not full-text search (which stays in an opt-in sidecar). See the LOOKUP Catalog Verb section below and /plans/lookup-verb.md.

LOOKUP Catalog Verb: DONE (merged to main 2026-05-30, PR #166; plugin tail closed by #168)

See plan. Tracks issue #113.

Given a subject, return which documents are about it and how important they are; a card-catalog supplement to the index hub so an agent finds the right doc without fetching and grepping the whole index. Catalog scan only: in-memory path → {tags, importance, title}, built on the startup walk and maintained inline on writes, ranked by match-count then importance. No body reads at query time, no relevance modeling. Full-text / semantic search is permanently out of core, in an opt-in sidecar.

Shipped:

• Protocol verb + spec §6.7.
• server/internal/catalog package + handler with read-auth result filtering.
• Client surfaces: fetch.Client.Lookup, demarkus lookup CLI, MCP mark_lookup.
• Broker parity: mark_lookup is the gateway's 14th tool.
• Publisher metadata: tags/importance settable at publish; CLI -meta key=value (repeatable), and a metadata object on MCP + broker mark_publish. This is what makes the tag/importance ranking actually usable (before it, LOOKUP was title-only with uniform 0.5 importance).
• End-to-end smoke test passed (publish with tags → lookup by tag → archive, over QUIC).
• Plugin surfacing (v0.3.0, #168, 2026-05-31): Claude Code plugin wires mark_lookup, fixes the knowledge-join tool-count text (13→14), and injects SessionStart guidance so sessions recall via lookup.

Remaining tail (small): mark_append metadata; deferred by design (you tag on publish, not append). The plugin knowledge-join/version-pin items are now shipped in v0.3.0.

Open Knowledge Format (OKF) Compatibility: DONE (merged 2026-06-22, PR #195)

demarkus interoperates with Google's Open Knowledge Format v0.1 (a bundle of markdown files with YAML frontmatter). The document content model is OKF-compatible; demarkus layers versioning, hash chain, QUIC transport, capability auth, and LOOKUP around it.

Shipped:

• Store metadata alignment: recognized OKF fields (type, title, description, resource, tags, timestamp) serialize as bare frontmatter; tags as a YAML flow list; non-spec keys keep the meta. prefix; reserved operational fields (version, previous-hash, archived) enforced by name. Back-compatible read of legacy meta.*. Caps raised to 50 keys / 1024 bytes (frontmatter budget 2048), tags counted at serialized length.
• demarkus okf codec (client/internal/okf); validate (v0.1 conformance), import (bundle → world, sanitize/cap with warnings, link rewrite, upsert), export (world subtree → conformant bundle, frontmatter reattach, type synthesis, RFC 3339 timestamp normalization). Verified live: import → export → validate round-trip, byte-identical bodies.
• Opinionated default: server assigns type: Document on PUBLISH and APPEND when none declared (reserved index.md/log.md exempt), so every served document is a typed OKF concept by construction. Non-breaking for existing typeless writers (ADR 0003).
• Spec §8.1/§9.4/§13/§14; ADR 0002 (metadata alignment), ADR 0003 (type default).
• Review hardening (PR #195 CodeRabbit): export path-traversal guard, import symlink rejection, LIST decode/cycle hardening, reserved-key read guard, cap re-validation, link query strip.

MCP Agent Ergonomics & Attach Surface: DONE (2026-07-04/05)

Two same-week phases that made MCP agents cheap to serve and demarkus pleasant to drive from Claude Code / Desktop. Plans: /plans/mcp-client-ergonomics.md and /plans/mcp-resources-prompts.md. Both fully shipped AND deployed on both MCP surfaces (local demarkus-mcp and the broker gateway).

Shipped:

• Size-adaptive mark_fetch: full body under 8KB; above it an outline (heading tree with GitHub-slug #anchors + per-section line counts + opening paragraph + hint). url#anchor slices one section at any size; force overrides. Killed the five-figure-token fetches of append-heavy docs (measured: a 52KB roadmap → 5.9KB outline).
• Session unchanged-dedup: re-fetch of an unchanged doc returns status: unchanged since vN (identity = version+etag; identity-flip edge cases spelled out). Process-scoped in the local client; MCP-session-keyed at the broker (multi-tenant safe: no session → no dedup; LRU eviction at cap, since mcp-go only unregisters sessions on explicit DELETE).
• mark_explore: one-call orientation card (outline head, opening paragraph, outbound links, backlinks, siblings; 10/section + "+N more").
• MCP resources: documents as client-attachable context, zero tool turns: mark:// URI templates, #anchor section attach, well-known + background-listed picker entries (client) / per-world index hubs (broker).
• MCP prompts: orient / recall / whats-new as server-vended slash commands; the broker variants sweep worlds via mark_worlds.
• Shared packages client/mdoutline (outline/slice + render helpers) and client/fetchdedup (dedup identity + notice text); one implementation for both surfaces (the changedNote mirror drifted twice in review before hoisting; lesson in /debugging.md).
• One #section grammar everywhere: mark_fetch, mark_explore, MCP resource URIs, and the demarkus-library librarian's open tool (adopted 2026-07-05) all share the same anchors from the same implementation.
• Deployed: plugin users via client v0.15.0→v0.17.0 (tools 0.4.1+, pin chain now self-driving after the #228 pin-bump automation fix); knowledge.demarkus.io via broker 0.5.0 (ergonomics) and 0.6.0 (resources+prompts).

Remaining: none. Follow-ups all landed (broker parity, resources/prompts, librarian adoption).

Distribution & Package Management: PLANNED

Homebrew tap for easier installation on macOS and Linux.

Build Targets

Supported platforms: macOS, Linux, Windows (WSL only).

Plugins: IN PROGRESS

Obsidian Plugin: v0.1.0 RELEASED

Standalone repo: latebit-io/obsidian-demarkus. Source in plugins/obsidian/.

Claude Code Plugin: v0.13.5 (current)

Merged into main via PR #96 (commit af8e210) on 2026-04-23 as v0.1.0. Source at plugins/claude-code/ in the monorepo; marketplace manifest at .claude-plugin/marketplace.json.

See plan for the full design. One-click marketplace install that lazy-spawns a local demarkus-server, auto-generates a token, and wires demarkus-mcp into the agent. Zero config on install. Includes /soul, /soul-journal, /soul-init, /soul-join, /soul-default, /knowledge-join slash commands and a soul-memory skill.

Shipped with a project-centric soul schema: /index.md is a project list, each project under /<slug>/ holds plan/tasks.md, architecture.md, patterns.md, roadmap.md, adr/, journal/<YYYY-MM-DD>.md.

Version history:

• v0.1.0 (2026-04-23, #96): initial marketplace plugin.
• v0.2.0 (2026-05-23, #152): /knowledge-join command for organizational broker-fronted knowledge systems.
• v0.3.0 (2026-05-31, #168): wires mark_lookup; injects standing SessionStart guidance (context/session-guidance.md) so sessions self-document to the soul and recall via lookup; fixes knowledge-join tool count (13→14); bumps binary pins to SERVER 0.17.13 / CLIENT 0.12.38 / TOOLS 0.1.28.
• v0.4.0–v0.9.x: not enumerated here (promote bridge, soul-refresh, binary-pin automation, OKF release pin, soul catalog groundwork). See the journals for detail.
• v0.10.0 / v0.10.1 (#200, #201): /soul-join: managed remote souls. Joins a direct-QUIC remote soul (token in a 0600 file via a launch wrapper, never inline in MCP config), records it in a catalog (~/.demarkus/souls) with a per-project binding (~/.demarkus/project-souls), and enforces the binding with a destination gate (PreToolUse on mark_publish/mark_append; a write to a non-bound soul is denied/asked/warned). Publish tag-gate extended to joined remote souls. 0.10.1 closed a reuse-server restart bug (never kill a healthy externally-managed server on binary upgrade).
• v0.11.0 (#202, 2026-06-22): /soul-default: set a project's default write target. Standalone command to re-point an already-joined repo at a different default without re-running /soul-join (until now the binding was only ever written as a --bind side-effect of join). Lists the joined-soul catalog (local managed soul + remotes), picks one, saves the binding the dest-gate enforces. Routing stays model-driven: the catalog is the discovery surface and each soul is its own MCP server, so no router or dispatch tool; reads and one-off writes to another joined soul go direct to its mcp__<slug>__mark_* tools (the gate fires only on publish/append). New lib.sh helpers (local_soul_present, soul_catalog, is_catalog_soul); no core/protocol change.
• v0.12.x (late June → 2026-07-05): pi-agent port fixes (#217), the ergonomics release-chain repins (tools 0.3.5 → 0.4.1, #221/#227), and the automated pin-bump sweep (#229) after the pin-bump workflow's closed-PR upsert bug was fixed (#228: the pin chain is self-driving again: release → auto bump PR → merge).
• v0.13.0 (2026-07-17, #261): join URLs. /soul-join accepts mark://host#token=... (one paste-able string from install.sh or demarkus-token join); new demarkus join CLI subcommand; pi-memory 0.13.0 ships the same command update.
• v0.13.x pins (2026-07-18 → 07-24, #264/#266/#268/#274/#275): automated pin-bump chain keeping managed binaries current (the release.yml dispatch fix in #265 revived the dead release trigger). Current: demarkus-memory 0.13.5, demarkus-knowledge 0.5.21, tools 0.14.3.

No core code changes landed. The plugin is built entirely on existing primitives: DEMARKUS_AUTH env var for token injection, /health endpoint plus ALPN "mark" negotiation for port probing, and shell redirection of demarkus-token generate stdout for writing the token to a file. Any future need for a new core primitive must be discussed and justified before landing.

TUI Polish

External Links: PLANNED (implementation ready to commit)

See plan. Closes issue #75 in full. Opens http/https/gemini/mailto links in the user's default handler via open/xdg-open/rundll32. Scheme allowlist, URL passed as argv only, no shell.

The original plan had a Phase 2 for same-server non-markdown files (images); dropped because the server is markdown-only and will not serve binaries. Inline image rendering, if ever pursued, is a separate client-rendering initiative (terminal graphics protocols like Kitty/Sixel), not a continuation of this work.

Features Not Prioritized: Backlog

• WebSub-style Subscriptions (removed from spec)
• Offline Mode (deferred)
• Full-Text / Semantic Search (out of core: opt-in sidecar reading via LIST/FETCH; in-core LOOKUP covers catalog-level subject lookup, see above)
• Diff / Changelog Between Versions (noted, not implemented)
• Blind Append with Content Deduplication (rejected)
• Non-markdown content types on the server (out of scope by design)
• TOFU cert-fingerprint pinning in join URLs (built on branch join-string, cut before merge; see the pinning decision in /plans/remote-token-issuance.md; any revival rides a redeem handshake)

Operational Polish: Backlog

• Plain-directory root returns silent not-found; the server only serves the versioned-symlink store layout. Pointing -root at a directory of plain .md files (e.g. docs/site) silently returns not-found for every path and builds a 0-entry hash index and catalog, with no startup warning. Bit setup during the 2026-06-02 load-test work. Flagged to revisit: at minimum a startup warning when the root has markdown files but no versions/ layout; ideally a clear hint.

• tools/demarkus-loadtest: concurrent read-only load generator (FETCH/LIST/ VERSIONS/LOOKUP) for capacity measurement. Reuses fetch.Client; warm (reused-conn) vs -fresh (handshake-per-request) regimes. Report and first measured numbers in tools/demarkus-loadtest/REPORT.md. See journal 2026-06-02. Key finding: server self-throttles per IP (default 50 req/s, burst 100, DEMARKUS_RATE_LIMIT=0 disables); warm ceiling ~7.7k req/s, cold ~1.4k req/s on one core.

Docs Hygiene

• SPEC.md markdownlint cleanup: DONE (PR #196, merged 2026-06-22). Added repo-wide .markdownlint.json disabling MD013 (line-length) and MD060 (table-column-style): both pervasive, intentional spec style. Fixed the substantive rules in docs/SPEC.md: fence languages (MD040, 29 fences → text), blank lines around fences/lists (MD031/MD032), bare URLs (MD034), and the emphasis footer (MD036). markdownlint-cli2 docs/SPEC.md → 0 errors; formatting only, no content changes.

• Single-host deployment page: DONE (2026-07-17). docs/site/deployment/single-host.md covers the server + broker + library stack on one VPS, the auth-tier table (read and tool-write need no IdP; browser editing needs OIDC, self-hosted Dex/Keycloak/Authentik included), OIDC app setup, library SSO wiring, and uninstall.

• Website brought current: DONE (2026-07-23, pages branch). The site had not moved since 2026-06-22; everything from the last month was missing. Added the five-minute appliance, the library and its librarian, the indexing agent, promote/soul-join, retention, MCP resources and prompts. Framing decision and the library-screenshots maintenance note are in journal 2026-07-23.

Knowledge Layer & Storage Backends: NEXT (added 2026-07-13)

Wire backlinks/graph tools to the published /graph.md: DONE (see Graph hub seeding below)

The federation agent already publishes the aggregated link graph to each hub at /graph.md (hourly, cross-server edges, same format as mark_graph_export), but mark_backlinks and mark_graph answered only from the client's local ~/.mark/graph.json crawl cache. A fresh client answered from an empty graph while an authoritative aggregate sat on the hub unread.

• Seed/refresh the MCP graphstore from the hub's /graph.md (etag-cached fetch, merge into local store) before falling back to local crawl.
• Fixes cold start and cross-agent consistency with a client-only change; days of work, no server change.
• Later upgrade, not the fix: publish-time edge extraction in the store (instant, transactional backlinks). Subject to the backend-parity principle below.

Principle: backend parity is system-level, not implementation-level

With two store backends (file, postgres), behavioral parity is a property of the system, not a detail of either implementation. The storetest conformance suite is the contract; both backends run it unchanged and divergence is a protocol bug.

Consequences for future work:

• Any new store-observable behavior lands in the conformance suite first, then in every backend, or it does not ship.
• Knowledge-layer features should prefer layers above the store (published documents like /graph.md, broker aggregation, client merge) which are backend-independent by construction.
• A feature that only one backend can provide efficiently (e.g. a transactional edges table in postgres) must still be defined system-level: either every backend implements the same observable surface, or the capability is exposed as an optional, discoverable feature rather than silently backend-dependent.

Remaining gaps from the 2026-07-13 knowledge-layer analysis

The /graph.md wiring item above came out of a gap analysis (knowledge-system-deploy session, 2026-07-13): demarkus is strong at the document layer (versioning, integrity, provenance, gates) and weak at the knowledge layer. Only gap 1 got recorded at the time; the rest are recorded here so they are not lost. Framing correction from that session: the plugins are the knowledge layer (recall-first, tag-at-publish, promote, doctors), so the governing principle is that the agent owns judgment while the server owns the accumulation of that judgment (edges, indexes, derived rank live in shared storage so the layer compounds across agents instead of being recomputed per client).

Edge semantics: give edges more information

Every link in the graph is an untyped "mentions". Knowledge graphs live on predicates: supersedes, implements, depends-on, derived-from. OKF types the nodes; nothing types the edges (ADR 0002 superseding its v1 draft was expressible only as prose). Edges also carry no provenance: graph.Edge/StoredEdge are bare from/to pairs and /graph.md exports a two-column table, so backlinks cannot say why or where a doc links (no link text, no source anchor, no occurrence count).

Shape: a rel: metadata axis (e.g. rel:supersedes=/adr/0002-....md) surfaced in mark_graph/mark_backlinks, plus richer edge records in the export (label, source anchor, count). This is a plugin and export-format convention, zero server change, so it can land before publish-time edge extraction; when the store later learns edge extraction it must capture the same fields (backend-parity principle above).

Semantic recall beyond tags

Lookup recall depends entirely on declared tags and titles; recall bounded by metadata hygiene loses to grep, and the agent compensates by exploring and fetching more every session (judgment tokens spent on enumeration). Path: Postgres tsvector full-text over bodies (cheap, phase-2 adjacent), then pgvector embeddings, hybrid-ranked with declared importance. Indexes amortize the agent's search loop into infrastructure; they do not replace its judgment.

Universe-wide LOOKUP

Lookup is scoped to one world; the broker routes but does not aggregate, so "search the whole system" is client-side fan-out. A broker-level aggregated LOOKUP (and eventually graph) makes a multi-world system feel like one system. Broker feature, no server change.

Derived ranking signals

importance is entirely self-declared, so a well-tagged orphan outranks the hub everything cites. Blend link-graph signals (in-degree, centrality) with declared importance. Link structure is not usage telemetry, so this respects the no-tracking principle.

Staleness at lookup time

modified exists but nothing decays or flags aging content at query time; the doctors catch it offline only.

Edge semantics: DONE (merged 2026-07-14, PR #251, commit 8798d0a)

The "Edge semantics: give edges more information" gap above is shipped; see /plans/edge-semantics.md and repo ADR 0004. Edges carry {From, To, Rel, Label, Anchor, Count} with identity {From, To, Rel}; the rel-<predicate> publisher-metadata convention (hyphen form, CSV values) is ingested as typed edges by both crawlers; the /graph.md export is six columns with legacy 2-column parse kept; mark_graph/mark_backlinks/mark_explore render provenance via the shared graph.EdgeAnnotation. Zero server changes. The fetch-callback debt item went with it (removed from /debt.md on merge). Review hardening from three CodeRabbit rounds: whitespace-strict rel refs, Merge replaces refreshed sources' outgoing edges so stale backlinks drop, BaseInline.Lines() panic guard for links inside inline formatting (lesson in /debugging.md), BlockStart provenance for empty-label links. Hubs pick up the enriched /graph.md once the federation agent redeploys with the new binary.

Graph hub seeding: DONE (merged 2026-07-14, PR #253 commit 8ab14aa; follow-ups in PR #254 commit 8fe9b98)

The "Wire backlinks/graph tools to the published /graph.md" item is shipped per /plans/graph-hub-seed.md. What shipped:

• graphstore SeedFromExport with the local-wins rule: authoritative = real fetched status AND non-zero CrawledAt; seeded nodes carry zero CrawledAt as the durable marker; the same rule arbitrates against Merge's replace-refreshed-sources logic (shared dropEdgesFromLocked/upsertEdgeLocked helpers plus the observedStatus predicate, tested from both directions).
• Seed etags persist in graph.json as an additive seed_etags map (host to etag); schema stays v1, legacy files load with a nil map.
• fetch.FetchConditional(host, path, token, etag) for explicit if-none-match (token'd souls skip the disk cache, so the seeder tracks freshness itself). First wire-level test of fetch.Client via an in-process QUIC server.
• demarkus-mcp: seedGraph(host) in mark_backlinks, mark_graph, and explore's backlinks section, seeding the resolved host from the tool URL (review round: originally defaultHost only); 5-minute per-host process-scoped throttle; never fatal (warn-log and degrade).
• Broker: seedWorldGraph through dispatchWithAuth in the same three handlers, per world, so cold pods answer backlinks; FetchConditional added to worldDispatcher/worldPool (trivial passthrough); MCP-API.md updated.
• Bonus fix required by the live check: the MCP backlinks/graph URL keys are now canonicalized through resolveURL (default port applied). Without it the seeded aggregate's mark://host:6309 rows never matched queries built from a portless -host flag; regression test pins it.
• ParseExport got its first production caller and the live soul /graph.md (still two-column legacy, v6) seeded correctly.
• PR 253 review also closed a SeedFromExport stale-edge gap: the drop set includes observed seed nodes, so a source whose outgoing set went to zero sheds its stale seeded edges.

Verified live: scratch-HOME cold start against the soul answered 4 backlinks for /patterns.md with zero crawls (133 nodes / 190 edges seeded, etag persisted); a depth-1 crawl of /conventions.md then flipped it authoritative and its enriched edges replaced the seeded rows while 130 nodes stayed seeded. TUI seeding and Exported-timestamp arbitration remain out of scope.

Follow-up shipped same day (PR #254, closes issue #222): Merge no longer clobbers a stored node when a re-crawl fails to read it (unobserved status), so titles converge across flaky crawls instead of regressing, and a stalled federation-agent crawl can no longer publish blanked titles into hub /graph.md for seeding to amplify. Two review findings declined with reasoning on the PR: preserving titles on ok-with-blank fetches (blank is affirmative "no H1") and zeroing CrawledAt on never-seen error nodes (observedStatus already gates authoritative). Released as client v0.21.1.

Same-day broker follow-ups, both found by post-deploy verification and closed (PR #256, PR #257): seeded rows are translated from world dial addresses to mark://{worldName}/... form (the hub aggregate keys rows by cluster-internal DNS, which no legal tool URL can address, so seeding was inert in the real topology), and graph calls seed every configured world instead of only the queried one (the aggregate lives only on the hub). A producer-consumer contract test now pins the agent's real export as a golden that the broker seed suite consumes, so either side drifting fails a build; lesson recorded in /debugging.md. Deployed and verified live on knowledge.demarkus.io (broker 0.12.4): the first graph call on a cold pod answers non-hub backlinks from the hub aggregate with zero crawls.

Two more consumers closed on 2026-07-15. demarkus-library PR 62 (repo latebit-io/demarkus-library, released 0.21.2 and deployed): the floor's hand-rolled /graph.md parser classified tables by column count and got zero edges plus phantom nodes from the enriched format; it now consumes graphstore.ParseExport behind a hexagonal GraphExportParser port returning domain types, with the export-format fixtures living only in the adapter tests. And monorepo #259 (client 0.21.3, agent deployed): fedcrawl falls back to the H1 for node titles when publisher metadata declares none, so soul-style docs stop publishing nameless into the hub graph; precedence pinned by test and silently by the contract golden. Live-verified: /graph.md v766 carries titles on all 147 soul rows.

Graph completeness analysis (2026-07-15)

Post-enhancement code survey recorded at /plans/graph-completeness.md. New Tier 1 findings beyond the gap list above: the 1 MiB /graph.md export wall with silent publish failure (no sharding), tombstone accumulation with no graphstore eviction path, Exported-timestamp arbitration confirmed unimplemented (header written, never parsed), and inconsistent URL canonicalization between crawlers. Suggested order in the plan doc: sharding + loud failure, eviction, timestamp arbitration, shared canonicalization, rel query filter + predicate vocabulary, in-degree into LOOKUP.

Remote Token Issuance: DONE (added 2026-07-15, closed 2026-07-17)

See plan. Two onboarding tiers shipped; invites and the capability front door both dropped as unnecessary.

Step 1 DONE (join URLs, PR #261): join URL mark://host#token=... as the one paste-able onboarding string; demarkus-token join; demarkus join CLI subcommand; install.sh ready-to-paste /soul-join line; /soul-join in claude-code 0.13.0 + pi-memory 0.13.0. TOFU cert pinning built then cut (see plan pinning decision).

Step 2 DONE (broker single-host mode, PR #262 + #263): the broker runs on a plain VPS in file-backend mode (SecretStore interface, tokens.toml writer that the server hot-reloads), so /knowledge-join, library SSO, and per-world authorization work without Kubernetes. install.sh --with-broker --with-library with hardened systemd units; docs/site/deployment/single-host.md. Verified live end to end (broker-signed bearer drives the gateway; mark_publish provisions a write token into the local tokens.toml and the doc lands; the library renders broker-written docs).

Invites DROPPED, capability front door DECLINED (2026-07-17): with the broker in single-host mode, invites are redundant, because the broker is OIDC-fronted by construction, so any broker deployment onboards via /knowledge-join plus an existing IdP login, which beats an anonymous invite code on identity, revocation, and secret-in-transit. The one remaining case (multi-user browser editing without any IdP) is declined: the library's cataloging desk (browser editor, shipped in library v0.22.0) writes only through broker mode, the broker needs OIDC, and OIDC means any provider including a self-hosted one (Dex/Keycloak/Authentik), so the ecosystem already covers the no-commercial-IdP case and a bespoke broker auth surface would be a worse Dex. Final tier map: read anywhere (no IdP); write via CLI/agent/Obsidian anywhere (no IdP, join URL); browser editing needs an OIDC provider (self-hosted fine). The earlier in-server options (A, B) and the sidecar framing are recorded in the plan as superseded.

Five-Minute Appliance & Install Hardening: DONE (2026-07-18 → 07-22, PRs #267, #269–#272)

The single-host stack became a one-command appliance, then a week of hardening driven by the soul.demarkus.io droplet migration (journal 2026-07-22). Journals 2026-07-18 through 07-22 carry the detail.

• Agent release asset (PR #267): demarkus-agent ships as a release tarball so the appliance can install the indexing agent without building from source.
• install-stack.sh appliance (PR #269): orchestrator that runs install.sh --with-broker --with-library then layers Authelia (all secrets generated by the authelia binary itself, mandatory claims_policy), Caddy (path-split reverse proxy owning 80/443, caddy validate gate), broker OIDC wiring with an abort-if-any-placeholder-survives guard, sslip.io default hostname, and a final card with library URL, owner login, and the /knowledge-join line. Split out of install.sh by decision (precedent: install-readonly.sh); install.sh stays standalone, its uninstall extended to tear down all flavors. Verified live in a debian bookworm container.
• Library on HTTPS :443 (PR #270): on TLS installs the library defaults to 443 sharing the world's cert (one hostname, one identity); --library-port override; kept /etc/demarkus-library/env is authoritative and validated (validate_port on kept values too); fail-closed ss port preflight keyed on the unit's MainPID, not process names; conditional CAP_NET_BIND_SERVICE; cert-group access and renewal hook follow the effective values. Three CodeRabbit rounds; 41-assertion harness.
• Self-dial probe (PR #271): after install with --with-broker/--with-library on a domain, ensure_self_dial probes mark://<domain>/health; on hairpin-NAT failure it pins the domain to loopback in /etc/hosts (exact-token idempotent) and re-probes; fail-closed diagnostics otherwise. Long-term fix stays open in demarkus-library: decouple display name from dial address.
• Cert renewal on re-run (PR #272): a re-run of install.sh silently skipped the renewal-hook setup (gotcha found during the droplet migration); now reachable on re-run and timer-safe, with a CI-run renewal test harness.
• Ops: soul.demarkus.io migrated off the home Orange Pi onto a DigitalOcean droplet with full version history intact (2026-07-22); migration gotchas and droplet hardening recorded in journal 2026-07-22 and /debugging.md.

Markdown-Only Content Contract: DONE (merged 2026-07-23, PR #273)

Implementation caught up with SPEC 2.3 ("content is markdown"): before this, a PNG published to a server round-tripped byte-perfect and every markdown-derived feature degraded silently, worst of all MCP outline mode injecting raw binary as mojibake into an agent's context. Full analysis and layer-split reasoning in journal 2026-07-23; architecture decision recorded in /architecture.md.

• Handler gate (protocol surface): PUBLISH/APPEND path must end in lowercase .md and body must be valid UTF-8, else bad-request with no version created; pure store.ValidateDocumentContent called from both write handlers (same one-choke pattern as ValidateMeta).
• Store invariant (defense in depth): store.ValidateBody (UTF-8 only) at both backends' write choke plus a RejectsBinaryBody conformance case; the .md rule deliberately stays out of the store, which is path-agnostic and whose traversal-safety tests rely on non-.md paths.
• MCP render guard: mark_fetch, mark_explore, and resource reads return a one-line notice for non-UTF-8 bodies instead of mojibake; force=true still pulls raw bytes. Raw protocol FETCH stays byte-faithful (refusing to serve would make any binary that reached disk permanently uninspectable).
• SPEC updates (4.4, 6.4, 6.6) and the §7 status table corrected: bad-request and conflict moved from "future" to active.

Broker follow-up DONE (merged 2026-07-25, PR #277, closes issue #276). #273 guarded only the local demarkus-mcp, leaving the broker gateway's render paths still able to put mojibake into agent context. The guard now lives once in client/mdoutline (BinaryBody, NonMarkdownNotice), shared by both MCP surfaces the same way fetchdedup and the outline helpers already were; the broker calls it at handleMarkFetch (before the anchor slice, mode: binary), handleMarkExplore (before the outline builder), and readResource (text/plain notice). As in #273, force=true bypasses the size gate but never the binary gate. Deployed to knowledge.demarkus.io as broker 0.14.5 and verified live (gateway fetch and explore render normally; the guard itself is now an unexercisable backstop, since worlds at server 0.22.0 refuse to accept binary in the first place).

OpenCode Memory Plugin: DONE (merged 2026-08-10, PR #281)

plugins/opencode-memory/ v0.13.8: the OpenCode port of demarkus-memory. Thin single-file TS adapter over the shared demarkus-plugin binary (pi pattern); config hook wires the MCP server + joined souls + slash commands, gates via tool.execute.before/after, guidance/recall via chat.message, journal nudge via session.idle. Installed by curl one-liner (no npm). Plan + follow-ups: /plans/opencode-memory-plugin.md.

Store: make LIST and FETCH agree on flat files: NEXT (added 2026-08-11)

Root cause behind issue #288. The store's ListDir deliberately shows flat files (legacy compat, store.go dirHasVisibleEntry path) while Get refuses anything without version history, so a flat file LISTs but 404s on FETCH forever. Worse, Write's next==1 branch skips migrateFlatFile (migrations only run under next > 1 via prepareExistingDoc), so publishing over a flat file silently clobbers it instead of promoting it to v1.

The fix: Write migrates a versionless flat current file to v1 before writing (extend the existing migrateFlatFile machinery to the next==1 path). A create-only publish over a flat file then conflicts with the migrated v1 instead of destroying it, and any write to a flat path heals it into a served document. Decide separately whether LIST should keep showing never-written flat files or hide them.

Sequencing with the plugin fix (branch fix/seed-via-publish): the plugin's seedDoc adoption branch for index.md becomes removable once the server pin bumps past this fix. The project-template.md adoption stays regardless, because store migration only triggers on a write and nothing else ever writes that path.

Update (same day): the plugin branch dropped its adoption paths entirely rather than carrying transitional code. cleanupLegacyTemplate is local-only (pristine hash match → delete; customized flat template left on disk for store migration), and seedDoc publishes the embedded seed create-only. Consequence: the store fix must ship first and the plugin's serverVersion pin must bump past it before this plugin release, because on an unfixed server the create-only seed publish over a legacy flat index.md clobbers it (the next==1 migrateFlatFile skip) instead of conflicting.

Resolution (2026-08-11, branch fix/store-flat-file-migration): Fritz ruled flat files are not documents, and the spec was amended instead of implementing adoption. SPEC 9.8 rewritten from "Flat File Migration" (publish MUST migrate flat to v1) to "Non-Document Files" (publish MUST NOT incorporate flat content; it creates v1 and replaces the file), SPEC 6.2/6.5/11.9 and DESIGN updated to match. Store change is LIST-side only: ListDir and the directory-visibility walks exclude versionless flat files in both views, and a directory whose subtree holds no documents is excluded even from the include-archived audit view. Legacy flat-layout docs (protocol-written, versions/doc.md.v{N}) remain visible and are unaffected. Write path needed no change; its replace behavior is now the specified behavior, which also removes the plugin PR's release-ordering constraint entirely.

Merged 2026-08-12 as PR #291 (9f1f062 on main): spec amendment, LIST exclusion of non-documents, eager legacy-layout migration in store.Open, legacy flat-layout code removed. The plugin PR #290 has no release-ordering dependency on it and remains open.

PR #290 merged 2026-08-12 (8571554); issue #288 resolved at both layers.

Related documents

• Five-minute appliance: the appliance work this roadmap section records
• OKF type adoption: the type policy behind the OKF section
• Postgres backend: the second store backend in the knowledge layer
• Version retention: the retention work named in the backlog
• Obsidian plugin plan: the plugin released in the plugins section

Production Shared Knowledge System: DONE (2026-08-23)

The GKE reference deployment now serves three logical worlds from one multi-replica GCS-backed knowledge server: root, latebit, and ontehfritz. Broker routing, Workload Identity, per-world buckets and policy, SNI aliases, writable token flows, Library broker mode, and the hourly federation agent are live. Client and agent 0.26.1 add the strict-policy metadata required by generated hash indexes and /graph.md; production publish verification passed.

Legacy per-world storage and credentials for root, servicing, latebit, origination, and world-a were removed after explicit review. The authenticated broker directory is now the three-world production topology. Current content worlds contain no user documents; populating latebit and ontehfritz is product/content work rather than deployment debt.