soul.demarkus.io:6309/roadmap.md/v51 draft reader meta

Roadmap

Where demarkus has been and where it's going.

Scope: Markdown-Only Server

The demarkus server serves markdown only. Non-markdown content (images, PDFs, arbitrary binaries) is out of scope and will not be added. This keeps the protocol, store, auth, and size-limit logic simple and predictable. Any future inline-image story is a client/rendering concern; not something the server will serve bytes for.

Phase 1: MVP (Read-Only): COMPLETE

Everything shipped:

  • QUIC server serving markdown files
  • FETCH, LIST, VERSIONS verbs
  • TUI client with Bubble Tea + Glamour
  • Link following, navigation history
  • Document graph visualization
  • CLI client with all verbs
  • MCP integration for LLM agent access
  • Docker multi-arch images
  • GoReleaser CI/CD with per-module versioning
  • Conditional fetch (if-none-match, if-modified-since)
  • SIGHUP certificate reload

Phase 2: Publish Operations: COMPLETE

Done:

  • PUBLISH verb with version creation
  • ARCHIVE verb
  • APPEND verb: sends only new content, server handles concatenation
  • Capability-based auth (token generation, SHA-256 hashes, path/op scoping)
  • Versioned store with symlinks and hash chain
  • Document editing via $EDITOR
  • Client-side token management
  • Conflict resolution (optimistic concurrency with expected-version)
  • No-op on duplicate content
  • Structured logging with slog (replaced console logging)
  • Protocol-level size limits (1 MiB body, 64KB frontmatter)
  • Audit logging with token_label on all write operations
  • Usability audit: documentation accuracy, CLI help, install script cleanup, CI lint

Phase 2 is fully complete. No remaining items.

Phase 3: Agent-Native & Advanced Features: COMPLETE

Done:

  • Agent manifest discovery (/.well-known/agent-manifest.md)
  • Bookmarks/favorites
  • MCP mark_append auto-resolve expected_version
  • Content-addressed fetch
  • Federation via MCP tools

Phase 4: The Information Graph: COMPLETE

Done:

  • The Demarkus Hub pattern
  • Persistent graph store
  • Backlinks
  • Graph as content (export/import)
  • Graph-aware navigation (TUI)
  • Agent discovery

Phase 5: The Demarkus Agent & Private Networks: IN PROGRESS

Read Auth for Private Networks: DONE

Per-path read token enforcement. Server-side and client-side shipped.

Security Hardening: DONE

See plan for full details.

  • Systemd hardening: install script generates units with ProtectSystem=strict, ReadWritePaths, NoNewPrivileges, etc. Conditional ProtectHome. Update path detects insecure config and prompts to harden with rollback.
  • -read-only mode: DEMARKUS_READ_ONLY env var / -read-only flag. Handler rejects PUBLISH/APPEND/ARCHIVE with not-permitted. Zero write access needed.
  • demarkus-publish: CLI tool that writes directly to the versioned store on disk. Enables local publishing when server runs read-only. Shares store.Write() with the server.
  • Read-only chroot install: install-readonly.sh. Chrooted server with ReadOnlyPaths=/, BindReadOnlyPaths=/dev/urandom. Maximum lockdown: Gemini-level security with full versioning.
  • Security documentation: attack surface analysis, threat model, comparison table, hardening guide.

Core Loop: Crawl & Index

  1. Seed: start from configured servers (hub, known peers)
  2. Crawl: follow mark:// links, discover new servers and documents
  3. Hash: collect content hashes from every document
  4. Index: publish updated hash indexes to configured hubs
  5. Repeat: on a configurable schedule, with conditional fetch (if-none-match) to be polite

Server-to-Server Sync

Rsync for the Mark Protocol. Replicate content between servers using content hashes as the diff mechanism.

How it works:

  1. LIST both source and destination servers
  2. Compare content hashes: skip documents that match
  3. FETCH changed/new documents from source
  4. PUBLISH to destination with the fetched content
  5. Optionally handle deletions (ARCHIVE on destination for docs removed from source)

Sync modes:

  • Mirror: destination becomes an exact copy of source (one-way)
  • Selective: sync specific paths or glob patterns (e.g., /docs/* only)
  • Multi-source: aggregate content from multiple servers into one destination

Key Design Points

  • Go binary, not an LLM agent: mechanical work, not reasoning
  • Polite crawling: conditional fetch, configurable rate limits
  • Hub-aware: reads and updates hub index documents
  • Daemon or cron: continuous or triggered
  • Auth-aware: passes read and write tokens for private servers

Implementation Sketch

  • New module: tools/demarkus-agent (the tools/ directory is the home for utility binaries; demarkus-token, demarkus-publish, and now sync/agent tooling)
  • Reuses: fetch.Client, graphstore, client/internal/index, graph.Crawl
  • CLI: demarkus-agent crawl, demarkus-agent sync source dest, demarkus-agent daemon

External Tool Sync: PLANNED

The shape that lets any markdown-emitting tool (OpenSPDD, ADR generators, doc generators, custom workflows) land its output on demarkus without integrating the protocol itself. The integration burden lives on demarkus's side; upstream tools change nothing.

demarkus-sync: single binary in tools/. Watches a local directory and PUBLISHes (or writes via demarkus-publish against a local store) on every changed .md. Versioning happens automatically through the existing store. Cross-team sharing emerges naturally from publishing to a team server.

Core flags:

  • --watch <dir>: directory to watch
  • --target mark://host/path/: remote demarkus server (uses fetch.Client PUBLISH)
  • --store <local-path>: alternative: write directly to a local versioned store via store.Write
  • --include / --exclude glob filters (e.g. **/*.canvas.md)
  • --token / DEMARKUS_AUTH for write auth

Project-flow integration is the killer use: drop into a Makefile target, pre-commit hook, CI step, or long-running dev daemon. Design contracts, ADRs, RFCs, runbooks, agent canvases all flow into the team world automatically; no per-tool integration work.

OpenSPDD as first concrete validation: REASONS Canvas docs are pure markdown contracts written to a local directory. demarkus-sync makes them a versioned, federated demarkus surface without OpenSPDD knowing. Same pattern works for any tool that emits markdown.

The deeper framing: demarkus shouldn't ask other tools to integrate it. It should make any markdown-emitting tool's output queryable as a versioned protocol surface, transparently. demarkus-sync is the generic shape of that.

Reuses: fetch.Client for remote PUBLISH, store.Write for local-store writes, fsnotify for the watch loop.

Phase 6: Universe Deployment: PLANNED

See plan for full details.

Make it cheap to stand up N worlds and onboard real users without hand-distributing tokens. Phase 5 gave us a single hardened server; Phase 6 makes a universe of them deployable with one Helm chart, one ApplicationSet, and one OIDC-fronted token broker.

Sub-phases:

  • 6.1: Helm chart (deploy/helm/demarkus-server/): StatefulSet + volumeClaimTemplates so each world owns its own PVC, Service, bootstrap Job that runs demarkus-token generate and writes to a Secret, optional read-only mode, SIGHUP reload wiring.
  • 6.2: Universe topology (deploy/k8s/examples/): reference Argo CD ApplicationSet over a worlds: list, plus a Kustomize overlay for clusters without Argo. No new code.
  • 6.3: Token broker (prototype in tools/demarkus-broker/, splits to latebit-io/demarkus-broker after 6.4): OIDC-fronted HTTP service that mints scoped tokens, writes hashes to world Secrets, SIGHUPs on revocation, audit log to stdout.
  • 6.4: User install flow (lives with broker): GET /me/install returns a one-shot shell script that writes ~/.config/demarkus/auth and idempotently patches ~/.claude.json with a demarkus-mcp server entry per world.
  • 6.5: Docs: /deployment.md for chart values + broker setup; plan doc maintained at /plans/universe-deployment.md.

Constraint: zero changes to protocol or core server. Phase 6 is packaging and lifecycle. Any need for a new core primitive must be discussed and justified before landing, same rule that applied to the Claude Code plugin.

Explicitly deferred to Phase 7+: multi-replica worlds with shared storage, cross-cluster universe federation, an operator with a World CRD, a web UI on top of the broker.

Verb Set: Complete

7 verbs: FETCH, LIST, VERSIONS, PUBLISH, APPEND, ARCHIVE, LOOKUP.

LOOKUP (issue #113) is the catalog verb: subject lookup over an in-memory, importance-ranked catalog of declared tags + title. It is a token-efficient supplement to the index, not full-text search (which stays in an opt-in sidecar). See the LOOKUP Catalog Verb section below and /plans/lookup-verb.md.

LOOKUP Catalog Verb: DONE (merged to main 2026-05-30, PR #166; plugin tail closed by #168)

See plan. Tracks issue #113.

Given a subject, return which documents are about it and how important they are; a card-catalog supplement to the index hub so an agent finds the right doc without fetching and grepping the whole index. Catalog scan only: in-memory path → {tags, importance, title}, built on the startup walk and maintained inline on writes, ranked by match-count then importance. No body reads at query time, no relevance modeling. Full-text / semantic search is permanently out of core, in an opt-in sidecar.

Shipped:

  • Protocol verb + spec §6.7.
  • server/internal/catalog package + handler with read-auth result filtering.
  • Client surfaces: fetch.Client.Lookup, demarkus lookup CLI, MCP mark_lookup.
  • Broker parity: mark_lookup is the gateway's 14th tool.
  • Publisher metadata: tags/importance settable at publish; CLI -meta key=value (repeatable), and a metadata object on MCP + broker mark_publish. This is what makes the tag/importance ranking actually usable (before it, LOOKUP was title-only with uniform 0.5 importance).
  • End-to-end smoke test passed (publish with tags → lookup by tag → archive, over QUIC).
  • Plugin surfacing (v0.3.0, #168, 2026-05-31): Claude Code plugin wires mark_lookup, fixes the knowledge-join tool-count text (13→14), and injects SessionStart guidance so sessions recall via lookup.

Remaining tail (small): mark_append metadata; deferred by design (you tag on publish, not append). The plugin knowledge-join/version-pin items are now shipped in v0.3.0.

Open Knowledge Format (OKF) Compatibility: DONE (merged 2026-06-22, PR #195)

demarkus interoperates with Google's Open Knowledge Format v0.1 (a bundle of markdown files with YAML frontmatter). The document content model is OKF-compatible; demarkus layers versioning, hash chain, QUIC transport, capability auth, and LOOKUP around it.

Shipped:

  • Store metadata alignment: recognized OKF fields (type, title, description, resource, tags, timestamp) serialize as bare frontmatter; tags as a YAML flow list; non-spec keys keep the meta. prefix; reserved operational fields (version, previous-hash, archived) enforced by name. Back-compatible read of legacy meta.*. Caps raised to 50 keys / 1024 bytes (frontmatter budget 2048), tags counted at serialized length.
  • demarkus okf codec (client/internal/okf); validate (v0.1 conformance), import (bundle → world, sanitize/cap with warnings, link rewrite, upsert), export (world subtree → conformant bundle, frontmatter reattach, type synthesis, RFC 3339 timestamp normalization). Verified live: import → export → validate round-trip, byte-identical bodies.
  • Opinionated default: server assigns type: Document on PUBLISH and APPEND when none declared (reserved index.md/log.md exempt), so every served document is a typed OKF concept by construction. Non-breaking for existing typeless writers (ADR 0003).
  • Spec §8.1/§9.4/§13/§14; ADR 0002 (metadata alignment), ADR 0003 (type default).
  • Review hardening (PR #195 CodeRabbit): export path-traversal guard, import symlink rejection, LIST decode/cycle hardening, reserved-key read guard, cap re-validation, link query strip.

MCP Agent Ergonomics & Attach Surface: DONE (2026-07-04/05)

Two same-week phases that made MCP agents cheap to serve and demarkus pleasant to drive from Claude Code / Desktop. Plans: /plans/mcp-client-ergonomics.md and /plans/mcp-resources-prompts.md. Both fully shipped AND deployed on both MCP surfaces (local demarkus-mcp and the broker gateway).

Shipped:

  • Size-adaptive mark_fetch: full body under 8KB; above it an outline (heading tree with GitHub-slug #anchors + per-section line counts + opening paragraph + hint). url#anchor slices one section at any size; force overrides. Killed the five-figure-token fetches of append-heavy docs (measured: a 52KB roadmap → 5.9KB outline).
  • Session unchanged-dedup: re-fetch of an unchanged doc returns status: unchanged since vN (identity = version+etag; identity-flip edge cases spelled out). Process-scoped in the local client; MCP-session-keyed at the broker (multi-tenant safe: no session → no dedup; LRU eviction at cap, since mcp-go only unregisters sessions on explicit DELETE).
  • mark_explore: one-call orientation card (outline head, opening paragraph, outbound links, backlinks, siblings; 10/section + "+N more").
  • MCP resources: documents as client-attachable context, zero tool turns: mark:// URI templates, #anchor section attach, well-known + background-listed picker entries (client) / per-world index hubs (broker).
  • MCP prompts: orient / recall / whats-new as server-vended slash commands; the broker variants sweep worlds via mark_worlds.
  • Shared packages client/mdoutline (outline/slice + render helpers) and client/fetchdedup (dedup identity + notice text); one implementation for both surfaces (the changedNote mirror drifted twice in review before hoisting; lesson in /debugging.md).
  • One #section grammar everywhere: mark_fetch, mark_explore, MCP resource URIs, and the demarkus-library librarian's open tool (adopted 2026-07-05) all share the same anchors from the same implementation.
  • Deployed: plugin users via client v0.15.0→v0.17.0 (tools 0.4.1+, pin chain now self-driving after the #228 pin-bump automation fix); knowledge.demarkus.io via broker 0.5.0 (ergonomics) and 0.6.0 (resources+prompts).

Remaining: none. Follow-ups all landed (broker parity, resources/prompts, librarian adoption).

Distribution & Package Management: PLANNED

Homebrew tap for easier installation on macOS and Linux.

Build Targets

Supported platforms: macOS, Linux, Windows (WSL only).

Plugins: IN PROGRESS

Obsidian Plugin: v0.1.0 RELEASED

Standalone repo: latebit-io/obsidian-demarkus. Source in plugins/obsidian/.

Claude Code Plugin: v0.12.5 (current)

Merged into main via PR #96 (commit af8e210) on 2026-04-23 as v0.1.0. Source at plugins/claude-code/ in the monorepo; marketplace manifest at .claude-plugin/marketplace.json.

See plan for the full design. One-click marketplace install that lazy-spawns a local demarkus-server, auto-generates a token, and wires demarkus-mcp into the agent. Zero config on install. Includes /soul, /soul-journal, /soul-init, /soul-join, /soul-default, /knowledge-join slash commands and a soul-memory skill.

Shipped with a project-centric soul schema: /index.md is a project list, each project under /<slug>/ holds plan/tasks.md, architecture.md, patterns.md, roadmap.md, adr/, journal/<YYYY-MM-DD>.md.

Version history:

  • v0.1.0 (2026-04-23, #96): initial marketplace plugin.
  • v0.2.0 (2026-05-23, #152): /knowledge-join command for organizational broker-fronted knowledge systems.
  • v0.3.0 (2026-05-31, #168): wires mark_lookup; injects standing SessionStart guidance (context/session-guidance.md) so sessions self-document to the soul and recall via lookup; fixes knowledge-join tool count (13→14); bumps binary pins to SERVER 0.17.13 / CLIENT 0.12.38 / TOOLS 0.1.28.
  • v0.4.0–v0.9.x: not enumerated here (promote bridge, soul-refresh, binary-pin automation, OKF release pin, soul catalog groundwork). See the journals for detail.
  • v0.10.0 / v0.10.1 (#200, #201): /soul-join: managed remote souls. Joins a direct-QUIC remote soul (token in a 0600 file via a launch wrapper, never inline in MCP config), records it in a catalog (~/.demarkus/souls) with a per-project binding (~/.demarkus/project-souls), and enforces the binding with a destination gate (PreToolUse on mark_publish/mark_append; a write to a non-bound soul is denied/asked/warned). Publish tag-gate extended to joined remote souls. 0.10.1 closed a reuse-server restart bug (never kill a healthy externally-managed server on binary upgrade).
  • v0.11.0 (#202, 2026-06-22): /soul-default: set a project's default write target. Standalone command to re-point an already-joined repo at a different default without re-running /soul-join (until now the binding was only ever written as a --bind side-effect of join). Lists the joined-soul catalog (local managed soul + remotes), picks one, saves the binding the dest-gate enforces. Routing stays model-driven: the catalog is the discovery surface and each soul is its own MCP server, so no router or dispatch tool; reads and one-off writes to another joined soul go direct to its mcp__<slug>__mark_* tools (the gate fires only on publish/append). New lib.sh helpers (local_soul_present, soul_catalog, is_catalog_soul); no core/protocol change.
  • v0.12.x (late June → 2026-07-05): pi-agent port fixes (#217), the ergonomics release-chain repins (tools 0.3.5 → 0.4.1, #221/#227), and the automated pin-bump sweep (#229) after the pin-bump workflow's closed-PR upsert bug was fixed (#228: the pin chain is self-driving again: release → auto bump PR → merge). Current: demarkus-memory 0.12.5, demarkus-knowledge 0.5.5.

No core code changes landed. The plugin is built entirely on existing primitives: DEMARKUS_AUTH env var for token injection, /health endpoint plus ALPN "mark" negotiation for port probing, and shell redirection of demarkus-token generate stdout for writing the token to a file. Any future need for a new core primitive must be discussed and justified before landing.

TUI Polish

See plan. Closes issue #75 in full. Opens http/https/gemini/mailto links in the user's default handler via open/xdg-open/rundll32. Scheme allowlist, URL passed as argv only, no shell.

The original plan had a Phase 2 for same-server non-markdown files (images); dropped because the server is markdown-only and will not serve binaries. Inline image rendering, if ever pursued, is a separate client-rendering initiative (terminal graphics protocols like Kitty/Sixel), not a continuation of this work.

Features Not Prioritized: Backlog

  • WebSub-style Subscriptions (removed from spec)
  • Offline Mode (deferred)
  • Full-Text / Semantic Search (out of core: opt-in sidecar reading via LIST/FETCH; in-core LOOKUP covers catalog-level subject lookup, see above)
  • Diff / Changelog Between Versions (noted, not implemented)
  • Blind Append with Content Deduplication (rejected)
  • Non-markdown content types on the server (out of scope by design)

Operational Polish: Backlog

  • Plain-directory root returns silent not-found; the server only serves the versioned-symlink store layout. Pointing -root at a directory of plain .md files (e.g. docs/site) silently returns not-found for every path and builds a 0-entry hash index and catalog, with no startup warning. Bit setup during the 2026-06-02 load-test work. Flagged to revisit: at minimum a startup warning when the root has markdown files but no versions/ layout; ideally a clear hint.

  • tools/demarkus-loadtest: concurrent read-only load generator (FETCH/LIST/ VERSIONS/LOOKUP) for capacity measurement. Reuses fetch.Client; warm (reused-conn) vs -fresh (handshake-per-request) regimes. Report and first measured numbers in tools/demarkus-loadtest/REPORT.md. See journal 2026-06-02. Key finding: server self-throttles per IP (default 50 req/s, burst 100, DEMARKUS_RATE_LIMIT=0 disables); warm ceiling ~7.7k req/s, cold ~1.4k req/s on one core.

Docs Hygiene

  • SPEC.md markdownlint cleanup: DONE (PR #196, merged 2026-06-22). Added repo-wide .markdownlint.json disabling MD013 (line-length) and MD060 (table-column-style): both pervasive, intentional spec style. Fixed the substantive rules in docs/SPEC.md: fence languages (MD040, 29 fences → text), blank lines around fences/lists (MD031/MD032), bare URLs (MD034), and the emphasis footer (MD036). markdownlint-cli2 docs/SPEC.md → 0 errors; formatting only, no content changes.

Knowledge Layer & Storage Backends: NEXT (added 2026-07-13)

Wire backlinks/graph tools to the published /graph.md: IMPLEMENTED (see Graph hub seeding below)

The federation agent already publishes the aggregated link graph to each hub at /graph.md (hourly, cross-server edges, same format as mark_graph_export), but mark_backlinks and mark_graph answered only from the client's local ~/.mark/graph.json crawl cache. A fresh client answered from an empty graph while an authoritative aggregate sat on the hub unread.

  • Seed/refresh the MCP graphstore from the hub's /graph.md (etag-cached fetch, merge into local store) before falling back to local crawl.
  • Fixes cold start and cross-agent consistency with a client-only change; days of work, no server change.
  • Later upgrade, not the fix: publish-time edge extraction in the store (instant, transactional backlinks). Subject to the backend-parity principle below.

Principle: backend parity is system-level, not implementation-level

With two store backends (file, postgres), behavioral parity is a property of the system, not a detail of either implementation. The storetest conformance suite is the contract; both backends run it unchanged and divergence is a protocol bug.

Consequences for future work:

  • Any new store-observable behavior lands in the conformance suite first, then in every backend, or it does not ship.
  • Knowledge-layer features should prefer layers above the store (published documents like /graph.md, broker aggregation, client merge) which are backend-independent by construction.
  • A feature that only one backend can provide efficiently (e.g. a transactional edges table in postgres) must still be defined system-level: either every backend implements the same observable surface, or the capability is exposed as an optional, discoverable feature rather than silently backend-dependent.

Remaining gaps from the 2026-07-13 knowledge-layer analysis

The /graph.md wiring item above came out of a gap analysis (knowledge-system-deploy session, 2026-07-13): demarkus is strong at the document layer (versioning, integrity, provenance, gates) and weak at the knowledge layer. Only gap 1 got recorded at the time; the rest are recorded here so they are not lost. Framing correction from that session: the plugins are the knowledge layer (recall-first, tag-at-publish, promote, doctors), so the governing principle is that the agent owns judgment while the server owns the accumulation of that judgment (edges, indexes, derived rank live in shared storage so the layer compounds across agents instead of being recomputed per client).

Edge semantics: give edges more information

Every link in the graph is an untyped "mentions". Knowledge graphs live on predicates: supersedes, implements, depends-on, derived-from. OKF types the nodes; nothing types the edges (ADR 0002 superseding its v1 draft was expressible only as prose). Edges also carry no provenance: graph.Edge/StoredEdge are bare from/to pairs and /graph.md exports a two-column table, so backlinks cannot say why or where a doc links (no link text, no source anchor, no occurrence count).

Shape: a rel: metadata axis (e.g. rel:supersedes=/adr/0002-....md) surfaced in mark_graph/mark_backlinks, plus richer edge records in the export (label, source anchor, count). This is a plugin and export-format convention, zero server change, so it can land before publish-time edge extraction; when the store later learns edge extraction it must capture the same fields (backend-parity principle above).

Semantic recall beyond tags

Lookup recall depends entirely on declared tags and titles; recall bounded by metadata hygiene loses to grep, and the agent compensates by exploring and fetching more every session (judgment tokens spent on enumeration). Path: Postgres tsvector full-text over bodies (cheap, phase-2 adjacent), then pgvector embeddings, hybrid-ranked with declared importance. Indexes amortize the agent's search loop into infrastructure; they do not replace its judgment.

Universe-wide LOOKUP

Lookup is scoped to one world; the broker routes but does not aggregate, so "search the whole system" is client-side fan-out. A broker-level aggregated LOOKUP (and eventually graph) makes a multi-world system feel like one system. Broker feature, no server change.

Derived ranking signals

importance is entirely self-declared, so a well-tagged orphan outranks the hub everything cites. Blend link-graph signals (in-degree, centrality) with declared importance. Link structure is not usage telemetry, so this respects the no-tracking principle.

Staleness at lookup time

modified exists but nothing decays or flags aging content at query time; the doctors catch it offline only.

Edge semantics: DONE (merged 2026-07-14, PR #251, commit 8798d0a)

The "Edge semantics: give edges more information" gap above is shipped; see /plans/edge-semantics.md and repo ADR 0004. Edges carry {From, To, Rel, Label, Anchor, Count} with identity {From, To, Rel}; the rel-<predicate> publisher-metadata convention (hyphen form, CSV values) is ingested as typed edges by both crawlers; the /graph.md export is six columns with legacy 2-column parse kept; mark_graph/mark_backlinks/mark_explore render provenance via the shared graph.EdgeAnnotation. Zero server changes. The fetch-callback debt item went with it (removed from /debt.md on merge). Review hardening from three CodeRabbit rounds: whitespace-strict rel refs, Merge replaces refreshed sources' outgoing edges so stale backlinks drop, BaseInline.Lines() panic guard for links inside inline formatting (lesson in /debugging.md), BlockStart provenance for empty-label links. Hubs pick up the enriched /graph.md once the federation agent redeploys with the new binary.

Graph hub seeding: IMPLEMENTED (2026-07-14, PR pending; flip to DONE on merge)

The "Wire backlinks/graph tools to the published /graph.md" item is implemented per /plans/graph-hub-seed.md, on branch graph-hub-seed. What shipped:

  • graphstore SeedFromExport with the local-wins rule: authoritative = real fetched status AND non-zero CrawledAt; seeded nodes carry zero CrawledAt as the durable marker; the same rule arbitrates against Merge's replace-refreshed-sources logic (shared dropEdgesFromLocked/upsertEdgeLocked helpers, tested from both directions).
  • Seed etags persist in graph.json as an additive seed_etags map (host to etag); schema stays v1, legacy files load with a nil map.
  • fetch.FetchConditional(host, path, token, etag) for explicit if-none-match (token'd souls skip the disk cache, so the seeder tracks freshness itself). First wire-level test of fetch.Client via an in-process QUIC server.
  • demarkus-mcp: seedGraph(defaultHost) in mark_backlinks, mark_graph, and explore's backlinks section; 5-minute per-host process-scoped throttle; never fatal (warn-log and degrade).
  • Broker: seedWorldGraph through dispatchWithAuth in the same three handlers, per world, so cold pods answer backlinks; FetchConditional added to worldDispatcher/worldPool (trivial passthrough); MCP-API.md updated.
  • Bonus fix required by the live check: the MCP backlinks/graph URL keys are now canonicalized through resolveURL (default port applied). Without it the seeded aggregate's mark://host:6309 rows never matched queries built from a portless -host flag; regression test pins it.
  • ParseExport got its first production caller and the live soul /graph.md (still two-column legacy, v6) seeded correctly.

Verified live: scratch-HOME cold start against the soul answered 4 backlinks for /patterns.md with zero crawls (133 nodes / 190 edges seeded, etag persisted); a depth-1 crawl of /conventions.md then flipped it authoritative and its enriched edges replaced the seeded rows while 130 nodes stayed seeded. TUI seeding and Exported-timestamp arbitration remain out of scope.

soul.demarkus.io:6309/plans/graph-hub-seed.md draft reader meta

Plan: Wire backlinks/graph tools to the published /graph.md (hub seeding)

Status: implemented 2026-07-14 on branch graph-hub-seed (PR pending); planned 2026-07-13. Closes roadmap "Wire backlinks/graph tools to the published /graph.md" (gap 1 of the 2026-07-13 knowledge-layer analysis). See the implementation notes at the end for deviations.

Context

The federation agent publishes the aggregated link graph to each hub at /graph.md (now with enriched six-column edges after the edge-semantics work, PR 251), but mark_backlinks and mark_graph answer only from the local crawl cache: ~/.mark/graph.json for demarkus-mcp, a per-pod ephemeral store for the broker. A fresh client or a recycled broker pod answers from an empty graph while an authoritative aggregate sits on the hub unread. The accumulation exists; the query path bypasses it.

Client-only change. No server, protocol, or store-backend work. ParseExport (currently test-only) becomes the production consumer of /graph.md.

Design decisions

  • Seed source is the attached world. demarkus-mcp seeds from defaultHost + "/graph.md" (the -host world; the soul server publishes its own /graph.md). The broker seeds per world from mark://{world}/graph.md through dispatchWithAuth. No new flags; a world without /graph.md degrades silently to today's behavior (not-found is not an error).
  • Local wins. The hub aggregate carries no per-node freshness, so locally crawled data is never overwritten by seed data. A stored node counts as locally authoritative iff its status is not one of "", "external", "error" AND CrawledAt is non-zero. Seeded nodes get zero CrawledAt, which is the durable "seeded, not locally observed" marker across restarts.
  • Seeding fills gaps only: nodes inserted when absent or when the existing node is non-authoritative; edges replace the outgoing set only for sources that are not locally authoritative. A later local crawl of a seeded source replaces its edges through the existing Merge refresh logic.
  • Conditional refresh with our own etag. fetch.Client's built-in if-none-match rides the unauthenticated disk cache, which token'd souls skip, so the seeder stores the last /graph.md etag itself: new seed_etags map (host -> etag) in the graph.json envelope, additive JSON, schema stays v1. Requests send if-none-match; not-modified means skip parse and merge.
  • Throttled per process. A conditional check costs one round trip; still, cap at one check per host per 5 minutes (package constant), process-scoped like the fetchdedup session state. First graph-tool call in a session always checks.
  • Never fatal. Any seed failure (fetch error, malformed document, oversized) logs at warn and falls through to the local store. Seeding must never make backlinks worse than today.

Steps

1. graphstore: seed support (client/graphstore/store.go, export.go)

  • document envelope gains SeedEtags map[string]string with json:"seed_etags,omitempty"; Store carries it; Load/Save round-trip it (nil-safe for legacy files).
  • SeedEtag(host) string and SetSeedEtag(host, etag) accessors (locked).
  • SeedFromExport(nodes []StoredNode, edges []StoredEdge) (added int):
    • classify locally authoritative sources (status not in {"", "external", "error"} and CrawledAt non-zero);
    • insert nodes when absent or existing node non-authoritative, forcing CrawledAt to zero on the seeded copy;
    • drop stored edges whose From is a seeded (non-authoritative) source being refreshed, then insert seed edges for those sources, Count normalized, dedup on edgeKey;
    • never touch nodes or edges of authoritative sources.
  • Optionally parse the > Exported: header line in ParseExport later; NOT in scope (freshness rule does not need it).

Tests: seed into empty store; local-wins (authoritative node and its edges untouched); seeded-then-crawled source flips to authoritative and Merge replaces its edges; seed refresh replaces prior seeded edges (no stale seeded backlinks); seed_etags round-trips through Save/Load; legacy file loads with nil map.

2. fetch: conditional fetch with explicit etag (client/fetch/fetch.go)

cachedRequest already threads extra metadata internally. Export a minimal surface:

// FetchConditional fetches with an if-none-match etag; status not-modified
// returns with an empty body.
func (c *Client) FetchConditional(host, path, token, etag string) (Result, error)

Implemented via the existing extra-metadata path (option-bearing requests already skip the disk cache, which is correct here). Test with the existing mock-stream harness: etag sent, not-modified passthrough.

3. demarkus-mcp: seed hook (client/cmd/demarkus-mcp/main.go, explore.go)

  • handler gains graphSeed state: map[string]time.Time last-checked per host + mutex (process-scoped, mirrors fetchdedup's session scoping).
  • seedGraph(host string): throttle check; FetchConditional(host, "/graph.md", token, store.SeedEtag(host)); on ok: ParseExport then SeedFromExport, SetSeedEtag, Save(); on not-modified/not-found/error: return silently (warn-log real errors).
  • Call seedGraph(defaultHost) at the top of markBacklinks, markGraph, and explore's writeBacklinksSection (before the store read; markGraph seeds so depth-limited crawls still benefit from hub context).
  • Tool descriptions: one sentence on mark_backlinks/mark_graph ("seeded from the world's published /graph.md when available; local crawls take precedence"). No em dashes.
  • The "No backlinks found ... run mark_graph" hint stays as the empty-store fallback text.

Tests (mock fetch func in main_test.go style): cold store + hub graph answers backlinks without a crawl; hub 404 degrades to the current hint; local crawl beats conflicting seed rows; second call within the throttle window does not refetch; etag round-trip sends if-none-match.

4. Broker: per-world seeding (tools/demarkus-broker/internal/broker/mcp_tools_graph.go, mcp_tools_explore.go)

  • Gateway gains graphSeed map keyed by worldName {etag string, checked time.Time} + mutex (per pod, like graphStore).
  • seedWorldGraph(ctx, claims, worldName): throttle; dispatchWithAuth Fetch of /graph.md (send if-none-match via the dispatcher's metadata path or accept a full fetch and compare the stored etag; pick during implementation, do not add dispatcher surface unless trivial); ParseExport + SeedFromExport + record etag. Silent degrade.
  • Call from handleMarkBacklinks, handleMarkGraph, and explore's backlinks section, scoped to the world parsed from the tool URL.
  • MCP-API.md: note the graph store is seeded from each world's published /graph.md on demand, so cold pods answer backlinks; local crawls still take precedence; ephemerality note stays (the seed makes restarts cheap, not durable).

Tests: cold gateway + world with /graph.md answers backlinks with no prior crawl (this is the headline behavior); world without /graph.md keeps today's empty hint; seeded then crawled world prefers crawl results; throttle respected across two calls.

5. Docs and bookkeeping

  • Roadmap: mark the wiring item IMPLEMENTED with plan link (soul, post-merge DONE edit as usual).
  • docs/site/client/index.md and architecture/index.md: one line each where mark_backlinks/graph persistence is described ("seeded from the published /graph.md").
  • ADR: not needed (no new convention; consumes ADR 0004's format). Note in the PR description instead.

Out of scope

  • TUI graph-view seeding (worthwhile follow-up; keep this PR to the MCP surfaces the roadmap item names).
  • Parsing the hub graph's Exported timestamp for freshness arbitration (local-wins does not need it).
  • Publish-time edge extraction in the store (the "later upgrade" per the roadmap; backend-parity applies there).
  • demarkus-library floor changes (its own repo; it already reads /graph.md).

Risks

  • Seed data quality: the hub aggregate may contain hosts unreachable from this client (cluster-internal names in the knowledge system's graph). Harmless for backlinks (they are just labels); mark_graph crawls starting from them will error per node as today. Do not filter by reachability.
  • Large /graph.md: the enriched table grows rows, not new fetch cost (single doc). ParseExport is linear; the 1 MiB protocol body cap bounds it. No action.
  • Merge interplay: SeedFromExport must not fight the new Merge replace-refreshed-sources logic; the shared authoritative-source rule (status + CrawledAt) is the single arbiter, tested from both directions.
  • Etag identity: /graph.md is republished wholesale with retention 20; etag changes on every publish even if content is identical except the Exported line, so not-modified hits only between publishes. Accept; the throttle bounds the cost.

Verification

  1. Unit suites: client graphstore, fetch, demarkus-mcp; broker package. bash pre-commit.sh.
  2. Live cold-start check: build demarkus-mcp, delete (or point HOME at a scratch dir for) ~/.mark/graph.json, attach to the soul, call mark_backlinks on a doc known to have hub-graph backlinks; expect answers with zero crawls. Then mark_graph a subtree and confirm local results win and persist.
  3. Broker check via its test harness (no cluster needed): cold gateway seeds from a mocked world /graph.md.
  4. Legacy: a /graph.md still in two-column form seeds correctly (ParseExport dual-format already tested; add one seed test using a legacy fixture).

Implementation notes (2026-07-14)

Implemented as planned, with these deviations and additions:

  • URL canonicalization fix (unplanned, required). The first live cold-start check failed: the soul's /graph.md keys rows on canonical mark://soul.demarkus.io:6309/... URLs, but mark_backlinks built its lookup key as defaultHost + path with no default-port normalization, and the plugin's -host flag omits the port. Fixed by canonicalizing through resolveURL ("mark://" + host + path) in markBacklinks, markGraph's start URL, and explore's backlinks section. This was a latent pre-existing mismatch (portless -host crawls and full-URL queries could already disagree); seeding surfaced it. Regression test: TestSeedGraph_DefaultHostWithoutPortCanonicalizes.
  • Step 2 test harness. The "existing mock-stream harness" is server-side only, so FetchConditional got the client's first wire-level test: an in-process QUIC listener with a self-signed cert (client/fetch/conditional_test.go) asserting etag sent, not-modified passthrough, and stale-etag refetch.
  • Step 4 dispatcher surface. Adding FetchConditional to worldDispatcher/worldPool was trivial (passthrough to fetch.Client), so the broker sends real if-none-match instead of comparing etags after a full fetch. The broker's seed etag lives in the in-memory graphstore's seed_etags map (keyed by worldName); only the throttle map lives on the gateway.
  • Shared helpers. Merge's drop-refreshed-edges and upsert logic were extracted (dropEdgesFromLocked, upsertEdgeLocked) so SeedFromExport and Merge share one implementation of the arbitration rule.
  • Live verification passed (2026-07-14): scratch-HOME cold start against the soul answered 4 backlinks for /patterns.md with zero crawls; 133 nodes / 190 edges seeded from the still-legacy two-column /graph.md (v6); seed etag persisted; all seeded nodes carried zero CrawledAt. A depth-1 mark_graph of /conventions.md flipped it authoritative (real CrawledAt) and its enriched crawl edges replaced the seeded legacy row in backlinks output, with the other 130 nodes untouched.

Review round (PR 253, CodeRabbit)

Three findings, all accepted:

  • Seed the resolved host, not defaultHost. The plan's "seed source is the attached world" left full mark:// URLs against other hosts unseeded. seedGraph now takes the canonical host from resolveURL (throttle, etag, and token resolution were already per-host, and the broker already seeded the URL's world). Test: TestSeedGraph_SeedsResolvedHostNotDefault.
  • Stale-edge gap in SeedFromExport. The drop set was built only from seed-edge Froms, so a re-seed where a source's outgoing set went to zero left its old seeded edges behind. Now every observed (real-status) non-authoritative seed node also joins the drop set, mirroring Merge's refreshed criterion; the status predicate is extracted as observedStatus and shared by Merge, authoritativeLocked, and SeedFromExport. Tests: TestSeedRefreshDropsEdgesOfEmptiedSource plus the counter-case TestSeedKeepsEdgesOfUnobservedSeedNode (an error-status seed node with no edges must not drop what it never read).
trail
  1. soul.demarkus.io:6309 v51
  2. graph-hub-seed