soul.demarkus.io:6309/plans/graph-hub-seed.md/v4 draft reader meta

Plan: Wire backlinks/graph tools to the published /graph.md (hub seeding)

Status: implemented 2026-07-14 on branch graph-hub-seed (PR pending); planned 2026-07-13. Closes roadmap "Wire backlinks/graph tools to the published /graph.md" (gap 1 of the 2026-07-13 knowledge-layer analysis). See the implementation notes at the end for deviations.

Context

The federation agent publishes the aggregated link graph to each hub at /graph.md (now with enriched six-column edges after the edge-semantics work, PR 251), but mark_backlinks and mark_graph answer only from the local crawl cache: ~/.mark/graph.json for demarkus-mcp, a per-pod ephemeral store for the broker. A fresh client or a recycled broker pod answers from an empty graph while an authoritative aggregate sits on the hub unread. The accumulation exists; the query path bypasses it.

Client-only change. No server, protocol, or store-backend work. ParseExport (currently test-only) becomes the production consumer of /graph.md.

Design decisions

  • Seed source is the attached world. demarkus-mcp seeds from defaultHost + "/graph.md" (the -host world; the soul server publishes its own /graph.md). The broker seeds per world from mark://{world}/graph.md through dispatchWithAuth. No new flags; a world without /graph.md degrades silently to today's behavior (not-found is not an error).
  • Local wins. The hub aggregate carries no per-node freshness, so locally crawled data is never overwritten by seed data. A stored node counts as locally authoritative iff its status is not one of "", "external", "error" AND CrawledAt is non-zero. Seeded nodes get zero CrawledAt, which is the durable "seeded, not locally observed" marker across restarts.
  • Seeding fills gaps only: nodes inserted when absent or when the existing node is non-authoritative; edges replace the outgoing set only for sources that are not locally authoritative. A later local crawl of a seeded source replaces its edges through the existing Merge refresh logic.
  • Conditional refresh with our own etag. fetch.Client's built-in if-none-match rides the unauthenticated disk cache, which token'd souls skip, so the seeder stores the last /graph.md etag itself: new seed_etags map (host -> etag) in the graph.json envelope, additive JSON, schema stays v1. Requests send if-none-match; not-modified means skip parse and merge.
  • Throttled per process. A conditional check costs one round trip; still, cap at one check per host per 5 minutes (package constant), process-scoped like the fetchdedup session state. First graph-tool call in a session always checks.
  • Never fatal. Any seed failure (fetch error, malformed document, oversized) logs at warn and falls through to the local store. Seeding must never make backlinks worse than today.

Steps

1. graphstore: seed support (client/graphstore/store.go, export.go)

  • document envelope gains SeedEtags map[string]string with json:"seed_etags,omitempty"; Store carries it; Load/Save round-trip it (nil-safe for legacy files).
  • SeedEtag(host) string and SetSeedEtag(host, etag) accessors (locked).
  • SeedFromExport(nodes []StoredNode, edges []StoredEdge) (added int):
    • classify locally authoritative sources (status not in {"", "external", "error"} and CrawledAt non-zero);
    • insert nodes when absent or existing node non-authoritative, forcing CrawledAt to zero on the seeded copy;
    • drop stored edges whose From is a seeded (non-authoritative) source being refreshed, then insert seed edges for those sources, Count normalized, dedup on edgeKey;
    • never touch nodes or edges of authoritative sources.
  • Optionally parse the > Exported: header line in ParseExport later; NOT in scope (freshness rule does not need it).

Tests: seed into empty store; local-wins (authoritative node and its edges untouched); seeded-then-crawled source flips to authoritative and Merge replaces its edges; seed refresh replaces prior seeded edges (no stale seeded backlinks); seed_etags round-trips through Save/Load; legacy file loads with nil map.

2. fetch: conditional fetch with explicit etag (client/fetch/fetch.go)

cachedRequest already threads extra metadata internally. Export a minimal surface:

// FetchConditional fetches with an if-none-match etag; status not-modified
// returns with an empty body.
func (c *Client) FetchConditional(host, path, token, etag string) (Result, error)

Implemented via the existing extra-metadata path (option-bearing requests already skip the disk cache, which is correct here). Test with the existing mock-stream harness: etag sent, not-modified passthrough.

3. demarkus-mcp: seed hook (client/cmd/demarkus-mcp/main.go, explore.go)

  • handler gains graphSeed state: map[string]time.Time last-checked per host + mutex (process-scoped, mirrors fetchdedup's session scoping).
  • seedGraph(host string): throttle check; FetchConditional(host, "/graph.md", token, store.SeedEtag(host)); on ok: ParseExport then SeedFromExport, SetSeedEtag, Save(); on not-modified/not-found/error: return silently (warn-log real errors).
  • Call seedGraph(defaultHost) at the top of markBacklinks, markGraph, and explore's writeBacklinksSection (before the store read; markGraph seeds so depth-limited crawls still benefit from hub context).
  • Tool descriptions: one sentence on mark_backlinks/mark_graph ("seeded from the world's published /graph.md when available; local crawls take precedence"). No em dashes.
  • The "No backlinks found ... run mark_graph" hint stays as the empty-store fallback text.

Tests (mock fetch func in main_test.go style): cold store + hub graph answers backlinks without a crawl; hub 404 degrades to the current hint; local crawl beats conflicting seed rows; second call within the throttle window does not refetch; etag round-trip sends if-none-match.

4. Broker: per-world seeding (tools/demarkus-broker/internal/broker/mcp_tools_graph.go, mcp_tools_explore.go)

  • Gateway gains graphSeed map keyed by worldName {etag string, checked time.Time} + mutex (per pod, like graphStore).
  • seedWorldGraph(ctx, claims, worldName): throttle; dispatchWithAuth Fetch of /graph.md (send if-none-match via the dispatcher's metadata path or accept a full fetch and compare the stored etag; pick during implementation, do not add dispatcher surface unless trivial); ParseExport + SeedFromExport + record etag. Silent degrade.
  • Call from handleMarkBacklinks, handleMarkGraph, and explore's backlinks section, scoped to the world parsed from the tool URL.
  • MCP-API.md: note the graph store is seeded from each world's published /graph.md on demand, so cold pods answer backlinks; local crawls still take precedence; ephemerality note stays (the seed makes restarts cheap, not durable).

Tests: cold gateway + world with /graph.md answers backlinks with no prior crawl (this is the headline behavior); world without /graph.md keeps today's empty hint; seeded then crawled world prefers crawl results; throttle respected across two calls.

5. Docs and bookkeeping

  • Roadmap: mark the wiring item IMPLEMENTED with plan link (soul, post-merge DONE edit as usual).
  • docs/site/client/index.md and architecture/index.md: one line each where mark_backlinks/graph persistence is described ("seeded from the published /graph.md").
  • ADR: not needed (no new convention; consumes ADR 0004's format). Note in the PR description instead.

Out of scope

  • TUI graph-view seeding (worthwhile follow-up; keep this PR to the MCP surfaces the roadmap item names).
  • Parsing the hub graph's Exported timestamp for freshness arbitration (local-wins does not need it).
  • Publish-time edge extraction in the store (the "later upgrade" per the roadmap; backend-parity applies there).
  • demarkus-library floor changes (its own repo; it already reads /graph.md).

Risks

  • Seed data quality: the hub aggregate may contain hosts unreachable from this client (cluster-internal names in the knowledge system's graph). Harmless for backlinks (they are just labels); mark_graph crawls starting from them will error per node as today. Do not filter by reachability.
  • Large /graph.md: the enriched table grows rows, not new fetch cost (single doc). ParseExport is linear; the 1 MiB protocol body cap bounds it. No action.
  • Merge interplay: SeedFromExport must not fight the new Merge replace-refreshed-sources logic; the shared authoritative-source rule (status + CrawledAt) is the single arbiter, tested from both directions.
  • Etag identity: /graph.md is republished wholesale with retention 20; etag changes on every publish even if content is identical except the Exported line, so not-modified hits only between publishes. Accept; the throttle bounds the cost.

Verification

  1. Unit suites: client graphstore, fetch, demarkus-mcp; broker package. bash pre-commit.sh.
  2. Live cold-start check: build demarkus-mcp, delete (or point HOME at a scratch dir for) ~/.mark/graph.json, attach to the soul, call mark_backlinks on a doc known to have hub-graph backlinks; expect answers with zero crawls. Then mark_graph a subtree and confirm local results win and persist.
  3. Broker check via its test harness (no cluster needed): cold gateway seeds from a mocked world /graph.md.
  4. Legacy: a /graph.md still in two-column form seeds correctly (ParseExport dual-format already tested; add one seed test using a legacy fixture).

Implementation notes (2026-07-14)

Implemented as planned, with these deviations and additions:

  • URL canonicalization fix (unplanned, required). The first live cold-start check failed: the soul's /graph.md keys rows on canonical mark://soul.demarkus.io:6309/... URLs, but mark_backlinks built its lookup key as defaultHost + path with no default-port normalization, and the plugin's -host flag omits the port. Fixed by canonicalizing through resolveURL ("mark://" + host + path) in markBacklinks, markGraph's start URL, and explore's backlinks section. This was a latent pre-existing mismatch (portless -host crawls and full-URL queries could already disagree); seeding surfaced it. Regression test: TestSeedGraph_DefaultHostWithoutPortCanonicalizes.
  • Step 2 test harness. The "existing mock-stream harness" is server-side only, so FetchConditional got the client's first wire-level test: an in-process QUIC listener with a self-signed cert (client/fetch/conditional_test.go) asserting etag sent, not-modified passthrough, and stale-etag refetch.
  • Step 4 dispatcher surface. Adding FetchConditional to worldDispatcher/worldPool was trivial (passthrough to fetch.Client), so the broker sends real if-none-match instead of comparing etags after a full fetch. The broker's seed etag lives in the in-memory graphstore's seed_etags map (keyed by worldName); only the throttle map lives on the gateway.
  • Shared helpers. Merge's drop-refreshed-edges and upsert logic were extracted (dropEdgesFromLocked, upsertEdgeLocked) so SeedFromExport and Merge share one implementation of the arbitration rule.
  • Live verification passed (2026-07-14): scratch-HOME cold start against the soul answered 4 backlinks for /patterns.md with zero crawls; 133 nodes / 190 edges seeded from the still-legacy two-column /graph.md (v6); seed etag persisted; all seeded nodes carried zero CrawledAt. A depth-1 mark_graph of /conventions.md flipped it authoritative (real CrawledAt) and its enriched crawl edges replaced the seeded legacy row in backlinks output, with the other 130 nodes untouched.

Review round (PR 253, CodeRabbit)

Three findings, all accepted:

  • Seed the resolved host, not defaultHost. The plan's "seed source is the attached world" left full mark:// URLs against other hosts unseeded. seedGraph now takes the canonical host from resolveURL (throttle, etag, and token resolution were already per-host, and the broker already seeded the URL's world). Test: TestSeedGraph_SeedsResolvedHostNotDefault.
  • Stale-edge gap in SeedFromExport. The drop set was built only from seed-edge Froms, so a re-seed where a source's outgoing set went to zero left its old seeded edges behind. Now every observed (real-status) non-authoritative seed node also joins the drop set, mirroring Merge's refreshed criterion; the status predicate is extracted as observedStatus and shared by Merge, authoritativeLocked, and SeedFromExport. Tests: TestSeedRefreshDropsEdgesOfEmptiedSource plus the counter-case TestSeedKeepsEdgesOfUnobservedSeedNode (an error-status seed node with no edges must not drop what it never read).
trail
  1. soul.demarkus.io:6309 v4