# Plan: Wire backlinks/graph tools to the published /graph.md (hub seeding)

Status: planned 2026-07-13, implementation scheduled next session. Closes roadmap "Wire backlinks/graph tools to the published /graph.md" (gap 1 of the 2026-07-13 knowledge-layer analysis).

## Context

The federation agent publishes the aggregated link graph to each hub at /graph.md (now with enriched six-column edges after the edge-semantics work, PR 251), but mark_backlinks and mark_graph answer only from the local crawl cache: ~/.mark/graph.json for demarkus-mcp, a per-pod ephemeral store for the broker. A fresh client or a recycled broker pod answers from an empty graph while an authoritative aggregate sits on the hub unread. The accumulation exists; the query path bypasses it.

Client-only change. No server, protocol, or store-backend work. ParseExport (currently test-only) becomes the production consumer of /graph.md.

## Design decisions

- **Seed source is the attached world.** demarkus-mcp seeds from `defaultHost + "/graph.md"` (the -host world; the soul server publishes its own /graph.md). The broker seeds per world from `mark://{world}/graph.md` through dispatchWithAuth. No new flags; a world without /graph.md degrades silently to today's behavior (not-found is not an error).
- **Local wins.** The hub aggregate carries no per-node freshness, so locally crawled data is never overwritten by seed data. A stored node counts as locally authoritative iff its status is not one of "", "external", "error" AND CrawledAt is non-zero. Seeded nodes get zero CrawledAt, which is the durable "seeded, not locally observed" marker across restarts.
- **Seeding fills gaps only**: nodes inserted when absent or when the existing node is non-authoritative; edges replace the outgoing set only for sources that are not locally authoritative. A later local crawl of a seeded source replaces its edges through the existing Merge refresh logic.
- **Conditional refresh with our own etag.** fetch.Client's built-in if-none-match rides the unauthenticated disk cache, which token'd souls skip, so the seeder stores the last /graph.md etag itself: new `seed_etags` map (host -> etag) in the graph.json envelope, additive JSON, schema stays v1. Requests send if-none-match; not-modified means skip parse and merge.
- **Throttled per process.** A conditional check costs one round trip; still, cap at one check per host per 5 minutes (package constant), process-scoped like the fetchdedup session state. First graph-tool call in a session always checks.
- **Never fatal.** Any seed failure (fetch error, malformed document, oversized) logs at warn and falls through to the local store. Seeding must never make backlinks worse than today.

## Steps

### 1. graphstore: seed support (client/graphstore/store.go, export.go)

- `document` envelope gains `SeedEtags map[string]string` with `json:"seed_etags,omitempty"`; Store carries it; Load/Save round-trip it (nil-safe for legacy files).
- `SeedEtag(host) string` and `SetSeedEtag(host, etag)` accessors (locked).
- `SeedFromExport(nodes []StoredNode, edges []StoredEdge) (added int)`:
  - classify locally authoritative sources (status not in {"", "external", "error"} and CrawledAt non-zero);
  - insert nodes when absent or existing node non-authoritative, forcing CrawledAt to zero on the seeded copy;
  - drop stored edges whose From is a seeded (non-authoritative) source being refreshed, then insert seed edges for those sources, Count normalized, dedup on edgeKey;
  - never touch nodes or edges of authoritative sources.
- Optionally parse the `> Exported:` header line in ParseExport later; NOT in scope (freshness rule does not need it).

Tests: seed into empty store; local-wins (authoritative node and its edges untouched); seeded-then-crawled source flips to authoritative and Merge replaces its edges; seed refresh replaces prior seeded edges (no stale seeded backlinks); seed_etags round-trips through Save/Load; legacy file loads with nil map.

### 2. fetch: conditional fetch with explicit etag (client/fetch/fetch.go)

`cachedRequest` already threads `extra` metadata internally. Export a minimal surface:

```go
// FetchConditional fetches with an if-none-match etag; status not-modified
// returns with an empty body.
func (c *Client) FetchConditional(host, path, token, etag string) (Result, error)
```

Implemented via the existing extra-metadata path (option-bearing requests already skip the disk cache, which is correct here). Test with the existing mock-stream harness: etag sent, not-modified passthrough.

### 3. demarkus-mcp: seed hook (client/cmd/demarkus-mcp/main.go, explore.go)

- `handler` gains `graphSeed` state: `map[string]time.Time` last-checked per host + mutex (process-scoped, mirrors fetchdedup's session scoping).
- `seedGraph(host string)`: throttle check; `FetchConditional(host, "/graph.md", token, store.SeedEtag(host))`; on ok: `ParseExport` then `SeedFromExport`, `SetSeedEtag`, `Save()`; on not-modified/not-found/error: return silently (warn-log real errors).
- Call `seedGraph(defaultHost)` at the top of markBacklinks, markGraph, and explore's writeBacklinksSection (before the store read; markGraph seeds so depth-limited crawls still benefit from hub context).
- Tool descriptions: one sentence on mark_backlinks/mark_graph ("seeded from the world's published /graph.md when available; local crawls take precedence"). No em dashes.
- The "No backlinks found ... run mark_graph" hint stays as the empty-store fallback text.

Tests (mock fetch func in main_test.go style): cold store + hub graph answers backlinks without a crawl; hub 404 degrades to the current hint; local crawl beats conflicting seed rows; second call within the throttle window does not refetch; etag round-trip sends if-none-match.

### 4. Broker: per-world seeding (tools/demarkus-broker/internal/broker/mcp_tools_graph.go, mcp_tools_explore.go)

- Gateway gains `graphSeed` map keyed by worldName {etag string, checked time.Time} + mutex (per pod, like graphStore).
- `seedWorldGraph(ctx, claims, worldName)`: throttle; dispatchWithAuth Fetch of `/graph.md` (send if-none-match via the dispatcher's metadata path or accept a full fetch and compare the stored etag; pick during implementation, do not add dispatcher surface unless trivial); ParseExport + SeedFromExport + record etag. Silent degrade.
- Call from handleMarkBacklinks, handleMarkGraph, and explore's backlinks section, scoped to the world parsed from the tool URL.
- MCP-API.md: note the graph store is seeded from each world's published /graph.md on demand, so cold pods answer backlinks; local crawls still take precedence; ephemerality note stays (the seed makes restarts cheap, not durable).

Tests: cold gateway + world with /graph.md answers backlinks with no prior crawl (this is the headline behavior); world without /graph.md keeps today's empty hint; seeded then crawled world prefers crawl results; throttle respected across two calls.

### 5. Docs and bookkeeping

- Roadmap: mark the wiring item IMPLEMENTED with plan link (soul, post-merge DONE edit as usual).
- docs/site/client/index.md and architecture/index.md: one line each where mark_backlinks/graph persistence is described ("seeded from the published /graph.md").
- ADR: not needed (no new convention; consumes ADR 0004's format). Note in the PR description instead.

## Out of scope

- TUI graph-view seeding (worthwhile follow-up; keep this PR to the MCP surfaces the roadmap item names).
- Parsing the hub graph's Exported timestamp for freshness arbitration (local-wins does not need it).
- Publish-time edge extraction in the store (the "later upgrade" per the roadmap; backend-parity applies there).
- demarkus-library floor changes (its own repo; it already reads /graph.md).

## Risks

- **Seed data quality**: the hub aggregate may contain hosts unreachable from this client (cluster-internal names in the knowledge system's graph). Harmless for backlinks (they are just labels); mark_graph crawls starting from them will error per node as today. Do not filter by reachability.
- **Large /graph.md**: the enriched table grows rows, not new fetch cost (single doc). ParseExport is linear; the 1 MiB protocol body cap bounds it. No action.
- **Merge interplay**: SeedFromExport must not fight the new Merge replace-refreshed-sources logic; the shared authoritative-source rule (status + CrawledAt) is the single arbiter, tested from both directions.
- **Etag identity**: /graph.md is republished wholesale with retention 20; etag changes on every publish even if content is identical except the Exported line, so not-modified hits only between publishes. Accept; the throttle bounds the cost.

## Verification

1. Unit suites: client graphstore, fetch, demarkus-mcp; broker package. `bash pre-commit.sh`.
2. Live cold-start check: build demarkus-mcp, delete (or point HOME at a scratch dir for) ~/.mark/graph.json, attach to the soul, call mark_backlinks on a doc known to have hub-graph backlinks; expect answers with zero crawls. Then mark_graph a subtree and confirm local results win and persist.
3. Broker check via its test harness (no cluster needed): cold gateway seeds from a mocked world /graph.md.
4. Legacy: a /graph.md still in two-column form seeds correctly (ParseExport dual-format already tested; add one seed test using a legacy fixture).