Knowledge graph completeness analysis (2026-07-15)

Code-level survey of the graph pipeline after the July enhancements (edge semantics PR #251, hub seeding PR #253/#254, broker follow-ups #256/#257, library ParseExport, fedcrawl H1 titles #259), answering: what remains for a complete, optimized graph. Findings verified against source with file:line refs.

What the recent work closed

• Typed edges with provenance: Edge{From, To, Rel, Label, Anchor, Count}, rel- metadata convention, ingested by both crawlers.
• Cold-start topology: SeedFromExport from hub /graph.md with local-wins arbitration, seed etags, FetchConditional; client MCP and broker (all worlds, world-name translation).
• Producer-consumer contract golden between agent export and broker seed.
• Title propagation into the hub graph (metadata title, then H1 fallback).
• Library floor consumes graphstore.ParseExport instead of a hand-rolled parser.

Tier 1: correctness and scale walls (new findings, not yet on the roadmap)

The 1 MiB export wall, failing silently

/graph.md is one monolithic export (client/graphstore/export.go:28-80); the protocol caps bodies at 1 MiB (protocol/request.go:27). When the aggregate outgrows the cap, the agent's publish is rejected (server handler.go:751) and the caller only warn-logs (demarkus-agent/main.go:189), so the hub serves a frozen last-good graph forever. The enriched 6-column edge rows make this arrive sooner. Needs sharding or pagination of the export (per-world or per-prefix shards, a /graph/ directory with an index) plus a loud failure mode. Biggest single gap: past the threshold the whole seeding layer quietly stops improving.

Tombstone accumulation, no eviction

graphstore has no delete or evict path at all (client/graphstore/store.go). Archived and not-found nodes are rewritten with their status but never removed, and every inbound edge to them persists (Merge only drops refreshed sources' outgoing edges, store.go:246-275). graph.json grows monotonically; the fedcrawl aggregate self-heals per run but still emits orphan edge targets with no node row for archived docs that live docs still link to (fedcrawl/crawl.go:486). Needs: an eviction policy (drop archived/not-found nodes after N crawls or T days, prune edges whose endpoints are gone) and a doctor-style compaction.

Exported-timestamp arbitration (known deferral, confirmed real)

export.go:56 writes the Exported header; ParseExport (export.go:100-176) never reads it. Arbitration is presence-based only: a locally crawled node never re-seeds even when the hub is newer, so local rows freeze against a fresher aggregate until the next manual crawl. Deferred out of the hub-seed plan; the survey confirms it matters once seeds refresh regularly.

URL identity is not normalized consistently

Only fedcrawl normalizes ports (crawl.go:513-522); the client crawler and graphstore key on exact strings, so mark://h/x and mark://h:6309/x can be duplicate nodes depending on ingestion path. The broker hit exactly this class of bug (PR #256 dial-address translation). Needs one canonicalization helper applied at every graph write, client and agent both.

Tier 2: the graph is stored but barely queried

• No rel filtering anywhere: mark_backlinks/mark_graph cannot ask "what supersedes this" even though typed edges exist (store.go:407-430). Cheap, high leverage now that rel edges are real.
• No predicate vocabulary: rel is free-form (supersedes vs supersede are distinct edges). The plugins should curate the small vocabulary already suggested in the edge-semantics follow-ups.
• Backlinks sort alphabetically, not by degree/importance (store.go:377-390); in-degree ranking exists only in the TUI view.
• Rel edges do not count toward LinkCount (graph crawl.go:164) and fedcrawl does not discover servers via rel targets (crawl.go:490-496).
• No path queries or centrality; Neighbors/InDegrees only.

Tier 3: known roadmap gaps, unchanged by this round

Still open exactly as recorded in /roadmap.md Knowledge Layer section: semantic recall beyond tags (tsvector then pgvector), universe-wide LOOKUP at the broker, derived ranking signals, staleness at lookup time, publish-time edge extraction in the store (backend-parity rule applies). The survey adds force to derived ranking: catalog and graph are fully disjoint subsystems today (server catalog.go vs client graph), so in-degree never informs LOOKUP importance, which is the cheapest ranking win available.

Tier 4: optimization polish

• Every query is an O(E) scan; no adjacency index (graph.go:126-173, store.go:383/412). Fine at hundreds of edges, wrong shape for the scale Tier 1 implies.
• graph.json is pretty-printed (store.go:180), roughly 2x disk and marshal cost; Save holds RLock across disk I/O; seed then save is not atomic.
• TUI never seeds from the hub (no SeedFromExport calls in demarkus-tui), so standalone TUI users get no hub topology until they crawl at depth 10.

Suggested order

1. Export sharding + loud publish failure (unblocks everything at scale).
2. Eviction/compaction for tombstones and orphan edges.
3. Exported-timestamp arbitration in SeedFromExport.
4. URL canonicalization shared by all writers.
5. Rel filter on backlinks/graph + curated predicate vocabulary (cheap, makes typed edges pay rent).
6. Derived ranking: in-degree into LOOKUP ordering (first graph-catalog crossing).
7. Adjacency index + compact JSON when store sizes warrant.

Tier 3 items keep their roadmap standing; 6 here is the same item as the roadmap's derived-ranking gap.