soul.demarkus.io:6309/journal/2026-07-14.md/v12 draft reader meta

Journal 2026-07-14

Graph hub seeding implemented (branch graph-hub-seed, PR pending)

Implemented /plans/graph-hub-seed.md end to end: graphstore SeedFromExport + persisted seed_etags (additive, schema v1), fetch.FetchConditional with explicit if-none-match, seedGraph hooks in demarkus-mcp's mark_backlinks/mark_graph/mark_explore, and per-world seedWorldGraph on the broker through dispatchWithAuth (FetchConditional added to worldDispatcher; trivial passthrough). Local-wins rule held as designed: authoritative = real fetched status plus non-zero CrawledAt, seeded nodes carry zero CrawledAt, and the rule is shared with Merge via extracted dropEdgesFromLocked/upsertEdgeLocked helpers. Never fatal, 5-minute per-host throttle. ParseExport's first production caller; the soul's live /graph.md is still two-column legacy (v6) and seeded fine.

The interesting find: the first live cold-start check returned no backlinks despite a successful seed. Root cause was URL canonicalization, not seeding. The hub aggregate keys rows on canonical mark://host:6309 URLs while mark_backlinks built its key as defaultHost + path, and the plugin's -host flag omits the port. A latent pre-existing mismatch (portless -host crawls vs full-URL queries) that seeding surfaced. Fixed by canonicalizing through resolveURL in markBacklinks, markGraph's start URL, and explore's backlinks section, with a regression test pinning the portless -host case. Recorded in the plan's implementation notes.

Verification: full client + tools suites green, pre-commit clean, live cold start (scratch HOME) answered 4 backlinks for /patterns.md with zero crawls (133 nodes / 190 edges seeded, etag persisted, all zero CrawledAt), then a depth-1 crawl of /conventions.md flipped it authoritative and its enriched edges replaced the seeded row while 130 nodes stayed seeded. FetchConditional got the client's first wire-level test (in-process QUIC listener, self-signed cert).

Roadmap updated (seeding item IMPLEMENTED; flip to DONE on merge). PR 251 bookkeeping (edge-semantics DONE flip, debt.md fetch-callback removal) was already done last session. Changes staged; Fritz commits.

Out of scope confirmed untouched: TUI seeding, Exported-timestamp arbitration, publish-time edge extraction.

PR 253 review round

CodeRabbit posted three findings; all valid, all fixed (details appended to the plan doc): seedGraph now seeds the resolved host from the tool URL instead of only defaultHost, and SeedFromExport's edge-drop set now includes observed seed nodes so a source whose outgoing set went to zero sheds its stale seeded edges (Merge's refreshed criterion, shared via a new observedStatus helper). One test-only guard: seedGraph tolerates a nil client since partial handlers in tests construct without one. Suites and pre-commit green, changes staged.

Issue 222 fix (branch fix/graphstore-merge-preserve-resolved)

Fixed the Merge title/status regression from issue 222: a crawl node with an unobserved status (error/external/blank) no longer clobbers a stored node; the stored copy stays untouched, keeping its CrawledAt so it remains authoritative against seeds. Never-seen nodes are still recorded. Reused the observedStatus predicate from the seeding review round, so Merge's node upsert, its edge-refresh set, and SeedFromExport all share one definition of "actually read". This also closes the seeding amplification path: a stalled federation-agent crawl can no longer publish blanked titles into hub /graph.md (fedcrawl merges through the same code). Tests pin the preserve case, the still-authoritative-against-seed interplay, and that new unfetched nodes are still recorded. Suites and pre-commit green, staged; Fritz commits.

PR 254 merged, day closed out

PR 254 landed as 8fe9b98 (squash), closing issue 222; released as client v0.21.1. Final review round: accepted the Merge contract-comment correction and the stronger CrawledAt-equality test assertion; declined two findings with posted rationale (ok-with-blank title is affirmative "no H1", and zeroing CrawledAt on never-seen error nodes is moot since observedStatus gates authoritative; the "critical" severity on the latter was factually wrong and TestSeedFromExportOverwritesNonAuthoritative pins the behavior). Decline replies posted to the PR and CodeRabbit threads resolved on request. Roadmap graph-hub-seeding entry flipped to DONE with both PRs recorded. Pin-bump workflow dispatched again for client 0.21.1; note for next time: the release event never triggers it (workflow-token limitation), so after any merge-day release either dispatch it manually or wait for the morning cron.

Broker seed URL translation (branch fix/broker-seed-url-translation)

Post-deploy verification of the knowledge system found broker seeding inert in the real topology: the federation agent keys the hub aggregate's rows by cluster-internal dial addresses (name.namespace.svc.cluster.local:6309), but every gateway query keys on world names and the tool URL grammar rejects port-carrying hosts, so seeded rows were unreachable by any legal query. The plan's "seed data quality: harmless labels" risk assessment was wrong for the broker; the demarkus-mcp surface was unaffected (host-form keys match end to end, verified live against the soul).

Fix: seedWorldGraph now translates node/edge URLs whose host matches a configured world's internal dial address (resolveWorldAddress, the same mapping mark_worlds prints) into mark://{worldName}/... form before SeedFromExport; unknown hosts stay as-is since they are labels. Test pins the production-shaped aggregate answering world-name backlinks and the external URL staying untouched. MCP-API.md notes the translation. Suites and pre-commit green, staged.

Deploy verification otherwise: broker healthy, agent publishing (root /graph.md v748, retention working), per-world degrade correct. Pod image confirmation still pending gcloud reauth. Root's aggregate carries blank titles for soul-world docs, issue 222 live in the wild; the new agent stops further regression but existing blanks heal only after a clean crawl. After this fix merges, the broker needs one more image bump.

Broker seeds all worlds (branch fix/broker-seed-all-worlds)

Closed the last seeding wrinkle: graph-tool calls now seed every configured world through seedGraphStore instead of only the queried one, so a cold pod answers non-hub backlinks on the very first query. Chose seed-all over a hub config flag: no new config surface, world count is tiny, non-hub worlds not-found instantly, and the per-world throttle bounds the cost. Test pins a team-a query finding the aggregate on a different (hub) world with one check per configured world. MCP-API.md updated. Green, staged. One more broker release + deploy bump after merge.

Producer-consumer contract test for /graph.md

Built the cross-module contract test from the post-mortem, riding fix/broker-seed-all-worlds. Producer half: fedcrawl's TestGraphExportContract drives the real crawler over an in-cluster-shaped topology (service-DNS seed, external and loopback links, a rel-supersedes edge) and pins the exact export body as a golden (testdata/graph-export-incluster.md, regenerate with -update; the Exported timestamp is normalized). Consumer half: the broker suite reads the same golden by relative path and asserts a cold gateway answers world-name backlinks from it, with typed-edge provenance intact and an addressability invariant (no .svc.cluster.local rows survive translation). fedcrawl is client-internal so the broker cannot import it; the committed golden is the shared contract artifact, storetest-style. Either side drifting fails a build, which is exactly the class of slip that cost two extra release+deploy cycles today.

Day wrap-up

Graph hub seeding went from plan to fully deployed and live-verified in one day, across four monorepo PRs (253 feature, 254 issue 222, 256 broker seed URL translation, 257 seed-all-worlds + contract test), four releases (client 0.21.0 to 0.21.2, tools 0.12.0 to 0.12.4), two plugin pin bumps, and three deploy-repo bumps (broker 0.12.2/0.12.3/0.12.4, agent 0.21.1). Final live state: cold demarkus-mcp clients answer soul backlinks with zero crawls, and a cold broker pod's very first graph call answers non-hub backlinks from the hub aggregate with translated keys and enriched provenance.

Retrospective: PRs 256 and 257 were avoidable; the plan's "cluster-internal names are harmless labels" assessment went unchallenged and the broker fixtures encoded it, so production topology refuted it only after deploy. The producer-consumer contract golden (fedcrawl generates, broker consumes) now closes that class at PR time; full post-mortem in /debugging.md. CodeRabbit now enforces the terse-comments guideline at review ("Source: Coding guidelines"), which caught four of my comments today; a PostToolUse hook remains the option for catching it mid-session.

Bookkeeping current: roadmap (seeding DONE with all four PRs + deploy verification), plan doc (implementation notes + review rounds), index hub (Completed Plans entry), debugging.md (fixtures lesson), deploy-repo journal (bumps + verifications). Plugin pins at server 0.21.0 / client 0.21.1 / tools 0.12.1 via PR 255; the morning cron will sweep up 0.21.2/0.12.4 (test-only and broker-only, nothing plugin-relevant). No open threads.

trail
  1. soul.demarkus.io:6309 v12