# Journal — 2026-03-08

## 2026-03-08 — Federation Phase 1 Shipped

`mark_index` and `mark_resolve` MCP tools merged. Agents can now crawl servers to build hash indexes and resolve content by hash across the federation.

Key features:
- `mark_index`: crawls source server, publishes hash index to hub, manifest checks (hard block + force override), dry_run, merge with existing aggregated indexes, truncation warnings at 1000 docs
- `mark_resolve`: looks up hash in hub index, tries each server, verifies content-hash on response
- `client/internal/index` package: Parse, Build, Merge with server URL canonicalization

PR review hardened the implementation:
- Sentinel error for truncated indexes (visible warning instead of silent)
- Server URL canonicalization (trailing slash, default port, case)
- Negative expected_version validation
- Source path used as crawl root (subtree indexing)
- Hash validation on crawled content-hash values
- Fail on existing index fetch failure (don't silently drop other servers' entries)
- Aggregated index uses target URL in Source header
- Extracted checkManifests to fix cyclomatic complexity
- Tool parameter descriptions use urlDesc() for bare path support

"The Agent as the Librarian" — core architectural principle documented. Server is a bookshelf, agent is the librarian.

---

## 2026-03-08 — Persistent Graph Store + Backlinks

Shipped the persistent graph store and backlinks — Phase 4's first functional increment after the Hub pattern.

### What Shipped

- **`client/internal/graphstore` package** — new persistent graph store at `~/.mark/graph.json`. `StoredNode` tracks URL, title, status, link count, etag, and crawl timestamp. `StoredEdge` tracks directed links. Atomic writes (`.tmp` + `os.Rename`), schema versioning (`version: 1`), incremental merge with deduplication.
- **`EtagFetcher`** — shared adapter implementing `graph.Fetcher` that collects etags during crawl. Replaces duplicated etag collection in MCP and TUI.
- **`CrawlAndPersist`** — unified method on `*Store` that runs `graph.Crawl`, merges results + etags, and saves. Nil-safe (nil store = crawl without persist). All three clients (CLI, TUI, MCP) now share this single code path.
- **`mark_backlinks` MCP tool** — reverse edge lookup from the persistent graph. "What links here?" Returns sorted markdown list with document titles.
- **TUI graph seeding** — graph view loads instantly from stored graph while background crawl runs. No more blank screen on graph toggle.
- **CLI persistence** — `demarkus graph` now also persists to the store (was missing before).
- **10 unit tests** — Load, Save, Merge, Backlinks, ToGraph, CrawlAndPersist, nil-safety, atomic writes.

### The Sharing Evolution

This increment went through three rounds of consolidation at Fritz's push for cohesive client design:

1. **First pass**: Duplicated etag collection + merge/save in both MCP and TUI
2. **Second pass**: Extracted `EtagFetcher` into graphstore, shared by MCP and TUI
3. **Third pass**: Extracted `CrawlAndPersist` into graphstore, shared by all three clients

The principle: CLI, TUI, and MCP are all clients. Core fundamentals must be shared. Differences are fine when unavoidable, but the default is elegant reuse.

### Also This Session

- **Dropped native Windows builds** — removed `windows` from `goos` in both `.goreleaser.yml` files. WSL users run Linux binaries; native Windows builds were untested overhead.
- **Updated roadmap** — Phase 3 marked COMPLETE (federation was already shipped but roadmap hadn't been updated). Phase 4 now shows persistent graph, backlinks, CrawlAndPersist, and graph seeding as Done.

### What's Next

- **Graph as content** — export the crawled graph as a markdown document with `mark://` links, publish it to a server, others fetch and import it
- **Graph-aware navigation** — TUI shows explored topology, graph proximity for related docs
- **Agent discovery** — agents crawl the graph to build knowledge maps

### Note to Future Sessions

`CrawlAndPersist` is the entry point for all graph crawling. Don't bypass it — it handles etag collection, merge, and atomic save. If the store is nil, it still works (just doesn't persist). The `EtagFetcher` is also in graphstore, not in the graph package — it's a persistence concern, not a graph concern.

---

## 2026-03-08 — Persistent Graph: Copilot Review Hardening

Post-merge review pass on the persistent graph store. Copilot raised ~15 comments; about half were valid fixes, the rest were over-engineering or duplicates.

### Fixes Applied

1. **Data race in `CrawlAndPersist`** — `nodeCount` for MaxNodes cap was a plain `int` incremented from concurrent crawler goroutines. Fixed with `atomic.Int32`. Race detector confirmed clean. Test's `OnNode` callback had the same race — fixed too.
2. **Error wrapping in `Load`** — read/parse errors returned raw without file path context. Wrapped with `fmt.Errorf("read graph store %q: %w", ...)` and `"parse graph store %q: %w"`, matching the bookmarks store pattern.
3. **Schema version validation** — `schemaVersion` constant existed but was never checked on load. Added `doc.Version != schemaVersion` guard with clear error message.
4. **MaxDepth semantics mismatch** — `graphstore.CrawlOptions` documented `0 = default 2` but `graph.CrawlOptions` uses `0 = start node only`. Initially added a 0→-1 translation, then Copilot correctly pointed out this breaks `-depth 0` from CLI. Reverted to pass-through with corrected comment: `0 = start node only, -1 = default 2`.
5. **Graph store load errors surfaced** — CLI now warns to stderr, TUI shows in status bar (preserves both bookmark + graph errors with `|` separator), MCP logs with `log.Printf`.
6. **MCP handler holds `graphStore` field** — instead of loading from disk on every `mark_graph`/`mark_backlinks` call, store is loaded once at startup and shared. Enables testability.
7. **Happy-path backlinks test** — added `TestHandlerMarkBacklinks_HappyPath` with pre-populated temp store, asserting returned titles and URLs. Also `TestHandlerMarkBacklinks_NilStore`.
8. **Tool description accuracy** — `mark_graph` description now says "When a local graph store is available" instead of unconditionally claiming persistence.
9. **Displaced comment** — `assertIsToolError` doc comment had drifted above a test function after insertion. Moved back.

### Copilot Comments Rejected

- **Loop variable aliasing** (×2) — Copilot claimed `n := doc.Nodes[i]; &n` aliases all entries. Wrong: `:=` creates a copy, and Go 1.22+ scopes loop vars per iteration.
- **Windows `os.Rename` doesn't overwrite** — not applicable, we dropped Windows native builds this session.
- **URL validation in backlinks** — "no backlinks found" is the correct answer for any URL not in the graph, regardless of format.
- **Flat tree from store** — valid observation (stored nodes have depth 0), but intentional trade-off. Flat list is still useful as instant preview; crawl provides depth.

### What I Learned

The data race was the most important catch. The `nodeCount++` inside `OnNode` was called from multiple crawler goroutines — a real `-race` failure, not theoretical. `atomic.Int32` is the right fix: no mutex needed, the counter is independent of other state.

The MaxDepth saga was instructive: Copilot's first comment (translate 0→-1) was right about the documentation mismatch but wrong about the fix. Its second comment (revert the translation) was right about the fix. The lesson: when two layers have different zero-value semantics, the wrapper should either document and pass through, or fully own the translation. Half-translating creates a third set of semantics that nobody understands.

### Note to Future Sessions

Graph store is now hardened: race-free, error-wrapped, version-checked, properly surfaced in all three clients. The MCP handler holds the store as a field — tests can inject pre-populated stores directly.


---

## 2026-03-08 — Graph as Content & Documentation Updates

### Graph as Content (Phase 4)

Implemented graph export as publishable markdown — the graph becomes a document you can publish to a demarkus server and others can crawl to discover the topology.

**New files:**
- `client/internal/graphstore/export.go` — `Export()` renders the store as markdown with mark:// links in tables; `ParseExport()` parses it back
- `client/internal/graphstore/export_test.go` — 4 tests: export, empty, parse, round-trip (with escaped pipes and backslashes)

**CLI:** `demarkus graph export [-o file.md]` — exports stored graph to stdout or file
**MCP:** `mark_graph_export` — returns graph as markdown for agents to publish

**Design insight:** No special import function needed. Crawling the exported doc discovers all the mark:// links naturally — the same link extraction the crawler uses parses the graph document. This matches the DESIGN.md vision.

**Copilot review fixes:**
- Cell escaping for titles containing `|` and `\` (escape backslashes first, unescape in reverse order)
- Regex captures link destination `(url)` not display text `[label]` for canonical URL
- Reject extra positional args in `graph export` subcommand
- Rejected: strconv.Atoi overflow (regex constrains to digits), URL special chars (mark:// doesn't use them)

### Documentation Updates

Updated 5 in-repo docs on main to document persistent graph store and backlinks:
- `README.md` — tool descriptions mention persistent graph and backlink queries
- `docs/DESIGN.md` — Phase 4 implementation status block
- `docs/site/client/index.md` — graph persistence, MCP tools list
- `docs/site/architecture/index.md` — expanded Document Graph section
- `docs/site/philosophy/agent-cookbook.md` — new backlinks discovery recipe

## 2026-03-08 — Phase 4: Graph-Aware Navigation Complete

Shipped the last Phase 4 feature: graph-aware navigation in the TUI.

**What shipped:**
- Three sub-views in graph mode: Links (`d`), Backlinks (`r`), Topology (`t`)
- Link density indicators `[N←]` showing backlink counts per node
- Topology view: all explored nodes sorted by importance (backlink count descending)
- `graphstore.BacklinksEnriched()` — shared enrichment logic used by both TUI and MCP
- `graph.InDegrees()` — O(E) single-pass in-degree computation, replaces per-node O(N*E) lookups
- Rune-safe truncation (`truncateRunes()`) for multi-byte UTF-8 status icons
- Null-safe sub-view switching during active crawls

**Design decisions:**
- Removed `Store.InDegrees()` to avoid duplicating `graph.Graph.InDegrees()` — callers use `store.ToGraph().InDegrees()` instead (topology view is user-triggered, not a hot path)
- `graph.InDegrees()` guards against phantom nodes (edges to URLs never added as nodes, e.g. from cancelled crawls)
- `AllNodes()` exists on both Graph and Store but returns different types (`*Node` vs `StoredNode`) — not duplication

Phase 4 marked COMPLETE on roadmap. Remaining item: Agent discovery (not started).