# Debugging

Lessons learned from bugs, investigations, and things that went wrong.

## General Principles

- Read the error message. The whole error message.
- Reproduce before fixing. If you can't trigger it, you don't understand it.
- Don't brute force. If something is blocked, step back and think about *why*.
- When a test fails, read the test first: the test might be wrong.

## Known Gotchas

### YAML Auto-Typing
YAML will silently convert strings like `yes`, `no`, `on`, `off` to booleans. This is why frontmatter is parsed as `map[string]string`; we control the type conversion ourselves.

### Symlink Resolution in Tests
When testing the versioned store, be careful with `os.ReadFile` vs following symlinks. The store creates `doc.md -> versions/doc.md.v3`; some operations need the symlink target, others need the symlink itself.

### Path Traversal
`filepath.Clean` handles most cases but you still need an explicit `..` check after cleaning. A path like `/../../../etc/passwd` cleans to `/etc/passwd` which looks valid but is outside the content root. The check is: clean the path, then verify it doesn't escape the root.

### QUIC Stream Lifecycle
Streams must be closed properly. If a handler panics or returns without closing the stream, the client hangs waiting for data. Always defer stream close.

## Debugging Approach for This Project

1. Write a failing test that reproduces the issue
2. Fix the code
3. Verify the test passes
4. Check if the fix breaks anything else (`make test`)
5. Run `bash pre-commit.sh`

The test-first approach isn't dogma here: it's practical. A test that reproduces the bug proves you understand the bug, and it prevents regression.

### QUIC UDP Buffer Size Warning

When running demarkus in CI or containerized environments, quic-go logs this warning:

```
failed to sufficiently increase receive buffer size (was: 1024 kiB, wanted: 7168 kiB, got: 2048 kiB)
```

**Not a failure.** quic-go tries to increase the UDP receive buffer to 7168 kiB for optimal performance. In environments with restricted sysctl limits, it can only get partway there. The connection still works: everything publishes fine.

**Fix (if needed):** Increase the system limit with `sysctl -w net.core.rmem_max=7168000`. Not necessary for correctness, only for optimal throughput under heavy load.

**First seen:** CI publishing content to demarkus-hub (`hub.demarkus.io`).


### OnNode Callback Concurrency
`graph.Crawl` invokes `OnNode` from multiple worker goroutines. Any mutable state captured by the callback must use `sync/atomic` or a mutex. The `CrawlAndPersist` MaxNodes counter uses `atomic.Int32` for this reason. Always run `go test -race` on packages that use `OnNode`.


### Glamour WithAutoStyle() Blocks TUI Rendering

**Symptom:** TUI shows "Loading..." for up to 60 seconds after fetch completes. Server logs confirm the FETCH is handled immediately. CLI works fine. A mouse scroll "unsticks" the display.

**Root cause:** `glamour.WithAutoStyle()` calls `termenv.HasDarkBackground()`, which sends an OSC 11 escape sequence to the terminal and reads the response from stdin. But Bubbletea is also reading stdin for input events. This causes two problems:

1. **Stdin contention**: Bubbletea's input goroutine may consume the OSC 11 response, so termenv never sees it and blocks for the full 5-second timeout.
2. **Render corruption**: The stdin race breaks Bubbletea's render cycle, leaving the view stuck on "Loading..." even though the model state has been updated. A mouse event forces a fresh render, which is why scrolling "fixes" it.

The TUI recreated the glamour renderer on every window resize (to update word wrap width), re-triggering the query each time. Multiple resize events at startup compounded into 30-60 seconds of blocking.

**Fix:** Detect dark/light background once in `main()` before `tea.NewProgram().Run()` takes over stdin. Pass the resolved style name (`"dark"` or `"light"`) into the model. Use `glamour.WithStandardStyle(styleName)` instead of `glamour.WithAutoStyle()`; this hits the `DefaultStyles` map lookup directly, bypassing the terminal query entirely. Also added a `-style` flag so users can skip detection.

**Guard:** Both stdin and stdout must be terminals before attempting detection. If either is piped, fall back to dark.

**Key insight:** Never do terminal escape sequence queries inside a Bubbletea event loop. Bubbletea owns stdin once it starts. Any library that reads stdin for terminal responses will race with Bubbletea's input goroutine.


### TUI Link Highlighting: Marker Injection and the Glamour Black Box

**Context:** The TUI needs to highlight the selected link in the rendered markdown and support clickable links. But glamour's API is `Render(string) → string`; a black box. It doesn't expose the goldmark AST it builds internally, and the rendered ANSI output doesn't preserve link structure. Link URLs are stripped from the output; only styled text remains.

**Problem:** We parse the markdown twice with goldmark. Once in `links.ExtractWithPositions` to get link destinations and their byte offsets in the source, and again inside glamour to render. If glamour exposed its pre-parsed AST (or accepted one), we could do a single parse, extract link positions, and annotate nodes directly for highlighting.

**Workaround: Marker injection.** Before passing markdown to glamour, we inject Unicode private-use-area codepoints (U+F0000 range) around each link's text inside the `[...]` brackets. Glamour treats them as literal characters and passes them through to the rendered output. After rendering, `processMarkers` scans the ANSI output, finds the markers, records their screen coordinates as `linkRegion` entries, and strips them (or replaces with ANSI reverse-video for highlighted links).

**Goldmark AST quirks for link positions:**
- `ast.Link` nodes don't expose the byte offset of `[` or `]` directly. You have to walk the link's child `ast.Text` nodes to get their `Segment.Start` and `Segment.Stop`, then scan backward/forward in the source to find the brackets.
- Links with inline formatting (e.g., `[hello **world**](url)`) have multiple child nodes. The `**` markers sit between text segments but aren't text nodes themselves, so `Segment.Stop` of one child doesn't equal `Segment.Start` of the next.
- `link.Text(source)` is deprecated in newer goldmark versions. Use the child text segments directly and build the text yourself.
- Links with no text nodes (e.g., `[](url)`) produce no text segments. `ExtractWithPositions` handles this by emitting a `LinkInfo` with `OpenBracket: -1` so the link is still in the navigation list but skipped during marker injection.

**Region extension for clickable URLs:** Glamour renders links as bold text followed by an underlined URL. The end marker sits after the text but before the URL. `processMarkers` extends each region past the end marker to also cover the URL portion, scanning forward through the space and URL characters and stopping at the next space. This makes both the text and URL clickable.

**Marker limit:** Start markers use U+F0000+i, end markers use U+F1000+i. Maximum 4096 links per document before the ranges overlap. `injectLinkMarkers` caps at `maxMarkedLinks`; excess links render normally but without highlighting or click detection.

**Mouse mode:** `tea.WithMouseCellMotion()` only delivers motion events when a button is held. Hover requires `tea.WithMouseAllMotion()`.



### Argo CD `selfHeal` Reverts Broker Writes to the World Tokens Secret

**Context:** demarkus-broker mints a token by appending a `[tokens.usr_xxx]` section to the world's `tokens.toml` Secret (the same Secret the chart created with the `admin` token at install time). When the world is deployed via Argo CD with `syncPolicy.automated.selfHeal: true` (the default for an "auto-sync" GitOps shape), Argo's reconciler treats the broker's appended sections as drift from the chart-rendered manifest and reverts them on the next sync pass.

**Symptom:**
- `POST /auth/callback` returns 200 with a tokens array. Broker logs `mint succeeded`.
- The broker-namespace `issuances` Secret has the new entry; broker side is fine.
- The world's `tokens.toml` Secret only contains the original `admin` entry. `kubectl get secret -o yaml | grep resourceVersion` shows a high number (e.g. 1705 after a few minutes): the Secret has been updated repeatedly, but every update is immediately overwritten by Argo's selfHeal cycle.
- demarkus-server returns "unauthorized" for every minted token because they never make it to the file the server reads.

**Why selfHeal does this:** Argo's drift detector compares the live Secret's `data` against the rendered manifest. The Helm chart only renders `data.tokens\.toml` with the admin entry. Anything else (including the broker's runtime-appended `[tokens.usr_xxx]` blocks) is "extra content" that selfHeal removes to bring the Secret back into compliance with the chart-defined source of truth. There's no warning. The mint API call succeeds because the broker's Secret-update API call returned 200; the revert happens milliseconds later.

**Fix:** add `ignoreDifferences` to the Application (or ApplicationSet template) so Argo stops reconciling `/data` of the per-world tokens Secret:

```yaml
spec:
  ignoreDifferences:
    - kind: Secret
      name: "<release>-demarkus-server-tokens"
      jsonPointers:
        - /data
```

This leaves the chart in charge of the Secret's existence + initial `admin` entry while letting the broker own its runtime mutations from then on. The narrower `/data/tokens.toml` jsonPointer also works but `/data` covers the case where the chart later adds another data field.

**Key insight:** Argo CD treats every Helm-rendered manifest as the desired state. Any resource whose contents are **mutated at runtime** by another controller (or by the broker, in our case) needs an `ignoreDifferences` carve-out for the field being mutated, OR the Argo Application needs `selfHeal: false`. Pick one: selfHeal=true with no ignoreDifferences is a silent data-loss machine.

**Detection: how to spot this fast.** When a runtime mutation against an Argo-managed resource appears to have no effect, check `kubectl get <kind> <name> -o yaml | grep resourceVersion`. If the version is climbing every few seconds without obvious cause, selfHeal is the cause.

**Where this bites in production:** any Helm-rendered Secret/ConfigMap/CRD that another component mutates at runtime. demarkus-broker writes to the world tokens Secret. cert-manager writes to TLS Secrets. external-secrets-operator writes to Secrets. Any operator that owns a CR's `.spec` may hit the inverse pattern (Argo trying to reset spec back to the rendered shape). The `ignoreDifferences` carve-out is the durable answer in every case.

**First seen:** Stage 4 of the kind harness (PR #134), 2026-05-14. Mint API returned 200 with valid tokens; the world Secrets stayed at the chart-default admin-only contents. Took ~30 minutes of broker-side debugging (logs, code-reading, RBAC checks) before checking the Argo CD Application diff revealed the selfHeal cycle. Now load-bearing knowledge for any production broker deployment with Argo CD.


## Agents improvise body frontmatter that demarkus never interprets (2026-06-10)

**Symptom.** Browsing a new project's index doc in the TUI showed garbled output; a stray `## name:` setext heading followed by loose `description:` / `type:` lines. The doc had been authored by a claude-code agent against an org knowledge system.

**Root cause: doubled frontmatter.** demarkus carries metadata out of band: the publisher passes a `metadata` object on PUBLISH, the store prepends its own version envelope (`buildVersionFile`, `protocol/store/store.go`), and on FETCH the server strips exactly one leading `---` … `---` block (`stripFrontmatter`, `server/internal/handler/handler.go`). The agent additionally hand-wrote a *second* frontmatter block into the document body:

```
---            ← store envelope (version/archived/meta.*), stripped on fetch
version: 1
meta.agent: claude-code
---
---            ← agent's own block, survives into Response.Body
name: ...
description: ...
type: reference
---
# real content
```

The server strips only the first block, so the agent's block reaches the TUI as literal body. glamour renders `name: ...\n---` as a setext H2, hence `## name:`. The block is also invisible to LOOKUP (which reads only `tags`/`importance`/`title`).

**Why the agent did it.** Two reinforcing causes. (1) The session guidance told it to set `tags`/`importance` via the `metadata` object but never mentioned `title`; so "record a name" had no documented home and it reached for the Hugo/Obsidian frontmatter reflex. (2) `name`/`description`/`type` is ubiquitous in LLM training data. Provenance check came back negative: nothing in the repo prescribes that block, the org template was the stock deploy default (unchanged), and `docs/site/reference/markdown.md` already warns against body `---`. Pure improvisation.

**Fix.** Guidance, not code. Added "all metadata travels in the `metadata` object, never the body; recognized keys are `title`/`tags`/`importance`; map name→H1/`title`, kind→a `type:` tag, description→first sentence under the H1" to both `context/session-guidance.md` files and `skills/soul-memory/SKILL.md`. Naming `title` is the gap-closer. Deliberately did **not** add defensive frontmatter stripping to the TUI; Fritz wants the raw render to stay a faithful knowledge-document debugger (it's exactly what surfaced this). Open follow-up: a write-side guard in `publish-gate.sh` to warn when a body opens with `---`.


## Managed soul server didn't restart onto an upgraded binary (2026-06-10)

**Symptom.** After a plugin update, the local demarkus-server stayed on the old version ("server binary behind").

**Root cause.** Two-stage propagation, only one stage existed. `ensure_binaries` (plugins/claude-code/scripts/lib.sh) compares a `.versions` sentinel against the pinned `SERVER_VERSION` and re-downloads the binary *file* on drift at session start; that worked. But `ensure_managed_server` reused any live PID unconditionally (`kill -0` only), so the already-running server kept serving the **old binary from memory** until the process died (reboot / manual kill). The on-disk upgrade never reached the running process.

**Fix.** Version-stamp the managed server and restart on mismatch. On spawn, write `SERVER_VERSION` to `${soul_dir}/.server-version` next to `.pid`. New predicate `managed_server_current PID_FILE VERSION_FILE` reuses only when the PID is alive AND the stamp equals the current pin; a live-but-stale (or unstamped, from an older plugin) server is killed (SIGTERM, bounded ~3s wait, SIGKILL fallback) and respawned on the new binary. Safe: soul is on-disk + versioned, and demarkus is QUIC/UDP so the port frees on exit with no TIME_WAIT. Restart only happens in managed default/isolated modes; reuse mode (user's own server) never calls this. Propagates automatically on next session start and on `/soul-init`. `demarkus-server` has no `--version` flag, which is why the stamp file (not a process query) is the source of truth. Branch `fix/server-auto-upgrade-on-drift`, demarkus-memory 0.5.1→0.5.2.

**Test gotcha found along the way.** `pid="$(helper)"` where helper does `sleep 30 & echo $!` does not reliably leave the process alive in the caller (command-substitution subshell ownership + the inherited-fd pipe trap). Start background test processes directly in the test function with fds redirected: `sleep 30 >/dev/null 2>&1 & local pid=$!`. The bug hid because "not-current" assertions pass for a dead process too; only the positive "reuse a live current server" case caught it.


## Release binaries shipped version="dev": goreleaser ldflags lacked -X main.version (2026-06-10)

**Finding (while adding `--version`).** The Makefile injects `-ldflags "-X main.version=$(VERSION)"`, but the **goreleaser** configs (`server/.goreleaser.yml`, `client/.goreleaser.yml`, `tools/.goreleaser.yml`) had `ldflags: -s -w` only. Releases go through goreleaser, so every released binary reported `main.version = "dev"` (the package default). Local `make` builds looked correct; the gap was invisible unless you ran a *released* binary. Fix: add `-X main.version={{ .Version }}` to each released build's ldflags. goreleaser runs per-module with the module tag, so `{{ .Version }}` resolves to the real release version (e.g. server/v0.17.x → `0.17.x`).

**`--version` added to demarkus-server, demarkus-mcp, demarkus-token.** Each prints the bare version to stdout and returns before any side effects (no config load / no server start / no token file). server & mcp use a `-version` bool flag (Go's flag pkg accepts `-version` and `--version`); token is subcommand-style so it takes a `version` subcommand plus `--version`/`-version` cases. Output is bare (just the version, node-style) so `ver=$("$bin" --version)` needs no parsing; that's the ops use case.

**Design constraint (carried from the auto-upgrade work):** `--version` reports the **on-disk** binary's version, not a running process's. So it can feed `ensure_binaries`' installed-version detection and `/soul-status` diagnostics, but it does **not** replace the `.server-version` launch-stamp used to decide whether to restart a live server. Two different questions.

**Sequencing gotcha for adoption:** the plugin can't call `--version` on its managed binaries until the pinned release actually includes it; old binaries error on the unknown flag/subcommand. So: land this → release server/client/tools → bump plugin pins → only then wire `--version` into `ensure_binaries`/`/soul-status`. Test approach: each binary's `version_test.go` builds itself with `-ldflags -X main.version=9.9.9-test` and asserts `--version` echoes it, which also catches a mistargeted `-X` package path (would silently stay "dev"). Branch `feat/binary-version-flag`.


## Plugin adopted --version for binary drift detection; dropped the .versions sidecar (2026-06-10)

Follow-on to the `--version` work, now that server 0.17.15 / client 0.12.39 / tools 0.1.32 ship the flag. The demarkus-memory plugin's `ensure_binaries` (plugins/claude-code/scripts/lib.sh) used a `.versions` sidecar file recording what it last installed. Replaced that with a **live query**: `_binary_version BIN --version` plus `_installed_versions` running each binary's `--version`, compared against the pinned `_desired_versions`. The binaries are now the source of truth: the sidecar could drift from reality (binary swapped out of band, sidecar deleted, partial install), and a live query also catches a corrupt or wrong-arch binary (fails to run → empty field → mismatch → re-download). Removed `PLUGIN_VERSION_FILE`, the end-of-install sidecar write, and the failure-path `rm`; a failed/partial install simply reads as a mismatch next session. `demarkus-token` accepts `--version` as well as its `version` subcommand, so one flag form (`--version`) covers all three. Pre-0.17.15 binaries error on the unknown flag → empty field → upgraded on next session start (the desired transition). `/soul-status` now queries `--version` per binary instead of `cat .versions`. Tested via stub scripts (working / old-erroring / missing / non-executable) and verified end-to-end against the real released binaries. Branch `feat/plugin-adopt-version-flag`, demarkus-memory 0.5.2→0.5.3. Pins bumped to 0.17.15 / 0.12.39 / 0.1.32.


## pi slash commands registered but never executed; sendMessage without triggerTurn (2026-06-25)

**Symptom.** In the pi-agent ports (`plugins/pi-memory`, `plugins/pi-knowledge`, #211), every slash command (`/soul-*`, `/promote`, `/knowledge-*`) appeared to do nothing. The command was recognized (no "unknown command" error) but the agent never acted on it.

**Root cause.** The command handlers inject the bundled skill body into the session with `pi.sendMessage({ customType, content, display: false })` and **no options**. Pi's `sendCustomMessage` (`@earendil-works/pi-coding-agent` `dist/core/agent-session.js`) branches three ways: streaming → steer/followUp; not-streaming **+ `triggerTurn`** → `_runAgentPrompt` (starts a turn); not-streaming **+ no trigger** → just append to state + emit message_start/message_end, **no LLM turn**. Typing a slash command happens while the session is idle (not streaming), so it hit the third branch: the skill instructions landed in history and the agent was never prompted to run them. `display: false` made it invisible too, so it looked like a complete no-op.

**Fix.** Pass `{ triggerTurn: true }` in the command handler's `sendMessage` (both `plugins/pi-memory/src/index.ts` and `plugins/pi-knowledge/src/index.ts`). Idle → starts a turn; mid-stream → pi ignores `triggerTurn` and steers, which is the desired behavior anyway. The nudge/gate `sendMessage` calls elsewhere correctly pass `{ triggerTurn: false }` (fire-and-forget reminders); only the user-invoked command path needs a turn.

**Key insight.** In pi, `sendMessage` is *not* a request to act; it's an entry append that only starts a turn when `triggerTurn: true` (or when streaming, via steer/followUp). `sendUserMessage` always triggers a turn. For a user-invoked command whose whole job is "make the agent do X now," you must trigger the turn explicitly; the claude-code mental model (inject prompt → it runs) does not carry over. No test caught this because the handlers were never exercised end-to-end against a live pi session; type-checking passes regardless.


## Ignoring the lock-PID write creates a poison lock (2026-06-25)

**Context.** The demarkus-plugin (`tools/demarkus-plugin`) guards provisioning and registry
writes with an atomic mkdir-mutex: `os.Mkdir(lockDir)` acquires, and the holder stamps its
PID into `lockDir/pid` so stale-lock recovery can reclaim a lock whose owner died. Two sites:
`internal/provision/provision.go` (`withProvisionLock`) and `internal/registry/lock.go`
(`withLock`).

**Bug (caught in PR #218 review).** Both did `_ = os.WriteFile(lockPid, …)`; swallowing the
error: then ran the critical section anyway. If that write fails, the lock dir exists with
no PID file. Stale-lock recovery reads `lockPid` to check whether the recorded owner is still
alive; with no file the read fails and the recovery path falls through to sleep-and-retry,
never reclaiming. Result: an unowned **poison lock** that wedges every future writer for the
full bounded wait (~180s for provision, ~2s for registry) and then errors; with no owner to
blame and nothing to clean it up.

**Fix.** Treat the PID stamp as part of acquisition: on write failure, `os.RemoveAll` the
lock dir and return the error (fail closed) instead of entering the critical section. Don't
hold a lock you can't prove ownership of.

**Lesson / sweep.** A `_ =` on a write that another code path later *reads to make a
decision* is not a benign ignore: it silently breaks the reader's invariant. When auditing,
separate "best-effort writes nobody depends on" (e.g. the guidance.go one-time-offer
sentinels: a lost write just re-offers, fine to ignore) from "writes that establish state a
recovery/cleanup path relies on" (lock PID stamps; must be checked). Grep for the lock
acquisition primitive (`os.Mkdir(` mutexes), not just `os.WriteFile`, to enumerate every
site: there were exactly two module-wide.

## `gh pr view <branch>` resolves CLOSED PRs; automation edited a dead PR for 10 days

The plugin-pin-bump workflow's "open or update PR" step used
`gh pr view "$branch"` to decide between `gh pr edit` and `gh pr create`.
`gh pr view` (and `gh pr edit`) resolve a branch name to the **most recent
PR for that branch, including closed ones**. After bump PR #210 was closed
unmerged (2026-06-24), every daily run force-pushed `auto/plugin-pin-bump`
and "successfully" edited the closed PR's title/body; no new PR, no
failure, pins silently drifting behind releases for 10 days. The tell:
#210's title named `server 0.19.0`, a version released a week *after* the
PR was closed, and `updatedAt` kept moving.

Rule: in automation, branch→PR resolution must be
`gh pr list --head "$branch" --state open --json number` and then operate
on the PR number. Never `gh pr view/edit <branch>` when a closed PR for
that branch may exist. Fixed in plugin-pin-bump.yml (2026-07-05).

General shape of the bug: an "upsert" whose existence check matches dead
records. Same family as matching stale locks or tombstoned rows; the
check must filter by liveness, not just identity.

## Mirrored code without a fidelity test absorbs every fix twice

PR #230 (broker ergonomics parity): `changedNote`/`seenDoc` were
copy-pasted between client cmd and broker as a "deliberate mirror". Two
consecutive review rounds found wording bugs that had to be fixed in both
places, and a later round found a guard present in one flow but not the
other. Hoisting into a shared package (`client/fetchdedup`) ended it.

Rule of thumb: a deliberate mirror is only safe when a byte-for-byte
fidelity TEST enforces it (as `formatResult`/`formatToolResult` has; that
mirror has never drifted). A mirror kept in sync by reviewer vigilance
will drift; hoist to a shared package at the SECOND two-place fix. The
tools module imports the client module, so client-side shared packages
(mdoutline, fetchdedup) are always available to the broker.

## A gateway behavior change needs a consumer census, not just agent acceptance

Broker 0.5.0's mark_fetch ergonomics (outline mode >8KB, session
unchanged-dedup) were verified against LLM agents on both MCP surfaces;
but the demarkus-library is a THIRD consumer of the same gateway (broker
mode reads every document through mark_fetch, one pooled MCP session per
reader), and it broke two ways, silently:

1. graph.md (74KB) returned an outline → the floor's hub-topology parser
   found no Edges table → federation portals (soul.demarkus.io) vanished.
   The library's designed degradation ("any read/parse failure yields an
   empty topology") masked it: degradation paths hide regressions.
2. Re-opening an already-opened document returned `status: unchanged`
   with no body → parseToolResult's default branch → "unreachable" in
   the UI. `unchanged` was a NEW status value in a response contract that
   programmatic consumers parse by exact status.

Fix: the library's gateway Fetch passes force=true; a programmatic
renderer always wants the real body; force bypasses both behaviors by
design (library fix/gateway-force-fetch). Diagnosis pattern that worked:
walk the data pipeline producer→consumer (root hub row → federated-
systems.md → agent graph.md → floor parser); the break is at the first
hop where healthy data stops being consumed correctly.

Rules of thumb:
- Changing a shared surface's RESPONSE SHAPE (new status values, new body
  modes) is a breaking change for every parser of that surface; census
  the consumers (agents, library, plugins, smoke scripts) before shipping.
- A consumer with graceful degradation needs a canary assertion somewhere
  (e.g. the floor should alert when the hub topology parses to zero edges
  but the doc fetch succeeded), or regressions land as quiet emptiness.

## Deleting old versions broke a hidden v1-existence assumption (#236)

Version retention (prune oldest, keep last N) looked structurally safe:
numbering derives from max-on-disk, `VerifyChain` checks consecutive present
pairs. The store tests caught what analysis missed; `migrateFlatFile`
detected "flat file needing migration" by checking whether **v1** exists.
After pruning v1, the next write misclassified the versioned document as a
flat file and "migrated" the current symlink's raw bytes into a resurrected
v1: doubled store frontmatter, broken hash chain, a version listing showing
deleted versions back from the dead. Fixed by detecting *any* per-doc version
history instead of v1 specifically.

Rules of thumb:
- Before adding deletion to an append-only store, grep for existence checks
  on specific low versions (`v1`, version==1 special cases); append-only
  code is full of "oldest version always exists" assumptions that deletion
  invalidates.
- Destructive filesystem ops in the store go through `os.OpenRoot` (Go
  1.24+): paths resolve inside the root at operation time, so planted or
  raced symlinks error instead of redirecting the delete outside the store,
  and a final-element symlink is unlinked, never followed. Retention's
  `pruneVersions` is the reference example; reuse the pattern for any future
  delete/rename in `protocol/store`.
- The debug loop that found it: identical logic passed in a scratch test but
  failed in the suite → diff the *sequence*, not the assertion; the extra
  later write was the trigger, and `find` on the versions dir between steps
  showed v1 reappearing with double frontmatter.

## Self-referential automation: the pin-bump/release loop (#244)

The plugin pin-bump workflow edited `fallbackToolsVersion` (a dev-build-only
const in `tools/`) on every run and also used it as the tools currency
signal. Consequence: a tools-only bump PR touched `tools/`, merging it cut a
new tools patch release (semantic-version bumps on any change_path commit,
chore included), the release event re-triggered the workflow, and a new bump
PR opened. Release, bump, release, forever, one junk release per merge.
Surfaced while running the second pin cycle for the style gate: PR #243's
only provision.go change was the fallback const.

Fix (#244): the currency signal is now the bootstrap `TOOLS_VERSION` (the
real pin, as the script's own header already said), sampled across EVERY
plugin bootstrap with the lowest value winning so a partially-bumped tree
self-heals; `fallbackToolsVersion` rides along only when provision.go is
already changing for a real server/client pin.

Rules of thumb:
- An automation that edits a path inside its own trigger's change_path is a
  loop until proven otherwise. Trace one full cycle (edit, release, trigger)
  before merging its output.
- Never use a value the automation itself writes as its convergence signal;
  key off the artifact users actually consume (here: the bootstrap pin).
- Same family as the gh-pr-view-closed-PRs bug: automation state checks must
  reference ground truth, not the automation's own byproducts.

## kqueue cannot see a same-name atomic symlink swap (configwatch flake, 2026-07-13)

CI failed TestDebounceCoalescesBurst; local -race sweeps then failed TestSymlinkRetargetTriggersReload 8 of 12 runs. Two distinct causes, one shared lesson.

**Debounce flake**: 10 writes at 5ms intervals under an 80ms debounce, asserting exactly one reload. Any scheduler stall > 80ms between writes fires the debounce mid-burst. Fix: widen the window relative to the pacing (400ms vs 2ms writes) and wait deterministically (waitForCount, then one more full window to prove no straggler).

**Watch-establishment race (the general class)**: every trigger test started Run on a goroutine, slept a flat 50ms, then fired one-shot filesystem ops. fsnotify only reports changes made after registration, so under load the ops land before Add and are lost forever. Fix: awaitWatchLive helper writes a sentinel file repeatedly (a single touch just re-loses the same race) until a reload is observed, pausing a full debounce window between touches so a pending reload can fire instead of being re-armed forever, then drains stragglers and resets the counter. Never sleep-and-hope to synchronize with a watcher goroutine; prove liveness with an observable round trip.

**Symlink swap invisibility (the deep one)**: with the watch provably live, the retarget still delivered zero events, deterministically. Diagnostic with timestamped raw fsnotify events showed CREATE current.new arrives instantly, but the rename's events never arrive: not after 3s of silence, not after a sentinel kick; only teardown's name changes surfaced anything. Mechanism: fsnotify's kqueue (macOS) backend detects directory changes by rescanning and diffing entry names. rename(current.new, current) changes no names once both ops coalesce into one rescan window, so it emits nothing. The test only ever passed when the scheduler let the rescan catch current.new briefly existing; -race and suite load made it lose that race almost always. Linux inotify reports renames explicitly by cookie, so CI never saw this one. Fix: skip the test on darwin with the mechanism documented, note the caveat in the package doc. Production (Linux) keeps the real guarantee; macOS dev still works in practice because staging the new link fires its own event.

Debugging pattern that cracked it: when a test is flaky, first make the precondition deterministic (watch-live sentinel); if the failure then becomes deterministic, the flake was hiding a real absence, not a timing wobble. Then drop below the abstraction (raw fsnotify with timestamps) and test delivery against silence and against a kick, which distinguishes "late" from "never".

## goldmark: BaseInline.Lines() panics by contract (2026-07-13)

Walking AST ancestors from an inline node (a link) and calling `p.Lines()` on each parent panics whenever the parent is another inline (emphasis wrapping a link: `**[text](url)**`): goldmark's `BaseInline.Lines()` is `panic("can not call with inline nodes.")`, not an empty result. Guard ancestor scans with `p.Type() == ast.TypeBlock` before touching `Lines()`.

Found by CodeRabbit on PR 251 after the edge-provenance work shipped a blockStart walk; every existing test had formatting *inside* links, none had links *inside* formatting, so the panic was invisible. Test both nesting directions when touching inline AST handling. Repro test added at links/mdoutline/crawl levels.


## Mock fixtures encoded a plan assumption; production data refuted it two deploys later (2026-07-14)

Graph hub seeding (PR 253) shipped broker-side with tests whose /graph.md fixtures used mark://{worldName}/... rows, because the plan's test sketch did. The production aggregate keys rows by cluster-internal dial addresses (name.namespace.svc.cluster.local:6309), which the gateway's URL grammar cannot even express, so seeding was inert in the real topology; and only the hub world publishes /graph.md at all, so per-queried-world seeding never found the aggregate. Two follow-up PRs (256, seed-all-worlds), each with its own release and deploy bump, fixed what one look at real data would have caught pre-merge.

The plan had even flagged the risk ("cluster-internal names in the aggregate") and mis-assessed it as "harmless, they are just labels"; they are lookup keys. The same bug class (key-form mismatch) was caught pre-merge on the demarkus-mcp side precisely because that surface got a live cold-start check against the real soul; the broker settled for its mock harness and met real data only after deploy.

Lessons:

1. When mock-tested code consumes a production-generated document, fetch a real sample and make it the fixture. The real aggregate was one mark_fetch away the whole time.
2. Re-derive a plan's risk assessments during implementation instead of inheriting them; "harmless" written at planning time is a hypothesis, not a fact.
3. If one surface gets a live verification and a sibling surface only gets mocks, expect the sibling to carry the bug the live check catches.


## GITHUB_TOKEN events never trigger workflows (dead release trigger, 2026-07-18)

Symptom: the plugin pin-bump PR for a release landed ~16 hours after the release, on the daily cron, despite plugin-pin-bump.yml declaring `release: [published]` as a trigger. The trigger had never fired once.

Cause: GitHub suppresses workflow triggering from events created by the default GITHUB_TOKEN (anti-recursion rule), and release.yml publishes via GoReleaser with exactly that token. So any `release:`/`push:`-triggered workflow downstream of a GITHUB_TOKEN-published release is dead by construction. The documented exceptions are workflow_dispatch and repository_dispatch: those DO fire even when invoked with GITHUB_TOKEN.

Fix (branch pin-bump-dispatch): release.yml gained a final `dispatch-pin-bump` job (`gh workflow run plugin-pin-bump.yml`, needs the three release jobs, fires when any succeeded; `actions: write` added to permissions). The dead `release:` trigger removed from plugin-pin-bump.yml with a comment explaining why; the daily cron stays as backstop.

Lesson: when a workflow chains off an event another workflow creates, check what token created the event. GITHUB_TOKEN-created events are invisible to triggers; chain via explicit workflow_dispatch instead (the sanctioned exception), or publish with a PAT/App token. The roadmap's "pin chain is self-driving: release to auto bump PR" claim was believed for weeks while the release leg had never fired.


## install-stack.sh indexing agent seeded the wrong world port (2026-07-18)

The appliance's WORLD_INTERNAL was `localhost:6399`; the world server listens on 6309 (protocol.DefaultPort, and install.sh uses 6309 everywhere). The indexing agent seeds/hubs pointed at 6399, so it could never reach the world: no crawl, no /graph.md republish, and the library's graph/backlink views stayed empty. Silent because the agent unit is Restart=on-failure and just kept retrying a dead address. Fixed to localhost:6309. Lesson: the agent's seed port must equal the server's actual listen port, not a transposed digit; there is no second internal port in the appliance.

## Library "world unreadable" on same-host installs: hairpin NAT

Symptom: reading room serves fine at https://soul.demarkus.io/ but the universe shows "soul.demarkus.io:6309 unreadable", while external clients read the world over QUIC without issue.

Cause: install.sh writes `DEMARKUS_HOST=<domain>` into the library env when a domain exists (chosen so cert verification works; the cert can never match localhost). The library then dials its own host's public address from inside, which requires hairpin NAT; the Pi's network does not hairpin UDP, so the dial fails. External paths work, the loopback path does not.

Fix applied on soul.demarkus.io: `DEMARKUS_HOST=localhost:6309`, `DEMARKUS_INSECURE=true`, restart. Cosmetic side effect: the universe tile is labeled localhost:6309 instead of the domain (QUIC-mode world name comes straight from DEMARKUS_HOST).

Diagnosis tell worth remembering: in QUIC mode the universe tile name IS the configured DEMARKUS_HOST, so a tile still showing the old host after an env edit means the process never picked up the change. That was the second layer here: the first restart never actually ran, and the tile name proved it from the outside.

Open installer follow-up: same-host library installs should not assume hairpin. Options: probe the domain dial at install time and fall back to localhost+insecure, or document the trade. Related: the world name/label should arguably be decoupled from the dial address.

Final resolution (superseding the localhost workaround above): `127.0.0.1 soul.demarkus.io` in the Pi's /etc/hosts, env back to `DEMARKUS_HOST=soul.demarkus.io` with `DEMARKUS_INSECURE=false`. The dial resolves to loopback so hairpin never enters the picture, the Let's Encrypt cert matches the dialed name so verification stays on, and the universe tile keeps the domain label. Verified end to end: tile readable, documents render, insecure=false confirms cert verification on the loopback dial.

## Plugin token minted with "/*" instead of "/**" (2026-08-10)

Provision minted the plugin token with `-paths "/*"`. Token path matching uses `path.Match` for patterns without `**`, and `*` does not cross `/`, so `/*` grants only single-segment paths (`/index.md` yes, `/<slug>/journal/x.md` no). Agents hit auth denials on nested writes right after install and had to widen the scope by hand. Only `/**` is recursive (server/internal/auth/auth.go matchPath).

Two-part fix in tools/demarkus-plugin/internal/provision:
1. Mint with `/**`.
2. The idempotent gate (token file + label present, return early) never re-checked scope, so existing installs would keep the narrow token forever. Added tokenScopeStale: an entry with `/*` and no `/**` falls through to revoke + remint, then SIGHUPs the server at that root so the running process reloads the token table (without the HUP the fresh raw token fails auth until restart).

Lesson: an idempotent "already provisioned" check must validate the provisioned state's content, not just its existence, or config fixes never reach existing installs. Note `demarkus-token generate` still defaults `-paths` to `/*`; hand-minted tokens without an explicit -paths are single-segment scoped.

### Follow-up: gate now verifies content, and commit implies reload (2026-08-10)

Review iteration on the same fix converged on two structural invariants for ensureTokenEntry:
1. The idempotent gate verifies the raw token against the TOML entry's `protocol.HashToken` hash (rawMatchesEntry), not just file existence. Any inconsistent raw/entry pair self-heals via remint.
2. The server SIGHUP happens **before** the token file commit. A committed token file therefore implies the server loaded its entry, which eliminated the fallible remove-on-failure recovery (and the marker-file design it was sliding toward). When a recovery path needs a fallible cleanup step to stay correct, reorder so the failure leaves the pre-commit state instead.

## Discovery-metadata move broke a second consumer (2026-08-11)

Broker 0.14.9 (#287) moved RFC 8414 metadata off the MCP gateway host to the issuer host to fix strict clients (pi's issuer-mismatch failure). The impact analysis concluded "the only known consumer was the kind smoke test" and called the change non-breaking. Wrong: demarkus-library derived its OAuth discovery URL as `{broker.url}/.well-known/oauth-authorization-server` against the gateway origin, and its own appset comment documented exactly that dependency. Library sign-in broke in production once its 5-minute discovery cache expired.

Lessons:

1. Before declaring a removed endpoint unconsumed, grep every sibling repo (library, agent, plugins, deploy) for the path literal, not just the repo being changed. The dependency was written down in the deploy repo the whole time.
2. A spec-violating surface that ships becomes a de-facto contract; in-house clients copy the shortcut (fetch AS metadata from the resource origin) instead of the spec chain (RFC 9728 PRM, then authorization_servers[0], then RFC 8414). Fix: the library now follows the chain with a single-origin fallback for old brokers.
3. Bonus find: the library's `/token/revoke` had been posting to the gateway host since inception, a silent 404 on every logout. Nobody noticed because Revoke errors were not user-visible. Silent-failure paths hide broken integrations; surface them.

## Test migrations against a copy of real data, not just fixtures

The eager legacy-layout migration (2026-08-11, `fix/store-flat-file-migration`) passed every synthetic fixture, then failed instantly against a copy of the real soul: `versions/..v1` (junk from an old bug that published base ".") made `legacyVersionBase` derive base "." and aim the current-pointer rename at the store root itself. Real stores accumulate junk that fixture authors never imagine. Before shipping anything that rewrites on-disk state, run it against a copy of production data and inspect what it skipped as well as what it moved; and when parsing filenames into paths, reject bases that are hidden, empty, ".", or otherwise unpublishable before joining them into a filesystem target.

## Hidden directories hold served documents; migrations must walk them

The 0.22.2 legacy-layout migration skipped dot-directories, but FETCH serves explicit hidden paths: `/.well-known/agent-manifest.md`. With the legacy read fallback deleted in the same release, updating soul.demarkus.io broke its agent manifest (LIST-hidden is not FETCH-hidden; `isHiddenEntry` governs listings only). Found minutes after running `demarkus-install update` because the post-update verification included `demarkus info`, which fetches the manifest. Fixed live by hand-migrating the file, then in code by removing the walker's dot-dir skip (branch `fix/migrate-hidden-dirs`). Lessons: a migration's skip list must match the serving path's reachability, not the listing's visibility; and always exercise the well-known endpoints in post-upgrade verification, they use the store differently than normal documents.

## Issue #289: configwatch log feedback loop (2026-08-12)

The plugin-managed soul server's `.log` grew to 1.5 GB. Two composed defects:
configwatch reloaded on ANY event in the tokens.toml directory (never checked
`event.Name`), and provision placed the server's log at `<soulDir>/.log`,
inside that watched directory. Each reload logged "configwatch: reloaded",
the append emitted an fsnotify event, and the 150 ms debounce re-armed forever.

Gotchas worth remembering:

- fsnotify's kqueue backend (macOS) reports Write events on files inside a
  watched directory, so co-locating any continuously written file with a
  watched config file self-sustains the loop.
- A pure `event.Name == target` filter breaks the symlink-retarget guarantee:
  the retarget event lands on the sibling `current` symlink, not the target.
  Fix filters by name AND op: sibling Write/Chmod drop; Create/Rename/Remove
  (structural swaps) still reload.
- The launchd install never looped because its log lives in `~/.demarkus/logs/`,
  a subdirectory: non-recursive watches don't see subdir writes. That asymmetry
  was the diagnostic clue.

Fix (branch fix/289-configwatch-log-loop): op+name filter in
`server/internal/configwatch/watcher.go`; managed server log moved to
`~/.demarkus/logs/server-<base>-<hash>.log` with legacy `.log` removed on
migration in `tools/demarkus-plugin/internal/provision/provision.go`. The
running old server keeps looping until the fixed release rolls out; truncation
alone regrows ~55 MB/day.

Update: same branch also moves the managed server's tokens.toml out of the
content root to `~/.demarkus/tokens/<soulID>/tokens.toml`, migrated by rename
(copy+remove across devices) only while the server is stopped in
ensureManagedServer; resolution (`tokensPathFor`) prefers legacy until
migrated. Reuse mode keeps `<root>/tokens.toml`: the adopted external server's
-tokens flag is not ours. ensureTokenEntry now takes root explicitly since
`filepath.Dir(tokensTOML)` no longer locates the server.


## OpenCode update-check empty output

`demarkus-plugin update-check` intentionally emits no stdout when a plugin is current, throttled, disabled, or offline. The Pi adapter already treats empty output as no update. Both OpenCode adapters instead passed empty stdout to `JSON.parse`, producing `Unexpected EOF` warnings during startup and commands such as `opencode mcp list`.

OpenCode `runBin` now accepts expected empty output only for update checks. Other helper operations still surface an empty response as an error, preserving gate and guidance diagnostics.


The same expected-empty contract also applies to `nudge` when no reminder is due and `guidance` when no context exists. OpenCode now maps allowed-empty responses to `{}`, preserving the important distinction between a successful no-op and a helper failure (`null`). Gate responses do not allow empty output and still surface an error.

## Hub truncated by a section fetch republished as the whole document (2026-08-17)

Symptom: `/index.md` showed `(no title)` in `mark_graph` and linked only Active Plans and RFC Review; every root section, plugin, sub-project hub, and plan archive was unreachable. Cause: at v69 (2026-08-12) an agent fetched `/index.md#active-plans`, edited it, and called `mark_publish` with that section as the full body. Eight later versions (cli, claude-code, opencode) edited the fragment without noticing. Found by `/soul-doctor`; last full version was v68; restored at v78 by publishing v68's body with the current Active Plans and RFC Review sections spliced in.

Rules:

- Never publish a body you obtained from a `#section` fetch or an outline. Before `mark_publish` on an existing document, `mark_fetch` with `force=true` and edit that body.
- A publish whose body has no `# H1` on a document whose previous version had one is a strong truncation signal. Candidate gate check for the plugin: warn or block when the H1 disappears or the body shrinks by more than half.
- `mark_graph` from the hub is the cheapest detector: a hub with `(no title)` or a suspiciously low link count means the hub, not the graph, is broken.


## Republishing a body through an LLM drops or adds its trailing newline (2026-08-17)

During a corpus-wide metadata sweep, four documents came back from `mark_publish` with a different `content-hash` than the version they were fetched from, even though the intent was a metadata-only write. The whole delta was the terminal `\n`: a body round-tripped through the model as a tool argument does not reliably preserve its final byte.

Consequences and handling:

- `content-hash` is exactly sha256 over the raw body, so any whitespace drift changes it. That makes a hash compare a complete verifier for "metadata only" writes.
- Detection: after a metadata-only publish, fetch again and compare `content-hash` against the pre-write value. Version must be old+1 and the hash must be identical.
- Recovery: fetch the pinned prior version (`mark_fetch /doc.md/vN force=true`), republish that exact body at the new `expected_version`, verify the hash again. History keeps both versions, which is the point of the store.
- Prevention: do not retype a body through the model at all. Fetch it to a file and publish from the file, byte for byte:
  `demarkus -auth "$(cat ~/.demarkus/soul-soul.token)" mark://host/path.md > body.md`, confirm `shasum -a 256 < body.md` equals the served `content-hash`, then `demarkus -X PUBLISH -expected-version N -meta ... mark://host/path.md < body.md`.
- Prefer `mark_append` when adding a section: it never retypes the existing body and carries catalog metadata forward.
- A `-X PUBLISH` invocation with neither `-body` nor a stdin redirect blocks reading stdin until the caller times out. Always redirect from the file.


## The CLI graph crawl keys nodes by the raw start URL, the MCP crawl canonicalizes (2026-08-17)

Rebuilding the soul graph doubled it: 388 nodes for 192 documents, split across two key spaces, 204 under `mark://soul.demarkus.io/...` and 167 under `mark://soul.demarkus.io:6309/...`. Backlinks answered from such a store find only the half that matches the caller's key form.

Cause: `demarkus graph` passes `fs.Arg(0)` straight into `CrawlAndPersist` as the start URL (`client/cmd/demarkus/main.go:349`). The parsed host is used to fetch, but the node key stays whatever the user typed, and `links.Resolve` then derives every child key from that raw base. The MCP handler does the opposite: it rebuilds `startURL` as `"mark://" + host + path` after `resolveURL` has added the default port (`client/cmd/demarkus-mcp/main.go:1276`), with a comment saying the canonical form is required so crawled rows share the key form of the hub `/graph.md` and backlink lookups. So the two crawlers disagree on node identity for the same document.

Notes:

- Typing the port explicitly (`mark://host:6309/path`) makes the CLI agree with the MCP. A fresh canonical re-crawl of this soul produced 220 nodes with zero bare-host duplicates, which also proves no document body links to an absolute bare-host URL. The whole duplication was crawler-introduced, not corpus rot.
- Fix candidate: canonicalize in the CLI the same way the MCP does, before the crawl, so node identity does not depend on how the operator typed the address. This is the URL-normalization gap already recorded in [/plans/graph-completeness.md](/plans/graph-completeness.md).
- Detection: group `~/.mark/graph.json` node URLs by host and look for the same host with and without a port.
- `links.Resolve` returns any destination containing `://` untouched, so an absolute link written without a port would fragment the graph the same way even under a canonical crawl. Prefer root-relative links (`/plans/x.md`) in document bodies.


## The installer updated the server but never the stack components (2026-08-19)

The droplet serving soul.demarkus.io was three library releases behind, which is how the reading room ended up showing pinned editions as backlink entries: the fix for that had shipped, but nothing on the host ever picked it up. Cause: `demarkus-install update` refreshed server, client, tui, mcp, token and publish, and never touched `demarkus-broker` or `demarkus-library`. Those are installed once by `install` or `install-stack` and had no update path at all, so a single-host deployment drifted for as long as it ran.

Fixed in #323 (`71e43aa`) by extracting `fetch_library_binary`, adding `update_stack_component` for the two optional components, and calling both from the update path.

Four things the review surfaced that are worth carrying forward, all of them variants of "reports success for work it did not do":

- The update returned early when the server was already current, so the component refresh was unreachable in exactly the common case: current server, stale component. That is the shape the original bug had too.
- An unresolvable tools release left the broker branch logging a skip and returning success.
- A failed `systemctl try-restart` was swallowed, so a component whose binary was replaced but never restarted looked like a clean update. It now records the failure and the command exits nonzero after finishing its other work.
- Writing over a live executable fails with ETXTBSY. The file already documented this hazard twice for the server, which stops the unit first; the component path now replaces by rename instead, which needs no downtime at all.

Method notes, because three separate checks in this one change passed for the wrong reason:

- When proving a test catches a bug by removing the fix, assert the mutation actually applied. One removal silently missed after gofmt rewrapped the condition.
- A "running binary" made of a shell script never triggers ETXTBSY, because the interpreter reads it rather than mapping it. Test the observable mechanism instead: replace-by-rename gives the destination a new inode, an in-place copy keeps the old one.
- A linear shell harness leaks state between cases. One case set `PLATFORM=darwin` and never restored it, which stayed invisible until a later case was added after it.


## Hand-written conformance tests left seven backend divergences; a differential found them in minutes (2026-08-19)

The pgstore conformance suite (25 subtests) passed on both backends, and CI never ran it against Postgres anyway (no service container; every pg test skipped). A seeded random differential harness (`storetest.RunDifferential`, file store as reference) found seven real divergences within the first few seeds: `..` handling, ENOTDIR leaking as a 500 under document paths, validation order vs version check, write-returned metadata spelling feeding the catalog, non-canonical keys in the hash index and catalog, `LookupHash` tiebreak on shared bodies, and LOOKUP scope cleaning. Several were bugs in the reference too (the file store was inconsistent with itself across a restart). Details and fixes in [/plans/store-parity.md](/plans/store-parity.md).

Lessons:

- A conformance suite proves the cases the author imagined. For "observably identical" claims, add a differential that compares full state after random ops; it costs a few hundred lines and finds what nobody asserted.
- A gated test that skips when its dependency is absent is a test that never runs in CI unless CI provides the dependency and a required flag turns the skip into a failure.
- Fuzz workers are separate processes; a fuzz target against one shared database needs `-parallel 1` or per-worker isolation, otherwise workers reset each other's data and report false divergences.
- When two code paths key a map by request path, canonicalize at the boundary once (`store.CanonicalPath`); comments claiming keys are canonical are not evidence.
## Hand-rolled layout knowledge in tests was silently wrong (2026-08-19, PR #331)

Two test tampers computed a version file's location as `root/versions/<rel>/vN`, but the real layout is `<dir>/versions/<base>/vN`. Both passed because every caller tampered a top-level path, where the two coincide. A simplify review agent caught it by diffing the helpers against `newVersionFilePath`. Fix: export the layout once (`store.(*Store).VersionFilePath`, sharing `versionRelPath` with `getVersion`) and pin it with a nested-path test that proves the tamper is visible to `Get` and breaks `VerifyChain`. Lesson: when a test must reach behind an abstraction, the abstraction should hand it the path; duplicated layout math in tests only fails on inputs the tests happen not to use.

## A green diff harness can be comparing identical failures (2026-08-19)

The first run of `scripts/e2e-backend-parity.sh` passed with zero divergence, and every single check was the same `Unauthorized` error page: the chart's admin token grants read on `/**`, which makes every seeded path private, and the sweep sent no auth. A differential harness that diffs two outputs proves nothing unless the outputs contain the behavior under test; identical error pages diff clean. Guard against it by spot-reading the captured output for expected content (real bodies, version tables, lookup rows) before trusting PASS, and by recording each command's exit status into the compared output so an error path must at least fail identically on both sides.


## A 64-bit constant that only fails on the release (2026-08-20)

`const schemaLockID = 0x64656d61726b7573` compiled everywhere I build and broke the release:

```
internal/pgstore/pgstore.go:185: cannot use schemaLockID (untyped int constant
7234308641739470195) as int value in argument to tx.ExecContext (overflows)
target=linux_arm_7
```

The constant is 63 bits. Go's `int` is 32 bits on `linux/arm/v7`, which goreleaser builds and nothing else does, so the untyped constant had no valid conversion there. Local builds, CI, and the container images are all 64-bit, so every gate passed and the tag was cut before the first 32-bit compile ever ran.

Two lessons, the second more useful than the first.

Type any constant that exceeds 32 bits, and type it as what the destination actually wants. Here `pg_advisory_xact_lock` takes `bigint`, so `int64` was right on its own terms and the platform break was a symptom of leaving it untyped.

The real gap was that the release was the first place five of six target platforms were ever compiled. CI built host amd64 only, so a whole class of error (32-bit int width, platform-specific syscalls, build tags) could only surface after a tag existed. `test-server` now cross-compiles every release target for all three binaries, both build flavors, which is seconds of work and reproduces exactly the failure above when run against the unfixed code. Any project whose release matrix is wider than its CI matrix has this hole.


### Federation agent generated artifacts must satisfy hub publish policy

After shared-server cutover, crawler reads succeeded and the root token matched, but every `/index/**` and `/graph.md` publish returned `server-error`. Server audit logs exposed the actual cause: `publish policy block: 1 violation(s)`. `fedcrawl.publishIndex` sent only `agent` and `retention`, while root policy required a `category:` tag and `type`. Generated artifacts must carry policy metadata too. Use `tags=category:federation` and `type=Reference`; do not weaken the hub policy or misdiagnose this as token/routing failure.