Debugging

Lessons learned from bugs, investigations, and things that went wrong.

General Principles

• Read the error message. The whole error message.
• Reproduce before fixing. If you can't trigger it, you don't understand it.
• Don't brute force. If something is blocked, step back and think about why.
• When a test fails, read the test first: the test might be wrong.

Known Gotchas

YAML Auto-Typing

YAML will silently convert strings like yes, no, on, off to booleans. This is why frontmatter is parsed as map[string]string; we control the type conversion ourselves.

Symlink Resolution in Tests

When testing the versioned store, be careful with os.ReadFile vs following symlinks. The store creates doc.md -> versions/doc.md.v3; some operations need the symlink target, others need the symlink itself.

Path Traversal

filepath.Clean handles most cases but you still need an explicit .. check after cleaning. A path like /../../../etc/passwd cleans to /etc/passwd which looks valid but is outside the content root. The check is: clean the path, then verify it doesn't escape the root.

QUIC Stream Lifecycle

Streams must be closed properly. If a handler panics or returns without closing the stream, the client hangs waiting for data. Always defer stream close.

Debugging Approach for This Project

1. Write a failing test that reproduces the issue
2. Fix the code
3. Verify the test passes
4. Check if the fix breaks anything else (make test)
5. Run bash pre-commit.sh

The test-first approach isn't dogma here: it's practical. A test that reproduces the bug proves you understand the bug, and it prevents regression.

QUIC UDP Buffer Size Warning

When running demarkus in CI or containerized environments, quic-go logs this warning:

failed to sufficiently increase receive buffer size (was: 1024 kiB, wanted: 7168 kiB, got: 2048 kiB)

Not a failure. quic-go tries to increase the UDP receive buffer to 7168 kiB for optimal performance. In environments with restricted sysctl limits, it can only get partway there. The connection still works: everything publishes fine.

Fix (if needed): Increase the system limit with sysctl -w net.core.rmem_max=7168000. Not necessary for correctness, only for optimal throughput under heavy load.

First seen: CI publishing content to demarkus-hub (hub.demarkus.io).

OnNode Callback Concurrency

graph.Crawl invokes OnNode from multiple worker goroutines. Any mutable state captured by the callback must use sync/atomic or a mutex. The CrawlAndPersist MaxNodes counter uses atomic.Int32 for this reason. Always run go test -race on packages that use OnNode.

Glamour WithAutoStyle() Blocks TUI Rendering

Symptom: TUI shows "Loading..." for up to 60 seconds after fetch completes. Server logs confirm the FETCH is handled immediately. CLI works fine. A mouse scroll "unsticks" the display.

Root cause: glamour.WithAutoStyle() calls termenv.HasDarkBackground(), which sends an OSC 11 escape sequence to the terminal and reads the response from stdin. But Bubbletea is also reading stdin for input events. This causes two problems:

1. Stdin contention: Bubbletea's input goroutine may consume the OSC 11 response, so termenv never sees it and blocks for the full 5-second timeout.
2. Render corruption: The stdin race breaks Bubbletea's render cycle, leaving the view stuck on "Loading..." even though the model state has been updated. A mouse event forces a fresh render, which is why scrolling "fixes" it.

The TUI recreated the glamour renderer on every window resize (to update word wrap width), re-triggering the query each time. Multiple resize events at startup compounded into 30-60 seconds of blocking.

Fix: Detect dark/light background once in main() before tea.NewProgram().Run() takes over stdin. Pass the resolved style name ("dark" or "light") into the model. Use glamour.WithStandardStyle(styleName) instead of glamour.WithAutoStyle(); this hits the DefaultStyles map lookup directly, bypassing the terminal query entirely. Also added a -style flag so users can skip detection.

Guard: Both stdin and stdout must be terminals before attempting detection. If either is piped, fall back to dark.

Key insight: Never do terminal escape sequence queries inside a Bubbletea event loop. Bubbletea owns stdin once it starts. Any library that reads stdin for terminal responses will race with Bubbletea's input goroutine.

TUI Link Highlighting: Marker Injection and the Glamour Black Box

Context: The TUI needs to highlight the selected link in the rendered markdown and support clickable links. But glamour's API is Render(string) → string; a black box. It doesn't expose the goldmark AST it builds internally, and the rendered ANSI output doesn't preserve link structure. Link URLs are stripped from the output; only styled text remains.

Problem: We parse the markdown twice with goldmark. Once in links.ExtractWithPositions to get link destinations and their byte offsets in the source, and again inside glamour to render. If glamour exposed its pre-parsed AST (or accepted one), we could do a single parse, extract link positions, and annotate nodes directly for highlighting.

Workaround: Marker injection. Before passing markdown to glamour, we inject Unicode private-use-area codepoints (U+F0000 range) around each link's text inside the [...] brackets. Glamour treats them as literal characters and passes them through to the rendered output. After rendering, processMarkers scans the ANSI output, finds the markers, records their screen coordinates as linkRegion entries, and strips them (or replaces with ANSI reverse-video for highlighted links).

Goldmark AST quirks for link positions:

• ast.Link nodes don't expose the byte offset of [ or ] directly. You have to walk the link's child ast.Text nodes to get their Segment.Start and Segment.Stop, then scan backward/forward in the source to find the brackets.
• Links with inline formatting (e.g., [hello **world**](url)) have multiple child nodes. The ** markers sit between text segments but aren't text nodes themselves, so Segment.Stop of one child doesn't equal Segment.Start of the next.
• link.Text(source) is deprecated in newer goldmark versions. Use the child text segments directly and build the text yourself.
• Links with no text nodes (e.g., [](url)) produce no text segments. ExtractWithPositions handles this by emitting a LinkInfo with OpenBracket: -1 so the link is still in the navigation list but skipped during marker injection.

Region extension for clickable URLs: Glamour renders links as bold text followed by an underlined URL. The end marker sits after the text but before the URL. processMarkers extends each region past the end marker to also cover the URL portion, scanning forward through the space and URL characters and stopping at the next space. This makes both the text and URL clickable.

Marker limit: Start markers use U+F0000+i, end markers use U+F1000+i. Maximum 4096 links per document before the ranges overlap. injectLinkMarkers caps at maxMarkedLinks; excess links render normally but without highlighting or click detection.

Mouse mode: tea.WithMouseCellMotion() only delivers motion events when a button is held. Hover requires tea.WithMouseAllMotion().

Argo CD selfHeal Reverts Broker Writes to the World Tokens Secret

Context: demarkus-broker mints a token by appending a [tokens.usr_xxx] section to the world's tokens.toml Secret (the same Secret the chart created with the admin token at install time). When the world is deployed via Argo CD with syncPolicy.automated.selfHeal: true (the default for an "auto-sync" GitOps shape), Argo's reconciler treats the broker's appended sections as drift from the chart-rendered manifest and reverts them on the next sync pass.

Symptom:

• POST /auth/callback returns 200 with a tokens array. Broker logs mint succeeded.
• The broker-namespace issuances Secret has the new entry; broker side is fine.
• The world's tokens.toml Secret only contains the original admin entry. kubectl get secret -o yaml | grep resourceVersion shows a high number (e.g. 1705 after a few minutes): the Secret has been updated repeatedly, but every update is immediately overwritten by Argo's selfHeal cycle.
• demarkus-server returns "unauthorized" for every minted token because they never make it to the file the server reads.

Why selfHeal does this: Argo's drift detector compares the live Secret's data against the rendered manifest. The Helm chart only renders data.tokens\.toml with the admin entry. Anything else (including the broker's runtime-appended [tokens.usr_xxx] blocks) is "extra content" that selfHeal removes to bring the Secret back into compliance with the chart-defined source of truth. There's no warning. The mint API call succeeds because the broker's Secret-update API call returned 200; the revert happens milliseconds later.

Fix: add ignoreDifferences to the Application (or ApplicationSet template) so Argo stops reconciling /data of the per-world tokens Secret:

spec:
  ignoreDifferences:
    - kind: Secret
      name: "<release>-demarkus-server-tokens"
      jsonPointers:
        - /data

This leaves the chart in charge of the Secret's existence + initial admin entry while letting the broker own its runtime mutations from then on. The narrower /data/tokens.toml jsonPointer also works but /data covers the case where the chart later adds another data field.

Key insight: Argo CD treats every Helm-rendered manifest as the desired state. Any resource whose contents are mutated at runtime by another controller (or by the broker, in our case) needs an ignoreDifferences carve-out for the field being mutated, OR the Argo Application needs selfHeal: false. Pick one: selfHeal=true with no ignoreDifferences is a silent data-loss machine.

Detection: how to spot this fast. When a runtime mutation against an Argo-managed resource appears to have no effect, check kubectl get <kind> <name> -o yaml | grep resourceVersion. If the version is climbing every few seconds without obvious cause, selfHeal is the cause.

Where this bites in production: any Helm-rendered Secret/ConfigMap/CRD that another component mutates at runtime. demarkus-broker writes to the world tokens Secret. cert-manager writes to TLS Secrets. external-secrets-operator writes to Secrets. Any operator that owns a CR's .spec may hit the inverse pattern (Argo trying to reset spec back to the rendered shape). The ignoreDifferences carve-out is the durable answer in every case.

First seen: Stage 4 of the kind harness (PR #134), 2026-05-14. Mint API returned 200 with valid tokens; the world Secrets stayed at the chart-default admin-only contents. Took ~30 minutes of broker-side debugging (logs, code-reading, RBAC checks) before checking the Argo CD Application diff revealed the selfHeal cycle. Now load-bearing knowledge for any production broker deployment with Argo CD.

Agents improvise body frontmatter that demarkus never interprets (2026-06-10)

Symptom. Browsing a new project's index doc in the TUI showed garbled output; a stray ## name: setext heading followed by loose description: / type: lines. The doc had been authored by a claude-code agent against an org knowledge system.

Root cause: doubled frontmatter. demarkus carries metadata out of band: the publisher passes a metadata object on PUBLISH, the store prepends its own version envelope (buildVersionFile, protocol/store/store.go), and on FETCH the server strips exactly one leading --- … --- block (stripFrontmatter, server/internal/handler/handler.go). The agent additionally hand-wrote a second frontmatter block into the document body:

---            ← store envelope (version/archived/meta.*), stripped on fetch
version: 1
meta.agent: claude-code
---
---            ← agent's own block, survives into Response.Body
name: ...
description: ...
type: reference
---
# real content

The server strips only the first block, so the agent's block reaches the TUI as literal body. glamour renders name: ...\n--- as a setext H2, hence ## name:. The block is also invisible to LOOKUP (which reads only tags/importance/title).

Why the agent did it. Two reinforcing causes. (1) The session guidance told it to set tags/importance via the metadata object but never mentioned title; so "record a name" had no documented home and it reached for the Hugo/Obsidian frontmatter reflex. (2) name/description/type is ubiquitous in LLM training data. Provenance check came back negative: nothing in the repo prescribes that block, the org template was the stock deploy default (unchanged), and docs/site/reference/markdown.md already warns against body ---. Pure improvisation.

Fix. Guidance, not code. Added "all metadata travels in the metadata object, never the body; recognized keys are title/tags/importance; map name→H1/title, kind→a type: tag, description→first sentence under the H1" to both context/session-guidance.md files and skills/soul-memory/SKILL.md. Naming title is the gap-closer. Deliberately did not add defensive frontmatter stripping to the TUI; Fritz wants the raw render to stay a faithful knowledge-document debugger (it's exactly what surfaced this). Open follow-up: a write-side guard in publish-gate.sh to warn when a body opens with ---.

Managed soul server didn't restart onto an upgraded binary (2026-06-10)

Symptom. After a plugin update, the local demarkus-server stayed on the old version ("server binary behind").

Root cause. Two-stage propagation, only one stage existed. ensure_binaries (plugins/claude-code/scripts/lib.sh) compares a .versions sentinel against the pinned SERVER_VERSION and re-downloads the binary file on drift at session start; that worked. But ensure_managed_server reused any live PID unconditionally (kill -0 only), so the already-running server kept serving the old binary from memory until the process died (reboot / manual kill). The on-disk upgrade never reached the running process.

Fix. Version-stamp the managed server and restart on mismatch. On spawn, write SERVER_VERSION to ${soul_dir}/.server-version next to .pid. New predicate managed_server_current PID_FILE VERSION_FILE reuses only when the PID is alive AND the stamp equals the current pin; a live-but-stale (or unstamped, from an older plugin) server is killed (SIGTERM, bounded ~3s wait, SIGKILL fallback) and respawned on the new binary. Safe: soul is on-disk + versioned, and demarkus is QUIC/UDP so the port frees on exit with no TIME_WAIT. Restart only happens in managed default/isolated modes; reuse mode (user's own server) never calls this. Propagates automatically on next session start and on /soul-init. demarkus-server has no --version flag, which is why the stamp file (not a process query) is the source of truth. Branch fix/server-auto-upgrade-on-drift, demarkus-memory 0.5.1→0.5.2.

Test gotcha found along the way. pid="$(helper)" where helper does sleep 30 & echo $! does not reliably leave the process alive in the caller (command-substitution subshell ownership + the inherited-fd pipe trap). Start background test processes directly in the test function with fds redirected: sleep 30 >/dev/null 2>&1 & local pid=$!. The bug hid because "not-current" assertions pass for a dead process too; only the positive "reuse a live current server" case caught it.

Release binaries shipped version="dev": goreleaser ldflags lacked -X main.version (2026-06-10)

Finding (while adding --version). The Makefile injects -ldflags "-X main.version=$(VERSION)", but the goreleaser configs (server/.goreleaser.yml, client/.goreleaser.yml, tools/.goreleaser.yml) had ldflags: -s -w only. Releases go through goreleaser, so every released binary reported main.version = "dev" (the package default). Local make builds looked correct; the gap was invisible unless you ran a released binary. Fix: add -X main.version={{ .Version }} to each released build's ldflags. goreleaser runs per-module with the module tag, so {{ .Version }} resolves to the real release version (e.g. server/v0.17.x → 0.17.x).

--version added to demarkus-server, demarkus-mcp, demarkus-token. Each prints the bare version to stdout and returns before any side effects (no config load / no server start / no token file). server & mcp use a -version bool flag (Go's flag pkg accepts -version and --version); token is subcommand-style so it takes a version subcommand plus --version/-version cases. Output is bare (just the version, node-style) so ver=$("$bin" --version) needs no parsing; that's the ops use case.

Design constraint (carried from the auto-upgrade work): --version reports the on-disk binary's version, not a running process's. So it can feed ensure_binaries' installed-version detection and /soul-status diagnostics, but it does not replace the .server-version launch-stamp used to decide whether to restart a live server. Two different questions.

Sequencing gotcha for adoption: the plugin can't call --version on its managed binaries until the pinned release actually includes it; old binaries error on the unknown flag/subcommand. So: land this → release server/client/tools → bump plugin pins → only then wire --version into ensure_binaries//soul-status. Test approach: each binary's version_test.go builds itself with -ldflags -X main.version=9.9.9-test and asserts --version echoes it, which also catches a mistargeted -X package path (would silently stay "dev"). Branch feat/binary-version-flag.

Plugin adopted --version for binary drift detection; dropped the .versions sidecar (2026-06-10)

Follow-on to the --version work, now that server 0.17.15 / client 0.12.39 / tools 0.1.32 ship the flag. The demarkus-memory plugin's ensure_binaries (plugins/claude-code/scripts/lib.sh) used a .versions sidecar file recording what it last installed. Replaced that with a live query: _binary_version BIN --version plus _installed_versions running each binary's --version, compared against the pinned _desired_versions. The binaries are now the source of truth: the sidecar could drift from reality (binary swapped out of band, sidecar deleted, partial install), and a live query also catches a corrupt or wrong-arch binary (fails to run → empty field → mismatch → re-download). Removed PLUGIN_VERSION_FILE, the end-of-install sidecar write, and the failure-path rm; a failed/partial install simply reads as a mismatch next session. demarkus-token accepts --version as well as its version subcommand, so one flag form (--version) covers all three. Pre-0.17.15 binaries error on the unknown flag → empty field → upgraded on next session start (the desired transition). /soul-status now queries --version per binary instead of cat .versions. Tested via stub scripts (working / old-erroring / missing / non-executable) and verified end-to-end against the real released binaries. Branch feat/plugin-adopt-version-flag, demarkus-memory 0.5.2→0.5.3. Pins bumped to 0.17.15 / 0.12.39 / 0.1.32.

pi slash commands registered but never executed; sendMessage without triggerTurn (2026-06-25)

Symptom. In the pi-agent ports (plugins/pi-memory, plugins/pi-knowledge, #211), every slash command (/soul-*, /promote, /knowledge-*) appeared to do nothing. The command was recognized (no "unknown command" error) but the agent never acted on it.

Root cause. The command handlers inject the bundled skill body into the session with pi.sendMessage({ customType, content, display: false }) and no options. Pi's sendCustomMessage (@earendil-works/pi-coding-agent dist/core/agent-session.js) branches three ways: streaming → steer/followUp; not-streaming + triggerTurn → _runAgentPrompt (starts a turn); not-streaming + no trigger → just append to state + emit message_start/message_end, no LLM turn. Typing a slash command happens while the session is idle (not streaming), so it hit the third branch: the skill instructions landed in history and the agent was never prompted to run them. display: false made it invisible too, so it looked like a complete no-op.

Fix. Pass { triggerTurn: true } in the command handler's sendMessage (both plugins/pi-memory/src/index.ts and plugins/pi-knowledge/src/index.ts). Idle → starts a turn; mid-stream → pi ignores triggerTurn and steers, which is the desired behavior anyway. The nudge/gate sendMessage calls elsewhere correctly pass { triggerTurn: false } (fire-and-forget reminders); only the user-invoked command path needs a turn.

Key insight. In pi, sendMessage is not a request to act; it's an entry append that only starts a turn when triggerTurn: true (or when streaming, via steer/followUp). sendUserMessage always triggers a turn. For a user-invoked command whose whole job is "make the agent do X now," you must trigger the turn explicitly; the claude-code mental model (inject prompt → it runs) does not carry over. No test caught this because the handlers were never exercised end-to-end against a live pi session; type-checking passes regardless.

Ignoring the lock-PID write creates a poison lock (2026-06-25)

Context. The demarkus-plugin (tools/demarkus-plugin) guards provisioning and registry writes with an atomic mkdir-mutex: os.Mkdir(lockDir) acquires, and the holder stamps its PID into lockDir/pid so stale-lock recovery can reclaim a lock whose owner died. Two sites: internal/provision/provision.go (withProvisionLock) and internal/registry/lock.go (withLock).

Bug (caught in PR #218 review). Both did _ = os.WriteFile(lockPid, …); swallowing the error: then ran the critical section anyway. If that write fails, the lock dir exists with no PID file. Stale-lock recovery reads lockPid to check whether the recorded owner is still alive; with no file the read fails and the recovery path falls through to sleep-and-retry, never reclaiming. Result: an unowned poison lock that wedges every future writer for the full bounded wait (~180s for provision, ~2s for registry) and then errors; with no owner to blame and nothing to clean it up.

Fix. Treat the PID stamp as part of acquisition: on write failure, os.RemoveAll the lock dir and return the error (fail closed) instead of entering the critical section. Don't hold a lock you can't prove ownership of.

Lesson / sweep. A _ = on a write that another code path later reads to make a decision is not a benign ignore: it silently breaks the reader's invariant. When auditing, separate "best-effort writes nobody depends on" (e.g. the guidance.go one-time-offer sentinels: a lost write just re-offers, fine to ignore) from "writes that establish state a recovery/cleanup path relies on" (lock PID stamps; must be checked). Grep for the lock acquisition primitive (os.Mkdir( mutexes), not just os.WriteFile, to enumerate every site: there were exactly two module-wide.

gh pr view <branch> resolves CLOSED PRs; automation edited a dead PR for 10 days

The plugin-pin-bump workflow's "open or update PR" step used gh pr view "$branch" to decide between gh pr edit and gh pr create. gh pr view (and gh pr edit) resolve a branch name to the most recent PR for that branch, including closed ones. After bump PR #210 was closed unmerged (2026-06-24), every daily run force-pushed auto/plugin-pin-bump and "successfully" edited the closed PR's title/body; no new PR, no failure, pins silently drifting behind releases for 10 days. The tell: #210's title named server 0.19.0, a version released a week after the PR was closed, and updatedAt kept moving.

Rule: in automation, branch→PR resolution must be gh pr list --head "$branch" --state open --json number and then operate on the PR number. Never gh pr view/edit <branch> when a closed PR for that branch may exist. Fixed in plugin-pin-bump.yml (2026-07-05).

General shape of the bug: an "upsert" whose existence check matches dead records. Same family as matching stale locks or tombstoned rows; the check must filter by liveness, not just identity.

Mirrored code without a fidelity test absorbs every fix twice

PR #230 (broker ergonomics parity): changedNote/seenDoc were copy-pasted between client cmd and broker as a "deliberate mirror". Two consecutive review rounds found wording bugs that had to be fixed in both places, and a later round found a guard present in one flow but not the other. Hoisting into a shared package (client/fetchdedup) ended it.

Rule of thumb: a deliberate mirror is only safe when a byte-for-byte fidelity TEST enforces it (as formatResult/formatToolResult has; that mirror has never drifted). A mirror kept in sync by reviewer vigilance will drift; hoist to a shared package at the SECOND two-place fix. The tools module imports the client module, so client-side shared packages (mdoutline, fetchdedup) are always available to the broker.

A gateway behavior change needs a consumer census, not just agent acceptance

Broker 0.5.0's mark_fetch ergonomics (outline mode >8KB, session unchanged-dedup) were verified against LLM agents on both MCP surfaces; but the demarkus-library is a THIRD consumer of the same gateway (broker mode reads every document through mark_fetch, one pooled MCP session per reader), and it broke two ways, silently:

1. graph.md (74KB) returned an outline → the floor's hub-topology parser found no Edges table → federation portals (soul.demarkus.io) vanished. The library's designed degradation ("any read/parse failure yields an empty topology") masked it: degradation paths hide regressions.
2. Re-opening an already-opened document returned status: unchanged with no body → parseToolResult's default branch → "unreachable" in the UI. unchanged was a NEW status value in a response contract that programmatic consumers parse by exact status.

Fix: the library's gateway Fetch passes force=true; a programmatic renderer always wants the real body; force bypasses both behaviors by design (library fix/gateway-force-fetch). Diagnosis pattern that worked: walk the data pipeline producer→consumer (root hub row → federated- systems.md → agent graph.md → floor parser); the break is at the first hop where healthy data stops being consumed correctly.

Rules of thumb:

• Changing a shared surface's RESPONSE SHAPE (new status values, new body modes) is a breaking change for every parser of that surface; census the consumers (agents, library, plugins, smoke scripts) before shipping.
• A consumer with graceful degradation needs a canary assertion somewhere (e.g. the floor should alert when the hub topology parses to zero edges but the doc fetch succeeded), or regressions land as quiet emptiness.

Deleting old versions broke a hidden v1-existence assumption (#236)

Version retention (prune oldest, keep last N) looked structurally safe: numbering derives from max-on-disk, VerifyChain checks consecutive present pairs. The store tests caught what analysis missed; migrateFlatFile detected "flat file needing migration" by checking whether v1 exists. After pruning v1, the next write misclassified the versioned document as a flat file and "migrated" the current symlink's raw bytes into a resurrected v1: doubled store frontmatter, broken hash chain, a version listing showing deleted versions back from the dead. Fixed by detecting any per-doc version history instead of v1 specifically.

Rules of thumb:

• Before adding deletion to an append-only store, grep for existence checks on specific low versions (v1, version==1 special cases); append-only code is full of "oldest version always exists" assumptions that deletion invalidates.
• Destructive filesystem ops in the store go through os.OpenRoot (Go 1.24+): paths resolve inside the root at operation time, so planted or raced symlinks error instead of redirecting the delete outside the store, and a final-element symlink is unlinked, never followed. Retention's pruneVersions is the reference example; reuse the pattern for any future delete/rename in protocol/store.
• The debug loop that found it: identical logic passed in a scratch test but failed in the suite → diff the sequence, not the assertion; the extra later write was the trigger, and find on the versions dir between steps showed v1 reappearing with double frontmatter.

Self-referential automation: the pin-bump/release loop (#244)

The plugin pin-bump workflow edited fallbackToolsVersion (a dev-build-only const in tools/) on every run and also used it as the tools currency signal. Consequence: a tools-only bump PR touched tools/, merging it cut a new tools patch release (semantic-version bumps on any change_path commit, chore included), the release event re-triggered the workflow, and a new bump PR opened. Release, bump, release, forever, one junk release per merge. Surfaced while running the second pin cycle for the style gate: PR #243's only provision.go change was the fallback const.

Fix (#244): the currency signal is now the bootstrap TOOLS_VERSION (the real pin, as the script's own header already said), sampled across EVERY plugin bootstrap with the lowest value winning so a partially-bumped tree self-heals; fallbackToolsVersion rides along only when provision.go is already changing for a real server/client pin.

Rules of thumb:

• An automation that edits a path inside its own trigger's change_path is a loop until proven otherwise. Trace one full cycle (edit, release, trigger) before merging its output.
• Never use a value the automation itself writes as its convergence signal; key off the artifact users actually consume (here: the bootstrap pin).
• Same family as the gh-pr-view-closed-PRs bug: automation state checks must reference ground truth, not the automation's own byproducts.

kqueue cannot see a same-name atomic symlink swap (configwatch flake, 2026-07-13)

CI failed TestDebounceCoalescesBurst; local -race sweeps then failed TestSymlinkRetargetTriggersReload 8 of 12 runs. Two distinct causes, one shared lesson.

Debounce flake: 10 writes at 5ms intervals under an 80ms debounce, asserting exactly one reload. Any scheduler stall > 80ms between writes fires the debounce mid-burst. Fix: widen the window relative to the pacing (400ms vs 2ms writes) and wait deterministically (waitForCount, then one more full window to prove no straggler).

Watch-establishment race (the general class): every trigger test started Run on a goroutine, slept a flat 50ms, then fired one-shot filesystem ops. fsnotify only reports changes made after registration, so under load the ops land before Add and are lost forever. Fix: awaitWatchLive helper writes a sentinel file repeatedly (a single touch just re-loses the same race) until a reload is observed, pausing a full debounce window between touches so a pending reload can fire instead of being re-armed forever, then drains stragglers and resets the counter. Never sleep-and-hope to synchronize with a watcher goroutine; prove liveness with an observable round trip.

Symlink swap invisibility (the deep one): with the watch provably live, the retarget still delivered zero events, deterministically. Diagnostic with timestamped raw fsnotify events showed CREATE current.new arrives instantly, but the rename's events never arrive: not after 3s of silence, not after a sentinel kick; only teardown's name changes surfaced anything. Mechanism: fsnotify's kqueue (macOS) backend detects directory changes by rescanning and diffing entry names. rename(current.new, current) changes no names once both ops coalesce into one rescan window, so it emits nothing. The test only ever passed when the scheduler let the rescan catch current.new briefly existing; -race and suite load made it lose that race almost always. Linux inotify reports renames explicitly by cookie, so CI never saw this one. Fix: skip the test on darwin with the mechanism documented, note the caveat in the package doc. Production (Linux) keeps the real guarantee; macOS dev still works in practice because staging the new link fires its own event.

Debugging pattern that cracked it: when a test is flaky, first make the precondition deterministic (watch-live sentinel); if the failure then becomes deterministic, the flake was hiding a real absence, not a timing wobble. Then drop below the abstraction (raw fsnotify with timestamps) and test delivery against silence and against a kick, which distinguishes "late" from "never".

goldmark: BaseInline.Lines() panics by contract (2026-07-13)

Walking AST ancestors from an inline node (a link) and calling p.Lines() on each parent panics whenever the parent is another inline (emphasis wrapping a link: **[text](url)**): goldmark's BaseInline.Lines() is panic("can not call with inline nodes."), not an empty result. Guard ancestor scans with p.Type() == ast.TypeBlock before touching Lines().

Found by CodeRabbit on PR 251 after the edge-provenance work shipped a blockStart walk; every existing test had formatting inside links, none had links inside formatting, so the panic was invisible. Test both nesting directions when touching inline AST handling. Repro test added at links/mdoutline/crawl levels.

Mock fixtures encoded a plan assumption; production data refuted it two deploys later (2026-07-14)

Graph hub seeding (PR 253) shipped broker-side with tests whose /graph.md fixtures used mark://{worldName}/... rows, because the plan's test sketch did. The production aggregate keys rows by cluster-internal dial addresses (name.namespace.svc.cluster.local:6309), which the gateway's URL grammar cannot even express, so seeding was inert in the real topology; and only the hub world publishes /graph.md at all, so per-queried-world seeding never found the aggregate. Two follow-up PRs (256, seed-all-worlds), each with its own release and deploy bump, fixed what one look at real data would have caught pre-merge.

The plan had even flagged the risk ("cluster-internal names in the aggregate") and mis-assessed it as "harmless, they are just labels"; they are lookup keys. The same bug class (key-form mismatch) was caught pre-merge on the demarkus-mcp side precisely because that surface got a live cold-start check against the real soul; the broker settled for its mock harness and met real data only after deploy.

Lessons:

1. When mock-tested code consumes a production-generated document, fetch a real sample and make it the fixture. The real aggregate was one mark_fetch away the whole time.
2. Re-derive a plan's risk assessments during implementation instead of inheriting them; "harmless" written at planning time is a hypothesis, not a fact.
3. If one surface gets a live verification and a sibling surface only gets mocks, expect the sibling to carry the bug the live check catches.

GITHUB_TOKEN events never trigger workflows (dead release trigger, 2026-07-18)

Symptom: the plugin pin-bump PR for a release landed ~16 hours after the release, on the daily cron, despite plugin-pin-bump.yml declaring release: [published] as a trigger. The trigger had never fired once.

Cause: GitHub suppresses workflow triggering from events created by the default GITHUB_TOKEN (anti-recursion rule), and release.yml publishes via GoReleaser with exactly that token. So any release:/push:-triggered workflow downstream of a GITHUB_TOKEN-published release is dead by construction. The documented exceptions are workflow_dispatch and repository_dispatch: those DO fire even when invoked with GITHUB_TOKEN.

Fix (branch pin-bump-dispatch): release.yml gained a final dispatch-pin-bump job (gh workflow run plugin-pin-bump.yml, needs the three release jobs, fires when any succeeded; actions: write added to permissions). The dead release: trigger removed from plugin-pin-bump.yml with a comment explaining why; the daily cron stays as backstop.

Lesson: when a workflow chains off an event another workflow creates, check what token created the event. GITHUB_TOKEN-created events are invisible to triggers; chain via explicit workflow_dispatch instead (the sanctioned exception), or publish with a PAT/App token. The roadmap's "pin chain is self-driving: release to auto bump PR" claim was believed for weeks while the release leg had never fired.

install-stack.sh indexing agent seeded the wrong world port (2026-07-18)

The appliance's WORLD_INTERNAL was localhost:6399; the world server listens on 6309 (protocol.DefaultPort, and install.sh uses 6309 everywhere). The indexing agent seeds/hubs pointed at 6399, so it could never reach the world: no crawl, no /graph.md republish, and the library's graph/backlink views stayed empty. Silent because the agent unit is Restart=on-failure and just kept retrying a dead address. Fixed to localhost:6309. Lesson: the agent's seed port must equal the server's actual listen port, not a transposed digit; there is no second internal port in the appliance.

Library "world unreadable" on same-host installs: hairpin NAT

Symptom: reading room serves fine at https://soul.demarkus.io/ but the universe shows "soul.demarkus.io:6309 unreadable", while external clients read the world over QUIC without issue.

Cause: install.sh writes DEMARKUS_HOST=<domain> into the library env when a domain exists (chosen so cert verification works; the cert can never match localhost). The library then dials its own host's public address from inside, which requires hairpin NAT; the Pi's network does not hairpin UDP, so the dial fails. External paths work, the loopback path does not.

Fix applied on soul.demarkus.io: DEMARKUS_HOST=localhost:6309, DEMARKUS_INSECURE=true, restart. Cosmetic side effect: the universe tile is labeled localhost:6309 instead of the domain (QUIC-mode world name comes straight from DEMARKUS_HOST).

Diagnosis tell worth remembering: in QUIC mode the universe tile name IS the configured DEMARKUS_HOST, so a tile still showing the old host after an env edit means the process never picked up the change. That was the second layer here: the first restart never actually ran, and the tile name proved it from the outside.

Open installer follow-up: same-host library installs should not assume hairpin. Options: probe the domain dial at install time and fall back to localhost+insecure, or document the trade. Related: the world name/label should arguably be decoupled from the dial address.

Final resolution (superseding the localhost workaround above): 127.0.0.1 soul.demarkus.io in the Pi's /etc/hosts, env back to DEMARKUS_HOST=soul.demarkus.io with DEMARKUS_INSECURE=false. The dial resolves to loopback so hairpin never enters the picture, the Let's Encrypt cert matches the dialed name so verification stays on, and the universe tile keeps the domain label. Verified end to end: tile readable, documents render, insecure=false confirms cert verification on the loopback dial.

Plugin token minted with "/*" instead of "/**" (2026-08-10)

Provision minted the plugin token with -paths "/*". Token path matching uses path.Match for patterns without **, and * does not cross /, so /* grants only single-segment paths (/index.md yes, /<slug>/journal/x.md no). Agents hit auth denials on nested writes right after install and had to widen the scope by hand. Only /** is recursive (server/internal/auth/auth.go matchPath).

Two-part fix in tools/demarkus-plugin/internal/provision:

1. Mint with /**.
2. The idempotent gate (token file + label present, return early) never re-checked scope, so existing installs would keep the narrow token forever. Added tokenScopeStale: an entry with /* and no /** falls through to revoke + remint, then SIGHUPs the server at that root so the running process reloads the token table (without the HUP the fresh raw token fails auth until restart).

Lesson: an idempotent "already provisioned" check must validate the provisioned state's content, not just its existence, or config fixes never reach existing installs. Note demarkus-token generate still defaults -paths to /*; hand-minted tokens without an explicit -paths are single-segment scoped.

Follow-up: gate now verifies content, and commit implies reload (2026-08-10)

Review iteration on the same fix converged on two structural invariants for ensureTokenEntry:

1. The idempotent gate verifies the raw token against the TOML entry's protocol.HashToken hash (rawMatchesEntry), not just file existence. Any inconsistent raw/entry pair self-heals via remint.
2. The server SIGHUP happens before the token file commit. A committed token file therefore implies the server loaded its entry, which eliminated the fallible remove-on-failure recovery (and the marker-file design it was sliding toward). When a recovery path needs a fallible cleanup step to stay correct, reorder so the failure leaves the pre-commit state instead.

Discovery-metadata move broke a second consumer (2026-08-11)

Broker 0.14.9 (#287) moved RFC 8414 metadata off the MCP gateway host to the issuer host to fix strict clients (pi's issuer-mismatch failure). The impact analysis concluded "the only known consumer was the kind smoke test" and called the change non-breaking. Wrong: demarkus-library derived its OAuth discovery URL as {broker.url}/.well-known/oauth-authorization-server against the gateway origin, and its own appset comment documented exactly that dependency. Library sign-in broke in production once its 5-minute discovery cache expired.

Lessons:

1. Before declaring a removed endpoint unconsumed, grep every sibling repo (library, agent, plugins, deploy) for the path literal, not just the repo being changed. The dependency was written down in the deploy repo the whole time.
2. A spec-violating surface that ships becomes a de-facto contract; in-house clients copy the shortcut (fetch AS metadata from the resource origin) instead of the spec chain (RFC 9728 PRM, then authorization_servers[0], then RFC 8414). Fix: the library now follows the chain with a single-origin fallback for old brokers.
3. Bonus find: the library's /token/revoke had been posting to the gateway host since inception, a silent 404 on every logout. Nobody noticed because Revoke errors were not user-visible. Silent-failure paths hide broken integrations; surface them.

Test migrations against a copy of real data, not just fixtures

The eager legacy-layout migration (2026-08-11, fix/store-flat-file-migration) passed every synthetic fixture, then failed instantly against a copy of the real soul: versions/..v1 (junk from an old bug that published base ".") made legacyVersionBase derive base "." and aim the current-pointer rename at the store root itself. Real stores accumulate junk that fixture authors never imagine. Before shipping anything that rewrites on-disk state, run it against a copy of production data and inspect what it skipped as well as what it moved; and when parsing filenames into paths, reject bases that are hidden, empty, ".", or otherwise unpublishable before joining them into a filesystem target.

Hidden directories hold served documents; migrations must walk them

The 0.22.2 legacy-layout migration skipped dot-directories, but FETCH serves explicit hidden paths: /.well-known/agent-manifest.md. With the legacy read fallback deleted in the same release, updating soul.demarkus.io broke its agent manifest (LIST-hidden is not FETCH-hidden; isHiddenEntry governs listings only). Found minutes after running demarkus-install update because the post-update verification included demarkus info, which fetches the manifest. Fixed live by hand-migrating the file, then in code by removing the walker's dot-dir skip (branch fix/migrate-hidden-dirs). Lessons: a migration's skip list must match the serving path's reachability, not the listing's visibility; and always exercise the well-known endpoints in post-upgrade verification, they use the store differently than normal documents.

Issue #289: configwatch log feedback loop (2026-08-12)

The plugin-managed soul server's .log grew to 1.5 GB. Two composed defects: configwatch reloaded on ANY event in the tokens.toml directory (never checked event.Name), and provision placed the server's log at <soulDir>/.log, inside that watched directory. Each reload logged "configwatch: reloaded", the append emitted an fsnotify event, and the 150 ms debounce re-armed forever.

Gotchas worth remembering:

• fsnotify's kqueue backend (macOS) reports Write events on files inside a watched directory, so co-locating any continuously written file with a watched config file self-sustains the loop.
• A pure event.Name == target filter breaks the symlink-retarget guarantee: the retarget event lands on the sibling current symlink, not the target. Fix filters by name AND op: sibling Write/Chmod drop; Create/Rename/Remove (structural swaps) still reload.
• The launchd install never looped because its log lives in ~/.demarkus/logs/, a subdirectory: non-recursive watches don't see subdir writes. That asymmetry was the diagnostic clue.

Fix (branch fix/289-configwatch-log-loop): op+name filter in server/internal/configwatch/watcher.go; managed server log moved to ~/.demarkus/logs/server-<base>-<hash>.log with legacy .log removed on migration in tools/demarkus-plugin/internal/provision/provision.go. The running old server keeps looping until the fixed release rolls out; truncation alone regrows ~55 MB/day.

Update: same branch also moves the managed server's tokens.toml out of the content root to ~/.demarkus/tokens/<soulID>/tokens.toml, migrated by rename (copy+remove across devices) only while the server is stopped in ensureManagedServer; resolution (tokensPathFor) prefers legacy until migrated. Reuse mode keeps <root>/tokens.toml: the adopted external server's -tokens flag is not ours. ensureTokenEntry now takes root explicitly since filepath.Dir(tokensTOML) no longer locates the server.

OpenCode update-check empty output

demarkus-plugin update-check intentionally emits no stdout when a plugin is current, throttled, disabled, or offline. The Pi adapter already treats empty output as no update. Both OpenCode adapters instead passed empty stdout to JSON.parse, producing Unexpected EOF warnings during startup and commands such as opencode mcp list.

OpenCode runBin now accepts expected empty output only for update checks. Other helper operations still surface an empty response as an error, preserving gate and guidance diagnostics.

The same expected-empty contract also applies to nudge when no reminder is due and guidance when no context exists. OpenCode now maps allowed-empty responses to {}, preserving the important distinction between a successful no-op and a helper failure (null). Gate responses do not allow empty output and still surface an error.

Hub truncated by a section fetch republished as the whole document (2026-08-17)

Symptom: /index.md showed (no title) in mark_graph and linked only Active Plans and RFC Review; every root section, plugin, sub-project hub, and plan archive was unreachable. Cause: at v69 (2026-08-12) an agent fetched /index.md#active-plans, edited it, and called mark_publish with that section as the full body. Eight later versions (cli, claude-code, opencode) edited the fragment without noticing. Found by /soul-doctor; last full version was v68; restored at v78 by publishing v68's body with the current Active Plans and RFC Review sections spliced in.

Rules:

• Never publish a body you obtained from a #section fetch or an outline. Before mark_publish on an existing document, mark_fetch with force=true and edit that body.
• A publish whose body has no # H1 on a document whose previous version had one is a strong truncation signal. Candidate gate check for the plugin: warn or block when the H1 disappears or the body shrinks by more than half.
• mark_graph from the hub is the cheapest detector: a hub with (no title) or a suspiciously low link count means the hub, not the graph, is broken.

Republishing a body through an LLM drops or adds its trailing newline (2026-08-17)

During a corpus-wide metadata sweep, four documents came back from mark_publish with a different content-hash than the version they were fetched from, even though the intent was a metadata-only write. The whole delta was the terminal \n: a body round-tripped through the model as a tool argument does not reliably preserve its final byte.

Consequences and handling:

• content-hash is exactly sha256 over the raw body, so any whitespace drift changes it. That makes a hash compare a complete verifier for "metadata only" writes.
• Detection: after a metadata-only publish, fetch again and compare content-hash against the pre-write value. Version must be old+1 and the hash must be identical.
• Recovery: fetch the pinned prior version (mark_fetch /doc.md/vN force=true), republish that exact body at the new expected_version, verify the hash again. History keeps both versions, which is the point of the store.
• Prevention: do not retype a body through the model at all. Fetch it to a file and publish from the file, byte for byte: demarkus -auth "$(cat ~/.demarkus/soul-soul.token)" mark://host/path.md > body.md, confirm shasum -a 256 < body.md equals the served content-hash, then demarkus -X PUBLISH -expected-version N -meta ... mark://host/path.md < body.md.
• Prefer mark_append when adding a section: it never retypes the existing body and carries catalog metadata forward.
• A -X PUBLISH invocation with neither -body nor a stdin redirect blocks reading stdin until the caller times out. Always redirect from the file.

The CLI graph crawl keys nodes by the raw start URL, the MCP crawl canonicalizes (2026-08-17)

Rebuilding the soul graph doubled it: 388 nodes for 192 documents, split across two key spaces, 204 under mark://soul.demarkus.io/... and 167 under mark://soul.demarkus.io:6309/.... Backlinks answered from such a store find only the half that matches the caller's key form.

Cause: demarkus graph passes fs.Arg(0) straight into CrawlAndPersist as the start URL (client/cmd/demarkus/main.go:349). The parsed host is used to fetch, but the node key stays whatever the user typed, and links.Resolve then derives every child key from that raw base. The MCP handler does the opposite: it rebuilds startURL as "mark://" + host + path after resolveURL has added the default port (client/cmd/demarkus-mcp/main.go:1276), with a comment saying the canonical form is required so crawled rows share the key form of the hub /graph.md and backlink lookups. So the two crawlers disagree on node identity for the same document.

Notes:

• Typing the port explicitly (mark://host:6309/path) makes the CLI agree with the MCP. A fresh canonical re-crawl of this soul produced 220 nodes with zero bare-host duplicates, which also proves no document body links to an absolute bare-host URL. The whole duplication was crawler-introduced, not corpus rot.
• Fix candidate: canonicalize in the CLI the same way the MCP does, before the crawl, so node identity does not depend on how the operator typed the address. This is the URL-normalization gap already recorded in /plans/graph-completeness.md.
• Detection: group ~/.mark/graph.json node URLs by host and look for the same host with and without a port.
• links.Resolve returns any destination containing :// untouched, so an absolute link written without a port would fragment the graph the same way even under a canonical crawl. Prefer root-relative links (/plans/x.md) in document bodies.

The installer updated the server but never the stack components (2026-08-19)

The droplet serving soul.demarkus.io was three library releases behind, which is how the reading room ended up showing pinned editions as backlink entries: the fix for that had shipped, but nothing on the host ever picked it up. Cause: demarkus-install update refreshed server, client, tui, mcp, token and publish, and never touched demarkus-broker or demarkus-library. Those are installed once by install or install-stack and had no update path at all, so a single-host deployment drifted for as long as it ran.

Fixed in #323 (71e43aa) by extracting fetch_library_binary, adding update_stack_component for the two optional components, and calling both from the update path.

Four things the review surfaced that are worth carrying forward, all of them variants of "reports success for work it did not do":

• The update returned early when the server was already current, so the component refresh was unreachable in exactly the common case: current server, stale component. That is the shape the original bug had too.
• An unresolvable tools release left the broker branch logging a skip and returning success.
• A failed systemctl try-restart was swallowed, so a component whose binary was replaced but never restarted looked like a clean update. It now records the failure and the command exits nonzero after finishing its other work.
• Writing over a live executable fails with ETXTBSY. The file already documented this hazard twice for the server, which stops the unit first; the component path now replaces by rename instead, which needs no downtime at all.

Method notes, because three separate checks in this one change passed for the wrong reason:

• When proving a test catches a bug by removing the fix, assert the mutation actually applied. One removal silently missed after gofmt rewrapped the condition.
• A "running binary" made of a shell script never triggers ETXTBSY, because the interpreter reads it rather than mapping it. Test the observable mechanism instead: replace-by-rename gives the destination a new inode, an in-place copy keeps the old one.
• A linear shell harness leaks state between cases. One case set PLATFORM=darwin and never restored it, which stayed invisible until a later case was added after it.

Hand-written conformance tests left seven backend divergences; a differential found them in minutes (2026-08-19)

The pgstore conformance suite (25 subtests) passed on both backends, and CI never ran it against Postgres anyway (no service container; every pg test skipped). A seeded random differential harness (storetest.RunDifferential, file store as reference) found seven real divergences within the first few seeds: .. handling, ENOTDIR leaking as a 500 under document paths, validation order vs version check, write-returned metadata spelling feeding the catalog, non-canonical keys in the hash index and catalog, LookupHash tiebreak on shared bodies, and LOOKUP scope cleaning. Several were bugs in the reference too (the file store was inconsistent with itself across a restart). Details and fixes in /plans/store-parity.md.

Lessons:

• A conformance suite proves the cases the author imagined. For "observably identical" claims, add a differential that compares full state after random ops; it costs a few hundred lines and finds what nobody asserted.
• A gated test that skips when its dependency is absent is a test that never runs in CI unless CI provides the dependency and a required flag turns the skip into a failure.
• Fuzz workers are separate processes; a fuzz target against one shared database needs -parallel 1 or per-worker isolation, otherwise workers reset each other's data and report false divergences.
• When two code paths key a map by request path, canonicalize at the boundary once (store.CanonicalPath); comments claiming keys are canonical are not evidence.

Hand-rolled layout knowledge in tests was silently wrong (2026-08-19, PR #331)

Two test tampers computed a version file's location as root/versions/<rel>/vN, but the real layout is <dir>/versions/<base>/vN. Both passed because every caller tampered a top-level path, where the two coincide. A simplify review agent caught it by diffing the helpers against newVersionFilePath. Fix: export the layout once (store.(*Store).VersionFilePath, sharing versionRelPath with getVersion) and pin it with a nested-path test that proves the tamper is visible to Get and breaks VerifyChain. Lesson: when a test must reach behind an abstraction, the abstraction should hand it the path; duplicated layout math in tests only fails on inputs the tests happen not to use.

A green diff harness can be comparing identical failures (2026-08-19)

The first run of scripts/e2e-backend-parity.sh passed with zero divergence, and every single check was the same Unauthorized error page: the chart's admin token grants read on /**, which makes every seeded path private, and the sweep sent no auth. A differential harness that diffs two outputs proves nothing unless the outputs contain the behavior under test; identical error pages diff clean. Guard against it by spot-reading the captured output for expected content (real bodies, version tables, lookup rows) before trusting PASS, and by recording each command's exit status into the compared output so an error path must at least fail identically on both sides.

A 64-bit constant that only fails on the release (2026-08-20)

const schemaLockID = 0x64656d61726b7573 compiled everywhere I build and broke the release:

internal/pgstore/pgstore.go:185: cannot use schemaLockID (untyped int constant
7234308641739470195) as int value in argument to tx.ExecContext (overflows)
target=linux_arm_7

The constant is 63 bits. Go's int is 32 bits on linux/arm/v7, which goreleaser builds and nothing else does, so the untyped constant had no valid conversion there. Local builds, CI, and the container images are all 64-bit, so every gate passed and the tag was cut before the first 32-bit compile ever ran.

Two lessons, the second more useful than the first.

Type any constant that exceeds 32 bits, and type it as what the destination actually wants. Here pg_advisory_xact_lock takes bigint, so int64 was right on its own terms and the platform break was a symptom of leaving it untyped.

The real gap was that the release was the first place five of six target platforms were ever compiled. CI built host amd64 only, so a whole class of error (32-bit int width, platform-specific syscalls, build tags) could only surface after a tag existed. test-server now cross-compiles every release target for all three binaries, both build flavors, which is seconds of work and reproduces exactly the failure above when run against the unfixed code. Any project whose release matrix is wider than its CI matrix has this hole.

Federation agent generated artifacts must satisfy hub publish policy

After shared-server cutover, crawler reads succeeded and the root token matched, but every /index/** and /graph.md publish returned server-error. Server audit logs exposed the actual cause: publish policy block: 1 violation(s). fedcrawl.publishIndex sent only agent and retention, while root policy required a category: tag and type. Generated artifacts must carry policy metadata too. Use tags=category:federation and type=Reference; do not weaken the hub policy or misdiagnose this as token/routing failure.

Concurrent GCS publishes to different paths contend on one world root CAS (2026-08-24)

Eighteen concurrent create-only publishes to different paths in the production latebit world produced seven successes and eleven server-error responses. LIST confirmed only the seven reported successes had committed; retrying missing paths serially succeeded. Different document paths are not independent in the GCS backend because every world mutation replaces one root object through generation CAS.

Treat a write-side server-error as an ambiguous outcome: reconcile the target path before retrying, especially for create-only writes. Multi-replica correctness does not imply parallel write acceptance. The backend needs a bounded rebase-and-CAS retry for independent mutations, or the client-facing error must identify root contention explicitly instead of returning a generic server error.

Correction: root-CAS retries already exist

The bucket store already performs bounded application-level CAS recovery: up to 12 candidate rebuilds, operation-receipt reconciliation for ambiguous outcomes, and 1.5-second per-process/world pacing. The 18-way production burst saturated that bounded envelope or request deadlines. Follow-up should expose contention clearly and add client backoff or broker-side serialization, not add a first retry loop.

Local soul writes broken by tokens.toml drift (2026-08-27, open)

All writes to the local managed soul (port 6309) return unauthorized. Root cause chain, verified live:

• The running server is NOT plugin-managed: it is the user's LaunchAgent (~/Library/LaunchAgents/io.latebit.demarkus.plist, ~/.local/bin/demarkus-server) with DEMARKUS_TOKENS=~/.demarkus/tokens.toml and DEMARKUS_ROOT=~/.demarkus/content.
• The plugin runs in MODE=reuse and adopted that server. Provision's reuse-mode contract assumes the adopted server reads <root>/tokens.toml (~/.demarkus/content/tokens.toml), and that file correctly lists the current plugin-memory.token hash (sha256-8bda6132...).
• The server never reads that file. Its watched ~/.demarkus/tokens.toml still holds a stale claude-code-plugin hash (sha256-d5002dbb...) matching neither the current token nor the plugin-memory.token.bak-repair-2026-07-27 backup. Likely fallout of the 2026-07-27 token repair rotating the token and writing the hash only to the root-relative file.
• Result: token rotation and server registry diverged silently; every MCP write, CLI write, and stdio mcp-serve write fails. Reads unaffected.

Immediate remedy (not yet applied): update the claude-code-plugin hash in ~/.demarkus/tokens.toml to sha256-8bda61324a244a86963e86a16669f51a7647c3a710af9d0f7f98660f07635377; the server hot-reloads via configwatch. Backup exists at tokens.toml.bak-2026-08-27.

Real bug to fix in the plugin: reuse mode cannot assume <root>/tokens.toml. An adopted server's DEMARKUS_TOKENS can point anywhere, so provision must either discover the adopted server's tokens path (or refuse adoption when it cannot), verify the token actually authenticates after rotation (a write probe), and /soul-doctor should flag a registry/token hash mismatch. Silent divergence violated the no-silent-error rule at the system level.

Cursor plugin spike findings (2026-09-03, Cursor 3.18.25)

Partial Phase 0 for /plans/cursor-memory-plugin.md; the agent-side checks were cut short by a free-plan model lock, so payload shapes for beforeMCPExecution, afterFileEdit, and stop stay unverified.

• Hooks bundled in a local plugin (~/.cursor/plugins/local/<name>/hooks/hooks.json) run without further enabling. The working directory is the plugin root, so ./hooks/x.sh commands resolve. CURSOR_VERSION is set; CURSOR_PROJECT_DIR and CLAUDE_PROJECT_DIR were empty (Cursor was open without a folder, so treat that as unverified rather than absent).
• Plugin mcp.json substitutes ${userHome} even though the plugin reference says only declared variables substitute. A relative command: ./scripts/x.sh never launched. Use "${userHome}/.demarkus/bin/demarkus-plugin" as the command.
• sessionStart fires per conversation with conversation_id, session_id (same value), model, composer_mode (agent, multitask), is_background_agent, workspace_roots, user_email, transcript_path (null when transcripts are off, the default). Do not depend on transcripts.
• beforeSubmitPrompt carries the raw prompt; MCP server prompts surface to the user as /<server>/<prompt> commands, so demarkus-mcp's orient, recall, whats-new come for free.

Cursor plugin install paths (2026-09-03, Cursor 3.18.25)

The plugin docs say custom marketplaces are Teams-only, but the Customize → Plugins → Add menu offers From Marketplace, From GitHub Repository, and From Local Repository to every user. So the public marketplace listing is a discoverability step, not the only distribution path: a free-plan user can point Cursor at latebit-io/demarkus directly. Do not run a ~/.cursor/plugins/local/ symlink and a repo install of the same plugin side by side: both load, every hook fires twice, and two demarkus-memory MCP servers spawn.

Cursor reads Claude Code hooks, not Claude Code plugins (2026-09-04)

Cursor's third-party-hooks reference: it loads hooks from .claude/settings.local.json, .claude/settings.json, and ~/.claude/settings.json, maps PreToolUse/PostToolUse/UserPromptSubmit/Stop/SubagentStop/SessionStart/SessionEnd/PreCompact to its own events, and translates permissionDecision to permission, permissionDecisionReason to user_message, updatedInput to updated_input, and a Stop decision: "block" plus reason to a follow-up message. Notification and PermissionRequest are unsupported. It does not read .claude-plugin/plugin.json, .claude-plugin/marketplace.json, a plugin's .mcp.json, commands, or skills, so a Claude Code plugin cannot be installed into Cursor as is; the compat layer only covers user-level hook settings. The Cursor plugin port stays necessary.

Correction: Cursor imports Claude Code plugins on boot (2026-09-04)

Cursor 3.18 has importPluginsFromClaudeOnBoot (default on) and importClaudeMarketplaces: at startup it reads ~/.claude/plugins/known_marketplaces.json and installed_plugins.json, registers every Claude marketplace, and installs the Claude plugins, including their .mcp.json servers (the Claude demarkus-memory plugin's server connected in Cursor before any Cursor plugin existed). The toggle lives in Cursor Settings under "Third-Party Imports": "Include Third-Party Plugins, Skills, and Other Configs"; it is reactive storage, not settings.json. Cleaning ~/.cursor/plugins/{cache,marketplaces,local} and the cursor.plugins.installedIds.* rows in User/globalStorage/state.vscdb resets Cursor's own state, but the import repopulates it unless the toggle is off. A marketplace added from GitHub is a sparse, blobless checkout that includes both /.cursor-plugin/ and /.claude-plugin/, so a repo that ships both manifests can list the same plugin name twice; precedence unverified. Open question for the port: how much of a Claude plugin (hooks with ${CLAUDE_PLUGIN_ROOT}, commands, skills) Cursor's import actually runs; if all of it, the Cursor-native plugin's remaining value is the native ask permission and the sentinel stop nudge.

CI "The operation was canceled" in test-agent-plugins (2026-09-04)

Not a cancel and not the npm warn allow-scripts lines (npm 11.12 warns about @google/genai and protobufjs install scripts pulled in by @earendil-works/pi-coding-agent; the passing runs print the same warning). The job has a 10-minute timeout-minutes, and GitHub reports a timeout as canceled. Two runs in a row saw each of the four sequential npm ci calls take 4 to 5 minutes against the registry instead of 10 to 19 seconds. Fix in PR #405: actions/setup-node with cache: npm over plugins/*/package-lock.json and timeout-minutes: 25. When a job "cancels" mid-step with no concurrency group in the workflow, check the step timestamps against the job timeout first.

Update: the npm ci hang is one request per install, not download speed (2026-09-04)

Four cancelled runs in a row on PR #405 shared one shape: the first npm ci finishes in 7 seconds, every later one reports "added 143 packages in 5m" with no "audited" line, then the next hangs. Five minutes is npm's default fetch-timeout, so a single request stalls per install until it times out; the missing audit line points at the bulk-advisories audit call. The same audit answers in about a second from a laptop, so it is runner egress or the registry's audit path from GitHub's network. Fix in PR #405: NPM_CONFIG_AUDIT=false, NPM_CONFIG_FUND=false, NPM_CONFIG_FETCH_TIMEOUT=60000, NPM_CONFIG_FETCH_RETRIES=1 on the install step, plus the npm cache from the earlier change and cache-dependency-path: tools/go.sum for setup-go (its restore had been failing with "Dependencies file is not found"). Caching alone did not help: the packages were already local, the stalled request was not a download.

Cursor MCP has one tool surface; bound souls are mark:// URLs (2026-09-04)

Live verification of /plans/cursor-memory-plugin.md in Cursor on the demarkus checkout, which is bound to soul.

The /memory command resolved the binding to id soul (mark://soul.demarkus.io), then looked for an MCP server named soul. Cursor only exposes demarkus-memory, whose tool descriptions say they are connected to mark://localhost:6309. The command treated that as disconnected and stopped. Soul reads were reachable the whole time: mark_fetch with url: mark://soul.demarkus.io/index.md returns soul documents. Bare /index.md is the local memory.

Soul writes from that same MCP return unauthorized. Plugin mcp.json launches mcp-serve with no --memory slug, so DEMARKUS_AUTH is the local token. Joined-memory writes need demarkus-plugin mcp-serve --memory soul (the /memory-join registration). Until that Cursor MCP entry exists, publish with the CLI and soul-soul.token.

Do not require a second MCP namespace per joined memory for reads. Pass the absolute mark:// URL of the bound memory on every mark_* call. Fix the /memory (and sibling) prompts so "server named <id>" is not the Cursor disconnect test. For writes, register the bound memory's mcp-serve --memory <slug> or the agent cannot publish.

Shell snippets in plugin prompts run under the user's shell (2026-09-04)

Command prompts that embed shell (memory-status, memory-join) are evaluated by the agent in the user's login shell, zsh on macOS, not bash. status is a read-only special parameter in zsh, so status=$? aborted the version-check loop in /memory-status and reported binaries as unknown (#407). Use bash-and-zsh-safe names (version_status, read_status); avoid status, path, argv, options. Hook scripts declare a bash shebang and are unaffected.

Prompt-only plugin changes still need a version bump

Editing a .tmpl under plugins/prompt-source and regenerating changes the shipped command text, but installed plugins only update on a manifest version change. #407 shipped without one; #408 cut the memory-lineage patch bump by hand (same six files .github/scripts/bump-plugin-pins.sh touches: claude-code plugin.json + marketplace, cursor-memory plugin.json + marketplace, pi-memory and opencode-memory package.json). Bump in the same PR as the prompt change.

CodeRabbit review loops on generated prompt copies (2026-09-04)

PR #412 went nine review rounds and 69 comments. Two causes. First, the assertive profile reviewed each template plus its four generated copies, so one finding arrived up to five times with slightly different wording; fixed by excluding plugins/*/commands|context|skills/** in .coderabbit.yaml (templates stay reviewed). Second, six threads that were already fixed stayed unresolved because CodeRabbit only auto-resolves when it recognizes the fix in a later diff; the PR stayed CHANGES_REQUESTED until each thread got a reply and resolveReviewThread. Before asking for approval, list unresolved threads with the GraphQL reviewThreads query, verify each at HEAD, reply with evidence, resolve, then @coderabbitai review.

Cursor drops commands whose frontmatter description has an inner colon (2026-09-05, Cursor 3.19.7)

Symptom: plugin slash commands missing from Cursor's / menu while the plugin loads with zero failures and CursorPluginsAgentSkillsService reports one skill instead of two. Cause: description: Audit the soul ... for catalog hygiene: orphans, ... is invalid YAML (a plain scalar cannot contain : ), and Cursor parses frontmatter with js-yaml, which rejects it ("bad indentation of a mapping entry"). Claude Code's parser is lenient, so the same files worked there for months. Five templates were affected: promote, soul-doctor, soul-refresh, knowledge-doctor, and the knowledge-promote skill. Fix: quote the description in the template; plugin-prompts now parses every frontmatter block with gopkg.in/yaml.v3 and fails the build on invalid YAML, and renderAlias inserts its prefix inside an existing quote. Verify a suspect file with the parser Cursor ships: node -e 'require("/Applications/Cursor.app/Contents/Resources/app/node_modules/js-yaml").load(fm)'. Useful Cursor forensics: ~/Library/Application Support/Cursor/logs/<session>/window*/exthost/anysphere.cursor-agent-exec/Cursor Plugins*.log for plugin loading, and the agentData.cacheStorage.agentEnvironment.slashMenuItems.* row in User/globalStorage/state.vscdb for what the slash menu last registered.

Cursor hides all plugin slash commands behind the third-party extensibility toggle (2026-09-05, Cursor 3.19.7)

Symptom: both Cursor plugins load with zero failures, skills count correctly, hooks and the MCP server run, yet no /soul*, /knowledge*, or /promote* command appears in the agent / menu; only the MCP prompts (/demarkus-memory/orient etc.) show. Cause, from workbench.glass.main.js: loadAllCommands computes allowExtensibilityCommands = thirdPartyExtensibilityEnabled && featureGate("enable_cc_plugin_import") and shouldLoadPluginCommand returns that flag for every plugin source, including cursor-third-party marketplace plugins, not only Claude imports. The toggle is stored as cursor/thirdPartyExtensibilityEnabled in User/globalStorage/state.vscdb (default true); Fritz's machine had it false, which the hooks log shows as Claude Code hooks disabled (thirdPartyExtensibilityEnabled off). UI label: "Include Third-Party Plugins, Skills, and Other Configs", description "Automatically import agent configs from other tools". Every earlier session where commands worked shows loadAllPlugins ... claude=true, meaning the commands came from the Claude marketplace import while the toggle was on; the Cursor-format commands had never been exercised on their own. Discovery itself is lenient (Xc falls back to regex when js-yaml fails), so the invalid-YAML fix in #420 mattered for the gray-matter content parse (Cu returns null and drops the command) and for skills, but not for this symptom. Forensics: Cursor Plugins*.log for loading, cursor.hooks.*.log for the toggle state, sqlite3 state.vscdb "select * from ItemTable where key like 'cursor/thirdParty%'".

Stop hook journal nudge fired mid-task and on sessions with no edits (2026-09-08)

Reported by a user: the session-end journal nudge blocked a Stop while a background agent was still running, and claimed files changed when nothing had been edited. Two causes in plugins/claude-code/hooks/session-journal-nudge.sh:

1. The "changed files" signal was a raw grep '"name":"(Edit|Write|NotebookEdit)"' over the transcript JSONL. Every transcript carries a prompt_snapshot attachment with the system prompt, which contains those exact bytes, so the grep hit on every session (one hit on a session with zero edits). Any tool output or pasted text with the same bytes hits too.
2. Claude Code fires Stop when the main turn yields to a background agent (Agent tool) or background Bash command, not only at the true end. The nudge's decision:block then forced an extra turn mid-task and burned the once-per-session sentinel.

Fix: the bash grep is gone; demarkus-plugin nudge --event session-end --transcript <path> parses the JSONL and derives signals from tool_use blocks only (internal/nudge/transcript.go). Pending background work is detected by joining launch receipts (Async agent launched for Agent, Command running in background prefix for Bash, matched by tool name so quoted text cannot fake it) against <tool-use-id> values in <task-notification> lines. Pending work returns no nudge and leaves the sentinel unset, so the nudge still fires at the real end. Scans a 4 MB transcript in about 60 ms.

Lesson: never derive facts from a transcript with a substring grep; parse the content blocks. And a Stop event is a pause, not an end, whenever a background task is outstanding.

Update after /simplify (same day): the phrase matching on launch receipts was dropped. Claude Code writes harness metadata beside each tool_result line as toolUseResult: isAsync:true on an Agent launch, backgroundTaskId on a backgrounded Bash command. The scan now keys pending work on that structure, so no tool allowlist and no wording dependency. The Stop adapter script also shrank to the recall-nudge shape: it pipes the raw Stop payload to demarkus-plugin nudge --event session-end --format claude; the binary reads transcript_path, stop_hook_active, and session_id itself and owns the once-per-session sentinel ($TMPDIR/demarkus-memory-nudge-<session_id>, same name the bash version used). The --transcript flag never shipped.

Plugin reuse mode never version-checks the reused server (2026-09-11)

Symptom. Every archived document in the local soul rendered as a live link in directory listings, then failed to open. Through the demarkus-library reading room that surfaced as a 404 reading "not found"; the reading room maps demarkus status archived onto ErrNotFound, so the archived state was invisible at the call site. Reproduced against the world directly: mark_list / returned pi-gate-test.md, pi-gate-test2.md and pi-gate-test3.md with include_archived defaulted false, and mark_fetch /pi-gate-test3.md answered status: archived. Same in subdirectories, where /demarkus-library/ listed three documents that are all archived on disk.

Root cause. The soul was being served by demarkus-server 0.17.19. Hiding archived documents from LIST landed in ef45913 (2026-07-02) and first shipped in server/v0.19.0, so the running server predated the filter by two minor versions. It listed everything and refused archived documents only on fetch.

The plugin had already downloaded a current binary. The machine carried two installs and ran the wrong one:

A launchd agent, io.latebit.demarkus, starts the ~/.local/bin copy at login with DEMARKUS_ROOT=~/.demarkus/content. The plugin never touches it, because ~/.demarkus/plugin-memory.conf says MODE=reuse. The 2026-06-10 fix above stamps and restarts the managed server on version drift, and deliberately exempts reuse mode as the user's own server. Correct as far as it goes, but reuse mode then performs no version check of any kind and reports nothing, so a stale user-owned server degrades silently.

Supporting state on the box, all of it pointing at a plugin that has lost track of what is actually running:

• ~/.demarkus/version reads 0.17.19 while ~/.demarkus/bin/demarkus-server is 0.35.0. Worth settling which install that file is meant to describe.
• ~/.demarkus/content/.server-version reads 0.18.0, stamped by some earlier managed run, matching neither.
• ~/.demarkus/content/.pid holds 36773, a dead process. The live listener is PID 775.

Fix direction for the plugin. In reuse mode, probe the reused server and warn on drift below the pinned floor, naming both the path and the version found. Do not auto-restart a server the plugin does not own, and do not silently adopt it either. A SessionStart warning is enough, because the two-install case is otherwise undetectable from inside a session: everything looks healthy until a listing entry fails to open. Note that demarkus-server -version now answers on both 0.17.19 and 0.35.0, so the stamp file is no longer the only available source of truth; the 2026-06-10 entry's claim that the binary has no version flag is out of date.

Generalize. A "reuse the user's own service" mode trades control for safety and then usually reports nothing at all. Any such path needs a version or capability probe at startup. Without one, feature drift reaches the caller disguised as data corruption, and the investigation starts several layers away from the cause. Here it started in an unrelated repo's web UI.

Adjacent finding, not chased. The same root listing advertises tokens.toml, tokens.toml.lock and a tokens.toml.bak file by name. They are 0600 on disk, but any reader of that world learns the paths exist. Worth checking whether a current server hides them.

World root listing advertises the server's own tokens file (2026-09-11)

Sharpens the adjacent finding noted at the end of the reuse-mode entry above, which recorded the symptom without establishing how far it goes.

What is exposed. The local world's root LIST names tokens.toml, tokens.toml.lock and tokens.toml.bak-repair-2026-07-27. They sit in ~/.demarkus/content, which is both the served tree and where the server keeps its own operational files, so they are listed like any other entry.

What is not. The content is refused. Every read path tested returns not-found: the reading room's raw route, its document route, and a direct fetch. The files are 0600 on disk. So this is name disclosure, not content disclosure, and the names alone say a token store exists and roughly when it was last repaired.

Scope. It is in the world's own listing, so every client of that world sees it. It is not a reading-room bug and no single client can fix it. Filtering the names in a client would be that client deciding what a world may publish, at a layer that cannot enforce the decision and does not own the data.

Where the fix belongs. The server should keep its operational files out of the served tree, or hide them from LIST the way archived documents are hidden. The second option is the smaller change and reuses machinery that already exists, but the first is the one that stops a future file from leaking by default. Note that the same root listing on 0.17.19 also fails to hide archived documents, so both bugs live in the same listing path and are worth fixing together.

Related, already handled downstream. demarkus-library now distinguishes the two failure modes a listing can hand a reader. A retired document answers 410 with "archived" instead of 404 "not found", so a stale-looking link is no longer indistinguishable from a broken one. That makes the client honest about what it was told; it does not make the listing correct.

Reuse mode now probes the adopted server, plus two corrections (2026-09-11)

Follows the reuse-mode entry above, which named the fix direction but left the state of the machine and two earlier entries unsettled.

What shipped. HealthWarning in reuse mode no longer stops at liveness. It resolves the adopted process's executable, runs --version on it, and warns when that sits below the pinned serverVersion, naming pid, path, found version and floor. A second check compares the binary's mtime against the process start time and warns when the binary was replaced under a running process, the case where a server keeps serving old code from memory. Neither path restarts anything, because reuse mode means the server belongs to the user. Both warnings reach SessionStart through guidance memory and appear in /soul-status.

Three things the implementation turned up.

A plain exec.CommandContext timeout does not bound a version probe. Killing the probe leaves a grandchild holding the stdout pipe, so Output() blocks until that child exits, and a stub sleeping 60 seconds still took 60 seconds. cmd.WaitDelay force closes the pipes after the kill and is what actually bounds it. This is not hypothetical for ~/.demarkus/bin, which already holds a wrapper script alongside the real binaries.

ps -o lstart= has one second granularity, so a naive "binary newer than process" comparison fires on any process started in the same second as its binary was written. A skew window absorbs it. The first test run caught this as a false positive against the test binary itself.

ps -o comm= returns the path the process was invoked with, so it is absolute for a launchd or plugin spawn and a bare name after a PATH lookup, while Linux comm is a truncated name rather than a path at all. The resolver prefers /proc/<pid>/exe and discards a non-absolute answer rather than guessing, which leaves a server started by relative path unreported.

Correction to the tokens.toml drift entry. Its remedy was applied, not left pending. plugin-memory.token hashes to sha256-8bda6132... and both ~/.demarkus/tokens.toml and ~/.demarkus/content/tokens.toml carry that hash, so local writes authenticate. The plugin-side bug it describes, reuse mode assuming the adopted server reads <root>/tokens.toml, is still real and still open.

Correction to the tokens file listing entry. Server 0.35.0 already keeps tokens.toml, its lock and its backups out of the root listing, and hides archived documents there too. Both listing bugs were fixed upstream and 0.17.19 was simply many releases behind. There is nothing left to fix in the server.

The machine. The launchd agent io.latebit.demarkus now runs ~/.demarkus/bin/demarkus-server instead of the standalone ~/.local/bin copy, so one binary serves rather than two. Mode stays reuse, root and port unchanged. launchd does not restart a job when its binary is replaced, so an upgrade needs launchctl kickstart -k gui/$UID/io.latebit.demarkus, which is exactly what the new second check now says out loud.

Two more from review on the same probe (2026-09-11)

/proc/<pid> ModTime is not a process start time. Procfs does not define it as one, and it can reflect when the entry was instantiated in the dcache, meaning roughly first lookup. A staleness check built on it silently misses the case it exists for, and only on Linux. The bounded ps lstart path is a real start time on both platforms, so the procfs branch went away rather than being special cased. The test now waits before its first lookup, which is what separates a launch time from a lookup time.

Checking a PID's owner and then resolving its executable is two lookups with a gap. A process that exits in that gap can have its PID recycled by another user, and the probe would then run their binary. Owner and path now come from one ps observation, and the resolved file is refused unless it is a regular executable owned by us or root that others cannot rewrite. A refusal is reported rather than swallowed, because a binary the probe will not run means drift goes unchecked, which is not the same as healthy.

Linux ps lstart can predate the binary it launched (2026-09-11)

Comparing a binary's mtime against the process start time to detect a replaced executable is unsound on Linux. GNU ps derives lstart from the boot time in /proc/stat, which has whole-second granularity, and the display truncates again. Measured in a container: a stub built at 19:08:29.285 and launched a few milliseconds later reported lstart 19:08:28, one full second before its own binary was written. Any mtime comparison then reports a replacement that never happened. macOS ps is accurate here, reading an exact start time from the kernel and only rounding down for display, so a one-second skew is sound there.

The fix is to stop treating mtime as a supplement and make it a fallback. Where procfs exists, compare the inode behind /proc/pid/exe against the inode at the path. Linux refuses to rewrite a running executable with ETXTBSY, so every real replacement is a rename or an unlink and recreate, and the inode always changes. That makes inode identity both exact and complete on Linux, with no blind window at all, while macOS keeps the mtime comparison it needs.

Two testing notes. Tests that overrode the skew variable to zero were hiding the production value entirely, so the shipped behavior had no coverage. Replacing a test binary with os.Chtimes is not a replacement at all under inode identity; staging a copy and renaming over the path exercises both platforms.

exec.CommandContext does not bound a process that forks (2026-09-11)

Cancelling the context kills the direct child only. A --version implementation that forks leaves the grandchild holding the probe's stdout pipe, and Output() then blocks forever. WaitDelay does not rescue it: measured, the call never returned at all rather than returning late. Set SysProcAttr Setpgid and a Cancel that kills the negative pid so the whole group goes. A test for this should guard the call behind a select, or a regression hangs to the package timeout instead of failing with a usable message.

macOS syspolicyd can stall every freshly built binary (2026-09-11)

Symptom: go test hangs with no output and dies at the package timeout, across unrelated packages at once. A trivial freshly built Go binary that prints one line also hangs at 0 percent CPU, and sampling it shows the process parked in _dyld_start with a 112K footprint, meaning it never reaches main. Cause is syspolicyd pinned at 100 percent CPU, stalling Gatekeeper assessment on first launch. sudo killall syspolicyd clears it, and it needs a real terminal because sudo wants a TTY. Adding the terminal to Privacy and Security, Developer Tools exempts its children from the assessment.

bucketstore.Initialize on an existing world costs a full Open (2026-09-11)

Initialize looks idempotent and cheap. It is idempotent, not cheap: the head-exists branch calls validateExistingWorld, which calls bucketstore.Open, and that loads the root, every shard, and indexes every document body. Calling it unconditionally at world open meant every restart of a provisioned world paid two complete snapshot loads.

EnsureWorld returns early on a head hit and lets the caller's own Open do the validating, including the world-ID mismatch check that loadHeadObject already performs. The check is not lost, only relocated, and the error it produces names both IDs.

Lesson: a constructor-shaped name can hide a full read of the store. Check what a function's idempotence costs before calling it on every start.

A test that Fatals under defer group.Wait deadlocks (2026-09-11)

TestWorldManagerStaticModeFailsFast asserted that an empty bucket fails a static world open. When that became legal behavior the assertion failed, and the package hung for the whole ten minute timeout instead of reporting a failure. The goroutine dump showed only parked background loops, which sent the first look in the wrong direction.

Cause: deferred functions in a test body run before t.Cleanup, so defer group.Wait() waited on loops whose context, t.Context(), is canceled only during cleanup. Fatal, then deadlock.

Put the wait in t.Cleanup alongside an explicit cancel. A broken expectation then fails loudly instead of stalling the package, which matters most when the expectation breaks because the behavior under test deliberately changed.

Benchmark binaries need the uncommitted read-fence patch

The answer benchmark proxy starts demarkus-mcp with a read-scope flag that lives only as an uncommitted 21-line patch in the benchmarks worktree (client/cmd/demarkus-mcp/main.go plus client/fetch/read_fence.go). A binary built from main rejects the flag, the reader gets no tools, and the first attempt fails with a missing graph-preparation.json after one abstaining model turn. To benchmark a main change: detached worktree at the main commit, git apply the saved patch, copy read_fence.go, build, and record the binary hash, the commit and a fingerprint of the applied diff beside the cohort. Seen 2026-09-17 on the lean schemas run, cost 778 tokens.

Offline call-reduction estimates must charge history resend

An attachment or extra result that the reader does not use is paid on the turn it arrives and again on every later turn, because the agent loop resends history. The first-hit lookup replay counted each attachment once and predicted a small net saving; the live run showed +9 percent on latebit. Charge speculative payload by the remaining turns of the episode, and require a hit rate that clears that bar before a paid run.

Pruning derived from current outputs cannot remove a dropped output (2026-09-18)

plugin-prompts pruned stale files only under the output dirs of the brands it had just rendered, so removing a brand from --brands left its whole directory behind and check stayed green. The same defect surfaced three times in review because each fix worked around it downstream (a git clean in the regen script) instead of at the source. Rule: when a generator owns a managed tree, prune and drift-check the tree itself (plugins/brands/*/), never a set derived from what was generated this run; a removed input produces no artifacts and so no prune scope. The lifecycle test renders a brand, rewrites the file to {"brands": []}, and asserts check flags the old directory and write removes it.

Reservation and routing must share one predicate (2026-09-18)

The branded MCP server alias went through four review rounds because each layer compared names differently: routing (ServerMatches) accepts <plugin>_<id> suffixes with hyphen and underscore folded, while the alias collision check used exact strings and the join-time reservation used normalized equality. Every mismatch was a misclassification (a joined acme-memory routed as local alias memory). Fix: one exported predicate (config.ResolvesToLocalMemory) used by the gate, by SetLocalMemoryAlias's collision scan, and by every catalog insert, all under the same catalog locks. Second lesson: a rollback keyed on the value it wrote is ABA-prone; the alias file stores <write-token> <alias> and a failed mcp-serve restores only while its token is still stored, with the replaced value observed under the lock rather than read beforehand.

Claude Code command hooks fail open at their timeout (2026-09-23, #504)

gate-pre.sh exited 2 on a non-zero demarkus-plugin gate (#488), so a crash denied the write. A hang did not: when a command hook reaches its timeout in plugin.json, Claude Code cancels it, drops its output and continues PreToolUse through normal permissions. Non-zero exit is not the only fail-open path; a hook that guards a write must bound its own work below the registered timeout and exit 2 on expiry. Cursor's hooks.json has failClosed: true, which covers timeouts; Claude Code has no equivalent. Fix: the bounded process-group helper from scripts/mcp-launch.sh inside the hook, TERM at 12s, 5s grace, KILL, all under the 20s hook timeout. The gate job keeps the hook's stdin and stdout (payload in, deny/ask JSON out); only the watchdog is detached with </dev/null. Found by CodeRabbit on a downstream repo's generated copy of the hook.

Unquoted ${CLAUDE_PLUGIN_ROOT} in hook commands fails open on spaced paths (2026-09-23, #507)

Claude Code runs a shell-form hook command through sh -c after substituting ${CLAUDE_PLUGIN_ROOT} as text. A plugin installed under a path with a space (a home directory like /Users/Jane Doe) splits the unquoted command, the shell exits 127, and Claude Code reports a non-blocking hook error and continues: the publish gate never runs. Same fail-open family as the hook timeout (#504): anything that stops the hook from returning a decision lets the write through. Every hook command now reads "${CLAUDE_PLUGIN_ROOT}"/hooks/<name>.sh, the form the Claude Code plugin docs use. .mcp.json stays unquoted: its command is substituted and spawned as one path, and quotes there would become part of the path. scripts/test-hook-commands.sh runs every manifest command through bash -c under a directory with a space.

OpenCode V1 plugin loader calls every named export (2026-09-24)

Symptom: opencode 1.18.31 log shows failed to load plugin .../demarkus-memory.ts error="The \"paths[1]\" property must be of type string, got object" (same for demarkus-knowledge.ts).

Cause: when default is a bare function, V1's readV1Plugin (detect mode) returns nothing and the legacy path iterates Object.values(mod), invoking each export as fn(input, options). loadCommand(input) then hits path.join with an object. Verified in packages/opencode/src/plugin/index.ts for v1.16.0 through v1.18.31.

Fix: default export must be an object { id, server } (V2 adds setup). V1 then calls only server and ignores named exports. PR #509 does this as a side effect of V2 support. demarkus-knowledge.ts needs the same change.

Also: V1 already globs {command,commands}/**/*.md in the config dir, so Markdown commands installed for V2 are discovered on V1 too and win over the plugin's config.command (??). V2 default export schema ignores the extra server key.

WATCH implementation lessons (2026-09-27)

• Two readers on one stream lose data. subscribeOn read the acknowledgement with one protocol.NewWatchReader(stream) and then built the watchStream with a fresh reader: the first reader's bufio had already slurped the events that followed the ack, so a resume skipped about 4 KB of replay (seq 1026 to 1033 in the slow-consumer test). The reader that reads the first block stays with the stream.
• A graceful drain waits on watches. quicserve.Shutdown waits for streams; an open WATCH never ends by itself, so the old process held UDP 6309 for the full 10 s and the restarted one failed to bind. worldruntime.Drain closes the change hub (every watch writes closing) before Shutdown; the drain then takes milliseconds.
• Pipe-backed test streams must close their read side. A watch handler blocked forever on the closing block written into an io.Pipe nobody read, and runtime.Close in t.Cleanup hung the package for the 10 minute test timeout. t.Cleanup(pr.Close) turns the write into an error the handler treats as a departed peer.
• Per-connection state lives in the stream context. quicserve has no connection handle at the endpoint; ConnState travels in handlerCtx, and a test that expects the per-connection cap must serve through a context that carries it, or the cap silently does not apply and the watch waits for events instead of being refused.
• The closing cursor is the last delivered position, not the head. A test feed that put the head in closing made the client resume past an event published between the terminal block and the reopen; the real server uses Subscription.Cursor().

Streaming lessons from the WATCH review (2026-09-27)

• A heartbeat timer keyed off "last event received" goes silent when events arrive that the receiver filters out. Key it off "last block written", or the liveness check and anything riding on it (the token recheck) never fires under load.
• In a resumable stream, an encode failure must be told apart from a write failure. Treated the same, the stream ends without a terminal block, the client resumes from before the bad event and replays it at once, forever, with no backoff. Encode to io.Discard first, degrade the block, then skip it.
• Every send on a bounded channel from the goroutine that Close waits on must select on the context, including the ones that "cannot" run with a full queue; the reopen path proved one could.
• A refusal status on reconnect is not always final. Retry-later statuses (rate-limited, server-error) hit exactly when many clients reconnect after a drain, which is when ending them for good hurts most.
• A per-write snapshot diff over every path is O(N log N) hidden inside a write. Diff only the shards whose ref changed, and report a local write from its own result, not from a scan that may see a neighbour's commit first.

Smoke against stale binaries (2026-09-27)

scripts/smoke.sh only checks that */bin/* exist; it never builds them. After a worktree move the main checkout's client/bin/demarkus was days old and demarkus watch failed with "invalid URL: unsupported scheme" (the old binary took watch for a URL). Fourteen WATCH checks failed while the server side was fine. Run make server knowledge-server client tools before the smoke, or compare the binary's mtime with the last commit when a smoke failure makes no sense against the code.

Sleeping for an overflow is not waiting for it (2026-09-27)

TestWatchCloseWithAFullQueueOnResync slept 300 ms after publishing 1025 events and assumed the client's queue had overflowed and cancelled its stream. On the CI runner under -race the pump was still reading, the consumer's one read made room, event 1025 fit, no reopen happened, and the test timed out on "subscribes = [], want 2" (the empty string since prints as []). Locally it passed every time. A timing assumption in a test is a bug that only CI finds: wait on the signal the code path emits (here the feed counts streams that ended after the acknowledgement, which is exactly the client's CancelRead on overflow), never on a duration.

Smoke-script lessons from the replica pair (2026-09-27)

• wc -l pads its count with spaces on macOS: match ^ *2$ or pipe through tr -d ' ' before comparing.
• The file server logs text (msg=watch path=...), the knowledge server JSON ("msg":"watch"); a grep written for one finds nothing in the other and looks like a reconnect that never happened.
• A resync hands the client the head, so a publish made before the client reconnects is never replayed: a smoke that publishes right after a restart must first wait for the server's watch log line, or it races the client's backoff.
• A python edit script fed through a shell heredoc must not use EOF as its delimiter when the text it edits contains a heredoc of its own; the shell ends the outer one early and reports a parse error far from the cause.

A ratchet diff that is really a proxy timeout (2026-09-28)

The CI lint-ratchet job failed on PR 516 with every protocol/ finding shown as fixed ('-') and three could not import github.com/yuin/goldmark ... TLS handshake timeout (typecheck) lines as new ('+'). golangci-lint could not fetch a module on the runner, so it reported typecheck errors and none of that module's findings; the script diffed that against the baseline as if the code had changed. Local run with the same v2.9.0 held. Fix: rerun the job; scripts/lint-ratchet.sh now aborts with "could not compile" when the linter output carries (typecheck) lines, so the next one reads as what it is.

CI lint-ratchet: uncached module fetches (2026-09-28)

Two runs in a row failed in the lint-ratchet job on network hiccups: a TLS handshake timeout to proxy.golang.org while linting, then a sum.golang.org stream error during go install golangci-lint. The ratchet script was never the cause; read the failing step before touching the script it runs.

Cause: the repo has four Go modules and no root go.sum, so setup-go's cache never restored ("Dependencies file is not found ... go.sum") and go install ...@v2.9.0 refetched about 200 modules with sumdb verification on every run.

Fix: cache-dependency-path: "**/go.sum" on setup-go, golangci-lint installed from the pinned release tarball with its checksum (two github.com downloads), and go mod download per module up front with one retry, so a proxy failure fails that step plainly instead of surfacing as typecheck errors in the ratchet diff. The test-* jobs show the same setup-go warning; they fetch fewer modules and have not flaked yet.

Also: printf '%s\n' "$out" | grep -q pattern under set -o pipefail can report false when grep exits early and printf takes SIGPIPE. Capture the grep output and test it instead.

Flaky: bucketstore TestReadViewTimeout on slow CI runners (2026-09-28)

The test opened the store with RequestTimeout: 25ms, and Open bounds its own world walk by the same timeout, so a loaded runner failed at the open ("open bucket store: context deadline exceeded") before the read-view path under test ran. Fix on fix/gitops-world-tokens: open with the default timeout, then set store.requestTimeout before the blocking head store. Pattern: a short timeout meant for one call must not also bound the fixture setup.

Argo CD shows Synced but never applies a volume type change (2026-09-28)

A chart change that swaps a volume from secret: to projected: cannot be server-side applied while another manager (here helm from the original install) still owns the old key: the merge keeps both and the API server rejects the object as having two volume types. Argo 2.13 with server-side diff logs serverSideDiff error once, then reports the app Synced, so nothing rolls and nothing looks wrong. Detect: app Synced to the new revision while operationState.startedAt is old and pods carry the old image; kubectl diff --server-side --field-manager=argocd-controller -f <rendered> reproduces the validation error. Fix: one kubectl replace with the rendered manifest, after which Argo owns the fields. Any future volume-type or list-key change on a resource with stale managed-field owners needs the same.

Knowledge replicas race genesis on an empty bucket

Found 2026-09-29 in the umbrella kind smoke (fake-gcs, two knowledge replicas starting together). The replica that loses the race to initialize a new world fails open world "root": genesis: ensure world: blob precondition failed: bucket holds 10 objects but no world head, exits, and succeeds on restart once the winner has committed the head. Fails closed and self-heals, but a fresh install shows a crash-looped pod. Genesis should treat "objects present, head absent" as another writer mid-genesis and wait or retry, not as a fatal precondition.

Fix: EnsureWorld finishes a genesis-only bucket (2026-09-29)

Branch fix/umbrella-notes-and-genesis-race. Initialize was already safe to run concurrently (immutable objects are create-or-verify, the head create-only then validate); only EnsureWorld's "any object but no head" refusal made the loser crash. It now lists the bucket and proceeds when every key belongs to this world's buildGenesis model, refusing only a foreign key (named in the error). This also heals a genesis interrupted mid-write, which previously needed manual cleanup. Kind control on the same harness: pre-fix image 3 genesis crashes in 5 cycles, fixed image 0 in 5. An in-memory concurrent unit test passed on the old code (no interleaving), so the regression test writes a genesis prefix instead.

Harness lessons: kubectl rollout status returns before the old pod is gone, so a client started right after a fake-gcs restart can hit dial tcp ... i/o timeout; wait for one pod and let kube-proxy settle. kubectl run -i intermittently drops a short pod's output; preload fake-gcs buckets from /data/<bucket> directories instead of creating them over HTTP. Classify crashes from kubectl logs --previous, not current logs.

Knowledge writer memory growth

Status: open, found 2026-09-29 in the production week review (knowledge server 0.46.0 and 0.47.0).

In every rollout of the last week one of three knowledge replicas grows about 150 to 200 MiB a day (non-evictable memory, 90 to 498 MiB over three days on 0.46.0) while the other two stay at 25 to 65 MiB. The growing pod is the one taking all PUBLISH and retention prunes (the agent's hub index and graph writes every 5 minutes); a replica serving about 15k reads in 4 hours stays flat, so the growth tracks writes, not reads. Head receipts are capped (maximumReceipts), so they are not it. No pprof endpoint and no GOMEMLIMIT in the server, so the next step is a heap profile of the writer: add pprof on the health port, or reproduce with a publish-and-prune loop against one replica on kind. At this rate a pod reaches its 1 GiB limit in about five days without a rollout.

Root cause (2026-09-29)

Status: root-caused and fixed on branch fix/knowledge-writer-memory (not yet released). Prune and head receipts were red herrings; the growth is in the body-match section index (server/internal/catalog).

• Retained structure: the process-wide term vocabulary in catalog/sections.go (a map[string]term that never removed entries, capped only at 2^20 tokens). Every new body the replica indexed added its unseen tokens for good.
• Why it grew so fast: the agent's /graph.md (about 800 KB) ends with one 253 KB spaceless JSON line of source observations carrying nanosecond timestamps. fieldTokens lowercases the whole field, and each word run it interns (for example 114798459z) is a substring of that copy, so every new token pinned the entire 253 KB line in the map. About 1,500 new tokens and one pinned line per graph write, 288 writes a day.
• Why only one pod: every root world request (reads and all writes) lands on one replica, per GKE request logs (the pod n2qzj served 9,805 root FETCH and 222 PUBLISH in 2 hours; the others served latebit and music). Indexing happens on whichever replica refreshes root, so it tracks the root replica, not writes as such. Since WATCH, every replica polls each world every 5 s, so on server 0.48 and later all replicas would index every root body and all would grow.
• Second defect behind the cap: at 2^20 tokens the vocabulary stopped accepting new ones and body match silently stopped indexing new words until restart.
• Also found: catalog.firstH1 returned a substring of string(body), so each document this replica wrote pinned a full body copy through its catalog title. Bounded per live document, not the leak, but writer-only overhead.

Evidence: an in-process harness (bucketstore on the memory blob store, agent-shaped graph bodies plus five index documents, retention 20) with heap profiles at cycles 50 and 200. The diff outside the in-memory bucket was all under catalog.IndexSections: strings.ToLower in fieldTokens (the pinned lines) and internTerms (map growth). Over 400 cycles the non-bucket heap went 15.9, 34.5, 46.7, 60.6 MB on the old code and stayed at 5.2 to 6.8 MB after the fix.

Fix: term ids are document-scoped. Each DocSections owns a packed, sorted vocabulary string, and sections hold sorted uint32 positions in it; a query resolves each term once per document. The index now dies with its snapshot and never pins source text. firstH1 clones the title. On the repo's 1,140-document markdown corpus, body lookup cost is unchanged (2.6 ms for three queries) and heap is 2.84x body bytes, down from 3.1 to 3.35x. Regression tests: TestRepublishKeepsIndexBounded (60 republishes of an agent-shaped body; the old code grew 16 MB) and TestFromDocumentTitleReleasesBody.

Lesson: a process-wide intern table needs eviction or document scope, and any interned substring must be cloned; otherwise it keeps its parent string alive for as long as the table lives.

Kind verification (2026-09-29, namespace ks-mem on kind-knowledge-system): two isolated stacks ran side by side, each with one replica, its own fake-gcs bucket, a 1 GiB limit and a busybox writer running the demarkus CLI. The writer republished an agent-shaped /graph.md (569 KB with a 240 KB timestamped JSON line, retention 20), about one write every 1.5 s. The first 3 minutes also wrote the five index documents. The old image (main at 55ab2c7c) went from 25.4 to 357 MiB RSS over 630 graph writes in 20 minutes, about 0.52 MiB per write, which is roughly 150 MiB a day at production's 288 graph writes a day and matches what GKE showed. The fixed image stayed at 20 to 26 MiB RSS for the whole run. Head commits are spaced 1.5 s apart (commitInterval), so a loop can reach at most one write per 1.5 s per store.

Follow-up from review: section.text could also pin the body. stripMarkup returns a substring when a section's own text is a single line, because strings.Join of one element returns it unchanged. newSection now clones the text, and TestIndexSectionsReleasesBody guards it (5.1 MB retained across 8 indexes of a 720 KB body before the clone). The rule for the index and catalog entries: nothing kept long-term may alias string(body).

Scope: the leak was in the shared server/internal/catalog package, so demarkus-server had it too. filestore calls catalog.Put on every publish, append and archive, which runs IndexSections. It has been present since body match landed in server v0.33.0. How fast it grows depends on write content: prose adds few new tokens, while agent-style churn (timestamps, hashes, graph exports) leaks roughly one pinned line per write. internal/filestore TestPublishPruneHeapTracksLiveData shows it: 14.6 MB over 60 agent-shaped cycles on the old catalog, under 1 KB on the fix. The same fix covers both servers.

Status: fixed on main in fb609084 (#527); not yet released or deployed.

WATCH memory audit (2026-09-29)

Audit of the watch path against the memory rules, on branch feat/bucket-store-change-journal.

• Leak fixed: peerhint connections. peerhint.Sender.conns was keyed by peer address and only dropped a connection when a send to it failed. Every rollout moves pods to new addresses, and departed peers are never sent to again, so each rollout added dead *quic.Conn entries for the life of the process. forgetGone now closes and drops connections to peers missing from each resolve; TestSenderForgetsDepartedPeers fails without it.
• Duplicate removed: file store journal tail. The journal kept its own ring-sized copy of the hub's events (about 320 KB of event headers at the default ring, plus reallocation churn under the write lock). It now rewrites from Hub.Retained(), a transient copy, and holds the tail only between reading the file and seeding the hub.
• Checked clean: event strings never alias request or file bodies (paths are rebuilt by CanonicalPath, request metadata and WATCH blocks decode into fresh strings, frontmatter values alias only the small copied header block); the hub keeps no reference to its subscriptions; retiring a world drains its hub, so open watches end and release the store; the client watch queue and the handler goroutines die with the watch.
• Retained-heap tests: bucketstore TestPublishPruneHeapTracksLiveData/watch (ring 16, sealing on) matches the plain run; TestBacklogResumeRetainsNothing runs 200 drained resumes through sealed blocks.

Lesson for memtest.Retained: code that fans out goroutines (runParallel) grows the runtime's threads, sudogs and pool caches for the first few dozen runs, about 25 to 80 KB that pprof shows under runtime.allocm, acquireSudog and sync.Pool, none of it project code. Warm up before measuring and set the bound from the size of the leak you are guarding against (a retained backlog is about 17 KB per resume, 3.3 MB over 200, which the test was checked to catch), not from the noise floor.

Left as is: each hub preallocates its ring (4096 events, about 320 KB of headers) at world open, even for a world nobody watches. It is bounded and evicts; lazy growth is an option if world count rises.

Knowledge broker memory growth: leaked MCP sessions (2026-09-30)

Symptom: the knowledge broker's memory grows linearly per pod, 15 to 92 MiB in 3.5 days on 0.40.0 and about 12 MiB a day on 0.42.0 (GKE memory baseline).

Cause: mcp-go v0.44.0's Streamable HTTP server registers every initialize in MCPServer.sessions and activeSessions and removes it only on a session DELETE. A GET listen stream that closes does not help, because the GET path skips cleanup when POST already registered the session. Claude Code never sends DELETE: ingress logs show 1,404 new sessions in 40 hours against 10 DELETEs, all of them from Go clients.

Churn: nginx closes each idle SSE GET at exactly 60 s, its default proxy-read-timeout, since the gateway sets no heartbeat. Every few minutes Claude Code then sends a POST that mcp-go answers with a 19-byte 404 for an invalid session ID, and it re-initializes. The result is 40 to 99 sessions an hour from a handful of real clients. The reconnect trigger inside Claude Code is inferred from the request order, not verified.

Measured: a harness on mcp-go v0.44.0 with the gateway's options retains about 9.6 KB per session, linear from 2,000 to 8,000 sessions, and closing the GET stream changes nothing. With a DELETE it retains 54 B. Over 0.42.0's life that gives 1,080 sessions, about 10 MiB, which matches the observed rise.

Fix path, not applied: mcp-go v1.1.1 adds WithSessionIdleTTL. Its sweeper calls UnregisterSession, which also fires the gateway hook that drops fetch dedup state. On that version with a 1 s TTL the harness returned to baseline. A GET stream refreshes the TTL only when it writes an event, so pair the TTL with WithHeartbeatInterval under 60 s; that also stops nginx cutting streams. Route the sweeper's "Sweeping expired session" log, which carries session IDs, through the gateway logger. The branch code knowledge/internal/broker/gateway/mcp_gateway.go still calls NewStreamableHTTPServer with no options, so the leak carries into the merged binary. The memory broker has the same code but sees about 30 requests a day, so it stays flat.

Leaked MCP sessions: the applied fix

Applied 2026-09-30 on feat/broker-in-process, for the leak traced above. mcp-go went from v0.44.0 to v1.1.1 in client, tools and knowledge; the API change was nil (two test loops over the grown mcp.Resource switched to indexing). The gateway's transport takes WithSessionIdleTTL(30m), and its sweeper calls UnregisterSession, which fires the fetch dedup hook. WithStreamableHTTPLogger puts "Sweeping expired session" on the gateway logger. Gateway.Shutdown stops the sweeper; Broker.Close calls it for both gateways.

Not applied: the heartbeat. mcp-go's heartbeat is a JSON-RPC ping the client must answer, and the answer is a POST through gatewayAuth and the per subject limiter (default 10 a minute, burst 5). A 25 s heartbeat would spend about 2.4 of those a minute per listening session and verify a JWT each time. Without it, nginx still cuts idle GET streams at 60 s and clients still re-initialize; that is churn, not a leak, since every idle session is now swept. Raise the Ingress proxy-read-timeout if the churn matters.

Found while testing: a client that comes back on a swept session ID is still served (200). mcp-go's default StatelessGeneratingSessionIdManager accepts any well-formed ID, in v0.44.0 too. So a sweep never breaks a client; the ID just carries fresh state.

Regression test: gateway/mcp_session_test.go sets a 1 s TTL, initializes without a DELETE, and polls the registry through DeleteSessionTools, which does not touch the idle clock, until the session is gone. It then checks that the swept ID is still served. With the TTL off, the test fails at 5 s.

Also from the same upgrade: mcp-go v1.1.1's stdio server cancels in-flight requests when stdin reaches EOF (Listen calls cancelRequests() before draining its workers); v0.44.0 finished them first. A caller that pipes a request and closes stdin now gets context canceled, or no answer at all. MCP hosts keep stdin open for the session, so they are unaffected. scripts/smoke.sh piped one-shot calls into demarkus-mcp and lost 10 checks. Its mcp_call now keeps stdin open until the id: 2 answer arrives, the way a host does, and passes on bash 3.2 without coproc.

A bucket-name IAM condition does authorize bucket creation

Verified 2026-10-01 on knowledge-49722: a project-level roles/storage.admin binding conditioned on resource.name.startsWith("projects/_/buckets/<prefix>") lets the principal call storage.buckets.create for a bucket under that prefix. The Admin Activity audit log shows demarkus-memory-broker@ creating knowledge-49722-memory-fritz-995e13f7 on 2026-08-29 with no other project binding. Reviewers (CodeRabbit on deploy PR 163) claim the create is checked on the project and needs an unconditional grant; it does not. Check with gcloud logging read 'protoPayload.methodName="storage.buckets.create"'.

Production broker move: two rollout blockers

Found 2026-10-01 rolling deploy PR 163 (knowledge 0.51.0 with the broker in process).

• tofu-ci creates buckets under one prefix only. tofu/bootstrap/ci grants tofu-ci a custom bucket role conditioned on projects/_/buckets/<projectId>-demarkus-. A bucket named outside it (knowledge-49722-broker-state) fails the apply with a 403 on storage.buckets.create, after the destroys in the same apply already ran. Name tofu-managed buckets <projectId>-demarkus-*; the state bucket became knowledge-49722-demarkus-broker-state.
• A renamed container under server-side diff. Chart 0.51.0 renamed the Deployment container knowledge-server to knowledge. A stale helm field manager from the original pre-Argo install still owned fields on the old container, so the server-side apply dry run kept a knowledge-server entry with no image: ComparisonError ... containers[0].image: Required value, and Argo never synced. Fix: remove that managedFields entry (kubectl patch --type=json with a test on /metadata/managedFields/<i>/manager == helm, then remove); argocd-controller, the remaining owner, then dropped the old container. Before renaming any list-map key (container, port, volume) in a chart Argo applies server-side, check --show-managed-fields for other managers.

client-go leader election retires a replica after one lost term (2026-10-01)

leaderelection.RunOrDie returns once a renewal fails and never contends again, and it starts OnStartedLeading in a goroutine it does not wait for. The sweeper called it once per process, so every lost renewal removed one contender for good; after each replica lost once, nothing swept until a restart, and rollouts hid it. knowledge/internal/leader.Run contends again after every term and runs the lead on its own goroutine, so terms never overlap and shutdown waits for the work.

Testing it against the fake clientset has two traps. The Lease stores its duration in whole seconds, so a sub-second test lease reads as already expired and two replicas both lead. Neither fake.NewSimpleClientset nor fake.NewClientset enforces resourceVersion, so a leader's cached renew overwrites a "stolen" Lease; simulate loss with a reactor that fails every Lease call (an apiserver partition) instead.

A WATCH resumed on a lagging replica resyncs (2026-10-01)

A bucket-store replica learns peers' commits only by polling (peer hints, a 5 second backstop while it has watches, or a read). A watch that resumed there with a cursor from a replica further ahead was past this hub's head, and Hub.Subscribe answered resync, so the client rebuilt everything it derived. SPEC §6.8 does not list that as a resync reason. Fixed on feat/incremental-federation: changefeed.Backlog.CatchUp (the bucket store polls) runs once before refusing, and the fan-out starts such a watcher at since+1 instead of at its own ring's next, which would have replayed what the client already saw.

Capacity limits must travel with every design that depends on them (2026-10-01)

The knowledge server's write limit had been documented since 2026-08-22:

• The limit: one commit per 1.5 s per world per replica, because each commit replaces the world's head.json and GCS allows about one update per second to one object.
• Where it lived: /plans/knowledge-server.md and its real-GCS sizing results.

It never reached the plans that write to worlds. The federation plan's Cost section left it out until a step 7 test took 1.5 s per write, and the operator learned of it only after committing to the design.

Rule: any design that writes to a world states its write count per change and what that costs at the commit interval, and links the limit. When a new design depends on a measured envelope (write rate, memory, cold load), restate the envelope in the new plan before step 1, not when a test trips over it.

helm install --dry-run reaches the current kube context (2026-10-02)

helm install --dry-run (and --dry-run=client on helm 3.13) contacts the cluster in the current context: version discovery, plus every lookup in the chart. On this machine the default context is GKE production, so two NOTES renders during step 9 read production's API, including the knowledge chart's lookup of its config Secret in default. A third failed only because gcloud auth had expired.

Rule: render charts with helm template or helm unittest, or under a scratch KUBECONFIG pointing at a throwaway kind cluster, never with the default context. NOTES render only through install or upgrade, so test them with helm unittest.

An in-process watch died with its handshake and reopened every 20 ms (2026-10-02)

The production cutover to knowledge 0.52.0 showed the federation leader logging 250 watch lines a second at 0.27 CPU, while the other replicas used 3m.

• Cause. fetch.Watch.subscribe dials under a handshake context it cancels on return. QUIC treats that context as setup-only. The knowledge server's in-process Watch, by documented contract, ends the stream when its context ends. fetch.ConnDialer passed one to the other, so every in-process stream closed right after its acknowledgement, and pump's "cut stream" path reopened with delay 0.
• Why nothing caught it. Events kept flowing through the reopens, so every unit test, real-server test and kind run passed. No test counted reopens, and I never looked at the watch log rate in kind.
• Fix (fix/watch-reopen-loop):
  • ConnDialer dials with context.WithoutCancel(ctx), as net.Dialer treats its context.
  • Watch.run backs off, from 100 ms doubling to 5 s, after a stream that carried nothing past its acknowledgement.
  • Each fix has a mutation-checked test, plus TestDeriverHoldsOneWatch on real servers: one open over the deriver's life.

Rule: a context passed to a dial or open is setup-only unless the callee documents otherwise. An adapter between the two meanings owns the conversion. Any long-lived loop that reconnects gets a test counting its reconnects while idle, not only one checking results. In kind and right after a deploy, look at log rates, not just pass or fail.

Correction: where the fix lives (PR 537)

CodeRabbit pointed out that context.WithoutCancel in ConnDialer also made the open uncancelable: a dial waiting on its context would hang NewWatch past the handshake timeout. The setup-only context now lives at the source. knowledgeserver.Server.Watch takes ctx for the open alone, as a dial does, and its conn carries blocks until closed. ConnDialer passes ctx through unchanged.

Tests:

• TestWatchOutlivesItsOpenContext replaces TestWatchEndsWithItsContext.
• TestWatchCloseLetsGoOfTheWorld keeps the blocked-write release.
• TestConnDialerOpenEndsWithTheHandshake covers cancelability.
• The backoff test now also requires a reopen.

Each is mutation-checked, and the end-to-end TestDeriverHoldsOneWatch fails with the server change reverted.

A model of creates alone hid a read-path defect (2026-10-04)

The first in-process model of three replicas writing one world (GCS-shaped latency, 64 writers a store) ran three writers as fast as one, while the GCS gate had measured 94 writes/s against 365. The model wrote new paths only; a new path's build reads nothing. The gate's load was 70% blind updates of random documents, whose builds read the document from the bucket, and the defect was in the rebuild after a lost slot: requeued batches were rebuilt without their warm-ups, one 25 ms read per member, serially, on the committer. With a seeded world and the gate's mix the model reproduced the collapse exactly (91 writes/s, a starved store). Lesson: a latency model must carry the production write mix, creates and updates, or it exercises only the paths that read nothing; and a recovery estimate counted in bucket calls must count the calls a rebuild makes, not just the ones the loser's own path makes. Fix and numbers: ADR 0036 amendment.

Deploy PRs: render the chart with the exact values first (2026-10-05)

A deploy repo PR set replicaCount: 0 on the knowledge chart to scale production down for a maintenance window. The chart's validate include refuses anything under 2, so Argo could not render the app (ComparisonError) and silently left the old pods running; the PR's apps-immutable-check passed because it compares StatefulSet fields only, and only broker-smoke (a helm template of the pinned chart with the repo's values) failed. Before any chart or values PR: helm template the pinned chart version from GHCR with the appset's values block, or run scripts/smoke-broker-allow-domains.sh, and read its output. The chart's guards are the contract, and an unrenderable app is a frozen app, which can be a useful lever (self-heal cannot revert a manual kubectl scale while the render fails) but never a plan.

Heap-sample tests race a short idle eviction

Release run for #553 failed TestIdleSectionIndexReleasesMemory with "eviction released 0" though no server code changed. The test set the section index idle to 50 ms, then measured two memtest.Retained deltas. Each sample forces two GCs; under -race on a shared runner those outlast 50 ms, so the eviction landed between the build sample and the eviction sample. The first sample still counted the index, the second saw nothing change, and the assertion read the whole index as leaked. Rule: a test that samples the heap around a timed eviction needs the idle period to outlast the sampling's GC work by an order of magnitude, and should assert the state it is about to measure still holds before sampling. Fixed on fix/section-index-memtest-timing (idle one second plus an active check).

The server answers created to every landed write, updates included

Found 2026-10-12 by the kind MCP smoke, after unit tests, the repo smoke and gateway tests were all green. hub.DropLinks accepted only ok as a landed hub edit, so on the real knowledge server an archive that correctly dropped a hub bullet reported note: hub ... not updated: created. Every other write path already used protocol.IsWriteSuccess (ok or created); the fake backend in client/fetchtest answered ok to publishes, so nothing caught the one path that did not. The fake now answers created, as server/internal/handler does for publish and append (new or update), and three tests that pinned ok moved with it. Lesson: a test double that answers a status the real server never sends hides exactly this class of bug; mirror the server's status, and when a check lists accepted statuses, use the shared predicate.

Atomic rewrite drops the tokens file's owner and mode

On a Linux install, the server stopped reading /etc/demarkus/tokens.toml until the user ran chmod 777 on it. The installer sets root:demarkus 0640 on the file. demarkus-token generate and revoke (through client/token AppendEntry and WriteFile) write a temp file at 0600 and rename it over the original. Run with sudo, as the install summary suggests, the result is a new root:root 0600 file that the demarkus service user cannot read. The fix is matchExisting in client/token/perm_unix.go, which chowns and chmods the temp file to match the file it replaces before the rename, and fails closed if it can't. Lesson: any temp-and-rename write over a file that another user reads has to carry over the original's owner and mode.

Eval sandbox homes leaked 146 managed servers (2026-10-18)

/soul-status on 2026-10-18 found 146 demarkus-server processes (about 1.3 GB RSS) beside the real soul on 6309, on ports 16310 to 16454, each rooted at /private/tmp/e-*/home/.demarkus/plugin-soul, a directory that no longer exists. 19 started 2026-10-12 and 127 on 2026-10-15; none after.

Cause: before PR 602 (2026-10-16) the plugin eval ran the real SessionStart hook in each case's temporary HOME. With no config there, provision ran the default setup, found 6309 taken by a server it did not own, chose isolated mode (isolatedPortStart = 16310) and started a managed server. The eval deleted the HOME and the server kept running. Since 602 the eval's stand-in hook prints a fixed header and provisions nothing, and since 615 the real hook never provisions without a non-empty plugin-memory.conf.

Lessons:

• A managed server does not notice its root being deleted; nothing ties its life to the HOME that started it. Any harness that runs a provisioning hook owns the server it starts and must stop it, or point the hook at a stub.
• provision detect lists such orphans as ordinary SERVERS rows; nothing flags a row whose root is gone. A candidate for H4 in prompt mechanics, which already has detect validate its own rows.

Flaky: bucketstore TestSeveralWritersModel throughput ratio on CI runners (2026-10-19)

• Release run on 8cbaf694 (PR 628, no bucketstore change) failed test-server on R=2 closed loop: 135 writes/s, under 70% of one writer's 230; R=3 did 234 the same run. A throughput ratio measured on a shared runner, not a correctness check. CI skips the test under -short; only Release runs it, so the flake costs a server release and the pin bump.
• Fix of the day: gh run rerun <id> --failed once the run completes; the skipped semver-server, release-server and dispatch-pin-bump resume. Follow-up: turn the 70 percent ratio into a log line, keep the starvation, slot-loss, recovery and p50 assertions. Same family as TestReadViewTimeout.

A sentinel that means two things acknowledges nothing (2026-10-21)

• blob.ErrPrecondition covers both a failed create-if-absent and rejected input (an oversized object). A create path that reads it as "already there" turns a rejection into a committed reference to a missing object; the failure surfaces one read later as integrity. When a sentinel is shared, the caller must disambiguate by observing state (Head), not by the category.
• A layout rule keyed on one dimension (document count) silently assumes the others are bounded. Find every field that goes into the object and ask what bounds it; the unbounded one here was a heading copied into a title. Bound the input where it enters the catalog, and make the writer measure the encoded object anyway.

A world token with read flips the world to read-auth mode

On the knowledge server a world token file is reloaded live from the <world>-tokens Secret. A token minted with -ops publish,read makes the world require read auth: open LOOKUPs still answer ok but filter every row the requester cannot read (lookup read authorization failed in the log), and matches: 0 looks like absence. For kind testing mint publish-only tokens (demarkus-token generate -ops publish); the chart's values comment says the same. Seen 2026-10-21 while testing the near-duplicate gate.