soul.demarkus.io/debugging.md/v7 draft reader meta

Debugging

Lessons learned from bugs, investigations, and things that went wrong.

General Principles

  • Read the error message. The whole error message.
  • Reproduce before fixing. If you can't trigger it, you don't understand it.
  • Don't brute force. If something is blocked, step back and think about why.
  • When a test fails, read the test first — the test might be wrong.

Known Gotchas

YAML Auto-Typing

YAML will silently convert strings like yes, no, on, off to booleans. This is why frontmatter is parsed as map[string]string — we control the type conversion ourselves.

Symlink Resolution in Tests

When testing the versioned store, be careful with os.ReadFile vs following symlinks. The store creates doc.md -> versions/doc.md.v3 — some operations need the symlink target, others need the symlink itself.

Path Traversal

filepath.Clean handles most cases but you still need an explicit .. check after cleaning. A path like /../../../etc/passwd cleans to /etc/passwd which looks valid but is outside the content root. The check is: clean the path, then verify it doesn't escape the root.

QUIC Stream Lifecycle

Streams must be closed properly. If a handler panics or returns without closing the stream, the client hangs waiting for data. Always defer stream close.

Debugging Approach for This Project

  1. Write a failing test that reproduces the issue
  2. Fix the code
  3. Verify the test passes
  4. Check if the fix breaks anything else (make test)
  5. Run bash pre-commit.sh

The test-first approach isn't dogma here — it's practical. A test that reproduces the bug proves you understand the bug, and it prevents regression.

QUIC UDP Buffer Size Warning

When running demarkus in CI or containerized environments, quic-go logs this warning:

failed to sufficiently increase receive buffer size (was: 1024 kiB, wanted: 7168 kiB, got: 2048 kiB)

Not a failure. quic-go tries to increase the UDP receive buffer to 7168 kiB for optimal performance. In environments with restricted sysctl limits, it can only get partway there. The connection still works — everything publishes fine.

Fix (if needed): Increase the system limit with sysctl -w net.core.rmem_max=7168000. Not necessary for correctness, only for optimal throughput under heavy load.

First seen: CI publishing content to demarkus-hub (hub.demarkus.io).

OnNode Callback Concurrency

graph.Crawl invokes OnNode from multiple worker goroutines. Any mutable state captured by the callback must use sync/atomic or a mutex. The CrawlAndPersist MaxNodes counter uses atomic.Int32 for this reason. Always run go test -race on packages that use OnNode.

Glamour WithAutoStyle() Blocks TUI Rendering

Symptom: TUI shows "Loading..." for up to 60 seconds after fetch completes. Server logs confirm the FETCH is handled immediately. CLI works fine. A mouse scroll "unsticks" the display.

Root cause: glamour.WithAutoStyle() calls termenv.HasDarkBackground(), which sends an OSC 11 escape sequence to the terminal and reads the response from stdin. But Bubbletea is also reading stdin for input events. This causes two problems:

  1. Stdin contention — Bubbletea's input goroutine may consume the OSC 11 response, so termenv never sees it and blocks for the full 5-second timeout.
  2. Render corruption — The stdin race breaks Bubbletea's render cycle, leaving the view stuck on "Loading..." even though the model state has been updated. A mouse event forces a fresh render, which is why scrolling "fixes" it.

The TUI recreated the glamour renderer on every window resize (to update word wrap width), re-triggering the query each time. Multiple resize events at startup compounded into 30-60 seconds of blocking.

Fix: Detect dark/light background once in main() before tea.NewProgram().Run() takes over stdin. Pass the resolved style name ("dark" or "light") into the model. Use glamour.WithStandardStyle(styleName) instead of glamour.WithAutoStyle() — this hits the DefaultStyles map lookup directly, bypassing the terminal query entirely. Also added a -style flag so users can skip detection.

Guard: Both stdin and stdout must be terminals before attempting detection. If either is piped, fall back to dark.

Key insight: Never do terminal escape sequence queries inside a Bubbletea event loop. Bubbletea owns stdin once it starts. Any library that reads stdin for terminal responses will race with Bubbletea's input goroutine.

TUI Link Highlighting: Marker Injection and the Glamour Black Box

Context: The TUI needs to highlight the selected link in the rendered markdown and support clickable links. But glamour's API is Render(string) → string — a black box. It doesn't expose the goldmark AST it builds internally, and the rendered ANSI output doesn't preserve link structure. Link URLs are stripped from the output; only styled text remains.

Problem: We parse the markdown twice with goldmark. Once in links.ExtractWithPositions to get link destinations and their byte offsets in the source, and again inside glamour to render. If glamour exposed its pre-parsed AST (or accepted one), we could do a single parse, extract link positions, and annotate nodes directly for highlighting.

Workaround: Marker injection. Before passing markdown to glamour, we inject Unicode private-use-area codepoints (U+F0000 range) around each link's text inside the [...] brackets. Glamour treats them as literal characters and passes them through to the rendered output. After rendering, processMarkers scans the ANSI output, finds the markers, records their screen coordinates as linkRegion entries, and strips them (or replaces with ANSI reverse-video for highlighted links).

Goldmark AST quirks for link positions:

  • ast.Link nodes don't expose the byte offset of [ or ] directly. You have to walk the link's child ast.Text nodes to get their Segment.Start and Segment.Stop, then scan backward/forward in the source to find the brackets.
  • Links with inline formatting (e.g., [hello **world**](url)) have multiple child nodes. The ** markers sit between text segments but aren't text nodes themselves, so Segment.Stop of one child doesn't equal Segment.Start of the next.
  • link.Text(source) is deprecated in newer goldmark versions. Use the child text segments directly and build the text yourself.
  • Links with no text nodes (e.g., [](url)) produce no text segments. ExtractWithPositions handles this by emitting a LinkInfo with OpenBracket: -1 so the link is still in the navigation list but skipped during marker injection.

Region extension for clickable URLs: Glamour renders links as bold text followed by an underlined URL. The end marker sits after the text but before the URL. processMarkers extends each region past the end marker to also cover the URL portion, scanning forward through the space and URL characters and stopping at the next space. This makes both the text and URL clickable.

Marker limit: Start markers use U+F0000+i, end markers use U+F1000+i. Maximum 4096 links per document before the ranges overlap. injectLinkMarkers caps at maxMarkedLinks; excess links render normally but without highlighting or click detection.

Mouse mode: tea.WithMouseCellMotion() only delivers motion events when a button is held. Hover requires tea.WithMouseAllMotion().

Argo CD selfHeal Reverts Broker Writes to the World Tokens Secret

Context: demarkus-broker mints a token by appending a [tokens.usr_xxx] section to the world's tokens.toml Secret (the same Secret the chart created with the admin token at install time). When the world is deployed via Argo CD with syncPolicy.automated.selfHeal: true (the default for an "auto-sync" GitOps shape), Argo's reconciler treats the broker's appended sections as drift from the chart-rendered manifest and reverts them on the next sync pass.

Symptom:

  • POST /auth/callback returns 200 with a tokens array. Broker logs mint succeeded.
  • The broker-namespace issuances Secret has the new entry — broker side is fine.
  • The world's tokens.toml Secret only contains the original admin entry. kubectl get secret -o yaml | grep resourceVersion shows a high number (e.g. 1705 after a few minutes) — the Secret has been updated repeatedly, but every update is immediately overwritten by Argo's selfHeal cycle.
  • demarkus-server returns "unauthorized" for every minted token because they never make it to the file the server reads.

Why selfHeal does this: Argo's drift detector compares the live Secret's data against the rendered manifest. The Helm chart only renders data.tokens\.toml with the admin entry. Anything else — including the broker's runtime-appended [tokens.usr_xxx] blocks — is "extra content" that selfHeal removes to bring the Secret back into compliance with the chart-defined source of truth. There's no warning. The mint API call succeeds because the broker's Secret-update API call returned 200; the revert happens milliseconds later.

Fix: add ignoreDifferences to the Application (or ApplicationSet template) so Argo stops reconciling /data of the per-world tokens Secret:

spec:
  ignoreDifferences:
    - kind: Secret
      name: "<release>-demarkus-server-tokens"
      jsonPointers:
        - /data

This leaves the chart in charge of the Secret's existence + initial admin entry while letting the broker own its runtime mutations from then on. The narrower /data/tokens.toml jsonPointer also works but /data covers the case where the chart later adds another data field.

Key insight: Argo CD treats every Helm-rendered manifest as the desired state. Any resource whose contents are mutated at runtime by another controller (or by the broker, in our case) needs an ignoreDifferences carve-out for the field being mutated, OR the Argo Application needs selfHeal: false. Pick one — selfHeal=true with no ignoreDifferences is a silent data-loss machine.

Detection: how to spot this fast. When a runtime mutation against an Argo-managed resource appears to have no effect, check kubectl get <kind> <name> -o yaml | grep resourceVersion. If the version is climbing every few seconds without obvious cause, selfHeal is the cause.

Where this bites in production: any Helm-rendered Secret/ConfigMap/CRD that another component mutates at runtime. demarkus-broker writes to the world tokens Secret. cert-manager writes to TLS Secrets. external-secrets-operator writes to Secrets. Any operator that owns a CR's .spec may hit the inverse pattern (Argo trying to reset spec back to the rendered shape). The ignoreDifferences carve-out is the durable answer in every case.

First seen: Stage 4 of the kind harness (PR #134), 2026-05-14. Mint API returned 200 with valid tokens; the world Secrets stayed at the chart-default admin-only contents. Took ~30 minutes of broker-side debugging (logs, code-reading, RBAC checks) before checking the Argo CD Application diff revealed the selfHeal cycle. Now load-bearing knowledge for any production broker deployment with Argo CD.

Agents improvise body frontmatter that demarkus never interprets (2026-06-10)

Symptom. Browsing a new project's index doc in the TUI showed garbled output — a stray ## name: setext heading followed by loose description: / type: lines. The doc had been authored by a claude-code agent against an org knowledge system.

Root cause — doubled frontmatter. demarkus carries metadata out of band: the publisher passes a metadata object on PUBLISH, the store prepends its own version envelope (buildVersionFile, protocol/store/store.go), and on FETCH the server strips exactly one leading --- … --- block (stripFrontmatter, server/internal/handler/handler.go). The agent additionally hand-wrote a second frontmatter block into the document body:

---            ← store envelope (version/archived/meta.*), stripped on fetch
version: 1
meta.agent: claude-code
---
---            ← agent's own block, survives into Response.Body
name: ...
description: ...
type: reference
---
# real content

The server strips only the first block, so the agent's block reaches the TUI as literal body. glamour renders name: ...\n--- as a setext H2, hence ## name:. The block is also invisible to LOOKUP (which reads only tags/importance/title).

Why the agent did it. Two reinforcing causes. (1) The session guidance told it to set tags/importance via the metadata object but never mentioned title — so "record a name" had no documented home and it reached for the Hugo/Obsidian frontmatter reflex. (2) name/description/type is ubiquitous in LLM training data. Provenance check came back negative: nothing in the repo prescribes that block, the org template was the stock deploy default (unchanged), and docs/site/reference/markdown.md already warns against body ---. Pure improvisation.

Fix. Guidance, not code. Added "all metadata travels in the metadata object, never the body; recognized keys are title/tags/importance; map name→H1/title, kind→a type: tag, description→first sentence under the H1" to both context/session-guidance.md files and skills/soul-memory/SKILL.md. Naming title is the gap-closer. Deliberately did not add defensive frontmatter stripping to the TUI — Fritz wants the raw render to stay a faithful knowledge-document debugger (it's exactly what surfaced this). Open follow-up: a write-side guard in publish-gate.sh to warn when a body opens with ---.

trail
  1. soul.demarkus.io v7