demarkus-library — Roadmap & Resume

Pick-up-where-we-left-off doc. Authoritative plan is /plans/universe-library.md; the Reading Room redesign lives in /plans/reading-room.md (invariants locked in ADR 0005). Build model: Opus 4.8 default, Fable 5 for load-bearing design.

Where we are (2026-06-13 — Reading Room fully LIVE in prod, R3 deployed)

main HEAD e8a136b (library). The whole Reading Room is deployed and working in prod: R1 (Tufte pane) → R2 (trail engine) → the floor (pane zero) → R3 (previews, backlinks, graph pane), both transports (decision 16). Cluster versions: library 0.1.7, broker 0.1.36, worlds server 0.17.18 — ArgoCD Synced/Healthy (R3 deployed via PR #17 → v0.1.7, rev 0.1.7 confirmed on the running pod). Next: floor enrichment / Phase 2. Detail in journal 2026-06-13 + journal 2026-06-12.

• Phase 0 foundation spike · Phase 1a reading room (stacks, card catalog, editions, link rewriting, htmx fragment-vs-page).
• Phase 1b — library card: DONE. Broker confidential web clients (demarkus 117e2a6) · oauth client (PKCE S256, Basic auth §2.3.1 escaping) · server-side sessions + single-flight refresh · ctx threading on ports · broker MCP WorldGateway with per-reader pooled sessions (1 request per read; fixed the live 429s) · login turnstile · composition-root swap DEMARKUS_TRANSPORT=quic|broker · native TLS for local e2e (env at ~/.config/demarkus-library/dev.env, client id library-web-dev — dedicated dev broker client; prod is library-web). E2e verified live.
• Multi-world floor + federation (fc4277c): (world, path) addressing, /w/<world>/... routes, federated composite gateway — KS names via broker, host[:port] via direct QUIC (DEMARKUS_FEDERATION default true). Read token never leaves home host (test-pinned). Cross-world mark:// links clickable.
• Deploy + CI/release — DONE (PR #10, v0.1.0): goreleaser-pro, distroless Dockerfile, helm chart (replicaCount 1 + Recreate — in-memory sessions), hardened workflows. Module paths on latebit-io. Releases auto-cut on merge to main (PaulHatch per-module semver). Deploy = ArgoCD watching latebit-io/demarkus-knowledge-system-deploy; image+chart pins in apps/<app>/applicationset.yaml, world list + webClients in deployment.yaml.
• Rendering — DONE (PRs #11/#12): chroma classes-mode · GFM alerts · emoji · mermaid + KaTeX islands (lazy, degrade to source).
• R1 — Tufte pane: MERGED (PR #13), LIVE. 60ch column + 250px margin · pinned margin trust signals (status badge: metadata status: axis → frontmatter fallback → draft; tags; provenance; parsed-frontmatter properties; mark:// + raw-source escape) · sidenote goldmark extension (footnote→margin) · lookup-backed tag pages · /w/:world/raw/* · search box removed · authoring contract published (mark://soul.demarkus.io/.well-known/library/authoring.md).
• R2 — trail engine: MERGED (PR #14), LIVE (plans §"R2 build decisions"):
  • Trail URL /t/<pane>/~/<pane>?focus=i; ~ = separator; pure codec + click algebra (truncate/append, dedup→focus, spine→focus). Cap 10, raw slashes. Pane kinds: <world>/d/<path> (doc; trailing slash/empty = listing), <world>/tags/<tag>, bare u = floor.
  • Rendered-document cache (decision 9): port.DocumentCache + LRU. Focused-live: every click costs exactly 1 world read.
  • Canvas: spines / body-only parent / focused Tufte pane; tombstone spine for unreadable unfocused panes; trailizeLinks rewrites /w/ hrefs to post-click trail URLs + active-link breadcrumb. Spec published (.well-known/library/trails.md).
  • Codec note: paneAddrFromParts(world,kind,value) (trail.go) is the single source of truth for pane semantics; parsePaneChunk (/t/) and paneAddrFromRoute (/w/) both call it (PR #16). Share semantics, not syntax — don't re-duplicate.
• The floor — universe view (pane zero): MERGED (PR #15), LIVE (plans §"Universe-view research" + 2 addenda):
  • demarkus prereqs: mark_worlds broker tool (#182) lists authorized worlds; LOOKUP match-all * (b8aedb7) returns the whole catalog in importance order. World must run server ≥ 0.17.18 or it tombstones.
  • WorldGateway.Worlds(): broker parses mark_worlds; QUIC returns the home world; federated prefers Names→Hosts.
  • service.Floor/FloorCached: worlds × Lookup("*"); failed world → tombstone (dimmed, never dropped); ErrUnauthorized → re-login.
  • Web: pane kind u; / → /t/u; SSR SVG floorSVG (worlds as circles r∝satellite count, top-10 docs on orbit r∝importance, stroke=status, every node a plain ). ADR 0003's canvas island stays UNSPENT.
  • PR #16 fix: parser rejected the world-root listing chunk (<world>/d/); fixed + consolidated the two parsers.
• R3 — previews + backlinks + graph pane: MERGED (PR #17), v0.1.7, LIVE (plans §"R3 build decisions"; verified live in QUIC):
  • Edge source = render-time observed-links map (the load-bearing decision). rewriteLinks already resolves every link to (world,path); capturing edges is free and library-internal → transport-symmetric (no broker graph store, so decision 16 holds). Rejected: broker store (broker-only), library-side crawl (over-built). Cold state fills as the room is read; durable hub-sourced topology (addendum 2) is the upgrade layered in front later — the service edge methods are the seam.
  • Architecture: domain.Ref + domain.Neighborhood; edge store is an internal service.linkGraph (fwd+reverse maps, mirrors floorCache — no new port/adapter/wiring). Inbound port gains RecordLinks/Backlinks/Neighborhood. rewriteLinks returns (html, []Ref), rewriteHref returns (route, Ref, isDoc); web resolves+records, core owns the store.
  • Hover previews: previewize wraps in-app doc anchors with a preview-host/preview-card popover + htmx (mouseenter delay:300ms once → /w/:world/preview/* fragment). Pure htmx+CSS, no JS. One component for body links AND backlinks. Snippet = first paragraph, from ReadCached (budget-safe). mouseenter overrides click so hx-boost nav still fires.
  • Backlinks "referenced by" block in shared doc-meta (trail URLs on the canvas, /w/ permalinks single-doc). Empty ⇒ absent.
  • Graph pane: kind <world>/g/<path> via shared paneAddrFromParts; route /w/:world/g/*; SSR SVG graph.go (center + backlinks left / outbound right, each node continues the trail). Canvas island still unspent. Margin "graph" affordance opens it.
  • docs/trail-format.md + in-universe trails.md (v3) updated with the g kind (decision 14).
  • Repo pre-commit.sh added (from demarkus, single-module): fmt/vet → golangci-lint (pinned to CI v2.9.0) → build → test. Run before wrapping a phase/task.
  • Broker-mode R3 smoke still pending (interactive OAuth); parity is architectural (edge source is library-internal, no broker-only path).

How to resume

cd ~/latebit/demarkus-library && ./pre-commit.sh   # fmt/vet/lint/build/test
go build -o /tmp/dl ./cmd/demarkus-library
# quic mode (no auth): open /t/u (floor), a doc, hover a link, click "graph".
PORT=8080 DEMARKUS_INSECURE=true DEMARKUS_HOST=soul.demarkus.io \
  DEMARKUS_AUTH=<token from .mcp.json> /tmp/dl
# broker mode: source ~/.config/demarkus-library/dev.env && /tmp/dl
#   OAuth on broker.knowledge.demarkus.io. Port held? lsof -ti :8443.
# cluster (GKE): kubectl ctx gke_knowledge-49722_...; creds expire mid-session
#   → ! gcloud auth login. ArgoCD ns argocd; check operationState, not just
#   sync.status (stale on a retrying op).

File map (hexagonal — ADR 0002)

• internal/core/{domain,port,service} — entities (Document, Floor, WorldInfo, Ref, Neighborhood), ports; ReadingService = Read/Browse/History/Search/Tag/Raw + *Cached + Floor/FloorCached + RecordLinks/Backlinks/Neighborhood; WorldGateway adds Worlds(); DocumentCache; service/floor.go (catalog parse), service/graph.go (linkGraph observed-links store)
• internal/adapter/inbound/web — Echo routes (/t/* canvas incl. u/g, /w/:world/... incl. tags/raw/g/preview), trail.go (codec + click algebra + paneAddrFromParts, incl. g), trail_links.go, links.go (rewriteLinks → edges), trail_handlers.go (+ floorPaneView + graphPaneView), floor.go + graph.go (SSR SVG), preview.go (cards + previewize + backlinkLinks), templates (+ preview/preview-link), static/
• internal/adapter/outbound/{world,broker,federated,oauth,markdown,cache} — direct QUIC, MCP pool (Worlds parses mark_worlds), shape-router, broker OAuth, goldmark, LRU cache
• cmd/demarkus-library — composition root, AppConfig, brokerAuth shim
• docs/ — authoring-contract.md + trail-format.md (sources of the in-universe /.well-known/library/ docs; keep both in sync)
• pre-commit.sh — the phase/task gate (golangci-lint pinned to CI v2.9.0)

Next sessions — queued work

1. Floor enrichment (pairs with R3's edge-source seam)

Hub-sourced topology (plans addendum 2 — hubs are the primary map: durable, versioned, transport-symmetric) layered in front of the observed-links map; portal nodes for externally-linked hosts in quic mode; world-view zoom level. Hub discovery convention still open (manifest self-declaration vs DEMARKUS_HUB env — lean manifest + override). Graph-pane aggregation/cap for high-degree hubs (a doc with ~40 outbound links crowds the right arc).

2. Phase 2 — restricted collections (per /plans/universe-library.md)

Per-world/per-path authz surfaced in the reading room. Depends on broker-side authz semantics — re-read the plan + broker authz.go first.

Maybe / observed

• Broker-mode R3 smoke (dev.env): hover preview, backlinks, graph pane behind the turnstile — last unrun acceptance check.
• Deploy CI papercut: apps-immutable-check runs the guard via ... | tee, so the pipe masks the script's exit — the check can NEVER fail the build, only posts a PR comment. Add set -o pipefail / drop the tee.
• Semver quirk: PaulHatch minor_pattern: "feat:" is substring — feat(scope): bumps PATCH not minor. Fix via regex /^feat(\(.+\))?:/.
• Broker rate-limit drop (60/30 → 30/15) on fix/broker-rate-limit-down — confirm merged/synced.
• Visual polish by eyeball (pane widths/gaps, spine legibility, dark mode, canvas scroll; floor at many-worlds scale + label crowding; preview-card placement near pane edges; graph pane at high node counts).

Standing deferred

• Find + fix publishers that wrote metadata fences into bodies (R1's friendly frontmatter reduces harm; source fix still open).
• Shared session store (or sticky sessions) → replicaCount > 1 + RollingUpdate
  • PDB — the R2 doc cache, single-slot floor cache, and R3 observed-links graph are all per-pod (fine at replicaCount 1; revisit together — per-pod graph means backlinks differ across replicas).
• Adopt demarkus's strict .golangci.yml (18 issues to clean first).
• Phase 3 cataloging desk (writes, CodeMirror island) · Phase 4 AI librarian (SSE; joins the canvas as a pane, trail = its context) · Phase 5 public face.

History (condensed — details in journals + ADRs)

• 2026-06-10: Phases 0 + 1a built + merged (PRs #1–2).
• 2026-06-11: Phase 1b planned + shipped (PRs #3–#9); floor + federation; frontmatter strip. Gotchas in debugging.md.
• 2026-06-12: v0.1.0 deploy/CI (#10) · latebit-io migration · v0.1.1/v0.1.2 rendering (#11/#12) · ADR 0005 · R1 (#13) · R2 (#14) (journal).
• 2026-06-13: universe-view research · demarkus mark_worlds (#182) + LOOKUP match-all (b8aedb7) · floor merged (#15) · cluster rollout of R1+R2+floor (server 0.17.18 / broker 0.1.36 / library → 0.1.6) · trail root-listing fix + parser consolidation (#16) · R3 merged + deployed (#17, v0.1.7) — observed-links edge source, hover previews, backlinks margin, graph pane; repo pre-commit.sh added (golangci-lint pinned to CI v2.9.0). Reading Room fully live in prod (library 0.1.7) (journal).

Git / process notes

• Work on feature/* / fix/* branches. Never auto-commit — Fritz commits and opens PRs; CodeRabbit reviews via .coderabbit.yaml. Releases auto-cut on merge.
• Run ./pre-commit.sh before wrapping a phase/task (fmt/vet/lint/build/ test; golangci-lint pinned to CI's v2.9.0).
• Deploy = ArgoCD on latebit-io/demarkus-knowledge-system-deploy. Bump chart targetRevision + image.tag together in apps/<app>/applicationset.yaml. Library/broker are Deployments (clean roll); worlds are StatefulSets.
• PR scopes: domain, port, service, web, world, markdown, cmd, view, deps, ci, broker, auth, session, oauth, federated, deploy, docker, helm.

⭐ CURRENT STATUS (2026-06-14) — supersedes "Where we are" above

The Reading Room is feature-complete and fully deployed. Cluster (all ArgoCD Synced/Healthy): library 0.1.10, broker 0.1.37, worlds server 0.17.19, agent 0.12.43.

Shipped + verified since the stale section above:

• R3 (PR #17, library 0.1.7): hover previews, backlinks margin, graph pane.
• Floor enrichment: hub-graph topology → cross-world edges + portal nodes.
  • library hub-topology reader (0.1.8) reading <hub>/graph.md; DEMARKUS_HUB chart wiring (0.1.9); host↔name join via broker mark_worlds address column (broker 0.1.37 + library 0.1.10).
  • agent publishes /graph.md (--publish-graph, client 0.12.42/43).
  • server token-ownership fix (0.17.19): token Secrets created by a pre-install bootstrap Job, not chart-templated (ArgoCD was pruning/ corrupting the runtime-mutated tokens.toml — see debugging.md).
• External federation verified: floor portals to public worlds reachable from the cluster (docs.demarkus.io cloud + soul.demarkus.io once its residential ingress was opened). docs in debugging.md.

Full detail: journal 2026-06-13 (latest entries run into 2026-06-14) + debugging.md.

Next phases (queued)

1. World-view zoom — deferred floor-enrichment piece: one world's catalog as dir clusters + aggregate bubbles → listing panes (plan §Universe-view research, zoom level 2).
2. Phase 2 — restricted collections — per-world/path authz in the room (re-read /plans/universe-library.md + broker authz.go).
3. Standing deferred — shared session store → replicaCount>1 (R2 cache, floor cache, R3 link graph all per-pod); strict .golangci.yml; deploy CI papercuts (apps-immutable-check tee-masks exit; PaulHatch feat(scope): bumps patch not minor).

Loose end

Demo links in world-a/index.md (mark://root/index.md + mark:// soul.demarkus.io/index.md) added to exercise floor edges/portals — keep as a live example or strip (then restart the agent to re-publish graph.md).

Process

feature/* branches; Fritz commits, never the agent; ./pre-commit.sh gate (golangci-lint pinned CI v2.9.0); CodeRabbit per PR; releases auto-cut on merge; deploy = bump apps/<app>/applicationset.yaml targetRevision+image.tag (ArgoCD). knowledge MCP joined for root/world-a writes.

Update (2026-06-14, later) — world-view zoom SHIPPED

Cluster now library 0.1.11 (others unchanged). Next-phases item 1 (world-view zoom) is DONE + deployed: PR #21, reader-only, floor world node zooms to <world>/u/ (catalog as dir clusters + "+N more" → listing panes) + doc-margin "map" affordance. In-universe trails.md → v4 with the <world>/u/ chunk. Loose end resolved: demo links in world-a/index.md KEPT (legit related-links footer; floor edges stay live). Remaining queued phases: Phase 2 (restricted collections) and standing deferred. Only open item on this phase: human browser smoke of the world map at library.knowledge.demarkus.io.

Phase 2 plan recorded (2026-06-14)

Detailed Phase 2 plan written: /plans/phase-2-restricted-collections.md — sub-world read-authz tier (collections), broker-led, library projects. Settles D1–D4 (rules in broker WorldConfig; thread read identity; per-collection locked-vs-hidden; path globs) with recommendations; broker/library work split, acceptance, build order. Important, NOT urgent — scheduled later. Authz today is world-level only (reads tokenless/open within an allowed world), so this is a real broker security change, not just a library projection.

Phase 3 plan recorded (2026-06-14) — flagged important

Detailed Phase 3 plan written: /plans/phase-3-cataloging-desk.md — the WRITE side: create/edit docs in markdown source, a projection of the broker's mark_publish/mark_append protocol. Settles D1–D5 (htmx textarea + live server preview first, CodeMirror island only on escalation; editing is a focused-pane mode NOT a trail chunk; metadata as separate form fields — no --- fence; expected_version + merge flow; write authz coupled with Phase 2). First write path through the library (ends the read-only invariant). Library is the bulk; broker work only if write-collections (D5) are in scope. Fritz: Phase 3 is important — likely ahead of Phase 2 in priority.

Phase 3 STARTED — edit slice live (2026-06-14)

Cluster now library 0.1.12. Phase 3 (cataloging desk) is underway: the first write slice — edit an existing document — is merged (PR #22) + deployed (ArgoCD Synced/Healthy). The room is now reader+writer. Margin "edit" → htmx textarea + live server preview → publish via mark_publish (metadata out of band, expected_version conflict guard) → 303 back to the doc. Auth-gated; no broker/worlds/agent change. Plan: /plans/phase-3-cataloging-desk.md. Remaining Phase 3 slices: create-new document, mark_append affordance, conflict merge-candidate flow, write-collections authz (couples with Phase 2). Open: human browser write-smoke at library.knowledge.demarkus.io.

Next session candidate — R4 reader mode (2026-06-14)

Fritz wants a reader mode: a focus overlay that centers the document you want to read over the dimmed trail (context kept behind, one click/Back to return). Plan: /plans/reader-mode.md. It's a lens over the trail (NOT a new pane), URL-state (?reader=<i>, ADR 0003 shareable + Back-closes + no client state), web-layer only — no port/service/ read/broker/deploy change. Settle before building: persist-on-navigate (recommend yes), Esc-to-close, margin shown/collapsible. Phase 3 (cataloging desk: edit + create + append) is live at library 0.1.14 — a clean pause point; reader mode is the next nice-to-have. Remaining Phase 3 slices (world-map "new", conflict merge, write-collections authz) still queued but lower priority.

R4 reader mode BUILT (2026-06-15, branch feature/reader-mode)

The focus overlay is built + locally verified (NOT yet merged). Web-only lens over the trail: ?reader=<paneIndex> renders the focused document centered over the dimmed canvas (context preserved behind), ✕/backdrop/Esc close to the bare trail. Decisions settled (Fritz, all matched recommendations): persist-on- navigate, standalone ?reader=, ✕+backdrop+Esc (vanilla in islands.js, not _hyperscript), margin shown. Load-bearing: a valid ?reader= also takes Focus, so the overlay reuses the focused pane's already-fetched doc (no 2nd read; ADR 0005 d9 holds). No port/service/domain/broker/deploy change. Gate green; live QUIC smoke passed. Detail: plans/reader-mode.md SETTLED+BUILT appendix + journal 2026-06-15. Next: Fritz + CodeRabbit → merge (library 0.1.15) → deploy bump (reader-only).

R4 reader mode COMPLETE — deployed + lens fix (2026-06-15)

Cluster now library 0.1.16. R4 shipped in two PRs:

• PR #25 (0.1.15) — reader overlay built (?reader=<paneIndex>); merged with a CodeRabbit DRY fix (trailBasePath extracted).
• PR #26 (0.1.16) — lens-not-navigation fix. The original "reader implies focus" decision was wrong: opening reader on a non-focused pane re-focused it, collapsing the canvas behind. Fixed by decoupling reader from focus — the overlay reuses the reader pane's already-fetched doc (every pane is read in the loop; cached if unfocused), trailReaderURL preserves a non-default focus, close keeps it. The overlay is now a pure lens; the canvas behind never relayouts. Both reader-only deploys (no broker/worlds/agent change). ArgoCD Synced/Healthy on 0.1.16; Fritz confirmed live. Detail: journal 2026-06-15.

Remaining queued: Phase 2 (restricted collections) + Phase 3 leftover slices (world-map "new" for empty worlds, conflict merge-candidate flow, write-collections authz) + standing deferred.

Update (2026-06-18) — Phase 3 leftover slices SHIPPED + strict golangci; releases out to v0.4.0

All merged to main; releases auto-cut. Cluster currently runs library 0.3.0 (apps/demarkus-library/applicationset.yaml targetRevision+tag 0.3.0; broker 0.2.0, worlds 0.17.19, agent 0.12.43) — so 0.3.1 and 0.4.0 are released but NOT yet deployed (bump the applicationset to roll them).

Releases since 0.1.16 (intermediate fixes: 0.1.17 floor TTL-cache #27 · 0.1.18 no-store #28 · 0.1.19 empty-floor zero-world #29 · 0.1.20 signed-in email nav #30 · 0.1.21 markdown H1-title + ci #31/#32):

• World-map "new" affordance — SHIPPED (PR #33, v0.2.0, d2ad556): create-doc entry from the floor for empty/unreadable worlds; degrade on unreadable catalog. Clears the queued Phase 3 "world-map new" slice. Deployed (cluster ≥ 0.3.0 supersedes).
• Conflict merge-candidate flow — SHIPPED (PR #34, v0.3.0, 633a9ab): cataloging-desk write path reviews the broker's mark_publish merged candidate (on_conflict=merge) before republish. Clears the queued Phase 3 "conflict merge-candidate flow" slice. Deployed — current cluster (0.3.0).
• Strict .golangci.yml — SHIPPED (PR #35, v0.3.1, 895aac1/ac13ca0): adopted demarkus's strict config (revive/gocritic/modernize/…), findings fixed. Clears the standing-deferred golangci item. Released, not yet deployed.

Phase 3 remaining: write-collections authz only (couples with Phase 2).

Also shipped, NOT a tracked phase — login knowledge-graph backdrop (PR #36, v0.4.0, be8fdc0): login.html redesigned to a B&W dark theme with a faint full-page SVG knowledge-graph backdrop (jittered-grid network, degree-sized hub nodes, center-fade opacity for text legibility; decorative, aria-hidden; ADR 0003 no-JS form behavior unchanged). Incidental visual polish — see journal 2026-06-18. Released, not yet deployed.

Deploy action queued: bump apps/demarkus-library/applicationset.yaml targetRevision+image.tag 0.3.0 → 0.4.0 to roll golangci + login backdrop.

Still next (unchanged)

1. Phase 2 — restricted collections (/plans/phase-2-restricted-collections.md) — couples with the last Phase 3 slice (write-collections authz).
2. Standing deferred — shared session store → replicaCount>1 (R2/floor/R3 caches per-pod); deploy CI papercuts (apps-immutable-check tee-masks exit; PaulHatch feat(scope): bumps PATCH not MINOR); publishers writing --- metadata fences into bodies.
3. Phase 4 AI librarian (SSE pane) · Phase 5 public face.

Decision (2026-06-18) — write-collections authz (Phase 3 D5) DROPPED

Fritz: not important. World-level write access is acceptable — if an identity may write to a world, it may write to ANY path in that world. The coarse grant (world AllowConfig + the broker's per-world write token, op ["publish"], path globs) is the intended model, not a stopgap.

Consequences:

• Phase 3 (cataloging desk) is COMPLETE. Edit (#22) · create (#23) · append (#24) · world-map "new" (#33) · conflict-merge (#34) all shipped; D5 was the only open slice and it's now a non-goal. No further write-authz work.
• The library keeps gating the edit/new affordance on writability and degrading to "not permitted" on a broker reject — that's just honoring the world-level token, no per-path tier.
• Decouples Phase 2 from Phase 3. Phase 2 (restricted READ collections) stands on its own if/when wanted; it no longer has a write-side twin to co-design. Phase 3 plan D5 + its "coordinate with Phase 2" coupling are void.

Phase 3 plan (/plans/phase-3-cataloging-desk.md) D5 should be read as superseded by this note.

Phase 5 anonymous-read question SETTLED — plan recorded (2026-06-18)

Plan: /plans/phase-5-public-face.md — settles the universe-library open question ("does the broker expose any anonymous-readable surface"). Recommendation: yes, but a narrow, default-closed, operator+author opt-in public carve-out — NOT a relaxation of the org-scoped default.

Load-bearing finding (grounded in broker ratelimit.go/mcp_tools_read.go): the world read layer is already identity-blind — reads dispatch with an empty bearer (tokens.toml is write-only), so requireAuth at the broker edge is the only thing making org reads private. Anonymous-public is therefore a narrow fail-closed bypass for explicitly-public resources, the inverse end of Phase 2's read-authz axis — build them on one model or a public world × restricted subtree will leak.

Decisions: D1★ tier yes/default-closed · D2 operator-eligible ∧ author-marked (visibility:public) · D3 one evaluator shared with Phase 2 (public: true rule flag) · D4 dedicated fail-closed anonymous read path (not requireAuth-bypassed gateway) · D5 follow-a-link only, NO anonymous catalog/list/worlds in v1 · D6 session-less library render + OG + bluemonday · D7 ipRateLimit + cacheable + residential worlds stay behind the broker. Settle D1/D2 with Fritz before building. Cross-repo (broker leads). Prefer Phase 2 first or joint.

Direction (2026-06-18) — DEFER forward phases; focus usage + robustness + code quality

Fritz: pause net-new feature phases. The reading room is feature-complete and deployed; next priorities are (1) get real usage, (2) make the system robust, (3) raise architecture/code quality before continuing the phase ladder.

Deferred (plans recorded, not scheduled): Phase 2 restricted collections · Phase 5 public face / anonymous read · Phase 4 AI librarian. Write-collections authz already dropped (world-level write is fine). These resume after the robustness bar is met.

Now (robustness / quality agenda — pick up here):

• Observability for usage — without metrics/structured logs/error tracking there's no "usage" signal and no way to know it's robust. Likely prerequisite to everything else.
• Resilience / no SPOF — the standing-deferred shared session store → replicaCount>1 (in-memory sessions + per-pod R2 doc cache / floor cache / R3 link graph are all single-pod today; HA needs them shared or sticky). This is the biggest reliability gap.
• Architecture + code-quality audit — verify hexagonal adherence (ADR 0002), ADR conformance, error-handling/degradation paths, test coverage gaps, the deploy CI papercuts (apps-immutable-check tee-masks exit; PaulHatch feat(scope): bumps PATCH not MINOR).

Starting point under discussion: an architecture + code-quality audit to set the bar, paired with observability so usage is measurable.

Architecture + code-quality audit DONE (2026-06-18)

Full report: /demarkus-library/audit-2026-06-18.md (5 parallel agents: architecture, concurrency, error-handling, security, tests/CI).

Verdict: strong foundation, clear hardening gap. Hexagon clean (core imports zero adapters — verified), go vet/go test -race clean, XSS closed, OAuth/PKCE/ session correct, .mcp.json git-ignored (NOT a committed secret — earlier worry void), coverage high (service 93 / web 90 / markdown 92.6 / broker 87.3%). Work is production-hardening + HA, not rearchitecting.

Robustness backlog (detail in the audit), do in order:

• Tier 1 hardening (small, first): CSRF on POST routes (CRITICAL — none today, only SameSite=Lax) · HTTP server timeouts + ctx into QUIC fetch · body-size limit on writes · post-turnstile ErrUnauthorized→re-login (dead-ends on a 401 today).
• Tier 2 leaks/staleness: inflight lock leak (manager.go:139) · worldMapCache has no TTL/eviction (永 stale) · cache invalidation on write · linkGraph unbounded · broker errors matched by substring (typed status instead) + tests.
• Tier 3 arch (optional): split ReadingService god-interface (port.go) into Reader/Graph/Map/Editor · move listing-vs-doc rule into a core Open dispatch.
• Tier 4 HA (big, when scaling >1 replica): shared session + pending store (Store iface is the seam) · shared linkGraph · cross-pod cache invalidation.
• Quick-win tests: mapWriteError branches · Append tool-error · config fail-fast.

Standing-deferred CI papercuts RESOLVED/MOOT: the apps-immutable-check tee-mask is NOT in this repo (likely the broker repo); the PaulHatch feat(scope): semver bug is already fixed (release.yml:73 anchored regex). Removed from the library's deferred list.

Recommended start: Tier 1 — small, high-value, directly serves "robust enough for real usage." CSRF first.

Tier 1 hardening BUILT (2026-06-18) — branch feature/tier1-hardening

All four Tier-1 items done, gate-green (0 lint, build, all tests + 6 new), NOT merged (Fritz commits). Detail: journal 2026-06-18.

• CSRF (the CRITICAL gap) — Echo v5 CSRF (Sec-Fetch-Site + token fallback), web.CSRFMiddleware, {{ csrf }} renderer injection, tokens in all forms + htmx preview header. Tested.
• HTTP timeouts — ReadHeaderTimeout/IdleTimeout via BeforeServeFunc (both plain + TLS), ContextTimeout 30s handler bound. WriteTimeout left unset (future SSE).
• Body limit — 2 MiB on write POSTs.
• Post-turnstile re-login — presentError ErrUnauthorized clears the session + re-logins (HX-Redirect for htmx); QUIC mode keeps plain 401. Tested.

Deferred follow-up: thread ctx into the QUIC fetch adapter (world.go ignores it) — ContextTimeout bounds the response but won't abort the QUIC goroutine.

Next: Tier 2 (leaks/staleness — inflight lock leak, worldMapCache TTL, cache invalidation on write, linkGraph cap, typed broker errors) then Tier 3 / Tier 4.

Tier 2 hardening BUILT (2026-06-18) — branch feature/tier2-hardening

Leaks/staleness/eviction + the quick-win tests. Gate-green, race-clean, NOT merged. Branched off main (independent of the Tier 1 PR #37; no file overlap).

• inflight lock leak — manager.go dead-grant path now dropLocks (was leaking a mutex per revoked-token session). Test asserts no leak.
• worldMapCache TTL + invalidate — per-world timestamped entries; mirror floorTTL; WorldMapCached rebuilds when stale (was serving the first-ever assembly forever).
• cache invalidation on write — Publish/Append now invalidateTopology (floor + world map) so an edit shows at once, not at TTL; a merge candidate (nothing written) leaves caches intact.
• linkGraph bounded — maxLinkGraphSources 5000 with LRU-by-last-observed eviction (was unbounded for a long-lived pod).
• broker error tests — mapWriteError/mapToolError branch coverage + Append tool-error path (the substring-brittleness the audit flagged). NOTE: the typed-status broker change stays a cross-repo follow-up; library-side this is test coverage, not a behavior change.
• config fail-fast tests — getEnvAsBoolStrict/getEnvAsDuration error paths + NewAppConfig rejects a bad strict switch (was 0% covered).

Remaining: Tier 3 (split ReadingService god-interface; core Open dispatch for listing-vs-doc) — optional; Tier 4 (HA: shared session/pending store, shared linkGraph, cross-pod invalidation) — the big one, when scaling >1 replica.

Tier 1 + Tier 2 MERGED; Tier 3 BUILT (2026-06-18)

• PR #37 MERGED — Tier 1 (CSRF, timeouts, body limit, re-login) + Tier 2 (inflight leak, worldMap TTL, write-invalidation, linkGraph bound, broker/config tests) + the CodeRabbit fixes, all folded into one PR, now on main.
• Tier 3 BUILT — branch feature/tier3-interface-split, gate-green, NOT merged. The two architecture refinements:
  • Split the ReadingService god-interface (port.go) into Reader / GraphService / MapService / Editor, composed as ReadingService. The concrete *service.ReadingService satisfies all four; a narrower consumer (preview-only handler, Phase 4 librarian) can depend on one slice. Pure organization — no behavior change; the web handler still uses the composite.
  • Core Open/OpenCached own the listing-vs-document read dispatch; the rule itself is domain.IsListingPath (single definition). Doc + readPane call Open/OpenCached instead of re-deciding by trailing slash; the margin/edge-source presentation checks route through domain.IsListingPath too, so the rule lives in exactly one place.

Robustness push status: Tiers 1–3 done (T1/T2 merged, T3 in review). Only Tier 4 (HA / replicaCount>1) remains — the big one: shared session + pending store (the Store interface is the seam), shared linkGraph, cross-pod cache invalidation. Do it when actually scaling past one replica; single-replica + Recreate is correct until then.

Deployed — library 0.5.1 LIVE (2026-06-19)

Cluster bumped 0.3.0 → 0.5.1 (deploy repo apps/demarkus-library/applicationset.yaml targetRevision + image.tag, branch bump/demarkus-library-0.5.1, merged). ArgoCD verified: app Synced, revision=0.5.1, operationState phase=Succeeded ("all tasks run"); pod ...c5cfcb554-gxg74 running image :0.5.1, ready. Single replica (Recreate), as intended pre-HA.

Now live: golangci (0.3.1) · login knowledge-graph backdrop (0.4.0) · Tier 1+2 robustness hardening (0.5.0) · Tier 3 interface split + Open dispatch (0.5.1). Reader-only deploy — broker 0.2.0 / worlds 0.17.19 / agent 0.12.43 unchanged.

Robustness push complete + deployed (Tiers 1–3). Tier 4 (HA) still deferred until

1 replica is needed. Next per direction: get usage → observability.

Deployed — library 0.5.2 LIVE (2026-06-19)

Cluster 0.5.1 → 0.5.2 (sign-in page restyled to the reading room's Tufte palette — paper+ink, Charter serif, light-dark() adaptive; knowledge-graph backdrop recolored to faint ink; CSS only). ArgoCD verified: Synced, rev 0.5.2, operationState Succeeded, Healthy; pod ...78c68d5c7f-cl2gn on image :0.5.2, ready. Reader-only (no broker/worlds/agent change), single replica.

Gotcha: after merging the deploy bump, the ApplicationSet git-generator hadn't regenerated the Application yet (live target still 0.5.1) — a hard refresh (kubectl annotate ... argocd.argoproj.io/refresh=hard) made it pick up 0.5.2 immediately instead of waiting for the ~3min appset requeue.

Universe overlay MERGED (2026-06-22) — PR #47

The universe floor now has the on-demand overlay treatment (parity with the §4 graph and §5 world-map overlays). The floor's "view as map" link summons a full-viewport lens (/u?overlay=1 → bare floorSVG fragment, lazy htmx into #universe-overlay); hx-boost="false" on the trigger with a JS-off fallback to inline ?view=map. Landing-only reach; worlds-as-cards stays the default.

Two things rode along in the same PR:

• Scalable floor layout — floorSVG went from single unbounded rows to centered grids (~√N landscape columns) for both systems and portals, so a lone world centers and a large universe wraps + grows downward into the scrollable overlay instead of sliding off one edge.
• Overlay accessibility parity — a shared showOverlay/hideOverlay focus helper across all three overlays (focus into panel on open, restore on close), and role="dialog"/aria-modal/aria-labelledby on each panel (the map overlay gained these; graph + universe aligned). Closed the CodeRabbit nitpicks on PR #47 and the pre-existing gaps it flagged.

Decision recorded: ADR 0006 §6 (also the first written form of the umbrella ADR 0006; §0–§5 indexed there). Web-layer only, reader-only — no broker/worlds/agent/deploy change; not yet deployed (bump the applicationset to roll it).

Open follow-up: satellite density at very large N — each system still draws its orbiting top-docs, so the grid gets busy per-cell at tens of worlds. Cap satellites per world or show them only on zoom-in (same family as graph-pane aggregation for high-degree hubs).

Deployed — library 0.13.0 LIVE (2026-06-23)

Cluster rolled to library 0.13.0 (deploy repo apps/demarkus-library/applicationset.yaml targetRevision + image.tag 0.12.0 → 0.13.0, branch bump/demarkus-library-0.13.0, merged to deploy main). ArgoCD verified: app Synced, revision=0.13.0, operationState phase=Succeeded, Healthy; pod demarkus-library-5c7b7c76f8-m8mc2 Running on image :0.13.0, ready. Single replica (Recreate), pre-HA.

Now live:

• OKF doc-meta grouping (0.13.0, PR #48) — the document margin's metadata is split into an OKF group (type, tags, timestamp — the Open Knowledge Format frontmatter fields) plus a complete metadata block that surfaces every other out-of-band catalog key (importance, etag, content-hash, publisher-custom) instead of dropping all but a curated handful. The OKF type field is newly threaded through domain.Document and both view models (page + paneVM); otherMeta() sorts the catch-all and skips already-surfaced keys. Web/core only, reader-only deploy.
• Universe overlay (0.12.0, PR #47) — already on the cluster (pin was 0.12.0), now documented in the deploy comment trail: the universe floor's on-demand full-viewport "view as map" overlay + scalable centered-grid layout + shared focus/ARIA across the graph/map/universe overlays.

Reader-only roll — broker 0.2.0 / worlds 0.17.19 / agent 0.12.43 unchanged.

Stale-version note: the "⭐ CURRENT STATUS (2026-06-14)" and "Where we are" sections above still cite library 0.5.2 — out of date. The deployed line moved 0.5.2 → 0.6.0 (palette) → 0.7.0 (dock) → 0.8.0 (worlds-only floor + reference map) → 0.9.0 (rich index) → 0.10.x (graph/map overlays) → 0.11.0 (overlay usability) → 0.12.0 (universe overlay) → 0.13.0 (OKF doc-meta). Treat this entry as the current deployed status.

Deployed — floor portal cleanup: library 0.13.2 + agent 0.13.1 LIVE (2026-06-23)

Fixed the universe floor showing phantom/duplicate federated portals (127.0.0.1:6401-6403, localhost, localhost:6309, plus soul.demarkus.io beside soul.demarkus.io:6309 etc., and a second root portal). Root cause: the durable hub graph (root/graph.md, published by the federation agent) had accumulated dev-crawl link targets and addressed hosts in mixed port-less/ported forms; the floor renders every distinct host as a portal.

Shipped across two repos + three releases, all deployed (ArgoCD Synced/Succeeded/ Healthy):

• demarkus-agent 0.13.1 (demarkus fix/fedcrawl-filter-loopback-portals, PR #209) — recordEdges now normalizes each edge target's host via ParseMarkURL (port-stable) and drops loopback/localhost/unspecified targets; discoverServers skips queuing them. Narrow filter by design — private + cluster-internal hosts are KEPT (real federated/cluster worlds use those). CodeRabbit round: hardened isLoopbackHost for unbracketed IPv6+port (::1:6309) + IPv6 test coverage. Cleans graph.md at the source on its next crawl (confirmed: 0 loopback mentions, port variants collapsed after the re-crawl).
• demarkus-library 0.13.1 (fix/floor-filter-loopback-portals) — defense-in- depth portalLabel/isLocalHost in service/hub.go: collapses port-variant portals (hostKey-canonical, default port elided) and drops loopback/unspecified/ link-local/localhost + unmatched cluster-internal portals. Private IPs KEPT (LAN federation). Makes the floor clean regardless of a dirty graph.
• demarkus-library 0.13.2 (fix/floor-match-world-by-name-port) — hostName now joins an authorized world referenced by name+port (mark://root:6309, the port-normalized form of a curated mark://root/... link from world-a/index.md) to the real world, instead of doubling it as its own root portal. This second root was a regression introduced by the agent's port-normalization (before, the port-less mark://root matched the authorized name directly). Render-time join — applies immediately, no re-crawl needed.

Result on the live floor: worlds root + world-a; real external portals soul.demarkus.io, docs.demarkus.io, hub.demarkus.io only. No loopback/local clutter, no port-variant or name+port duplicates.

Cluster now: library 0.13.2 / agent 0.13.1 (broker / worlds / etc. unchanged).

Gotcha (reusable): "world identity" in the hub graph has THREE forms that must all join to one authorized world — the name (root), the dial address (root.root.svc.cluster.local:6309, from mark_worlds), and a name+port (root:6309, from a port-normalized name link). host2name only covers the dial address; authorized[name] only the bare name; the name+port case needs an explicit port-strip-and-recheck. A world that is the hub is never crawled (not seeded), so it appears in graph.md ONLY via inbound name links — making the name+port form its only representation there.

Phase 4 plan recorded — AI librarian NEXT (2026-07-02)

Fritz: work on the AI librarian now, overriding the 06-18 defer direction. Detailed plan written: /plans/phase-4-ai-librarian.md — embedded assistant as a canvas pane (kind a), streamed answer + visible tool-call trace over SSE (ADR 0003's earmarked island), no-JS full-page degrade. Load-bearing D1 supersedes the universe-library sketch: tools are the library's own Reader/Graph/Map ports (transport-symmetric, reader-identity via ctx, read-only — no Editor slice), NOT broker MCP. Loop in service/librarian.go behind a new outbound port.Assistant; adapter = official Go SDK, manual loop, claude-opus-4-8 + adaptive thinking + prompt caching; key via env/Secret, feature dark when unset; caps day one. Settle the 4 open decisions (presentation, model tier, trail-context depth, trace verbosity) with Fritz, then build order starts with the SSE spike.

Amendment (same day): Phase 4 adopts latebit-io/nib (nib/agent + nib/ai/llm) instead of a hand-built loop/port — supersedes the plan's D2/D3; multi-provider (Anthropic + OpenAI-compatible/local) via nib's llmconfig. See the plan's addendum.

Phase 4 — AI librarian: steps 1–3 MERGED (2026-07-03, PR #51 → v0.14.0)

The reading room is now reader+writer+asker. Everything through the UX pass is on main; release v0.14.0 auto-cut. NOT yet deployed.

Shipped: nib-backed librarian (internal/adapter/librarian — per-conversation nib Agent, MaxTurns 15, LRU conversation store, 4 read-only port tools) · port.Librarian + domain events · pane kind a (transcript through the document pipeline; citations are trail-continuing links) · CSRF'd ask form (htmx SSE live exchange via one-shot ask tokens — the question never rides a GET URL; no-JS runs synchronously + PRG) · mark:// scheme allowed through bluemonday (pre-existing bug: in-body cross-world links were losing hrefs) · nav Librarian door (trail-appending) · chat-scroll transcript · attention marker + click engagement (ADR 0003 concessions recorded, incl. the SSE island) · ?slow= soak kept as the transport diagnostic.

Feature-dark by design: the librarian enables only when nib llmconfig resolves a provider (LLM_API_KEY / llm.json). Deploying v0.14.0 is safe (dark without a key) but enabling it in the cluster still needs chart work: an LLM-key Secret + env wiring in the helm chart — NOT done yet.

Next: deploy bump 0.13.2 → 0.14.0 (dark) or chart+Secret first (live) · cluster soak (/a/stream?slow=45 through the GKE ingress) · broker-mode smoke · step 4: trail-context injection (decision open: titles + focused body recommended) · nib packaging fix upstream (tag agent/ai modules, drop the consumer-side replace).

Phase 4 build order COMPLETE — trail context merged (2026-07-03, PR #52 → v0.15.0)

Step 4 on main: each ask carries the reader's view ( — every pane labeled with cached titles, focus marked, focused document's text from the cached render, 8KB rune-safe cap). Injected via nib TransformContext on the model transcript only — History/pane stay clean; cleared at run end. Ask form gains hidden focus (idx = librarian pane for link algebra; focus = whose text rides). CodeRabbit round: document-derived text is defanged against </reader-context> wrapper-breakout (case-insensitive neutralizer). Verified live: "what does this document decide?" answered from the focused ADR, cited as a trail link.

All five build-order steps of the Phase 4 plan are done and merged (v0.13.x spike → v0.14.0 pane+UX → v0.15.0 context). Remaining around the phase, unchanged: deploy decision (dark bump 0.13.2→0.15.0 vs helm chart + LLM-key Secret first, then one roll) · cluster soak (/a/stream?slow=45 through GKE ingress) · broker-mode smoke · nib packaging fix upstream (tag agent/ai modules, drop the consumer-side replace) · independent-pane-scroll design experiment (parked).

Deployed — library 0.15.1 LIVE, PHASE 4 COMPLETE (2026-07-03)

Fritz confirmed in prod: the librarian answers with streaming tokens through the turnstile at library.knowledge.demarkus.io. That one interaction closed the last three checks at once — broker-mode smoke (tool reads with the reader's bearer), GKE-ingress SSE (progressive tokens = no buffering; X-Accel-Buffering honored), and MiniMax-M3 live with the four tools.

Phase 4 — the AI librarian — is COMPLETE and in production. The plan's exit criterion ("returns the doc with its catalog reasoning shown") is met: asks stream a visible tool-call trace and answers cite documents as trail-continuing links, grounded in the reader's own trail context. Built on nib (agent kit + multi-provider llmconfig: fugu in dev, MiniMax-M3 in prod, zero code difference); read-only by design; feature-dark without a key.

Cluster: library 0.15.1, broker 0.2.0, worlds 0.17.19, agent 0.13.1.

Remaining queued (unchanged priorities):

• nib packaging fix upstream (tag agent/ai modules → drop our replace pin)
• observability (the pre-Phase-4 direction: usage signal + robustness)
• Tier 4 HA (now also: conversation store is per-pod)
• independent-pane-scroll design experiment (parked)
• deferred phases: Phase 2 restricted collections · Phase 5 public face

UX round MERGED — v0.16.0 (2026-07-04); cluster still on 0.15.1

PR #55 (feature/library-ux) on main, release v0.16.0 auto-cut, CI green. Reader-side only: hover/settle scroll fix (islands.js afterSettle gating + horizontal-only recenter) · preview cards anchor-positioned (per-pair inline anchor names, position:fixed escapes pane clips, flip fallbacks) · graph arc cap 18/side with "+N more" · floor satellites scale with world count · doc-meta metadata fold · metadata lens (?meta=<pane>, head "meta" affordance) · pane-scroll experiment (DEMARKUS_PANE_SCROLL, default off: fixed-viewport room, panes scroll internally, marginless wide columns) · librarian nib-keystore fallback (DEMARKUS_LLM_KEYSTORE, default true). Detail + gotchas: journal 2026-07-03-ux.

Deploy action queued: bump apps/demarkus-library/applicationset.yaml 0.15.1 → 0.16.0 (safe: experiment flags dark by default; cluster env key preempts the keystore path). Decide separately whether to set DEMARKUS_PANE_SCROLL=true in prod once the experiment verdict lands — if it wins, it's ADR territory (margins/dock/mobile).

ADR 0007 — pane-scroll room is now the DEFAULT (2026-07-04, uncommitted)

Experiment verdict: it wins. Recorded as ADR 0007; built on the working tree (Fritz commits):

• DEMARKUS_PANE_SCROLL default true — the flag is the opt-out back to the page-scroll room, kept until that room is retired.
• Status chip in the pane head (paneVM.HeadStatus, pane-scroll room only): the at-rest trust signal, since the margin badge waits behind the ?meta= lens.
• Tests: config default-on/opt-out; chip presence in the lens test.

Open per ADR consequences: mobile (46ch columns pan horizontally — neither room designed for phones) · retire page-scroll margin CSS with the flag · Firefox card-clip degrade until anchor positioning ships · sidenotes hidden in-room (live in reader overlay).

Deployed — library 0.17.0 LIVE, pane-scroll room default in prod (2026-07-04)

Cluster rolled 0.15.1 → 0.17.0 (deploy branch bump/demarkus-library-0.17.0, merged; appset regenerated without the hard-refresh nudge this time). ArgoCD Synced/Healthy/Succeeded; pod on image :0.17.0, ready; librarian enabled (MiniMax-M3 — Secret env key resolves first, nib-keystore fallback inert); /health 200. Reader-only roll (broker 0.2.0 / worlds / agent unchanged), single replica pre-HA.

Now live: the whole UX round (v0.16.0 — hover/settle fix, anchor-positioned preview cards, graph/floor density caps, metadata fold + ?meta= lens) and ADR 0007's pane-scroll room as the default (v0.17.0 — fixed-viewport canvas, per-pane scrolling, wide marginless columns, head status chip; DEMARKUS_PANE_SCROLL=false is the opt-out).

Remaining human check: browser walk of the room at library.knowledge.demarkus.io (authed) — chip, meta lens, hover cards through the turnstile.

Observability DEFERRED again (2026-07-04)

Recommended as next (blind on usage/cost; librarian spends tokens); Fritz: defer. Design sketch agreed for when it's picked up — prom client library only, no cluster infra: /metrics on a separate internal port (not in Service/ingress; port-forward to read; GKE managed collection or Prometheus scrapes it later via pod annotation, zero code change). Metric set: HTTP by route-class/status + latency (Echo middleware) · world reads per transport (WorldGateway decorator) · cache hit/miss (nil-safe metrics iface in service) · librarian asks/outcomes/duration/turns (the money surface). Rejected: expvar (conversion cost later); Grafana in-cluster (not wanted).