soul.demarkus.io:6309/demarkus-library/roadmap.md/v25 draft reader meta

demarkus-library — Roadmap & Resume

Pick-up-where-we-left-off doc. Authoritative plan is /plans/universe-library.md; the Reading Room redesign lives in /plans/reading-room.md (invariants locked in ADR 0005). Build model: Opus 4.8 default, Fable 5 for load-bearing design.

Where we are (2026-06-13 — Reading Room fully LIVE in prod, R3 deployed)

main HEAD e8a136b (library). The whole Reading Room is deployed and working in prod: R1 (Tufte pane) → R2 (trail engine) → the floor (pane zero) → R3 (previews, backlinks, graph pane), both transports (decision 16). Cluster versions: library 0.1.7, broker 0.1.36, worlds server 0.17.18 — ArgoCD Synced/Healthy (R3 deployed via PR #17 → v0.1.7, rev 0.1.7 confirmed on the running pod). Next: floor enrichment / Phase 2. Detail in journal 2026-06-13 + journal 2026-06-12.

  • Phase 0 foundation spike · Phase 1a reading room (stacks, card catalog, editions, link rewriting, htmx fragment-vs-page).
  • Phase 1b — library card: DONE. Broker confidential web clients (demarkus 117e2a6) · oauth client (PKCE S256, Basic auth §2.3.1 escaping) · server-side sessions + single-flight refresh · ctx threading on ports · broker MCP WorldGateway with per-reader pooled sessions (1 request per read; fixed the live 429s) · login turnstile · composition-root swap DEMARKUS_TRANSPORT=quic|broker · native TLS for local e2e (env at ~/.config/demarkus-library/dev.env, client id library-web-dev — dedicated dev broker client; prod is library-web). E2e verified live.
  • Multi-world floor + federation (fc4277c): (world, path) addressing, /w/<world>/... routes, federated composite gateway — KS names via broker, host[:port] via direct QUIC (DEMARKUS_FEDERATION default true). Read token never leaves home host (test-pinned). Cross-world mark:// links clickable.
  • Deploy + CI/release — DONE (PR #10, v0.1.0): goreleaser-pro, distroless Dockerfile, helm chart (replicaCount 1 + Recreate — in-memory sessions), hardened workflows. Module paths on latebit-io. Releases auto-cut on merge to main (PaulHatch per-module semver). Deploy = ArgoCD watching latebit-io/demarkus-knowledge-system-deploy; image+chart pins in apps/<app>/applicationset.yaml, world list + webClients in deployment.yaml.
  • Rendering — DONE (PRs #11/#12): chroma classes-mode · GFM alerts · emoji · mermaid + KaTeX islands (lazy, degrade to source).
  • R1 — Tufte pane: MERGED (PR #13), LIVE. 60ch column + 250px margin · pinned margin trust signals (status badge: metadata status: axis → frontmatter fallback → draft; tags; provenance; parsed-frontmatter properties; mark:// + raw-source escape) · sidenote goldmark extension (footnote→margin) · lookup-backed tag pages · /w/:world/raw/* · search box removed · authoring contract published (mark://soul.demarkus.io/.well-known/library/authoring.md).
  • R2 — trail engine: MERGED (PR #14), LIVE (plans §"R2 build decisions"):
    • Trail URL /t/<pane>/~/<pane>?focus=i; ~ = separator; pure codec + click algebra (truncate/append, dedup→focus, spine→focus). Cap 10, raw slashes. Pane kinds: <world>/d/<path> (doc; trailing slash/empty = listing), <world>/tags/<tag>, bare u = floor.
    • Rendered-document cache (decision 9): port.DocumentCache + LRU. Focused-live: every click costs exactly 1 world read.
    • Canvas: spines / body-only parent / focused Tufte pane; tombstone spine for unreadable unfocused panes; trailizeLinks rewrites /w/ hrefs to post-click trail URLs + active-link breadcrumb. Spec published (.well-known/library/trails.md).
    • Codec note: paneAddrFromParts(world,kind,value) (trail.go) is the single source of truth for pane semantics; parsePaneChunk (/t/) and paneAddrFromRoute (/w/) both call it (PR #16). Share semantics, not syntax — don't re-duplicate.
  • The floor — universe view (pane zero): MERGED (PR #15), LIVE (plans §"Universe-view research" + 2 addenda):
    • demarkus prereqs: mark_worlds broker tool (#182) lists authorized worlds; LOOKUP match-all * (b8aedb7) returns the whole catalog in importance order. World must run server ≥ 0.17.18 or it tombstones.
    • WorldGateway.Worlds(): broker parses mark_worlds; QUIC returns the home world; federated prefers Names→Hosts.
    • service.Floor/FloorCached: worlds × Lookup("*"); failed world → tombstone (dimmed, never dropped); ErrUnauthorized → re-login.
    • Web: pane kind u; / → /t/u; SSR SVG floorSVG (worlds as circles r∝satellite count, top-10 docs on orbit r∝importance, stroke=status, every node a plain ). ADR 0003's canvas island stays UNSPENT.
    • PR #16 fix: parser rejected the world-root listing chunk (<world>/d/); fixed + consolidated the two parsers.
  • R3 — previews + backlinks + graph pane: MERGED (PR #17), v0.1.7, LIVE (plans §"R3 build decisions"; verified live in QUIC):
    • Edge source = render-time observed-links map (the load-bearing decision). rewriteLinks already resolves every link to (world,path); capturing edges is free and library-internal → transport-symmetric (no broker graph store, so decision 16 holds). Rejected: broker store (broker-only), library-side crawl (over-built). Cold state fills as the room is read; durable hub-sourced topology (addendum 2) is the upgrade layered in front later — the service edge methods are the seam.
    • Architecture: domain.Ref + domain.Neighborhood; edge store is an internal service.linkGraph (fwd+reverse maps, mirrors floorCache — no new port/adapter/wiring). Inbound port gains RecordLinks/Backlinks/Neighborhood. rewriteLinks returns (html, []Ref), rewriteHref returns (route, Ref, isDoc); web resolves+records, core owns the store.
    • Hover previews: previewize wraps in-app doc anchors with a preview-host/preview-card popover + htmx (mouseenter delay:300ms once → /w/:world/preview/* fragment). Pure htmx+CSS, no JS. One component for body links AND backlinks. Snippet = first paragraph, from ReadCached (budget-safe). mouseenter overrides click so hx-boost nav still fires.
    • Backlinks "referenced by" block in shared doc-meta (trail URLs on the canvas, /w/ permalinks single-doc). Empty ⇒ absent.
    • Graph pane: kind <world>/g/<path> via shared paneAddrFromParts; route /w/:world/g/*; SSR SVG graph.go (center + backlinks left / outbound right, each node continues the trail). Canvas island still unspent. Margin "graph" affordance opens it.
    • docs/trail-format.md + in-universe trails.md (v3) updated with the g kind (decision 14).
    • Repo pre-commit.sh added (from demarkus, single-module): fmt/vet → golangci-lint (pinned to CI v2.9.0) → build → test. Run before wrapping a phase/task.
    • Broker-mode R3 smoke still pending (interactive OAuth); parity is architectural (edge source is library-internal, no broker-only path).

How to resume

cd ~/latebit/demarkus-library && ./pre-commit.sh   # fmt/vet/lint/build/test
go build -o /tmp/dl ./cmd/demarkus-library
# quic mode (no auth): open /t/u (floor), a doc, hover a link, click "graph".
PORT=8080 DEMARKUS_INSECURE=true DEMARKUS_HOST=soul.demarkus.io \
  DEMARKUS_AUTH=<token from .mcp.json> /tmp/dl
# broker mode: source ~/.config/demarkus-library/dev.env && /tmp/dl
#   OAuth on broker.knowledge.demarkus.io. Port held? lsof -ti :8443.
# cluster (GKE): kubectl ctx gke_knowledge-49722_...; creds expire mid-session
#   → ! gcloud auth login. ArgoCD ns argocd; check operationState, not just
#   sync.status (stale on a retrying op).

File map (hexagonal — ADR 0002)

  • internal/core/{domain,port,service} — entities (Document, Floor, WorldInfo, Ref, Neighborhood), ports; ReadingService = Read/Browse/History/Search/Tag/Raw + *Cached + Floor/FloorCached + RecordLinks/Backlinks/Neighborhood; WorldGateway adds Worlds(); DocumentCache; service/floor.go (catalog parse), service/graph.go (linkGraph observed-links store)
  • internal/adapter/inbound/web — Echo routes (/t/* canvas incl. u/g, /w/:world/... incl. tags/raw/g/preview), trail.go (codec + click algebra + paneAddrFromParts, incl. g), trail_links.go, links.go (rewriteLinks → edges), trail_handlers.go (+ floorPaneView + graphPaneView), floor.go + graph.go (SSR SVG), preview.go (cards + previewize + backlinkLinks), templates (+ preview/preview-link), static/
  • internal/adapter/outbound/{world,broker,federated,oauth,markdown,cache} — direct QUIC, MCP pool (Worlds parses mark_worlds), shape-router, broker OAuth, goldmark, LRU cache
  • cmd/demarkus-library — composition root, AppConfig, brokerAuth shim
  • docs/ — authoring-contract.md + trail-format.md (sources of the in-universe /.well-known/library/ docs; keep both in sync)
  • pre-commit.sh — the phase/task gate (golangci-lint pinned to CI v2.9.0)

Next sessions — queued work

1. Floor enrichment (pairs with R3's edge-source seam)

Hub-sourced topology (plans addendum 2 — hubs are the primary map: durable, versioned, transport-symmetric) layered in front of the observed-links map; portal nodes for externally-linked hosts in quic mode; world-view zoom level. Hub discovery convention still open (manifest self-declaration vs DEMARKUS_HUB env — lean manifest + override). Graph-pane aggregation/cap for high-degree hubs (a doc with ~40 outbound links crowds the right arc).

2. Phase 2 — restricted collections (per /plans/universe-library.md)

Per-world/per-path authz surfaced in the reading room. Depends on broker-side authz semantics — re-read the plan + broker authz.go first.

Maybe / observed

  • Broker-mode R3 smoke (dev.env): hover preview, backlinks, graph pane behind the turnstile — last unrun acceptance check.
  • Deploy CI papercut: apps-immutable-check runs the guard via ... | tee, so the pipe masks the script's exit — the check can NEVER fail the build, only posts a PR comment. Add set -o pipefail / drop the tee.
  • Semver quirk: PaulHatch minor_pattern: "feat:" is substring — feat(scope): bumps PATCH not minor. Fix via regex /^feat(\(.+\))?:/.
  • Broker rate-limit drop (60/30 → 30/15) on fix/broker-rate-limit-down — confirm merged/synced.
  • Visual polish by eyeball (pane widths/gaps, spine legibility, dark mode, canvas scroll; floor at many-worlds scale + label crowding; preview-card placement near pane edges; graph pane at high node counts).

Standing deferred

  • Find + fix publishers that wrote metadata fences into bodies (R1's friendly frontmatter reduces harm; source fix still open).
  • Shared session store (or sticky sessions) → replicaCount > 1 + RollingUpdate
    • PDB — the R2 doc cache, single-slot floor cache, and R3 observed-links graph are all per-pod (fine at replicaCount 1; revisit together — per-pod graph means backlinks differ across replicas).
  • Adopt demarkus's strict .golangci.yml (18 issues to clean first).
  • Phase 3 cataloging desk (writes, CodeMirror island) · Phase 4 AI librarian (SSE; joins the canvas as a pane, trail = its context) · Phase 5 public face.

History (condensed — details in journals + ADRs)

  • 2026-06-10: Phases 0 + 1a built + merged (PRs #1–2).
  • 2026-06-11: Phase 1b planned + shipped (PRs #3–#9); floor + federation; frontmatter strip. Gotchas in debugging.md.
  • 2026-06-12: v0.1.0 deploy/CI (#10) · latebit-io migration · v0.1.1/v0.1.2 rendering (#11/#12) · ADR 0005 · R1 (#13) · R2 (#14) (journal).
  • 2026-06-13: universe-view research · demarkus mark_worlds (#182) + LOOKUP match-all (b8aedb7) · floor merged (#15) · cluster rollout of R1+R2+floor (server 0.17.18 / broker 0.1.36 / library → 0.1.6) · trail root-listing fix + parser consolidation (#16) · R3 merged + deployed (#17, v0.1.7) — observed-links edge source, hover previews, backlinks margin, graph pane; repo pre-commit.sh added (golangci-lint pinned to CI v2.9.0). Reading Room fully live in prod (library 0.1.7) (journal).

Git / process notes

  • Work on feature/* / fix/* branches. Never auto-commit — Fritz commits and opens PRs; CodeRabbit reviews via .coderabbit.yaml. Releases auto-cut on merge.
  • Run ./pre-commit.sh before wrapping a phase/task (fmt/vet/lint/build/ test; golangci-lint pinned to CI's v2.9.0).
  • Deploy = ArgoCD on latebit-io/demarkus-knowledge-system-deploy. Bump chart targetRevision + image.tag together in apps/<app>/applicationset.yaml. Library/broker are Deployments (clean roll); worlds are StatefulSets.
  • PR scopes: domain, port, service, web, world, markdown, cmd, view, deps, ci, broker, auth, session, oauth, federated, deploy, docker, helm.

⭐ CURRENT STATUS (2026-06-14) — supersedes "Where we are" above

The Reading Room is feature-complete and fully deployed. Cluster (all ArgoCD Synced/Healthy): library 0.1.10, broker 0.1.37, worlds server 0.17.19, agent 0.12.43.

Shipped + verified since the stale section above:

  • R3 (PR #17, library 0.1.7): hover previews, backlinks margin, graph pane.
  • Floor enrichment: hub-graph topology → cross-world edges + portal nodes.
    • library hub-topology reader (0.1.8) reading <hub>/graph.md; DEMARKUS_HUB chart wiring (0.1.9); host↔name join via broker mark_worlds address column (broker 0.1.37 + library 0.1.10).
    • agent publishes /graph.md (--publish-graph, client 0.12.42/43).
    • server token-ownership fix (0.17.19): token Secrets created by a pre-install bootstrap Job, not chart-templated (ArgoCD was pruning/ corrupting the runtime-mutated tokens.toml — see debugging.md).
  • External federation verified: floor portals to public worlds reachable from the cluster (docs.demarkus.io cloud + soul.demarkus.io once its residential ingress was opened). docs in debugging.md.

Full detail: journal 2026-06-13 (latest entries run into 2026-06-14) + debugging.md.

Next phases (queued)

  1. World-view zoom — deferred floor-enrichment piece: one world's catalog as dir clusters + aggregate bubbles → listing panes (plan §Universe-view research, zoom level 2).
  2. Phase 2 — restricted collections — per-world/path authz in the room (re-read /plans/universe-library.md + broker authz.go).
  3. Standing deferred — shared session store → replicaCount>1 (R2 cache, floor cache, R3 link graph all per-pod); strict .golangci.yml; deploy CI papercuts (apps-immutable-check tee-masks exit; PaulHatch feat(scope): bumps patch not minor).

Loose end

Demo links in world-a/index.md (mark://root/index.md + mark:// soul.demarkus.io/index.md) added to exercise floor edges/portals — keep as a live example or strip (then restart the agent to re-publish graph.md).

Process

feature/* branches; Fritz commits, never the agent; ./pre-commit.sh gate (golangci-lint pinned CI v2.9.0); CodeRabbit per PR; releases auto-cut on merge; deploy = bump apps/<app>/applicationset.yaml targetRevision+image.tag (ArgoCD). knowledge MCP joined for root/world-a writes.

Update (2026-06-14, later) — world-view zoom SHIPPED

Cluster now library 0.1.11 (others unchanged). Next-phases item 1 (world-view zoom) is DONE + deployed: PR #21, reader-only, floor world node zooms to <world>/u/ (catalog as dir clusters + "+N more" → listing panes) + doc-margin "map" affordance. In-universe trails.md → v4 with the <world>/u/ chunk. Loose end resolved: demo links in world-a/index.md KEPT (legit related-links footer; floor edges stay live). Remaining queued phases: Phase 2 (restricted collections) and standing deferred. Only open item on this phase: human browser smoke of the world map at library.knowledge.demarkus.io.

Phase 2 plan recorded (2026-06-14)

Detailed Phase 2 plan written: /plans/phase-2-restricted-collections.md — sub-world read-authz tier (collections), broker-led, library projects. Settles D1–D4 (rules in broker WorldConfig; thread read identity; per-collection locked-vs-hidden; path globs) with recommendations; broker/library work split, acceptance, build order. Important, NOT urgent — scheduled later. Authz today is world-level only (reads tokenless/open within an allowed world), so this is a real broker security change, not just a library projection.

Phase 3 plan recorded (2026-06-14) — flagged important

Detailed Phase 3 plan written: /plans/phase-3-cataloging-desk.md — the WRITE side: create/edit docs in markdown source, a projection of the broker's mark_publish/mark_append protocol. Settles D1–D5 (htmx textarea + live server preview first, CodeMirror island only on escalation; editing is a focused-pane mode NOT a trail chunk; metadata as separate form fields — no --- fence; expected_version + merge flow; write authz coupled with Phase 2). First write path through the library (ends the read-only invariant). Library is the bulk; broker work only if write-collections (D5) are in scope. Fritz: Phase 3 is important — likely ahead of Phase 2 in priority.

Phase 3 STARTED — edit slice live (2026-06-14)

Cluster now library 0.1.12. Phase 3 (cataloging desk) is underway: the first write slice — edit an existing document — is merged (PR #22) + deployed (ArgoCD Synced/Healthy). The room is now reader+writer. Margin "edit" → htmx textarea + live server preview → publish via mark_publish (metadata out of band, expected_version conflict guard) → 303 back to the doc. Auth-gated; no broker/worlds/agent change. Plan: /plans/phase-3-cataloging-desk.md. Remaining Phase 3 slices: create-new document, mark_append affordance, conflict merge-candidate flow, write-collections authz (couples with Phase 2). Open: human browser write-smoke at library.knowledge.demarkus.io.

trail
  1. soul.demarkus.io:6309 v25