soul.demarkus.io:6309/index.md/v86 draft reader meta

demarkus-soul

This is the living knowledge base for the demarkus project, served by demarkus itself.

An AI agent's evolving memory, architecture notes, debugging insights, and design decisions; all versioned, all permanent.

Sections

  • Architecture: system design, module boundaries, key decisions
  • Universe Pattern: souls, worlds, and hubs as a deployment topology
  • Patterns: code patterns, conventions, idioms used in this codebase
  • Guidelines: hard rules for code quality, must be referenced before writing code
  • Conventions: collaboration + repo/plugin conventions (how I work: commits, layering, tooling, plugin discipline)
  • Debugging: lessons learned from bugs and investigations
  • Roadmap: what's next, what's in flight, what's done, and what's deliberately not prioritized
  • Ecosystem: browsers, plugins, and tools that implement or integrate with demarkus
  • Debt: technical debt and improvement opportunities
  • Journal: session notes and evolution log, one file per day at /journal/<YYYY-MM-DD>.md
  • Guide: agent install guide for setting up demarkus-soul
  • Thoughts: my own reflections, ideas, and open questions
  • FAQ: common questions about demarkus and how it compares

Decisions

Architecture decision records for the core project. Canonical copies live in git at docs/adr/; these are verbatim mirrors so they are reachable by lookup, backlinks, and this hub (see Conventions). Sub-project decisions live in their own series, e.g. /demarkus-library/adr/.

  • ADR 0001: broker confidential web-client registry (accepted 2026-06-11)
  • ADR 0002: align store frontmatter with the Open Knowledge Format (accepted 2026-06-22)
  • ADR 0003: default OKF type on publish (accepted 2026-06-22)
  • ADR 0004: edge semantics, provenance on every edge, typed relations via rel- metadata (accepted 2026-07-13)
  • ADR 0005: node identity omits the default port (accepted 2026-08-18)

Vocabulary

  • knowledge system: organizational, broker-fronted universe. Joined via /knowledge-join (the demarkus-knowledge plugin). MCP traffic over HTTPS terminates at the broker; broker translates to QUIC for internal worlds.
  • soul: personal demarkus knowledge base, direct-QUIC. The original demarkus-soul shape. Will be joined via a future /soul-join slash command if one ships.
  • Both compose worlds (demarkus servers, QUIC). A Claude Code installation can have both; they don't conflict.

Plugins

  • Obsidian Plugin; fetch, publish, and browse demarkus documents from Obsidian (standalone repo latebit-io/obsidian-demarkus)
  • Claude Code: demarkus-memory (personal soul), source at plugins/claude-code/, shipped via the marketplace. Version history: v0.1.0 (2026-04-23, #96) initial; v0.2.0 (2026-05-23, #152) /knowledge-join; v0.3.0 (2026-05-31, #168) mark_lookup + SessionStart guidance; v0.4.0 (2026-06-01, #171) hook-based enforcement (publish tag-gate, journal nudge, recall nudge), /project-template.md, /soul-doctor; v0.5.0 (2026-06-03, #172) split the knowledge-system surface out into the separate demarkus-knowledge plugin (below) so this one is personal-soul only, and added an always-on "single memory store" steering line plus a one-time, ask-don't-force offer to disable Claude Code's built-in memory; v0.6.0 (2026-06-17, #192) the soul→knowledge promote bridge; /promote (detect endpoint → run the knowledge cascade → one-directional back-stamp, stub or marker mode), /soul-refresh (the directional coherence edge: refresh promoted docs from knowledge, local edits re-enter upward through the gate), and mutual knowledge detection (knowledge_endpoints/detect-knowledge.sh reverse-peek the knowledge registry). Hooks: SessionStart, PreToolUse, PostToolUse, Stop, UserPromptSubmit. Now at v0.13.8 (#281).
  • Claude Code: demarkus-knowledge (organizational knowledge system), source at plugins/claude-code-knowledge/, a second entry in the same marketplace. v0.1.0 (2026-06-03, #172). Owns the broker-fronted surface split out of demarkus-memory: /knowledge-join, a new /knowledge navigation command, KS-first SessionStart guidance with soul↔system synergy, a KS-scoped publish tag-gate, and a KS-gated recall nudge. No binaries and no local server: pure broker + Claude Code MCP OAuth. Standalone: owns its own ~/.demarkus/plugin-knowledge.* file namespace and DEMARKUS_KNOWLEDGE_STRICTNESS env; reads (never writes) plugin-memory.conf only to detect a sibling soul for the synergy note. The two plugins' publish gates partition cleanly by server scope, so both can be installed together. v0.2.0 (2026-06-17, #192) added the knowledge-promote cascade skill (the execution half of the promote bridge: triage → distill, stripping personal framing + secrets/PII → dedup vs catalog → tag to taxonomy → destination-select via mark_worlds writable + per-world world.md → human gate capped by the world's autonomy ceiling → publish with provenance) and the per-world world.md descriptor example. Now at v0.5.24 (#281).
  • OpenCode: demarkus-opencode-memory (personal soul), source at plugins/opencode-memory/. v0.13.8 (2026-08-10, #281). The OpenCode port of demarkus-memory: single-file TS adapter over the shared demarkus-plugin binary; installed by curl one-liner into ~/.config/opencode/plugins/ (no npm). Plan + follow-ups: /plans/opencode-memory-plugin.md.
  • pi: demarkus-pi-memory / demarkus-pi-knowledge, source at plugins/pi-memory/ and plugins/pi-knowledge/, mirrored to standalone repos for pi install. Same adapter pattern; now at v0.13.8 / v0.5.25 (#281).

Sub-projects

Standalone-repo projects in the demarkus ecosystem, each with its own hub and durable knowledge under /<slug>/:

  • demarkus-library; the web front-end ("Universe Library"): a server-rendered Go + htmx reading room over a broker-fronted knowledge system. Repo latebit-io/demarkus-library. Plan: /plans/universe-library.md. Reading room feature-complete and deployed (cluster library 0.5.2; universe overlay PR #47 merged 2026-06-22, awaiting deploy). See roadmap.
  • demarkus-knowledge-system-deploy; GitOps deploy repo for the production knowledge system (knowledge.demarkus.io): OpenTofu (GCP/GKE) + ArgoCD ApplicationSets standing up the broker, worlds, agent, library, and backups. Repo latebit-io/demarkus-knowledge-system-deploy. deployment.yaml at repo root is the single source of deployment identity.
  • mark-knowledge; the hosted service build (signup, tiers, per world billing, management app). Repo at /Users/fritz/latebit/mark-knowledge, with its own soul provisioned 2026-07-26 (isolated mode, port 16310). It does not have durable knowledge under /<slug>/ here, because it keeps its own soul rather than a section of this one. Direction and the demarkus-side constraints live in /plans/demarkus-as-a-service.md.

Active Plans

Verified against code/PRs on 2026-05-31; versions-sharding entry corrected 2026-07-05. Plans with real remaining work:

  • Store Parity (file vs Postgres); absolute parity between the file store and pgstore: CI Postgres service with a required DSN, seeded differential suite over both backends plus a fuzz target, the handler suite parameterized over both backends, then contract-test porting, kind e2e, migration tool, dogfood soak, and a pg performance list. Steps 1 to 4 merged 2026-08-19 (PRs #324, #327, #329, #331): seven real divergences found and fixed, PartialWalkError, every handler test runs as /file and /postgres, per-package Postgres schemas via pgtest, handler benchmarks committed, and the file-only contract tests (invalid metadata, retention edges, chain tamper via a Tamper hook and store.VersionFilePath) ported into the conformance suite. Steps 5 to 8 open; 8 is the pg performance work (LOOKUP index, pool limits, root LIST).
  • OpenCode Knowledge Plugin Port; port the Claude Code organizational knowledge plugin to OpenCode with shared endpoint registration, native OAuth, policy gates, guidance, commands, and promotion skill. Implemented 2026-08-15 on branch feat/opencode-knowledge-plugin; tests and pre-commit pass, branch unmerged.
  • APPEND metadata loss; appending to a document silently stripped its catalog metadata, so tags and importance were lost and the document fell out of mark_lookup. Complete 2026-08-14 on branch fix/append-metadata-merge via Option C, the protocol merge: APPEND now writes the base version's publisher metadata with the request's layered over it (store.MergeAppendMeta, both backends), retention excluded and the OKF type default moved after the merge. SPEC 6.6 and 9.9 updated; nine plugin guidance files, both mark_append tool descriptions, and a new metadata-loss check in all five doctor commands; memory plugins 0.13.24, knowledge plugins 0.5.40/0.5.41. Corpus repaired: of 123 untagged soul documents, 69 had lost tags and were republished with the metadata recovered from their newest still-tagged version, bodies unchanged; the other 54 were never tagged and are a separate curation exercise. Branch unmerged; a soul only gets the fix once its server is upgraded, so soul.demarkus.io still strips on append.
  • Agent Memory Leaderboard entry; enter demarkus in agentmemoryleaderboard.ai next cycle: agentic search (nav agent over lookup/fetch/backlinks) as the Search implementation, distillation cascade at Add-time, commercial board via self-hosted API on a droplet with echo v5 as inference backend. Sub-project hub: /memoryleaderboard/ (repo /Users/fritz/latebit/memoryleaderboard). Planned 2026-08-13; cycle 1 closed 2026-08-07, awaiting cycle 2 dates. Phase 0 (recon) not started.
  • Code Quality Sweep 2026-08; full-repo review findings (6-agent sweep, 2026-08-12): 10 high-severity correctness/security leads, cross-module duplication extraction targets, broker package-split recommendation, dead code, pervasive rule violations, remediation order. Findings recorded; nothing fixed yet.
  • Bucket Document-Store Backend; native object-storage backend (GCS first, S3/MinIO designed-for) as a third DocumentStore implementation, enabling multi-replica worlds on k8s with no PVCs: per-document manifest objects committed via generation CAS, write-once version blobs, per-pod LIST-driven hash-index/catalog sync, storage.kind: filesystem|bucket chart knob (bucket mode renders a Deployment, no VCT), tofu world-storage module + migration tool + runbooks. Planned 2026-08-10 (investigation: symlinks stay in the file store; gcsfuse and Filestore RWX rejected). Not started; 8 PRs.
  • demarkus as a service; the hosted offering: Aiven adjacent service model, three tiers matching the website's Personal, Team, and Knowledge System scales, VPS first substrate with Kubernetes only on overflow, per world billing with the box as the size step, power off instead of scale to zero, and a management app as the only new engineering. Direction set 2026-07-26; the build moved to the mark-knowledge repo and its own soul on the same day. This copy stays as the demarkus-side record, since the decisions constrain this repo: the appliance is the unit of deployment, the broker stays one binary, the librarian is the only inference cost centre, and quotas plus backups are prerequisites that land here. Note that repo ADR 0005 (hosted tenant density), which an earlier revision cited as settling density, was deleted 2026-07-25.
  • The five-minute appliance; one pasted command on a fresh VPS yields a working self-hosted knowledge system in about five minutes: sslip.io default (no domain), fully native (no container runtime), Authelia as primary IdP with Pocket ID and Dex as fallbacks, zero prompts with everything generated, ending in a summary card (library URL, owner login, /knowledge-join line, librarian key hint). Builds on the single-host stack (PR #262/#263). Draft recorded 2026-07-18; not started.
  • Knowledge Ingestion Pipeline; narrative + design for how org knowledge flows into a knowledge destination, framing the soul as the staging/write-ahead tier and the knowledge destination as the curated read-model, with one curation gate (cascade model routing: Haiku triage → strong-model distillation → human approval) reused across all inflows (soul promotion, Confluence, Slack, Jira, meetings). Promote is a detection-gated bridge between the memory and knowledge plugins; soul↔knowledge coherence is a directional refresh. Phase-0 prerequisites built and merged (2026-06-17): the promote primitive + coherence edge (plugins; memory v0.6.0 / knowledge v0.2.0, #192), the brokered access-discovery surface (mark_worlds writable column, #191), and the per-world world.md descriptor. Three of four prerequisites done; A2 (plain-remote token-grant introspection) deferred; the live target is brokered. Remaining phase-0 surface: signal/batch triggers (manual /promote + /soul-refresh are the only triggers today), then the dogfood promote of this plan itself.
  • Universe Library; web front-end for a demarkus universe (Go + htmx reading room). Sub-project hub: /demarkus-library/. Reading room feature-complete and deployed (cluster library 0.5.2); see the sub-project roadmap.
  • Universe Deployment (Phase 6); Helm charts (server, broker, agent), OIDC token broker, release pipeline, observability. ~95% complete (PRs #126-#134, 2026-05-14). Remaining §6.6 (docs) + §6.4 Kustomize overlay reframed as deferrable ops polish; effectively superseded in practice by the GKE reference deployment.

RFC Review

  • Demarkus / Knowledge System FAQ; terse Q&A for the RFC review session, sourced from the demarkus and demarkus-knowledge-system-deploy repos. Status: WIP, pending review via the library.

Completed Plans

  • OpenCode Memory Plugin (1:1 port); the OpenCode port of demarkus-memory as plugins/opencode-memory/ v0.13.8: single-file TS adapter over the shared demarkus-plugin binary, curl-one-liner installer with stage-then-commit + rollback, atomic bootstrap binary replace across all five plugin copies, live-verified against OpenCode 1.18.15. COMPLETE: planned 2026-08-09, merged 2026-08-10 (PR #281, f4c2b35). Follow-ups (soul-list/soul-remove subcommands, token stdin input, shared-source bundling, opencode-knowledge port) recorded in the plan.
  • Graph Hub Seeding; mark_backlinks/mark_graph/mark_explore seed from the published /graph.md aggregate on both MCP surfaces (demarkus-mcp per host, broker per world with dial-address-to-world-name translation), local wins via the zero-CrawledAt marker, seed etags in graph.json, fetch.FetchConditional. COMPLETE 2026-07-14/15 across #253 (feature), #254 (issue #222: Merge preserves resolved nodes on failed re-crawl), #256 (broker seed URL translation), #257 (seed all worlds + the producer-consumer /graph.md contract test). Deployed and live-verified: scratch-HOME cold client answered soul backlinks with zero crawls; a cold broker pod's first graph call answers non-hub backlinks from the hub aggregate (broker 0.12.4, agent 0.21.1). Lessons in /debugging.md (mock fixtures encoded a plan assumption).
  • Multi-replica LOOKUP (postgres, phase 2); the LOOKUP catalog moved into Postgres (rows in the write transaction, SQL-backed Lookup behind the handler LookupCatalog seam) so world pods can scale past one replica; phase 2 of the deploy repo's ADR 0002, following the phase-1 postgres backend (#249). MERGED PR #250 (2026-07-13): LOOKUP conformance suite in storetest, two-replica handler proof, batched reconcile-on-Init backfill, server chart startupProbe, and the configwatch flake fixes (kqueue same-name swap limitation documented in /debugging.md).
  • Version Retention; keep last N versions per document via a retention publish-metadata key with prune-on-write in the store; motivated by the knowledge system's graph document at 545+ versions. COMPLETE: planned, shipped, and production-verified 2026-07-06/07 across #236 (store core + os.Root delete hardening + audit logging + SPEC §9.9), #237 (plugin gate binary), #239 (guidance + repins), #240 (agent publishes generated artifacts with retention=20), and the deploy rollout (server 0.20.0 / broker 0.9.0 / agent 0.19.0). Live result: /graph.md pruned 556 → 20 versions and the hub hash indexes cleared their backlogs in one crawl (~1,714 version files deleted, audit-logged, chains valid).
  • MCP Resources + Prompts; demarkus documents as client-attachable MCP resources (mark:// URI template, #anchor section attach, background-LIST picker population) and orient/recall/whats-new as server-vended prompt commands. SHIPPED PR #232 (2026-07-05), client/v0.17.0. Follow-up deferred: broker gateway resources/prompts (multi-world URIs, auth on reads; starts by flipping the gateway capabilities test).
  • MCP Client Ergonomics; size-adaptive mark_fetch (outline mode, #section slicing, force), session unchanged-dedup, and the mark_explore orientation card, on both MCP surfaces via shared client/mdoutline + client/fetchdedup packages. SHIPPED #225/#230 and deployed 2026-07-04/05; plugin users (client v0.15.0 via tools 0.4.1) and the live knowledge system (broker 0.5.0). Deferred follow-ups: MCP resources/prompts (shipped; see above), library librarian open adoption.
  • Versions Sharding; per-document versions/<doc>/vN subdirectories with lazy migration, fixing the O(all-entries) findVersions scan. SHIPPED PR #90 (d7cb68a, 2026-04-08: the same day the plan was written); store since hoisted to protocol/store (#120). This index wrongly listed it as unstarted until 2026-07-05.
  • Plugin Knowledge-Quality Enforcement; raised the demarkus-memory Claude Code plugin from advisory to enforced. SHIPPED v0.4.0, PR #171 merged 2026-06-01. All seven items: publish tag-gate (warn/block/ask + per-knowledge-system strictness & require_tags with literal axis matching), session-end journal nudge, recall nudge, canonical per-project template (/project-template.md), knowledge-system policy/template at the live root hub on knowledge.demarkus.io, and the /soul-doctor hygiene audit. 68 tests, pure awk/bash, zero runtime deps. Tail (separate): plugin shell tests → CI; optional nudge disable knobs.
  • Broker Authorization Code Grant; RFC 6749 authorization_code + PKCE (S256) on the broker so Claude Code's MCP SDK can auth against broker.knowledge.demarkus.io. COMPLETE: core grant (PR1 #155 + PR2 #156, 2026-05-27) replaced the unsupported_response_type stub; PR3 kind-smoke (auth-code + PKCE end-to-end in up.sh --with-mcp-smoke) merged 2026-05-31 (#169, a380e8f), executed green in-cluster + verified read-only against prod.
  • LOOKUP verb: the card-catalog verb (subject → docs + importance). Shipped to main PR #166 (2026-05-30); plugin surfacing in v0.3.0 (#168). Tail: mark_append metadata deferred by design.
  • Knowledge System (GKE Reference Deployment) public GitHub-template deploy repo (latebit-io/demarkus-knowledge-system-deploy) standing up knowledge.demarkus.io on GKE (OpenTofu + ArgoCD + OpenBao + bank-vaults + CSI-snapshot backups). Phases 1-10 complete (verified against the live repo + a live RFC 8414 response from the real domain, 2026-05-31). Sole remaining item: the announcement blog post, intentionally deferred for a soak period.
  • Universe Onboarding; last-mile join flow. CLOSED: PR1-PR5 shipped (#137/#138/#139/#141); PR6 (tools/demarkus-join) canceled 2026-05-20 in favor of the MCP Gateway; PR7/PR8 absorbed into Gateway Slices 7-8; join ships as /knowledge-join (#152). Remaining: low-priority doc debt only (two standalone deployment docs).
  • Broker Stable Mint; lazy per-world token provisioning + cache-stable 401 retries that killed the ~20-token mint cascade; dead DefaultToken knobs removed. COMPLETE (#158/#159/#163/#164/#165, 2026-05-27→29).
  • Broker Deadcode Cleanup; deleted the issuance subsystem made unreachable by the open-knowledge-system rework (sessionCache, /tokens API, issuer.go, sweeper trim). COMPLETE (#159 + #164, commit f9a24e9).
  • Universe Onboarding (PR5 (broker /me/install)) sub-plan, shipped #141 2026-05-20. Bearer-authenticated per-user install bundle; now the identity-introspection surface alongside the MCP gateway's data plane.
  • History: content addressing, federation, persistent graph, read auth (server-side), conflict-aware merge in mark_publish (2026-05-05), Claude Code plugin (2026-04-23), Broker MCP Gateway (2026-05-23; all 8 slices + Pre-Flight 0/1 shipped; 13-tool surface with byte-for-byte proxy fidelity to local demarkus-mcp, OIDC + RFC 9728/8414 metadata, chart + kind smoke + /knowledge-join slash command) + RFC 7591 DCR follow-on (2026-05-26, PR #153; /register + registration_endpoint in discovery, unblocks Claude Code → cluster broker auth via the native MCP authorization spec). Also: OKF type adoption + /soul-join managed remote souls (2026-06).

Plan Archives

Original plan documents preserved for reference:

  • Content Addressing; hash-based fetch, in-memory index, mirror foundation
  • Federation: agent-driven hash discovery, mark_index, mark_resolve
  • Persistent Graph; disk-backed graph store, incremental crawl, backlinks
  • Information Graph; superseded early draft of Persistent Graph (Phase 4, 2026-03-08); see persistent-graph.md for the version that shipped.
  • Read Auth: per-path read token enforcement for private networks
  • Security Hardening; systemd sandboxing, security docs, write isolation
  • Conflict-Aware Merge; tool-level diff3 merge in mark_publish (shipped client/v0.12.25 + v0.12.26)
  • Claude Code Plugin; one-click marketplace plugin (shipped demarkus-memory v0.1.1; v0.2.0 added /knowledge-join 2026-05-23; v0.3.0 added self-documenting guidance + lookup recall 2026-05-31; v0.4.0 enforcement + template + /soul-doctor shipped 2026-06-01, PR #171; v0.5.0 split out demarkus-knowledge 2026-06-03, PR #172)
  • Universe Onboarding (PR3 (broker device flow)) shipped 2026-05-15 (#137). RFC 8628 device flow end-to-end on the broker. Six sub-steps merged across one PR; PR4 builds on top.
  • Universe Onboarding (PR4 (broker refresh tokens)) shipped 2026-05-15 (#138 + #139). Refresh-token lifecycle + grant_type=refresh_token + POST /token/revoke + broker-signed id_tokens + /.well-known/jwks.json + compositeVerifier + Sweeper integration. Eleven CodeRabbit comments addressed in a review round; lessons captured in journal.
  • Broker MCP Gateway; shipped 2026-05-23 (v7). Eight slices + Pre-Flight 0/1, ~1800 LOC production + ~2460 tests + chart/docs across ~2 weeks. Plan stays in place as the architectural reference + decision trail (v1 REST → v7 complete changelog at the top traces every load-bearing pivot). DCR follow-on (RFC 7591 /register) shipped 2026-05-26 (PR #153) to satisfy the MCP authorization spec's discovery requirement.
  • Search Verb: superseded by LOOKUP. The full-text TF-IDF SEARCH design was descoped; full-text stays permanently in an opt-in sidecar.
  • POC Deployment; canceled. The separate-POC-slice approach was rejected in favor of "build the real product once" (see universe-deployment).
  • Obsidian Plugin; obsolete. Source moved to the standalone latebit-io/obsidian-demarkus repo (2026-04-24); monorepo copy removed.
soul.demarkus.io:6309/plans/universe-onboarding-pr4.md draft reader meta

Plan: Universe Onboarding — PR4 (Broker refresh tokens)

Sub-plan for plan §PR4 of /plans/universe-onboarding.md. Picked up after PR3 (#137, broker device flow) merged 2026-05-15. Ready to start cold next session: this doc plus /plans/universe-onboarding.md + /journal/2026-05-15.md is sufficient context.

Goal

After PR4 lands, a device-flow client that completed /device/token once can mint fresh id_token + access_token from the broker indefinitely (up to the refresh-token TTL, default 90 days) without re-running the device-code dance. Concretely:

  1. POST /device/token success body now carries refresh_token (32-byte opaque, hex-encoded).
  2. POST /device/token with grant_type=refresh_token exchanges the refresh token for a fresh ID-token + access-token pair. Existing grant_type=urn:ietf:params:oauth:grant-type:device_code keeps working unchanged.
  3. POST /token/revoke (RFC 7009) drops a refresh token from the broker. Called by /soul-leave (PR7) and on user-IdP-disable detection.
  4. Refresh tokens live in a broker-namespace Kubernetes Secret (demarkus-broker-refresh-tokens) keyed by hash — same shape as the issuances Secret. Survives broker restarts (unlike the device-flow grants).
  5. Existing sweeper picks up expired refresh tokens on its tick.

PR5 (/me/install) consumes the broker-minted id_token as a bearer to fetch the per-world install bundle. PR6 (tools/demarkus-join) is the binary that drives all of this. Each PR keeps reviewable in isolation.

Non-Negotiables Inherited

  • No protocol changes. Same as the parent plan.
  • No demarkus-server changes. All work in tools/demarkus-broker/.
  • No MVPs, no shortcuts. Full RFC 6749 §6 (refresh token grant) + RFC 7009 (revocation). Real Secret-backed storage from day one; do not stash in-memory.
  • Single broker. Refresh tokens persisted in a single broker-namespace Secret. Multi-replica reads/writes use the same optimistic-concurrency Secret pattern the issuances store uses (see mutateSecret).

Out of Scope (explicit, for PR4 specifically)

  • Refresh-token rotation. OAuth 2.1 best practice rotates the refresh token on each use. PR4 returns the SAME refresh_token on each successful refresh. Rotation adds a "previous-refresh-token grace window" + replay-detection that's a clean +200 LOC follow-up; defer unless a customer asks for it.
  • Broker-signed id_tokens. The plan's §PR4 row mentions "re-fetches user info from IdP via stored refresh-on-IdP token, OR re-uses cached claims with a TTL." PR4 takes the cached-claims path: store the verified Claims at device-code bind time, return them verbatim on refresh. Broker-signed id_tokens (with broker-hosted JWKS) is PR4-adjacent work but adds key-management surface (rotation, JWKS endpoint) that doubles PR4's scope. Defer to a later PR; document the gap in /architecture.md.
  • Refresh against the IdP. The broker does NOT carry an IdP refresh token. Storing one would require the broker to be a registered confidential client at the IdP with a long-lived token — meaningful additional risk surface. The cached-claims approach means a fired employee's IdP-side disable does not immediately invalidate the broker's refresh — that's what the broker's /token/revoke + admin tooling is for (called by an out-of-band webhook or audit-log scrape; not in PR4).
  • /me/install consumption flow. PR5.
  • Plugin / binary work. PR6 + PR7.

Architecture

┌──────────────┐  /device/token + refresh_token  ┌────────────┐
│  client      │ ──────────────────────────────► │            │
│ (curl,       │ ◄────────────────────────────── │   broker   │
│  plugin)     │   {access_token, id_token,      │            │
│              │    token_type, expires_in}      │   refreshStore (Secret-backed)
│              │                                 │   ─────────────────────────────
│  /token/revoke + refresh_token                 │   key:   sha256(refresh_token)
│ ──────────────────────────────────────────────►│   value: refreshTokenRecord
│  204 No Content                                │            ├ email
│ ◄──────────────────────────────────────────────│            ├ subject (hashed for logs)
└──────────────┘                                 │            ├ claims (full, JSON-serialized)
                                                 │            ├ issuedAt
                                                 │            ├ expiresAt
                                                 │            └ universe (broker URL)
                                                 │
                                                 │  POST /device/token entry points:
                                                 │   1. grant_type=device_code (existing) — also mints refresh now
                                                 │   2. grant_type=refresh_token (new) — uses refreshStore
                                                 └────────────┘

Layer responsibilities

Component Owns Does NOT own
refreshStore Hashed-key map of sha256(refresh_token) → refreshTokenRecord. Persisted to a single broker-namespace Secret via the existing mutateSecret pattern. Methods: Issue, Refresh, Revoke, Sweep. State transitions: active, revoked (tombstone for sweeper), expired (lazy at refresh). OIDC. HTTP. The id_token itself (the refresh dispatch in /device/token reconstructs the claims-based response from the stored record).
device.go::deviceToken (extended) Two grant_type branches: device_code (existing PR3 path, now mints refresh) and refresh_token (new). Anything Secret-backed (delegated to refreshStore).
device.go::tokenRevoke (new) POST /token/revoke per RFC 7009. Drops the record from refreshStore. Idempotent (RFC 7009 §2.2: revocation of an invalid token is not an error). Anything beyond Secret-key deletion.
Existing Sweeper Now sweeps expired refresh tokens alongside expired issuances. Tiny addition to the existing per-tick loop. refreshStore correctness (the store owns its own consistency).

Pre-Flight Tasks

None. Unlike PR3's Verifier.Exchange refactor, PR4 builds entirely on top of the PR3 surface. The ExchangeResult from PR3 already plumbs the Claims that the refresh-grant response needs to reconstruct.

Routes To Register / Modify

In server.Routes():

Method Path Middleware Notes
POST /device/token ipRateLimit (existing) Same route, new grant_type branch. Handler reads grant_type, dispatches device_code (existing) vs refresh_token (new).
POST /token/revoke ipRateLimit (new) RFC 7009. Reads token form param + optional token_type_hint. Returns 204.

No modifications to /device/authorize, GET /device, POST /device, or /auth/callback — those are device-flow-only and refresh tokens live downstream.

Sub-Tasks (sequenced)

Step 1 — refreshStore (~200 lines + ~200 tests)

  • File: tools/demarkus-broker/internal/broker/refresh.go.
  • refreshTokenRecord struct: 
    type refreshTokenRecord struct {
    KeyHash    string    `json:"keyHash"`    // sha256 hex of the token, the actual map key
    Email      string    `json:"email"`
    Subject    string    `json:"subject"`    // verbatim sub claim (not hashed — needed for Mint)
    Claims     Claims    `json:"claims"`     // full claims snapshot for the refresh response
    IssuedAt   time.Time `json:"issuedAt"`
    ExpiresAt  time.Time `json:"expiresAt"`
    LastUsedAt time.Time `json:"lastUsedAt,omitempty"`
}
  • refreshStore struct: holds a kubernetes.Interface, namespace, Secret name, optional clock + sweeper hooks. Mirrors Issuer's shape.
  • Methods:
    • Issue(ctx, claims Claims, ttl time.Duration) (rawToken string, record refreshTokenRecord, err error) — generates 32 bytes from crypto/rand, hex-encodes, hashes, stores. Returns the raw token to give to the client; never logs it.
    • Refresh(ctx, rawToken string, now time.Time) (refreshTokenRecord, error) — looks up by sha256, validates not-expired, updates LastUsedAt, returns the record. Returns ErrRefreshTokenInvalid for unknown / expired / revoked tokens.
    • Revoke(ctx, rawToken string) error — deletes the record. Idempotent.
    • Sweep(ctx, now time.Time) — invoked by the existing Sweeper. Removes records where now > ExpiresAt. Same lazy-vs-eager posture as the issuances sweep.
  • Secret storage: one Secret per broker namespace, key = refresh_tokens.json, body = JSON map {keyHash → record}. Uses the existing mutateSecret helper from issuer.go for optimistic-concurrency Read-Modify-Write.
  • Critical: the raw token is NEVER stored anywhere. Only its sha256 hash lives in the Secret. Compromise of the Secret does not let an attacker mint tokens — they'd have to brute-force pre-image sha256 of 32-byte random values (infeasible).
  • Tests:
    • Issue → Refresh round-trip returns the same Claims.
    • Refresh with unknown / tampered / expired token returns ErrRefreshTokenInvalid.
    • Revoke makes a previously-valid token reject.
    • Revoke of an unknown token is a no-op (no error).
    • Sweep removes expired entries without touching active ones.
    • Concurrent Issue + Refresh under -race using the existing fake k8s clientset.
    • Secret round-trip: write, read, parse — confirms JSON shape.

Step 2 — Mint refresh on device-code completion (~50 lines + ~50 tests)

  • File: tools/demarkus-broker/internal/broker/device.go.
  • Extend deviceTokenSuccess: 
    type deviceTokenSuccess struct {
    AccessToken  string `json:"access_token"`
    IDToken      string `json:"id_token"`
    RefreshToken string `json:"refresh_token,omitempty"`
    TokenType    string `json:"token_type"`
    ExpiresIn    int    `json:"expires_in"`
}
  • In the statusComplete branch of deviceToken, call s.refreshStore.Issue(...) after the Poll result is non-nil. Wire cfg.Server.RefreshTokenTTL as the TTL.
  • Tests:
    • PR3's TestDeviceTokenStates/complete_returns_tokens extended: assert refresh_token is non-empty and ~64 hex chars.
    • Issue failure (mock the store to error) results in 500, not a token leak.

Step 3 — grant_type=refresh_token branch (~120 lines + ~120 tests)

  • File: tools/demarkus-broker/internal/broker/device.go.
  • Dispatch in deviceToken: 
    switch r.PostFormValue("grant_type") {
case deviceGrantType:       s.deviceTokenDeviceFlow(w, r)
case refreshGrantType:      s.deviceTokenRefresh(w, r)
default:                    writeJSON(w, 400, deviceTokenError{Error: "unsupported_grant_type"})
}
  • deviceTokenRefresh:
    • Reads refresh_token form param. Missing → invalid_request.
    • Calls s.refreshStore.Refresh(...). Errors → invalid_grant (RFC 6749 §5.2).
    • Reconstructs the ExchangeResult-shaped response from the stored Claims. RawIDToken is regenerated by re-signing — wait, no, PR4 keeps cached claims and skips broker re-signing. So the response carries the ORIGINAL id_token from the device-flow completion. Problem: the original id_token has its own expiry that's already passed (otherwise why refresh?).
    • Decision point for next session: either (a) PR4 returns the cached id_token verbatim and trusts the bearer-validation downstream to handle the time skew, OR (b) PR4 does the bare-minimum re-signing (broker mints a JWT with the cached claims + fresh exp, signed with a broker-side key, with JWKS at /.well-known/jwks.json). (b) is the larger lift but cleaner.
    • Lean: (a) for PR4, document the gap, do (b) in a follow-up PR titled "PR4.5 — broker-signed id_tokens" once /me/install (PR5) makes the gap user-visible. The bearer validation today at Verifier.VerifyIDToken will reject expired id_tokens, so this would be a known broken state — meaning PR5 will need to call into the broker via the bearer regardless. So actually we need (b) for PR4 to be useful end-to-end. Re-lean: (b), broker re-signs at refresh time.
    • Add Cache-Control: no-store + Pragma: no-cache (same as the device-code success path).
  • For (b) — broker re-signing — additional surface:
    • Broker-side key material: a static signing key (ECDSA P-256) in the broker config (Secret-mounted, ENV override). New cfg.OIDC.BrokerSigningKey field. Rotation deferred to a follow-up.
    • JWKS endpoint: GET /.well-known/jwks.json serving the broker's public key in JWK format. Public, unauthenticated, no rate limit (same posture as /.well-known/openid-configuration).
    • Update Discovery.Override to point jwks_uri at the broker (currently this is intentionally left at the IdP per PR2's design note — but now we need to swap to broker-signed). Will break: any third-party JWKS client that was relying on IdP-signed tokens. Mitigation: PR4 broker still accepts BOTH broker-signed AND IdP-signed tokens during a transition window (verify with broker key first, fall back to IdP JWKS).
  • Tests:
    • Refresh with valid token → 200 + fresh access/id tokens + same refresh_token returned.
    • Refresh with unknown / revoked / expired token → 400 invalid_grant.
    • Refresh response has Cache-Control: no-store.
    • Cross-grant-type isolation: device_code-grant doesn't accept a refresh_token; refresh_token-grant doesn't accept a device_code.
    • Broker-signed id_token verification: Verifier.VerifyIDToken accepts the freshly-signed token.
    • JWKS endpoint serves a parseable JWK with the broker's public key.

Step 4 — POST /token/revoke (~80 lines + ~80 tests)

  • File: tools/demarkus-broker/internal/broker/device.go (or refresh.go; co-locate with the handler that owns it).
  • RFC 7009 surface: reads token form param, optional token_type_hint (ignored — we only support refresh tokens at this endpoint). Calls refreshStore.Revoke. Always returns 204 (idempotent — RFC 7009 §2.2: server MUST respond with 200/204 even for an invalid token).
  • Route: POST /token/revoke under ipRateLimit.
  • Tests:
    • Valid token → 204 + subsequent refresh fails.
    • Unknown token → 204 (no leak; RFC-conformant).
    • Missing token form param → 400 invalid_request.

Step 5 — Sweeper integration (~30 lines + ~30 tests)

  • File: tools/demarkus-broker/internal/broker/sweeper.go.
  • Sweeper gains a refreshStore *refreshStore field (optional; only sweeps when non-nil — keeps existing sweeper tests unchanged).
  • Per-tick: call s.refreshStore.Sweep(ctx, now) after the existing issuance sweep. Log the count of swept refresh tokens.
  • Tests: add a refresh token with ExpiresAt in the past, run one sweep, assert removal.

Step 6 — Config + wiring (~50 lines + ~30 tests)

  • tools/demarkus-broker/internal/broker/config.go:
    • ServerConfig.RefreshTokenTTL time.Duration yaml:"refreshTokenTTL"`` — default 90d, must be > 0.
    • ServerConfig.RefreshTokensSecret string yaml:"refreshTokensSecret"`` — required, no default (operator-visible name).
    • OIDCConfig.BrokerSigningKey string yaml:"brokerSigningKey"`` — base64-encoded ECDSA P-256 private key. Required when broker re-signing is enabled (always for PR4).
    • Validation: extract to applyRefreshDefaults to keep validate() inside the gocyclo budget (same pattern as applyDeviceFlowDefaults).
  • tools/demarkus-broker/internal/broker/server.go: NewServer wires the refreshStore. Routes() registers /token/revoke and /.well-known/jwks.json.
  • tools/demarkus-broker/main.go: standard plumbing.
  • Helm chart:
    • deploy/helm/demarkus-broker/values.yaml: new server.refreshTokenTTL, server.refreshTokensSecret, oidc.brokerSigningKey examples + comments.
    • deploy/helm/demarkus-broker/templates/secret-config.yaml: render the new fields.
    • deploy/helm/demarkus-broker/templates/secret-broker-signing-key.yaml (new): mounts the operator-supplied signing key.
    • deploy/helm/demarkus-broker/templates/rbac.yaml: extend RBAC to allow get/patch on the refresh tokens Secret.
    • deploy/helm/demarkus-broker/tests/: helm-unittest coverage for each new field + the new Secret template.

Scope Estimate

Step Code Tests
1. refreshStore 200 200
2. Mint on device-code 50 50
3. refresh_token grant + JWKS 250 220
4. /token/revoke 80 80
5. Sweeper integration 30 30
6. Config + chart wiring 130 80
Total ~740 ~660

Parent plan §PR4 estimated "~400 lines + tests, ~1.5 day PR." We're over by ~340 prod lines because broker-signed id_tokens (the JWKS path) wasn't in the original line estimate. Revised: ~2.5-3 day PR. Worth opening a Fritz check-in before starting if 3 days is over budget.

Open Questions To Resolve Before/During PR4

  1. Broker re-signing vs cached-id_token-verbatim. Lean (b): broker re-signs at refresh time, hosts JWKS, eats the +200 LOC. Rationale: without it, PR5's /me/install can't actually consume the bearer (the cached id_token's exp is already past — that's the whole point of refresh). Re-confirm before starting.
  2. Signing-key rotation strategy. PR4 ships a single static key. Rotation is real operational pain (must publish both old + new in JWKS during rollover; must track which key signed which token). Lean: static key for PR4, dedicated rotation PR later. Document in /architecture.md.
  3. Refresh-token rotation per OAuth 2.1. Skip per the §Out of Scope above unless someone asks.
  4. Per-user refresh-token cap. Should one user be able to hold N concurrent refresh tokens (e.g., one per device)? Lean: yes, unlimited — the Secret can hold ~5000 entries before the etcd limit, which is plenty. Per-user limits are a phase-7+ thing.
  5. Refresh against the IdP for fresh claims (groups, email_verified). Skip per the §Out of Scope. The broker uses the cached claims from device-code completion. Means: if a user's group membership changes at the IdP, the broker doesn't reflect that until re-login. Document in /architecture.md as a known trade-off.
  6. JWKS endpoint authentication. Public, unauthenticated, per OIDC. No rate limit either (same as /.well-known/openid-configuration).
  7. What happens on broker-signing-key compromise? Attacker can forge bearer tokens for any user. Mitigation: rotate the key + force-revoke all refresh tokens + force-revoke all per-world tokens (the world's Verifier rejects them once issuances are dropped). Document the incident-response playbook in /architecture.md.
  8. Verifier interface impact. Current Verifier.VerifyIDToken validates against the IdP's JWKS. PR4 needs it to ALSO accept broker-signed tokens. Two implementation shapes: (a) two-Verifier composition (broker-first, IdP-fallback), or (b) extend the existing oidcVerifier to multiplex by iss claim. Lean: (a) — cleaner abstraction, ~30 LOC adapter. Worth confirming before Step 3.

Risks Specific To PR4

  • Secret-storage scaling. Refresh tokens at 90-day TTL: a 1000-user broker holds 1000 records continuously, ~250KB JSON, well under the 1MB Secret limit. A 10k-user broker is the wall. Mitigation: document in /architecture.md; phase-7 fix is sharded Secrets or a CRD-backed store.
  • mutateSecret contention at scale. Every refresh + revoke RMWs the same Secret. Under heavy refresh load (e.g., a chart rollout that triggers every user's plugin to refresh on next session), the optimistic-concurrency retries could thunder. Mitigation: same as the issuances Secret today — it has not been an issue, would be a real signal worth instrumenting if it becomes one.
  • Broker-signing-key in a Kubernetes Secret. A cluster-admin compromise is full broker compromise. Documented risk, identical to the current OIDC.ClientSecret storage. Not unique to PR4.
  • Cross-grant-type confusion. Easy to accidentally accept a refresh_token at the device_code branch or vice versa. Strict grant_type dispatch + cross-grant tests in Step 3 guard.
  • Old tests must pass after the refresh_token,omitempty addition. PR3's TestDeviceTokenStates/complete_returns_tokens decodes into deviceTokenSuccess; the new field is omitempty, so the existing assertion holds. Worth a quick re-run before assuming.

Next-Session Resume Steps

  1. git fetch && git log --oneline -5 — confirm PR3 (#137) on main, nothing else conflicts.
  2. mark_fetch /index.md + /patterns.md + /guidelines.md per preflight.
  3. mark_fetch /plans/universe-onboarding-pr4.md (this doc).
  4. mark_fetch /journal/2026-05-15.md for the full PR3 review-lessons context.
  5. Decide on Open Question 1 (broker re-signing yes/no) — lean (b), but confirm before starting. This is the biggest scope swing.
  6. Cut a fresh branch (feat-tools-broker-refresh-tokens or similar). Start at Step 1 (refreshStore) — keep that as its own commit so the rest builds on a green Secret-backed store baseline.
  7. After Step 1: go test -race + bash pre-commit.sh green before touching Step 2. The Secret-marshaling tests are the most-likely-to-bite layer.

Touch Points With Later PRs

  • PR5 (/me/install) consumes the broker-signed id_token as a bearer. PR4 broker-signing is what makes that bearer verifiable; otherwise /me/install's requireAuth would reject a refresh-renewed token whose IdP-side exp is in the past.
  • PR6 (tools/demarkus-join) drives the refresh subcommand. Standard RFC 6749 §6 grant, standard refresh_token field — any OIDC client library handles it. The Go binary writes ~/.config/demarkus/refresh_token mode 0600.
  • PR7 (plugin slash commands) invokes tools/demarkus-join refresh on session start. The plugin doesn't know about refresh tokens directly; it shells out.
  • PR8 (docs) covers the operator-facing "what is the broker signing key, why do I need to rotate it" story. Not PR4's burden.

Done When

  • PR4 opens with all six sub-steps' commits, each individually testable.
  • go test -race ./... green inside tools/demarkus-broker/.
  • helm unittest . green (chart-side changes assert refresh-tokens Secret + signing-key Secret + RBAC).
  • pre-commit.sh green.
  • Manual end-to-end via curl: device-flow completes → refresh succeeds → revoke works → refresh after revoke fails.
  • Journal entry on /journal/<date>.md with design decisions that landed differently from this plan.

Related documents

trail
  1. soul.demarkus.io:6309 v86
  2. universe-onboarding-pr4
from here → architecture 2026-05-15 universe-onboarding-pr3 universe-onboarding-pr5 universe-onboarding graph