soul.demarkus.io/journal/2026-05-11.md/v2 draft reader meta

Journal — 2026-05-11

2026-05-11 — Universe deployment planning + agent verification

Worked through the production-deployment story for demarkus end-to-end with Fritz. Several reframes happened in sequence:

Reframe 1: "POC" means customer trial, not Claude-built slice. Initial framing treated the work as a one-week demo cut. Fritz pushed back: the deliverable is the productized k8s deployment itself (chart, broker, agent), what's "POC" is a customer (like nesto) trialing the solid product on their own infra. No corner-cuts. This collapsed the earlier /plans/poc-deployment.md into /plans/universe-deployment.md and the former was archived.

Reframe 2: No core changes, even for observability. Initially proposed a /metrics endpoint on server + broker for Prometheus. Fritz refused per the existing rule. Considered OTel SDK — same problem, still core instrumentation. Landed on log-derived metrics via collector sidecar/DaemonSet (Datadog Agent, OTel Collector, Vector, Fluent Bit, Grafana Alloy — customer picks one). Charts ship structured slog + autodiscovery annotations + reference configs. Backend-agnostic. Customer's SRE wires their backend. Captured in plan v3.

Reframe 3: Phase 5 agent was already built. Path A was "land Phase 5 agent first." On recon discovered client/cmd/demarkus-agent/ + client/internal/fedcrawl/ already implements crawl + daemon + per-host token auth + aggregated/per-server hub indexing. The roadmap saying Phase 5 was "IN PROGRESS" was misleading — the agent's been there. Memory note from 11 days ago said new utility binaries live in tools/, but the agent is at client/cmd/. Resolved by sharpening the rule: client/cmd/ = protocol clients (uses fedcrawl/fetch/tokens), tools/ = utilities (token mint, direct-store-write, sidecars). Agent stays where it is; future demarkus-broker and demarkus-sync go in tools/.

Smoke test surfaced two bugs in fedcrawl/crawl.go publishIndex:

  1. Status check only accepted protocol.StatusOK. First publish returns created, which was emitted as a misleading warning even though the publish succeeded. Fix: accept both ok and created.
  2. Always published with expected_version=0 (create-only), so every re-publish in daemon mode failed with conflict. Fix: use -1 (no check). Hub indexes are regenerated whole-doc each crawl; server's no-op-on-duplicate-content prevents version churn. The TODO comment in the original code anticipated this.

Also discovered the Makefile didn't build demarkus-agent — only demarkus, demarkus-tui, demarkus-mcp. Added it to make client.

Tests added in client/internal/fedcrawl/crawl_test.go: TestPublishIndex covers status acceptance and the -1 expected-version invariant; TestPublishIndexRePublish covers the daemon-mode case (first call created, second call ok, both succeed); TestPublishToHubs covers aggregated and per-server modes. bash pre-commit.sh clean.

End-to-end smoke verified: 3-server local setup (team-a, team-b, hub), agent crawl publishes a hub index with all 4 docs, re-runs without conflict, only 1 version on the hub (no-op kicked in when content was identical between passes). Daemon mode is now safe to run on a schedule.

Net plan state at end of session: /plans/universe-deployment.md v4. Phase 5 prereq cleared. 6.0 reduced to the agent Helm chart (binary done). Next: 6.0 chart, then 6.1 server chart. Estimate down from 4-8 weeks to 2-4 weeks.

Bug-fix changes are staged on main (working tree), not yet committed — Fritz commits himself per the project rule.

2026-05-11 (late) — Phase 6 charts landed

Three PRs in sequence after the agent verification + bugfixes:

  • #105 — fedcrawl publishIndex bugfixes (accept created, idempotent re-publish via expected_version=-1), Makefile adds demarkus-agent to make client. Two rounds of CodeRabbit comments addressed.
  • #106 — Phase 6.0 demarkus-agent Helm chart. Production-grade: Deployment + ConfigMap + optional inline-tokens Secret + ServiceAccount with Workload Identity hook. Pod is non-root, read-only root FS, seccompProfile RuntimeDefault, all caps dropped. Probes intentionally omitted (outbound scheduled job). State PVC opt-in. helm-unittest suites included. Eight rounds of CodeRabbit comments addressed before merge, plus one follow-up for deterministic Secret key ordering (sortAlpha).
  • #107 — Phase 6.1 demarkus-server Helm chart. StatefulSet + volumeClaimTemplates, LoadBalancer Service on UDP/6309 (configurable port), three TLS modes (existingSecret / cert-manager Certificate / server-generated dev cert), two-Secret token bootstrap. Critical correctness pattern: both Secrets render in one file so a single $rawToken value flows into both, keeping raw and hash consistent within one render pass. lookup against the existing token-values Secret preserves the admin token across helm upgrade. Two CodeRabbit comments addressed (label-aware lookup, README markdown formatting).

Known caveat carried forward: the current server/Dockerfile ships only demarkus-server. The server chart's exec probes need /demarkus (CLI), and the future broker integration uses /demarkus-token. Multi-binary image is Phase 6.7 (release pipeline). Charts are YAML-correct; probes will fail against the current single-binary image until 6.7 lands.

State for next session: Phase 6.2 — the demarkus-broker Go binary — is the next thing. Recommended slicing into A/B/C/D PRs (library refactor → broker skeleton → install page + DELETE + rate limit + integration test → broker chart). See bootstrap notes for Slice A scope.

2026-05-11 — Slice A merged: token-mint library extracted

PR #108 (fix(server): Centralize token hashing and implement token generation) merged. First concrete delivery toward Phase 6.2 (broker).

What landed:

  • HashToken promoted from server/internal/auth/ to protocol/ as the single source of truth for the sha256-<hex> token-hash contract. Server, CLI, and future broker all reference the same function.
  • New protocol/token/ package: Generate(label, paths, ops) Minted (pure, no I/O), ReadFile, AppendEntry, WriteFile, FormatEntry. Atomic temp+rename publishing, fsync data + parent dir for crash durability, advisory flock(2) on a sidecar .lock for cross-process serialization, defensive slices.Clone on capability slices, label quoting that handles dotted/whitespace/email-style labels correctly.
  • server/cmd/demarkus-token/main.go rewritten to delegate to protocol/token. ~80 lines of duplicated TOML/format/write logic deleted; the CLI inherits correctness (label quoting, atomicity, durability, locking) from the library.
  • Makefile + pre-commit.sh extended to cover the tools/ module.

Architectural reversal worth flagging for future agents:

Mid-PR we chose "option C" — library at tools/internal/token/, CLI stays put with protocol.HashToken only. Plan §6.2 was updated to match. CodeRabbit then blocked approval on the grounds that the PR's stated goal ("centralize token hashing") wasn't actually being met for the CLI consumer — module-boundary rules made tools/internal/token unreachable from server/. We pivoted to "option B-prime": library at protocol/token/, CLI wired to it.

The plan at /plans/universe-deployment.md §6.2 still says the library lives at tools/internal/token/ and the repository-layout block reflects the same. That needs updating before agents picking up 6.2 (broker) get confused. Action: edit §6.2 + Repository Layout to point at protocol/token/. Broker's import path is github.com/latebit/demarkus/protocol/token from tools/demarkus-broker/.

Lessons:

  • Option C was the right call for the CLI binary location (release pipeline stays unchanged). It was the wrong call for the library location. The two decisions were conflated; they should have been independent. Library placement should follow "where can all consumers reach it" first, "where does it feel architecturally clean" second.
  • protocol/ is not just wire format — it's the shared contract layer. Anything that two modules need to agree on byte-for-byte (hash format, token TOML shape) belongs there.
  • CodeRabbit's "CLI doesn't use the centralized code" critique was correct. Naming a PR "centralize X" while not wiring the existing consumer creates a credibility gap.

Next per plan sequencing: 6.0 agent Helm chart (quick win — binary already verified 2026-05-11 earlier), then 6.1 server Helm chart with helm.sh/resource-policy: keep persistence pattern, then 6.2 broker binary (can import protocol/token directly).

2026-05-11 (late) — Slice B: broker scaffold + single-world mint flow

Picked up Phase 6.2 (broker binary). Plan v7 §6.0 status was stale (chart already shipped PR #106) — Fritz redirected to 6.2. Followed the A/B/C/D slicing from the prior journal entry; this is Slice B, the broker scaffold + single-world OIDC mint flow.

Decisions locked in before writing (Fritz approved):

  • k8s.io/client-go direct, not controller-runtime — the broker doesn't reconcile, it just CRUDs Secrets.
  • coreos/go-oidc/v3 + golang.org/x/oauth2 for OIDC code flow.
  • Broker config in YAML (KnownFields(true) catches typos at startup); world tokens stay TOML (protocol contract).
  • OIDC state cookie: signed HMAC-SHA256 over JSON {nonce, expiresAt} payload; cookie is HttpOnly+Secure+SameSite=Lax on /auth/callback path only. 5-minute default TTL.
  • Code goes under tools/demarkus-broker/, internal package at internal/broker/.

Architectural reframe mid-slice: Originally planned a broker-issued session cookie for /tokens and DELETE /tokens/:label. Dropped it. Browser flow only uses the state cookie for CSRF during the OIDC dance; everything else is bearer-token authenticated using the user's OIDC ID token. Added VerifyIDToken(ctx, raw) (Claims, error) to the Verifier interface. The CLI is the primary consumer of /tokens / DELETE and already deals with token lifetimes; the browser only ever sees the one-time JSON callback response.

Touches to protocol/token/: Added two in-memory helpers AppendBytes(existing, label, *Entry) ([]byte, error) and RemoveBytes(existing, label) ([]byte, error) so the broker can do the same duplicate-check + format-or-rewrite logic as the CLI without touching disk. This is exactly the kind of additive helper the Slice A library promotion was designed for — both consumers (server/cmd/demarkus-token and tools/demarkus-broker/) read from protocol/token, byte-shape compatibility is automatic.

What landed (Slice B, ready for a PR):

  • tools/demarkus-broker/main.go — thin entry: config load, OIDC discovery, k8s client (in-cluster or kubeconfig), HTTP server, signal-driven graceful shutdown.
  • tools/demarkus-broker/internal/broker/:
    • labels.go — usr_<8 hex> opaque labels, 4 bytes of entropy + collision-retry in the issuer.
    • config.go — YAML config with eager validation. KnownFields(true) rejects typos. Rejects zero expiresAfter because short-lived tokens are the identity-lifecycle mechanism.
    • session.go — Signer for the OIDC state cookie. HMAC-SHA256, constant-time signature compare. Rejects ≥16-byte keys, expired payloads, malformed envelopes.
    • oidc.go — Verifier interface (3 methods: AuthCodeURL, Exchange, VerifyIDToken). Production impl wraps coreos/go-oidc + oauth2. Test double in oidc_test.go is the same interface, hard-codes claims, mocks discovery via httptest.
    • issuer.go — Issuer.Mint/List/Revoke against two Secrets: per-world tokens.toml Secret (tokens.toml key) + broker-namespace issuances.json Secret. Optimistic concurrency via resourceVersion with retry-on-conflict up to 5. Label collisions also retry up to 5. Mint order: world Secret first (token works immediately), then issuance record (orphan-in-issuances is the documented partial-failure mode the future sweeper prunes).
    • server.go — http.ServeMux with Go-1.22 method+path patterns. Routes: GET /healthz, GET /readyz, GET /auth/login, GET /auth/callback, GET /tokens, DELETE /tokens/{label}.
    • 87% line coverage. helm-unittest-equivalent end-to-end tests via httptest.NewServer + fake.Clientset + fakeVerifier.

Bugs I caught during development:

  • First server-test run: every callback returned 401 "invalid state". Cause: in newTestServer, I pinned s.clock = issuer.clock to a 2026-05-11 12:00 UTC fixed date. State cookie's ExpiresAt = clock() + 5min = 12:05 UTC. Real wall-clock during test run was hours later → cookie deemed expired by Verify. Decoupled the two clocks: server stays on time.Now, issuer stays pinned for predictable token-expiry assertions.

Pre-commit run clean. Several gocritic + revive fixes along the way: hugeParam on Issuance, rangeValCopy on the entries slice, exported-type docstrings, http.NoBody over nil request bodies.

Carried-forward caveats for Slice C:

  • No expiry sweeper, no leader-election Lease, no rotate, no rate limit, no SCIM webhook. All accepted backlog per plan §6.2.
  • Authorization is domain-allowlist only. Groups claim → Slice C.
  • Partial-mint behavior when multi-world iteration fails mid-loop: returns successful results + error to caller, leaves orphans in the Secrets the loop got to. Acceptable for Slice B since multi-world is mostly a Slice C concern (Slice B realistic config has one world).
  • client/go.mod and server/go.mod picked up cosmetic go 1.26 → 1.26.0 updates from go mod tidy; protocol/go.mod gained 3 indirect deps (gopkg.in/check.v1, kr/pretty, rogpeppe/go-internal) surfaced by tidy. Benign — those were latent indirects, just now recorded.

Plan §6.2 staleness to fix: the section's "Token-mint library" subsection should be updated to reference protocol/token.AppendBytes and protocol/token.RemoveBytes as the in-memory primitives. The earlier list of Generate/AppendEntry/WriteFile/ReadFile/FormatEntry is still correct but doesn't mention the two new helpers the broker actually uses.

Bug-fix and code changes staged on main (working tree), not yet committed — Fritz commits himself.

2026-05-11 (very late) — Slice B merged as PR #109 (broker binary)

PR #109 (feat(broker): Implement Kubernetes-backed OIDC token broker) merged to main as 2a6aae6. The branch went through 7 rounds of CodeRabbit review (~20 comments) between the initial commit (fd18dce) and merge. Most comments produced material behavioral changes, not cosmetic fixes — worth recording the rationale so a future Slice C / hardening pass doesn't unwind them.

Security invariants baked into Issuer.Mint (do not remove without thought):

  • !claims.EmailVerified → ErrEmailUnverified. The production Verifier already rejects unverified IDs, but Mint is reachable from test doubles and future Verifier impls. Belt-and-suspenders, not redundant.
  • strings.TrimSpace(claims.Email) == "" → ErrNotAuthorized. A world with empty AllowDomains (the "any verified user" knob) would otherwise authorize an empty/whitespace email because domainMatches short-circuits to true on empty allow-list. Worse, that empty string would land as the owner in the issuances Secret, collapsing every future no-identity caller into a shared List/Revoke namespace. The trim catches "", " ", "\t".

Correctness invariants in the mint/revoke flow:

  • Global label uniqueness in appendIssuance. removeIssuance filters by label across the whole issuances Secret. A cross-world label collision (2^32 space is small once you accumulate worlds) would let Revoke(label) find the first matching entry, remove it from world A's tokens.toml, then drop both issuance entries — leaving world B's token live and untracked. Fix: appendIssuance scans for the label and returns token.ErrLabelExists; mintForWorld treats that error as a collision, rolls back the world-Secret write, and continues the retry loop. Tested in TestMintCrossWorldLabelCollisionRetries.
  • Rollback context detaches from caller. When appendIssuance fails after appendToWorldSecret succeeded, the world Secret has an active token with no ownership record. Rollback runs on context.WithoutCancel(ctx) + 5s WithTimeout so a client hangup mid-request doesn't skip the compensating delete and leave an orphan token live. defer cancel() was rejected by gocritic (defer-in-loop); replaced with explicit cancel() after the call.
  • Revoke errors on missing world. If lookupWorld(found.World) returns nil (operator removed the world between mint and revoke), silently dropping the issuance entry would leave the token live in the orphaned world's tokens.toml. Now returns an error naming the orphaned world; the issuance record stays so the operator can investigate. Tested in TestRevokeMissingWorldPreservesIssuance.

HTTP/API hygiene:

  • Bearer parsing is case-insensitive per RFC 6750 §2.1 (strings.Fields + strings.EqualFold). Clients in the wild send Bearer, bearer, BEARER.
  • Partial mint returns 200 with {tokens: [...], partialFailure: "one_or_more_worlds_failed"} — stable code, not err.Error() (which leaks Secret names + backend failure modes). Full err text goes to the structured log. The HTTP layer previously dropped results on err != nil from Issuer.Mint, throwing away tokens that had already been activated in their Secrets.
  • Log subjects are sha256-prefixed via hashSubject(claims.Subject) rather than raw email/caller. The issuances Secret is the authoritative identity store; logs only need a stable correlation fingerprint.
  • StateTTL < 0 rejected at config load (negative durations parse fine in YAML).
  • verifier := broker.NewVerifier(context.Background(), cfg.OIDC) in main.go — coreos/go-oidc does not honor the context for JWKS refresh (it builds its own background context internally via WithoutCancel), so the prior WithCancel/defer cancel() was theatre.

Test infrastructure that catches real regressions:

  • httptest.NewTLSServer (not NewServer). The state cookie is set with Secure: true + Path: /auth/callback; without HTTPS the prior tests bypassed those attributes via manual req.AddCookie. loginAndExtract now returns a jar-backed *http.Client (shallow copy of srv.Client() so per-test mutations don't leak) and asserts the cookie made it into the jar for /auth/callback. A regression that widened the cookie path or dropped Secure fails fast.
  • testHTTPTimeout = 5s on every test client. In-process handlers should complete in microseconds; a deadlocked handler now fails the offending test in seconds instead of stalling the suite for Go's 10-minute default.
  • url.QueryEscape(nonce) consistently on all callback URLs. NewNonce is hex today so this is a no-op in practice, but locks down the contract for future encoding changes.
  • Negative-mint tests use assertNoSecretsWritten(t, k8s) which checks both the world namespace and the broker namespace. A regression that writes the issuances Secret before returning the error would otherwise slip past a world-only check.
  • TestMintLabelCollisionExhaustsRetries exercises the terminal exhaustion path (all maxLabelRetries attempts collide) and asserts labelGen was called exactly maxLabelRetries times — guards against a future change that bails early.

Things still open for Slice C (unchanged from plan §6.2):

Groups-claim authorization, expiry sweeper + Lease-based leader election, rotate endpoint, rate limit, SCIM webhook, drift-pruning sweeper test, RBAC-denied test, owner-check 403 test. Plus allowEmails: [...] on WorldConfig if individual-user carve-outs are wanted alongside groups.

trail
  1. soul.demarkus.io v2