# Journal — 2026-05-11

## 2026-05-11 — Universe deployment planning + agent verification

Worked through the production-deployment story for demarkus end-to-end with Fritz. Several reframes happened in sequence:

**Reframe 1: "POC" means customer trial, not Claude-built slice.** Initial framing treated the work as a one-week demo cut. Fritz pushed back: the deliverable is the productized k8s deployment itself (chart, broker, agent), what's "POC" is a customer (like nesto) trialing the solid product on their own infra. No corner-cuts. This collapsed the earlier `/plans/poc-deployment.md` into `/plans/universe-deployment.md` and the former was archived.

**Reframe 2: No core changes, even for observability.** Initially proposed a `/metrics` endpoint on server + broker for Prometheus. Fritz refused per the existing rule. Considered OTel SDK — same problem, still core instrumentation. Landed on log-derived metrics via collector sidecar/DaemonSet (Datadog Agent, OTel Collector, Vector, Fluent Bit, Grafana Alloy — customer picks one). Charts ship structured slog + autodiscovery annotations + reference configs. Backend-agnostic. Customer's SRE wires their backend. Captured in plan v3.

**Reframe 3: Phase 5 agent was already built.** Path A was "land Phase 5 agent first." On recon discovered `client/cmd/demarkus-agent/` + `client/internal/fedcrawl/` already implements crawl + daemon + per-host token auth + aggregated/per-server hub indexing. The roadmap saying Phase 5 was "IN PROGRESS" was misleading — the agent's been there. Memory note from 11 days ago said new utility binaries live in `tools/`, but the agent is at `client/cmd/`. Resolved by sharpening the rule: `client/cmd/` = protocol clients (uses `fedcrawl`/`fetch`/`tokens`), `tools/` = utilities (token mint, direct-store-write, sidecars). Agent stays where it is; future `demarkus-broker` and `demarkus-sync` go in `tools/`.

**Smoke test surfaced two bugs in `fedcrawl/crawl.go publishIndex`:**

1. Status check only accepted `protocol.StatusOK`. First publish returns `created`, which was emitted as a misleading warning even though the publish succeeded. Fix: accept both `ok` and `created`.
2. Always published with `expected_version=0` (create-only), so every re-publish in daemon mode failed with `conflict`. Fix: use `-1` (no check). Hub indexes are regenerated whole-doc each crawl; server's no-op-on-duplicate-content prevents version churn. The TODO comment in the original code anticipated this.

Also discovered the `Makefile` didn't build `demarkus-agent` — only `demarkus`, `demarkus-tui`, `demarkus-mcp`. Added it to `make client`.

Tests added in `client/internal/fedcrawl/crawl_test.go`: `TestPublishIndex` covers status acceptance and the `-1` expected-version invariant; `TestPublishIndexRePublish` covers the daemon-mode case (first call `created`, second call `ok`, both succeed); `TestPublishToHubs` covers aggregated and per-server modes. `bash pre-commit.sh` clean.

End-to-end smoke verified: 3-server local setup (team-a, team-b, hub), agent crawl publishes a hub index with all 4 docs, re-runs without conflict, only 1 version on the hub (no-op kicked in when content was identical between passes). Daemon mode is now safe to run on a schedule.

Net plan state at end of session: `/plans/universe-deployment.md` v4. Phase 5 prereq cleared. 6.0 reduced to the agent Helm chart (binary done). Next: 6.0 chart, then 6.1 server chart. Estimate down from 4-8 weeks to 2-4 weeks.

Bug-fix changes are staged on `main` (working tree), not yet committed — Fritz commits himself per the project rule.

## 2026-05-11 (late) — Phase 6 charts landed

Three PRs in sequence after the agent verification + bugfixes:

- **#105** — fedcrawl publishIndex bugfixes (accept `created`, idempotent re-publish via `expected_version=-1`), Makefile adds `demarkus-agent` to `make client`. Two rounds of CodeRabbit comments addressed.
- **#106** — Phase 6.0 demarkus-agent Helm chart. Production-grade: Deployment + ConfigMap + optional inline-tokens Secret + ServiceAccount with Workload Identity hook. Pod is non-root, read-only root FS, seccompProfile RuntimeDefault, all caps dropped. Probes intentionally omitted (outbound scheduled job). State PVC opt-in. helm-unittest suites included. Eight rounds of CodeRabbit comments addressed before merge, plus one follow-up for deterministic Secret key ordering (`sortAlpha`).
- **#107** — Phase 6.1 demarkus-server Helm chart. StatefulSet + `volumeClaimTemplates`, LoadBalancer Service on UDP/6309 (configurable port), three TLS modes (existingSecret / cert-manager Certificate / server-generated dev cert), two-Secret token bootstrap. Critical correctness pattern: both Secrets render in one file so a single `$rawToken` value flows into both, keeping raw and hash consistent within one render pass. `lookup` against the existing token-values Secret preserves the admin token across `helm upgrade`. Two CodeRabbit comments addressed (label-aware lookup, README markdown formatting).

**Known caveat carried forward:** the current `server/Dockerfile` ships only `demarkus-server`. The server chart's exec probes need `/demarkus` (CLI), and the future broker integration uses `/demarkus-token`. Multi-binary image is Phase 6.7 (release pipeline). Charts are YAML-correct; probes will fail against the current single-binary image until 6.7 lands.

**State for next session:** Phase 6.2 — the `demarkus-broker` Go binary — is the next thing. Recommended slicing into A/B/C/D PRs (library refactor → broker skeleton → install page + DELETE + rate limit + integration test → broker chart). See bootstrap notes for Slice A scope.

## 2026-05-11 — Slice A merged: token-mint library extracted

PR #108 (`fix(server): Centralize token hashing and implement token generation`) merged. First concrete delivery toward Phase 6.2 (broker).

**What landed:**

- `HashToken` promoted from `server/internal/auth/` to `protocol/` as the single source of truth for the `sha256-<hex>` token-hash contract. Server, CLI, and future broker all reference the same function.
- New `protocol/token/` package: `Generate(label, paths, ops) Minted` (pure, no I/O), `ReadFile`, `AppendEntry`, `WriteFile`, `FormatEntry`. Atomic temp+rename publishing, `fsync` data + parent dir for crash durability, advisory `flock(2)` on a sidecar `.lock` for cross-process serialization, defensive `slices.Clone` on capability slices, label quoting that handles dotted/whitespace/email-style labels correctly.
- `server/cmd/demarkus-token/main.go` rewritten to delegate to `protocol/token`. ~80 lines of duplicated TOML/format/write logic deleted; the CLI inherits correctness (label quoting, atomicity, durability, locking) from the library.
- `Makefile` + `pre-commit.sh` extended to cover the `tools/` module.

**Architectural reversal worth flagging for future agents:**

Mid-PR we chose "option C" — library at `tools/internal/token/`, CLI stays put with `protocol.HashToken` only. Plan §6.2 was updated to match. CodeRabbit then blocked approval on the grounds that the PR's stated goal ("centralize token hashing") wasn't actually being met for the CLI consumer — module-boundary rules made `tools/internal/token` unreachable from `server/`. We pivoted to "option B-prime": library at `protocol/token/`, CLI wired to it.

The plan at `/plans/universe-deployment.md` §6.2 still says the library lives at `tools/internal/token/` and the repository-layout block reflects the same. **That needs updating** before agents picking up 6.2 (broker) get confused. Action: edit §6.2 + Repository Layout to point at `protocol/token/`. Broker's import path is `github.com/latebit/demarkus/protocol/token` from `tools/demarkus-broker/`.

**Lessons:**

- Option C was the right call for the CLI binary location (release pipeline stays unchanged). It was the wrong call for the library location. The two decisions were conflated; they should have been independent. Library placement should follow "where can all consumers reach it" first, "where does it feel architecturally clean" second.
- `protocol/` is not just wire format — it's the shared contract layer. Anything that two modules need to agree on byte-for-byte (hash format, token TOML shape) belongs there.
- CodeRabbit's "CLI doesn't use the centralized code" critique was correct. Naming a PR "centralize X" while not wiring the existing consumer creates a credibility gap.

**Next per plan sequencing:** 6.0 agent Helm chart (quick win — binary already verified 2026-05-11 earlier), then 6.1 server Helm chart with `helm.sh/resource-policy: keep` persistence pattern, then 6.2 broker binary (can import `protocol/token` directly).

## 2026-05-11 (late) — Slice B: broker scaffold + single-world mint flow

Picked up Phase 6.2 (broker binary). Plan v7 §6.0 status was stale (chart already shipped PR #106) — Fritz redirected to 6.2. Followed the A/B/C/D slicing from the prior journal entry; this is Slice B, the broker scaffold + single-world OIDC mint flow.

**Decisions locked in before writing (Fritz approved):**

- `k8s.io/client-go` direct, not controller-runtime — the broker doesn't reconcile, it just CRUDs Secrets.
- `coreos/go-oidc/v3` + `golang.org/x/oauth2` for OIDC code flow.
- Broker config in YAML (`KnownFields(true)` catches typos at startup); world tokens stay TOML (protocol contract).
- OIDC state cookie: signed HMAC-SHA256 over JSON `{nonce, expiresAt}` payload; cookie is HttpOnly+Secure+SameSite=Lax on `/auth/callback` path only. 5-minute default TTL.
- Code goes under `tools/demarkus-broker/`, internal package at `internal/broker/`.

**Architectural reframe mid-slice:** Originally planned a broker-issued session cookie for `/tokens` and `DELETE /tokens/:label`. Dropped it. Browser flow only uses the state cookie for CSRF during the OIDC dance; everything else is bearer-token authenticated using the user's OIDC ID token. Added `VerifyIDToken(ctx, raw) (Claims, error)` to the `Verifier` interface. The CLI is the primary consumer of `/tokens` / `DELETE` and already deals with token lifetimes; the browser only ever sees the one-time JSON callback response.

**Touches to `protocol/token/`:** Added two in-memory helpers `AppendBytes(existing, label, *Entry) ([]byte, error)` and `RemoveBytes(existing, label) ([]byte, error)` so the broker can do the same duplicate-check + format-or-rewrite logic as the CLI without touching disk. This is exactly the kind of additive helper the Slice A library promotion was designed for — both consumers (`server/cmd/demarkus-token` and `tools/demarkus-broker/`) read from `protocol/token`, byte-shape compatibility is automatic.

**What landed (Slice B, ready for a PR):**

- `tools/demarkus-broker/main.go` — thin entry: config load, OIDC discovery, k8s client (in-cluster or kubeconfig), HTTP server, signal-driven graceful shutdown.
- `tools/demarkus-broker/internal/broker/`:
  - `labels.go` — `usr_<8 hex>` opaque labels, 4 bytes of entropy + collision-retry in the issuer.
  - `config.go` — YAML config with eager validation. `KnownFields(true)` rejects typos. Rejects zero `expiresAfter` because short-lived tokens are the identity-lifecycle mechanism.
  - `session.go` — `Signer` for the OIDC state cookie. HMAC-SHA256, constant-time signature compare. Rejects ≥16-byte keys, expired payloads, malformed envelopes.
  - `oidc.go` — `Verifier` interface (3 methods: `AuthCodeURL`, `Exchange`, `VerifyIDToken`). Production impl wraps `coreos/go-oidc` + `oauth2`. Test double in `oidc_test.go` is the same interface, hard-codes claims, mocks discovery via `httptest`.
  - `issuer.go` — `Issuer.Mint/List/Revoke` against two Secrets: per-world `tokens.toml` Secret (`tokens.toml` key) + broker-namespace `issuances.json` Secret. Optimistic concurrency via `resourceVersion` with retry-on-conflict up to 5. Label collisions also retry up to 5. Mint order: world Secret first (token works immediately), then issuance record (orphan-in-issuances is the documented partial-failure mode the future sweeper prunes).
  - `server.go` — `http.ServeMux` with Go-1.22 method+path patterns. Routes: `GET /healthz`, `GET /readyz`, `GET /auth/login`, `GET /auth/callback`, `GET /tokens`, `DELETE /tokens/{label}`.
  - 87% line coverage. `helm-unittest`-equivalent end-to-end tests via `httptest.NewServer` + `fake.Clientset` + `fakeVerifier`.

**Bugs I caught during development:**

- First server-test run: every callback returned 401 "invalid state". Cause: in `newTestServer`, I pinned `s.clock = issuer.clock` to a 2026-05-11 12:00 UTC fixed date. State cookie's `ExpiresAt = clock() + 5min = 12:05 UTC`. Real wall-clock during test run was hours later → cookie deemed expired by `Verify`. Decoupled the two clocks: server stays on `time.Now`, issuer stays pinned for predictable token-expiry assertions.

**Pre-commit run clean.** Several gocritic + revive fixes along the way: hugeParam on `Issuance`, rangeValCopy on the entries slice, exported-type docstrings, http.NoBody over nil request bodies.

**Carried-forward caveats for Slice C:**

- No expiry sweeper, no leader-election Lease, no rotate, no rate limit, no SCIM webhook. All accepted backlog per plan §6.2.
- Authorization is domain-allowlist only. Groups claim → Slice C.
- Partial-mint behavior when multi-world iteration fails mid-loop: returns successful results + error to caller, leaves orphans in the Secrets the loop got to. Acceptable for Slice B since multi-world is mostly a Slice C concern (Slice B realistic config has one world).
- `client/go.mod` and `server/go.mod` picked up cosmetic `go 1.26 → 1.26.0` updates from `go mod tidy`; `protocol/go.mod` gained 3 indirect deps (`gopkg.in/check.v1`, `kr/pretty`, `rogpeppe/go-internal`) surfaced by tidy. Benign — those were latent indirects, just now recorded.

**Plan §6.2 staleness to fix:** the section's "Token-mint library" subsection should be updated to reference `protocol/token.AppendBytes` and `protocol/token.RemoveBytes` as the in-memory primitives. The earlier list of `Generate/AppendEntry/WriteFile/ReadFile/FormatEntry` is still correct but doesn't mention the two new helpers the broker actually uses.

Bug-fix and code changes staged on `main` (working tree), not yet committed — Fritz commits himself.

## 2026-05-11 (very late) — Slice B merged as PR #109 (broker binary)

PR #109 (`feat(broker): Implement Kubernetes-backed OIDC token broker`) merged to main as `2a6aae6`. The branch went through 7 rounds of CodeRabbit review (~20 comments) between the initial commit (`fd18dce`) and merge. Most comments produced material behavioral changes, not cosmetic fixes — worth recording the rationale so a future Slice C / hardening pass doesn't unwind them.

**Security invariants baked into `Issuer.Mint` (do not remove without thought):**

- `!claims.EmailVerified` → `ErrEmailUnverified`. The production `Verifier` already rejects unverified IDs, but `Mint` is reachable from test doubles and future Verifier impls. Belt-and-suspenders, not redundant.
- `strings.TrimSpace(claims.Email) == ""` → `ErrNotAuthorized`. A world with empty `AllowDomains` (the "any verified user" knob) would otherwise authorize an empty/whitespace email because `domainMatches` short-circuits to `true` on empty allow-list. Worse, that empty string would land as the *owner* in the issuances Secret, collapsing every future no-identity caller into a shared `List`/`Revoke` namespace. The trim catches `""`, `"   "`, `"\t"`.

**Correctness invariants in the mint/revoke flow:**

- **Global label uniqueness in `appendIssuance`.** `removeIssuance` filters by label across the whole issuances Secret. A cross-world label collision (2^32 space is small once you accumulate worlds) would let `Revoke(label)` find the first matching entry, remove it from world A's tokens.toml, then drop *both* issuance entries — leaving world B's token live and untracked. Fix: `appendIssuance` scans for the label and returns `token.ErrLabelExists`; `mintForWorld` treats that error as a collision, rolls back the world-Secret write, and `continue`s the retry loop. Tested in `TestMintCrossWorldLabelCollisionRetries`.
- **Rollback context detaches from caller.** When `appendIssuance` fails after `appendToWorldSecret` succeeded, the world Secret has an active token with no ownership record. Rollback runs on `context.WithoutCancel(ctx)` + `5s WithTimeout` so a client hangup mid-request doesn't skip the compensating delete and leave an orphan token live. `defer cancel()` was rejected by gocritic (defer-in-loop); replaced with explicit `cancel()` after the call.
- **`Revoke` errors on missing world.** If `lookupWorld(found.World)` returns nil (operator removed the world between mint and revoke), silently dropping the issuance entry would leave the token live in the orphaned world's tokens.toml. Now returns an error naming the orphaned world; the issuance record stays so the operator can investigate. Tested in `TestRevokeMissingWorldPreservesIssuance`.

**HTTP/API hygiene:**

- Bearer parsing is case-insensitive per RFC 6750 §2.1 (`strings.Fields` + `strings.EqualFold`). Clients in the wild send `Bearer`, `bearer`, `BEARER`.
- Partial mint returns `200` with `{tokens: [...], partialFailure: "one_or_more_worlds_failed"}` — stable code, not `err.Error()` (which leaks Secret names + backend failure modes). Full err text goes to the structured log. The HTTP layer previously dropped `results` on `err != nil` from `Issuer.Mint`, throwing away tokens that had already been activated in their Secrets.
- Log subjects are sha256-prefixed via `hashSubject(claims.Subject)` rather than raw email/caller. The issuances Secret is the authoritative identity store; logs only need a stable correlation fingerprint.
- `StateTTL < 0` rejected at config load (negative durations parse fine in YAML).
- `verifier := broker.NewVerifier(context.Background(), cfg.OIDC)` in `main.go` — coreos/go-oidc does not honor the context for JWKS refresh (it builds its own background context internally via `WithoutCancel`), so the prior `WithCancel`/`defer cancel()` was theatre.

**Test infrastructure that catches real regressions:**

- `httptest.NewTLSServer` (not `NewServer`). The state cookie is set with `Secure: true` + `Path: /auth/callback`; without HTTPS the prior tests bypassed those attributes via manual `req.AddCookie`. `loginAndExtract` now returns a jar-backed `*http.Client` (shallow copy of `srv.Client()` so per-test mutations don't leak) and asserts the cookie made it into the jar for `/auth/callback`. A regression that widened the cookie path or dropped `Secure` fails fast.
- `testHTTPTimeout = 5s` on every test client. In-process handlers should complete in microseconds; a deadlocked handler now fails the offending test in seconds instead of stalling the suite for Go's 10-minute default.
- `url.QueryEscape(nonce)` consistently on all callback URLs. `NewNonce` is hex today so this is a no-op in practice, but locks down the contract for future encoding changes.
- Negative-mint tests use `assertNoSecretsWritten(t, k8s)` which checks both the world namespace *and* the broker namespace. A regression that writes the issuances Secret before returning the error would otherwise slip past a world-only check.
- `TestMintLabelCollisionExhaustsRetries` exercises the terminal exhaustion path (all `maxLabelRetries` attempts collide) and asserts `labelGen` was called exactly `maxLabelRetries` times — guards against a future change that bails early.

**Things still open for Slice C (unchanged from plan §6.2):**

Groups-claim authorization, expiry sweeper + Lease-based leader election, rotate endpoint, rate limit, SCIM webhook, drift-pruning sweeper test, RBAC-denied test, owner-check 403 test. Plus `allowEmails: [...]` on `WorldConfig` if individual-user carve-outs are wanted alongside groups.