# Journal — 2026-05-13 ## 2026-05-13 — Slice 6.3.A merged: demarkus-broker chart skeleton + security defaults PR #115 merged to main as `c71f499`. First of four §6.3 sub-PRs. Lays the chart skeleton: Deployment + Service + config Secret + issuances Secret + ServiceAccount + values schema. Templates for RBAC, NetworkPolicy, PDB, Ingress, and the cert-manager Certificate ship in later slices but the values schema is declared up front so subsequent slices add templates without churning operator-facing values.yaml. **What landed** - **`deploy/helm/demarkus-broker/`** — Chart.yaml + values.yaml + 7 starting templates (`deployment.yaml`, `service.yaml`, `serviceaccount.yaml`, `secret-config.yaml`, `secret-issuances.yaml`, `NOTES.txt`, `_helpers.tpl`). - **Cookie-key resolution helper** (`demarkus-broker.resolveCookieKey` in `_helpers.tpl`). Precedence: explicit `server.cookieKey` → `lookup` of the live config Secret's `cookie-key` data field → freshly-generated 32-byte base64. The lookup branch is what keeps the HMAC key (and therefore every in-flight state cookie) stable across `helm upgrade`. The same `$cookieKey` variable feeds both the standalone `cookie-key` data field AND the `cookieKey:` field embedded inside `config.yaml` in the rendered Secret — single value, single render pass, never drifts. - **Issuances Secret seed** (`secret-issuances.yaml`). Renders an empty `data: {}` Secret on first install with `helm.sh/resource-policy: keep` on the metadata. Subsequent installs/upgrades hit the `lookup`-skip branch so helm sees no rendered resource and the keep-annotation on the LIVE Secret prevents helm GC. Net effect: chart updates never touch broker-minted issuance state. - **OIDC client-secret schema, mutual-exclusion guarded at render-time.** `oidc.clientSecret` (cleartext via values) and `oidc.existingSecretRef.name` (operator-managed Secret) are mutually exclusive. Setting both fails `helm template` with a clear message; setting neither fails. Render-time fail-fast rather than pod-startup `CreateContainerConfigError` is the whole point — the binary's `applyEnvOverrides` env-var path arrives in §6.3.C, so 6.3.A ships the cleartext path only but pins the mutual-exclusion guard now. - **PodSecurityContext + container securityContext lockdown.** `runAsNonRoot: true`, `runAsUser: 65532`, `runAsGroup: 65532`, `fsGroup: 65532`, `seccompProfile: RuntimeDefault`, `readOnlyRootFilesystem: true`, `allowPrivilegeEscalation: false`, `capabilities: drop ["ALL"]`. Five hardening defaults pinned in values.yaml; a "convenience" relaxation in any one of them should surface as a test failure in the §6.3.D helm-unittest sweep. - **TopologySpreadConstraints by `kubernetes.io/hostname`, `whenUnsatisfiable: ScheduleAnyway`.** Two replicas spread across nodes when possible; single-node clusters still schedule. The empty `matchLabels: {}` in values.yaml is intentional — the deployment template overrides it with `selectorLabels` so the constraint always targets the right pods even if a future operator mistakes the empty default for "fill these in." Easy to break in a refactor; worth pinning in §6.3.D. - **`checksum/config` pod annotation** so a config Secret change triggers a rollout. Without it, broker pods stale-mount after a config update. - **`rateLimit.trustForwardedFor: true` chart default**, inverting the binary's `false` default (C.4 §Decisions). The chart's deployment assumption is "behind an Ingress that strips spoofed X-Forwarded-For"; flipping the binary default at the chart level keeps the per-IP `/auth/login` limiter effective. Operator-facing docs in the chart README call out the failure mode (flipping the flag in a directly-internet-exposed broker). **Decisions that landed** - **Explicit `metadata.namespace` on both config and issuances Secrets.** CodeRabbit flagged that omitting it lets helm default to the release namespace silently — fine for the default case, surprising when an operator sets `server.brokerNamespace` to put the issuances Secret elsewhere. Both Secrets now resolve their namespace through the `demarkus-broker.brokerNamespace` helper (which falls back to `.Release.Namespace`), so lookup + render always agree. - **`worldAllows` nil-`.allow` template guard.** CodeRabbit found the `secret-config.yaml` `range .Values.worlds` block would panic when a world omitted the `allow:` block entirely (the production-meaningful "all empty = open world" predicate path from C.1). Wrapped with `{{- $allow := default (dict) .allow }}` so `.allow.domains` etc. each get `default (list)` fallbacks. Pre-fix the chart aborted; post-fix the rendered config emits empty arrays for all three dimensions and the binary's `worldAllows` predicate takes the back-compat all-empty match. - **`lookup`-skip namespace consistency.** The issuances Secret's `lookup` call uses the same `brokerNamespace` helper as the rendered metadata, so operators that override `server.brokerNamespace` get the lookup hitting the correct namespace and the keep-annotation surviving across upgrades. CodeRabbit fix. **Next: §6.3.B** — RBAC (per-world Roles + broker-namespace Role for sweeper Lease + issuances Secret), NetworkPolicy default-on, PodDisruptionBudget `minAvailable: 1`. Schema declared in A; templates ship in B. ## 2026-05-13 — Slice 6.3.B merged: per-world RBAC, broker-namespace RBAC, NetworkPolicy, PodDisruptionBudget PR #116 merged to main as `175e230`. Second of four §6.3 sub-PRs. Adds the RBAC + NetworkPolicy + PDB templates whose values schema 6.3.A pre-declared. Two new template files for RBAC (broker-ns and per-world) plus one each for NetworkPolicy and PDB. **What landed** - **`rbac-worlds.yaml`** — per-world Role + RoleBinding loop. `range .Values.worlds` emits N×2 resources, each Role scoped to a single `resourceNames: []` with verbs `get/update` (NOT `create`). The world's tokens Secret must already exist from the §6.1 server chart's bootstrap Job; the broker only needs to read+patch, never create. If a world's Secret is missing at first mint the broker surfaces a clear "Secret not found" error rather than silently bootstrapping into a namespace it shouldn't own. - **`rbac-broker-ns.yaml`** — broker-namespace Role + RoleBinding. Two rules: `coordination.k8s.io/leases` `get/create/update` scoped to `resourceNames: []` (the C.2 leader-election Lease), and `secrets` `get/update` scoped to `resourceNames: []`. The leases rule includes `create` so first-leadership-acquisition can seed the Lease — the k8s RBAC caveat "create cannot be restricted by resourceName" means create effectively allows any Lease in the namespace, acceptable scope since the namespace is broker-owned. The doc-comment in the template spells this out so a future tightening to just `update`+`get` doesn't break first-install silently. - **`networkpolicy.yaml`** — default-on (`networkPolicy.enabled: true`). Ingress allows traffic only from the configured `ingressFromNamespace` (default `ingress-nginx`) on the server port via the automatic `kubernetes.io/metadata.name` label (k8s 1.21+). Egress allows DNS to `kube-system` (UDP+TCP 53) and unrestricted TCP 443 — the latter intentionally NOT pinned to apiserver/OIDC issuer CIDRs because pinning breaks across managed-k8s providers and every IdP frontend rotation. Doc-comment notes the failure mode for self-hosted OIDC on non-443 (e.g. Keycloak) and older clusters lacking the automatic namespace label. - **`poddisruptionbudget.yaml`** — default `minAvailable: 1`. With two replicas and PDB-1, a single voluntary disruption (node drain, helm upgrade rollout) leaves the broker servable. Cheap insurance against in-cluster maintenance churn. - **`serviceAccount.workloadIdentity.gsa`** values plumbing into the ServiceAccount annotations. Off by default; setting it stamps `iam.gke.io/gcp-service-account` on the SA so GKE Workload Identity bindings can map broker→GSA without static keys. Validated values-shape only; integration is operator-side. **Decisions that landed** - **`rbac.create: true` as the default** so opinionated installs work out of the box; flippable to false for clusters that manage RBAC out-of-band (Argo CD AppProject, OPA, GitOps Roles). Same shape as `serviceAccount.create: true`. - **Verbs `get/update` on world tokens Secrets, NOT `create`.** Pinning the minimal set keeps the per-world Role aligned with what the broker actually does (read-modify-write via `mutateSecret`'s `resourceVersion` retry). A future refactor that adds `create` should land with a test failure so the question "do we want the broker bootstrapping world Secrets?" gets re-asked deliberately. §6.3.D helm-unittest will pin this. - **NetworkPolicy egress unrestricted on TCP 443 by design.** Restricting it to specific CIDRs trades the egress-blast-radius gain for an upgrade-and-multi-cluster brittleness that's worse in practice (apiserver IP changes per cluster; IdP frontend rotation is on-call paging territory). The default-on ingress namespace scope is the dominant security gain; egress 443 wide-open is the operator-friendly tradeoff. Documented in the doc-comment. **Next: §6.3.C** — Ingress + cert-manager Certificate + `oidc.existingSecretRef` plumbing via a new `OIDC_CLIENT_SECRET` env-var override path in `broker.LoadConfig.applyEnvOverrides`. The env var is a small additive surface on §6.2's "broker binary complete" framing — explicitly a one-line escape hatch for the production-secret-ref deployment shape, not a feature-set expansion. ## 2026-05-13 — Slice 6.3.C merged: Ingress + cert-manager Certificate + external OIDC secrets PR #117 merged to main as `599f63a`. Third of four §6.3 sub-PRs. Adds Ingress + cert-manager Certificate templates and the `oidc.existingSecretRef` deployment path — the production shape where the OAuth client secret lives in an externally-managed Kubernetes Secret (External Secrets Operator, Sealed Secrets, Vault) instead of being baked into the chart-rendered config Secret via helm release history. Pairs the chart-side wiring with a small additive surface on the broker binary: an `OIDC_CLIENT_SECRET` env-var override in `broker.LoadConfig`. **What landed** - **`ingress.yaml`** — opt-in (`ingress.enabled: false` default). When enabled, requires `ingress.host`; supports three TLS modes mutually-exclusive at the template level: `tls.existingSecret` (operator-provided `kubernetes.io/tls`), `tls.certManager.enabled: true` (chart provisions a cert-manager Certificate via `certificate.yaml`), or neither (Ingress controller terminates TLS externally). Default `ingressClassName: nginx`; default path `/` `Prefix`. `annotations` block left to the operator for controller-specific knobs (`proxy-body-size`, `cert-manager.io/cluster-issuer`, etc.). - **`certificate.yaml`** — cert-manager `Certificate` resource conditionally rendered when both `ingress.enabled: true` and `ingress.tls.certManager.enabled: true`. Targets the chart-named TLS Secret (`-tls`); the Ingress references the same name so cert-manager writes the Secret and the Ingress picks it up by reference. DNS01 vs HTTP01 lives on the issuer, not on the Certificate resource — issuer config is operator-side. - **`oidc.existingSecretRef`** values block (name + key). When `name` is set, the deployment template renders an `OIDC_CLIENT_SECRET` env on the container via `secretKeyRef` pointing at the operator-managed Secret, AND the rendered config.yaml leaves `clientSecret: ""` blank so the env-var override path is the unambiguous runtime source. - **`broker.LoadConfig.applyEnvOverrides()`** (`tools/demarkus-broker/internal/broker/config.go:262`) — pure additive helper invoked between YAML parse and `validate()`. Today only `OIDC_CLIENT_SECRET` is recognized. Env wins over file when both set; empty env treated as unset so an accidentally-cleared variable doesn't blank out a file-supplied value at runtime. Net: production deployments keep the OAuth secret out of helm release history without changing the binary's startup contract. - **README expanded** with production checklist (Ingress + cert-manager + existingSecretRef pattern) and the `rateLimit.trustForwardedFor: true` invariant call-out (only safe behind an XFF-stripping Ingress; the failure mode of flipping it on a directly-internet-exposed broker is the documented footgun). - **`NOTES.txt`** post-install message summarizing rendered endpoints, RBAC scope, TLS mode, and the OIDC redirect URI the operator needs to register at the IdP. **Decisions that landed** - **`existingSecretRef.key` required at render-time when `name` is set.** Render-time fail-fast on a blank `key` instead of `CreateContainerConfigError` at pod schedule. Three fail-render guards now live in `secret-config.yaml` (clientSecret + existingSecretRef.name both set, neither set, name set but key blank); §6.3.D will pin all three so the guards never regress to silent precedence. - **`tls.existingSecret` and `tls.certManager.enabled: true` mutually exclusive.** Allowing both would render an Ingress pointing at one Secret while `certificate.yaml` provisions a different Secret — silent TLS-source ambiguity. Render-time `fail` keeps the contract unambiguous. CodeRabbit fix. - **Env wins over file, empty env is unset.** The two-line `applyEnvOverrides` body is small enough that the test surface (4-row `TestLoadConfigOIDCClientSecretEnvOverride` in `config_test.go`) is the primary documentation of the contract. Critical row: "empty env does not clobber file value" — deterministic via `t.Setenv("OIDC_CLIENT_SECRET", "")` so the test asserts the empty-as-unset property regardless of whether the parent CI shell happens to have the env var exported. **Framing: env-var override is not a §6.2 reopening.** It's a one-line escape hatch for the production-secret-ref deployment shape, not a feature-set expansion. §6.2 "broker binary complete" stands; the plan v14 bump will capture this as a §6.3 §Decisions bullet with a footnote on §6.2 §Status rather than reopening §6.2 in §Sequencing. **Test surface** — `TestLoadConfigOIDCClientSecretEnvOverride` 4 rows: env overrides file, env supplies value when file empty, empty env does not clobber file value, no env + no file → `validate()` rejects `oidc.clientSecret is required`. Coverage stable at 83.4%. **Next: §6.3.D** — helm-unittest test files for the broker chart covering the chart invariants (resource-policy: keep, RBAC verbs, NetworkPolicy egress/ingress shape, OIDC mutual-exclusion guards, OIDC_CLIENT_SECRET env wiring, securityContext lockdown, topology spread, checksum/config), CI plumbing to run helm-unittest against both demarkus-server and demarkus-broker charts on PR, and folding in a `test-broker` Go-test job (the §6.2 broker binary currently has zero CI coverage — `.github/workflows/ci.yml`'s `detect-changes` only filters protocol/server/client, not tools/). Kind upgrade-wipe regression test deferred to a sidecar PR bundled with the §6.1 chart's missing `helm.sh/resource-policy: keep` annotation on the demarkus-server tokens Secret — plan v13 §6.1 prose claims it's set but `deploy/helm/demarkus-server/templates/tokens.yaml` ships without it. Real prior-work debt; tracked separately so §6.3.D isn't blocked behind the §6.1 fix. ## 2026-05-13 — Slice 6.3.D.1 merged: chart unittest suites + CI plumbing, broker binary CI gap closed PR #118 merged to main as part of §6.3 broker chart work. Fourth and last sub-PR for §6.3 chart proper. One CodeRabbit round with five comments (four inline + one out-of-diff); all addressed without pushback — none were strategic enough to warrant a defense. **What landed** - **`deploy/helm/demarkus-broker/tests/`** — seven helm-unittest test suites covering the chart's load-bearing invariants: Deployment (replicas, security context lockdown, HTTP probes, POD_NAME env, OIDC_CLIENT_SECRET env wiring in both modes, populated topology-spread matchLabels, `checksum/config` rollout-on-config-change), broker-ns + per-world RBAC (verbs scoped to `get/update`-not-`create`, lease verbs `get/create/update`, `rbac.create=false` skip), NetworkPolicy (default-on, ingress namespace scoping, DNS+443 egress shape), Secret config (rate-limit + sweeper defaults explicitly rendered, OIDC mutex guards × 3 fail-render cases, world-without-allow-block doesn't panic), Secret issuances (`helm.sh/resource-policy: keep` + brokerNamespace helper consistency), Ingress + cert-manager Certificate (TLS mode mutex, host-required, className override), PodDisruptionBudget (default `minAvailable: 1`, selector labels). 49 tests across 7 suites. - **`.github/workflows/ci.yml`** — added `broker` (`tools/demarkus-broker/**` + `tools/go.mod` + `tools/go.sum` + `protocol/**`) and `charts` (`deploy/helm/**`) paths to the change-detection filter. New `test-broker` job: Go test/vet/lint/build for `tools/demarkus-broker/` with `working-directory: tools/demarkus-broker` (narrowed past the broader `tools/` scope to avoid lint failures on `tools/tools.go`). New `test-charts` job: `helm lint` on all three charts + `helm unittest` on server + broker. Both new jobs `timeout-minutes: 20`. helm pinned to v3.13.2; helm-unittest pinned to v0.6.2 (compatible across the local helm version). - **`deploy/helm/demarkus-server/tests/statefulset_test.yaml`** — fixed real prior-work bug. The existing suite only loaded `statefulset.yaml`, so the StatefulSet template's `include "...tokens.yaml"` for the `checksum/config` annotation failed render and all 13 tests errored out. Added `tokens.yaml` to the suite templates list + `template: statefulset.yaml` per-test to scope assertions. Nobody had noticed because helm-unittest never ran in CI. The §6.1 chart-test claims in plan v13 weren't actually executing — until now. **Decisions / observations** - **Fold-in of `test-broker` was the right call.** The handoff's §6.3.D scope proposed adding helm-unittest to CI + flagged separately that the §6.2 broker binary had zero CI coverage. Folding the binary-CI job into the same PR meant the broker chart shipped paired with a binary that finally has real CI gates. Without the fold-in we'd have had the absurdity of testing the chart's behavior in CI while the binary it wraps remained CI-uncovered. - **helm-unittest v0.6.2 over v1.x.** v1.0+ uses `platformHooks` in plugin.yaml which needs helm v3.16+; Fritz's local helm is v3.13.2. Pinning v0.6.2 in CI (matching the local-dev validation surface) keeps the test behavior identical between developer machines and CI. Bump together later when helm itself bumps. - **Per-test `template:` scope is required when one template `include`s another.** The shared `checksum/config: {{ include "...secret-config.yaml" }}` pattern (and the §6.1 chart's analogous `tokens.yaml` include) means tests for the parent template must list BOTH templates in the suite `templates:` filter, and then narrow each test's assertion target via the singular `template:` field. Discovered while debugging the broker `deployment_test.yaml`; the same fix retroactively rescued the §6.1 statefulset suite. **CodeRabbit comments addressed (all five, no pushback)** 1. **Broker path filter narrowing** — `tools/**` → `tools/demarkus-broker/**` + `tools/go.mod` + `tools/go.sum`. CodeRabbit's static analysis confirmed `tools/tools.go` sits outside the broker package; the broad filter would trigger broker CI on unrelated tool changes. 2. **Job timeouts** — `timeout-minutes: 20` on `test-broker` and `test-charts`. Default GitHub Actions timeout (360min) is too lenient for these jobs. 3. **golangci-lint scope** — `working-directory: tools` → `tools/demarkus-broker`. Same reasoning as #1; verified locally that golangci-lint walks up to `tools/go.mod` for module resolution. 4. **CLAUDE.md "Records" fragment removed** — line 38 had a malformed header ("Records (one per decision, zero-padded 4-digit sequence)") followed by a sub-bullet describing journals not records. Collapsed into a single journal entry matching the surrounding flat-list style. Surfaced to Fritz that a separate ADR-style Records concept can be added back if intended. 5. **CLAUDE.md journal-path contradiction (out-of-diff)** — line 27 said `/journal.md`, line 39 said `//journal/.md`. Updated line 27 to match the dated convention. Caveat flagged: this made CLAUDE.md aspirational ahead of the soul, which still had the legacy `/journal.md` shape at PR-merge time. Migration to dated files happens in this very session, immediately after PR merge. **Prior-work debt surfaced (still open, sidecar PR §6.3.D.2 scope)** - **§6.1 demarkus-server `tokens.yaml` Secret missing `helm.sh/resource-policy: keep`.** Plan v13 §6.1 §Status claims the annotation is set and a kind integration test pins the upgrade-wipe regression. Reality at PR-merge time: the annotation is NOT on the rendered Secret and no kind test exists in CI. Real bug in shipped §6.1 chart; bundle the fix with the §6.3.D.2 kind harness so the regression test catches it directly. - **§6.2 broker binary had zero CI coverage** at the start of this session. Closed by the `test-broker` job in this PR; surfacing it here so the gap doesn't get lost in the diff. - **helm-unittest wasn't running anywhere.** The §6.1 chart's existing test files weren't executed by CI, AND the `statefulset_test.yaml` suite was structurally broken from the day it was written. Both fixed in this PR. Lesson: chart-test files in the repo don't equal CI test coverage of charts; the workflow plumbing matters at least as much as the test files. **Next: §6.3.D.2 sidecar** — kind upgrade-wipe regression for both charts, bundled with the §6.1 `helm.sh/resource-policy: keep` + `helm.sh/hook: pre-install` fix on `demarkus-server/templates/tokens.yaml`. Test shape: install chart, kubectl-patch the rendered tokens/issuances Secret to seed fake data, helm upgrade with a values change, assert seeded data persists. Broker pod doesn't need to be ready for the property to hold — kind harness can skip OIDC discovery via a values stub. Plan bump to v14 after §6.3.D.2 lands so v14 captures coherent §6.3-complete landed-state. ## 2026-05-13 — Journal migrated to dated per-day files Operational note. Until this session, all session notes lived in a single growing `/journal.md` file (v57 at migration time, ~140KB / 51 entries across 23 unique dates). PR #118 merged a CLAUDE.md edit committing to a `/journal/.md` per-day convention; this entry is the first written under that convention. Migration mechanic: the legacy `/journal.md` body was parsed by `## YYYY-MM-DD —` heading boundaries, grouped by date (multiple entries on the same date stayed in one file as sequential `##` sections), and published as `/journal/.md` for each unique date. Same content, different topology. The old `/journal.md` was kept (mark_archive not used — the version history at the legacy path is still readable) and `/index.md` updated to point at the new directory. Why per-day: a 140KB single file was bumping fetch-result size limits (had to be saved to file rather than returned inline; agents had to read the saved file in chunks). Per-day files cap at ~40KB worst case (2026-05-12 was the longest day, with 5 entries totaling 38KB) and most days are well under 10KB. Future fetches by date go straight to the right file rather than scrolling a 51-entry catalog. Convention going forward: `mark_append` to the current date's file when adding a session note. If the file doesn't exist yet, `mark_publish` it. The CLAUDE.md "End of Session" line points operators (and agents) at the right path.