Journal — 2026-05-13

2026-05-13 — Slice 6.3.A merged: demarkus-broker chart skeleton + security defaults

PR #115 merged to main as c71f499. First of four §6.3 sub-PRs. Lays the chart skeleton: Deployment + Service + config Secret + issuances Secret + ServiceAccount + values schema. Templates for RBAC, NetworkPolicy, PDB, Ingress, and the cert-manager Certificate ship in later slices but the values schema is declared up front so subsequent slices add templates without churning operator-facing values.yaml.

What landed

deploy/helm/demarkus-broker/ — Chart.yaml + values.yaml + 7 starting templates (deployment.yaml, service.yaml, serviceaccount.yaml, secret-config.yaml, secret-issuances.yaml, NOTES.txt, _helpers.tpl).
Cookie-key resolution helper (demarkus-broker.resolveCookieKey in _helpers.tpl). Precedence: explicit server.cookieKey → lookup of the live config Secret's cookie-key data field → freshly-generated 32-byte base64. The lookup branch is what keeps the HMAC key (and therefore every in-flight state cookie) stable across helm upgrade. The same $cookieKey variable feeds both the standalone cookie-key data field AND the cookieKey: field embedded inside config.yaml in the rendered Secret — single value, single render pass, never drifts.
Issuances Secret seed (secret-issuances.yaml). Renders an empty data: {} Secret on first install with helm.sh/resource-policy: keep on the metadata. Subsequent installs/upgrades hit the lookup-skip branch so helm sees no rendered resource and the keep-annotation on the LIVE Secret prevents helm GC. Net effect: chart updates never touch broker-minted issuance state.
OIDC client-secret schema, mutual-exclusion guarded at render-time. oidc.clientSecret (cleartext via values) and oidc.existingSecretRef.name (operator-managed Secret) are mutually exclusive. Setting both fails helm template with a clear message; setting neither fails. Render-time fail-fast rather than pod-startup CreateContainerConfigError is the whole point — the binary's applyEnvOverrides env-var path arrives in §6.3.C, so 6.3.A ships the cleartext path only but pins the mutual-exclusion guard now.
PodSecurityContext + container securityContext lockdown. runAsNonRoot: true, runAsUser: 65532, runAsGroup: 65532, fsGroup: 65532, seccompProfile: RuntimeDefault, readOnlyRootFilesystem: true, allowPrivilegeEscalation: false, capabilities: drop ["ALL"]. Five hardening defaults pinned in values.yaml; a "convenience" relaxation in any one of them should surface as a test failure in the §6.3.D helm-unittest sweep.
TopologySpreadConstraints by kubernetes.io/hostname, whenUnsatisfiable: ScheduleAnyway. Two replicas spread across nodes when possible; single-node clusters still schedule. The empty matchLabels: {} in values.yaml is intentional — the deployment template overrides it with selectorLabels so the constraint always targets the right pods even if a future operator mistakes the empty default for "fill these in." Easy to break in a refactor; worth pinning in §6.3.D.
checksum/config pod annotation so a config Secret change triggers a rollout. Without it, broker pods stale-mount after a config update.
rateLimit.trustForwardedFor: true chart default, inverting the binary's false default (C.4 §Decisions). The chart's deployment assumption is "behind an Ingress that strips spoofed X-Forwarded-For"; flipping the binary default at the chart level keeps the per-IP /auth/login limiter effective. Operator-facing docs in the chart README call out the failure mode (flipping the flag in a directly-internet-exposed broker).

Decisions that landed

Explicit metadata.namespace on both config and issuances Secrets. CodeRabbit flagged that omitting it lets helm default to the release namespace silently — fine for the default case, surprising when an operator sets server.brokerNamespace to put the issuances Secret elsewhere. Both Secrets now resolve their namespace through the demarkus-broker.brokerNamespace helper (which falls back to .Release.Namespace), so lookup + render always agree.
worldAllows nil-.allow template guard. CodeRabbit found the secret-config.yaml range .Values.worlds block would panic when a world omitted the allow: block entirely (the production-meaningful "all empty = open world" predicate path from C.1). Wrapped with {{- $allow := default (dict) .allow }} so .allow.domains etc. each get default (list) fallbacks. Pre-fix the chart aborted; post-fix the rendered config emits empty arrays for all three dimensions and the binary's worldAllows predicate takes the back-compat all-empty match.
lookup-skip namespace consistency. The issuances Secret's lookup call uses the same brokerNamespace helper as the rendered metadata, so operators that override server.brokerNamespace get the lookup hitting the correct namespace and the keep-annotation surviving across upgrades. CodeRabbit fix.

Next: §6.3.B — RBAC (per-world Roles + broker-namespace Role for sweeper Lease + issuances Secret), NetworkPolicy default-on, PodDisruptionBudget minAvailable: 1. Schema declared in A; templates ship in B.

2026-05-13 — Slice 6.3.B merged: per-world RBAC, broker-namespace RBAC, NetworkPolicy, PodDisruptionBudget

PR #116 merged to main as 175e230. Second of four §6.3 sub-PRs. Adds the RBAC + NetworkPolicy + PDB templates whose values schema 6.3.A pre-declared. Two new template files for RBAC (broker-ns and per-world) plus one each for NetworkPolicy and PDB.

What landed

rbac-worlds.yaml — per-world Role + RoleBinding loop. range .Values.worlds emits N×2 resources, each Role scoped to a single resourceNames: [<tokensSecret>] with verbs get/update (NOT create). The world's tokens Secret must already exist from the §6.1 server chart's bootstrap Job; the broker only needs to read+patch, never create. If a world's Secret is missing at first mint the broker surfaces a clear "Secret not found" error rather than silently bootstrapping into a namespace it shouldn't own.
rbac-broker-ns.yaml — broker-namespace Role + RoleBinding. Two rules: coordination.k8s.io/leases get/create/update scoped to resourceNames: [<sweeper.leaseName>] (the C.2 leader-election Lease), and secrets get/update scoped to resourceNames: [<issuancesSecret>]. The leases rule includes create so first-leadership-acquisition can seed the Lease — the k8s RBAC caveat "create cannot be restricted by resourceName" means create effectively allows any Lease in the namespace, acceptable scope since the namespace is broker-owned. The doc-comment in the template spells this out so a future tightening to just update+get doesn't break first-install silently.
networkpolicy.yaml — default-on (networkPolicy.enabled: true). Ingress allows traffic only from the configured ingressFromNamespace (default ingress-nginx) on the server port via the automatic kubernetes.io/metadata.name label (k8s 1.21+). Egress allows DNS to kube-system (UDP+TCP 53) and unrestricted TCP 443 — the latter intentionally NOT pinned to apiserver/OIDC issuer CIDRs because pinning breaks across managed-k8s providers and every IdP frontend rotation. Doc-comment notes the failure mode for self-hosted OIDC on non-443 (e.g. Keycloak) and older clusters lacking the automatic namespace label.
poddisruptionbudget.yaml — default minAvailable: 1. With two replicas and PDB-1, a single voluntary disruption (node drain, helm upgrade rollout) leaves the broker servable. Cheap insurance against in-cluster maintenance churn.
serviceAccount.workloadIdentity.gsa values plumbing into the ServiceAccount annotations. Off by default; setting it stamps iam.gke.io/gcp-service-account on the SA so GKE Workload Identity bindings can map broker→GSA without static keys. Validated values-shape only; integration is operator-side.

Decisions that landed

rbac.create: true as the default so opinionated installs work out of the box; flippable to false for clusters that manage RBAC out-of-band (Argo CD AppProject, OPA, GitOps Roles). Same shape as serviceAccount.create: true.
Verbs get/update on world tokens Secrets, NOT create. Pinning the minimal set keeps the per-world Role aligned with what the broker actually does (read-modify-write via mutateSecret's resourceVersion retry). A future refactor that adds create should land with a test failure so the question "do we want the broker bootstrapping world Secrets?" gets re-asked deliberately. §6.3.D helm-unittest will pin this.
NetworkPolicy egress unrestricted on TCP 443 by design. Restricting it to specific CIDRs trades the egress-blast-radius gain for an upgrade-and-multi-cluster brittleness that's worse in practice (apiserver IP changes per cluster; IdP frontend rotation is on-call paging territory). The default-on ingress namespace scope is the dominant security gain; egress 443 wide-open is the operator-friendly tradeoff. Documented in the doc-comment.

Next: §6.3.C — Ingress + cert-manager Certificate + oidc.existingSecretRef plumbing via a new OIDC_CLIENT_SECRET env-var override path in broker.LoadConfig.applyEnvOverrides. The env var is a small additive surface on §6.2's "broker binary complete" framing — explicitly a one-line escape hatch for the production-secret-ref deployment shape, not a feature-set expansion.

2026-05-13 — Slice 6.3.C merged: Ingress + cert-manager Certificate + external OIDC secrets

PR #117 merged to main as 599f63a. Third of four §6.3 sub-PRs. Adds Ingress + cert-manager Certificate templates and the oidc.existingSecretRef deployment path — the production shape where the OAuth client secret lives in an externally-managed Kubernetes Secret (External Secrets Operator, Sealed Secrets, Vault) instead of being baked into the chart-rendered config Secret via helm release history. Pairs the chart-side wiring with a small additive surface on the broker binary: an OIDC_CLIENT_SECRET env-var override in broker.LoadConfig.

What landed

ingress.yaml — opt-in (ingress.enabled: false default). When enabled, requires ingress.host; supports three TLS modes mutually-exclusive at the template level: tls.existingSecret (operator-provided kubernetes.io/tls), tls.certManager.enabled: true (chart provisions a cert-manager Certificate via certificate.yaml), or neither (Ingress controller terminates TLS externally). Default ingressClassName: nginx; default path / Prefix. annotations block left to the operator for controller-specific knobs (proxy-body-size, cert-manager.io/cluster-issuer, etc.).
certificate.yaml — cert-manager Certificate resource conditionally rendered when both ingress.enabled: true and ingress.tls.certManager.enabled: true. Targets the chart-named TLS Secret (<fullname>-tls); the Ingress references the same name so cert-manager writes the Secret and the Ingress picks it up by reference. DNS01 vs HTTP01 lives on the issuer, not on the Certificate resource — issuer config is operator-side.
oidc.existingSecretRef values block (name + key). When name is set, the deployment template renders an OIDC_CLIENT_SECRET env on the container via secretKeyRef pointing at the operator-managed Secret, AND the rendered config.yaml leaves clientSecret: "" blank so the env-var override path is the unambiguous runtime source.
broker.LoadConfig.applyEnvOverrides() (tools/demarkus-broker/internal/broker/config.go:262) — pure additive helper invoked between YAML parse and validate(). Today only OIDC_CLIENT_SECRET is recognized. Env wins over file when both set; empty env treated as unset so an accidentally-cleared variable doesn't blank out a file-supplied value at runtime. Net: production deployments keep the OAuth secret out of helm release history without changing the binary's startup contract.
README expanded with production checklist (Ingress + cert-manager + existingSecretRef pattern) and the rateLimit.trustForwardedFor: true invariant call-out (only safe behind an XFF-stripping Ingress; the failure mode of flipping it on a directly-internet-exposed broker is the documented footgun).
NOTES.txt post-install message summarizing rendered endpoints, RBAC scope, TLS mode, and the OIDC redirect URI the operator needs to register at the IdP.

Decisions that landed

existingSecretRef.key required at render-time when name is set. Render-time fail-fast on a blank key instead of CreateContainerConfigError at pod schedule. Three fail-render guards now live in secret-config.yaml (clientSecret + existingSecretRef.name both set, neither set, name set but key blank); §6.3.D will pin all three so the guards never regress to silent precedence.
tls.existingSecret and tls.certManager.enabled: true mutually exclusive. Allowing both would render an Ingress pointing at one Secret while certificate.yaml provisions a different Secret — silent TLS-source ambiguity. Render-time fail keeps the contract unambiguous. CodeRabbit fix.
Env wins over file, empty env is unset. The two-line applyEnvOverrides body is small enough that the test surface (4-row TestLoadConfigOIDCClientSecretEnvOverride in config_test.go) is the primary documentation of the contract. Critical row: "empty env does not clobber file value" — deterministic via t.Setenv("OIDC_CLIENT_SECRET", "") so the test asserts the empty-as-unset property regardless of whether the parent CI shell happens to have the env var exported.

Framing: env-var override is not a §6.2 reopening. It's a one-line escape hatch for the production-secret-ref deployment shape, not a feature-set expansion. §6.2 "broker binary complete" stands; the plan v14 bump will capture this as a §6.3 §Decisions bullet with a footnote on §6.2 §Status rather than reopening §6.2 in §Sequencing.

Test surface — TestLoadConfigOIDCClientSecretEnvOverride 4 rows: env overrides file, env supplies value when file empty, empty env does not clobber file value, no env + no file → validate() rejects oidc.clientSecret is required. Coverage stable at 83.4%.

Next: §6.3.D — helm-unittest test files for the broker chart covering the chart invariants (resource-policy: keep, RBAC verbs, NetworkPolicy egress/ingress shape, OIDC mutual-exclusion guards, OIDC_CLIENT_SECRET env wiring, securityContext lockdown, topology spread, checksum/config), CI plumbing to run helm-unittest against both demarkus-server and demarkus-broker charts on PR, and folding in a test-broker Go-test job (the §6.2 broker binary currently has zero CI coverage — .github/workflows/ci.yml's detect-changes only filters protocol/server/client, not tools/). Kind upgrade-wipe regression test deferred to a sidecar PR bundled with the §6.1 chart's missing helm.sh/resource-policy: keep annotation on the demarkus-server tokens Secret — plan v13 §6.1 prose claims it's set but deploy/helm/demarkus-server/templates/tokens.yaml ships without it. Real prior-work debt; tracked separately so §6.3.D isn't blocked behind the §6.1 fix.

2026-05-13 — Slice 6.3.D.1 merged: chart unittest suites + CI plumbing, broker binary CI gap closed

PR #118 merged to main as part of §6.3 broker chart work. Fourth and last sub-PR for §6.3 chart proper. One CodeRabbit round with five comments (four inline + one out-of-diff); all addressed without pushback — none were strategic enough to warrant a defense.

What landed

deploy/helm/demarkus-broker/tests/ — seven helm-unittest test suites covering the chart's load-bearing invariants: Deployment (replicas, security context lockdown, HTTP probes, POD_NAME env, OIDC_CLIENT_SECRET env wiring in both modes, populated topology-spread matchLabels, checksum/config rollout-on-config-change), broker-ns + per-world RBAC (verbs scoped to get/update-not-create, lease verbs get/create/update, rbac.create=false skip), NetworkPolicy (default-on, ingress namespace scoping, DNS+443 egress shape), Secret config (rate-limit + sweeper defaults explicitly rendered, OIDC mutex guards × 3 fail-render cases, world-without-allow-block doesn't panic), Secret issuances (helm.sh/resource-policy: keep + brokerNamespace helper consistency), Ingress + cert-manager Certificate (TLS mode mutex, host-required, className override), PodDisruptionBudget (default minAvailable: 1, selector labels). 49 tests across 7 suites.
.github/workflows/ci.yml — added broker (tools/demarkus-broker/** + tools/go.mod + tools/go.sum + protocol/**) and charts (deploy/helm/**) paths to the change-detection filter. New test-broker job: Go test/vet/lint/build for tools/demarkus-broker/ with working-directory: tools/demarkus-broker (narrowed past the broader tools/ scope to avoid lint failures on tools/tools.go). New test-charts job: helm lint on all three charts + helm unittest on server + broker. Both new jobs timeout-minutes: 20. helm pinned to v3.13.2; helm-unittest pinned to v0.6.2 (compatible across the local helm version).
deploy/helm/demarkus-server/tests/statefulset_test.yaml — fixed real prior-work bug. The existing suite only loaded statefulset.yaml, so the StatefulSet template's include "...tokens.yaml" for the checksum/config annotation failed render and all 13 tests errored out. Added tokens.yaml to the suite templates list + template: statefulset.yaml per-test to scope assertions. Nobody had noticed because helm-unittest never ran in CI. The §6.1 chart-test claims in plan v13 weren't actually executing — until now.

Decisions / observations

Fold-in of test-broker was the right call. The handoff's §6.3.D scope proposed adding helm-unittest to CI + flagged separately that the §6.2 broker binary had zero CI coverage. Folding the binary-CI job into the same PR meant the broker chart shipped paired with a binary that finally has real CI gates. Without the fold-in we'd have had the absurdity of testing the chart's behavior in CI while the binary it wraps remained CI-uncovered.
helm-unittest v0.6.2 over v1.x. v1.0+ uses platformHooks in plugin.yaml which needs helm v3.16+; Fritz's local helm is v3.13.2. Pinning v0.6.2 in CI (matching the local-dev validation surface) keeps the test behavior identical between developer machines and CI. Bump together later when helm itself bumps.
Per-test template: scope is required when one template includes another. The shared checksum/config: {{ include "...secret-config.yaml" }} pattern (and the §6.1 chart's analogous tokens.yaml include) means tests for the parent template must list BOTH templates in the suite templates: filter, and then narrow each test's assertion target via the singular template: field. Discovered while debugging the broker deployment_test.yaml; the same fix retroactively rescued the §6.1 statefulset suite.

CodeRabbit comments addressed (all five, no pushback)

Broker path filter narrowing — tools/** → tools/demarkus-broker/** + tools/go.mod + tools/go.sum. CodeRabbit's static analysis confirmed tools/tools.go sits outside the broker package; the broad filter would trigger broker CI on unrelated tool changes.
Job timeouts — timeout-minutes: 20 on test-broker and test-charts. Default GitHub Actions timeout (360min) is too lenient for these jobs.
golangci-lint scope — working-directory: tools → tools/demarkus-broker. Same reasoning as #1; verified locally that golangci-lint walks up to tools/go.mod for module resolution.
CLAUDE.md "Records" fragment removed — line 38 had a malformed header ("Records (one per decision, zero-padded 4-digit sequence)") followed by a sub-bullet describing journals not records. Collapsed into a single journal entry matching the surrounding flat-list style. Surfaced to Fritz that a separate ADR-style Records concept can be added back if intended.
CLAUDE.md journal-path contradiction (out-of-diff) — line 27 said /journal.md, line 39 said /<project>/journal/<YYYY-MM-DD>.md. Updated line 27 to match the dated convention. Caveat flagged: this made CLAUDE.md aspirational ahead of the soul, which still had the legacy /journal.md shape at PR-merge time. Migration to dated files happens in this very session, immediately after PR merge.

Prior-work debt surfaced (still open, sidecar PR §6.3.D.2 scope)

§6.1 demarkus-server tokens.yaml Secret missing helm.sh/resource-policy: keep. Plan v13 §6.1 §Status claims the annotation is set and a kind integration test pins the upgrade-wipe regression. Reality at PR-merge time: the annotation is NOT on the rendered Secret and no kind test exists in CI. Real bug in shipped §6.1 chart; bundle the fix with the §6.3.D.2 kind harness so the regression test catches it directly.
§6.2 broker binary had zero CI coverage at the start of this session. Closed by the test-broker job in this PR; surfacing it here so the gap doesn't get lost in the diff.
helm-unittest wasn't running anywhere. The §6.1 chart's existing test files weren't executed by CI, AND the statefulset_test.yaml suite was structurally broken from the day it was written. Both fixed in this PR. Lesson: chart-test files in the repo don't equal CI test coverage of charts; the workflow plumbing matters at least as much as the test files.

Next: §6.3.D.2 sidecar — kind upgrade-wipe regression for both charts, bundled with the §6.1 helm.sh/resource-policy: keep + helm.sh/hook: pre-install fix on demarkus-server/templates/tokens.yaml. Test shape: install chart, kubectl-patch the rendered tokens/issuances Secret to seed fake data, helm upgrade with a values change, assert seeded data persists. Broker pod doesn't need to be ready for the property to hold — kind harness can skip OIDC discovery via a values stub. Plan bump to v14 after §6.3.D.2 lands so v14 captures coherent §6.3-complete landed-state.

2026-05-13 — Journal migrated to dated per-day files

Operational note. Until this session, all session notes lived in a single growing /journal.md file (v57 at migration time, ~140KB / 51 entries across 23 unique dates). PR #118 merged a CLAUDE.md edit committing to a /journal/<YYYY-MM-DD>.md per-day convention; this entry is the first written under that convention.

Migration mechanic: the legacy /journal.md body was parsed by ## YYYY-MM-DD — heading boundaries, grouped by date (multiple entries on the same date stayed in one file as sequential ## sections), and published as /journal/<YYYY-MM-DD>.md for each unique date. Same content, different topology. The old /journal.md was kept (mark_archive not used — the version history at the legacy path is still readable) and /index.md updated to point at the new directory.

Why per-day: a 140KB single file was bumping fetch-result size limits (had to be saved to file rather than returned inline; agents had to read the saved file in chunks). Per-day files cap at ~40KB worst case (2026-05-12 was the longest day, with 5 entries totaling 38KB) and most days are well under 10KB. Future fetches by date go straight to the right file rather than scrolling a 51-entry catalog.

Convention going forward: mark_append to the current date's file when adding a session note. If the file doesn't exist yet, mark_publish it. The CLAUDE.md "End of Session" line points operators (and agents) at the right path.