Journal — 2026-05-13

2026-05-13 — Slice 6.3.A merged: demarkus-broker chart skeleton + security defaults

PR #115 merged to main as c71f499. First of four §6.3 sub-PRs. Lays the chart skeleton: Deployment + Service + config Secret + issuances Secret + ServiceAccount + values schema. Templates for RBAC, NetworkPolicy, PDB, Ingress, and the cert-manager Certificate ship in later slices but the values schema is declared up front so subsequent slices add templates without churning operator-facing values.yaml.

What landed

• deploy/helm/demarkus-broker/ — Chart.yaml + values.yaml + 7 starting templates (deployment.yaml, service.yaml, serviceaccount.yaml, secret-config.yaml, secret-issuances.yaml, NOTES.txt, _helpers.tpl).
• Cookie-key resolution helper (demarkus-broker.resolveCookieKey in _helpers.tpl). Precedence: explicit server.cookieKey → lookup of the live config Secret's cookie-key data field → freshly-generated 32-byte base64. The lookup branch is what keeps the HMAC key (and therefore every in-flight state cookie) stable across helm upgrade. The same $cookieKey variable feeds both the standalone cookie-key data field AND the cookieKey: field embedded inside config.yaml in the rendered Secret — single value, single render pass, never drifts.
• Issuances Secret seed (secret-issuances.yaml). Renders an empty data: {} Secret on first install with helm.sh/resource-policy: keep on the metadata. Subsequent installs/upgrades hit the lookup-skip branch so helm sees no rendered resource and the keep-annotation on the LIVE Secret prevents helm GC. Net effect: chart updates never touch broker-minted issuance state.
• OIDC client-secret schema, mutual-exclusion guarded at render-time. oidc.clientSecret (cleartext via values) and oidc.existingSecretRef.name (operator-managed Secret) are mutually exclusive. Setting both fails helm template with a clear message; setting neither fails. Render-time fail-fast rather than pod-startup CreateContainerConfigError is the whole point — the binary's applyEnvOverrides env-var path arrives in §6.3.C, so 6.3.A ships the cleartext path only but pins the mutual-exclusion guard now.
• PodSecurityContext + container securityContext lockdown. runAsNonRoot: true, runAsUser: 65532, runAsGroup: 65532, fsGroup: 65532, seccompProfile: RuntimeDefault, readOnlyRootFilesystem: true, allowPrivilegeEscalation: false, capabilities: drop ["ALL"]. Five hardening defaults pinned in values.yaml; a "convenience" relaxation in any one of them should surface as a test failure in the §6.3.D helm-unittest sweep.
• TopologySpreadConstraints by kubernetes.io/hostname, whenUnsatisfiable: ScheduleAnyway. Two replicas spread across nodes when possible; single-node clusters still schedule. The empty matchLabels: {} in values.yaml is intentional — the deployment template overrides it with selectorLabels so the constraint always targets the right pods even if a future operator mistakes the empty default for "fill these in." Easy to break in a refactor; worth pinning in §6.3.D.
• checksum/config pod annotation so a config Secret change triggers a rollout. Without it, broker pods stale-mount after a config update.
• rateLimit.trustForwardedFor: true chart default, inverting the binary's false default (C.4 §Decisions). The chart's deployment assumption is "behind an Ingress that strips spoofed X-Forwarded-For"; flipping the binary default at the chart level keeps the per-IP /auth/login limiter effective. Operator-facing docs in the chart README call out the failure mode (flipping the flag in a directly-internet-exposed broker).

Decisions that landed

• Explicit metadata.namespace on both config and issuances Secrets. CodeRabbit flagged that omitting it lets helm default to the release namespace silently — fine for the default case, surprising when an operator sets server.brokerNamespace to put the issuances Secret elsewhere. Both Secrets now resolve their namespace through the demarkus-broker.brokerNamespace helper (which falls back to .Release.Namespace), so lookup + render always agree.
• worldAllows nil-.allow template guard. CodeRabbit found the secret-config.yaml range .Values.worlds block would panic when a world omitted the allow: block entirely (the production-meaningful "all empty = open world" predicate path from C.1). Wrapped with {{- $allow := default (dict) .allow }} so .allow.domains etc. each get default (list) fallbacks. Pre-fix the chart aborted; post-fix the rendered config emits empty arrays for all three dimensions and the binary's worldAllows predicate takes the back-compat all-empty match.
• lookup-skip namespace consistency. The issuances Secret's lookup call uses the same brokerNamespace helper as the rendered metadata, so operators that override server.brokerNamespace get the lookup hitting the correct namespace and the keep-annotation surviving across upgrades. CodeRabbit fix.

Next: §6.3.B — RBAC (per-world Roles + broker-namespace Role for sweeper Lease + issuances Secret), NetworkPolicy default-on, PodDisruptionBudget minAvailable: 1. Schema declared in A; templates ship in B.

2026-05-13 — Slice 6.3.B merged: per-world RBAC, broker-namespace RBAC, NetworkPolicy, PodDisruptionBudget

PR #116 merged to main as 175e230. Second of four §6.3 sub-PRs. Adds the RBAC + NetworkPolicy + PDB templates whose values schema 6.3.A pre-declared. Two new template files for RBAC (broker-ns and per-world) plus one each for NetworkPolicy and PDB.

What landed

• rbac-worlds.yaml — per-world Role + RoleBinding loop. range .Values.worlds emits N×2 resources, each Role scoped to a single resourceNames: [<tokensSecret>] with verbs get/update (NOT create). The world's tokens Secret must already exist from the §6.1 server chart's bootstrap Job; the broker only needs to read+patch, never create. If a world's Secret is missing at first mint the broker surfaces a clear "Secret not found" error rather than silently bootstrapping into a namespace it shouldn't own.
• rbac-broker-ns.yaml — broker-namespace Role + RoleBinding. Two rules: coordination.k8s.io/leases get/create/update scoped to resourceNames: [<sweeper.leaseName>] (the C.2 leader-election Lease), and secrets get/update scoped to resourceNames: [<issuancesSecret>]. The leases rule includes create so first-leadership-acquisition can seed the Lease — the k8s RBAC caveat "create cannot be restricted by resourceName" means create effectively allows any Lease in the namespace, acceptable scope since the namespace is broker-owned. The doc-comment in the template spells this out so a future tightening to just update+get doesn't break first-install silently.
• networkpolicy.yaml — default-on (networkPolicy.enabled: true). Ingress allows traffic only from the configured ingressFromNamespace (default ingress-nginx) on the server port via the automatic kubernetes.io/metadata.name label (k8s 1.21+). Egress allows DNS to kube-system (UDP+TCP 53) and unrestricted TCP 443 — the latter intentionally NOT pinned to apiserver/OIDC issuer CIDRs because pinning breaks across managed-k8s providers and every IdP frontend rotation. Doc-comment notes the failure mode for self-hosted OIDC on non-443 (e.g. Keycloak) and older clusters lacking the automatic namespace label.
• poddisruptionbudget.yaml — default minAvailable: 1. With two replicas and PDB-1, a single voluntary disruption (node drain, helm upgrade rollout) leaves the broker servable. Cheap insurance against in-cluster maintenance churn.
• serviceAccount.workloadIdentity.gsa values plumbing into the ServiceAccount annotations. Off by default; setting it stamps iam.gke.io/gcp-service-account on the SA so GKE Workload Identity bindings can map broker→GSA without static keys. Validated values-shape only; integration is operator-side.

Decisions that landed

• rbac.create: true as the default so opinionated installs work out of the box; flippable to false for clusters that manage RBAC out-of-band (Argo CD AppProject, OPA, GitOps Roles). Same shape as serviceAccount.create: true.
• Verbs get/update on world tokens Secrets, NOT create. Pinning the minimal set keeps the per-world Role aligned with what the broker actually does (read-modify-write via mutateSecret's resourceVersion retry). A future refactor that adds create should land with a test failure so the question "do we want the broker bootstrapping world Secrets?" gets re-asked deliberately. §6.3.D helm-unittest will pin this.
• NetworkPolicy egress unrestricted on TCP 443 by design. Restricting it to specific CIDRs trades the egress-blast-radius gain for an upgrade-and-multi-cluster brittleness that's worse in practice (apiserver IP changes per cluster; IdP frontend rotation is on-call paging territory). The default-on ingress namespace scope is the dominant security gain; egress 443 wide-open is the operator-friendly tradeoff. Documented in the doc-comment.

Next: §6.3.C — Ingress + cert-manager Certificate + oidc.existingSecretRef plumbing via a new OIDC_CLIENT_SECRET env-var override path in broker.LoadConfig.applyEnvOverrides. The env var is a small additive surface on §6.2's "broker binary complete" framing — explicitly a one-line escape hatch for the production-secret-ref deployment shape, not a feature-set expansion.

2026-05-13 — Slice 6.3.C merged: Ingress + cert-manager Certificate + external OIDC secrets

PR #117 merged to main as 599f63a. Third of four §6.3 sub-PRs. Adds Ingress + cert-manager Certificate templates and the oidc.existingSecretRef deployment path — the production shape where the OAuth client secret lives in an externally-managed Kubernetes Secret (External Secrets Operator, Sealed Secrets, Vault) instead of being baked into the chart-rendered config Secret via helm release history. Pairs the chart-side wiring with a small additive surface on the broker binary: an OIDC_CLIENT_SECRET env-var override in broker.LoadConfig.

What landed

• ingress.yaml — opt-in (ingress.enabled: false default). When enabled, requires ingress.host; supports three TLS modes mutually-exclusive at the template level: tls.existingSecret (operator-provided kubernetes.io/tls), tls.certManager.enabled: true (chart provisions a cert-manager Certificate via certificate.yaml), or neither (Ingress controller terminates TLS externally). Default ingressClassName: nginx; default path / Prefix. annotations block left to the operator for controller-specific knobs (proxy-body-size, cert-manager.io/cluster-issuer, etc.).
• certificate.yaml — cert-manager Certificate resource conditionally rendered when both ingress.enabled: true and ingress.tls.certManager.enabled: true. Targets the chart-named TLS Secret (<fullname>-tls); the Ingress references the same name so cert-manager writes the Secret and the Ingress picks it up by reference. DNS01 vs HTTP01 lives on the issuer, not on the Certificate resource — issuer config is operator-side.
• oidc.existingSecretRef values block (name + key). When name is set, the deployment template renders an OIDC_CLIENT_SECRET env on the container via secretKeyRef pointing at the operator-managed Secret, AND the rendered config.yaml leaves clientSecret: "" blank so the env-var override path is the unambiguous runtime source.
• broker.LoadConfig.applyEnvOverrides() (tools/demarkus-broker/internal/broker/config.go:262) — pure additive helper invoked between YAML parse and validate(). Today only OIDC_CLIENT_SECRET is recognized. Env wins over file when both set; empty env treated as unset so an accidentally-cleared variable doesn't blank out a file-supplied value at runtime. Net: production deployments keep the OAuth secret out of helm release history without changing the binary's startup contract.
• README expanded with production checklist (Ingress + cert-manager + existingSecretRef pattern) and the rateLimit.trustForwardedFor: true invariant call-out (only safe behind an XFF-stripping Ingress; the failure mode of flipping it on a directly-internet-exposed broker is the documented footgun).
• NOTES.txt post-install message summarizing rendered endpoints, RBAC scope, TLS mode, and the OIDC redirect URI the operator needs to register at the IdP.

Decisions that landed

• existingSecretRef.key required at render-time when name is set. Render-time fail-fast on a blank key instead of CreateContainerConfigError at pod schedule. Three fail-render guards now live in secret-config.yaml (clientSecret + existingSecretRef.name both set, neither set, name set but key blank); §6.3.D will pin all three so the guards never regress to silent precedence.
• tls.existingSecret and tls.certManager.enabled: true mutually exclusive. Allowing both would render an Ingress pointing at one Secret while certificate.yaml provisions a different Secret — silent TLS-source ambiguity. Render-time fail keeps the contract unambiguous. CodeRabbit fix.
• Env wins over file, empty env is unset. The two-line applyEnvOverrides body is small enough that the test surface (4-row TestLoadConfigOIDCClientSecretEnvOverride in config_test.go) is the primary documentation of the contract. Critical row: "empty env does not clobber file value" — deterministic via t.Setenv("OIDC_CLIENT_SECRET", "") so the test asserts the empty-as-unset property regardless of whether the parent CI shell happens to have the env var exported.

Framing: env-var override is not a §6.2 reopening. It's a one-line escape hatch for the production-secret-ref deployment shape, not a feature-set expansion. §6.2 "broker binary complete" stands; the plan v14 bump will capture this as a §6.3 §Decisions bullet with a footnote on §6.2 §Status rather than reopening §6.2 in §Sequencing.

Test surface — TestLoadConfigOIDCClientSecretEnvOverride 4 rows: env overrides file, env supplies value when file empty, empty env does not clobber file value, no env + no file → validate() rejects oidc.clientSecret is required. Coverage stable at 83.4%.

Next: §6.3.D — helm-unittest test files for the broker chart covering the chart invariants (resource-policy: keep, RBAC verbs, NetworkPolicy egress/ingress shape, OIDC mutual-exclusion guards, OIDC_CLIENT_SECRET env wiring, securityContext lockdown, topology spread, checksum/config), CI plumbing to run helm-unittest against both demarkus-server and demarkus-broker charts on PR, and folding in a test-broker Go-test job (the §6.2 broker binary currently has zero CI coverage — .github/workflows/ci.yml's detect-changes only filters protocol/server/client, not tools/). Kind upgrade-wipe regression test deferred to a sidecar PR bundled with the §6.1 chart's missing helm.sh/resource-policy: keep annotation on the demarkus-server tokens Secret — plan v13 §6.1 prose claims it's set but deploy/helm/demarkus-server/templates/tokens.yaml ships without it. Real prior-work debt; tracked separately so §6.3.D isn't blocked behind the §6.1 fix.

2026-05-13 — Slice 6.3.D.1 merged: chart unittest suites + CI plumbing, broker binary CI gap closed

PR #118 merged to main as part of §6.3 broker chart work. Fourth and last sub-PR for §6.3 chart proper. One CodeRabbit round with five comments (four inline + one out-of-diff); all addressed without pushback — none were strategic enough to warrant a defense.

What landed

• deploy/helm/demarkus-broker/tests/ — seven helm-unittest test suites covering the chart's load-bearing invariants: Deployment (replicas, security context lockdown, HTTP probes, POD_NAME env, OIDC_CLIENT_SECRET env wiring in both modes, populated topology-spread matchLabels, checksum/config rollout-on-config-change), broker-ns + per-world RBAC (verbs scoped to get/update-not-create, lease verbs get/create/update, rbac.create=false skip), NetworkPolicy (default-on, ingress namespace scoping, DNS+443 egress shape), Secret config (rate-limit + sweeper defaults explicitly rendered, OIDC mutex guards × 3 fail-render cases, world-without-allow-block doesn't panic), Secret issuances (helm.sh/resource-policy: keep + brokerNamespace helper consistency), Ingress + cert-manager Certificate (TLS mode mutex, host-required, className override), PodDisruptionBudget (default minAvailable: 1, selector labels). 49 tests across 7 suites.
• .github/workflows/ci.yml — added broker (tools/demarkus-broker/** + tools/go.mod + tools/go.sum + protocol/**) and charts (deploy/helm/**) paths to the change-detection filter. New test-broker job: Go test/vet/lint/build for tools/demarkus-broker/ with working-directory: tools/demarkus-broker (narrowed past the broader tools/ scope to avoid lint failures on tools/tools.go). New test-charts job: helm lint on all three charts + helm unittest on server + broker. Both new jobs timeout-minutes: 20. helm pinned to v3.13.2; helm-unittest pinned to v0.6.2 (compatible across the local helm version).
• deploy/helm/demarkus-server/tests/statefulset_test.yaml — fixed real prior-work bug. The existing suite only loaded statefulset.yaml, so the StatefulSet template's include "...tokens.yaml" for the checksum/config annotation failed render and all 13 tests errored out. Added tokens.yaml to the suite templates list + template: statefulset.yaml per-test to scope assertions. Nobody had noticed because helm-unittest never ran in CI. The §6.1 chart-test claims in plan v13 weren't actually executing — until now.

Decisions / observations

• Fold-in of test-broker was the right call. The handoff's §6.3.D scope proposed adding helm-unittest to CI + flagged separately that the §6.2 broker binary had zero CI coverage. Folding the binary-CI job into the same PR meant the broker chart shipped paired with a binary that finally has real CI gates. Without the fold-in we'd have had the absurdity of testing the chart's behavior in CI while the binary it wraps remained CI-uncovered.
• helm-unittest v0.6.2 over v1.x. v1.0+ uses platformHooks in plugin.yaml which needs helm v3.16+; Fritz's local helm is v3.13.2. Pinning v0.6.2 in CI (matching the local-dev validation surface) keeps the test behavior identical between developer machines and CI. Bump together later when helm itself bumps.
• Per-test template: scope is required when one template includes another. The shared checksum/config: {{ include "...secret-config.yaml" }} pattern (and the §6.1 chart's analogous tokens.yaml include) means tests for the parent template must list BOTH templates in the suite templates: filter, and then narrow each test's assertion target via the singular template: field. Discovered while debugging the broker deployment_test.yaml; the same fix retroactively rescued the §6.1 statefulset suite.

CodeRabbit comments addressed (all five, no pushback)

1. Broker path filter narrowing — tools/** → tools/demarkus-broker/** + tools/go.mod + tools/go.sum. CodeRabbit's static analysis confirmed tools/tools.go sits outside the broker package; the broad filter would trigger broker CI on unrelated tool changes.
2. Job timeouts — timeout-minutes: 20 on test-broker and test-charts. Default GitHub Actions timeout (360min) is too lenient for these jobs.
3. golangci-lint scope — working-directory: tools → tools/demarkus-broker. Same reasoning as #1; verified locally that golangci-lint walks up to tools/go.mod for module resolution.
4. CLAUDE.md "Records" fragment removed — line 38 had a malformed header ("Records (one per decision, zero-padded 4-digit sequence)") followed by a sub-bullet describing journals not records. Collapsed into a single journal entry matching the surrounding flat-list style. Surfaced to Fritz that a separate ADR-style Records concept can be added back if intended.
5. CLAUDE.md journal-path contradiction (out-of-diff) — line 27 said /journal.md, line 39 said /<project>/journal/<YYYY-MM-DD>.md. Updated line 27 to match the dated convention. Caveat flagged: this made CLAUDE.md aspirational ahead of the soul, which still had the legacy /journal.md shape at PR-merge time. Migration to dated files happens in this very session, immediately after PR merge.

Prior-work debt surfaced (still open, sidecar PR §6.3.D.2 scope)

• §6.1 demarkus-server tokens.yaml Secret missing helm.sh/resource-policy: keep. Plan v13 §6.1 §Status claims the annotation is set and a kind integration test pins the upgrade-wipe regression. Reality at PR-merge time: the annotation is NOT on the rendered Secret and no kind test exists in CI. Real bug in shipped §6.1 chart; bundle the fix with the §6.3.D.2 kind harness so the regression test catches it directly.
• §6.2 broker binary had zero CI coverage at the start of this session. Closed by the test-broker job in this PR; surfacing it here so the gap doesn't get lost in the diff.
• helm-unittest wasn't running anywhere. The §6.1 chart's existing test files weren't executed by CI, AND the statefulset_test.yaml suite was structurally broken from the day it was written. Both fixed in this PR. Lesson: chart-test files in the repo don't equal CI test coverage of charts; the workflow plumbing matters at least as much as the test files.

Next: §6.3.D.2 sidecar — kind upgrade-wipe regression for both charts, bundled with the §6.1 helm.sh/resource-policy: keep + helm.sh/hook: pre-install fix on demarkus-server/templates/tokens.yaml. Test shape: install chart, kubectl-patch the rendered tokens/issuances Secret to seed fake data, helm upgrade with a values change, assert seeded data persists. Broker pod doesn't need to be ready for the property to hold — kind harness can skip OIDC discovery via a values stub. Plan bump to v14 after §6.3.D.2 lands so v14 captures coherent §6.3-complete landed-state.

2026-05-13 — Journal migrated to dated per-day files

Operational note. Until this session, all session notes lived in a single growing /journal.md file (v57 at migration time, ~140KB / 51 entries across 23 unique dates). PR #118 merged a CLAUDE.md edit committing to a /journal/<YYYY-MM-DD>.md per-day convention; this entry is the first written under that convention.

Migration mechanic: the legacy /journal.md body was parsed by ## YYYY-MM-DD — heading boundaries, grouped by date (multiple entries on the same date stayed in one file as sequential ## sections), and published as /journal/<YYYY-MM-DD>.md for each unique date. Same content, different topology. The old /journal.md was kept (mark_archive not used — the version history at the legacy path is still readable) and /index.md updated to point at the new directory.

Why per-day: a 140KB single file was bumping fetch-result size limits (had to be saved to file rather than returned inline; agents had to read the saved file in chunks). Per-day files cap at ~40KB worst case (2026-05-12 was the longest day, with 5 entries totaling 38KB) and most days are well under 10KB. Future fetches by date go straight to the right file rather than scrolling a 51-entry catalog.

Convention going forward: mark_append to the current date's file when adding a session note. If the file doesn't exist yet, mark_publish it. The CLAUDE.md "End of Session" line points operators (and agents) at the right path.

2026-05-13 — Slice 6.3.D.2 merged: §6.1 server chart resource-policy fix + kind upgrade-wipe regression

PR #119 merged to main as 3d3b4bc. Sidecar to §6.3.D.1; closes the §6.1 prior-work debt that earlier plan versions claimed was already in place. Lands the helm.sh/resource-policy: keep annotation on the demarkus-server tokens Secret AND an end-to-end kind regression that proves the upgrade-wipe property — for BOTH charts — rather than re-asserting it in prose. With this, all five §6.3 sub-slices (A/B/C/D.1/D.2) are merged on 2026-05-13 and §6.3 broker chart work is fully complete.

What landed

• Three-branch hybrid template at deploy/helm/demarkus-server/templates/tokens.yaml keyed on the live Secret's state:
  1. First install (lookup returns nil) — render fresh content with helm.sh/resource-policy: keep. $rawToken resolves to a freshly-generated admin token; both the tokens Secret and the token-values Secret ship with consistent raw+hash.
  2. Legacy upgrade (Secret exists but LACKS the keep annotation) — render the EXISTING data verbatim plus the keep annotation. Migration path from §6.1's original template (PR #107) which shipped without the annotation; without this branch, the §6.1.D.2 lookup-skip branch would emit nothing, helm would diff old-annotation-less against new-absent, and DELETE the Secret on apply, wiping every broker-minted user-token entry along the way. Re-rendering existing data makes helm's apply a content no-op while the annotation lands so future upgrades flip to branch 3.
  3. Race-free no-op (Secret exists AND has the keep annotation) — emit nothing. helm sees the Secret removed from the rendered manifest, would normally GC it, but the live annotation blocks deletion. After the first migration, upgrades NEVER touch the Secret content, leaving the broker as the sole writer and eliminating any race window between helm template and helm apply.
• deploy/helm/test-upgrade-wipe.sh — 142-line generic kind upgrade-wipe regression. Usage: install chart with values stub, kubectl-patch the rendered Secret to seed fake data, helm upgrade with a values change, assert seeded data persists, uninstall, assert Secret survives uninstall under the keep annotation. SIMULATE_LEGACY_NO_KEEP env flag exercises the migration path by stripping the annotation from the live Secret between install and upgrade — the test then asserts the legacy-upgrade branch re-applies the annotation AND preserves existing data.
• deploy/helm/demarkus-server/tests/tokens_test.yaml — three new assertions added (10 total tests in the file now): tokens Secret carries helm.sh/resource-policy: keep, token-values Secret carries the same, and the rendered Secret has explicit metadata.namespace so the lookup-skip branch checks the right namespace on upgrade (rather than helm's release-namespace default which silently misaligns when an operator deploys with --namespace).
• .github/workflows/ci.yml — new test-charts-kind job (47 lines). Spins up a kind cluster, runs test-upgrade-wipe.sh three times: server normal mode + server SIMULATE_LEGACY_NO_KEEP mode + broker normal mode. Broker pod does NOT need to be ready (OIDC discovery is irrelevant to the upgrade-wipe property); the harness uses a values stub. Job runs only on charts-path changes.

Decisions

• Three-branch hybrid over "always render" (Option A). Option A keeps helm "managing" the Secret on every upgrade — helm template renders content (from lookup), helm apply writes it. That opens a race window: between helm's lookup snapshot and the apply, the broker is concurrently writing via mutateSecret (resourceVersion optimistic concurrency). Apply uses last-writer-wins on fields not under strategic-merge-patch, so a concurrent broker write between lookup and apply gets clobbered. The race-free branch 3 of the hybrid sidesteps this entirely: helm emits nothing, the live keep annotation blocks GC, broker stays the sole writer. The cost is a one-time legacy branch for migration; once a release passes through branch 2, all subsequent upgrades hit branch 3.
• SIMULATE_LEGACY_NO_KEEP env flag rather than a separate script. Same regression script exercises both the normal three-branch path AND the migration path. Two scripts would have drifted in the assertion shape; one script with a flag keeps the property-under-test identical across modes.
• CI runs three matrix entries, not nine. Server normal + server legacy + broker normal covers the property. Broker doesn't have a legacy state to migrate from (it shipped with the keep annotation from §6.3.A day one), so a broker legacy mode would test nothing. Agent chart doesn't have a stateful Secret to wipe — out of scope.

CodeRabbit rounds

Two rounds during PR #119 review.

• Round 1 caught two critical issues. (a) The original draft of tokens.yaml had only two branches (lookup-skip + first-install) — it would have wiped Secrets on the FIRST upgrade of any pre-D.2 release because legacy Secrets had no keep annotation, so lookup-skip would emit nothing, helm would diff against the missing-from-rendered-manifest state, and delete. The three-branch hybrid was the response: branch 2 catches the legacy state, re-renders existing data verbatim plus the annotation, then future upgrades land on branch 3. (b) The original test script didn't exercise the legacy-upgrade path at all — only the first-install + race-free no-op. SIMULATE_LEGACY_NO_KEEP was the response: identical script with an env flag that strips the annotation between install and upgrade to simulate a pre-D.2 release's state, then asserts the legacy branch re-applies the annotation while preserving data.
• Round 2 caught a kubectl jsonpath issue. The original script used dot-notation for the data field lookup: kubectl get secret ... -o jsonpath="{.data.tokens-toml-b64}". The hyphenated key happens to work on kubectl's permissive parser, but isn't spec-conformant — a stricter k8s version or a downstream tool reading the same expression could break silently. Switched to bracket form: {.data['tokens-toml-b64']}. Empirically the dot form was passing; the bracket form is defensive against the permissive-parser regression.

Test surface

• deploy/helm/demarkus-server/tests/tokens_test.yaml: 10 tests total, 3 new (keep annotation × 2 Secrets + explicit namespace).
• deploy/helm/test-upgrade-wipe.sh: end-to-end kind cluster + helm install/upgrade/uninstall + Secret data assertion. Doesn't add test counts; adds a property that prose can't carry.
• test-charts-kind CI job: three matrix entries (server normal + server legacy + broker normal).

Resolved risks

• The §6.1 chart upgrade-wipe gap that plan v14 carried as an open risk (the prose claim of "resource-policy: keep is in place" vs. the reality of the missing annotation) is now resolved — the kind regression pins the property end-to-end so the prose is no longer load-bearing. Plan v15 risk entry can be removed or downgraded to "resolved (pinned by kind regression)".
• The §6.1 chart-test plumbing latent failure surfaced in §6.3.D.1 is also closed end-to-end: the unittest suites run in CI AND the kind harness exercises the live-cluster behavior the unittests can't reach (lookup against an actual Secret, helm upgrade flow, uninstall).

Next

§6.3 fully complete. Phase 6 remaining sub-phases: §6.4 universe topology examples (ApplicationSet + Kustomize overlay), §6.5 observability recipes (Datadog / OTel Collector / Vector / Fluent Bit / Grafana Alloy), §6.6 documentation suite (install / security / operations / observability / per-chart READMEs), §6.7 release pipeline (GHCR images + OCI charts, multi-binary server image, CLI relocation to tools/). Open question for Fritz: order of attack. §6.7 is the most-unblocking — without it, §6.4 examples and §6.6 docs both reference not-yet-released artifacts. §6.5 is independent of §6.7. Likely sequence: §6.7 → §6.5 (parallel-able) → §6.6 incremental → §6.4 last. Waiting on Fritz to pick.

2026-05-13 — §6.7.0 merged: hoist server/internal/store → protocol/store

PR #120 merged to main as b210015. Precursor refactor to §6.7.A; no behavior change, no new feature. Purely a namespace move so demarkus-publish (and any future tool that needs versioned content storage) can consume the store package without server/ becoming a heavyweight dependency.

What landed

• server/internal/store/store.go → protocol/store/store.go (git mv).
• server/internal/store/store_test.go → protocol/store/store_test.go (git mv).
• Import paths rewritten in four files: server/internal/handler/handler.go, server/internal/handler/handler_test.go, server/cmd/demarkus-server/main.go, server/cmd/demarkus-publish/main.go. goimports reordered the import blocks alphabetically (cosmetic).

Why protocol/store/ over alternatives. The package only depends on stdlib + protocol; it has no QUIC, no handlers, no auth — just versioned-content disk-storage primitives. protocol/ already owns token/ (the disk-shape of capability tokens) and auth.go (the HashToken contract). The store fits the same conceptual layer. Hoisting to a new top-level module (store/) would have over-engineered for an abstraction that doesn't span beyond protocol's scope. Hoisting to server/store/ (out of internal/) would have left tools/ importing the heavy server/ module just to access disk primitives — wrong dependency direction.

Verification. protocol, server, client, tools all tested green; pre-commit.sh format/vet/lint clean across all four modules.

Lesson. Earlier handoff prose claimed hoisting store was "non-trivial — gets imported by 8+ files in the server module." Actual reach was four files, all import-path-only rewrites (no API changes). The non-trivial framing was an overstatement; the move is a 5-minute refactor when scoped correctly. Useful calibration for future "is this load-bearing" judgment calls.

2026-05-13 — §6.7.A merged: split per-service container images + CLI relocation

PR #121 merged to main as aa9a3c6. Closes the §6.1 caveat that has been carried since PR #107 (Phase 6.1 server chart shipped with exec probes referencing /demarkus, but the server image only contained /demarkus-server). The chart was YAML-correct from day one; the image side finally catches up. Two-hour rework arc inside the PR: started as a unified single-image bundle, pivoted to per-service split images after an architectural-fit discussion with Fritz.

What landed

• Three per-service container images, each ENTRYPOINTed to its primary binary:
  • ghcr.io/latebit-io/demarkus-server (server/Dockerfile) — bundles demarkus-server + demarkus (CLI for exec probes). 14MB.
  • ghcr.io/latebit-io/demarkus-broker (tools/demarkus-broker/Dockerfile) — single binary; HTTP probes; CA bundle for outbound OIDC. 25MB.
  • ghcr.io/latebit-io/demarkus-agent (client/cmd/demarkus-agent/Dockerfile) — single binary; CA bundle for outbound mark:// TLS. 7MB.
• Multi-stage Dockerfiles, each with build context = repo root so cross-module source is reachable (server image's CLI lives in client/; broker pulls protocol/; agent pulls protocol/).
• /etc/ssl/certs/ca-certificates.crt copied from build stage into broker + agent runtime images. Server skips it — no outbound TLS today; bundled CLI is for localhost probes with -insecure.
• USER 65532:65532 in all three runtime stages matching the charts' podSecurityContext.runAsUser. Defense-in-depth so the image runs non-root even outside k8s.
• go build -C <module> instead of RUN cd <module> && go build across all three Dockerfiles. Idiomatic Go tooling (1.20+ -C flag), removes the RUN cd pattern Trivy flags as DS-0013.
• CLI relocation to tools/: server/cmd/demarkus-token → tools/demarkus-token/, server/cmd/demarkus-publish → tools/demarkus-publish/. demarkus-token was a clean move (only imports protocol/token); demarkus-publish became clean after §6.7.0's protocol/store hoist landed.
• server/Dockerfile deleted and recreated at the same path with the new multi-stage shape. Previous file was 4 lines of COPY demarkus-server / + ENTRYPOINT and depended on goreleaser-built binaries in the build context.
• server/.goreleaser.yml slimmed — removed demarkus-token + demarkus-publish build entries (they live in tools/ now) and removed the dockers: + docker_manifests: blocks (the new image build lives at the per-component Dockerfiles, wiring into release.yml is §6.7.B scope).
• Makefile image target with sub-targets image-server, image-broker, image-agent. make image TAG=dev builds all three locally.
• Chart values reverted to per-component image refs (each chart's image.repository points at its own image; no command: field — relies on each image's ENTRYPOINT).
• Chart unittests updated: server +1 assertion (image regex + notExists command), broker +1 (same shape on broker tag), agent +1 (same shape on agent tag). Fixed a pre-existing structural bug in deploy/helm/demarkus-agent/tests/deployment_test.yaml — same template: scoping issue as §6.3.D.1 caught in the server's statefulset suite; agent's 10 deployment tests had been silently erroring because configmap.yaml + secret.yaml were missing from the suite templates list.

Decisions

• Per-service split images over unified single-image bundle. Initial implementation packaged all six binaries (server, broker, agent, CLI, token, publish) into one ghcr.io/latebit-io/demarkus image; each chart picked its binary via the pod's command: field. Worked end-to-end (one image, 52.8MB) but Fritz pushed back on the abstraction shape: demarkus is closer to Kubernetes (multi-component product — server speaks QUIC, broker speaks HTTP-OIDC, agent is a crawler) than Vault (single-purpose product). Multi-component products idiomatically split images so each pod's CVE blast radius covers only its own dependency tree; a CVE in broker's OIDC library shouldn't fail a server pod's security scan. Split images: 48MB total across 3 images, each pod pulls only ~10-26MB. Cost: 3 CI builds vs 1; minimal Go-static-binary footprint makes this negligible. The split is also reversible later if a customer requests a single-image distribution.
• Admin CLIs (demarkus-token, demarkus-publish) NOT bundled in any runtime image. They ship as standalone binaries via the existing goreleaser archives. Operators run them locally (port-forward) or in dedicated Job pods with the right PVC mount, not via kubectl exec into a long-running pod. The principle: runtime pods carry only what they need to serve their primary role. "I want demarkus-token for kubectl exec convenience" is a leaked abstraction — admin tooling lives outside the pod blast radius.
• CA bundle is selective. Broker (calls OIDC issuer discovery/token endpoints over HTTPS) and agent (crawls remote mark:// servers with insecure: false in production) NEED /etc/ssl/certs/ca-certificates.crt to validate certs. Server doesn't — it's the QUIC listener; bundled CLI exec probes use -insecure for localhost. Adding CA bundle to server "just in case" would leak "this image makes outbound TLS calls" into its abstraction; if needed later, it's a one-line add.
• No HEALTHCHECK in Dockerfiles (Checkov CKV_DOCKER_2 nit). K8s livenessProbe/readinessProbe are authoritative for deployed pods. HEALTHCHECK in a Dockerfile is silently ignored by k8s and would be a confusing artifact for anyone reading the image expecting it to be the source of truth. Skipped.
• No SHA256 digest pinning on golang:1.26-alpine (CodeRabbit 🔵 trivial nit). Floating the minor tag lets us pick up Go stdlib security patches automatically; pinning to a digest locks all three Dockerfiles to whatever 1.26.x was current at pin time and requires coordinated manual bumps on every Go patch release. Without renovate/dependabot wired to track base-image digests, pinning trades reproducibility for a maintenance hazard. Natural follow-up when base-image automation arrives alongside module-bump automation.

CodeRabbit rounds (3)

• Round 1 (commit 080e54f, unified-image draft): one comment, 🛠️ Refactor / 🟠 Major — "Set explicit non-root USER in the final image." Valid concern. Initial draft's unified Dockerfile defaulted to root in the scratch stage. Applied to all three Dockerfiles after the split: USER 65532:65532 matching chart values.
• Round 2 (commit 4af56e6, split-image draft): two comments, both ⚠️ Potential / 🟠 Major — "Missing CA trust store in scratch runtime breaks HTTPS/OIDC egress" (broker) and "Scratch runtime is missing CA certificates for outbound TLS" (agent). Both correct, both fixed by copying /etc/ssl/certs/ca-certificates.crt from the build stage. Server Dockerfile was NOT flagged (correctly — server has no outbound TLS).
• Round 3 (commit 3c2ea8e, ENTRYPOINT/CA hardening): one comment, ⚠️ Potential / 🔴 Critical — "Build output path can fail because /out is never created." False alarm — Go 1.26 go build -o /out/binary auto-creates the parent directory. Verified with a clean container test (mkdir-less, exit 0, /out materializes). Trivy's nested DS-0013 warning (RUN cd shouldn't change directory) had merit, addressed by switching to go build -C <module> instead of RUN cd <module> && go build. CodeRabbit's mkdir -p /out suggestion explicitly skipped — defensive code for a non-issue.
• Round 4 (commit dffec77, go build -C fix): one comment, 🧹 Nitpick / 🔵 Trivial — "Pin golang image to immutable digest for reproducibility." Skipped per the §Decisions rationale above; replied on the PR explaining the trade.

Useful learnings

• CodeRabbit "Critical" severity isn't always accurate. The /out directory comment was marked 🔴 Critical despite being a false alarm in modern Go. Worth running a focused test against a base-image-clean container before accepting/skipping these — same principle as treating any "you should fix this" advice from any tool: verify the premise, then act.
• go build -C <module> is the idiomatic way to build from sibling modules in a monorepo. It's been in Go since 1.20; cleaner than cd module && go build in Docker RUN statements and addresses Trivy DS-0013 simultaneously.
• Three-image build cost: ~3min for all images cold, ~30s incremental. Each Dockerfile reuses the alpine base layer + Go module cache between builds. Layer caching makes the per-component split fast in practice; the conceptual cleanliness is the dominant win.

Test surface

• Server chart: 38 → 37 (consolidated a test by merging assertions, no coverage loss).
• Broker chart: 50 → 49 (same merge consolidation).
• Agent chart: 22 → 21 (same, plus the agent's pre-existing templates: scoping bug fix rescued 10 tests that had been silently erroring).
• Total: 107 helm-unittest assertions across 16 suites. All green.
• Kind smoke: server pod went Ready in 19s as UID 65532 from the split server image; exec probe (/demarkus -insecure -no-cache mark://localhost:6309/.well-known/agent-manifest.md) succeeded against the bundled CLI. The §6.1 probes-will-fail-until-6.7 caveat is now empirically closed.

Next: §6.7.B — wire per-image docker build + docker push ghcr.io/latebit-io/<service>:<tag> into .github/workflows/release.yml; add tools/.goreleaser.yml for demarkus-broker + demarkus-token + demarkus-publish binary archives; add OCI chart publish via helm push oci://ghcr.io/latebit-io/charts/<chart> for all three charts. Cosign signing stays backlogged. Transition window note: until §6.7.B lands, charts' default image.repository references images that don't yet exist on GHCR — operators must build locally via make image or override image.repository/image.tag at install time.

2026-05-13 — §6.7.B release pipeline complete (3 PRs: per-image release, tools release, OCI chart publish)

Three sub-PRs merged in same-day succession to close out Phase 6.7. Total ~200 lines of .github/workflows/release.yml additions plus a new tools/.goreleaser.yml. End-to-end the three runtime images and three OCI Helm charts all have a release path tied to their corresponding module-version tag.

PR #122 (§6.7.B.1, commit 0e1a4ae) — image release on existing tag flow. Added docker/build-push-action@v6 steps to the existing release-server and release-client jobs. release-server was already set up with qemu+buildx+login from the (now-removed) goreleaser dockers: block; just plugged in build-push-action against server/Dockerfile. release-client had none of the docker setup actions — added qemu+buildx+login plus the build-push step for client/cmd/demarkus-agent/Dockerfile. Both build multi-arch (linux/amd64 + arm64 + arm/v7) and push to ghcr.io/latebit-io/demarkus-{server,agent}:${version} + :latest.

PR #123 (§6.7.B.2, commit afb6155) — tools/v* release flow. New tools/.goreleaser.yml mirroring server/.goreleaser.yml's shape with three independent builds (demarkus-broker, demarkus-token, demarkus-publish) producing three independent archives. New workflow plumbing: tools filter on detect-changes, test-tools job, semver-tools (PaulHatch with tag_prefix=tools/v, change_path=tools/), release-tools that creates tag + runs goreleaser + builds-and-pushes the broker image via tools/demarkus-broker/Dockerfile. Admin CLIs (token, publish) ship as archive-only — no runtime image — per the §6.7.A "minimal pod blast radius" decision.

PR #124 (§6.7.B.3, commit 9cf6303) — OCI chart publish. Added azure/setup-helm@v4 + helm package + helm push steps at the end of each of the three release jobs. Each helm package --version $VER --app-version $VER overrides both Chart.yaml fields to the just-released module version, so a chart pulled from the registry references an image at the same tag (image.tag | default .Chart.AppVersion). helm push oci://ghcr.io/latebit-io/charts/<chart> relies on the ~/.docker/config.json written by docker/login-action — helm 3.13+ reads it natively for OCI registry auth; no separate helm registry login step needed.

Mapping (chart ↔ image ↔ module tag):

• demarkus-server chart + demarkus-server image ← server/v* tag
• demarkus-agent chart + demarkus-agent image ← client/v* tag (agent lives in client/cmd/)
• demarkus-broker chart + demarkus-broker image ← tools/v* tag

Decisions

• docker/build-push-action@v6 over re-introducing goreleaser's dockers: block (§6.7.B.1). The current multi-stage Dockerfiles compile from source via Go's cross-compilation under buildx, not by copying goreleaser-built binaries. Wiring image release through build-push-action keeps the Dockerfiles unchanged and portable (anyone can docker build locally and get the same image without needing goreleaser-pro). Cost: binaries compile twice on release (once for archives, once inside image build) — Go is fast, ~30s extra per platform, negligible.
• Chart-version + appVersion pinned 1:1 to module version on release (§6.7.B.3). Coupled lifecycle: chart-of-version-X always references image-of-version-X. Independent chart versioning would be useful for chart-only fixes (broken template, NOTES.txt edit), but adds tag-scheme complexity (charts/<name>/v* per chart) for a use case we don't have evidence of yet. Revisit if it materializes.
• No -X main.version=... stamping in tools/.goreleaser.yml (§6.7.B.2). Matches the existing pattern in server/.goreleaser.yml + client/.goreleaser.yml — neither stamps version via goreleaser. Pre-existing oversight (the agent's var version = "dev" is wired in the source but never overridden at release time). Fixing this is a separate cleanup; not in §6.7.B scope.
• tag_prefix: tools/v + change_path: tools/ (§6.7.B.2 semver-tools job). Mirrors the server/client semver jobs exactly — tools/ is now a peer module in the release-pipeline taxonomy. The PaulHatch/semantic-version action handles auto-bumping based on conventional commit prefixes in commit subjects + bodies on paths under tools/.

Verification

• All three workflow YAMLs parse via Ruby's YAML loader at each branch handoff.
• Single-platform docker buildx build against each Dockerfile with --build-arg VERSION=test-X.Y.Z confirmed each builds cleanly.
• Multi-arch builds (linux/amd64,linux/arm64 via docker-container driver — same as setup-buildx-action@v3 in CI) verified for server and agent Dockerfiles. Broker Dockerfile is structurally identical to agent; skipped local multi-arch smoke after the user-interrupt on the QEMU-emulated build (Fritz asked what was taking so long).
• helm package --version 9.9.9 --app-version 9.9.9 against all three charts confirmed produces a tarball whose extracted Chart.yaml has both fields stamped to 9.9.9. Push not exercised locally (would require ghcr.io push perms).

Closes

• The §6.7.A transition window (charts referenced non-existent images on GHCR) is now fully closed — next module-version tag fire on any of server/client/tools pushes both the image and the chart.
• The §6.7 sub-phase as a whole. Phase 6 is now 80% complete by sub-phase: §6.0/§6.1/§6.2/§6.3/§6.7 done; §6.4/§6.5/§6.6 remaining.

Useful learnings

• PaulHatch/semantic-version + change_path is monorepo-friendly. Each module gets its own tag-prefix-and-change-path pair, so independent bumps don't cross-contaminate. The bump_each_commit: false flag keeps the version stable until a feat/BREAKING CHANGE commit on that path actually warrants a bump.
• docker/login-action writes a docker config that helm 3.13+ trusts for OCI auth. No additional helm registry login step needed in CI when the docker login step is already there. This saved a redundant credential-handling layer.
• GitHub Actions YAML supports comments densely. Each new step in the release jobs gets a doc-comment explaining WHY (e.g., "build context = repo root because the agent module's tools/ replace directive needs protocol/ alongside"). The release.yml is now self-documenting for any future maintainer adding a fourth module to the release pipeline.

Next

§6.5 / §6.6 / §6.4 remain. §6.5 (observability recipes) is independent of any release-pipeline artifact — pure config files for Datadog / OTel Collector / Vector / Fluent Bit / Grafana Alloy mapping demarkus's existing structured slog output to each backend. §6.6 (docs) and §6.4 (topology examples) can start referencing concrete chart pull URLs now that §6.7.B closed the gap. Two-week deadline at 2026-05-27 with ~11 days remaining.