# Journal — 2026-05-13 ## 2026-05-13 — Slice 6.3.A merged: demarkus-broker chart skeleton + security defaults PR #115 merged to main as `c71f499`. First of four §6.3 sub-PRs. Lays the chart skeleton: Deployment + Service + config Secret + issuances Secret + ServiceAccount + values schema. Templates for RBAC, NetworkPolicy, PDB, Ingress, and the cert-manager Certificate ship in later slices but the values schema is declared up front so subsequent slices add templates without churning operator-facing values.yaml. **What landed** - **`deploy/helm/demarkus-broker/`** — Chart.yaml + values.yaml + 7 starting templates (`deployment.yaml`, `service.yaml`, `serviceaccount.yaml`, `secret-config.yaml`, `secret-issuances.yaml`, `NOTES.txt`, `_helpers.tpl`). - **Cookie-key resolution helper** (`demarkus-broker.resolveCookieKey` in `_helpers.tpl`). Precedence: explicit `server.cookieKey` → `lookup` of the live config Secret's `cookie-key` data field → freshly-generated 32-byte base64. The lookup branch is what keeps the HMAC key (and therefore every in-flight state cookie) stable across `helm upgrade`. The same `$cookieKey` variable feeds both the standalone `cookie-key` data field AND the `cookieKey:` field embedded inside `config.yaml` in the rendered Secret — single value, single render pass, never drifts. - **Issuances Secret seed** (`secret-issuances.yaml`). Renders an empty `data: {}` Secret on first install with `helm.sh/resource-policy: keep` on the metadata. Subsequent installs/upgrades hit the `lookup`-skip branch so helm sees no rendered resource and the keep-annotation on the LIVE Secret prevents helm GC. Net effect: chart updates never touch broker-minted issuance state. - **OIDC client-secret schema, mutual-exclusion guarded at render-time.** `oidc.clientSecret` (cleartext via values) and `oidc.existingSecretRef.name` (operator-managed Secret) are mutually exclusive. Setting both fails `helm template` with a clear message; setting neither fails. Render-time fail-fast rather than pod-startup `CreateContainerConfigError` is the whole point — the binary's `applyEnvOverrides` env-var path arrives in §6.3.C, so 6.3.A ships the cleartext path only but pins the mutual-exclusion guard now. - **PodSecurityContext + container securityContext lockdown.** `runAsNonRoot: true`, `runAsUser: 65532`, `runAsGroup: 65532`, `fsGroup: 65532`, `seccompProfile: RuntimeDefault`, `readOnlyRootFilesystem: true`, `allowPrivilegeEscalation: false`, `capabilities: drop ["ALL"]`. Five hardening defaults pinned in values.yaml; a "convenience" relaxation in any one of them should surface as a test failure in the §6.3.D helm-unittest sweep. - **TopologySpreadConstraints by `kubernetes.io/hostname`, `whenUnsatisfiable: ScheduleAnyway`.** Two replicas spread across nodes when possible; single-node clusters still schedule. The empty `matchLabels: {}` in values.yaml is intentional — the deployment template overrides it with `selectorLabels` so the constraint always targets the right pods even if a future operator mistakes the empty default for "fill these in." Easy to break in a refactor; worth pinning in §6.3.D. - **`checksum/config` pod annotation** so a config Secret change triggers a rollout. Without it, broker pods stale-mount after a config update. - **`rateLimit.trustForwardedFor: true` chart default**, inverting the binary's `false` default (C.4 §Decisions). The chart's deployment assumption is "behind an Ingress that strips spoofed X-Forwarded-For"; flipping the binary default at the chart level keeps the per-IP `/auth/login` limiter effective. Operator-facing docs in the chart README call out the failure mode (flipping the flag in a directly-internet-exposed broker). **Decisions that landed** - **Explicit `metadata.namespace` on both config and issuances Secrets.** CodeRabbit flagged that omitting it lets helm default to the release namespace silently — fine for the default case, surprising when an operator sets `server.brokerNamespace` to put the issuances Secret elsewhere. Both Secrets now resolve their namespace through the `demarkus-broker.brokerNamespace` helper (which falls back to `.Release.Namespace`), so lookup + render always agree. - **`worldAllows` nil-`.allow` template guard.** CodeRabbit found the `secret-config.yaml` `range .Values.worlds` block would panic when a world omitted the `allow:` block entirely (the production-meaningful "all empty = open world" predicate path from C.1). Wrapped with `{{- $allow := default (dict) .allow }}` so `.allow.domains` etc. each get `default (list)` fallbacks. Pre-fix the chart aborted; post-fix the rendered config emits empty arrays for all three dimensions and the binary's `worldAllows` predicate takes the back-compat all-empty match. - **`lookup`-skip namespace consistency.** The issuances Secret's `lookup` call uses the same `brokerNamespace` helper as the rendered metadata, so operators that override `server.brokerNamespace` get the lookup hitting the correct namespace and the keep-annotation surviving across upgrades. CodeRabbit fix. **Next: §6.3.B** — RBAC (per-world Roles + broker-namespace Role for sweeper Lease + issuances Secret), NetworkPolicy default-on, PodDisruptionBudget `minAvailable: 1`. Schema declared in A; templates ship in B. ## 2026-05-13 — Slice 6.3.B merged: per-world RBAC, broker-namespace RBAC, NetworkPolicy, PodDisruptionBudget PR #116 merged to main as `175e230`. Second of four §6.3 sub-PRs. Adds the RBAC + NetworkPolicy + PDB templates whose values schema 6.3.A pre-declared. Two new template files for RBAC (broker-ns and per-world) plus one each for NetworkPolicy and PDB. **What landed** - **`rbac-worlds.yaml`** — per-world Role + RoleBinding loop. `range .Values.worlds` emits N×2 resources, each Role scoped to a single `resourceNames: []` with verbs `get/update` (NOT `create`). The world's tokens Secret must already exist from the §6.1 server chart's bootstrap Job; the broker only needs to read+patch, never create. If a world's Secret is missing at first mint the broker surfaces a clear "Secret not found" error rather than silently bootstrapping into a namespace it shouldn't own. - **`rbac-broker-ns.yaml`** — broker-namespace Role + RoleBinding. Two rules: `coordination.k8s.io/leases` `get/create/update` scoped to `resourceNames: []` (the C.2 leader-election Lease), and `secrets` `get/update` scoped to `resourceNames: []`. The leases rule includes `create` so first-leadership-acquisition can seed the Lease — the k8s RBAC caveat "create cannot be restricted by resourceName" means create effectively allows any Lease in the namespace, acceptable scope since the namespace is broker-owned. The doc-comment in the template spells this out so a future tightening to just `update`+`get` doesn't break first-install silently. - **`networkpolicy.yaml`** — default-on (`networkPolicy.enabled: true`). Ingress allows traffic only from the configured `ingressFromNamespace` (default `ingress-nginx`) on the server port via the automatic `kubernetes.io/metadata.name` label (k8s 1.21+). Egress allows DNS to `kube-system` (UDP+TCP 53) and unrestricted TCP 443 — the latter intentionally NOT pinned to apiserver/OIDC issuer CIDRs because pinning breaks across managed-k8s providers and every IdP frontend rotation. Doc-comment notes the failure mode for self-hosted OIDC on non-443 (e.g. Keycloak) and older clusters lacking the automatic namespace label. - **`poddisruptionbudget.yaml`** — default `minAvailable: 1`. With two replicas and PDB-1, a single voluntary disruption (node drain, helm upgrade rollout) leaves the broker servable. Cheap insurance against in-cluster maintenance churn. - **`serviceAccount.workloadIdentity.gsa`** values plumbing into the ServiceAccount annotations. Off by default; setting it stamps `iam.gke.io/gcp-service-account` on the SA so GKE Workload Identity bindings can map broker→GSA without static keys. Validated values-shape only; integration is operator-side. **Decisions that landed** - **`rbac.create: true` as the default** so opinionated installs work out of the box; flippable to false for clusters that manage RBAC out-of-band (Argo CD AppProject, OPA, GitOps Roles). Same shape as `serviceAccount.create: true`. - **Verbs `get/update` on world tokens Secrets, NOT `create`.** Pinning the minimal set keeps the per-world Role aligned with what the broker actually does (read-modify-write via `mutateSecret`'s `resourceVersion` retry). A future refactor that adds `create` should land with a test failure so the question "do we want the broker bootstrapping world Secrets?" gets re-asked deliberately. §6.3.D helm-unittest will pin this. - **NetworkPolicy egress unrestricted on TCP 443 by design.** Restricting it to specific CIDRs trades the egress-blast-radius gain for an upgrade-and-multi-cluster brittleness that's worse in practice (apiserver IP changes per cluster; IdP frontend rotation is on-call paging territory). The default-on ingress namespace scope is the dominant security gain; egress 443 wide-open is the operator-friendly tradeoff. Documented in the doc-comment. **Next: §6.3.C** — Ingress + cert-manager Certificate + `oidc.existingSecretRef` plumbing via a new `OIDC_CLIENT_SECRET` env-var override path in `broker.LoadConfig.applyEnvOverrides`. The env var is a small additive surface on §6.2's "broker binary complete" framing — explicitly a one-line escape hatch for the production-secret-ref deployment shape, not a feature-set expansion. ## 2026-05-13 — Slice 6.3.C merged: Ingress + cert-manager Certificate + external OIDC secrets PR #117 merged to main as `599f63a`. Third of four §6.3 sub-PRs. Adds Ingress + cert-manager Certificate templates and the `oidc.existingSecretRef` deployment path — the production shape where the OAuth client secret lives in an externally-managed Kubernetes Secret (External Secrets Operator, Sealed Secrets, Vault) instead of being baked into the chart-rendered config Secret via helm release history. Pairs the chart-side wiring with a small additive surface on the broker binary: an `OIDC_CLIENT_SECRET` env-var override in `broker.LoadConfig`. **What landed** - **`ingress.yaml`** — opt-in (`ingress.enabled: false` default). When enabled, requires `ingress.host`; supports three TLS modes mutually-exclusive at the template level: `tls.existingSecret` (operator-provided `kubernetes.io/tls`), `tls.certManager.enabled: true` (chart provisions a cert-manager Certificate via `certificate.yaml`), or neither (Ingress controller terminates TLS externally). Default `ingressClassName: nginx`; default path `/` `Prefix`. `annotations` block left to the operator for controller-specific knobs (`proxy-body-size`, `cert-manager.io/cluster-issuer`, etc.). - **`certificate.yaml`** — cert-manager `Certificate` resource conditionally rendered when both `ingress.enabled: true` and `ingress.tls.certManager.enabled: true`. Targets the chart-named TLS Secret (`-tls`); the Ingress references the same name so cert-manager writes the Secret and the Ingress picks it up by reference. DNS01 vs HTTP01 lives on the issuer, not on the Certificate resource — issuer config is operator-side. - **`oidc.existingSecretRef`** values block (name + key). When `name` is set, the deployment template renders an `OIDC_CLIENT_SECRET` env on the container via `secretKeyRef` pointing at the operator-managed Secret, AND the rendered config.yaml leaves `clientSecret: ""` blank so the env-var override path is the unambiguous runtime source. - **`broker.LoadConfig.applyEnvOverrides()`** (`tools/demarkus-broker/internal/broker/config.go:262`) — pure additive helper invoked between YAML parse and `validate()`. Today only `OIDC_CLIENT_SECRET` is recognized. Env wins over file when both set; empty env treated as unset so an accidentally-cleared variable doesn't blank out a file-supplied value at runtime. Net: production deployments keep the OAuth secret out of helm release history without changing the binary's startup contract. - **README expanded** with production checklist (Ingress + cert-manager + existingSecretRef pattern) and the `rateLimit.trustForwardedFor: true` invariant call-out (only safe behind an XFF-stripping Ingress; the failure mode of flipping it on a directly-internet-exposed broker is the documented footgun). - **`NOTES.txt`** post-install message summarizing rendered endpoints, RBAC scope, TLS mode, and the OIDC redirect URI the operator needs to register at the IdP. **Decisions that landed** - **`existingSecretRef.key` required at render-time when `name` is set.** Render-time fail-fast on a blank `key` instead of `CreateContainerConfigError` at pod schedule. Three fail-render guards now live in `secret-config.yaml` (clientSecret + existingSecretRef.name both set, neither set, name set but key blank); §6.3.D will pin all three so the guards never regress to silent precedence. - **`tls.existingSecret` and `tls.certManager.enabled: true` mutually exclusive.** Allowing both would render an Ingress pointing at one Secret while `certificate.yaml` provisions a different Secret — silent TLS-source ambiguity. Render-time `fail` keeps the contract unambiguous. CodeRabbit fix. - **Env wins over file, empty env is unset.** The two-line `applyEnvOverrides` body is small enough that the test surface (4-row `TestLoadConfigOIDCClientSecretEnvOverride` in `config_test.go`) is the primary documentation of the contract. Critical row: "empty env does not clobber file value" — deterministic via `t.Setenv("OIDC_CLIENT_SECRET", "")` so the test asserts the empty-as-unset property regardless of whether the parent CI shell happens to have the env var exported. **Framing: env-var override is not a §6.2 reopening.** It's a one-line escape hatch for the production-secret-ref deployment shape, not a feature-set expansion. §6.2 "broker binary complete" stands; the plan v14 bump will capture this as a §6.3 §Decisions bullet with a footnote on §6.2 §Status rather than reopening §6.2 in §Sequencing. **Test surface** — `TestLoadConfigOIDCClientSecretEnvOverride` 4 rows: env overrides file, env supplies value when file empty, empty env does not clobber file value, no env + no file → `validate()` rejects `oidc.clientSecret is required`. Coverage stable at 83.4%. **Next: §6.3.D** — helm-unittest test files for the broker chart covering the chart invariants (resource-policy: keep, RBAC verbs, NetworkPolicy egress/ingress shape, OIDC mutual-exclusion guards, OIDC_CLIENT_SECRET env wiring, securityContext lockdown, topology spread, checksum/config), CI plumbing to run helm-unittest against both demarkus-server and demarkus-broker charts on PR, and folding in a `test-broker` Go-test job (the §6.2 broker binary currently has zero CI coverage — `.github/workflows/ci.yml`'s `detect-changes` only filters protocol/server/client, not tools/). Kind upgrade-wipe regression test deferred to a sidecar PR bundled with the §6.1 chart's missing `helm.sh/resource-policy: keep` annotation on the demarkus-server tokens Secret — plan v13 §6.1 prose claims it's set but `deploy/helm/demarkus-server/templates/tokens.yaml` ships without it. Real prior-work debt; tracked separately so §6.3.D isn't blocked behind the §6.1 fix. ## 2026-05-13 — Slice 6.3.D.1 merged: chart unittest suites + CI plumbing, broker binary CI gap closed PR #118 merged to main as part of §6.3 broker chart work. Fourth and last sub-PR for §6.3 chart proper. One CodeRabbit round with five comments (four inline + one out-of-diff); all addressed without pushback — none were strategic enough to warrant a defense. **What landed** - **`deploy/helm/demarkus-broker/tests/`** — seven helm-unittest test suites covering the chart's load-bearing invariants: Deployment (replicas, security context lockdown, HTTP probes, POD_NAME env, OIDC_CLIENT_SECRET env wiring in both modes, populated topology-spread matchLabels, `checksum/config` rollout-on-config-change), broker-ns + per-world RBAC (verbs scoped to `get/update`-not-`create`, lease verbs `get/create/update`, `rbac.create=false` skip), NetworkPolicy (default-on, ingress namespace scoping, DNS+443 egress shape), Secret config (rate-limit + sweeper defaults explicitly rendered, OIDC mutex guards × 3 fail-render cases, world-without-allow-block doesn't panic), Secret issuances (`helm.sh/resource-policy: keep` + brokerNamespace helper consistency), Ingress + cert-manager Certificate (TLS mode mutex, host-required, className override), PodDisruptionBudget (default `minAvailable: 1`, selector labels). 49 tests across 7 suites. - **`.github/workflows/ci.yml`** — added `broker` (`tools/demarkus-broker/**` + `tools/go.mod` + `tools/go.sum` + `protocol/**`) and `charts` (`deploy/helm/**`) paths to the change-detection filter. New `test-broker` job: Go test/vet/lint/build for `tools/demarkus-broker/` with `working-directory: tools/demarkus-broker` (narrowed past the broader `tools/` scope to avoid lint failures on `tools/tools.go`). New `test-charts` job: `helm lint` on all three charts + `helm unittest` on server + broker. Both new jobs `timeout-minutes: 20`. helm pinned to v3.13.2; helm-unittest pinned to v0.6.2 (compatible across the local helm version). - **`deploy/helm/demarkus-server/tests/statefulset_test.yaml`** — fixed real prior-work bug. The existing suite only loaded `statefulset.yaml`, so the StatefulSet template's `include "...tokens.yaml"` for the `checksum/config` annotation failed render and all 13 tests errored out. Added `tokens.yaml` to the suite templates list + `template: statefulset.yaml` per-test to scope assertions. Nobody had noticed because helm-unittest never ran in CI. The §6.1 chart-test claims in plan v13 weren't actually executing — until now. **Decisions / observations** - **Fold-in of `test-broker` was the right call.** The handoff's §6.3.D scope proposed adding helm-unittest to CI + flagged separately that the §6.2 broker binary had zero CI coverage. Folding the binary-CI job into the same PR meant the broker chart shipped paired with a binary that finally has real CI gates. Without the fold-in we'd have had the absurdity of testing the chart's behavior in CI while the binary it wraps remained CI-uncovered. - **helm-unittest v0.6.2 over v1.x.** v1.0+ uses `platformHooks` in plugin.yaml which needs helm v3.16+; Fritz's local helm is v3.13.2. Pinning v0.6.2 in CI (matching the local-dev validation surface) keeps the test behavior identical between developer machines and CI. Bump together later when helm itself bumps. - **Per-test `template:` scope is required when one template `include`s another.** The shared `checksum/config: {{ include "...secret-config.yaml" }}` pattern (and the §6.1 chart's analogous `tokens.yaml` include) means tests for the parent template must list BOTH templates in the suite `templates:` filter, and then narrow each test's assertion target via the singular `template:` field. Discovered while debugging the broker `deployment_test.yaml`; the same fix retroactively rescued the §6.1 statefulset suite. **CodeRabbit comments addressed (all five, no pushback)** 1. **Broker path filter narrowing** — `tools/**` → `tools/demarkus-broker/**` + `tools/go.mod` + `tools/go.sum`. CodeRabbit's static analysis confirmed `tools/tools.go` sits outside the broker package; the broad filter would trigger broker CI on unrelated tool changes. 2. **Job timeouts** — `timeout-minutes: 20` on `test-broker` and `test-charts`. Default GitHub Actions timeout (360min) is too lenient for these jobs. 3. **golangci-lint scope** — `working-directory: tools` → `tools/demarkus-broker`. Same reasoning as #1; verified locally that golangci-lint walks up to `tools/go.mod` for module resolution. 4. **CLAUDE.md "Records" fragment removed** — line 38 had a malformed header ("Records (one per decision, zero-padded 4-digit sequence)") followed by a sub-bullet describing journals not records. Collapsed into a single journal entry matching the surrounding flat-list style. Surfaced to Fritz that a separate ADR-style Records concept can be added back if intended. 5. **CLAUDE.md journal-path contradiction (out-of-diff)** — line 27 said `/journal.md`, line 39 said `//journal/.md`. Updated line 27 to match the dated convention. Caveat flagged: this made CLAUDE.md aspirational ahead of the soul, which still had the legacy `/journal.md` shape at PR-merge time. Migration to dated files happens in this very session, immediately after PR merge. **Prior-work debt surfaced (still open, sidecar PR §6.3.D.2 scope)** - **§6.1 demarkus-server `tokens.yaml` Secret missing `helm.sh/resource-policy: keep`.** Plan v13 §6.1 §Status claims the annotation is set and a kind integration test pins the upgrade-wipe regression. Reality at PR-merge time: the annotation is NOT on the rendered Secret and no kind test exists in CI. Real bug in shipped §6.1 chart; bundle the fix with the §6.3.D.2 kind harness so the regression test catches it directly. - **§6.2 broker binary had zero CI coverage** at the start of this session. Closed by the `test-broker` job in this PR; surfacing it here so the gap doesn't get lost in the diff. - **helm-unittest wasn't running anywhere.** The §6.1 chart's existing test files weren't executed by CI, AND the `statefulset_test.yaml` suite was structurally broken from the day it was written. Both fixed in this PR. Lesson: chart-test files in the repo don't equal CI test coverage of charts; the workflow plumbing matters at least as much as the test files. **Next: §6.3.D.2 sidecar** — kind upgrade-wipe regression for both charts, bundled with the §6.1 `helm.sh/resource-policy: keep` + `helm.sh/hook: pre-install` fix on `demarkus-server/templates/tokens.yaml`. Test shape: install chart, kubectl-patch the rendered tokens/issuances Secret to seed fake data, helm upgrade with a values change, assert seeded data persists. Broker pod doesn't need to be ready for the property to hold — kind harness can skip OIDC discovery via a values stub. Plan bump to v14 after §6.3.D.2 lands so v14 captures coherent §6.3-complete landed-state. ## 2026-05-13 — Journal migrated to dated per-day files Operational note. Until this session, all session notes lived in a single growing `/journal.md` file (v57 at migration time, ~140KB / 51 entries across 23 unique dates). PR #118 merged a CLAUDE.md edit committing to a `/journal/.md` per-day convention; this entry is the first written under that convention. Migration mechanic: the legacy `/journal.md` body was parsed by `## YYYY-MM-DD —` heading boundaries, grouped by date (multiple entries on the same date stayed in one file as sequential `##` sections), and published as `/journal/.md` for each unique date. Same content, different topology. The old `/journal.md` was kept (mark_archive not used — the version history at the legacy path is still readable) and `/index.md` updated to point at the new directory. Why per-day: a 140KB single file was bumping fetch-result size limits (had to be saved to file rather than returned inline; agents had to read the saved file in chunks). Per-day files cap at ~40KB worst case (2026-05-12 was the longest day, with 5 entries totaling 38KB) and most days are well under 10KB. Future fetches by date go straight to the right file rather than scrolling a 51-entry catalog. Convention going forward: `mark_append` to the current date's file when adding a session note. If the file doesn't exist yet, `mark_publish` it. The CLAUDE.md "End of Session" line points operators (and agents) at the right path. ## 2026-05-13 — Slice 6.3.D.2 merged: §6.1 server chart resource-policy fix + kind upgrade-wipe regression PR #119 merged to main as `3d3b4bc`. Sidecar to §6.3.D.1; closes the §6.1 prior-work debt that earlier plan versions claimed was already in place. Lands the `helm.sh/resource-policy: keep` annotation on the demarkus-server tokens Secret AND an end-to-end kind regression that proves the upgrade-wipe property — for BOTH charts — rather than re-asserting it in prose. With this, all five §6.3 sub-slices (A/B/C/D.1/D.2) are merged on 2026-05-13 and §6.3 broker chart work is fully complete. **What landed** - **Three-branch hybrid template** at `deploy/helm/demarkus-server/templates/tokens.yaml` keyed on the live Secret's state: 1. **First install** (`lookup` returns nil) — render fresh content with `helm.sh/resource-policy: keep`. `$rawToken` resolves to a freshly-generated admin token; both the tokens Secret and the token-values Secret ship with consistent raw+hash. 2. **Legacy upgrade** (Secret exists but LACKS the keep annotation) — render the EXISTING data verbatim plus the keep annotation. Migration path from §6.1's original template (PR #107) which shipped without the annotation; without this branch, the §6.1.D.2 lookup-skip branch would emit nothing, helm would diff old-annotation-less against new-absent, and DELETE the Secret on apply, wiping every broker-minted user-token entry along the way. Re-rendering existing data makes helm's apply a content no-op while the annotation lands so future upgrades flip to branch 3. 3. **Race-free no-op** (Secret exists AND has the keep annotation) — emit nothing. helm sees the Secret removed from the rendered manifest, would normally GC it, but the live annotation blocks deletion. After the first migration, upgrades NEVER touch the Secret content, leaving the broker as the sole writer and eliminating any race window between `helm template` and `helm apply`. - **`deploy/helm/test-upgrade-wipe.sh`** — 142-line generic kind upgrade-wipe regression. Usage: install chart with values stub, kubectl-patch the rendered Secret to seed fake data, helm upgrade with a values change, assert seeded data persists, uninstall, assert Secret survives uninstall under the keep annotation. `SIMULATE_LEGACY_NO_KEEP` env flag exercises the migration path by stripping the annotation from the live Secret between install and upgrade — the test then asserts the legacy-upgrade branch re-applies the annotation AND preserves existing data. - **`deploy/helm/demarkus-server/tests/tokens_test.yaml`** — three new assertions added (10 total tests in the file now): tokens Secret carries `helm.sh/resource-policy: keep`, token-values Secret carries the same, and the rendered Secret has explicit `metadata.namespace` so the `lookup`-skip branch checks the right namespace on upgrade (rather than helm's release-namespace default which silently misaligns when an operator deploys with `--namespace`). - **`.github/workflows/ci.yml`** — new `test-charts-kind` job (47 lines). Spins up a kind cluster, runs `test-upgrade-wipe.sh` three times: server normal mode + server `SIMULATE_LEGACY_NO_KEEP` mode + broker normal mode. Broker pod does NOT need to be ready (OIDC discovery is irrelevant to the upgrade-wipe property); the harness uses a values stub. Job runs only on charts-path changes. **Decisions** - **Three-branch hybrid over "always render" (Option A).** Option A keeps helm "managing" the Secret on every upgrade — helm template renders content (from lookup), helm apply writes it. That opens a race window: between helm's lookup snapshot and the apply, the broker is concurrently writing via `mutateSecret` (resourceVersion optimistic concurrency). Apply uses last-writer-wins on fields not under strategic-merge-patch, so a concurrent broker write between lookup and apply gets clobbered. The race-free branch 3 of the hybrid sidesteps this entirely: helm emits nothing, the live keep annotation blocks GC, broker stays the sole writer. The cost is a one-time legacy branch for migration; once a release passes through branch 2, all subsequent upgrades hit branch 3. - **`SIMULATE_LEGACY_NO_KEEP` env flag rather than a separate script.** Same regression script exercises both the normal three-branch path AND the migration path. Two scripts would have drifted in the assertion shape; one script with a flag keeps the property-under-test identical across modes. - **CI runs three matrix entries, not nine.** Server normal + server legacy + broker normal covers the property. Broker doesn't have a legacy state to migrate from (it shipped with the keep annotation from §6.3.A day one), so a broker legacy mode would test nothing. Agent chart doesn't have a stateful Secret to wipe — out of scope. **CodeRabbit rounds** Two rounds during PR #119 review. - **Round 1** caught two critical issues. (a) The original draft of `tokens.yaml` had only two branches (lookup-skip + first-install) — it would have wiped Secrets on the FIRST upgrade of any pre-D.2 release because legacy Secrets had no keep annotation, so lookup-skip would emit nothing, helm would diff against the missing-from-rendered-manifest state, and delete. The three-branch hybrid was the response: branch 2 catches the legacy state, re-renders existing data verbatim plus the annotation, then future upgrades land on branch 3. (b) The original test script didn't exercise the legacy-upgrade path at all — only the first-install + race-free no-op. `SIMULATE_LEGACY_NO_KEEP` was the response: identical script with an env flag that strips the annotation between install and upgrade to simulate a pre-D.2 release's state, then asserts the legacy branch re-applies the annotation while preserving data. - **Round 2** caught a `kubectl jsonpath` issue. The original script used dot-notation for the data field lookup: `kubectl get secret ... -o jsonpath="{.data.tokens-toml-b64}"`. The hyphenated key happens to work on kubectl's permissive parser, but isn't spec-conformant — a stricter k8s version or a downstream tool reading the same expression could break silently. Switched to bracket form: `{.data['tokens-toml-b64']}`. Empirically the dot form was passing; the bracket form is defensive against the permissive-parser regression. **Test surface** - `deploy/helm/demarkus-server/tests/tokens_test.yaml`: 10 tests total, 3 new (keep annotation × 2 Secrets + explicit namespace). - `deploy/helm/test-upgrade-wipe.sh`: end-to-end kind cluster + helm install/upgrade/uninstall + Secret data assertion. Doesn't add test counts; adds a property that prose can't carry. - `test-charts-kind` CI job: three matrix entries (server normal + server legacy + broker normal). **Resolved risks** - The §6.1 chart upgrade-wipe gap that plan v14 carried as an open risk (the prose claim of "`resource-policy: keep` is in place" vs. the reality of the missing annotation) is now resolved — the kind regression pins the property end-to-end so the prose is no longer load-bearing. Plan v15 risk entry can be removed or downgraded to "resolved (pinned by kind regression)". - The §6.1 chart-test plumbing latent failure surfaced in §6.3.D.1 is also closed end-to-end: the unittest suites run in CI AND the kind harness exercises the live-cluster behavior the unittests can't reach (lookup against an actual Secret, helm upgrade flow, uninstall). **Next** §6.3 fully complete. Phase 6 remaining sub-phases: §6.4 universe topology examples (ApplicationSet + Kustomize overlay), §6.5 observability recipes (Datadog / OTel Collector / Vector / Fluent Bit / Grafana Alloy), §6.6 documentation suite (install / security / operations / observability / per-chart READMEs), §6.7 release pipeline (GHCR images + OCI charts, multi-binary server image, CLI relocation to `tools/`). Open question for Fritz: order of attack. §6.7 is the most-unblocking — without it, §6.4 examples and §6.6 docs both reference not-yet-released artifacts. §6.5 is independent of §6.7. Likely sequence: §6.7 → §6.5 (parallel-able) → §6.6 incremental → §6.4 last. Waiting on Fritz to pick. ## 2026-05-13 — §6.7.0 merged: hoist server/internal/store → protocol/store PR #120 merged to main as `b210015`. Precursor refactor to §6.7.A; no behavior change, no new feature. Purely a namespace move so `demarkus-publish` (and any future tool that needs versioned content storage) can consume the store package without `server/` becoming a heavyweight dependency. **What landed** - `server/internal/store/store.go` → `protocol/store/store.go` (git mv). - `server/internal/store/store_test.go` → `protocol/store/store_test.go` (git mv). - Import paths rewritten in four files: `server/internal/handler/handler.go`, `server/internal/handler/handler_test.go`, `server/cmd/demarkus-server/main.go`, `server/cmd/demarkus-publish/main.go`. goimports reordered the import blocks alphabetically (cosmetic). **Why protocol/store/ over alternatives.** The package only depends on stdlib + protocol; it has no QUIC, no handlers, no auth — just versioned-content disk-storage primitives. `protocol/` already owns `token/` (the disk-shape of capability tokens) and `auth.go` (the HashToken contract). The store fits the same conceptual layer. Hoisting to a new top-level module (`store/`) would have over-engineered for an abstraction that doesn't span beyond protocol's scope. Hoisting to `server/store/` (out of internal/) would have left `tools/` importing the heavy `server/` module just to access disk primitives — wrong dependency direction. **Verification.** protocol, server, client, tools all tested green; pre-commit.sh format/vet/lint clean across all four modules. **Lesson.** Earlier handoff prose claimed hoisting store was "non-trivial — gets imported by 8+ files in the server module." Actual reach was four files, all import-path-only rewrites (no API changes). The non-trivial framing was an overstatement; the move is a 5-minute refactor when scoped correctly. Useful calibration for future "is this load-bearing" judgment calls. ## 2026-05-13 — §6.7.A merged: split per-service container images + CLI relocation PR #121 merged to main as `aa9a3c6`. Closes the §6.1 caveat that has been carried since PR #107 (Phase 6.1 server chart shipped with exec probes referencing `/demarkus`, but the server image only contained `/demarkus-server`). The chart was YAML-correct from day one; the image side finally catches up. Two-hour rework arc inside the PR: started as a unified single-image bundle, pivoted to per-service split images after an architectural-fit discussion with Fritz. **What landed** - **Three per-service container images**, each ENTRYPOINTed to its primary binary: - `ghcr.io/latebit-io/demarkus-server` (`server/Dockerfile`) — bundles `demarkus-server` + `demarkus` (CLI for exec probes). 14MB. - `ghcr.io/latebit-io/demarkus-broker` (`tools/demarkus-broker/Dockerfile`) — single binary; HTTP probes; CA bundle for outbound OIDC. 25MB. - `ghcr.io/latebit-io/demarkus-agent` (`client/cmd/demarkus-agent/Dockerfile`) — single binary; CA bundle for outbound `mark://` TLS. 7MB. - **Multi-stage Dockerfiles**, each with build context = repo root so cross-module source is reachable (server image's CLI lives in `client/`; broker pulls `protocol/`; agent pulls `protocol/`). - **`/etc/ssl/certs/ca-certificates.crt` copied from build stage** into broker + agent runtime images. Server skips it — no outbound TLS today; bundled CLI is for `localhost` probes with `-insecure`. - **`USER 65532:65532` in all three runtime stages** matching the charts' `podSecurityContext.runAsUser`. Defense-in-depth so the image runs non-root even outside k8s. - **`go build -C ` instead of `RUN cd && go build`** across all three Dockerfiles. Idiomatic Go tooling (1.20+ `-C` flag), removes the `RUN cd` pattern Trivy flags as DS-0013. - **CLI relocation to `tools/`**: `server/cmd/demarkus-token` → `tools/demarkus-token/`, `server/cmd/demarkus-publish` → `tools/demarkus-publish/`. `demarkus-token` was a clean move (only imports `protocol/token`); `demarkus-publish` became clean after §6.7.0's `protocol/store` hoist landed. - **`server/Dockerfile` deleted and recreated** at the same path with the new multi-stage shape. Previous file was 4 lines of `COPY demarkus-server /` + `ENTRYPOINT` and depended on goreleaser-built binaries in the build context. - **`server/.goreleaser.yml` slimmed** — removed `demarkus-token` + `demarkus-publish` build entries (they live in `tools/` now) and removed the `dockers:` + `docker_manifests:` blocks (the new image build lives at the per-component Dockerfiles, wiring into release.yml is §6.7.B scope). - **`Makefile` `image` target** with sub-targets `image-server`, `image-broker`, `image-agent`. `make image TAG=dev` builds all three locally. - **Chart values reverted to per-component image refs** (each chart's `image.repository` points at its own image; no `command:` field — relies on each image's ENTRYPOINT). - **Chart unittests updated**: server +1 assertion (image regex + `notExists command`), broker +1 (same shape on broker tag), agent +1 (same shape on agent tag). Fixed a pre-existing structural bug in `deploy/helm/demarkus-agent/tests/deployment_test.yaml` — same `template:` scoping issue as §6.3.D.1 caught in the server's statefulset suite; agent's 10 deployment tests had been silently erroring because configmap.yaml + secret.yaml were missing from the suite templates list. **Decisions** - **Per-service split images over unified single-image bundle.** Initial implementation packaged all six binaries (server, broker, agent, CLI, token, publish) into one `ghcr.io/latebit-io/demarkus` image; each chart picked its binary via the pod's `command:` field. Worked end-to-end (one image, 52.8MB) but Fritz pushed back on the abstraction shape: demarkus is closer to Kubernetes (multi-component product — server speaks QUIC, broker speaks HTTP-OIDC, agent is a crawler) than Vault (single-purpose product). Multi-component products idiomatically split images so each pod's CVE blast radius covers only its own dependency tree; a CVE in broker's OIDC library shouldn't fail a server pod's security scan. Split images: 48MB total across 3 images, each pod pulls only ~10-26MB. Cost: 3 CI builds vs 1; minimal Go-static-binary footprint makes this negligible. The split is also reversible later if a customer requests a single-image distribution. - **Admin CLIs (demarkus-token, demarkus-publish) NOT bundled in any runtime image.** They ship as standalone binaries via the existing goreleaser archives. Operators run them locally (port-forward) or in dedicated Job pods with the right PVC mount, not via `kubectl exec` into a long-running pod. The principle: runtime pods carry only what they need to serve their primary role. "I want demarkus-token for kubectl exec convenience" is a leaked abstraction — admin tooling lives outside the pod blast radius. - **CA bundle is selective.** Broker (calls OIDC issuer discovery/token endpoints over HTTPS) and agent (crawls remote `mark://` servers with `insecure: false` in production) NEED `/etc/ssl/certs/ca-certificates.crt` to validate certs. Server doesn't — it's the QUIC listener; bundled CLI exec probes use `-insecure` for localhost. Adding CA bundle to server "just in case" would leak "this image makes outbound TLS calls" into its abstraction; if needed later, it's a one-line add. - **No HEALTHCHECK in Dockerfiles** (Checkov CKV_DOCKER_2 nit). K8s livenessProbe/readinessProbe are authoritative for deployed pods. HEALTHCHECK in a Dockerfile is silently ignored by k8s and would be a confusing artifact for anyone reading the image expecting it to be the source of truth. Skipped. - **No SHA256 digest pinning on `golang:1.26-alpine`** (CodeRabbit 🔵 trivial nit). Floating the minor tag lets us pick up Go stdlib security patches automatically; pinning to a digest locks all three Dockerfiles to whatever 1.26.x was current at pin time and requires coordinated manual bumps on every Go patch release. Without renovate/dependabot wired to track base-image digests, pinning trades reproducibility for a maintenance hazard. Natural follow-up when base-image automation arrives alongside module-bump automation. **CodeRabbit rounds (3)** - **Round 1** (commit `080e54f`, unified-image draft): one comment, 🛠️ Refactor / 🟠 Major — "Set explicit non-root USER in the final image." Valid concern. Initial draft's unified Dockerfile defaulted to root in the scratch stage. Applied to all three Dockerfiles after the split: `USER 65532:65532` matching chart values. - **Round 2** (commit `4af56e6`, split-image draft): two comments, both ⚠️ Potential / 🟠 Major — "Missing CA trust store in scratch runtime breaks HTTPS/OIDC egress" (broker) and "Scratch runtime is missing CA certificates for outbound TLS" (agent). Both correct, both fixed by copying `/etc/ssl/certs/ca-certificates.crt` from the build stage. Server Dockerfile was NOT flagged (correctly — server has no outbound TLS). - **Round 3** (commit `3c2ea8e`, ENTRYPOINT/CA hardening): one comment, ⚠️ Potential / 🔴 Critical — "Build output path can fail because `/out` is never created." False alarm — Go 1.26 `go build -o /out/binary` auto-creates the parent directory. Verified with a clean container test (mkdir-less, exit 0, /out materializes). Trivy's nested DS-0013 warning (`RUN cd` shouldn't change directory) had merit, addressed by switching to `go build -C ` instead of `RUN cd && go build`. CodeRabbit's `mkdir -p /out` suggestion explicitly skipped — defensive code for a non-issue. - **Round 4** (commit `dffec77`, go build -C fix): one comment, 🧹 Nitpick / 🔵 Trivial — "Pin golang image to immutable digest for reproducibility." Skipped per the §Decisions rationale above; replied on the PR explaining the trade. **Useful learnings** - **CodeRabbit "Critical" severity isn't always accurate.** The `/out` directory comment was marked 🔴 Critical despite being a false alarm in modern Go. Worth running a focused test against a base-image-clean container before accepting/skipping these — same principle as treating any "you should fix this" advice from any tool: verify the premise, then act. - **`go build -C ` is the idiomatic way to build from sibling modules** in a monorepo. It's been in Go since 1.20; cleaner than `cd module && go build` in Docker RUN statements and addresses Trivy DS-0013 simultaneously. - **Three-image build cost: ~3min for all images cold, ~30s incremental.** Each Dockerfile reuses the alpine base layer + Go module cache between builds. Layer caching makes the per-component split fast in practice; the conceptual cleanliness is the dominant win. **Test surface** - Server chart: 38 → 37 (consolidated a test by merging assertions, no coverage loss). - Broker chart: 50 → 49 (same merge consolidation). - Agent chart: 22 → 21 (same, plus the agent's pre-existing `templates:` scoping bug fix rescued 10 tests that had been silently erroring). - Total: 107 helm-unittest assertions across 16 suites. All green. - Kind smoke: server pod went Ready in 19s as UID 65532 from the split server image; exec probe (`/demarkus -insecure -no-cache mark://localhost:6309/.well-known/agent-manifest.md`) succeeded against the bundled CLI. The §6.1 probes-will-fail-until-6.7 caveat is now empirically closed. **Next: §6.7.B** — wire per-image `docker build` + `docker push ghcr.io/latebit-io/:` into `.github/workflows/release.yml`; add `tools/.goreleaser.yml` for `demarkus-broker` + `demarkus-token` + `demarkus-publish` binary archives; add OCI chart publish via `helm push oci://ghcr.io/latebit-io/charts/` for all three charts. Cosign signing stays backlogged. Transition window note: until §6.7.B lands, charts' default `image.repository` references images that don't yet exist on GHCR — operators must build locally via `make image` or override `image.repository`/`image.tag` at install time. ## 2026-05-13 — §6.7.B release pipeline complete (3 PRs: per-image release, tools release, OCI chart publish) Three sub-PRs merged in same-day succession to close out Phase 6.7. Total ~200 lines of `.github/workflows/release.yml` additions plus a new `tools/.goreleaser.yml`. End-to-end the three runtime images and three OCI Helm charts all have a release path tied to their corresponding module-version tag. **PR #122 (§6.7.B.1, commit `0e1a4ae`) — image release on existing tag flow.** Added `docker/build-push-action@v6` steps to the existing `release-server` and `release-client` jobs. `release-server` was already set up with qemu+buildx+login from the (now-removed) goreleaser `dockers:` block; just plugged in build-push-action against `server/Dockerfile`. `release-client` had none of the docker setup actions — added qemu+buildx+login plus the build-push step for `client/cmd/demarkus-agent/Dockerfile`. Both build multi-arch (linux/amd64 + arm64 + arm/v7) and push to `ghcr.io/latebit-io/demarkus-{server,agent}:${version}` + `:latest`. **PR #123 (§6.7.B.2, commit `afb6155`) — `tools/v*` release flow.** New `tools/.goreleaser.yml` mirroring `server/.goreleaser.yml`'s shape with three independent builds (`demarkus-broker`, `demarkus-token`, `demarkus-publish`) producing three independent archives. New workflow plumbing: `tools` filter on detect-changes, `test-tools` job, `semver-tools` (PaulHatch with tag_prefix=tools/v, change_path=tools/), `release-tools` that creates tag + runs goreleaser + builds-and-pushes the broker image via `tools/demarkus-broker/Dockerfile`. Admin CLIs (token, publish) ship as archive-only — no runtime image — per the §6.7.A "minimal pod blast radius" decision. **PR #124 (§6.7.B.3, commit `9cf6303`) — OCI chart publish.** Added `azure/setup-helm@v4` + `helm package` + `helm push` steps at the end of each of the three release jobs. Each `helm package --version $VER --app-version $VER` overrides both `Chart.yaml` fields to the just-released module version, so a chart pulled from the registry references an image at the same tag (`image.tag | default .Chart.AppVersion`). `helm push oci://ghcr.io/latebit-io/charts/` relies on the `~/.docker/config.json` written by `docker/login-action` — helm 3.13+ reads it natively for OCI registry auth; no separate `helm registry login` step needed. **Mapping (chart ↔ image ↔ module tag):** - `demarkus-server` chart + `demarkus-server` image ← `server/v*` tag - `demarkus-agent` chart + `demarkus-agent` image ← `client/v*` tag (agent lives in `client/cmd/`) - `demarkus-broker` chart + `demarkus-broker` image ← `tools/v*` tag **Decisions** - **`docker/build-push-action@v6` over re-introducing goreleaser's `dockers:` block** (§6.7.B.1). The current multi-stage Dockerfiles compile from source via Go's cross-compilation under buildx, not by copying goreleaser-built binaries. Wiring image release through build-push-action keeps the Dockerfiles unchanged and portable (anyone can `docker build` locally and get the same image without needing goreleaser-pro). Cost: binaries compile twice on release (once for archives, once inside image build) — Go is fast, ~30s extra per platform, negligible. - **Chart-version + appVersion pinned 1:1 to module version on release** (§6.7.B.3). Coupled lifecycle: chart-of-version-X always references image-of-version-X. Independent chart versioning would be useful for chart-only fixes (broken template, NOTES.txt edit), but adds tag-scheme complexity (`charts//v*` per chart) for a use case we don't have evidence of yet. Revisit if it materializes. - **No `-X main.version=...` stamping in `tools/.goreleaser.yml`** (§6.7.B.2). Matches the existing pattern in `server/.goreleaser.yml` + `client/.goreleaser.yml` — neither stamps version via goreleaser. Pre-existing oversight (the agent's `var version = "dev"` is wired in the source but never overridden at release time). Fixing this is a separate cleanup; not in §6.7.B scope. - **`tag_prefix: tools/v` + `change_path: tools/`** (§6.7.B.2 semver-tools job). Mirrors the server/client semver jobs exactly — tools/ is now a peer module in the release-pipeline taxonomy. The PaulHatch/semantic-version action handles auto-bumping based on conventional commit prefixes in commit subjects + bodies on paths under `tools/`. **Verification** - All three workflow YAMLs parse via Ruby's YAML loader at each branch handoff. - Single-platform `docker buildx build` against each Dockerfile with `--build-arg VERSION=test-X.Y.Z` confirmed each builds cleanly. - Multi-arch builds (`linux/amd64,linux/arm64` via `docker-container` driver — same as `setup-buildx-action@v3` in CI) verified for server and agent Dockerfiles. Broker Dockerfile is structurally identical to agent; skipped local multi-arch smoke after the user-interrupt on the QEMU-emulated build (Fritz asked what was taking so long). - `helm package --version 9.9.9 --app-version 9.9.9` against all three charts confirmed produces a tarball whose extracted `Chart.yaml` has both fields stamped to 9.9.9. Push not exercised locally (would require ghcr.io push perms). **Closes** - The §6.7.A transition window (charts referenced non-existent images on GHCR) is now fully closed — next module-version tag fire on any of server/client/tools pushes both the image and the chart. - The §6.7 sub-phase as a whole. Phase 6 is now 80% complete by sub-phase: §6.0/§6.1/§6.2/§6.3/§6.7 done; §6.4/§6.5/§6.6 remaining. **Useful learnings** - **`PaulHatch/semantic-version` + `change_path` is monorepo-friendly.** Each module gets its own tag-prefix-and-change-path pair, so independent bumps don't cross-contaminate. The `bump_each_commit: false` flag keeps the version stable until a feat/BREAKING CHANGE commit on that path actually warrants a bump. - **`docker/login-action` writes a docker config that helm 3.13+ trusts for OCI auth.** No additional `helm registry login` step needed in CI when the docker login step is already there. This saved a redundant credential-handling layer. - **GitHub Actions YAML supports comments densely.** Each new step in the release jobs gets a doc-comment explaining WHY (e.g., "build context = repo root because the agent module's tools/ replace directive needs protocol/ alongside"). The release.yml is now self-documenting for any future maintainer adding a fourth module to the release pipeline. **Next** §6.5 / §6.6 / §6.4 remain. §6.5 (observability recipes) is independent of any release-pipeline artifact — pure config files for Datadog / OTel Collector / Vector / Fluent Bit / Grafana Alloy mapping demarkus's existing structured slog output to each backend. §6.6 (docs) and §6.4 (topology examples) can start referencing concrete chart pull URLs now that §6.7.B closed the gap. Two-week deadline at 2026-05-27 with ~11 days remaining. ## 2026-05-13 — §6.5 merged (skinny version): agent slog migration + observability docs PR #125 merged to main as `1643806`. Reframed from the original §6.5 scope (5 per-backend log-shipper configs) to a skinny version after a cost/value pushback from Fritz on whether observability recipes advance the knowledge universe at all. **Reframing** The plan v17 §6.5 originally called for `deploy/observability/{datadog,otel-collector,vector,fluent-bit,grafana-alloy}/` directories with per-backend config files (~700–1000 lines across 5 backends, ~10 hours). Fritz pushed back: "I am just checking, I mean observability is good, but what is included in this." Honest assessment: per-backend recipes are operator ergonomics, not knowledge-universe expansion. They help an SRE team wire demarkus logs into Datadog/OTel/etc. without writing their own grok patterns. They don't change what demarkus does as a product. Worse: the recipes would be aspirational rather than verified. Without real Datadog/OTel/Vector/etc. infrastructure, the configs would be "follows documented patterns" not "tested end-to-end against a backend." Backend config languages drift; a `vector.yaml` written today is wrong six months later. Committing untested configs to the repo creates a maintenance burden that decays into noise. The skinny answer is a single doc that documents the stable thing — the slog field schema demarkus actually emits — and points operators at their backend's own JSON-log docs. That doc stays correct because it's the actual code path. ~1 hour of work + the agent slog conversion that's worth doing regardless. **What landed** - **`client/cmd/demarkus-agent/main.go`** — converted from stdlib `log` to `log/slog`. Package-level `logger` reads `DEMARKUS_LOG_FORMAT` env var; defaults text, JSON in production. Strict validation on the env var (typo → exit 2) since the agent's env-var contract is brand-new in this PR — no deployed agent has it set, so no backwards-compat risk to flipping it from lenient to strict. Stdout summary line in non-verbose `crawl` mode preserved (it's user-facing output for shell scripts, not a log event). 269 → 291 lines. - **`deploy/helm/demarkus-server/values.yaml` + `templates/statefulset.yaml`** — added `server.logFormat: json` chart default. Production deployments get machine-parseable logs out of the box without operator intervention. The binary's standalone default stays text (lenient, unchanged for backwards compat). - **`deploy/helm/demarkus-agent/values.yaml` + `templates/deployment.yaml`** — same shape for the agent chart. - **`docs/deployment/observability.md`** (new, 136 lines) — single-doc reference covering: format/routing (which service emits what default and how to override), field schema tables for server/broker/agent (one row per significant `msg`, with level, key fields, what it means), "useful operator queries" per service (count by verb, mint rate, sweeper health, etc.), pointers to each major backend's JSON-log ingestion docs (Datadog Agent, OTel filelog receiver, Vector, Fluent Bit, Grafana Alloy/Loki), and a closing "what's NOT in the logs" section (no raw subjects/emails — subject hashing pinned, no tokens, no bodies, no /metrics endpoint). **Decisions** - **Strict validation on agent's `DEMARKUS_LOG_FORMAT`, lenient on server's** (CodeRabbit round + Fritz pushback). Initial fix made both strict for symmetry, but Fritz pointed out the server's env var has shipped with lenient silent-fallback behavior since whenever it was wired in; flipping to strict-error breaks existing deployments that may have typos set. The agent is brand-new slog surface in this PR — no compat to break — so strict is fine there. `logging.New` library function stays lenient with the documented "unknown format defaults to text" behavior; the existing `TestNew` "unknown format defaults to text" assertion keeps passing. - **Skip per-backend recipe files entirely; ship a schema doc instead** (Fritz cost/value pushback). Schema is what stays stable; recipes drift with backend config languages. A doc that reads "here are the fields demarkus emits — your backend's JSON-log docs explain how to parse them" is more truthful and lower-maintenance than 5 aspirational YAMLs nobody verified end-to-end. - **Doc location at `docs/deployment/observability.md`** (Fritz confirmed earlier session). Stays consistent with the existing `docs/SPEC.md`, `docs/DESIGN.md` top-level convention. - **No subject/email in logs** (existing broker behavior, documented in the new doc's "what's NOT in the logs" section). `hashSubject(claims.Subject)` is the only identity surface; raw `claims.Email` never logged. Audit trails for "what did user X do" live in the issuances Secret. Documented as a deliberate alignment with the capability model — server is identity-blind; broker knows identity but doesn't publish it through logs. **CodeRabbit rounds (2)** Round 1 (commit `54166c2`): 1. **Agent `DEMARKUS_LOG_FORMAT` silently falls back to text on typo.** Valid — `jsno` would silently downgrade. Fixed with explicit switch + `os.Exit(2)` on invalid. 2. **Datadog paragraph names wrong integration.** `kubernetes_state_core` is a metrics integration, not log collection. Fixed to reference `logs_enabled` + `containerCollectAll` (Operator equivalents `features.logCollection.enabled`). 3. **Fluent Bit link points at `kubernetes-events`** (which collects k8s API events, not container logs). Fixed to `tail` input + `kubernetes` filter. Round 2 (commit `b95eb3f`): 4. **`tools/demarkus-broker/main.go:48` will drift** when the broker main changes. Dropped the `:48`. 5. **`§6.3.D.2` cross-reference** in the doc — operators reading this file have no way to find that section (it's a plan-doc section on the soul). Dropped the parenthetical; sentence still reads cleanly. **Useful learnings** - **"Aspirational" config files are a tax, not a feature.** I was about to commit 5 directories of per-backend YAML/conf files I couldn't verify against a real backend. Fritz's pushback caught it. The schema-doc approach is more honest and lower-maintenance. Pattern: when you can't verify a thing end-to-end, ship the stable contract (the schema) and point at the moving-target thing (backend docs) by reference. - **Backwards-compat surface for env vars matters even when "fixing" them.** Server's `DEMARKUS_LOG_FORMAT` has been lenient since it shipped; flipping to strict-error on typo is a defensible cleanup in isolation but breaks deployments that rely on the lenient behavior. Agent's env var didn't exist before this PR, so strict is OK there. Different histories deserve different treatment even when the env var name is the same. - **Library function behavior + binary validation belong at different layers.** Keeping `logging.New` lenient (so library callers + tests can pass any string and get a working logger) while validating at the binary boundary (where startup-failure is appropriate) is the cleaner split than putting validation inside the library and dealing with API breakage. **Next** Phase 6 remaining: §6.4 (universe topology examples — ApplicationSet + Kustomize) and §6.6 (docs suite). Both are ops polish per Fritz's earlier framing — they don't materially advance the knowledge universe. Open question for Fritz: do we keep grinding through Phase 6 polish, or pivot to something that moves the knowledge universe (`/plans/conflict-merge.md` for content-loss reduction; `/plans/claude-code-plugin.md` for distribution; obsidian plugin polish; agent/federation feature work; TUI/CLI ergonomics)? ## Related documents - [Universe deployment plan](/plans/universe-deployment.md): sections 6.3, 6.5, 6.7 implemented in this session - [Journal 2026-05-12](/journal/2026-05-12.md): continues the broker Slice C work