# Plan: Universe Deployment

Ship the enterprise-grade deployment for demarkus. The deliverable *is* the deployment: a customer's ops team installs a Helm chart, runs a broker, runs an agent, and has a federated demarkus universe their org can use. Any company evaluating demarkus (their internal "POC") installs the same product an established customer runs in production. There is no separate "POC slice" — the slice mentality is rejected. We build it once, right, and customers trial the real thing.

## Goal

Deliver a complete, supportable, production-grade Kubernetes deployment package for demarkus, including:

1. A Helm chart for `demarkus-server` (one world). ✓
2. A Helm chart for `demarkus-broker` (OIDC token issuance + revocation). ✓
3. A Helm chart for `demarkus-agent` (hub aggregator, crawl-and-index). ✓
4. Reference topology examples (Argo CD `ApplicationSet`, Kustomize overlay).
5. Backend-agnostic observability — structured slog emission + reference configs for Datadog, OTel Collector, Vector, Fluent Bit, Grafana Alloy.
6. Customer-facing documentation — installation, security/threat model, operations, upgrade path, per-provider OIDC setup, observability recipes.
7. A release pipeline producing images and chart releases consumable from GHCR. ✓

The same artifacts power first-customer trial and steady-state operations.

## Non-Goals (Phase 7+ territory)

- Multi-replica worlds with shared storage (RWX / object backend).
- Cross-cluster universe federation.
- Operator with a `World` CRD.
- Hosted / managed SaaS.
- Non-markdown content.

## Constraints

- **No core protocol/server changes.** Period. Per prior precedent (Claude Code plugin, Obsidian plugin, `feedback_plugin_scope.md`). Observability is achieved by log derivation in a collector, not by adding `/metrics` or OTel SDK calls to the server. Note: in Slice A we promoted `HashToken` and the token-mint library to `protocol/` (under `protocol/auth.go` and `protocol/token/`). These are additive helper relocations consumed by server + CLI + future broker; no wire-protocol or server behavior changed. Slice C.2 added `protocol/token.ParseBytes` in the same additive spirit — a read-side helper so the broker's drift sweeper can inspect a world's `tokens.toml` payload via the map shape rather than substring matching on serialized TOML. §6.7.0 hoisted `server/internal/store/` → `protocol/store/` in the same additive spirit — the disk-shape of versioned content lives at the protocol layer alongside `token/` and the `HashToken` contract; no API change, four import-path rewrites.
- Capability-based auth model is non-negotiable. The server never learns identity — only labels.
- Markdown-only scope is non-negotiable.

## Decisions (resolved during planning)

- **Health probes use `exec`, not HTTP.** Both charts ship liveness/readiness `exec` probes that invoke the `demarkus` CLI fetching `/.well-known/agent-manifest.md` (always public per `/architecture.md`). No core change. Same pattern as `redis-cli ping` / `pg_isready`.
- **UDP port is a values knob.** Default `6309`, override via `server.udpPort`. Documented option: switch to `443` for VPN/middlebox-hostile networks (Cloudflare Warp Zero Trust, corp firewalls that filter non-standard UDP). Protocol default stays `6309`.
- **Wildcard TLS via cert-manager DNS-01.** Single `*.<root>` cert covers all worlds + broker hostnames. HTTP-01 cannot work for worlds (no HTTP); DNS-01 is the standard path for QUIC services.
- **DNS topology.** One A record per world + one for the broker, all under the same wildcard zone. Each world exposed by `Service: LoadBalancer` `protocol: UDP`; broker by standard Ingress (HTTPS).
- **Hub aggregator is the existing `demarkus-agent`** at `client/cmd/demarkus-agent/`. Already implements `crawl` + `daemon` subcommands with TOML config, multi-worker crawl, per-host token auth, per-server + aggregated index publishing. Verified end-to-end on 2026-05-11. **Phase 5 is no longer a blocking prerequisite.**
- **`demarkus-agent` lives in `client/cmd/`, not `tools/`.** It's a protocol client (uses `fedcrawl`, `fetch`, `tokens`, `links`), not a utility. Memory rule sharpened: `client/cmd/` = protocol clients (CLI, TUI, MCP, agent); `tools/` = utilities (broker, future sync/ops binaries).
- **Token-mint library lives at `protocol/token/`, not `tools/internal/token/`** (revised mid-Slice-A after CodeRabbit review). `protocol/` is reachable by both `server/` (where `demarkus-token` CLI lives) and `tools/` (where the broker will live), which `tools/internal/` was not. `protocol/` already owns the `HashToken` byte-shape contract, so it's the natural home for the on-disk token primitives that must round-trip identically across CLI mints and broker mints.
- **`demarkus-token` and `demarkus-publish` CLIs live at `tools/`** (§6.7.A merged 2026-05-13 PR #121). `demarkus-token` had been a clean candidate since Slice A (only imports `protocol/token`). `demarkus-publish` followed once §6.7.0 (PR #120) hoisted `server/internal/store/` → `protocol/store/`, removing the server-internal dependency. Both ship as standalone binaries via `tools/.goreleaser.yml` archives (§6.7.B PR #123); they are NOT bundled into any runtime image.
- **Multi-OIDC.** Broker speaks generic OIDC, not Google-specific. Provider behind a `Verifier` interface. Google validated first; Okta, Entra ID, Auth0 follow with config only.
- **Workload Identity (GKE) as a broker option.** Values flag annotates the broker ServiceAccount with the GSA mapping. Off by default; on for GKE customers wanting no static SA keys. Not applicable to world servers.
- **Broker is HA.** Multi-replica with `resourceVersion` optimistic concurrency on Secret writes, retry on conflict.
- **Observability is log-derived, backend-agnostic.** demarkus already emits structured slog. Charts ship Datadog autodiscovery annotations + reference configs for OTel Collector, Vector, Fluent Bit, Grafana Alloy. Customer's SRE picks the agent/backend. Latency histograms deferred — depends on what current slog includes per request; revisit during trials if needed.
- **Backup/DR is documented, not built.** The chart deliberately does not run backup CronJobs. Operations doc covers Velero, VolumeSnapshot, and `demarkus-agent sync` as DR options.
- **Image hosting:** `ghcr.io/latebit-io/{demarkus-server,demarkus-broker,demarkus-agent}`. Three separate images, one per service. Released by §6.7.B per-module tag flow.
- **Chart registry:** `oci://ghcr.io/latebit-io/charts/{demarkus-server,demarkus-broker,demarkus-agent}`. Three OCI helm charts released by §6.7.B per-module tag flow alongside their images.
- **Cosign signing:** deferred to backlog. Half-day CI add when wanted.
- **Groups-claim case sensitivity** (revised mid-Slice-C.1 after CodeRabbit back-and-forth). Group names match case-insensitively: `AllowConfig.Groups` is lowercased+trimmed at config load (same as `Domains` and `Emails`) and `groupsMatch` lowercases the claim's groups before compare. The validated IdP set (Google, Okta, Entra ID, Auth0) enforces case-insensitive group-name uniqueness, so distinct case-variant groups can't exist; a case-sensitive compare would silently fail when an operator writes "Engineering" but the IdP emits "engineering" after upstream normalization. Keycloak with deliberately distinct case-variant groups is out of scope; revisit if a customer asks. Domains and emails get the same lowercase-at-load + plain `==` treatment for symmetry.
- **Sweeper interval capped at 24h** (Slice C.2). The `defaultToken.expiresAfter` typical is 24h; a sweep interval longer than that would let expired tokens linger past their ExpiresAt for up to a whole sweep cycle, defeating the short-lived-tokens identity-lifecycle model (§Revocation §3). `validate()` rejects with `sweeper.interval must be <= 24h`. Lease-election timings (15s lease / 10s renew / 2s retry) intentionally not exposed in YAML — the client-go defaults match our failover budget and adding knobs invites misconfiguration. Revisit if a customer needs sub-15s failover.
- **`sweeper.disabled` rather than `sweeper.enabled`** (Slice C.2). YAML zero-value (false) is the production-safe default so an operator who omits the block gets the sweeper running. Naming the field positively (`Enabled`) would have made omission silently disable the janitor — exactly the wrong default for multi-replica deployments where expired tokens accumulate forever without it.
- **Rotate is re-login semantics, not refresh** (Slice C.3). `POST /tokens/:label/rotate` re-runs `worldAllows(world.Allow, claims)` on every call, so a user removed from the world's allowlist between mint and rotate can NOT extend access via rotation. The bearer ID-token verification at the HTTP layer already proves the IdP identity is still active; the per-world predicate re-check catches in-org permission changes (group removed, domain renamed, email taken off carve-out). Picked over rotation-as-refresh because rotation extends the lifetime — without re-auth, a user whose IdP account stayed live could rotate forever even after losing world-scope authorization.
- **Rotate scope vs. lifetime asymmetry** (Slice C.3). Scope (paths + operations) stays frozen to the issuance record on rotate: operator narrowing of `DefaultToken.Paths` between mint and rotate does NOT shrink the rotated token's reach, because the user shouldn't be surprised by reduced access mid-session — they re-login to pick up new scope. Lifetime resets to `now + DefaultToken.ExpiresAfter` from the operator's CURRENT config: operator tightening of expiry DOES apply on rotate, because shorter lifetimes are an explicit security-tightening lever and rotation must not bypass it. The asymmetry is intentional and pinned by `TestRotateLabelPreservesIssuanceScope` + `TestRotateLabelLifetimeUpdatesToCurrentConfig`.
- **Rotate sequence: mint new, then revoke old** (Slice C.3). On revoke failure the new token is returned anyway with a wrapped soft error; the sweeper retires the orphan old label on expiry (or via drift if an operator hand-cleans). Picked over revoke-then-mint because that alternative leaves the user with no token if mint then fails, forcing a full re-login — worse UX than a brief two-valid-tokens window for the same user. Mirrors the Slice B partial-mint convention: `(MintResult{Label:""}, err)` for hard failures, `(MintResult{non-empty}, err)` for soft partial success.
- **Middleware-not-handler auth + rate limit** (Slice C.4). The three authed `/tokens` routes go through `requireAuth → subjectRateLimit → handler`; `/auth/login` goes through `ipRateLimit → authLogin`; `/auth/callback`, `/healthz`, `/readyz` stay middleware-free. The old per-handler `s.authenticate(w, r)` was extracted into the `requireAuth` middleware that stashes verified `Claims` on `r.Context()` via a typed key; handlers read claims via `claimsFromCtx`. The alternative — keeping auth in handlers and adding rate-limit inside each one — would either double-verify the bearer (one verify for the limit key, one in the handler) or hide the limiter inside each handler, opaque to the route-registration block. Middleware composition keeps the per-route policy visible in `Routes()`.
- **One shared subject bucket across the three /tokens routes** (Slice C.4). `subjectRateLimit` keys on `hashSubject(claims.Subject)` and the same registry covers GET `/tokens`, DELETE `/tokens/:label`, and POST `/tokens/:label/rotate`. Picked over three per-route buckets so a misbehaving client cannot multiply effective throughput by fanning out (3 routes × 10/min would give 30/min effective rather than the operator-intended 10/min). Pinned by `TestRateLimitTokensSharedBucketAcrossRoutes`.
- **`Reserve()` + `Cancel()` on denial, not `Allow()`** (Slice C.4). Functional equivalence — neither pattern consumes budget on denial — but `Reserve().Delay()` gives the precise wait time, which we surface as `Retry-After` with a 1s minimum floor. (`Retry-After: 0` reads as "retry immediately" to aggressive clients and would defeat the limiter.) Pinned by `TestRateLimitRegistryDenialDoesNotConsumeBudget` (50 denials in a tight loop, then 150ms regen window allows the 51st request) and the `Retry-After` assertions on the integration 429 tests.
- **`trustForwardedFor: false` default in the binary** (Slice C.4). The broker behind an Ingress sees the controller's IP in `r.RemoteAddr`, so the per-IP limiter on `/auth/login` collapses into one bucket for every client unless we honor XFF. But trusting XFF when NOT behind a proxy lets an attacker rotate the header to bypass per-IP limits. The binary ships safe; chart-side §6.3.A inverts the default to `true` (the chart's "behind an Ingress" assumption holds in deployment). `TestRateLimitLoginIPIgnoresForwardedForByDefault` pins the default-untrusted behavior at the binary level; `TestRateLimitLoginIPCrossIPIsolation` pins the trust-enabled behavior.
- **`rateLimit.disabled` not `enabled`** (Slice C.4). Same shape as `sweeper.disabled`. Zero-value (false) gives the production-safe behavior so an operator who omits the `rateLimit:` block in their values file still gets the protection. Defaults applied at validate-time: 10/min subject burst 5, 20/min IP burst 5. Field validation skipped entirely when `Disabled: true` so an operator opting out can leave the per-route knobs empty.
- **Per-replica unbounded registry, by design** (Slice C.4). The `rateLimitRegistry` keeps one `*rate.Limiter` per key with no TTL eviction and no max-entry cap. CodeRabbit flagged this as a memory-DoS vector; rejected after re-examining the threat model. Realistic worst-case sizing is ~500KB at ~10k authenticated subjects — not memory-DoS-shaped. Cluster-shared limiter is the Phase-7+ fix; documented in §Backlog.
- **`OIDC_CLIENT_SECRET` env-var override** (§6.3.C, additive to §6.2). `broker.LoadConfig.applyEnvOverrides()` (`tools/demarkus-broker/internal/broker/config.go:262`) lets the OAuth client secret come from the environment instead of the on-disk config file. Env wins over file when both set; empty env is treated as unset so an accidentally-cleared variable can't blank out a file-supplied value. Production deployments keep the OAuth secret in an externally-managed Kubernetes Secret (External Secrets Operator, Sealed Secrets, Vault) mounted via `secretKeyRef`, instead of baking it into the chart-rendered config Secret where it would leak into helm release history. This is a one-line escape hatch for the production-secret-ref deployment shape, NOT a §6.2 reopening — `§6.2 broker binary complete` framing stands. Pinned by `TestLoadConfigOIDCClientSecretEnvOverride` (4-row table).
- **helm-unittest pinned to v0.6.2 in CI** (§6.3.D.1). v1.0+ uses `platformHooks` in plugin.yaml which needs helm v3.16+; Fritz's local helm is v3.13.2. Pinning v0.6.2 in CI matches the local-dev validation surface so test behavior is identical between developer machines and CI. Bump together when helm itself bumps. Helm pinned to v3.13.2 alongside.
- **`test-broker` Go CI job folded into §6.3.D.1**. Documented above.
- **Three-branch hybrid template for upgrade-safe Secrets** (§6.3.D.2). `deploy/helm/demarkus-server/templates/tokens.yaml` keys on the live Secret's state: (1) first install → render fresh + keep annotation; (2) legacy upgrade → re-render existing data verbatim + add the annotation; (3) race-free no-op → emit nothing, helm GC blocked by live annotation, broker stays sole writer. Picked over Option A (always render) which keeps a race window between `helm template`'s lookup snapshot and `helm apply` while broker writes via `mutateSecret`.
- **`SIMULATE_LEGACY_NO_KEEP` test-mode flag over a separate script** (§6.3.D.2). One script with env-flag exercises both the normal three-branch path and the migration path.
- **`test-charts-kind` CI job** (§6.3.D.2). 47-line job in `.github/workflows/ci.yml` runs `test-upgrade-wipe.sh` three times: server normal + server `SIMULATE_LEGACY_NO_KEEP` + broker normal.
- **`kubectl jsonpath` bracket notation for hyphenated keys** (§6.3.D.2 CodeRabbit). `{.data['key-with-hyphens']}`, NOT `{.data.key-with-hyphens}`. Spec-conformant bracket form. Pin this convention for any future jsonpath against k8s data fields.
- **Per-service split container images over a unified single-image bundle** (§6.7.A). Three images: `ghcr.io/latebit-io/demarkus-server` (server + CLI for probes), `demarkus-broker` (single binary + CA bundle for OIDC), `demarkus-agent` (single binary + CA bundle for remote `mark://` crawl TLS). Initial §6.7.A draft packaged all six binaries into one `ghcr.io/latebit-io/demarkus` image; reverted to split because demarkus's shape (multi-component product) calls for per-service blast-radius isolation. A CVE in broker's OIDC dependency shouldn't fail the server pod's image scan.
- **Admin CLIs not bundled in runtime images** (§6.7.A). `demarkus-token` and `demarkus-publish` ship as standalone binaries via `tools/.goreleaser.yml` archives. Operators run them locally with port-forward or in dedicated Job pods, NOT via `kubectl exec` into long-running pods. Bundling admin tools into runtime images would leak "admin tooling" into the pod's blast radius.
- **CA bundle selectively included** (§6.7.A). Broker + agent images copy `/etc/ssl/certs/ca-certificates.crt` from the build stage. Server image skips it — no outbound TLS today; bundled CLI's exec probes use `-insecure` for localhost.
- **`USER 65532:65532` in all three runtime stages** (§6.7.A). Matches each chart's `podSecurityContext.runAsUser` for defense-in-depth.
- **`go build -C <module>` over `RUN cd <module> && go build`** (§6.7.A). Go 1.20+ idiom; auto-creates `-o` parent directory.
- **No SHA256 digest pinning on `golang:1.26-alpine`** (§6.7.A CodeRabbit nit skipped). Floating the minor tag picks up Go stdlib security patches automatically; pinning trades reproducibility for a maintenance hazard without renovate automation. Revisit when base-image automation lands.
- **`docker/build-push-action@v6` over re-introducing goreleaser's `dockers:` block** (§6.7.B.1 PR #122). The new multi-stage Dockerfiles compile from source via Go's cross-compilation under buildx, not by COPY-ing goreleaser-built binaries. Wiring image release through build-push-action keeps the Dockerfiles unchanged and portable (anyone can `docker build` locally without goreleaser-pro). Cost: binaries compile twice on release (once for archives, once inside image build) — Go is fast, ~30s extra per platform, negligible.
- **Chart-version + appVersion pinned 1:1 to module version on release** (§6.7.B.3 PR #124). Coupled lifecycle: chart-of-version-X always references image-of-version-X. Independent chart versioning would be useful for chart-only fixes (broken template, NOTES.txt edit), but adds tag-scheme complexity for a use case without evidence yet. Revisit if it materializes.
- **OCI helm auth via shared `~/.docker/config.json`** (§6.7.B.3). Helm 3.13+ reads the docker config natively for OCI registry auth; the existing `docker/login-action` step satisfies both image push and chart push. No separate `helm registry login` step needed.
- **`tag_prefix: tools/v` + `change_path: tools/`** (§6.7.B.2 semver-tools job). PaulHatch/semantic-version mirror of server/client semver jobs; tools/ is now a peer module in the release-pipeline taxonomy with independent version bumps.

## Open Questions

1. **First customer trial.** Nesto (`*.library.nesto.ca`) is path-B — trial waits for product. Trial runbook lands at `/trials/nesto.md` when scoping starts.
2. **Two-week deadline (2026-05-27):** §6.7 fully landed; §6.5 / §6.6 / §6.4 remain. ~11 days runway. Suggested order: §6.5 → §6.6 → §6.4 (per Fritz's earlier confirmation). §6.5 is independent and parallelizable; §6.6 grows incrementally; §6.4 depends on §6.7.B published artifacts (now available).
3. **First module-release tag firing of the §6.7.B pipeline.** Triggered automatically by the next push to main that touches `server/`, `client/`, or `tools/`. Until a first push fires, no image or chart is yet on GHCR despite the pipeline being live. Operators building from main still need `make image` locally or `--set image.repository=...` overrides. Resolves on the first `feat:` or `fix:` commit to a module path.
4. **Hardened broker chart symmetry (deferred from §6.3.D.2).** The broker chart's `secret-issuances.yaml` uses the simpler lookup-skip pattern. Defense-in-depth follow-up; not load-bearing. Only act on if Fritz brings it up.

## Repository Layout

Reflects state as of §6.7.B.3 merge (2026-05-14):

```
deploy/
  helm/
    demarkus-server/         # one-world chart (Phase 6.1) — shipped PR #107
    demarkus-broker/         # OIDC token broker chart (Phase 6.3)
    demarkus-agent/          # crawl/index agent chart (Phase 6.0) — shipped PR #106
  test-upgrade-wipe.sh       # kind regression: §6.3.D.2 PR #119
  k8s/
    examples/
      applicationset.yaml    # Argo CD ApplicationSet (§6.4 — pending)
      kustomize-overlay/     # Kustomize alternative (§6.4 — pending)
  observability/             # per-backend log-shipper configs (§6.5 — pending)
docs/
  deployment/                # install / security / operations / observability
                             # (§6.6 — pending; location confirmed 2026-05-13)

.dockerignore                # repo-root, applies to all three image builds
                             # (§6.7.A PR #121)

.github/
  workflows/
    release.yml              # full per-module release pipeline:
                             # - server/v*  → goreleaser archives
                             #               + docker buildx push demarkus-server image
                             #               + helm package + push OCI chart
                             # - client/v*  → goreleaser archives
                             #               + docker buildx push demarkus-agent image
                             #               + helm package + push OCI chart
                             # - tools/v*   → goreleaser archives (broker+token+publish)
                             #               + docker buildx push demarkus-broker image
                             #               + helm package + push OCI chart
                             # (§6.7.B PRs #122 + #123 + #124)
    ci.yml                   # PR + push tests; test-broker + test-charts +
                             # test-charts-kind (§6.3.D.1, §6.3.D.2)

protocol/
  auth.go                    # HashToken — sha256-<hex> contract (Slice A)
  token/                     # Generate, ReadFile, AppendEntry, WriteFile,
                             # FormatEntry, flock helpers (Slice A) +
                             # AppendBytes, RemoveBytes in-memory (Slice B) +
                             # ParseBytes (Slice C.2).
  store/                     # versioned content-store primitives, hoisted
                             # from server/internal/store (§6.7.0 PR #120).

server/
  Dockerfile                 # multi-stage; bundles demarkus-server + CLI
                             # (§6.7.A PR #121)
  .goreleaser.yml            # binary archives only
  cmd/
    demarkus-server/

client/
  .goreleaser.yml            # binary archives for demarkus, demarkus-tui,
                             # demarkus-mcp (demarkus-agent ships via Dockerfile
                             # build in release-client)
  cmd/
    demarkus/
    demarkus-tui/
    demarkus-mcp/
    demarkus-agent/
      Dockerfile             # (§6.7.A PR #121)

tools/
  .goreleaser.yml            # binary archives for demarkus-broker,
                             # demarkus-token, demarkus-publish (§6.7.B PR #123)
  demarkus-broker/
    Dockerfile               # (§6.7.A PR #121)
  demarkus-token/            # admin CLI (no Dockerfile — archive-only)
  demarkus-publish/          # admin CLI (no Dockerfile — archive-only)
```

## Sub-Phases

### 6.0 — `demarkus-agent` verified + chart ✓ (merged PR #106)

Existing `demarkus-agent` binary verified end-to-end (2 team worlds + 1 hub smoke test) and chart at `deploy/helm/demarkus-agent/` shipped. Stateless Deployment; ConfigMap holds TOML agent config; Secret holds per-host tokens; exec liveness probe; outbound-only (no Service). §6.7.A PR #121 fixed a pre-existing structural bug in the agent deployment test suite (10 silently-erroring tests, same shape as the server-statefulset suite). Agent ships per-image (`ghcr.io/latebit-io/demarkus-agent`) with CA bundle for outbound TLS, released via `client/v*` tag flow (§6.7.B PR #122 + #124).

### 6.1 — `demarkus-server` Helm chart ✓ (merged PR #107) + §6.3.D.2 ✓ + §6.7.A image fix ✓ + §6.7.B release ✓

`deploy/helm/demarkus-server/` — StatefulSet, 1 replica, `volumeClaimTemplates`, exec probes against `/.well-known/agent-manifest.md`, `Service` type `LoadBalancer` `protocol: UDP`. Bootstrap path seeds the initial `admin` token via `protocol/token.Generate` + `AppendEntry`. Cert-manager `Certificate` resource behind a flag. helm-unittest test files at `tests/` wired into CI by §6.3.D.1 PR #118 and validated end-to-end against a kind cluster by §6.3.D.2 PR #119.

**§6.3.D.2 sidecar:** tokens Secret now carries `helm.sh/resource-policy: keep` via three-branch hybrid template. PR #119 merged 2026-05-13. `test-charts-kind` CI job pins the upgrade-wipe property end-to-end.

**§6.7.A image fix:** chart's `image.repository` (default `ghcr.io/latebit-io/demarkus-server`) is built from `server/Dockerfile` as multi-binary image (`demarkus-server` + `demarkus` CLI). PR #121. The §6.1 probe caveat is empirically closed (kind smoke verified Ready in 19s).

**§6.7.B release plumbing:** next push to main touching `server/` triggers `server/v*` tag → goreleaser archives + multi-arch image push to `ghcr.io/latebit-io/demarkus-server:<version>` + OCI chart push to `oci://ghcr.io/latebit-io/charts/demarkus-server:<version>`. PRs #122 + #124.

### 6.2 — `demarkus-broker` binary (`tools/demarkus-broker/`) ✓ complete

Slice B (single-world OIDC mint flow) **merged as PR #109** (commit `2a6aae6`, 2026-05-11). Slice C.1 **PR #110**. Slice C.2 **PR #111**. Slice C.3 **PR #112**. Slice C.4 **PR #114**. **§6.2 broker binary complete.**

The §6.3.C `OIDC_CLIENT_SECRET` env-var override is a one-line additive surface that does NOT reopen §6.2.

### 6.3 — `demarkus-broker` Helm chart ✓ complete (all five sub-slices merged) + §6.7.B release ✓

Multi-replica HA at `deploy/helm/demarkus-broker/`, shipped across five sub-PRs (#115, #116, #117, #118, #119), all merged 2026-05-13. Chart now released via `tools/v*` tag flow (§6.7.B PR #123 + #124): image push to `ghcr.io/latebit-io/demarkus-broker:<version>` + OCI chart push to `oci://ghcr.io/latebit-io/charts/demarkus-broker:<version>`.

### 6.4 — Universe topology examples (pending)

ApplicationSet + Kustomize overlay at `deploy/k8s/examples/`. Both reference the chart pull URLs from §6.7.B (now live). Argo CD `ApplicationSet` template generator iterates a `worlds:` list and templates one `Application` per world. Kustomize overlay is the GitOps-without-Argo alternative. Customer's ops team picks one. Verification surface: `argocd app create --dry-run` or `kustomize build` against a kind cluster.

### 6.5 — Observability recipes (pending — independent of any prior phase, can start immediately)

Per-backend configs in `deploy/observability/`. Slice C.2 added sweeper-side log lines (`broker: swept`, `broker: sweep failed`, `broker: sweeper observing new leader`, `broker: sweeper lost leadership`); Slice C.3 added rotate-side log lines; Slice C.4 added rate-limit log lines (`broker: rate limit exceeded` with `route`, `subject` or `ip`, `retryAfter` fields). Recipes are pure config — no core code changes. Datadog autodiscovery via pod annotations + a dashboard JSON; OTel Collector via a `config.yaml`; Vector via a `[transforms.parse]` block; Fluent Bit via parser + filter; Grafana Alloy via `discovery.kubernetes` + `loki.process`.

### 6.6 — Documentation suite (pending — incremental)

`docs/deployment/*.md` (location confirmed 2026-05-13). Per-chart READMEs at `deploy/helm/<chart>/README.md` (broker README exists from §6.3.C; server + agent need similar treatment). Install guide (per-provider OIDC setup for Google / Okta / Entra ID / Auth0; chart install order: server → broker → agent; cert-manager + DNS topology). Security/threat model. Operations (upgrade path including the now-tested resource-policy:keep contract, backup/DR via Velero or `demarkus-agent sync`). Observability recipes (links to §6.5).

### 6.7 — Release pipeline ✓ complete

**§6.7.0** ✓ merged (PR #120, commit `b210015`) — Precursor refactor. Hoisted `server/internal/store/` → `protocol/store/`. Pure namespace move; four import-path rewrites.

**§6.7.A** ✓ merged (PR #121, commit `aa9a3c6`) — Per-service split container images + CLI relocation. Three Dockerfiles co-located with their primary binaries. Server image bundles CLI for probes; broker + agent images include CA bundle for outbound TLS. All three runtime stages USER 65532. Admin CLIs (token, publish) ship as standalone binaries — not bundled in any runtime image.

**§6.7.B** ✓ complete via three sub-PRs:

- **§6.7.B.1** ✓ merged (PR #122, commit `0e1a4ae`, 2026-05-13) — added `docker/build-push-action@v6` to existing `release-server` + `release-client` jobs. Multi-arch (amd64/arm64/armv7) push to `ghcr.io/latebit-io/{demarkus-server,demarkus-agent}:<version>` on `server/v*` / `client/v*` tag fire.
- **§6.7.B.2** ✓ merged (PR #123, commit `afb6155`, 2026-05-13) — new `tools/.goreleaser.yml` with three archive builds (broker + token + publish). New workflow plumbing: `tools` filter, `test-tools`, `semver-tools`, `release-tools`. `release-tools` creates `tools/v*` tag, runs goreleaser, builds + pushes broker image to `ghcr.io/latebit-io/demarkus-broker:<version>`.
- **§6.7.B.3** ✓ merged (PR #124, commit `9cf6303`, 2026-05-14) — `azure/setup-helm@v4` + `helm package` + `helm push` at the end of each of the three release jobs. Chart `version` + `appVersion` pinned 1:1 to module version. Charts push to `oci://ghcr.io/latebit-io/charts/{demarkus-server,demarkus-broker,demarkus-agent}`. Helm 3.13+ uses the docker login config natively for OCI auth.

**§6.7 fully complete.** All three runtime images + all three OCI helm charts have a release path tied to their corresponding module-version tag.

## Sequencing

1. **Slice A — token-mint library** ✓ merged 2026-05-11 (PR #108).
2. **6.0 chart** ✓ merged 2026-05-11 (PR #106).
3. **6.1 server chart** ✓ merged 2026-05-11 (PR #107). §6.3.D.2 ✓. §6.7.A image fix ✓. §6.7.B release ✓.
4. **6.2 broker binary — Slices B + C.1–C.4** ✓ all merged 2026-05-11–12 (PRs #109, #110, #111, #112, #114).
5. **6.3 broker chart** ✓ all five sub-slices merged 2026-05-13 (PRs #115, #116, #117, #118, #119).
6. **6.7.0 store hoist** ✓ merged 2026-05-13 (PR #120, `b210015`).
7. **6.7.A image consolidation + CLI relocation** ✓ merged 2026-05-13 (PR #121, `aa9a3c6`).
8. **6.7.B release pipeline** ✓ three sub-PRs merged 2026-05-13–14:
   - **6.7.B.1** (PR #122, `0e1a4ae`) — image release on existing server/v* + client/v* tags.
   - **6.7.B.2** (PR #123, `afb6155`) — tools/v* tag flow + broker image release.
   - **6.7.B.3** (PR #124, `9cf6303`) — OCI chart publish for all three charts.
9. **6.5 observability recipes** — NEXT. Independent of any prior phase. Pure config files.
10. **6.6 docs** — incremental throughout 6.5/6.4 plus dedicated polish pass.
11. **6.4 topology examples** — last; references chart pull URLs from §6.7.B (now live).

Rough effort: ~10 days of focused work remaining for §6.5 + §6.6 + §6.4 against the 2026-05-27 deadline. Phase 6 ~85% complete by sub-phase count (6.0/6.1/6.2/6.3/6.7 done; 6.4/6.5/6.6 remaining).

## Backlog (deferred, easy to add later)

- **Cosign signing** of images + chart releases. Half-day CI add.
- **Latency log-enrichment** (`duration_ms` field on request slog lines). Tiny additive change.
- **SCIM lifecycle webhook** on the broker. Optional add when a customer asks.
- **Userinfo-based groups** in the broker `Verifier`. Slice C.1 ships ID-token-only; `AllowEmails` is the documented workaround.
- **Case-sensitive group matching** for Keycloak. Slice C.1 chose case-insensitive for the validated IdP set.
- **Configurable lease timings** (`sweeper.leaseDuration`, `renewDeadline`, `retryPeriod`). C.2 hardcodes the client-go defaults.
- **SIGHUP on revoke + sweep + rotate** to shrink the revoke-to-effect window. Needs `pods/exec` RBAC across world namespaces.
- **Sweeper backing-store migration** when the issuances Secret hits the ~1MB / ~5000-issuance k8s storage ceiling.
- **Cluster-shared rate limiter** (Slice C.4 follow-up).
- **Idle-key GC for the rate-limit registry** (Slice C.4 follow-up).
- **Timeouts on the existing `test-protocol`/`test-server`/`test-client` CI jobs.** §6.3.D.1 added `timeout-minutes: 20` to the new jobs; the older jobs still rely on the GitHub Actions default (360 min). One-line cleanup.
- **Hardened broker chart Secret symmetry.** Broker's `secret-issuances.yaml` uses the simpler lookup-skip pattern (vs. the §6.3.D.2 three-branch hybrid on the server chart). Defense-in-depth follow-up; not load-bearing.
- **Base-image digest pinning** (§6.7.A CodeRabbit nit skipped). When renovate/dependabot is wired to bump base-image digests, pin all three Dockerfiles to specific SHAs for reproducibility.
- **Single-image distribution mode** (§6.7.A architectural alternative). If a customer asks for one-image-with-all-binaries, the unified-Dockerfile shape from the §6.7.A draft can be added as an opt-in. The split-images shape stays the default.
- **Server image CA bundle** (§6.7.A deferred). If a need arises for `kubectl exec` debugging or future outbound-TLS server features. One-line add.
- **Version-stamping for `demarkus-server` + `demarkus` CLI + `demarkus-token` + `demarkus-publish`** (§6.7.B deferred). Currently `var version` is wired in `demarkus-agent` and `demarkus-broker` only; goreleaser's `-X` flag on the others is a no-op. Add `var version = "dev"` + a `-version` flag to each missing binary; surface via `--version` subcommand. Out of §6.7 scope; small follow-up PR.
- **Independent chart versioning** (§6.7.B.3 deferred). Currently chart-version is pinned 1:1 to module version on release. If we need to ship a chart-only fix (broken template, NOTES.txt edit) without bumping the image, introduce a `charts/<name>/v*` tag scheme. Adds tag-scheme complexity; revisit when we have evidence of need.

## Risks

- **Observability via logs ceiling.** If customers want signals not derivable from current slog, we hit a wall. Mitigation: log enrichment is a small additive change.
- **Broker secret-write blast radius.** Mitigated by namespace-scoped `Role`s, audit log, optional `NetworkPolicy`. Per-world `Role` is non-negotiable. `TestMintRBACDeniedNoPartialState` validates clean failure.
- **OIDC provider coupling.** First impl is Google; structure so second provider is a one-day add.
- **Token revocation in-flight latency.** Revocation latency = kubelet Secret propagation + whatever the server does on re-mount. Acceptable for the deployment posture (~60s default kubelet propagation).
- **Rotate transient two-tokens window.** Documented; verified by `TestRotateTokenSoftPartial`.
- **§6.1 chart upgrade-wipe gap.** ✓ Resolved by §6.3.D.2 PR #119.
- **§6.1 chart-test plumbing latent failure.** ✓ Resolved by §6.3.D.1 PR #118; agent chart equivalent resolved by §6.7.A PR #121.
- **§6.7 multi-binary image hand-off.** ✓ Resolved by §6.7.A PR #121.
- **Release-pipeline image gap.** ✓ Resolved by §6.7.B PRs #122 + #123 + #124. First module-touching commit to main will fire the corresponding release flow and produce a fully-aligned chart+image bundle.
- **Issuances Secret scaling wall** at the ~1MB / ~5000-issuance k8s storage ceiling. Phase-7+ fix is a different backing store.
- **Chart proliferation.** Three charts + examples + dashboards. Mitigate with shared common-labels templates.
- **Trial scope creep.** First customer is path-B.
- **Per-replica rate limit ⇒ N× effective rate.** Slice C.4 documented; deferred fix in §Backlog.
- **Rate-limit registry growth under misconfigured XFF trust.** Documented in C.4 §Decisions and chart README.
- **First module-release tag fire is in-cycle on a `feat:` or `fix:` commit to a module path.** Until that fires, no image or chart is on GHCR despite the pipeline being live. Operators building from main need `make image` locally in the interim. Not a code risk; an operational one. Closes naturally on the first qualifying merge to main.

## Status

Plan v17, 2026-05-14. **Slice A, 6.0 chart, 6.1 chart (incl. §6.3.D.2 + §6.7.A image fix + §6.7.B release), 6.2 broker binary (Slices B + C.1–C.4), 6.3 broker chart (all five sub-slices), 6.7.0 store hoist, 6.7.A image consolidation + CLI relocation, 6.7.B release pipeline all merged** (PRs #106, #107, #108, #109, #110, #111, #112, #114, #115, #116, #117, #118, #119, #120, #121, #122, #123, #124).

**14 PRs merged across 2026-05-13–14 (the §6.3 → §6.7 sprint):** #115 (6.3.A) → #116 (6.3.B) → #117 (6.3.C) → #118 (6.3.D.1) → #119 (6.3.D.2) → #120 (6.7.0 store hoist) → #121 (6.7.A image consolidation) → #122 (6.7.B.1 image release) → #123 (6.7.B.2 tools release) → #124 (6.7.B.3 OCI chart publish). Phase 6 ~85% complete by sub-phase: §6.0 / §6.1 / §6.2 / §6.3 / §6.7 done; §6.4 / §6.5 / §6.6 remaining.

§6.7.B trajectory recap (three sub-PRs, same-day):
- **6.7.B.1** (PR #122, `0e1a4ae`) — `docker/build-push-action@v6` added to existing server/v* + client/v* release jobs. Multi-arch push to demarkus-server + demarkus-agent images. Chose build-push-action over re-introducing goreleaser's `dockers:` block to keep multi-stage Dockerfiles unchanged and portable.
- **6.7.B.2** (PR #123, `afb6155`) — new `tools/.goreleaser.yml` (broker + token + publish archives) + new `tools` filter/test-tools/semver-tools/release-tools jobs in workflow. tools is now a peer module with independent `tools/v*` tag-driven release cadence. Admin CLIs (token, publish) ship as archives only — no runtime image per §6.7.A decision on minimal pod blast radius.
- **6.7.B.3** (PR #124, `9cf6303`) — OCI chart publish via `helm package --version $VER --app-version $VER` + `helm push oci://ghcr.io/latebit-io/charts/<chart>`. Chart-version + appVersion pinned 1:1 to module version on release. Helm 3.13+ uses the docker login config natively for OCI auth, so no separate registry login step.

**Next: §6.5 observability recipes.** Independent of any prior phase. Per-backend log-shipper configs in `deploy/observability/{datadog,otel-collector,vector,fluent-bit,grafana-alloy}/` mapping demarkus's existing structured slog output to each backend. Pure config files; no Go changes. Customer's SRE picks one. Parallelizable with §6.6 docs work. Then §6.4 topology examples last (now unblocked by the §6.7.B chart OCI publish — examples can reference `oci://ghcr.io/latebit-io/charts/<chart>` pull URLs). Two-week deadline at 2026-05-27 with ~13 days remaining.