# Plan: Universe Deployment Ship the enterprise-grade deployment for demarkus. The deliverable *is* the deployment: a customer's ops team installs a Helm chart, runs a broker, runs an agent, and has a federated demarkus universe their org can use. Any company evaluating demarkus (their internal "POC") installs the same product an established customer runs in production. There is no separate "POC slice" — the slice mentality is rejected. We build it once, right, and customers trial the real thing. ## Goal Deliver a complete, supportable, production-grade Kubernetes deployment package for demarkus, including: 1. A Helm chart for `demarkus-server` (one world). 2. A Helm chart for `demarkus-broker` (OIDC token issuance + revocation). 3. A Helm chart for `demarkus-agent` (hub aggregator, crawl-and-index). 4. Reference topology examples (Argo CD `ApplicationSet`, Kustomize overlay). 5. Backend-agnostic observability — structured slog emission + reference configs for Datadog, OTel Collector, Vector, Fluent Bit, Grafana Alloy. 6. Customer-facing documentation — installation, security/threat model, operations, upgrade path, per-provider OIDC setup, observability recipes. 7. A release pipeline producing images and chart releases consumable from GHCR. The same artifacts power first-customer trial and steady-state operations. ## Non-Goals (Phase 7+ territory) - Multi-replica worlds with shared storage (RWX / object backend). - Cross-cluster universe federation. - Operator with a `World` CRD. - Hosted / managed SaaS. - Non-markdown content. ## Constraints - **No core protocol/server changes.** Period. Per prior precedent (Claude Code plugin, Obsidian plugin, `feedback_plugin_scope.md`). Observability is achieved by log derivation in a collector, not by adding `/metrics` or OTel SDK calls to the server. Note: in Slice A we promoted `HashToken` and the token-mint library to `protocol/` (under `protocol/auth.go` and `protocol/token/`). These are additive helper relocations consumed by server + CLI + future broker; no wire-protocol or server behavior changed. Slice C.2 added `protocol/token.ParseBytes` in the same additive spirit — a read-side helper so the broker's drift sweeper can inspect a world's `tokens.toml` payload via the map shape rather than substring matching on serialized TOML. - Capability-based auth model is non-negotiable. The server never learns identity — only labels. - Markdown-only scope is non-negotiable. ## Decisions (resolved during planning) - **Health probes use `exec`, not HTTP.** Both charts ship liveness/readiness `exec` probes that invoke the `demarkus` CLI fetching `/.well-known/agent-manifest.md` (always public per `/architecture.md`). No core change. Same pattern as `redis-cli ping` / `pg_isready`. - **UDP port is a values knob.** Default `6309`, override via `server.udpPort`. Documented option: switch to `443` for VPN/middlebox-hostile networks (Cloudflare Warp Zero Trust, corp firewalls that filter non-standard UDP). Protocol default stays `6309`. - **Wildcard TLS via cert-manager DNS-01.** Single `*.` cert covers all worlds + broker hostnames. HTTP-01 cannot work for worlds (no HTTP); DNS-01 is the standard path for QUIC services. - **DNS topology.** One A record per world + one for the broker, all under the same wildcard zone. Each world exposed by `Service: LoadBalancer` `protocol: UDP`; broker by standard Ingress (HTTPS). - **Hub aggregator is the existing `demarkus-agent`** at `client/cmd/demarkus-agent/`. Already implements `crawl` + `daemon` subcommands with TOML config, multi-worker crawl, per-host token auth, per-server + aggregated index publishing. Verified end-to-end on 2026-05-11. **Phase 5 is no longer a blocking prerequisite.** - **`demarkus-agent` lives in `client/cmd/`, not `tools/`.** It's a protocol client (uses `fedcrawl`, `fetch`, `tokens`, `links`), not a utility. Memory rule sharpened: `client/cmd/` = protocol clients (CLI, TUI, MCP, agent); `tools/` = utilities (broker, future sync/ops binaries). - **Token-mint library lives at `protocol/token/`, not `tools/internal/token/`** (revised mid-Slice-A after CodeRabbit review). `protocol/` is reachable by both `server/` (where `demarkus-token` CLI lives) and `tools/` (where the broker will live), which `tools/internal/` was not. `protocol/` already owns the `HashToken` byte-shape contract, so it's the natural home for the on-disk token primitives that must round-trip identically across CLI mints and broker mints. - **`demarkus-token` CLI stays at `server/cmd/demarkus-token/`** for now (rewired to import `protocol/token`). Moving the binary to `tools/` was deferred to a follow-up that splits the release pipeline (server release currently ships demarkus-token; tools has no release cadence yet). Same applies to `demarkus-publish` — still at `server/cmd/demarkus-publish/`, deferred because hoisting it requires also hoisting `server/internal/store/` out of internal-package scope. - **Multi-OIDC.** Broker speaks generic OIDC, not Google-specific. Provider behind a `Verifier` interface. Google validated first; Okta, Entra ID, Auth0 follow with config only. - **Workload Identity (GKE) as a broker option.** Values flag annotates the broker ServiceAccount with the GSA mapping. Off by default; on for GKE customers wanting no static SA keys. Not applicable to world servers. - **Broker is HA.** Multi-replica with `resourceVersion` optimistic concurrency on Secret writes, retry on conflict. - **Observability is log-derived, backend-agnostic.** demarkus already emits structured slog. Charts ship Datadog autodiscovery annotations + reference configs for OTel Collector, Vector, Fluent Bit, Grafana Alloy. Customer's SRE picks the agent/backend. Latency histograms deferred — depends on what current slog includes per request; revisit during trials if needed. - **Backup/DR is documented, not built.** The chart deliberately does not run backup CronJobs. Operations doc covers Velero, VolumeSnapshot, and `demarkus-agent sync` as DR options. - **Image hosting:** `ghcr.io/latebit-io/{demarkus-server,demarkus-broker,demarkus-agent}`. - **Chart registry:** OCI charts in GHCR. - **Cosign signing:** deferred to backlog. Half-day CI add when wanted. - **Groups-claim case sensitivity** (revised mid-Slice-C.1 after CodeRabbit back-and-forth). Group names match case-insensitively: `AllowConfig.Groups` is lowercased+trimmed at config load (same as `Domains` and `Emails`) and `groupsMatch` lowercases the claim's groups before compare. The validated IdP set (Google, Okta, Entra ID, Auth0) enforces case-insensitive group-name uniqueness, so distinct case-variant groups can't exist; a case-sensitive compare would silently fail when an operator writes "Engineering" but the IdP emits "engineering" after upstream normalization. Keycloak with deliberately distinct case-variant groups is out of scope; revisit if a customer asks. Domains and emails get the same lowercase-at-load + plain `==` treatment for symmetry. - **Sweeper interval capped at 24h** (Slice C.2). The `defaultToken.expiresAfter` typical is 24h; a sweep interval longer than that would let expired tokens linger past their ExpiresAt for up to a whole sweep cycle, defeating the short-lived-tokens identity-lifecycle model (§Revocation §3). `validate()` rejects with `sweeper.interval must be <= 24h`. Lease-election timings (15s lease / 10s renew / 2s retry) intentionally not exposed in YAML — the client-go defaults match our failover budget and adding knobs invites misconfiguration. Revisit if a customer needs sub-15s failover. - **`sweeper.disabled` rather than `sweeper.enabled`** (Slice C.2). YAML zero-value (false) is the production-safe default so an operator who omits the block gets the sweeper running. Naming the field positively (`Enabled`) would have made omission silently disable the janitor — exactly the wrong default for multi-replica deployments where expired tokens accumulate forever without it. - **Rotate is re-login semantics, not refresh** (Slice C.3). `POST /tokens/:label/rotate` re-runs `worldAllows(world.Allow, claims)` on every call, so a user removed from the world's allowlist between mint and rotate can NOT extend access via rotation. The bearer ID-token verification at the HTTP layer already proves the IdP identity is still active; the per-world predicate re-check catches in-org permission changes (group removed, domain renamed, email taken off carve-out). Picked over rotation-as-refresh because rotation extends the lifetime — without re-auth, a user whose IdP account stayed live could rotate forever even after losing world-scope authorization. - **Rotate scope vs. lifetime asymmetry** (Slice C.3). Scope (paths + operations) stays frozen to the issuance record on rotate: operator narrowing of `DefaultToken.Paths` between mint and rotate does NOT shrink the rotated token's reach, because the user shouldn't be surprised by reduced access mid-session — they re-login to pick up new scope. Lifetime resets to `now + DefaultToken.ExpiresAfter` from the operator's CURRENT config: operator tightening of expiry DOES apply on rotate, because shorter lifetimes are an explicit security-tightening lever and rotation must not bypass it. The asymmetry is intentional and pinned by `TestRotateLabelPreservesIssuanceScope` + `TestRotateLabelLifetimeUpdatesToCurrentConfig`. - **Rotate sequence: mint new, then revoke old** (Slice C.3). On revoke failure the new token is returned anyway with a wrapped soft error; the sweeper retires the orphan old label on expiry (or via drift if an operator hand-cleans). Picked over revoke-then-mint because that alternative leaves the user with no token if mint then fails, forcing a full re-login — worse UX than a brief two-valid-tokens window for the same user. Mirrors the Slice B partial-mint convention: `(MintResult{Label:""}, err)` for hard failures, `(MintResult{non-empty}, err)` for soft partial success. - **Middleware-not-handler auth + rate limit** (Slice C.4). The three authed `/tokens` routes go through `requireAuth → subjectRateLimit → handler`; `/auth/login` goes through `ipRateLimit → authLogin`; `/auth/callback`, `/healthz`, `/readyz` stay middleware-free. The old per-handler `s.authenticate(w, r)` was extracted into the `requireAuth` middleware that stashes verified `Claims` on `r.Context()` via a typed key; handlers read claims via `claimsFromCtx`. The alternative — keeping auth in handlers and adding rate-limit inside each one — would either double-verify the bearer (one verify for the limit key, one in the handler) or hide the limiter inside each handler, opaque to the route-registration block. Middleware composition keeps the per-route policy visible in `Routes()`. - **One shared subject bucket across the three /tokens routes** (Slice C.4). `subjectRateLimit` keys on `hashSubject(claims.Subject)` and the same registry covers GET `/tokens`, DELETE `/tokens/:label`, and POST `/tokens/:label/rotate`. Picked over three per-route buckets so a misbehaving client cannot multiply effective throughput by fanning out (3 routes × 10/min would give 30/min effective rather than the operator-intended 10/min). Pinned by `TestRateLimitTokensSharedBucketAcrossRoutes`. - **`Reserve()` + `Cancel()` on denial, not `Allow()`** (Slice C.4). Functional equivalence — neither pattern consumes budget on denial — but `Reserve().Delay()` gives the precise wait time, which we surface as `Retry-After` with a 1s minimum floor. (`Retry-After: 0` reads as "retry immediately" to aggressive clients and would defeat the limiter.) Pinned by `TestRateLimitRegistryDenialDoesNotConsumeBudget` (50 denials in a tight loop, then 150ms regen window allows the 51st request) and the `Retry-After` assertions on the integration 429 tests. - **`trustForwardedFor: false` default in the binary** (Slice C.4). The broker behind an Ingress sees the controller's IP in `r.RemoteAddr`, so the per-IP limiter on `/auth/login` collapses into one bucket for every client unless we honor XFF. But trusting XFF when NOT behind a proxy lets an attacker rotate the header to bypass per-IP limits. The binary ships safe; chart-side §6.3.A inverts the default to `true` (the chart's "behind an Ingress" assumption holds in deployment). `TestRateLimitLoginIPIgnoresForwardedForByDefault` pins the default-untrusted behavior at the binary level; `TestRateLimitLoginIPCrossIPIsolation` pins the trust-enabled behavior. - **`rateLimit.disabled` not `enabled`** (Slice C.4). Same shape as `sweeper.disabled`. Zero-value (false) gives the production-safe behavior so an operator who omits the `rateLimit:` block in their values file still gets the protection. Defaults applied at validate-time: 10/min subject burst 5, 20/min IP burst 5. Field validation skipped entirely when `Disabled: true` so an operator opting out can leave the per-route knobs empty. - **Per-replica unbounded registry, by design** (Slice C.4). The `rateLimitRegistry` keeps one `*rate.Limiter` per key with no TTL eviction and no max-entry cap. CodeRabbit flagged this as a memory-DoS vector; rejected after re-examining the threat model: (a) `subjectReg` growth is bounded by IdP user count (every key required a successful `requireAuth` pass, which means a valid IdP-issued token), (b) `loginReg` growth is bounded by the deployment posture (with `trustForwardedFor=false` the key is `r.RemoteAddr`, bounded by clients reaching the listener; with `trustForwardedFor=true` plus a correctly-configured XFF-stripping Ingress, the key space is bounded by real client IPs), and (c) TTL/LRU eviction *makes the property worse under the stated attack*: an attacker rotating keys faster than the TTL gets fresh buckets indefinitely, and LRU eviction during attack evicts legitimate users' older buckets first while the attacker's fresh buckets stay in the map. The right Phase-7+ fix lives at a different layer (cluster-shared rate limiter so per-replica eviction can't reset attacker state, or per-IP rate-limit annotations at the ingress controller). Realistic worst-case sizing is ~500KB at ~10k authenticated subjects — not memory-DoS-shaped. Mirrors the C.2 "sweep is unbounded" pushback in shape. - **`OIDC_CLIENT_SECRET` env-var override** (§6.3.C, additive to §6.2). `broker.LoadConfig.applyEnvOverrides()` (`tools/demarkus-broker/internal/broker/config.go:262`) lets the OAuth client secret come from the environment instead of the on-disk config file. Env wins over file when both set; empty env is treated as unset so an accidentally-cleared variable can't blank out a file-supplied value. Production deployments keep the OAuth secret in an externally-managed Kubernetes Secret (External Secrets Operator, Sealed Secrets, Vault) mounted via `secretKeyRef`, instead of baking it into the chart-rendered config Secret where it would leak into helm release history. This is a one-line escape hatch for the production-secret-ref deployment shape, NOT a §6.2 reopening — `§6.2 broker binary complete` framing stands. Pinned by `TestLoadConfigOIDCClientSecretEnvOverride` (4-row table). - **helm-unittest pinned to v0.6.2 in CI** (§6.3.D.1). v1.0+ uses `platformHooks` in plugin.yaml which needs helm v3.16+; Fritz's local helm is v3.13.2. Pinning v0.6.2 in CI matches the local-dev validation surface so test behavior is identical between developer machines and CI. Bump together when helm itself bumps. Helm pinned to v3.13.2 alongside. - **`test-broker` Go CI job folded into §6.3.D.1**. The §6.2 broker binary had zero CI coverage at the start of §6.3 chart work — `.github/workflows/ci.yml` only filtered protocol/server/client, not tools/. Adding helm-unittest plumbing while leaving the binary CI-uncovered would have been incoherent. New `test-broker` job runs Go test/vet/lint/build for `tools/demarkus-broker/`, with `working-directory: tools/demarkus-broker` (narrowed past the broader `tools/` scope to avoid lint failures on the package-marker file `tools/tools.go`). Path filter scoped to `tools/demarkus-broker/**` + `tools/go.mod` + `tools/go.sum` + `protocol/**` — broker CI does NOT run on unrelated tool changes. ## Open Questions 1. **First customer trial.** Nesto (`*.library.nesto.ca`) is path-B — trial waits for product. Trial runbook lands at `/trials/nesto.md` when scoping starts. 2. **CLI relocation to `tools/`.** Follow-up to Slice A. Requires `tools/.goreleaser.yml`, a `release-tools` job in the workflow, an `install.sh` block fetching the tools archive, and updates to plugin scripts + helm image build. Acceptable to defer until the release-pipeline work has its own dedicated PR. 3. **§6.3.D.2 sidecar PR remains open**: add `helm.sh/resource-policy: keep` + `helm.sh/hook: pre-install` to the §6.1 demarkus-server `tokens.yaml` Secret (the §6.1 prose has claimed this is in place since §6.1 shipped, but the chart never had the annotation), and add a kind upgrade-wipe regression test in CI for BOTH charts. Tracked as immediate next work, NOT folded into §6.3 chart proper because the §6.1 chart fix is independent scope. ## Repository Layout Reflects state as of §6.3.D.1 merge (2026-05-13): ``` deploy/ helm/ demarkus-server/ # one-world chart (Phase 6.1) — shipped PR #107 tests/ # helm-unittest suites — wired into CI by §6.3.D.1 PR #118 demarkus-broker/ # OIDC token broker chart (Phase 6.3) — shipped via PR #115, #116, #117 tests/ # helm-unittest suites — shipped by §6.3.D.1 PR #118 demarkus-agent/ # crawl/index agent chart (Phase 6.0) — shipped PR #106 k8s/ examples/ applicationset.yaml # Argo CD ApplicationSet over a worlds: list kustomize-overlay/ # Kustomize alternative observability/ datadog/ # autodiscovery annotations + dashboard JSON otel-collector/ # collector config recipes vector/ # vector config recipes fluent-bit/ # fluent-bit parser + filter recipes scripts/ # operator helpers (cert pre-check, MTU probe, etc.) protocol/ auth.go # HashToken — sha256- contract (Slice A) token/ # Generate, ReadFile, AppendEntry, WriteFile, # FormatEntry, flock helpers (Slice A) + # AppendBytes, RemoveBytes in-memory helpers # (Slice B) + ParseBytes read-side helper # (Slice C.2) for the broker's drift sweep. server/cmd/ demarkus-server/ # the server binary (existing) demarkus-token/ # token mint CLI — uses protocol/token demarkus-publish/ # publish CLI (existing; deferred move) client/cmd/ demarkus-agent/ # protocol client — federation crawler (existing) tools/ demarkus-broker/ # broker binary (Slice B PR #109, Slice C.1 PR #110, # Slice C.2 PR #111, Slice C.3 PR #112, Slice C.4 # PR #114). main.go + internal/broker/ # { config, session, oidc, issuer, server, # labels, sweeper, ratelimit }. §6.2 broker # binary complete; SCIM webhook stays in backlog. ``` ## Sub-Phases ### 6.0 — `demarkus-agent` verified + chart ✓ (merged PR #106) Existing `demarkus-agent` binary verified end-to-end (2 team worlds + 1 hub smoke test) and chart at `deploy/helm/demarkus-agent/` shipped. Stateless Deployment; ConfigMap holds TOML agent config; Secret holds per-host tokens; exec liveness probe; outbound-only (no Service). Three publish bugs fixed during verification (`publishIndex` status acceptance, `expected_version=-1` for idempotent re-publish, Makefile `make client` build). ### 6.1 — `demarkus-server` Helm chart ✓ (merged PR #107) + §6.3.D.2 follow-up pending `deploy/helm/demarkus-server/` — StatefulSet, 1 replica, `volumeClaimTemplates` (each world owns its PVC, never shared), exec probes against `/.well-known/agent-manifest.md`, `Service` type `LoadBalancer` `protocol: UDP`. Bootstrap path seeds the initial `admin` token via `protocol/token.Generate` + `AppendEntry` (Slice A). Cert-manager `Certificate` resource behind a flag. helm-unittest test files at `tests/` — initially shipped broken (the `statefulset_test.yaml` suite only loaded `statefulset.yaml`, so the StatefulSet template's `include "...tokens.yaml"` for `checksum/config` failed render and all 13 tests errored), then fixed in §6.3.D.1 PR #118 and wired into CI for the first time. **Known §6.1 chart gap (sidecar §6.3.D.2 scope):** the tokens Secret rendered by `templates/tokens.yaml` is MISSING the `helm.sh/resource-policy: keep` annotation + `helm.sh/hook: pre-install` hook. Earlier plan versions claimed the annotation was in place and "helm-unittest + kind integration test pin the upgrade-wipe regression" — neither claim was true. §6.3.D.2 bundles the chart fix with a kind upgrade-wipe regression test for both charts, so the property is verified end-to-end rather than re-asserted in prose. ### 6.2 — `demarkus-broker` binary (`tools/demarkus-broker/`) ✓ complete Slice B (single-world OIDC mint flow) **merged as PR #109** (commit `2a6aae6`, 2026-05-11). Slice C.1 (groups-claim authorization + AllowEmails carve-out + email canonicalization) **merged as PR #110** (commit `343f808`, 2026-05-12). Slice C.2 (expiry + drift sweeper with Lease-based leader election, RBAC-denied mint guard) **merged as PR #111** (commit `cb5f800`, 2026-05-12). Slice C.3 (`POST /tokens/:label/rotate` with re-login semantics + scope-frozen lifetime-reset asymmetry) **merged as PR #112** (commit `6ee0d8b`, 2026-05-12). Slice C.4 (per-subject + per-IP rate-limit middleware with shared subject buckets across the three `/tokens` routes, optional leftmost-XFF trust for the IP limiter, `Reserve()+Cancel()` for denial-doesn't-consume-budget) **merged as PR #114** (commit `e2933ff`, 2026-05-12). **§6.2 broker binary complete.** The §6.3.C `OIDC_CLIENT_SECRET` env-var override (`applyEnvOverrides` in `config.go:262`) is a one-line additive surface that does NOT reopen §6.2 — see §Decisions for the framing. #### Role & topology - **Issuance authority, not a request proxy.** Broker sits on the `demarkus login` path; clients then talk to world servers directly carrying the raw token. World servers stay identity-blind; broker never sees `mark://` requests. Capability model preserved. - **One broker per universe (cluster) by default.** Single OIDC client registration, single `worlds:` list in values, single issuance state Secret. - Broker SA holds a namespace-scoped `Role` + `RoleBinding` in **each world's namespace**, with `get/update` limited to **that world's `tokens` Secret only**. No `ClusterRole`. Slice C.2 sweeper adds `coordination.k8s.io/leases` `get/create/update` in the **broker namespace** for leader election; §6.3 chart bundles both into the broker SA. #### State Two distinct Kubernetes Secrets, never merged: | Secret | Lives in | Contents | Reader | |---|---|---|---| | `-tokens` | each world's namespace | TOML: `[tokens.