soul.demarkus.io/plans/universe-deployment.md/v14 draft reader meta

Plan: Universe Deployment

Ship the enterprise-grade deployment for demarkus. The deliverable is the deployment: a customer's ops team installs a Helm chart, runs a broker, runs an agent, and has a federated demarkus universe their org can use. Any company evaluating demarkus (their internal "POC") installs the same product an established customer runs in production. There is no separate "POC slice" — the slice mentality is rejected. We build it once, right, and customers trial the real thing.

Goal

Deliver a complete, supportable, production-grade Kubernetes deployment package for demarkus, including:

  1. A Helm chart for demarkus-server (one world).
  2. A Helm chart for demarkus-broker (OIDC token issuance + revocation).
  3. A Helm chart for demarkus-agent (hub aggregator, crawl-and-index).
  4. Reference topology examples (Argo CD ApplicationSet, Kustomize overlay).
  5. Backend-agnostic observability — structured slog emission + reference configs for Datadog, OTel Collector, Vector, Fluent Bit, Grafana Alloy.
  6. Customer-facing documentation — installation, security/threat model, operations, upgrade path, per-provider OIDC setup, observability recipes.
  7. A release pipeline producing images and chart releases consumable from GHCR.

The same artifacts power first-customer trial and steady-state operations.

Non-Goals (Phase 7+ territory)

  • Multi-replica worlds with shared storage (RWX / object backend).
  • Cross-cluster universe federation.
  • Operator with a World CRD.
  • Hosted / managed SaaS.
  • Non-markdown content.

Constraints

  • No core protocol/server changes. Period. Per prior precedent (Claude Code plugin, Obsidian plugin, feedback_plugin_scope.md). Observability is achieved by log derivation in a collector, not by adding /metrics or OTel SDK calls to the server. Note: in Slice A we promoted HashToken and the token-mint library to protocol/ (under protocol/auth.go and protocol/token/). These are additive helper relocations consumed by server + CLI + future broker; no wire-protocol or server behavior changed. Slice C.2 added protocol/token.ParseBytes in the same additive spirit — a read-side helper so the broker's drift sweeper can inspect a world's tokens.toml payload via the map shape rather than substring matching on serialized TOML.
  • Capability-based auth model is non-negotiable. The server never learns identity — only labels.
  • Markdown-only scope is non-negotiable.

Decisions (resolved during planning)

  • Health probes use exec, not HTTP. Both charts ship liveness/readiness exec probes that invoke the demarkus CLI fetching /.well-known/agent-manifest.md (always public per /architecture.md). No core change. Same pattern as redis-cli ping / pg_isready.
  • UDP port is a values knob. Default 6309, override via server.udpPort. Documented option: switch to 443 for VPN/middlebox-hostile networks (Cloudflare Warp Zero Trust, corp firewalls that filter non-standard UDP). Protocol default stays 6309.
  • Wildcard TLS via cert-manager DNS-01. Single *.<root> cert covers all worlds + broker hostnames. HTTP-01 cannot work for worlds (no HTTP); DNS-01 is the standard path for QUIC services.
  • DNS topology. One A record per world + one for the broker, all under the same wildcard zone. Each world exposed by Service: LoadBalancer protocol: UDP; broker by standard Ingress (HTTPS).
  • Hub aggregator is the existing demarkus-agent at client/cmd/demarkus-agent/. Already implements crawl + daemon subcommands with TOML config, multi-worker crawl, per-host token auth, per-server + aggregated index publishing. Verified end-to-end on 2026-05-11. Phase 5 is no longer a blocking prerequisite.
  • demarkus-agent lives in client/cmd/, not tools/. It's a protocol client (uses fedcrawl, fetch, tokens, links), not a utility. Memory rule sharpened: client/cmd/ = protocol clients (CLI, TUI, MCP, agent); tools/ = utilities (broker, future sync/ops binaries).
  • Token-mint library lives at protocol/token/, not tools/internal/token/ (revised mid-Slice-A after CodeRabbit review). protocol/ is reachable by both server/ (where demarkus-token CLI lives) and tools/ (where the broker will live), which tools/internal/ was not. protocol/ already owns the HashToken byte-shape contract, so it's the natural home for the on-disk token primitives that must round-trip identically across CLI mints and broker mints.
  • demarkus-token CLI stays at server/cmd/demarkus-token/ for now (rewired to import protocol/token). Moving the binary to tools/ was deferred to a follow-up that splits the release pipeline (server release currently ships demarkus-token; tools has no release cadence yet). Same applies to demarkus-publish — still at server/cmd/demarkus-publish/, deferred because hoisting it requires also hoisting server/internal/store/ out of internal-package scope.
  • Multi-OIDC. Broker speaks generic OIDC, not Google-specific. Provider behind a Verifier interface. Google validated first; Okta, Entra ID, Auth0 follow with config only.
  • Workload Identity (GKE) as a broker option. Values flag annotates the broker ServiceAccount with the GSA mapping. Off by default; on for GKE customers wanting no static SA keys. Not applicable to world servers.
  • Broker is HA. Multi-replica with resourceVersion optimistic concurrency on Secret writes, retry on conflict.
  • Observability is log-derived, backend-agnostic. demarkus already emits structured slog. Charts ship Datadog autodiscovery annotations + reference configs for OTel Collector, Vector, Fluent Bit, Grafana Alloy. Customer's SRE picks the agent/backend. Latency histograms deferred — depends on what current slog includes per request; revisit during trials if needed.
  • Backup/DR is documented, not built. The chart deliberately does not run backup CronJobs. Operations doc covers Velero, VolumeSnapshot, and demarkus-agent sync as DR options.
  • Image hosting: ghcr.io/latebit-io/{demarkus-server,demarkus-broker,demarkus-agent}.
  • Chart registry: OCI charts in GHCR.
  • Cosign signing: deferred to backlog. Half-day CI add when wanted.
  • Groups-claim case sensitivity (revised mid-Slice-C.1 after CodeRabbit back-and-forth). Group names match case-insensitively: AllowConfig.Groups is lowercased+trimmed at config load (same as Domains and Emails) and groupsMatch lowercases the claim's groups before compare. The validated IdP set (Google, Okta, Entra ID, Auth0) enforces case-insensitive group-name uniqueness, so distinct case-variant groups can't exist; a case-sensitive compare would silently fail when an operator writes "Engineering" but the IdP emits "engineering" after upstream normalization. Keycloak with deliberately distinct case-variant groups is out of scope; revisit if a customer asks. Domains and emails get the same lowercase-at-load + plain == treatment for symmetry.
  • Sweeper interval capped at 24h (Slice C.2). The defaultToken.expiresAfter typical is 24h; a sweep interval longer than that would let expired tokens linger past their ExpiresAt for up to a whole sweep cycle, defeating the short-lived-tokens identity-lifecycle model (§Revocation §3). validate() rejects with sweeper.interval must be <= 24h. Lease-election timings (15s lease / 10s renew / 2s retry) intentionally not exposed in YAML — the client-go defaults match our failover budget and adding knobs invites misconfiguration. Revisit if a customer needs sub-15s failover.
  • sweeper.disabled rather than sweeper.enabled (Slice C.2). YAML zero-value (false) is the production-safe default so an operator who omits the block gets the sweeper running. Naming the field positively (Enabled) would have made omission silently disable the janitor — exactly the wrong default for multi-replica deployments where expired tokens accumulate forever without it.
  • Rotate is re-login semantics, not refresh (Slice C.3). POST /tokens/:label/rotate re-runs worldAllows(world.Allow, claims) on every call, so a user removed from the world's allowlist between mint and rotate can NOT extend access via rotation. The bearer ID-token verification at the HTTP layer already proves the IdP identity is still active; the per-world predicate re-check catches in-org permission changes (group removed, domain renamed, email taken off carve-out). Picked over rotation-as-refresh because rotation extends the lifetime — without re-auth, a user whose IdP account stayed live could rotate forever even after losing world-scope authorization.
  • Rotate scope vs. lifetime asymmetry (Slice C.3). Scope (paths + operations) stays frozen to the issuance record on rotate: operator narrowing of DefaultToken.Paths between mint and rotate does NOT shrink the rotated token's reach, because the user shouldn't be surprised by reduced access mid-session — they re-login to pick up new scope. Lifetime resets to now + DefaultToken.ExpiresAfter from the operator's CURRENT config: operator tightening of expiry DOES apply on rotate, because shorter lifetimes are an explicit security-tightening lever and rotation must not bypass it. The asymmetry is intentional and pinned by TestRotateLabelPreservesIssuanceScope + TestRotateLabelLifetimeUpdatesToCurrentConfig.
  • Rotate sequence: mint new, then revoke old (Slice C.3). On revoke failure the new token is returned anyway with a wrapped soft error; the sweeper retires the orphan old label on expiry (or via drift if an operator hand-cleans). Picked over revoke-then-mint because that alternative leaves the user with no token if mint then fails, forcing a full re-login — worse UX than a brief two-valid-tokens window for the same user. Mirrors the Slice B partial-mint convention: (MintResult{Label:""}, err) for hard failures, (MintResult{non-empty}, err) for soft partial success.
  • Middleware-not-handler auth + rate limit (Slice C.4). The three authed /tokens routes go through requireAuth → subjectRateLimit → handler; /auth/login goes through ipRateLimit → authLogin; /auth/callback, /healthz, /readyz stay middleware-free. The old per-handler s.authenticate(w, r) was extracted into the requireAuth middleware that stashes verified Claims on r.Context() via a typed key; handlers read claims via claimsFromCtx. The alternative — keeping auth in handlers and adding rate-limit inside each one — would either double-verify the bearer (one verify for the limit key, one in the handler) or hide the limiter inside each handler, opaque to the route-registration block. Middleware composition keeps the per-route policy visible in Routes().
  • One shared subject bucket across the three /tokens routes (Slice C.4). subjectRateLimit keys on hashSubject(claims.Subject) and the same registry covers GET /tokens, DELETE /tokens/:label, and POST /tokens/:label/rotate. Picked over three per-route buckets so a misbehaving client cannot multiply effective throughput by fanning out (3 routes × 10/min would give 30/min effective rather than the operator-intended 10/min). Pinned by TestRateLimitTokensSharedBucketAcrossRoutes.
  • Reserve() + Cancel() on denial, not Allow() (Slice C.4). Functional equivalence — neither pattern consumes budget on denial — but Reserve().Delay() gives the precise wait time, which we surface as Retry-After with a 1s minimum floor. (Retry-After: 0 reads as "retry immediately" to aggressive clients and would defeat the limiter.) Pinned by TestRateLimitRegistryDenialDoesNotConsumeBudget (50 denials in a tight loop, then 150ms regen window allows the 51st request) and the Retry-After assertions on the integration 429 tests.
  • trustForwardedFor: false default in the binary (Slice C.4). The broker behind an Ingress sees the controller's IP in r.RemoteAddr, so the per-IP limiter on /auth/login collapses into one bucket for every client unless we honor XFF. But trusting XFF when NOT behind a proxy lets an attacker rotate the header to bypass per-IP limits. The binary ships safe; chart-side §6.3.A inverts the default to true (the chart's "behind an Ingress" assumption holds in deployment). TestRateLimitLoginIPIgnoresForwardedForByDefault pins the default-untrusted behavior at the binary level; TestRateLimitLoginIPCrossIPIsolation pins the trust-enabled behavior.
  • rateLimit.disabled not enabled (Slice C.4). Same shape as sweeper.disabled. Zero-value (false) gives the production-safe behavior so an operator who omits the rateLimit: block in their values file still gets the protection. Defaults applied at validate-time: 10/min subject burst 5, 20/min IP burst 5. Field validation skipped entirely when Disabled: true so an operator opting out can leave the per-route knobs empty.
  • Per-replica unbounded registry, by design (Slice C.4). The rateLimitRegistry keeps one *rate.Limiter per key with no TTL eviction and no max-entry cap. CodeRabbit flagged this as a memory-DoS vector; rejected after re-examining the threat model: (a) subjectReg growth is bounded by IdP user count (every key required a successful requireAuth pass, which means a valid IdP-issued token), (b) loginReg growth is bounded by the deployment posture (with trustForwardedFor=false the key is r.RemoteAddr, bounded by clients reaching the listener; with trustForwardedFor=true plus a correctly-configured XFF-stripping Ingress, the key space is bounded by real client IPs), and (c) TTL/LRU eviction makes the property worse under the stated attack: an attacker rotating keys faster than the TTL gets fresh buckets indefinitely, and LRU eviction during attack evicts legitimate users' older buckets first while the attacker's fresh buckets stay in the map. The right Phase-7+ fix lives at a different layer (cluster-shared rate limiter so per-replica eviction can't reset attacker state, or per-IP rate-limit annotations at the ingress controller). Realistic worst-case sizing is ~500KB at ~10k authenticated subjects — not memory-DoS-shaped. Mirrors the C.2 "sweep is unbounded" pushback in shape.
  • OIDC_CLIENT_SECRET env-var override (§6.3.C, additive to §6.2). broker.LoadConfig.applyEnvOverrides() (tools/demarkus-broker/internal/broker/config.go:262) lets the OAuth client secret come from the environment instead of the on-disk config file. Env wins over file when both set; empty env is treated as unset so an accidentally-cleared variable can't blank out a file-supplied value. Production deployments keep the OAuth secret in an externally-managed Kubernetes Secret (External Secrets Operator, Sealed Secrets, Vault) mounted via secretKeyRef, instead of baking it into the chart-rendered config Secret where it would leak into helm release history. This is a one-line escape hatch for the production-secret-ref deployment shape, NOT a §6.2 reopening — §6.2 broker binary complete framing stands. Pinned by TestLoadConfigOIDCClientSecretEnvOverride (4-row table).
  • helm-unittest pinned to v0.6.2 in CI (§6.3.D.1). v1.0+ uses platformHooks in plugin.yaml which needs helm v3.16+; Fritz's local helm is v3.13.2. Pinning v0.6.2 in CI matches the local-dev validation surface so test behavior is identical between developer machines and CI. Bump together when helm itself bumps. Helm pinned to v3.13.2 alongside.
  • test-broker Go CI job folded into §6.3.D.1. The §6.2 broker binary had zero CI coverage at the start of §6.3 chart work — .github/workflows/ci.yml only filtered protocol/server/client, not tools/. Adding helm-unittest plumbing while leaving the binary CI-uncovered would have been incoherent. New test-broker job runs Go test/vet/lint/build for tools/demarkus-broker/, with working-directory: tools/demarkus-broker (narrowed past the broader tools/ scope to avoid lint failures on the package-marker file tools/tools.go). Path filter scoped to tools/demarkus-broker/** + tools/go.mod + tools/go.sum + protocol/** — broker CI does NOT run on unrelated tool changes.

Open Questions

  1. First customer trial. Nesto (*.library.nesto.ca) is path-B — trial waits for product. Trial runbook lands at /trials/nesto.md when scoping starts.
  2. CLI relocation to tools/. Follow-up to Slice A. Requires tools/.goreleaser.yml, a release-tools job in the workflow, an install.sh block fetching the tools archive, and updates to plugin scripts + helm image build. Acceptable to defer until the release-pipeline work has its own dedicated PR.
  3. §6.3.D.2 sidecar PR remains open: add helm.sh/resource-policy: keep + helm.sh/hook: pre-install to the §6.1 demarkus-server tokens.yaml Secret (the §6.1 prose has claimed this is in place since §6.1 shipped, but the chart never had the annotation), and add a kind upgrade-wipe regression test in CI for BOTH charts. Tracked as immediate next work, NOT folded into §6.3 chart proper because the §6.1 chart fix is independent scope.

Repository Layout

Reflects state as of §6.3.D.1 merge (2026-05-13):

deploy/
  helm/
    demarkus-server/         # one-world chart (Phase 6.1) — shipped PR #107
      tests/                 # helm-unittest suites — wired into CI by §6.3.D.1 PR #118
    demarkus-broker/         # OIDC token broker chart (Phase 6.3) — shipped via PR #115, #116, #117
      tests/                 # helm-unittest suites — shipped by §6.3.D.1 PR #118
    demarkus-agent/          # crawl/index agent chart (Phase 6.0) — shipped PR #106
  k8s/
    examples/
      applicationset.yaml    # Argo CD ApplicationSet over a worlds: list
      kustomize-overlay/     # Kustomize alternative
  observability/
    datadog/                 # autodiscovery annotations + dashboard JSON
    otel-collector/          # collector config recipes
    vector/                  # vector config recipes
    fluent-bit/              # fluent-bit parser + filter recipes
  scripts/                   # operator helpers (cert pre-check, MTU probe, etc.)

protocol/
  auth.go                    # HashToken — sha256-<hex> contract (Slice A)
  token/                     # Generate, ReadFile, AppendEntry, WriteFile,
                             # FormatEntry, flock helpers (Slice A) +
                             # AppendBytes, RemoveBytes in-memory helpers
                             # (Slice B) + ParseBytes read-side helper
                             # (Slice C.2) for the broker's drift sweep.

server/cmd/
  demarkus-server/           # the server binary (existing)
  demarkus-token/            # token mint CLI — uses protocol/token
  demarkus-publish/          # publish CLI (existing; deferred move)

client/cmd/
  demarkus-agent/            # protocol client — federation crawler (existing)

tools/
  demarkus-broker/           # broker binary (Slice B PR #109, Slice C.1 PR #110,
                             # Slice C.2 PR #111, Slice C.3 PR #112, Slice C.4
                             # PR #114). main.go + internal/broker/
                             # { config, session, oidc, issuer, server,
                             #   labels, sweeper, ratelimit }. §6.2 broker
                             # binary complete; SCIM webhook stays in backlog.

Sub-Phases

6.0 — demarkus-agent verified + chart ✓ (merged PR #106)

Existing demarkus-agent binary verified end-to-end (2 team worlds + 1 hub smoke test) and chart at deploy/helm/demarkus-agent/ shipped. Stateless Deployment; ConfigMap holds TOML agent config; Secret holds per-host tokens; exec liveness probe; outbound-only (no Service). Three publish bugs fixed during verification (publishIndex status acceptance, expected_version=-1 for idempotent re-publish, Makefile make client build).

6.1 — demarkus-server Helm chart ✓ (merged PR #107) + §6.3.D.2 follow-up pending

deploy/helm/demarkus-server/ — StatefulSet, 1 replica, volumeClaimTemplates (each world owns its PVC, never shared), exec probes against /.well-known/agent-manifest.md, Service type LoadBalancer protocol: UDP. Bootstrap path seeds the initial admin token via protocol/token.Generate + AppendEntry (Slice A). Cert-manager Certificate resource behind a flag. helm-unittest test files at tests/ — initially shipped broken (the statefulset_test.yaml suite only loaded statefulset.yaml, so the StatefulSet template's include "...tokens.yaml" for checksum/config failed render and all 13 tests errored), then fixed in §6.3.D.1 PR #118 and wired into CI for the first time.

Known §6.1 chart gap (sidecar §6.3.D.2 scope): the tokens Secret rendered by templates/tokens.yaml is MISSING the helm.sh/resource-policy: keep annotation + helm.sh/hook: pre-install hook. Earlier plan versions claimed the annotation was in place and "helm-unittest + kind integration test pin the upgrade-wipe regression" — neither claim was true. §6.3.D.2 bundles the chart fix with a kind upgrade-wipe regression test for both charts, so the property is verified end-to-end rather than re-asserted in prose.

6.2 — demarkus-broker binary (tools/demarkus-broker/) ✓ complete

Slice B (single-world OIDC mint flow) merged as PR #109 (commit 2a6aae6, 2026-05-11). Slice C.1 (groups-claim authorization + AllowEmails carve-out + email canonicalization) merged as PR #110 (commit 343f808, 2026-05-12). Slice C.2 (expiry + drift sweeper with Lease-based leader election, RBAC-denied mint guard) merged as PR #111 (commit cb5f800, 2026-05-12). Slice C.3 (POST /tokens/:label/rotate with re-login semantics + scope-frozen lifetime-reset asymmetry) merged as PR #112 (commit 6ee0d8b, 2026-05-12). Slice C.4 (per-subject + per-IP rate-limit middleware with shared subject buckets across the three /tokens routes, optional leftmost-XFF trust for the IP limiter, Reserve()+Cancel() for denial-doesn't-consume-budget) merged as PR #114 (commit e2933ff, 2026-05-12). §6.2 broker binary complete.

The §6.3.C OIDC_CLIENT_SECRET env-var override (applyEnvOverrides in config.go:262) is a one-line additive surface that does NOT reopen §6.2 — see §Decisions for the framing.

Role & topology

  • Issuance authority, not a request proxy. Broker sits on the demarkus login path; clients then talk to world servers directly carrying the raw token. World servers stay identity-blind; broker never sees mark:// requests. Capability model preserved.
  • One broker per universe (cluster) by default. Single OIDC client registration, single worlds: list in values, single issuance state Secret.
  • Broker SA holds a namespace-scoped Role + RoleBinding in each world's namespace, with get/update limited to that world's tokens Secret only. No ClusterRole. Slice C.2 sweeper adds coordination.k8s.io/leases get/create/update in the broker namespace for leader election; §6.3 chart bundles both into the broker SA.

State

Two distinct Kubernetes Secrets, never merged:

Secret Lives in Contents Reader
<world>-tokens each world's namespace TOML: [tokens.<label>] hash=, paths=, operations=, expires= demarkus-server
<broker>-issuances broker namespace JSON: label → {email, world, paths, operations, issued_at, expires} + secondary index email → [labels] broker only

World servers see only hashes. Email and identity live in broker state alone. The broker-side <broker>-issuances Secret carries helm.sh/resource-policy: keep (shipped in §6.3.A); the world-side <world>-tokens Secret needs the same annotation + a pre-install hook on the §6.1 chart — sidecar §6.3.D.2 fixes this.

Plus one cluster-scoped coordination object (Slice C.2):

Resource Lives in Contents Reader
demarkus-broker-sweeper Lease broker namespace holderIdentity, renewTime, leaseDurationSeconds leader-election library only

Per-world authorization (values schema as shipped in C.1)

worlds:
  - name: team-a
    namespace: team-a
    tokensSecret: team-a-tokens
    allow:
      domains: ["nesto.ca"]
      groups: ["engineering"]
      emails: ["alice@example.com"]
    defaultToken:
      paths: ["/team-a/*"]
      operations: ["read", "publish"]
      expiresAfter: 24h
sweeper:
  disabled: false
  interval: 5m
  leaseName: demarkus-broker-sweeper
rateLimit:
  disabled: false
  tokens:
    perMinute: 10
    burst: 5
  login:
    perMinute: 20
    burst: 5
  trustForwardedFor: false   # binary default; chart §6.3.A inverts to true

WorldConfig.Allow is a nested struct. All three lists lowercased+trimmed at config load with empty entries rejected. Match is case-insensitive everywhere. The authorization predicate (worldAllows):

  1. All three lists empty → match (back-compat).
  2. Email in Allow.Emails → match (per-user carve-out).
  3. Only Emails was configured and step 2 didn't fire → reject.
  4. domainMatches AND groupsMatch.

Mint and RotateLabel canonicalize claims.Email (trim + lowercase) once at the top before authorization runs.

Token-mint library

Broker imports github.com/latebit/demarkus/protocol/token. Slice A on-disk + pure helpers (Generate, AppendEntry, WriteFile, ReadFile, FormatEntry). Slice B in-memory helpers (AppendBytes, RemoveBytes) for k8s-Secret round-trips. Slice C.2 in-memory read-side helper (ParseBytes) for drift detection — empty input gives an empty File with a non-nil Tokens map. Slice C.3 refactored Issuer.mintForWorld to take explicit paths, operations []string parameters so Mint and RotateLabel share the cross-world-collision + rollback machinery.

Revocation

Three triggers, one cleanup index (the issuances Secret):

  1. User-initiated (DELETE /tokens/:label or POST /tokens/:label/rotate, both bearer-authed via OIDC ID token). Owner check + revokeIssuance (Slice C.2 helper extracted from Revoke; shared with the sweeper). Rotate re-runs worldAllows before minting (re-login semantics).
  2. Expiry sweeper (Slice C.2, leader-elected via Lease). Reads issuances once per tick, groups non-expired entries by world, reads each affected world's tokens Secret once via ParseBytes, retires (expired ∪ drifted) entries via revokeIssuance.
  3. Identity lifecycle — Default: short-lived tokens (defaultToken.expiresAfter ~24h). SCIM webhook backlogged.

SIGHUP intentionally not implemented in C.2/C.3/C.4 — revocation latency equals kubelet Secret-mount propagation + server re-read behavior. Verified during §6.3 chart work as acceptable for the deployment posture (kubelet's default ~60s propagation is within the operator-visible bound).

Cleanup edge cases

  • Orphan in tokens.toml (admin-minted via CLI): broker never claims it.
  • Orphan in issuances Secret (admin hand-deleted from world tokens Secret): Slice C.2 sweeper detects drift via ParseBytes and prunes.
  • Issuance for an unconfigured world: conservative skip (log, never silently drop). Verified by TestSweepSkipsUnconfiguredWorld and TestRotateLabelMissingWorldErrors.
  • In-flight request after revoke/rotate: a raw token already authenticated on a live QUIC connection completes its current request. Property of model.
  • Partial mint across multiple worlds: failed world has zero state in either Secret; prior worlds in the same Mint stay committed. Verified by TestMintRBACDeniedNoPartialState.
  • Soft-partial rotate (Slice C.3): mint succeeded but old revoke failed → both labels coexist until sweeper retires the old. User gets 200 + new token; operator sees WARN log. Verified by TestRotateTokenSoftPartial.

Routes (HTTP, behind Ingress) — middleware chains as shipped in C.4

  • GET /auth/login — ipRateLimit → authLogin. OIDC redirect entry point; sets signed state cookie (HMAC-SHA256, HttpOnly+Secure+SameSite=Lax, path-scoped to /auth/callback, default 5-min TTL).
  • GET /auth/callback — no middleware. State cookie HMAC + 5-min TTL is the natural rate gate. Verifies state cookie, exchanges code, mints tokens for every qualifying world, returns JSON {world → raw_token} once. No broker session cookie.
  • GET /tokens — requireAuth → subjectRateLimit → listTokens. Bearer-authed (OIDC ID token).
  • DELETE /tokens/:label — same chain. Bearer-authed.
  • POST /tokens/:label/rotate — same chain (Slice C.3). Bearer-authed. Owner check + worldAllows re-validation against current Allow config.
  • GET /healthz, GET /readyz — no middleware. Probes must succeed without auth or rate-limit gating.

Rationale for skipping a broker session: the CLI is the primary /tokens consumer and already handles token lifetimes; the browser only sees the one-time JSON callback response. The Verifier interface exposes VerifyIDToken(ctx, raw) (Claims, error) so handlers authenticate bearer tokens uniformly.

OIDC providers

Provider behind a Verifier interface (AuthCodeURL, Exchange, VerifyIDToken). Initial implementation wraps coreos/go-oidc/v3 + golang.org/x/oauth2. Follow-ons (Okta, Entra ID, Auth0) are config only. Groups claim sourcing (Slice C.1 scope): ID-token only; userinfo-based groups in backlog.

HA

Multi-replica. Issuance writes to k8s Secrets use resourceVersion optimistic concurrency with retry-on-conflict. Sweeper (C.2) uses a coordination.k8s.io/Lease for leader election via k8s.io/client-go/tools/leaderelection; ReleaseOnCancel=true for sub-second handoff on rolling restarts. Mint/List/Revoke/Rotate handlers stay on every replica. Rate limit (C.4) is per-replica per the rateLimitRegistry doc-comment — effective rate seen by an abusive client is up to N× the configured value in a multi-replica deployment. Cluster-shared rate limiting backlogged.

Slice B test surface (shipped)

87% line coverage. Mint flow + state cookie + config + issuer (resourceVersion retry, label collision retry, mint ordering) + server (clock decoupling).

Slice C.1 test surface (shipped — PR #110)

  • 15-row TestMintAuthorizationPredicate covers every (Domains × Groups × Emails) × accept/reject cell.
  • TestMintCanonicalizesEmail — pins email trim+lowercase at top of Mint.
  • TestLoadConfig rows for lowercase normalization + empty-entry rejection on all three allow dimensions.
  • TestDeleteTokenNotOwner — owner-check 403.

Slice C.2 test surface (shipped — PR #111)

TestSweepRetiresExpiredKeepsFresh, TestSweepPrunesDrift, TestSweepSkipsUnconfiguredWorld, TestSweepEmptyIssuancesIsNoop, TestSweeperLeaderElection (two replicas + handoff), TestMintRBACDeniedNoPartialState, TestLoadConfig rows for sweeper.interval defaults + cap + negative-rejection. Coverage 81.0%.

Slice C.3 test surface (shipped — PR #112)

Issuer-layer (8): TestRotateLabelHappyPath, TestRotateLabelRejectsUnverifiedEmail, TestRotateLabelNotOwner, TestRotateLabelUnknownLabel, TestRotateLabelPreservesIssuanceScope, TestRotateLabelLifetimeUpdatesToCurrentConfig, TestRotateLabelReauthorizesAgainstCurrentAllow, TestRotateLabelMissingWorldErrors. HTTP-layer (5): success, not-owner, no-longer-authorized, not-found, soft-partial, unauthenticated. Coverage 81.1%.

Slice C.4 test surface (shipped — PR #114)

Registry-level (ratelimit_test.go):

  • TestRateLimitRegistryNilWhenDisabledOrZero (5-row table): nil for non-positive perMinute or non-positive burst. Pins the "disabled = no enforcement" contract.
  • TestRateLimitRegistryAllowsBurstThenDenies: burst+1 → first two allowed, third denied, retryAfter >= 1s (1s floor).
  • TestRateLimitRegistryCrossKeyIsolation: k1 exhausted, k2 still allowed.
  • TestRateLimitRegistryDenialDoesNotConsumeBudget: 50 denials in a tight loop + 150ms regen recovers. Pins Reserve()+Cancel() no-consume-on-denial.

ClientIP / middleware misuse:

  • TestServerClientIP (7-row table): untrusted XFF ignored, trusted single-hop, trusted multi-hop takes leftmost, trusted-but-empty falls back, trusted-whitespace-only falls back, IPv6, garbage RemoteAddr.
  • TestSubjectRateLimitMissingClaimsIs500: regression guard for misuse — 500 + ERROR log rather than silent enforcement disable.

HTTP integration:

  • TestRateLimitTokensExhaustsAndReturns429WithRetryAfter: burst=2 exhaust → 3rd returns 429 + non-zero Retry-After.
  • TestRateLimitTokensCrossSubjectIsolation: alice exhausts; bob (different Subject via multi-bearer verifier) still passes.
  • TestRateLimitTokensSharedBucketAcrossRoutes: 1 GET + 1 DELETE consumes burst 2; 3rd on a different route 429s. Pins single-bucket-for-three.
  • TestRateLimitLoginIPExhaustsAndReturns429WithRetryAfter: XFF trusted, same XFF entry 3× → 3rd 429s.
  • TestRateLimitLoginIPCrossIPIsolation: XFF trusted, two different spoofs → distinct buckets.
  • TestRateLimitLoginIPIgnoresForwardedForByDefault: XFF untrusted, three different XFF spoofs collapse to one bucket.
  • TestRateLimitDisabledBypasses: 20 rapid requests all 200 when rateLimit.disabled: true.

Config-level (config_test.go): defaults applied when block omitted (10/5 + 20/5), operator overrides honored, disabled: true bypasses field defaults, four "negative value rejected" rows. Coverage 83.4% (up from C.3's 81.1%).

6.3 — demarkus-broker Helm chart ✓ (A/B/C merged; D.1 merged; D.2 sidecar pending)

Multi-replica HA at deploy/helm/demarkus-broker/, shipped across four sub-PRs:

  • §6.3.A ✓ merged 2026-05-13 (PR #115, commit c71f499) — chart skeleton: Deployment + Service + config Secret + issuances Secret + ServiceAccount + values schema. Cookie-key lookup-preserve across upgrades. Issuances Secret seeded with helm.sh/resource-policy: keep. OIDC client-secret mutual-exclusion + existingSecretRef.key required guards at template-render time. PodSecurityContext + container hardening (runAsNonRoot, readOnlyRootFilesystem, seccompProfile: RuntimeDefault, capabilities: drop ["ALL"], allowPrivilegeEscalation: false). TopologySpreadConstraints by kubernetes.io/hostname, ScheduleAnyway. checksum/config pod annotation. rateLimit.trustForwardedFor: true chart default (inverts binary).
  • §6.3.B ✓ merged 2026-05-13 (PR #116, commit 175e230) — RBAC + NetworkPolicy + PDB. Per-world Role + RoleBinding loop, verbs scoped to get/update on the named tokens Secret (NOT create). Broker-namespace Role for coordination.k8s.io/leases get/create/update (create necessarily unscoped per k8s RBAC; documented) + secrets get/update on the issuances Secret. NetworkPolicy default-on with ingress from the configured ingress-controller namespace and egress allowing DNS to kube-system + TCP 443 unrestricted. PDB default minAvailable: 1. ServiceAccount Workload Identity annotation plumbing.
  • §6.3.C ✓ merged 2026-05-13 (PR #117, commit 599f63a) — Ingress + cert-manager Certificate templates + oidc.existingSecretRef deployment path. Three Ingress TLS modes mutex-guarded at render-time. OIDC_CLIENT_SECRET env-var injected via secretKeyRef when existingSecretRef path is taken; rendered config.yaml leaves clientSecret blank. Binary additive surface: applyEnvOverrides() in tools/demarkus-broker/internal/broker/config.go — env wins over file, empty env is unset, 4-row TestLoadConfigOIDCClientSecretEnvOverride pins all rows. README + NOTES.txt expanded.
  • §6.3.D.1 ✓ merged 2026-05-13 (PR #118) — helm-unittest test files for the broker chart at deploy/helm/demarkus-broker/tests/ (49 tests across 7 suites), CI plumbing wiring helm-unittest into PR runs for BOTH demarkus-server and demarkus-broker charts, and a folded-in test-broker Go CI job (tools/demarkus-broker/) closing the §6.2 binary-CI gap. Path-filter narrowed to broker-specific paths to keep unrelated tool changes from triggering broker CI. helm-unittest pinned to v0.6.2; both new jobs timeout-minutes: 20. Fixed a real prior-work bug in deploy/helm/demarkus-server/tests/statefulset_test.yaml that had silently errored all 13 §6.1 tests since the day it was written (template-include issue masked by helm-unittest never running in CI).
  • §6.3.D.2 (sidecar, pending) — kind upgrade-wipe regression test in CI for BOTH charts, bundled with the §6.1 helm.sh/resource-policy: keep + helm.sh/hook: pre-install fix on deploy/helm/demarkus-server/templates/tokens.yaml. The §6.1 chart's claim that the keep annotation is set on the tokens Secret has been false since §6.1 shipped; D.2 fixes the chart AND adds a kind harness that proves the upgrade-wipe property end-to-end rather than re-asserting it in prose. Kind test shape: helm install chart, kubectl-patch the rendered Secret to seed fake data, helm upgrade with a values change, assert seeded data persists. Broker pod doesn't need to be ready for the property to hold — kind harness can skip OIDC discovery via a values stub.

§6.3 chart test surface (shipped — PR #118)

helm-unittest test files at deploy/helm/demarkus-broker/tests/:

  • deployment_test.yaml (12 tests, template: deployment.yaml): replicas + RollingUpdate strategy, image tag default = AppVersion, checksum/config annotation regex-pinned, pod-level securityContext (runAsNonRoot, runAsUser/Group: 65532, fsGroup: 65532, seccompProfile: RuntimeDefault), container securityContext (readOnlyRootFilesystem, allowPrivilegeEscalation: false, capabilities: drop ALL), HTTP probes on /healthz + /readyz, probes omitted when disabled, POD_NAME env from fieldRef, OIDC_CLIENT_SECRET env ABSENT when cleartext clientSecret only, OIDC_CLIENT_SECRET env present via secretKeyRef when existingSecretRef.name set, topologySpread matchLabels populated from selector labels (NOT the empty {} in values), container args reference /etc/demarkus-broker/config.yaml.
  • rbac_test.yaml (7 tests): broker-ns Role + RoleBinding render in brokerNamespace, leases rule pinned to get/create/update on configured leaseName, secrets rule pinned to get/update on issuancesSecret (regression-guarded against create/delete/list drift), per-world template emits N×2 resources for N worlds, per-world Role verbs get/update only (NOT create/delete), per-world RoleBinding subject references broker SA in release namespace, rbac.create=false skips all.
  • networkpolicy_test.yaml (7 tests): default-on, policyTypes includes both Ingress AND Egress, ingress scoped to ingress-nginx namespace by default + override, egress DNS to kube-system on UDP+TCP 53, egress TCP 443 unrestricted (no to: selector), disabled-state.
  • secret-config_test.yaml (8 tests): rendered Secret metadata.namespace explicit, rate-limit defaults rendered (trustForwardedFor: true, 10/5, 20/5), sweeper defaults rendered, existingSecretRef leaves clientSecret blank, clientSecret cleartext rendered verbatim, fail-render mutex (both set), fail-render required (neither set), fail-render existingSecretRef.key blank, world with no allow block renders empty arrays for all three dimensions.
  • secret-issuances_test.yaml (4 tests): empty Opaque Secret + data: {}, helm.sh/resource-policy: keep annotation, metadata.namespace tracks brokerNamespace helper, server.brokerNamespace override moves the Secret.
  • ingress_test.yaml (8 tests): both absent by default, host-required fail-render, TLS mode mutex fail-render, existingSecret references provided Secret, certManager-mode Certificate emitted with correct dnsName/secretName/issuerRef, Ingress in certManager mode references chart-generated TLS Secret, ingressClassName override propagates.
  • pdb_test.yaml (3 tests): default minAvailable: 1 + selector labels, override propagates, disabled-state.

CI plumbing (.github/workflows/ci.yml):

  • detect-changes filter adds broker (tools/demarkus-broker/** + tools/go.mod + tools/go.sum + protocol/**) and charts (deploy/helm/**).
  • test-broker job — Go test/vet/lint/build for tools/demarkus-broker/. Closes the §6.2 binary-CI gap.
  • test-charts job — helm lint for all three charts + helm unittest for server + broker.

6.4 — Universe topology examples

ApplicationSet + Kustomize overlay.

6.5 — Observability recipes

Per-backend configs in deploy/observability/. Slice C.2 added sweeper-side log lines (broker: swept, broker: sweep failed, broker: sweeper observing new leader, broker: sweeper lost leadership); Slice C.3 added rotate-side log lines; Slice C.4 added rate-limit log lines (broker: rate limit exceeded with route, subject or ip, retryAfter fields) — same audit-log routing target.

6.6 — Documentation suite

/deployment/*.md + per-chart READMEs.

6.7 — Release pipeline

GHCR images + OCI charts. Cosign deferred. Also folds in the deferred CLI relocation (demarkus-token, demarkus-publish) from server/cmd/ to tools/.

Sequencing

  1. Slice A — token-mint library ✓ merged 2026-05-11 (PR #108).
  2. 6.0 chart ✓ merged 2026-05-11 (PR #106).
  3. 6.1 server chart ✓ merged 2026-05-11 (PR #107). §6.3.D.2 sidecar pending for the missing resource-policy: keep.
  4. 6.2 broker binary — Slice B ✓ merged 2026-05-11 (PR #109, commit 2a6aae6).
  5. 6.2 broker binary — Slice C, split into four landed PRs:
    • C.1 ✓ merged 2026-05-12 (PR #110, commit 343f808) — groups-claim + AllowEmails authorization.
    • C.2 ✓ merged 2026-05-12 (PR #111, commit cb5f800) — expiry + drift sweeper with coordination.k8s.io/Lease leader election.
    • C.3 ✓ merged 2026-05-12 (PR #112, commit 6ee0d8b) — POST /tokens/:label/rotate with re-login semantics + scope-frozen/lifetime-reset asymmetry.
    • C.4 ✓ merged 2026-05-12 (PR #114, commit e2933ff) — per-subject + per-IP rate limit middleware via golang.org/x/time/rate, in-memory per-replica.
    • SCIM webhook stays in backlog.
  6. 6.3 broker chart ✓ four-slice trajectory all merged 2026-05-13:
    • 6.3.A ✓ PR #115, commit c71f499 — chart skeleton + security defaults.
    • 6.3.B ✓ PR #116, commit 175e230 — RBAC + NetworkPolicy + PDB.
    • 6.3.C ✓ PR #117, commit 599f63a — Ingress + cert-manager Certificate + oidc.existingSecretRef + OIDC_CLIENT_SECRET env-var override.
    • 6.3.D.1 ✓ PR #118 — helm-unittest test files for broker chart, CI plumbing for both charts, folded-in test-broker Go CI job.
    • 6.3.D.2 sidecar — kind upgrade-wipe regression for both charts + §6.1 chart resource-policy: keep fix. NEXT.
  7. 6.4 topology examples.
  8. 6.5 observability recipes.
  9. 6.6 docs — incremental throughout.
  10. 6.7 release pipeline — final.

Rough effort: ~3-5 days of focused work remaining (§6.3.D.2 sidecar + §6.4 examples + §6.5 recipes + §6.6 docs + §6.7 release pipeline).

Backlog (deferred, easy to add later)

  • Cosign signing of images + chart releases. Half-day CI add.
  • Latency log-enrichment (duration_ms field on request slog lines). Tiny additive change.
  • SCIM lifecycle webhook on the broker. Optional add when a customer asks.
  • CLI relocation to tools/ (demarkus-token, demarkus-publish). Bundled with 6.7.
  • Userinfo-based groups in the broker Verifier. Slice C.1 ships ID-token-only; AllowEmails is the documented workaround.
  • Case-sensitive group matching for Keycloak. Slice C.1 chose case-insensitive for the validated IdP set.
  • Configurable lease timings (sweeper.leaseDuration, renewDeadline, retryPeriod). C.2 hardcodes the client-go defaults.
  • SIGHUP on revoke + sweep + rotate to shrink the revoke-to-effect window. Needs pods/exec RBAC across world namespaces.
  • Sweeper backing-store migration when the issuances Secret hits the ~1MB / ~5000-issuance k8s storage ceiling.
  • Cluster-shared rate limiter (Slice C.4 follow-up). Backend (Redis / memcached / k8s Lease-coordinated tokens) so a multi-replica broker enforces one budget across all replicas, instead of N× the configured rate. Required if a customer's failure model is "an attacker who can spread requests across replicas" rather than "a noisy legitimate user."
  • Idle-key GC for the rate-limit registry (Slice C.4 follow-up). Background goroutine evicting limiters whose token bucket is full and whose lastSeen exceeds a TTL. Only worth adding if a customer hits a sized-up deployment with millions of distinct authenticated subjects over the broker's uptime; per the C.4 doc-comment, ~10k subjects is ~500KB. Eviction does NOT defend against attacker-driven inflation — that wants cluster-shared rate limiting upstream of the eviction point.
  • Timeouts on the existing test-protocol/test-server/test-client CI jobs. §6.3.D.1 added timeout-minutes: 20 to the new test-broker and test-charts jobs per CodeRabbit feedback; the older jobs still rely on the GitHub Actions default (360 min). One-line cleanup.

Risks

  • Observability via logs ceiling. If customers want signals not derivable from current slog, we hit a wall. Mitigation: log enrichment is a small additive change.
  • Broker secret-write blast radius. Mitigated by namespace-scoped Roles, audit log, optional NetworkPolicy. Per-world Role is non-negotiable. TestMintRBACDeniedNoPartialState validates clean failure.
  • OIDC provider coupling. First impl is Google; structure so second provider is a one-day add.
  • Token revocation in-flight latency. Revocation latency = kubelet Secret propagation + whatever the server does on re-mount. Acceptable for the deployment posture (~60s default kubelet propagation).
  • Rotate transient two-tokens window. Mint-then-revoke means both labels are briefly valid; widens to next sweep cycle on soft-partial. Documented; verified by TestRotateTokenSoftPartial.
  • §6.1 chart upgrade-wipe gap. The demarkus-server tokens Secret rendered by deploy/helm/demarkus-server/templates/tokens.yaml is MISSING the helm.sh/resource-policy: keep annotation + helm.sh/hook: pre-install hook that earlier plan versions claimed were in place. Without them, helm upgrade of a §6.1 release can recreate the Secret with a fresh admin token, breaking every issued tokens.toml entry the broker has minted into the world's Secret. §6.3.D.2 sidecar fixes the chart AND adds a kind upgrade-wipe regression test that pins the property end-to-end. The broker-side issuances Secret already carries the annotation correctly (shipped in §6.3.A), so this gap is server-side only.
  • §6.1 chart-test plumbing latent failure. The helm-unittest test files at deploy/helm/demarkus-server/tests/ shipped with §6.1 (PR #107) but never ran in CI, AND statefulset_test.yaml had a structural bug (only loaded statefulset.yaml, not the tokens.yaml it includes) that silently errored all 13 tests since day one. §6.3.D.1 PR #118 fixed the structural bug AND wired helm-unittest into CI. Mitigation in place.
  • Issuances Secret scaling wall at the ~1MB / ~5000-issuance k8s storage ceiling. Phase-7+ fix is a different backing store.
  • Chart proliferation. Three charts + examples + dashboards. Mitigate with shared common-labels templates.
  • Trial scope creep. First customer is path-B.
  • Per-replica rate limit ⇒ N× effective rate. Slice C.4's rate limiter is per-pod, so a multi-replica broker enforces up to N× the configured rate. Same shape as the Sweeper.ReleaseOnCancel posture: a known per-replica property documented in the rateLimitRegistry doc-comment, deferred fix (cluster-shared limiter) sized for Phase-7+. If a customer's threat model requires strict global rate-limiting, the cluster-shared backend in §Backlog needs to land before they go live.
  • Rate-limit registry growth under misconfigured XFF trust. Slice C.4's loginReg keys on s.clientIP(r). With trustForwardedFor=true and no trusted proxy stripping spoofed XFF in front, an attacker can rotate the header to inflate the map. The defense at this layer (TTL/LRU eviction) makes the property worse under the same attack — see the per-replica unbounded Decision bullet. The correct fix is at the deployment layer: the §6.3 chart README + rateLimit.trustForwardedFor doc-comment call out that the chart default true is only safe behind an Ingress that strips spoofed XFF. Operators flipping the flag in a directly-internet-exposed broker is the failure mode.

Status

Plan v14, 2026-05-13. Slice A, 6.0 chart, 6.1 chart, 6.2 broker binary (Slices B + C.1–C.4), 6.3 broker chart (Slices A + B + C + D.1) all merged (PRs #106, #107, #108, #109, #110, #111, #112, #114, #115, #116, #117, #118). §6.2 broker binary complete. §6.3 broker chart code complete; D.2 sidecar (§6.1 resource-policy: keep fix + kind upgrade-wipe regression) is the immediate next work.

§6.3 trajectory recap (all 2026-05-13):

  • 6.3.A (PR #115, c71f499): chart skeleton + security defaults. Cookie-key lookup-preserve, issuances Secret resource-policy: keep, OIDC mutual-exclusion guards at render-time, full PodSecurityContext lockdown, topology spread, checksum/config annotation, rateLimit.trustForwardedFor: true chart default.
  • 6.3.B (PR #116, 175e230): per-world + broker-ns RBAC, NetworkPolicy default-on, PodDisruptionBudget, Workload Identity SA annotation. Per-world Role verbs pinned to get/update (NOT create); broker-ns leases rule includes create per k8s RBAC caveat (resourceName scoping doesn't apply to create).
  • 6.3.C (PR #117, 599f63a): Ingress + cert-manager Certificate + oidc.existingSecretRef deployment shape + OIDC_CLIENT_SECRET env-var override in the binary (config.go:262). Three TLS modes mutex-guarded; existingSecretRef.key required when name set. README + NOTES.txt expanded with production checklist.
  • 6.3.D.1 (PR #118): seven helm-unittest test suites (49 tests) covering the chart's load-bearing invariants, CI plumbing for both charts via new test-charts job, folded-in test-broker Go CI job closing the §6.2 binary-CI gap. helm-unittest pinned to v0.6.2. Fixed real prior-work bug in §6.1's statefulset_test.yaml that had silently errored since day one.

Next: §6.3.D.2 sidecar. Add helm.sh/resource-policy: keep + helm.sh/hook: pre-install to deploy/helm/demarkus-server/templates/tokens.yaml. Add a kind upgrade-wipe regression test job in CI exercising BOTH the server and broker charts: helm install → kubectl-patch the Secret with fake data → helm upgrade → assert seeded data persists → uninstall → assert Secret survives uninstall (resource-policy: keep). Broker pod doesn't need to be ready; OIDC issuer reachability is irrelevant to the upgrade-wipe property. Kind test runtime is the dominant cost (~3-5min per chart); accept it as a per-PR-on-charts cost on the test-charts-kind job.

Then 6.4 examples → 6.5 observability recipes → 6.6 docs → 6.7 release pipeline.

trail
  1. soul.demarkus.io v14