soul.demarkus.io/plans/universe-deployment.md/v10 draft reader meta

Plan: Universe Deployment

Ship the enterprise-grade deployment for demarkus. The deliverable is the deployment: a customer's ops team installs a Helm chart, runs a broker, runs an agent, and has a federated demarkus universe their org can use. Any company evaluating demarkus (their internal "POC") installs the same product an established customer runs in production. There is no separate "POC slice" — the slice mentality is rejected. We build it once, right, and customers trial the real thing.

Goal

Deliver a complete, supportable, production-grade Kubernetes deployment package for demarkus, including:

  1. A Helm chart for demarkus-server (one world).
  2. A Helm chart for demarkus-broker (OIDC token issuance + revocation).
  3. A Helm chart for demarkus-agent (hub aggregator, crawl-and-index).
  4. Reference topology examples (Argo CD ApplicationSet, Kustomize overlay).
  5. Backend-agnostic observability — structured slog emission + reference configs for Datadog, OTel Collector, Vector, Fluent Bit, Grafana Alloy.
  6. Customer-facing documentation — installation, security/threat model, operations, upgrade path, per-provider OIDC setup, observability recipes.
  7. A release pipeline producing images and chart releases consumable from GHCR.

The same artifacts power first-customer trial and steady-state operations.

Non-Goals (Phase 7+ territory)

  • Multi-replica worlds with shared storage (RWX / object backend).
  • Cross-cluster universe federation.
  • Operator with a World CRD.
  • Hosted / managed SaaS.
  • Non-markdown content.

Constraints

  • No core protocol/server changes. Period. Per prior precedent (Claude Code plugin, Obsidian plugin, feedback_plugin_scope.md). Observability is achieved by log derivation in a collector, not by adding /metrics or OTel SDK calls to the server. Note: in Slice A we promoted HashToken and the token-mint library to protocol/ (under protocol/auth.go and protocol/token/). These are additive helper relocations consumed by server + CLI + future broker; no wire-protocol or server behavior changed.
  • Capability-based auth model is non-negotiable. The server never learns identity — only labels.
  • Markdown-only scope is non-negotiable.

Decisions (resolved during planning)

  • Health probes use exec, not HTTP. Both charts ship liveness/readiness exec probes that invoke the demarkus CLI fetching /.well-known/agent-manifest.md (always public per /architecture.md). No core change. Same pattern as redis-cli ping / pg_isready.
  • UDP port is a values knob. Default 6309, override via server.udpPort. Documented option: switch to 443 for VPN/middlebox-hostile networks (Cloudflare Warp Zero Trust, corp firewalls that filter non-standard UDP). Protocol default stays 6309.
  • Wildcard TLS via cert-manager DNS-01. Single *.<root> cert covers all worlds + broker hostnames. HTTP-01 cannot work for worlds (no HTTP); DNS-01 is the standard path for QUIC services.
  • DNS topology. One A record per world + one for the broker, all under the same wildcard zone. Each world exposed by Service: LoadBalancer protocol: UDP; broker by standard Ingress (HTTPS).
  • Hub aggregator is the existing demarkus-agent at client/cmd/demarkus-agent/. Already implements crawl + daemon subcommands with TOML config, multi-worker crawl, per-host token auth, per-server + aggregated index publishing. Verified end-to-end on 2026-05-11. Phase 5 is no longer a blocking prerequisite.
  • demarkus-agent lives in client/cmd/, not tools/. It's a protocol client (uses fedcrawl, fetch, tokens, links), not a utility. Memory rule sharpened: client/cmd/ = protocol clients (CLI, TUI, MCP, agent); tools/ = utilities (broker, future sync/ops binaries).
  • Token-mint library lives at protocol/token/, not tools/internal/token/ (revised mid-Slice-A after CodeRabbit review). protocol/ is reachable by both server/ (where demarkus-token CLI lives) and tools/ (where the broker will live), which tools/internal/ was not. protocol/ already owns the HashToken byte-shape contract, so it's the natural home for the on-disk token primitives that must round-trip identically across CLI mints and broker mints.
  • demarkus-token CLI stays at server/cmd/demarkus-token/ for now (rewired to import protocol/token). Moving the binary to tools/ was deferred to a follow-up that splits the release pipeline (server release currently ships demarkus-token; tools has no release cadence yet). Same applies to demarkus-publish — still at server/cmd/demarkus-publish/, deferred because hoisting it requires also hoisting server/internal/store/ out of internal-package scope.
  • Multi-OIDC. Broker speaks generic OIDC, not Google-specific. Provider behind a Verifier interface. Google validated first; Okta, Entra ID, Auth0 follow with config only.
  • Workload Identity (GKE) as a broker option. Values flag annotates the broker ServiceAccount with the GSA mapping. Off by default; on for GKE customers wanting no static SA keys. Not applicable to world servers.
  • Broker is HA. Multi-replica with resourceVersion optimistic concurrency on Secret writes, retry on conflict.
  • Observability is log-derived, backend-agnostic. demarkus already emits structured slog. Charts ship Datadog autodiscovery annotations + reference configs for OTel Collector, Vector, Fluent Bit, Grafana Alloy. Customer's SRE picks the agent/backend. Latency histograms deferred — depends on what current slog includes per request; revisit during trials if needed.
  • Backup/DR is documented, not built. The chart deliberately does not run backup CronJobs. Operations doc covers Velero, VolumeSnapshot, and demarkus-agent sync as DR options.
  • Image hosting: ghcr.io/latebit-io/{demarkus-server,demarkus-broker,demarkus-agent}.
  • Chart registry: OCI charts in GHCR.
  • Cosign signing: deferred to backlog. Half-day CI add when wanted.
  • Groups-claim case sensitivity (revised mid-Slice-C.1 after CodeRabbit back-and-forth). Group names match case-insensitively: AllowConfig.Groups is lowercased+trimmed at config load (same as Domains and Emails) and groupsMatch lowercases the claim's groups before compare. The validated IdP set (Google, Okta, Entra ID, Auth0) enforces case-insensitive group-name uniqueness, so distinct case-variant groups can't exist; a case-sensitive compare would silently fail when an operator writes "Engineering" but the IdP emits "engineering" after upstream normalization. Keycloak with deliberately distinct case-variant groups is out of scope; revisit if a customer asks. Domains and emails get the same lowercase-at-load + plain == treatment for symmetry.

Open Questions

  1. First customer trial. Nesto (*.library.nesto.ca) is path-B — trial waits for product. Trial runbook lands at /trials/nesto.md when scoping starts.
  2. CLI relocation to tools/. Follow-up to Slice A. Requires tools/.goreleaser.yml, a release-tools job in the workflow, an install.sh block fetching the tools archive, and updates to plugin scripts + helm image build. Acceptable to defer until the release-pipeline work has its own dedicated PR.

Repository Layout

Reflects state as of Slice C.1 merge (2026-05-12):

deploy/
  helm/
    demarkus-server/         # one-world chart (Phase 6.1) — shipped PR #107
    demarkus-broker/         # OIDC token broker chart (Phase 6.3)
    demarkus-agent/          # crawl/index agent chart (Phase 6.0) — shipped PR #106
  k8s/
    examples/
      applicationset.yaml    # Argo CD ApplicationSet over a worlds: list
      kustomize-overlay/     # Kustomize alternative
  observability/
    datadog/                 # autodiscovery annotations + dashboard JSON
    otel-collector/          # collector config recipes
    vector/                  # vector config recipes
    fluent-bit/              # fluent-bit parser + filter recipes
  scripts/                   # operator helpers (cert pre-check, MTU probe, etc.)

protocol/
  auth.go                    # HashToken — sha256-<hex> contract (Slice A)
  token/                     # Generate, ReadFile, AppendEntry, WriteFile,
                             # FormatEntry, flock helpers (Slice A) +
                             # AppendBytes, RemoveBytes in-memory helpers
                             # (Slice B) for callers that round-trip TOML
                             # through k8s Secrets instead of disk.
                             # Shared by CLI and broker; on-disk path is
                             # atomic temp+rename, fsync data + parent dir
                             # for durability, advisory flock(2) for
                             # cross-process serialization, quoted-key
                             # handling for non-bare TOML labels.

server/cmd/
  demarkus-server/           # the server binary (existing)
  demarkus-token/            # token mint CLI — uses protocol/token
                             # (Slice A: rewired, not moved; tools/
                             # relocation deferred — see Open Questions)
  demarkus-publish/          # publish CLI (existing; deferred move)

client/cmd/
  demarkus-agent/            # protocol client — federation crawler (existing)

tools/
  demarkus-broker/           # broker binary (Slice B PR #109, Slice C.1
                             # PR #110) — imports protocol/token for
                             # byte-identical hashes (Phase 6.2).
                             # main.go + internal/broker/
                             # { config, session, oidc, issuer, server,
                             #   labels }. Slice B shipped the
                             # single-world OIDC mint flow; Slice C.1
                             # added groups-claim authorization,
                             # AllowEmails carve-out, and email
                             # canonicalization in Mint. Remaining
                             # Slice C work: C.2 sweeper +
                             # leader-election + RBAC test, C.3 rotate,
                             # C.4 rate limit. SCIM webhook stays in
                             # backlog.

Sub-Phases

6.0 — demarkus-agent verified + chart ✓ binary, ✓ chart (merged PR #106)

Binary status (2026-05-11): existing demarkus-agent verified end-to-end against a 3-server smoke test (2 team worlds + 1 hub). Crawls, builds aggregated and per-server indexes, publishes to hub. Two bugs fixed during verification:

  • fedcrawl/crawl.go publishIndex accepted only ok status; first publish returns created, surfaced as a misleading warning despite the publish succeeding. Now accepts both.
  • publishIndex always used expected_version=0 (create-only), causing every re-publish in daemon mode to fail with conflict. Now uses -1 (no check) for idempotent hub re-publish; server's no-op-on-duplicate-content prevents version churn.
  • Makefile did not build demarkus-agent; the agent had to be hand-built. Now builds with make client.

Tests in client/internal/fedcrawl/crawl_test.go cover all three: create + re-publish + status acceptance + per-server / aggregated modes. bash pre-commit.sh clean.

Agent Helm chart shipped in PR #106 at deploy/helm/demarkus-agent/:

  • Deployment (not StatefulSet — agent is stateless modulo state file; recoverable from next crawl).
  • ConfigMap holding the TOML agent config (seeds, hubs, crawl, politeness, schedule).
  • Secret holding per-host tokens for publishing to hub(s).
  • ServiceAccount, no special RBAC needed (no k8s API calls).
  • Exec liveness probe (demarkus-agent version returns 0).
  • No Service (agent is outbound-only).
  • Pod annotations for Datadog autodiscovery; log fields documented for mint/crawl events.

6.1 — demarkus-server Helm chart ✓ (merged PR #107)

deploy/helm/demarkus-server/. Production-grade.

Workload:

  • StatefulSet, 1 replica (multi-replica is Phase 7).
  • volumeClaimTemplates — each world owns its PVC. Never a shared PVC.
  • Container image bundles demarkus-server + demarkus CLI (for exec probes).
  • Exec liveness + readiness probes against /.well-known/agent-manifest.md.
  • Resource requests/limits with sane defaults, overridable.
  • Service type LoadBalancer, protocol: UDP, port from server.udpPort (default 6309). Annotations for cloud-specific LB type (NLB on AWS, etc.).

Secrets (world namespace):

  • <release>-tokens — TOML of SHA-256 hashes. Server-mounted.
    • Persistence across helm upgrade is non-negotiable. Rendered with helm.sh/resource-policy: keep + helm.sh/hook: pre-install and empty data: {} in the template body. Helm creates the Secret on first install and never overwrites it on subsequent upgrades; broker writes via k8s API are preserved.
    • Bootstrap Job (below) seeds the initial admin token on first install only. The broker manages all subsequent entries at runtime via the k8s API.
    • Without this pattern, every helm upgrade would silently wipe every broker-minted token. This is the single most important Helm correctness property in the chart and must be unit-tested.
    • kubelet propagation: Secret data changes propagate to the mounted file with a delay (default ~60s). Server re-reads via SIGHUP (preferred) or fsnotify; verify which during chart implementation.
  • Broker state (raw tokens are never persisted; email→label mappings and issuance metadata) lives in the broker namespace, not the world namespace. See §6.2 for the broker's issuances Secret. The world's tokens Secret holds only hashes and is the sole source of truth the server reads.

Persistence boundary recap: pod restarts, node failures, control-plane restarts → fine. helm upgrade → fine iff resource-policy: keep. kubectl delete namespace → everything gone (operations doc covers Velero / VolumeSnapshot recovery). helm uninstall without keep annotation → tokens Secret gone; with keep, retained.

Auth + TLS:

  • TLS Secret mounted via volumeMounts; cert/key paths via flags.
  • Optional cert-manager Certificate resource (behind a flag) requesting *.<root> from a configured ClusterIssuer.

Bootstrap Job:

  • Mints initial admin token on first install via protocol/token.Generate + protocol/token.AppendEntry (Slice A).
  • helm.sh/hook: pre-install only — does not run on upgrade.
  • Idempotent guard: reads the tokens Secret; skips if admin label already exists.
  • SIGHUPs pod after writing (no-op on first install before pod is up).

Observability:

  • Pod annotations for Datadog autodiscovery.
  • Reference configs for OTel Collector, Vector, Fluent Bit in deploy/observability/.
  • slog output already structured; no chart-side instrumentation needed.

RBAC:

  • Bootstrap Job SA with get/update on the named Secret, get/list/create on pods/exec for SIGHUP. Namespace-scoped Role, not ClusterRole.

Tests:

  • helm-unittest for templates, including a test that asserts resource-policy: keep is present on the tokens Secret and that the rendered Secret has empty data: {}.
  • Kind-based integration test in CI: install chart → exec into pod → verify health → publish via CLI → verify version increments → helm upgrade with a changed value → re-verify token-based auth still works (regression guard against the upgrade-wipe footgun).

6.2 — demarkus-broker binary (tools/demarkus-broker/)

Slice B (single-world OIDC mint flow) merged as PR #109 (commit 2a6aae6, 2026-05-11). Slice C.1 (groups-claim authorization + AllowEmails carve-out + email canonicalization) merged as PR #110 (commit 343f808, 2026-05-12). Remaining Slice C work — C.2 expiry sweeper + leader election, C.3 rotate endpoint, C.4 rate limit — is open; see §Sequencing for the ordering and §Status for the agreed sub-slicing.

Role & topology

  • Issuance authority, not a request proxy. Broker sits on the demarkus login path; clients then talk to world servers directly carrying the raw token. World servers stay identity-blind; broker never sees mark:// requests. Capability model preserved.
  • One broker per universe (cluster) by default. Single OIDC client registration, single worlds: list in values, single issuance state Secret. Multiple brokers only when (a) multiple OIDC providers must coexist, (b) hard tenant isolation between orgs sharing a cluster, or (c) Phase-7 geo split. N>1 is supported but not the common case.
  • Broker SA holds a namespace-scoped Role + RoleBinding in each world's namespace, with get/patch limited to that world's tokens Secret only. No ClusterRole. Blast radius bounded to the token Secrets it's explicitly granted.

State

Two distinct Kubernetes Secrets, never merged:

Secret Lives in Contents Reader
<world>-tokens each world's namespace TOML: [tokens.<label>] hash=, paths=, operations=, expires= demarkus-server
<broker>-issuances broker namespace JSON: label → {email, world, paths, operations, issued_at, expires} + secondary index email → [labels] broker only

World servers see only hashes. Email and identity live in broker state alone. Both Secrets follow the §6.1 resource-policy: keep pattern — broker state must also survive helm upgrade of the broker chart.

Labels

Opaque IDs (usr_<random8>; 4 bytes of entropy, collision-retry up to 5 in the issuer). Not email-derived. Reasons:

  • Sanitizing emails (dots, plus signs, IDN) into TOML keys is fragile — though note protocol/token.FormatEntry does handle quoted-key emission for non-bare labels if you ever need it.
  • Token rotation yields a new label each time; opaque IDs reflect that naturally.
  • "Revoke everything for fredrick@x" becomes a broker-state index lookup, not a label string scan.

Per-world authorization (values schema, as shipped in C.1)

worlds:
  - name: team-a
    namespace: team-a
    tokensSecret: team-a-tokens
    allow:
      domains: ["nesto.ca"]
      groups: ["engineering"]      # OIDC `groups` claim — Slice C.1
      emails: ["alice@example.com"] # per-user carve-out — Slice C.1
    defaultToken:
      paths: ["/team-a/*"]
      operations: ["read", "publish"]
      expiresAfter: 24h

WorldConfig.Allow is a nested struct (the prior AllowDomains flat field was promoted into Allow.Domains during C.1 — Slice B was 2 days old with no deployed configs, so the breaking schema change was free). All three lists are lowercased+trimmed at config load with empty entries rejected at validation. Match is case-insensitive everywhere.

On demarkus login, broker evaluates the OIDC identity against every world in its config and mints one token per world the user qualifies for. Client may request narrower scope; never broader.

The authorization predicate (worldAllows in issuer.go):

  1. All three lists empty → match (back-compat for worlds with no allowlist configured, same as the pre-Slice-C behavior).
  2. Email in Allow.Emails → match (per-user carve-out bypasses both domain and group requirements; the documented escape hatch when an IdP doesn't surface usable groups in the ID token).
  3. Only Emails was configured and step 2 didn't fire → reject. Without this guard, allow.emails: [alice@x] with no other allowlist would silently mean "everyone plus alice" because the AND of empty Domains + empty Groups evaluates to true. Detected as len(Domains)+len(Groups) == 0 when we reach this branch.
  4. domainMatches AND groupsMatch, where an empty list in either dimension is a no-op for that dimension only. This is the standard "domain restriction intersected with groups restriction" — RBAC inside an org, not across orgs.

Mint also canonicalizes claims.Email (trim + lowercase) once at the top before authorization runs, so the persisted issuance owner is canonical and untrimmed-IdP-emit casing doesn't silently fail domainMatches (which previously kept trailing whitespace in the domain slice via LastIndex("@")).

Token-mint library

Broker imports github.com/latebit/demarkus/protocol/token (Slice A + Slice B helpers). Available primitives:

Slice A — on-disk and pure helpers (used by demarkus-token CLI):

  • token.Generate(label, paths, operations) (Minted, error) — pure, no I/O. Returns Minted{Label, Raw, Entry} with Raw as the one-time secret to hand to the user via the OIDC callback response, and Entry as the hash+capabilities to persist.
  • token.AppendEntry(path, label, *Entry) — adds an entry to a tokens.toml file with atomic temp+rename publishing, fsync data + parent dir for crash durability, advisory flock(2) for cross-process serialization, and rejects duplicate labels via ErrLabelExists.
  • token.WriteFile(path, File) — full-replace with the same atomicity/durability/locking guarantees (used for revoke from CLI).
  • token.ReadFile(path) — decode tokens.toml from disk.
  • token.FormatEntry(label, *Entry) — render a single labeled entry; handles bare vs quoted TOML keys correctly.

Slice B — in-memory helpers (used by the broker, which round-trips TOML through k8s Secrets instead of files):

  • token.AppendBytes(existing []byte, label string, entry *Entry) ([]byte, error) — decode existing, append the new labeled entry, re-encode. Same duplicate-check (ErrLabelExists) and quoted-key handling as AppendEntry, minus the disk-side concerns (no temp+rename, no fsync, no flock — k8s API + resourceVersion provide the equivalent ordering guarantees at a different layer).
  • token.RemoveBytes(existing []byte, label string) ([]byte, error) — decode, drop the named label, re-encode. Used for DELETE /tokens/:label and the expiry sweeper.

Broker writes the returned []byte into the world's tokens Secret via Patch(StrategicMergePatchType), paired with resourceVersion optimistic concurrency and retry-on-conflict (up to 5). protocol.HashToken guarantees byte-identical hashes to what world servers read regardless of mint origin.

Revocation

Three triggers, one cleanup index (the issuances Secret):

  1. User-initiated (demarkus token revoke <label> or rotate):

    • DELETE /tokens/:label carrying OIDC ID token (bearer-authed; no broker session cookie — see "Routes" below).
    • Broker confirms entry.email == claims.email (owner check — users can only revoke their own).
    • Patches the world's tokens Secret via RemoveBytes to drop the label.
    • Drops the entry from issuances Secret.
    • SIGHUPs world server pod(s) so tokens.toml is re-read. (Or relies on server fsnotify if present — verify during implementation.)

  2. Expiry sweeper (Slice C.2 — broker in-process ticker, default every 5 min, leader-elected via Lease so only one replica sweeps):

    • Queries issuances where expires < now.
    • For each: patch the world's tokens Secret + drop from issuances + SIGHUP.

  3. Identity lifecycle (user leaves the org):

    • Default: short-lived tokens. defaultToken.expiresAfter of 24h means stale tokens age out within a day. Next demarkus login reruns OIDC; if the user's IdP account is disabled, login fails. No broker→IdP coupling needed. Same approach as AWS STS / gcloud auth login.
    • Backlog: SCIM lifecycle webhook (POST /scim/v2/Users/:id) for enterprise IdPs that push deprovisioning events (Okta, Entra). Optional add when a customer asks.
    • Periodic IdP re-validation rejected — worse trade-off than either above (couples broker to IdP rate limits, slower than SCIM, less simple than short-lived).

Cleanup edge cases

  • Orphan in tokens.toml (admin-minted via legacy demarkus-token CLI): broker never claims it. Admins manage via CLI. Broker API only operates on labels it minted.
  • Orphan in issuances Secret (admin hand-deleted from world tokens Secret): expiry sweeper (Slice C.2) detects drift on each pass and prunes broker-state entries whose label no longer exists in the world's TOML.
  • In-flight request after revoke: a raw token already authenticated on a live QUIC connection completes its current request. Property of the capability + connection-reuse model. Documented; not fixable without core server changes.
  • Partial mint across multiple worlds (Slice B behavior): if world N+1's Secret patch fails mid-iteration, successful mints from worlds 1..N stay live and the caller gets back the partial result plus the error. Orphans land in those worlds' tokens Secrets until the Slice-C.2 sweeper prunes them. Acceptable for Slice B since realistic single-broker config has one world; revisit when multi-world is the common case.

Routes (HTTP, behind Ingress)

  • GET /auth/login — OIDC redirect entry point. Sets a signed state cookie (HMAC-SHA256 over JSON {nonce, expiresAt}, HttpOnly+Secure+SameSite=Lax, path-scoped to /auth/callback, default 5-minute TTL).
  • GET /auth/callback — OIDC callback. Verifies state cookie (CSRF), exchanges code, mints tokens for every qualifying world, returns JSON {world → raw_token} once. No broker session cookie is issued — the state cookie is the only cookie in the system, and it exists purely to defend the OIDC dance against CSRF.
  • GET /tokens — list caller's tokens from broker state (labels + metadata only, never raw tokens). Bearer-authed with the user's OIDC ID token, not a broker session.
  • DELETE /tokens/:label — revoke caller-owned token. Bearer-authed (OIDC ID token).
  • POST /tokens/:label/rotate — Slice C.3. Revoke + mint with same scope (preserved from the issuance record, not reread from current config); returns new raw token.
  • GET /healthz, GET /readyz.

Rationale for skipping a broker session: the CLI is the primary /tokens consumer and already handles token lifetimes; the browser only ever sees the one-time JSON callback response. Adding a session cookie would couple broker state to browser state for no gain. The Verifier interface accordingly exposes VerifyIDToken(ctx, raw) (Claims, error) so handlers can authenticate bearer tokens uniformly.

OIDC providers

Provider behind a Verifier interface (AuthCodeURL, Exchange, VerifyIDToken). Initial implementation wraps coreos/go-oidc/v3 + golang.org/x/oauth2. Follow-ons: Okta, Entra ID, Auth0 — config only. Group-claim availability varies per provider; doc page per provider explains required claim configuration (e.g., Entra needs the groups optional claim enabled; some providers require a userinfo call to fetch groups not in the ID token).

Groups claim sourcing (Slice C.1 scope): ID-token only. The Claims struct's Groups []string field is populated from the groups claim in the verified ID token. Providers that surface groups in userinfo instead of the ID token (Google, some Entra configs) need a richer Verifier impl that runs a userinfo call after Verify. Out of Slice C scope; revisit when a customer needs it. The AllowEmails carve-out is the documented workaround in the meantime.

HA

Multi-replica. Issuance writes to k8s Secrets use resourceVersion optimistic concurrency with retry-on-conflict. Sweeper uses a coordination.k8s.io/Lease for leader election so only one replica runs the periodic loop (Slice C.2).

Slice B test surface (shipped)

  • Mint flow: table-driven, Verifier test double in oidc_test.go, fake k8s clientset, OIDC discovery mocked via httptest. End-to-end coverage at the HTTP layer via httptest.NewServer.
  • State cookie: HMAC verify path, expiry path, malformed-envelope path, short-key rejection.
  • Config: YAML decode with KnownFields(true) rejecting typos; zero-expiresAfter rejected at startup.
  • Issuer: resourceVersion retry-on-conflict, label collision retry, mint ordering (world Secret first, issuance record second — orphan-in-issuances is the documented partial-failure mode).
  • Server: clock decoupling between server and issuer (state-cookie expiry vs token-expiry assertions need different clocks).

87% line coverage on internal/broker/. bash pre-commit.sh clean.

Slice C.1 test surface (shipped — PR #110)

  • 15-row table TestMintAuthorizationPredicate covers every (Domains × Groups × Emails) × accept/reject cell, including the (domain AND groups) OR email-carve-out predicate, the only-emails-set reject case, and case-insensitive group matching (claim-side).
  • TestMintCanonicalizesEmail: passes " Alice@Example.COM\t" and asserts both domain match succeeded and the persisted issuance email is alice@example.com. Regression guard against the IdP-side whitespace/case footgun that motivated Mint's top-of-function trim+lowercase.
  • TestLoadConfig rows for each allowlist dimension: lowercase normalization at load (domains, emails, groups) and empty-entry rejection (allow.<dim>[N] is empty). Pins the load-side half of the case-insensitive contract documented above.
  • TestDeleteTokenNotOwner (existed since Slice B PR #109): different OIDC subject than the issuance email → 403. Already covered the owner-check 403 path that the original Slice C surface listed as "open"; flagged during C.1 review.

Slice C remaining test surface (open)

  • C.2: sweeper happy path + drift-pruning (orphan-in-issuances) using fake clock + fake clientset; two-replica leader-election fake-clientset test asserting only the leader runs the loop; RBAC permission-denied test (broker SA without patch on a world's Secret → mint fails cleanly with no partial state).
  • C.3: rotate happy path; rotate owner-mismatch 403; rotate of unknown label 404; rotate preserves issuance-recorded scope (not current DefaultToken).
  • C.4: per-subject rate limit smoke test on /tokens (assert N+1 returns 429); separate IP-keyed limit on /auth/login.

6.3 — demarkus-broker Helm chart

Multi-replica HA. As previously specified.

6.4 — Universe topology examples

ApplicationSet + Kustomize overlay. As previously specified.

6.5 — Observability recipes

Per-backend configs in deploy/observability/. As previously specified.

6.6 — Documentation suite

/deployment/*.md + per-chart READMEs. As previously specified.

6.7 — Release pipeline

GHCR images + OCI charts. Cosign deferred to backlog. Also folds in the deferred CLI relocation (demarkus-token, demarkus-publish) from server/cmd/ to tools/ — that change requires a new tools/.goreleaser.yml + release-tools workflow job + install.sh block, all of which naturally belong in the release-pipeline slice.

Sequencing

  1. Slice A — token-mint library ✓ merged 2026-05-11 (PR #108).
  2. 6.0 chart ✓ merged 2026-05-11 (PR #106).
  3. 6.1 server chart ✓ merged 2026-05-11 (PR #107).
  4. 6.2 broker binary — Slice B ✓ merged 2026-05-11 (PR #109, commit 2a6aae6). Adds protocol/token.AppendBytes / RemoveBytes in-memory helpers for k8s-Secret-backed callers.
  5. 6.2 broker binary — Slice C, split into four landed PRs:
    • C.1 ✓ merged 2026-05-12 (PR #110, commit 343f808) — groups-claim + AllowEmails authorization. Nested WorldConfig.Allow schema (breaking change vs Slice B's AllowDomains); lowercase-at-load + plain == everywhere; case-insensitive group matching; email canonicalization in Mint; configurable groups via ID-token claim.
    • C.2 — expiry sweeper + Lease-based leader election + drift-pruning + RBAC-denied test. New sweeper.go + tests; depends on client-go/tools/leaderelection.
    • C.3 — POST /tokens/:label/rotate. New Issuer.RotateLabel (reuses cross-world-collision + rollback machinery from mintForWorld); thin handler.
    • C.4 — per-subject rate limit middleware (golang.org/x/time/rate, in-memory per-replica). Keyed by hashSubject(claims.Subject) for authed endpoints, source IP for /auth/login.
    • SCIM webhook stays in backlog (no customer ask yet).
  6. 6.3 broker chart — after 6.2 Slice C testable.
  7. 6.4 topology examples.
  8. 6.5 observability recipes.
  9. 6.6 docs — incremental throughout.
  10. 6.7 release pipeline — final; also relocates demarkus-token / demarkus-publish to tools/.

Rough effort: 1–3 weeks of focused work remaining (3 core code units + Slice C.1 shipped; what's left is Slice C in three more PRs, the broker chart, examples, recipes, docs, and the release pipeline).

Backlog (deferred, easy to add later)

  • Cosign signing of images + chart releases. Half-day CI add when wanted.
  • Latency log-enrichment (duration_ms field on request slog lines). Tiny additive change. Defer until trials show it's needed.
  • SCIM lifecycle webhook on the broker (POST /scim/v2/Users/:id) for enterprise IdPs that push deprovisioning events. Adds responsiveness beyond what short-lived tokens give. Optional add when a customer asks.
  • CLI relocation to tools/ (demarkus-token, demarkus-publish). Bundled with 6.7 release-pipeline work — see Open Questions.
  • Userinfo-based groups in the broker Verifier for providers (Google, some Entra configs) that don't surface groups in the ID token. Slice C.1 ships ID-token-only; AllowEmails is the documented workaround.
  • Case-sensitive group matching for Keycloak (or any future case-sensitive IdP) where an operator deliberately maintains distinct case-variant groups for different access levels. Slice C.1 chose case-insensitive matching for the validated IdP set; revisit if a customer asks.

Risks

  • Observability via logs ceiling. If customers want signals not derivable from current slog (latency, internal state like version counts or PVC fullness), we hit a wall. Mitigation: log enrichment is a small additive change; internal-state metrics derivable by a cluster-side sidecar that calls LIST periodically.
  • Broker secret-write blast radius. Holds k8s API creds across world namespaces. Mitigated by namespace-scoped Roles (one per world), audit log, optional NetworkPolicy. Per-world Role is non-negotiable — never ClusterRole.
  • OIDC provider coupling. First impl is Google; structure so second provider is a one-day add.
  • Token revocation in-flight latency. SIGHUP reloads tokens.toml, but a request already authenticated on a live QUIC connection completes. Property of model. Slice C.2 sweeper does not currently SIGHUP world servers; revocation latency = kubelet Secret propagation + whatever the server actually does on re-mount. Verify server reload behavior during chart implementation (§6.3).
  • Helm upgrade wiping tokens. Classic templating footgun: rendering a Secret on every upgrade overwrites broker-minted content. Mitigated by helm.sh/resource-policy: keep + pre-install hook on the world tokens Secret (and the broker issuances Secret). Regression-tested in CI via the kind integration test.
  • Chart proliferation. Three charts + examples + dashboards. Mitigate with shared common-labels templates.
  • Trial scope creep. First customer is path-B. If feedback pulls scope back to "demo slice" mid-build, decide explicitly.

Status

Plan v10, 2026-05-12. Slice A, 6.0 chart, 6.1 chart, 6.2 broker binary Slice B, 6.2 broker binary Slice C.1 all merged (PRs #106, #107, #108, #109, #110).

  • Slice B (PR #109, commit 2a6aae6, 2026-05-11): single-world OIDC mint flow with Verifier-interface OIDC provider, signed state cookie (no broker session), bearer-auth on /tokens and DELETE /tokens/:label, k8s Secret writes via resourceVersion retry, opaque usr_<8 hex> labels with collision-retry, 87% line coverage. Slice B also added protocol/token.AppendBytes / RemoveBytes in-memory helpers — the library that started Slice A on disk now serves both disk callers (CLI) and Secret callers (broker) from the same byte-shape contract.
  • Slice C.1 (PR #110, commit 343f808, 2026-05-12): groups-claim + AllowEmails authorization. Nested WorldConfig.Allow schema replaces the flat AllowDomains field (breaking config change; Slice B was 2 days old with no deployed configs). Authorization predicate is (domain AND groups) OR email-carve-out with an explicit reject for the only-emails-set case. Email canonicalized (trim+lowercase) once in Mint to fix an untrimmed-email footgun in domainMatches. Groups, domains, and emails all lowercased+trimmed at config load with empty-entry rejection; group matching is case-insensitive because the validated IdP set (Google/Okta/Entra/Auth0) enforces case-insensitive group-name uniqueness. ID-token-sourced groups only; userinfo-based groups in backlog.

Next: 6.2 Slice C remaining:

  • C.2 — expiry sweeper + Lease-based leader election + drift-pruning sweeper test + RBAC-denied test. Sweeper does not SIGHUP world servers in Slice C; revocation latency window documented in §Risks, revisited in 6.3 chart work.
  • C.3 — POST /tokens/:label/rotate. Preserves issuance-recorded scope (not current DefaultToken); operator scope changes only take effect on next demarkus login.
  • C.4 — per-subject rate limit middleware (golang.org/x/time/rate, in-memory per-replica). Defaults proposed: 10/min subject on /tokens+revoke+rotate (burst 5), 20/min IP on /auth/login (burst 5); both override-able via config.

Then 6.3 broker chart → examples/recipes/docs → 6.7 release pipeline.

trail
  1. soul.demarkus.io v10