soul.demarkus.io/plans/universe-deployment.md/v11 draft reader meta

Plan: Universe Deployment

Ship the enterprise-grade deployment for demarkus. The deliverable is the deployment: a customer's ops team installs a Helm chart, runs a broker, runs an agent, and has a federated demarkus universe their org can use. Any company evaluating demarkus (their internal "POC") installs the same product an established customer runs in production. There is no separate "POC slice" — the slice mentality is rejected. We build it once, right, and customers trial the real thing.

Goal

Deliver a complete, supportable, production-grade Kubernetes deployment package for demarkus, including:

  1. A Helm chart for demarkus-server (one world).
  2. A Helm chart for demarkus-broker (OIDC token issuance + revocation).
  3. A Helm chart for demarkus-agent (hub aggregator, crawl-and-index).
  4. Reference topology examples (Argo CD ApplicationSet, Kustomize overlay).
  5. Backend-agnostic observability — structured slog emission + reference configs for Datadog, OTel Collector, Vector, Fluent Bit, Grafana Alloy.
  6. Customer-facing documentation — installation, security/threat model, operations, upgrade path, per-provider OIDC setup, observability recipes.
  7. A release pipeline producing images and chart releases consumable from GHCR.

The same artifacts power first-customer trial and steady-state operations.

Non-Goals (Phase 7+ territory)

  • Multi-replica worlds with shared storage (RWX / object backend).
  • Cross-cluster universe federation.
  • Operator with a World CRD.
  • Hosted / managed SaaS.
  • Non-markdown content.

Constraints

  • No core protocol/server changes. Period. Per prior precedent (Claude Code plugin, Obsidian plugin, feedback_plugin_scope.md). Observability is achieved by log derivation in a collector, not by adding /metrics or OTel SDK calls to the server. Note: in Slice A we promoted HashToken and the token-mint library to protocol/ (under protocol/auth.go and protocol/token/). These are additive helper relocations consumed by server + CLI + future broker; no wire-protocol or server behavior changed. Slice C.2 added protocol/token.ParseBytes in the same additive spirit — a read-side helper so the broker's drift sweeper can inspect a world's tokens.toml payload via the map shape rather than substring matching on serialized TOML.
  • Capability-based auth model is non-negotiable. The server never learns identity — only labels.
  • Markdown-only scope is non-negotiable.

Decisions (resolved during planning)

  • Health probes use exec, not HTTP. Both charts ship liveness/readiness exec probes that invoke the demarkus CLI fetching /.well-known/agent-manifest.md (always public per /architecture.md). No core change. Same pattern as redis-cli ping / pg_isready.
  • UDP port is a values knob. Default 6309, override via server.udpPort. Documented option: switch to 443 for VPN/middlebox-hostile networks (Cloudflare Warp Zero Trust, corp firewalls that filter non-standard UDP). Protocol default stays 6309.
  • Wildcard TLS via cert-manager DNS-01. Single *.<root> cert covers all worlds + broker hostnames. HTTP-01 cannot work for worlds (no HTTP); DNS-01 is the standard path for QUIC services.
  • DNS topology. One A record per world + one for the broker, all under the same wildcard zone. Each world exposed by Service: LoadBalancer protocol: UDP; broker by standard Ingress (HTTPS).
  • Hub aggregator is the existing demarkus-agent at client/cmd/demarkus-agent/. Already implements crawl + daemon subcommands with TOML config, multi-worker crawl, per-host token auth, per-server + aggregated index publishing. Verified end-to-end on 2026-05-11. Phase 5 is no longer a blocking prerequisite.
  • demarkus-agent lives in client/cmd/, not tools/. It's a protocol client (uses fedcrawl, fetch, tokens, links), not a utility. Memory rule sharpened: client/cmd/ = protocol clients (CLI, TUI, MCP, agent); tools/ = utilities (broker, future sync/ops binaries).
  • Token-mint library lives at protocol/token/, not tools/internal/token/ (revised mid-Slice-A after CodeRabbit review). protocol/ is reachable by both server/ (where demarkus-token CLI lives) and tools/ (where the broker will live), which tools/internal/ was not. protocol/ already owns the HashToken byte-shape contract, so it's the natural home for the on-disk token primitives that must round-trip identically across CLI mints and broker mints.
  • demarkus-token CLI stays at server/cmd/demarkus-token/ for now (rewired to import protocol/token). Moving the binary to tools/ was deferred to a follow-up that splits the release pipeline (server release currently ships demarkus-token; tools has no release cadence yet). Same applies to demarkus-publish — still at server/cmd/demarkus-publish/, deferred because hoisting it requires also hoisting server/internal/store/ out of internal-package scope.
  • Multi-OIDC. Broker speaks generic OIDC, not Google-specific. Provider behind a Verifier interface. Google validated first; Okta, Entra ID, Auth0 follow with config only.
  • Workload Identity (GKE) as a broker option. Values flag annotates the broker ServiceAccount with the GSA mapping. Off by default; on for GKE customers wanting no static SA keys. Not applicable to world servers.
  • Broker is HA. Multi-replica with resourceVersion optimistic concurrency on Secret writes, retry on conflict.
  • Observability is log-derived, backend-agnostic. demarkus already emits structured slog. Charts ship Datadog autodiscovery annotations + reference configs for OTel Collector, Vector, Fluent Bit, Grafana Alloy. Customer's SRE picks the agent/backend. Latency histograms deferred — depends on what current slog includes per request; revisit during trials if needed.
  • Backup/DR is documented, not built. The chart deliberately does not run backup CronJobs. Operations doc covers Velero, VolumeSnapshot, and demarkus-agent sync as DR options.
  • Image hosting: ghcr.io/latebit-io/{demarkus-server,demarkus-broker,demarkus-agent}.
  • Chart registry: OCI charts in GHCR.
  • Cosign signing: deferred to backlog. Half-day CI add when wanted.
  • Groups-claim case sensitivity (revised mid-Slice-C.1 after CodeRabbit back-and-forth). Group names match case-insensitively: AllowConfig.Groups is lowercased+trimmed at config load (same as Domains and Emails) and groupsMatch lowercases the claim's groups before compare. The validated IdP set (Google, Okta, Entra ID, Auth0) enforces case-insensitive group-name uniqueness, so distinct case-variant groups can't exist; a case-sensitive compare would silently fail when an operator writes "Engineering" but the IdP emits "engineering" after upstream normalization. Keycloak with deliberately distinct case-variant groups is out of scope; revisit if a customer asks. Domains and emails get the same lowercase-at-load + plain == treatment for symmetry.
  • Sweeper interval capped at 24h (Slice C.2). The defaultToken.expiresAfter typical is 24h; a sweep interval longer than that would let expired tokens linger past their ExpiresAt for up to a whole sweep cycle, defeating the short-lived-tokens identity-lifecycle model (§Revocation §3). validate() rejects with sweeper.interval must be <= 24h. Lease-election timings (15s lease / 10s renew / 2s retry) intentionally not exposed in YAML — the client-go defaults match our failover budget and adding knobs invites misconfiguration. Revisit if a customer needs sub-15s failover.
  • sweeper.disabled rather than sweeper.enabled (Slice C.2). YAML zero-value (false) is the production-safe default so an operator who omits the block gets the sweeper running. Naming the field positively (Enabled) would have made omission silently disable the janitor — exactly the wrong default for multi-replica deployments where expired tokens accumulate forever without it.

Open Questions

  1. First customer trial. Nesto (*.library.nesto.ca) is path-B — trial waits for product. Trial runbook lands at /trials/nesto.md when scoping starts.
  2. CLI relocation to tools/. Follow-up to Slice A. Requires tools/.goreleaser.yml, a release-tools job in the workflow, an install.sh block fetching the tools archive, and updates to plugin scripts + helm image build. Acceptable to defer until the release-pipeline work has its own dedicated PR.

Repository Layout

Reflects state as of Slice C.2 merge (2026-05-12):

deploy/
  helm/
    demarkus-server/         # one-world chart (Phase 6.1) — shipped PR #107
    demarkus-broker/         # OIDC token broker chart (Phase 6.3)
    demarkus-agent/          # crawl/index agent chart (Phase 6.0) — shipped PR #106
  k8s/
    examples/
      applicationset.yaml    # Argo CD ApplicationSet over a worlds: list
      kustomize-overlay/     # Kustomize alternative
  observability/
    datadog/                 # autodiscovery annotations + dashboard JSON
    otel-collector/          # collector config recipes
    vector/                  # vector config recipes
    fluent-bit/              # fluent-bit parser + filter recipes
  scripts/                   # operator helpers (cert pre-check, MTU probe, etc.)

protocol/
  auth.go                    # HashToken — sha256-<hex> contract (Slice A)
  token/                     # Generate, ReadFile, AppendEntry, WriteFile,
                             # FormatEntry, flock helpers (Slice A) +
                             # AppendBytes, RemoveBytes in-memory helpers
                             # (Slice B) + ParseBytes read-side helper
                             # (Slice C.2) for the broker's drift sweep.
                             # Shared by CLI and broker; on-disk path is
                             # atomic temp+rename, fsync data + parent dir
                             # for durability, advisory flock(2) for
                             # cross-process serialization, quoted-key
                             # handling for non-bare TOML labels.

server/cmd/
  demarkus-server/           # the server binary (existing)
  demarkus-token/            # token mint CLI — uses protocol/token
                             # (Slice A: rewired, not moved; tools/
                             # relocation deferred — see Open Questions)
  demarkus-publish/          # publish CLI (existing; deferred move)

client/cmd/
  demarkus-agent/            # protocol client — federation crawler (existing)

tools/
  demarkus-broker/           # broker binary (Slice B PR #109, Slice C.1
                             # PR #110, Slice C.2 PR #111) — imports
                             # protocol/token for byte-identical hashes
                             # (Phase 6.2). main.go + internal/broker/
                             # { config, session, oidc, issuer, server,
                             #   labels, sweeper }. Slice B shipped the
                             # single-world OIDC mint flow; Slice C.1
                             # added groups-claim authorization,
                             # AllowEmails carve-out, and email
                             # canonicalization in Mint; Slice C.2 added
                             # the expiry+drift sweeper with
                             # coordination.k8s.io/Lease leader election
                             # (singleton across replicas). Remaining
                             # Slice C work: C.3 rotate, C.4 rate limit.
                             # SCIM webhook stays in backlog.

Sub-Phases

6.0 — demarkus-agent verified + chart ✓ binary, ✓ chart (merged PR #106)

Binary status (2026-05-11): existing demarkus-agent verified end-to-end against a 3-server smoke test (2 team worlds + 1 hub). Crawls, builds aggregated and per-server indexes, publishes to hub. Two bugs fixed during verification:

  • fedcrawl/crawl.go publishIndex accepted only ok status; first publish returns created, surfaced as a misleading warning despite the publish succeeding. Now accepts both.
  • publishIndex always used expected_version=0 (create-only), causing every re-publish in daemon mode to fail with conflict. Now uses -1 (no check) for idempotent hub re-publish; server's no-op-on-duplicate-content prevents version churn.
  • Makefile did not build demarkus-agent; the agent had to be hand-built. Now builds with make client.

Tests in client/internal/fedcrawl/crawl_test.go cover all three: create + re-publish + status acceptance + per-server / aggregated modes. bash pre-commit.sh clean.

Agent Helm chart shipped in PR #106 at deploy/helm/demarkus-agent/:

  • Deployment (not StatefulSet — agent is stateless modulo state file; recoverable from next crawl).
  • ConfigMap holding the TOML agent config (seeds, hubs, crawl, politeness, schedule).
  • Secret holding per-host tokens for publishing to hub(s).
  • ServiceAccount, no special RBAC needed (no k8s API calls).
  • Exec liveness probe (demarkus-agent version returns 0).
  • No Service (agent is outbound-only).
  • Pod annotations for Datadog autodiscovery; log fields documented for mint/crawl events.

6.1 — demarkus-server Helm chart ✓ (merged PR #107)

deploy/helm/demarkus-server/. Production-grade.

Workload:

  • StatefulSet, 1 replica (multi-replica is Phase 7).
  • volumeClaimTemplates — each world owns its PVC. Never a shared PVC.
  • Container image bundles demarkus-server + demarkus CLI (for exec probes).
  • Exec liveness + readiness probes against /.well-known/agent-manifest.md.
  • Resource requests/limits with sane defaults, overridable.
  • Service type LoadBalancer, protocol: UDP, port from server.udpPort (default 6309). Annotations for cloud-specific LB type (NLB on AWS, etc.).

Secrets (world namespace):

  • <release>-tokens — TOML of SHA-256 hashes. Server-mounted.
    • Persistence across helm upgrade is non-negotiable. Rendered with helm.sh/resource-policy: keep + helm.sh/hook: pre-install and empty data: {} in the template body. Helm creates the Secret on first install and never overwrites it on subsequent upgrades; broker writes via k8s API are preserved.
    • Bootstrap Job (below) seeds the initial admin token on first install only. The broker manages all subsequent entries at runtime via the k8s API.
    • Without this pattern, every helm upgrade would silently wipe every broker-minted token. This is the single most important Helm correctness property in the chart and must be unit-tested.
    • kubelet propagation: Secret data changes propagate to the mounted file with a delay (default ~60s). Server re-reads via SIGHUP (preferred) or fsnotify; verify which during chart implementation.
  • Broker state (raw tokens are never persisted; email→label mappings and issuance metadata) lives in the broker namespace, not the world namespace. See §6.2 for the broker's issuances Secret. The world's tokens Secret holds only hashes and is the sole source of truth the server reads.

Persistence boundary recap: pod restarts, node failures, control-plane restarts → fine. helm upgrade → fine iff resource-policy: keep. kubectl delete namespace → everything gone (operations doc covers Velero / VolumeSnapshot recovery). helm uninstall without keep annotation → tokens Secret gone; with keep, retained.

Auth + TLS:

  • TLS Secret mounted via volumeMounts; cert/key paths via flags.
  • Optional cert-manager Certificate resource (behind a flag) requesting *.<root> from a configured ClusterIssuer.

Bootstrap Job:

  • Mints initial admin token on first install via protocol/token.Generate + protocol/token.AppendEntry (Slice A).
  • helm.sh/hook: pre-install only — does not run on upgrade.
  • Idempotent guard: reads the tokens Secret; skips if admin label already exists.
  • SIGHUPs pod after writing (no-op on first install before pod is up).

Observability:

  • Pod annotations for Datadog autodiscovery.
  • Reference configs for OTel Collector, Vector, Fluent Bit in deploy/observability/.
  • slog output already structured; no chart-side instrumentation needed.

RBAC:

  • Bootstrap Job SA with get/update on the named Secret, get/list/create on pods/exec for SIGHUP. Namespace-scoped Role, not ClusterRole.

Tests:

  • helm-unittest for templates, including a test that asserts resource-policy: keep is present on the tokens Secret and that the rendered Secret has empty data: {}.
  • Kind-based integration test in CI: install chart → exec into pod → verify health → publish via CLI → verify version increments → helm upgrade with a changed value → re-verify token-based auth still works (regression guard against the upgrade-wipe footgun).

6.2 — demarkus-broker binary (tools/demarkus-broker/)

Slice B (single-world OIDC mint flow) merged as PR #109 (commit 2a6aae6, 2026-05-11). Slice C.1 (groups-claim authorization + AllowEmails carve-out + email canonicalization) merged as PR #110 (commit 343f808, 2026-05-12). Slice C.2 (expiry + drift sweeper with Lease-based leader election, RBAC-denied mint guard) merged as PR #111 (commit cb5f800, 2026-05-12). Remaining Slice C work — C.3 rotate endpoint, C.4 rate limit — is open; see §Sequencing for the ordering and §Status for the agreed sub-slicing.

Role & topology

  • Issuance authority, not a request proxy. Broker sits on the demarkus login path; clients then talk to world servers directly carrying the raw token. World servers stay identity-blind; broker never sees mark:// requests. Capability model preserved.
  • One broker per universe (cluster) by default. Single OIDC client registration, single worlds: list in values, single issuance state Secret. Multiple brokers only when (a) multiple OIDC providers must coexist, (b) hard tenant isolation between orgs sharing a cluster, or (c) Phase-7 geo split. N>1 is supported but not the common case.
  • Broker SA holds a namespace-scoped Role + RoleBinding in each world's namespace, with get/patch limited to that world's tokens Secret only. No ClusterRole. Blast radius bounded to the token Secrets it's explicitly granted. Slice C.2 sweeper also needs coordination.k8s.io/leases get/create/update in the broker namespace for leader election; chart RBAC (6.3) bundles both into the broker SA.

State

Two distinct Kubernetes Secrets, never merged:

Secret Lives in Contents Reader
<world>-tokens each world's namespace TOML: [tokens.<label>] hash=, paths=, operations=, expires= demarkus-server
<broker>-issuances broker namespace JSON: label → {email, world, paths, operations, issued_at, expires} + secondary index email → [labels] broker only

World servers see only hashes. Email and identity live in broker state alone. Both Secrets follow the §6.1 resource-policy: keep pattern — broker state must also survive helm upgrade of the broker chart.

Plus one cluster-scoped coordination object (Slice C.2):

Resource Lives in Contents Reader
demarkus-broker-sweeper Lease broker namespace holderIdentity, renewTime, leaseDurationSeconds leader-election library only

The Lease is recreated on demand; no migration concern. ReleaseOnCancel=true so rolling restarts hand it off in milliseconds.

Labels

Opaque IDs (usr_<random8>; 4 bytes of entropy, collision-retry up to 5 in the issuer). Not email-derived. Reasons:

  • Sanitizing emails (dots, plus signs, IDN) into TOML keys is fragile — though note protocol/token.FormatEntry does handle quoted-key emission for non-bare labels if you ever need it.
  • Token rotation yields a new label each time; opaque IDs reflect that naturally.
  • "Revoke everything for fredrick@x" becomes a broker-state index lookup, not a label string scan.

Per-world authorization (values schema, as shipped in C.1)

worlds:
  - name: team-a
    namespace: team-a
    tokensSecret: team-a-tokens
    allow:
      domains: ["nesto.ca"]
      groups: ["engineering"]      # OIDC `groups` claim — Slice C.1
      emails: ["alice@example.com"] # per-user carve-out — Slice C.1
    defaultToken:
      paths: ["/team-a/*"]
      operations: ["read", "publish"]
      expiresAfter: 24h
sweeper:
  disabled: false                   # default: sweeper runs
  interval: 5m                      # default: 5 min; capped at 24h
  leaseName: demarkus-broker-sweeper # default

WorldConfig.Allow is a nested struct (the prior AllowDomains flat field was promoted into Allow.Domains during C.1 — Slice B was 2 days old with no deployed configs, so the breaking schema change was free). All three lists are lowercased+trimmed at config load with empty entries rejected at validation. Match is case-insensitive everywhere.

SweeperConfig (Slice C.2) defaults to running with a 5-min cadence; omitting the block gives production-correct behavior. See §Decisions for why the field is named disabled and why interval is capped at 24h.

On demarkus login, broker evaluates the OIDC identity against every world in its config and mints one token per world the user qualifies for. Client may request narrower scope; never broader.

The authorization predicate (worldAllows in issuer.go):

  1. All three lists empty → match (back-compat for worlds with no allowlist configured, same as the pre-Slice-C behavior).
  2. Email in Allow.Emails → match (per-user carve-out bypasses both domain and group requirements; the documented escape hatch when an IdP doesn't surface usable groups in the ID token).
  3. Only Emails was configured and step 2 didn't fire → reject. Without this guard, allow.emails: [alice@x] with no other allowlist would silently mean "everyone plus alice" because the AND of empty Domains + empty Groups evaluates to true. Detected as len(Domains)+len(Groups) == 0 when we reach this branch.
  4. domainMatches AND groupsMatch, where an empty list in either dimension is a no-op for that dimension only. This is the standard "domain restriction intersected with groups restriction" — RBAC inside an org, not across orgs.

Mint also canonicalizes claims.Email (trim + lowercase) once at the top before authorization runs, so the persisted issuance owner is canonical and untrimmed-IdP-emit casing doesn't silently fail domainMatches (which previously kept trailing whitespace in the domain slice via LastIndex("@")).

Token-mint library

Broker imports github.com/latebit/demarkus/protocol/token (Slice A + Slice B + Slice C.2 helpers). Available primitives:

Slice A — on-disk and pure helpers (used by demarkus-token CLI):

  • token.Generate(label, paths, operations) (Minted, error) — pure, no I/O. Returns Minted{Label, Raw, Entry} with Raw as the one-time secret to hand to the user via the OIDC callback response, and Entry as the hash+capabilities to persist.
  • token.AppendEntry(path, label, *Entry) — adds an entry to a tokens.toml file with atomic temp+rename publishing, fsync data + parent dir for crash durability, advisory flock(2) for cross-process serialization, and rejects duplicate labels via ErrLabelExists.
  • token.WriteFile(path, File) — full-replace with the same atomicity/durability/locking guarantees (used for revoke from CLI).
  • token.ReadFile(path) — decode tokens.toml from disk.
  • token.FormatEntry(label, *Entry) — render a single labeled entry; handles bare vs quoted TOML keys correctly.

Slice B — in-memory helpers (used by the broker, which round-trips TOML through k8s Secrets instead of files):

  • token.AppendBytes(existing []byte, label string, entry *Entry) ([]byte, error) — decode existing, append the new labeled entry, re-encode. Same duplicate-check (ErrLabelExists) and quoted-key handling as AppendEntry, minus the disk-side concerns (no temp+rename, no fsync, no flock — k8s API + resourceVersion provide the equivalent ordering guarantees at a different layer).
  • token.RemoveBytes(existing []byte, label string) ([]byte, error) — decode, drop the named label, re-encode. Used for DELETE /tokens/:label and the expiry sweeper.

Slice C.2 — in-memory read-side helper (used by the broker's drift sweep):

  • token.ParseBytes(existing []byte) (File, error) — decode tokens.toml bytes into the same File shape ReadFile returns. Empty input gives an empty File with a non-nil Tokens map (matches ReadFile's "missing file = empty" convention) so membership checks don't need to special-case nil. Used by Sweeper.readWorldLabels to build a label set and compare against broker-side issuances — quoted-key labels survive the round trip, which substring search on the serialized bytes wouldn't.

Broker writes the returned []byte into the world's tokens Secret via Get + Update (read-modify-write with resourceVersion optimistic concurrency, retry-on-conflict up to 5). protocol.HashToken guarantees byte-identical hashes to what world servers read regardless of mint origin.

Revocation

Three triggers, one cleanup index (the issuances Secret):

  1. User-initiated (demarkus token revoke <label> or rotate):

    • DELETE /tokens/:label carrying OIDC ID token (bearer-authed; no broker session cookie — see "Routes" below).
    • Broker confirms entry.email == claims.email (owner check — users can only revoke their own).
    • Calls revokeIssuance (Slice C.2 helper extracted from the original Revoke): patch the world's tokens Secret via RemoveBytes, then drop the entry from the issuances Secret. The sweeper uses the same helper, so the two-Secret mutation logic lives in one place.

  2. Expiry sweeper (Slice C.2, shipped — broker in-process ticker, default every 5 min, leader-elected via Lease so only one replica sweeps):

    • Reads the issuances Secret once per tick.
    • Splits entries into expired (now.After(iss.ExpiresAt)) and not-expired; not-expired entries grouped by world.
    • For each world with non-expired entries: read that world's tokens Secret once via ParseBytes, identify drifted entries (issuance record exists, label gone from world TOML — operator hand-edit, or a prior Revoke that failed midway between the two-Secret writes).
    • Calls revokeIssuance on each (expired ∪ drifted) entry. Per-entry failures log and continue — one stuck label doesn't starve the rest.

  3. Identity lifecycle (user leaves the org):

    • Default: short-lived tokens. defaultToken.expiresAfter of 24h means stale tokens age out within a day. Next demarkus login reruns OIDC; if the user's IdP account is disabled, login fails. No broker→IdP coupling needed. Same approach as AWS STS / gcloud auth login.
    • Backlog: SCIM lifecycle webhook (POST /scim/v2/Users/:id) for enterprise IdPs that push deprovisioning events (Okta, Entra). Optional add when a customer asks.
    • Periodic IdP re-validation rejected — worse trade-off than either above (couples broker to IdP rate limits, slower than SCIM, less simple than short-lived).

SIGHUP intentionally not implemented in C.2 — revocation latency between the world Secret patch and the server stopping to accept the token equals kubelet Secret-mount propagation + whatever the server does on re-mount. Documented in §Risks; revisited in 6.3 chart work where we'll either prove the server already re-reads on mount-time mtime changes or add a SIGHUP path with pods/exec RBAC.

Cleanup edge cases

  • Orphan in tokens.toml (admin-minted via legacy demarkus-token CLI): broker never claims it. Admins manage via CLI. Broker API only operates on labels it minted.
  • Orphan in issuances Secret (admin hand-deleted from world tokens Secret): Slice C.2 sweeper detects drift on each pass via ParseBytes of the world Secret and a label-set membership check; the orphan is dropped from the issuances Secret. Verified by TestSweepPrunesDrift.
  • Issuance for an unconfigured world (operator removed the world between mint and the sweep): same conservative posture as Revoke — log and skip, never silently drop. The issuance record stays so the operator can investigate (e.g. re-add the world to clean up properly). Verified by TestSweepSkipsUnconfiguredWorld.
  • In-flight request after revoke: a raw token already authenticated on a live QUIC connection completes its current request. Property of the capability + connection-reuse model. Documented; not fixable without core server changes.
  • Partial mint across multiple worlds: if a world's Secret patch fails mid-iteration (e.g. RBAC denial), successful mints from prior worlds stay live and the caller gets back the partial result plus the error. The failing world has no orphan in either Secret — mintForWorld only commits to the issuances Secret after the world Secret write succeeds, and the rollback on appendIssuance failure removes the world-Secret entry. Verified by TestMintRBACDeniedNoPartialState (Slice C.2).

Routes (HTTP, behind Ingress)

  • GET /auth/login — OIDC redirect entry point. Sets a signed state cookie (HMAC-SHA256 over JSON {nonce, expiresAt}, HttpOnly+Secure+SameSite=Lax, path-scoped to /auth/callback, default 5-minute TTL).
  • GET /auth/callback — OIDC callback. Verifies state cookie (CSRF), exchanges code, mints tokens for every qualifying world, returns JSON {world → raw_token} once. No broker session cookie is issued — the state cookie is the only cookie in the system, and it exists purely to defend the OIDC dance against CSRF.
  • GET /tokens — list caller's tokens from broker state (labels + metadata only, never raw tokens). Bearer-authed with the user's OIDC ID token, not a broker session.
  • DELETE /tokens/:label — revoke caller-owned token. Bearer-authed (OIDC ID token).
  • POST /tokens/:label/rotate — Slice C.3. Revoke + mint with same scope (preserved from the issuance record, not reread from current config); returns new raw token.
  • GET /healthz, GET /readyz.

Rationale for skipping a broker session: the CLI is the primary /tokens consumer and already handles token lifetimes; the browser only ever sees the one-time JSON callback response. Adding a session cookie would couple broker state to browser state for no gain. The Verifier interface accordingly exposes VerifyIDToken(ctx, raw) (Claims, error) so handlers can authenticate bearer tokens uniformly.

OIDC providers

Provider behind a Verifier interface (AuthCodeURL, Exchange, VerifyIDToken). Initial implementation wraps coreos/go-oidc/v3 + golang.org/x/oauth2. Follow-ons: Okta, Entra ID, Auth0 — config only. Group-claim availability varies per provider; doc page per provider explains required claim configuration (e.g., Entra needs the groups optional claim enabled; some providers require a userinfo call to fetch groups not in the ID token).

Groups claim sourcing (Slice C.1 scope): ID-token only. The Claims struct's Groups []string field is populated from the groups claim in the verified ID token. Providers that surface groups in userinfo instead of the ID token (Google, some Entra configs) need a richer Verifier impl that runs a userinfo call after Verify. Out of Slice C scope; revisit when a customer needs it. The AllowEmails carve-out is the documented workaround in the meantime.

HA

Multi-replica. Issuance writes to k8s Secrets use resourceVersion optimistic concurrency with retry-on-conflict. Sweeper (shipped in C.2) uses a coordination.k8s.io/Lease for leader election via k8s.io/client-go/tools/leaderelection, so only one replica runs the periodic loop at any moment. ReleaseOnCancel=true on graceful shutdown so rollouts hand off the lease in milliseconds rather than waiting out the 15s expiry. Mint/List/Revoke handlers remain on every replica — those are request-driven, not periodic, and don't need coordination beyond the per-Secret optimistic-concurrency retry inside mutateSecret.

Slice B test surface (shipped)

  • Mint flow: table-driven, Verifier test double in oidc_test.go, fake k8s clientset, OIDC discovery mocked via httptest. End-to-end coverage at the HTTP layer via httptest.NewServer.
  • State cookie: HMAC verify path, expiry path, malformed-envelope path, short-key rejection.
  • Config: YAML decode with KnownFields(true) rejecting typos; zero-expiresAfter rejected at startup.
  • Issuer: resourceVersion retry-on-conflict, label collision retry, mint ordering (world Secret first, issuance record second — orphan-in-issuances is the documented partial-failure mode).
  • Server: clock decoupling between server and issuer (state-cookie expiry vs token-expiry assertions need different clocks).

87% line coverage on internal/broker/. bash pre-commit.sh clean.

Slice C.1 test surface (shipped — PR #110)

  • 15-row table TestMintAuthorizationPredicate covers every (Domains × Groups × Emails) × accept/reject cell, including the (domain AND groups) OR email-carve-out predicate, the only-emails-set reject case, and case-insensitive group matching (claim-side).
  • TestMintCanonicalizesEmail: passes " Alice@Example.COM\t" and asserts both domain match succeeded and the persisted issuance email is alice@example.com. Regression guard against the IdP-side whitespace/case footgun that motivated Mint's top-of-function trim+lowercase.
  • TestLoadConfig rows for each allowlist dimension: lowercase normalization at load (domains, emails, groups) and empty-entry rejection (allow.<dim>[N] is empty). Pins the load-side half of the case-insensitive contract documented above.
  • TestDeleteTokenNotOwner (existed since Slice B PR #109): different OIDC subject than the issuance email → 403. Already covered the owner-check 403 path that the original Slice C surface listed as "open"; flagged during C.1 review.

Slice C.2 test surface (shipped — PR #111)

  • TestSweepRetiresExpiredKeepsFresh: pinned-clock sweeper against two issuances (one past ExpiresAt, one fresh). After one sweep(ctx) pass, the expired entry is removed from both Secrets and the fresh one is intact in both. Exercises the primary expiry path.
  • TestSweepPrunesDrift: two issuance records against one world; only one has a matching entry in the world's tokens Secret. After one pass, the drifted issuance record is dropped, the present one is left alone. Verifies ParseBytes-backed drift detection.
  • TestSweepSkipsUnconfiguredWorld: issuance points at a world the broker is no longer configured for; sweeper logs and skips rather than silently dropping. The issuance stays so an operator can investigate. Matches the conservative posture from Revoke.
  • TestSweepEmptyIssuancesIsNoop: empty initial state; no panic, no spurious Secret creation.
  • TestSweeperLeaderElection: two RunLeaderElected against a shared fake clientset. Exactly one is elected, runs sweep; the follower stays at zero sweep count. Cancel the leader's context; ReleaseOnCancel hands the Lease back; the follower takes over and starts sweeping. Both goroutines explicitly cancelled and joined before test return so the next test starts clean.
  • TestMintRBACDeniedNoPartialState (in issuer_test.go): multi-world config; fake-clientset reactor returns Forbidden on update for team-b's tokens Secret. Asserts team-a fully committed (world Secret + issuance record), team-b has zero state in either Secret, and the error names the failing world. Validates the "no orphan on RBAC failure" property the plan §Cleanup edge cases promises.
  • TestLoadConfig rows: sweeper.interval defaults to 5m when omitted; over 24h rejected with explicit error; negative rejected.

bash pre-commit.sh clean. Package coverage 81.0% (up from C.1's 78.9%).

Slice C remaining test surface (open)

  • C.3: rotate happy path; rotate owner-mismatch 403; rotate of unknown label 404; rotate preserves issuance-recorded scope (not current DefaultToken).
  • C.4: per-subject rate limit smoke test on /tokens (assert N+1 returns 429); separate IP-keyed limit on /auth/login.

6.3 — demarkus-broker Helm chart

Multi-replica HA. RBAC bundles per-world Roles for get/patch on each world's tokens Secret plus a broker-namespace Role for get/create/update on coordination.k8s.io/leases (the sweeper's lease). sweeper: block exposed in values with sensible defaults so customers running single-replica dev installs can disable it cleanly.

6.4 — Universe topology examples

ApplicationSet + Kustomize overlay. As previously specified.

6.5 — Observability recipes

Per-backend configs in deploy/observability/. As previously specified. Slice C.2 added sweeper-side log lines (broker: swept, broker: sweep failed, broker: sweeper observing new leader, broker: sweeper lost leadership) that the recipes' parsers should route to the same audit-log fields as mint/revoke.

6.6 — Documentation suite

/deployment/*.md + per-chart READMEs. As previously specified.

6.7 — Release pipeline

GHCR images + OCI charts. Cosign deferred to backlog. Also folds in the deferred CLI relocation (demarkus-token, demarkus-publish) from server/cmd/ to tools/ — that change requires a new tools/.goreleaser.yml + release-tools workflow job + install.sh block, all of which naturally belong in the release-pipeline slice.

Sequencing

  1. Slice A — token-mint library ✓ merged 2026-05-11 (PR #108).
  2. 6.0 chart ✓ merged 2026-05-11 (PR #106).
  3. 6.1 server chart ✓ merged 2026-05-11 (PR #107).
  4. 6.2 broker binary — Slice B ✓ merged 2026-05-11 (PR #109, commit 2a6aae6). Adds protocol/token.AppendBytes / RemoveBytes in-memory helpers for k8s-Secret-backed callers.
  5. 6.2 broker binary — Slice C, split into four landed PRs:
    • C.1 ✓ merged 2026-05-12 (PR #110, commit 343f808) — groups-claim + AllowEmails authorization. Nested WorldConfig.Allow schema (breaking change vs Slice B's AllowDomains); lowercase-at-load + plain == everywhere; case-insensitive group matching; email canonicalization in Mint; configurable groups via ID-token claim.
    • C.2 ✓ merged 2026-05-12 (PR #111, commit cb5f800) — expiry + drift sweeper with coordination.k8s.io/Lease leader election via k8s.io/client-go/tools/leaderelection. Singleton across replicas, ReleaseOnCancel for fast handoff. protocol/token.ParseBytes additive helper for drift detection. Issuer.revokeIssuance extracted as shared helper between user-Revoke and sweeper. sweeper.interval capped at 24h; sweeper.disabled named for safe zero-value default. RBAC-denied mint test pins "no partial state" guarantee.
    • C.3 — POST /tokens/:label/rotate. New Issuer.RotateLabel (reuses cross-world-collision + rollback machinery from mintForWorld); thin handler.
    • C.4 — per-subject rate limit middleware (golang.org/x/time/rate, in-memory per-replica). Keyed by hashSubject(claims.Subject) for authed endpoints, source IP for /auth/login.
    • SCIM webhook stays in backlog (no customer ask yet).
  6. 6.3 broker chart — after 6.2 Slice C testable.
  7. 6.4 topology examples.
  8. 6.5 observability recipes.
  9. 6.6 docs — incremental throughout.
  10. 6.7 release pipeline — final; also relocates demarkus-token / demarkus-publish to tools/.

Rough effort: 1–2 weeks of focused work remaining (3 core code units + Slice C.1 + Slice C.2 shipped; what's left is C.3 + C.4 in the broker, the broker chart, examples, recipes, docs, and the release pipeline).

Backlog (deferred, easy to add later)

  • Cosign signing of images + chart releases. Half-day CI add when wanted.
  • Latency log-enrichment (duration_ms field on request slog lines). Tiny additive change. Defer until trials show it's needed.
  • SCIM lifecycle webhook on the broker (POST /scim/v2/Users/:id) for enterprise IdPs that push deprovisioning events. Adds responsiveness beyond what short-lived tokens give. Optional add when a customer asks.
  • CLI relocation to tools/ (demarkus-token, demarkus-publish). Bundled with 6.7 release-pipeline work — see Open Questions.
  • Userinfo-based groups in the broker Verifier for providers (Google, some Entra configs) that don't surface groups in the ID token. Slice C.1 ships ID-token-only; AllowEmails is the documented workaround.
  • Case-sensitive group matching for Keycloak (or any future case-sensitive IdP) where an operator deliberately maintains distinct case-variant groups for different access levels. Slice C.1 chose case-insensitive matching for the validated IdP set; revisit if a customer asks.
  • Configurable lease timings (sweeper.leaseDuration, renewDeadline, retryPeriod). C.2 hardcodes the client-go defaults (15s/10s/2s); expose only when a customer's failover budget needs different numbers.
  • SIGHUP on revoke + sweep to shrink the revoke-to-effect window from "kubelet Secret-mount propagation" to "near-zero." Needs pods/exec RBAC across world namespaces — non-trivial blast-radius expansion. Verify whether the server already re-reads on mount-time changes during 6.3 chart work; SIGHUP may not be needed at all.
  • Sweeper backing-store migration when the issuances Secret hits the ~1MB / ~5000-issuance k8s storage ceiling. Pagination of sweep() doesn't help (Secrets aren't a paginated resource); the real fix is a different store (CRD with watch/list, ConfigMap-per-user, external KV). Phase 7+.

Risks

  • Observability via logs ceiling. If customers want signals not derivable from current slog (latency, internal state like version counts or PVC fullness), we hit a wall. Mitigation: log enrichment is a small additive change; internal-state metrics derivable by a cluster-side sidecar that calls LIST periodically.
  • Broker secret-write blast radius. Holds k8s API creds across world namespaces. Mitigated by namespace-scoped Roles (one per world), audit log, optional NetworkPolicy. Per-world Role is non-negotiable — never ClusterRole. The Slice C.2 RBAC-denied mint test (TestMintRBACDeniedNoPartialState) validates clean failure when the SA's permission scope shrinks: failing world has zero state, prior worlds in the same Mint stay committed.
  • OIDC provider coupling. First impl is Google; structure so second provider is a one-day add.
  • Token revocation in-flight latency. SIGHUP reloads tokens.toml, but a request already authenticated on a live QUIC connection completes. Property of model. Slice C.2 sweeper does not SIGHUP world servers; revocation latency = kubelet Secret propagation + whatever the server actually does on re-mount. Verify server reload behavior during chart implementation (§6.3); SIGHUP path is backlogged.
  • Helm upgrade wiping tokens. Classic templating footgun: rendering a Secret on every upgrade overwrites broker-minted content. Mitigated by helm.sh/resource-policy: keep + pre-install hook on the world tokens Secret (and the broker issuances Secret). Regression-tested in CI via the kind integration test.
  • Issuances Secret scaling wall. The broker's issuances Secret has a ~1MB hard cap from etcd encoding, roughly 5000-7000 records at ~150 bytes apiece. Long before sweep memory becomes a problem, the Secret stops being writable — that's the real scaling ceiling for the current design. Pagination doesn't apply (Secrets aren't paginated). When we hit this, the Phase-7+ fix is a different backing store (CRD with watch+list, ConfigMap-per-user, external KV), not a streaming sweep. Documented in sweep()'s doc comment.
  • Chart proliferation. Three charts + examples + dashboards. Mitigate with shared common-labels templates.
  • Trial scope creep. First customer is path-B. If feedback pulls scope back to "demo slice" mid-build, decide explicitly.

Status

Plan v11, 2026-05-12. Slice A, 6.0 chart, 6.1 chart, 6.2 broker binary Slice B, 6.2 broker binary Slice C.1, 6.2 broker binary Slice C.2 all merged (PRs #106, #107, #108, #109, #110, #111).

  • Slice B (PR #109, commit 2a6aae6, 2026-05-11): single-world OIDC mint flow with Verifier-interface OIDC provider, signed state cookie (no broker session), bearer-auth on /tokens and DELETE /tokens/:label, k8s Secret writes via resourceVersion retry, opaque usr_<8 hex> labels with collision-retry, 87% line coverage. Slice B also added protocol/token.AppendBytes / RemoveBytes in-memory helpers — the library that started Slice A on disk now serves both disk callers (CLI) and Secret callers (broker) from the same byte-shape contract.
  • Slice C.1 (PR #110, commit 343f808, 2026-05-12): groups-claim + AllowEmails authorization. Nested WorldConfig.Allow schema replaces the flat AllowDomains field (breaking config change; Slice B was 2 days old with no deployed configs). Authorization predicate is (domain AND groups) OR email-carve-out with an explicit reject for the only-emails-set case. Email canonicalized (trim+lowercase) once in Mint to fix an untrimmed-email footgun in domainMatches. Groups, domains, and emails all lowercased+trimmed at config load with empty-entry rejection; group matching is case-insensitive because the validated IdP set (Google/Okta/Entra/Auth0) enforces case-insensitive group-name uniqueness. ID-token-sourced groups only; userinfo-based groups in backlog.
  • Slice C.2 (PR #111, commit cb5f800, 2026-05-12): expiry + drift sweeper, leader-elected via coordination.k8s.io/Lease so only one replica sweeps at a time. Single-pass per tick: read issuances once, group non-expired entries by world, read each affected world's tokens Secret at most once via the new protocol/token.ParseBytes helper, retire (expired ∪ drifted) entries via Issuer.revokeIssuance (extracted from Revoke so user-revoke and sweeper share the two-Secret mutation). Sweeper.RunLeaderElected wraps k8s.io/client-go/tools/leaderelection.RunOrDie with ReleaseOnCancel=true for sub-second handoff on rolling restarts. Lease timings hardcoded to client-go defaults (15s/10s/2s); only sweeper.disabled (default false), sweeper.interval (default 5m, capped 24h), and sweeper.leaseName are exposed in YAML. main.go wires the sweeper alongside HTTP with explicit context cancellation + WaitGroup join on shutdown. RBAC-denied mint guard (TestMintRBACDeniedNoPartialState) pins the "no partial state on a failed world's mint" property. Coverage 81.0% (up from C.1's 78.9%).

Next: 6.2 Slice C remaining:

  • C.3 — POST /tokens/:label/rotate. Preserves issuance-recorded scope (not current DefaultToken); operator scope changes only take effect on next demarkus login.
  • C.4 — per-subject rate limit middleware (golang.org/x/time/rate, in-memory per-replica). Defaults proposed: 10/min subject on /tokens+revoke+rotate (burst 5), 20/min IP on /auth/login (burst 5); both override-able via config.

Then 6.3 broker chart → examples/recipes/docs → 6.7 release pipeline.

trail
  1. soul.demarkus.io v11