# Plan: Universe Deployment

Ship the enterprise-grade deployment for demarkus. The deliverable *is* the deployment: a customer's ops team installs a Helm chart, runs a broker, runs an agent, and has a federated demarkus universe their org can use. Any company evaluating demarkus (their internal "POC") installs the same product an established customer runs in production. There is no separate "POC slice" — the slice mentality is rejected. We build it once, right, and customers trial the real thing.

## Goal

Deliver a complete, supportable, production-grade Kubernetes deployment package for demarkus, including:

1. A Helm chart for `demarkus-server` (one world).
2. A Helm chart for `demarkus-broker` (OIDC token issuance + revocation).
3. A Helm chart for `demarkus-agent` (hub aggregator, crawl-and-index).
4. Reference topology examples (Argo CD `ApplicationSet`, Kustomize overlay).
5. Backend-agnostic observability — structured slog emission + reference configs for Datadog, OTel Collector, Vector, Fluent Bit, Grafana Alloy.
6. Customer-facing documentation — installation, security/threat model, operations, upgrade path, per-provider OIDC setup, observability recipes.
7. A release pipeline producing images and chart releases consumable from GHCR.

The same artifacts power first-customer trial and steady-state operations.

## Non-Goals (Phase 7+ territory)

- Multi-replica worlds with shared storage (RWX / object backend).
- Cross-cluster universe federation.
- Operator with a `World` CRD.
- Hosted / managed SaaS.
- Non-markdown content.

## Constraints

- **No core protocol/server changes.** Period. Per prior precedent (Claude Code plugin, Obsidian plugin, `feedback_plugin_scope.md`). Observability is achieved by log derivation in a collector, not by adding `/metrics` or OTel SDK calls to the server. Note: in Slice A we promoted `HashToken` and the token-mint library to `protocol/` (under `protocol/auth.go` and `protocol/token/`). These are additive helper relocations consumed by server + CLI + future broker; no wire-protocol or server behavior changed.
- Capability-based auth model is non-negotiable. The server never learns identity — only labels.
- Markdown-only scope is non-negotiable.

## Decisions (resolved during planning)

- **Health probes use `exec`, not HTTP.** Both charts ship liveness/readiness `exec` probes that invoke the `demarkus` CLI fetching `/.well-known/agent-manifest.md` (always public per `/architecture.md`). No core change. Same pattern as `redis-cli ping` / `pg_isready`.
- **UDP port is a values knob.** Default `6309`, override via `server.udpPort`. Documented option: switch to `443` for VPN/middlebox-hostile networks (Cloudflare Warp Zero Trust, corp firewalls that filter non-standard UDP). Protocol default stays `6309`.
- **Wildcard TLS via cert-manager DNS-01.** Single `*.<root>` cert covers all worlds + broker hostnames. HTTP-01 cannot work for worlds (no HTTP); DNS-01 is the standard path for QUIC services.
- **DNS topology.** One A record per world + one for the broker, all under the same wildcard zone. Each world exposed by `Service: LoadBalancer` `protocol: UDP`; broker by standard Ingress (HTTPS).
- **Hub aggregator is the existing `demarkus-agent`** at `client/cmd/demarkus-agent/`. Already implements `crawl` + `daemon` subcommands with TOML config, multi-worker crawl, per-host token auth, per-server + aggregated index publishing. Verified end-to-end on 2026-05-11. **Phase 5 is no longer a blocking prerequisite.**
- **`demarkus-agent` lives in `client/cmd/`, not `tools/`.** It's a protocol client (uses `fedcrawl`, `fetch`, `tokens`, `links`), not a utility. Memory rule sharpened: `client/cmd/` = protocol clients (CLI, TUI, MCP, agent); `tools/` = utilities (broker, future sync/ops binaries).
- **Token-mint library lives at `protocol/token/`, not `tools/internal/token/`** (revised mid-Slice-A after CodeRabbit review). `protocol/` is reachable by both `server/` (where `demarkus-token` CLI lives) and `tools/` (where the broker will live), which `tools/internal/` was not. `protocol/` already owns the `HashToken` byte-shape contract, so it's the natural home for the on-disk token primitives that must round-trip identically across CLI mints and broker mints.
- **`demarkus-token` CLI stays at `server/cmd/demarkus-token/`** for now (rewired to import `protocol/token`). Moving the binary to `tools/` was deferred to a follow-up that splits the release pipeline (server release currently ships demarkus-token; tools has no release cadence yet). Same applies to `demarkus-publish` — still at `server/cmd/demarkus-publish/`, deferred because hoisting it requires also hoisting `server/internal/store/` out of internal-package scope.
- **Multi-OIDC.** Broker speaks generic OIDC, not Google-specific. Provider behind a `Verifier` interface. Google validated first; Okta, Entra ID, Auth0 follow with config only.
- **Workload Identity (GKE) as a broker option.** Values flag annotates the broker ServiceAccount with the GSA mapping. Off by default; on for GKE customers wanting no static SA keys. Not applicable to world servers.
- **Broker is HA.** Multi-replica with `resourceVersion` optimistic concurrency on Secret writes, retry on conflict.
- **Observability is log-derived, backend-agnostic.** demarkus already emits structured slog. Charts ship Datadog autodiscovery annotations + reference configs for OTel Collector, Vector, Fluent Bit, Grafana Alloy. Customer's SRE picks the agent/backend. Latency histograms deferred — depends on what current slog includes per request; revisit during trials if needed.
- **Backup/DR is documented, not built.** The chart deliberately does not run backup CronJobs. Operations doc covers Velero, VolumeSnapshot, and `demarkus-agent sync` as DR options.
- **Image hosting:** `ghcr.io/latebit-io/{demarkus-server,demarkus-broker,demarkus-agent}`.
- **Chart registry:** OCI charts in GHCR.
- **Cosign signing:** deferred to backlog. Half-day CI add when wanted.

## Open Questions

1. **First customer trial.** Nesto (`*.library.nesto.ca`) is path-B — trial waits for product. Trial runbook lands at `/trials/nesto.md` when scoping starts.
2. **CLI relocation to `tools/`.** Follow-up to Slice A. Requires `tools/.goreleaser.yml`, a `release-tools` job in the workflow, an `install.sh` block fetching the tools archive, and updates to plugin scripts + helm image build. Acceptable to defer until the release-pipeline work has its own dedicated PR.

## Repository Layout

Reflects state as of Slice B merge (2026-05-11):

```
deploy/
  helm/
    demarkus-server/         # one-world chart (Phase 6.1) — shipped PR #107
    demarkus-broker/         # OIDC token broker chart (Phase 6.3)
    demarkus-agent/          # crawl/index agent chart (Phase 6.0) — shipped PR #106
  k8s/
    examples/
      applicationset.yaml    # Argo CD ApplicationSet over a worlds: list
      kustomize-overlay/     # Kustomize alternative
  observability/
    datadog/                 # autodiscovery annotations + dashboard JSON
    otel-collector/          # collector config recipes
    vector/                  # vector config recipes
    fluent-bit/              # fluent-bit parser + filter recipes
  scripts/                   # operator helpers (cert pre-check, MTU probe, etc.)

protocol/
  auth.go                    # HashToken — sha256-<hex> contract (Slice A)
  token/                     # Generate, ReadFile, AppendEntry, WriteFile,
                             # FormatEntry, flock helpers (Slice A) +
                             # AppendBytes, RemoveBytes in-memory helpers
                             # (Slice B) for callers that round-trip TOML
                             # through k8s Secrets instead of disk.
                             # Shared by CLI and broker; on-disk path is
                             # atomic temp+rename, fsync data + parent dir
                             # for durability, advisory flock(2) for
                             # cross-process serialization, quoted-key
                             # handling for non-bare TOML labels.

server/cmd/
  demarkus-server/           # the server binary (existing)
  demarkus-token/            # token mint CLI — uses protocol/token
                             # (Slice A: rewired, not moved; tools/
                             # relocation deferred — see Open Questions)
  demarkus-publish/          # publish CLI (existing; deferred move)

client/cmd/
  demarkus-agent/            # protocol client — federation crawler (existing)

tools/
  demarkus-broker/           # broker binary (Slice B, PR #109) — imports
                             # protocol/token for byte-identical hashes
                             # (Phase 6.2). main.go + internal/broker/
                             # { config, session, oidc, issuer, server,
                             #   labels }. Single-world OIDC mint flow
                             # merged via PR #109; Slice C splits the
                             # remaining work into C.1 authz + owner-test,
                             # C.2 sweeper + leader-election + RBAC test,
                             # C.3 rotate, C.4 rate limit. SCIM webhook
                             # stays in backlog.
```

## Sub-Phases

### 6.0 — `demarkus-agent` verified + chart ✓ binary, ✓ chart (merged PR #106)

**Binary status (2026-05-11):** existing `demarkus-agent` verified end-to-end against a 3-server smoke test (2 team worlds + 1 hub). Crawls, builds aggregated and per-server indexes, publishes to hub. Two bugs fixed during verification:

- `fedcrawl/crawl.go publishIndex` accepted only `ok` status; first publish returns `created`, surfaced as a misleading warning despite the publish succeeding. Now accepts both.
- `publishIndex` always used `expected_version=0` (create-only), causing every re-publish in daemon mode to fail with `conflict`. Now uses `-1` (no check) for idempotent hub re-publish; server's no-op-on-duplicate-content prevents version churn.
- `Makefile` did not build `demarkus-agent`; the agent had to be hand-built. Now builds with `make client`.

Tests in `client/internal/fedcrawl/crawl_test.go` cover all three: create + re-publish + status acceptance + per-server / aggregated modes. `bash pre-commit.sh` clean.

**Agent Helm chart shipped in PR #106** at `deploy/helm/demarkus-agent/`:
- `Deployment` (not StatefulSet — agent is stateless modulo state file; recoverable from next crawl).
- ConfigMap holding the TOML agent config (`seeds`, `hubs`, `crawl`, `politeness`, `schedule`).
- Secret holding per-host tokens for publishing to hub(s).
- ServiceAccount, no special RBAC needed (no k8s API calls).
- Exec liveness probe (`demarkus-agent version` returns 0).
- No `Service` (agent is outbound-only).
- Pod annotations for Datadog autodiscovery; log fields documented for `mint`/`crawl` events.

### 6.1 — `demarkus-server` Helm chart ✓ (merged PR #107)

`deploy/helm/demarkus-server/`. Production-grade.

Workload:
- `StatefulSet`, 1 replica (multi-replica is Phase 7).
- `volumeClaimTemplates` — each world owns its PVC. Never a shared PVC.
- Container image bundles `demarkus-server` + `demarkus` CLI (for exec probes).
- Exec liveness + readiness probes against `/.well-known/agent-manifest.md`.
- Resource requests/limits with sane defaults, overridable.
- `Service` type `LoadBalancer`, `protocol: UDP`, port from `server.udpPort` (default 6309). Annotations for cloud-specific LB type (NLB on AWS, etc.).

Secrets (world namespace):
- `<release>-tokens` — TOML of SHA-256 hashes. Server-mounted.
  - **Persistence across `helm upgrade` is non-negotiable.** Rendered with `helm.sh/resource-policy: keep` + `helm.sh/hook: pre-install` and empty `data: {}` in the template body. Helm creates the Secret on first install and never overwrites it on subsequent upgrades; broker writes via k8s API are preserved.
  - Bootstrap Job (below) seeds the initial `admin` token on first install only. The broker manages all subsequent entries at runtime via the k8s API.
  - **Without this pattern, every `helm upgrade` would silently wipe every broker-minted token.** This is the single most important Helm correctness property in the chart and must be unit-tested.
  - kubelet propagation: Secret data changes propagate to the mounted file with a delay (default ~60s). Server re-reads via SIGHUP (preferred) or fsnotify; verify which during chart implementation.
- Broker state (raw tokens are never persisted; email→label mappings and issuance metadata) lives in the **broker namespace**, not the world namespace. See §6.2 for the broker's `issuances` Secret. The world's `tokens` Secret holds only hashes and is the sole source of truth the server reads.

Persistence boundary recap: pod restarts, node failures, control-plane restarts → fine. `helm upgrade` → fine *iff* `resource-policy: keep`. `kubectl delete namespace` → everything gone (operations doc covers Velero / VolumeSnapshot recovery). `helm uninstall` without `keep` annotation → tokens Secret gone; with `keep`, retained.

Auth + TLS:
- TLS Secret mounted via `volumeMounts`; cert/key paths via flags.
- Optional `cert-manager` `Certificate` resource (behind a flag) requesting `*.<root>` from a configured `ClusterIssuer`.

Bootstrap Job:
- Mints initial `admin` token on first install via `protocol/token.Generate` + `protocol/token.AppendEntry` (Slice A).
- `helm.sh/hook: pre-install` only — does not run on upgrade.
- Idempotent guard: reads the tokens Secret; skips if `admin` label already exists.
- SIGHUPs pod after writing (no-op on first install before pod is up).

Observability:
- Pod annotations for Datadog autodiscovery.
- Reference configs for OTel Collector, Vector, Fluent Bit in `deploy/observability/`.
- slog output already structured; no chart-side instrumentation needed.

RBAC:
- Bootstrap Job SA with `get/update` on the named Secret, `get/list/create` on `pods/exec` for SIGHUP. Namespace-scoped `Role`, not `ClusterRole`.

Tests:
- `helm-unittest` for templates, including a test that asserts `resource-policy: keep` is present on the tokens Secret and that the rendered Secret has empty `data: {}`.
- Kind-based integration test in CI: install chart → exec into pod → verify health → publish via CLI → verify version increments → `helm upgrade` with a changed value → re-verify token-based auth still works (regression guard against the upgrade-wipe footgun).

### 6.2 — `demarkus-broker` binary (`tools/demarkus-broker/`)

Slice B (single-world OIDC mint flow) **merged as PR #109** (commit `2a6aae6`, 2026-05-11). Slice C (groups-claim authorization, expiry sweeper, leader election, rotate, rate limit, SCIM webhook) is open and split into four landed PRs (C.1–C.4) — see §Sequencing for the ordering and §Status for the agreed sub-slicing.

#### Role & topology

- **Issuance authority, not a request proxy.** Broker sits on the `demarkus login` path; clients then talk to world servers directly carrying the raw token. World servers stay identity-blind; broker never sees `mark://` requests. Capability model preserved.
- **One broker per universe (cluster) by default.** Single OIDC client registration, single `worlds:` list in values, single issuance state Secret. Multiple brokers only when (a) multiple OIDC providers must coexist, (b) hard tenant isolation between orgs sharing a cluster, or (c) Phase-7 geo split. N>1 is supported but not the common case.
- Broker SA holds a namespace-scoped `Role` + `RoleBinding` in **each world's namespace**, with `get/patch` limited to **that world's `tokens` Secret only**. No `ClusterRole`. Blast radius bounded to the token Secrets it's explicitly granted.

#### State

Two distinct Kubernetes Secrets, never merged:

| Secret | Lives in | Contents | Reader |
|---|---|---|---|
| `<world>-tokens` | each world's namespace | TOML: `[tokens.<label>] hash=, paths=, operations=, expires=` | `demarkus-server` |
| `<broker>-issuances` | broker namespace | JSON: `label → {email, world, paths, operations, issued_at, expires}` + secondary index `email → [labels]` | broker only |

World servers see only hashes. Email and identity live in broker state alone. Both Secrets follow the §6.1 `resource-policy: keep` pattern — broker state must also survive `helm upgrade` of the broker chart.

#### Labels

Opaque IDs (`usr_<random8>`; 4 bytes of entropy, collision-retry up to 5 in the issuer). **Not** email-derived. Reasons:

- Sanitizing emails (dots, plus signs, IDN) into TOML keys is fragile — though note `protocol/token.FormatEntry` does handle quoted-key emission for non-bare labels if you ever need it.
- Token rotation yields a new label each time; opaque IDs reflect that naturally.
- "Revoke everything for `fredrick@x`" becomes a broker-state index lookup, not a label string scan.

#### Per-world authorization (values schema)

```yaml
worlds:
  - name: team-a
    namespace: team-a
    tokensSecret: team-a-tokens
    allow:
      domains: ["nesto.ca"]
      groups: ["engineering"]      # OIDC `groups` claim — Slice C.1
      emails: ["alice@example.com"] # per-user carve-out — Slice C.1
    defaultToken:
      paths: ["/team-a/*"]
      operations: ["read", "publish"]
      expiresAfter: 24h
```

On `demarkus login`, broker evaluates the OIDC identity against every world in its config and mints one token per world the user qualifies for. Client may request narrower scope; never broader.

Slice B authorization is domain-allowlist only. Slice C.1 adds `(domain AND groups) OR email-in-allowlist`: groups intersects with domains when groups is non-empty, and the per-user emails list is an OR carve-out for cases where the IdP doesn't surface usable groups in the ID token. All three lists empty preserves current "any verified user" behavior (back-compat).

#### Token-mint library

Broker imports `github.com/latebit/demarkus/protocol/token` (Slice A + Slice B helpers). Available primitives:

**Slice A — on-disk and pure helpers (used by `demarkus-token` CLI):**

- `token.Generate(label, paths, operations) (Minted, error)` — pure, no I/O. Returns `Minted{Label, Raw, Entry}` with `Raw` as the one-time secret to hand to the user via the OIDC callback response, and `Entry` as the hash+capabilities to persist.
- `token.AppendEntry(path, label, *Entry)` — adds an entry to a `tokens.toml` file with atomic temp+rename publishing, `fsync` data + parent dir for crash durability, advisory `flock(2)` for cross-process serialization, and rejects duplicate labels via `ErrLabelExists`.
- `token.WriteFile(path, File)` — full-replace with the same atomicity/durability/locking guarantees (used for revoke from CLI).
- `token.ReadFile(path)` — decode `tokens.toml` from disk.
- `token.FormatEntry(label, *Entry)` — render a single labeled entry; handles bare vs quoted TOML keys correctly.

**Slice B — in-memory helpers (used by the broker, which round-trips TOML through k8s Secrets instead of files):**

- `token.AppendBytes(existing []byte, label string, entry *Entry) ([]byte, error)` — decode `existing`, append the new labeled entry, re-encode. Same duplicate-check (`ErrLabelExists`) and quoted-key handling as `AppendEntry`, minus the disk-side concerns (no temp+rename, no fsync, no flock — k8s API + `resourceVersion` provide the equivalent ordering guarantees at a different layer).
- `token.RemoveBytes(existing []byte, label string) ([]byte, error)` — decode, drop the named label, re-encode. Used for `DELETE /tokens/:label` and the expiry sweeper.

Broker writes the returned `[]byte` into the world's `tokens` Secret via `Patch(StrategicMergePatchType)`, paired with `resourceVersion` optimistic concurrency and retry-on-conflict (up to 5). `protocol.HashToken` guarantees byte-identical hashes to what world servers read regardless of mint origin.

#### Revocation

Three triggers, one cleanup index (the issuances Secret):

1. **User-initiated** (`demarkus token revoke <label>` or `rotate`):
   - `DELETE /tokens/:label` carrying OIDC ID token (bearer-authed; no broker session cookie — see "Routes" below).
   - Broker confirms `entry.email == claims.email` (owner check — users can only revoke their own).
   - Patches the world's `tokens` Secret via `RemoveBytes` to drop the label.
   - Drops the entry from issuances Secret.
   - SIGHUPs world server pod(s) so `tokens.toml` is re-read. (Or relies on server fsnotify if present — verify during implementation.)

2. **Expiry sweeper** (Slice C.2 — broker in-process ticker, default every 5 min, leader-elected via `Lease` so only one replica sweeps):
   - Queries issuances where `expires < now`.
   - For each: patch the world's `tokens` Secret + drop from issuances + SIGHUP.

3. **Identity lifecycle** (user leaves the org):
   - **Default: short-lived tokens.** `defaultToken.expiresAfter` of 24h means stale tokens age out within a day. Next `demarkus login` reruns OIDC; if the user's IdP account is disabled, login fails. No broker→IdP coupling needed. Same approach as AWS STS / `gcloud auth login`.
   - **Backlog: SCIM lifecycle webhook** (`POST /scim/v2/Users/:id`) for enterprise IdPs that push deprovisioning events (Okta, Entra). Optional add when a customer asks.
   - **Periodic IdP re-validation rejected** — worse trade-off than either above (couples broker to IdP rate limits, slower than SCIM, less simple than short-lived).

#### Cleanup edge cases

- **Orphan in `tokens.toml`** (admin-minted via legacy `demarkus-token` CLI): broker never claims it. Admins manage via CLI. Broker API only operates on labels it minted.
- **Orphan in issuances Secret** (admin hand-deleted from world tokens Secret): expiry sweeper (Slice C.2) detects drift on each pass and prunes broker-state entries whose label no longer exists in the world's TOML.
- **In-flight request after revoke**: a raw token already authenticated on a live QUIC connection completes its current request. Property of the capability + connection-reuse model. Documented; not fixable without core server changes.
- **Partial mint across multiple worlds** (Slice B behavior): if world N+1's Secret patch fails mid-iteration, successful mints from worlds 1..N stay live and the caller gets back the partial result plus the error. Orphans land in those worlds' `tokens` Secrets until the Slice-C.2 sweeper prunes them. Acceptable for Slice B since realistic single-broker config has one world; revisit when multi-world is the common case.

#### Routes (HTTP, behind Ingress)

- `GET /auth/login` — OIDC redirect entry point. Sets a signed state cookie (HMAC-SHA256 over JSON `{nonce, expiresAt}`, HttpOnly+Secure+SameSite=Lax, path-scoped to `/auth/callback`, default 5-minute TTL).
- `GET /auth/callback` — OIDC callback. Verifies state cookie (CSRF), exchanges code, mints tokens for every qualifying world, returns JSON `{world → raw_token}` once. **No broker session cookie is issued** — the state cookie is the only cookie in the system, and it exists purely to defend the OIDC dance against CSRF.
- `GET /tokens` — list caller's tokens from broker state (labels + metadata only, never raw tokens). **Bearer-authed with the user's OIDC ID token**, not a broker session.
- `DELETE /tokens/:label` — revoke caller-owned token. Bearer-authed (OIDC ID token).
- `POST /tokens/:label/rotate` — Slice C.3. Revoke + mint with same scope (preserved from the issuance record, not reread from current config); returns new raw token.
- `GET /healthz`, `GET /readyz`.

Rationale for skipping a broker session: the CLI is the primary `/tokens` consumer and already handles token lifetimes; the browser only ever sees the one-time JSON callback response. Adding a session cookie would couple broker state to browser state for no gain. The `Verifier` interface accordingly exposes `VerifyIDToken(ctx, raw) (Claims, error)` so handlers can authenticate bearer tokens uniformly.

#### OIDC providers

Provider behind a `Verifier` interface (`AuthCodeURL`, `Exchange`, `VerifyIDToken`). Initial implementation wraps `coreos/go-oidc/v3` + `golang.org/x/oauth2`. Follow-ons: Okta, Entra ID, Auth0 — config only. Group-claim availability varies per provider; doc page per provider explains required claim configuration (e.g., Entra needs the `groups` optional claim enabled; some providers require a userinfo call to fetch groups not in the ID token).

**Groups claim sourcing (Slice C.1 scope):** ID-token only. Providers that surface groups in `userinfo` instead of the ID token (Google, some Entra configs) need a richer `Verifier` impl that runs a userinfo call after `Verify`. Out of Slice C scope; revisit when a customer needs it. The `AllowEmails` carve-out is the documented workaround in the meantime.

#### HA

Multi-replica. Issuance writes to k8s Secrets use `resourceVersion` optimistic concurrency with retry-on-conflict. Sweeper uses a `coordination.k8s.io/Lease` for leader election so only one replica runs the periodic loop (Slice C.2).

#### Slice B test surface (shipped)

- Mint flow: table-driven, `Verifier` test double in `oidc_test.go`, fake k8s clientset, OIDC discovery mocked via `httptest`. End-to-end coverage at the HTTP layer via `httptest.NewServer`.
- State cookie: HMAC verify path, expiry path, malformed-envelope path, short-key rejection.
- Config: YAML decode with `KnownFields(true)` rejecting typos; zero-`expiresAfter` rejected at startup.
- Issuer: `resourceVersion` retry-on-conflict, label collision retry, mint ordering (world Secret first, issuance record second — orphan-in-issuances is the documented partial-failure mode).
- Server: clock decoupling between server and issuer (state-cookie expiry vs token-expiry assertions need different clocks).

87% line coverage on `internal/broker/`. `bash pre-commit.sh` clean.

#### Slice C test surface (open)

- **C.1**: groups-claim authz table (allow/deny crosses domain × groups × emails); empty-all-three preserves "any verified user"; `AllowEmails` carve-out fires when groups don't match; owner-check 403 on `DELETE /tokens/:label` (existing untested path).
- **C.2**: sweeper happy path + drift-pruning (orphan-in-issuances) using fake clock + fake clientset; two-replica leader-election fake-clientset test asserting only the leader runs the loop; RBAC permission-denied test (broker SA without `patch` on a world's Secret → mint fails cleanly with no partial state).
- **C.3**: rotate happy path; rotate owner-mismatch 403; rotate of unknown label 404; rotate preserves issuance-recorded scope (not current `DefaultToken`).
- **C.4**: per-subject rate limit smoke test on `/tokens` (assert N+1 returns 429); separate IP-keyed limit on `/auth/login`.

### 6.3 — `demarkus-broker` Helm chart

Multi-replica HA. As previously specified.

### 6.4 — Universe topology examples

ApplicationSet + Kustomize overlay. As previously specified.

### 6.5 — Observability recipes

Per-backend configs in `deploy/observability/`. As previously specified.

### 6.6 — Documentation suite

`/deployment/*.md` + per-chart READMEs. As previously specified.

### 6.7 — Release pipeline

GHCR images + OCI charts. Cosign deferred to backlog. **Also folds in the deferred CLI relocation** (`demarkus-token`, `demarkus-publish`) from `server/cmd/` to `tools/` — that change requires a new `tools/.goreleaser.yml` + `release-tools` workflow job + `install.sh` block, all of which naturally belong in the release-pipeline slice.

## Sequencing

1. **Slice A — token-mint library** ✓ merged 2026-05-11 (PR #108). `HashToken` and the on-disk token primitives live at `protocol/auth.go` + `protocol/token/`. Server's `demarkus-token` CLI rewired to the library; legacy bugs (unquoted TOML keys, non-atomic appends) fixed in passing. Build infra (`Makefile`, `pre-commit.sh`) extended to lint `tools/`.
2. **6.0 chart** ✓ merged 2026-05-11 (PR #106).
3. **6.1 server chart** ✓ merged 2026-05-11 (PR #107).
4. **6.2 broker binary — Slice B** ✓ merged 2026-05-11 (PR #109, commit `2a6aae6`). Adds `protocol/token.AppendBytes` / `RemoveBytes` in-memory helpers for k8s-Secret-backed callers.
5. **6.2 broker binary — Slice C**, split into four landed PRs:
   - **C.1** — groups-claim + `AllowEmails` authorization + owner-check 403 test. Touches `oidc.go`, `config.go`, `issuer.go` and their test files. No new deps.
   - **C.2** — expiry sweeper + Lease-based leader election + drift-pruning + RBAC-denied test. New `sweeper.go` + tests; depends on `client-go/tools/leaderelection`.
   - **C.3** — `POST /tokens/:label/rotate`. New `Issuer.RotateLabel` (reuses cross-world-collision + rollback machinery from `mintForWorld`); thin handler.
   - **C.4** — per-subject rate limit middleware (`golang.org/x/time/rate`, in-memory per-replica). Keyed by `hashSubject(claims.Subject)` for authed endpoints, source IP for `/auth/login`.
   - SCIM webhook stays in backlog (no customer ask yet).
6. **6.3 broker chart** — after 6.2 Slice C testable.
7. **6.4 topology examples**.
8. **6.5 observability recipes**.
9. **6.6 docs** — incremental throughout.
10. **6.7 release pipeline** — final; also relocates `demarkus-token` / `demarkus-publish` to `tools/`.

Rough effort: 1–3 weeks of focused work remaining (3 of 4 core code units shipped; broker binary Slice B merged; what's left is Slice C in four PRs, the broker chart, examples, recipes, docs, and the release pipeline).

## Backlog (deferred, easy to add later)

- **Cosign signing** of images + chart releases. Half-day CI add when wanted.
- **Latency log-enrichment** (`duration_ms` field on request slog lines). Tiny additive change. Defer until trials show it's needed.
- **SCIM lifecycle webhook** on the broker (`POST /scim/v2/Users/:id`) for enterprise IdPs that push deprovisioning events. Adds responsiveness beyond what short-lived tokens give. Optional add when a customer asks.
- **CLI relocation to `tools/`** (`demarkus-token`, `demarkus-publish`). Bundled with 6.7 release-pipeline work — see Open Questions.
- **Userinfo-based groups** in the broker `Verifier` for providers (Google, some Entra configs) that don't surface groups in the ID token. Slice C.1 ships ID-token-only; `AllowEmails` is the documented workaround.

## Risks

- **Observability via logs ceiling.** If customers want signals not derivable from current slog (latency, internal state like version counts or PVC fullness), we hit a wall. Mitigation: log enrichment is a small additive change; internal-state metrics derivable by a cluster-side sidecar that calls `LIST` periodically.
- **Broker secret-write blast radius.** Holds k8s API creds across world namespaces. Mitigated by namespace-scoped `Role`s (one per world), audit log, optional `NetworkPolicy`. Per-world `Role` is non-negotiable — never `ClusterRole`.
- **OIDC provider coupling.** First impl is Google; structure so second provider is a one-day add.
- **Token revocation in-flight latency.** SIGHUP reloads `tokens.toml`, but a request already authenticated on a live QUIC connection completes. Property of model. Slice C.2 sweeper does not currently SIGHUP world servers; revocation latency = kubelet Secret propagation + whatever the server actually does on re-mount. Verify server reload behavior during chart implementation (§6.3).
- **Helm upgrade wiping tokens.** Classic templating footgun: rendering a Secret on every upgrade overwrites broker-minted content. Mitigated by `helm.sh/resource-policy: keep` + `pre-install` hook on the world tokens Secret (and the broker issuances Secret). Regression-tested in CI via the kind integration test.
- **Chart proliferation.** Three charts + examples + dashboards. Mitigate with shared common-labels templates.
- **Trial scope creep.** First customer is path-B. If feedback pulls scope back to "demo slice" mid-build, decide explicitly.

## Status

Plan v9, 2026-05-12. **Slice A, 6.0 chart, 6.1 chart, 6.2 broker binary Slice B all merged** (PRs #106, #107, #108, #109). Slice B (commit `2a6aae6`, merged 2026-05-11): single-world OIDC mint flow with `Verifier`-interface OIDC provider, signed state cookie (no broker session), bearer-auth on `/tokens` and `DELETE /tokens/:label`, k8s Secret writes via `resourceVersion` retry, opaque `usr_<8 hex>` labels with collision-retry, 87% line coverage. Slice B also added `protocol/token.AppendBytes` / `RemoveBytes` in-memory helpers — the library that started Slice A on disk now serves both disk callers (CLI) and Secret callers (broker) from the same byte-shape contract. **Next: 6.2 Slice C** is broken into four landed PRs:

- **C.1** — groups-claim + `AllowEmails` authorization + owner-check 403 test. `(domain AND groups) OR email-in-list` predicate; empty-all-three preserves current "any verified user" backwards-compat. Groups sourced from the ID token only; userinfo-based groups deferred to backlog.
- **C.2** — expiry sweeper + Lease-based leader election + drift-pruning sweeper test + RBAC-denied test. Sweeper does not SIGHUP world servers in Slice C; revocation latency window documented in §Risks, revisited in 6.3 chart work.
- **C.3** — `POST /tokens/:label/rotate`. Preserves issuance-recorded scope (not current `DefaultToken`); operator scope changes only take effect on next `demarkus login`.
- **C.4** — per-subject rate limit middleware (`golang.org/x/time/rate`, in-memory per-replica). Defaults proposed: 10/min subject on `/tokens`+revoke+rotate (burst 5), 20/min IP on `/auth/login` (burst 5); both override-able via config.

Then **6.3 broker chart** → examples/recipes/docs → 6.7 release pipeline.