soul.demarkus.io/plans/universe-deployment.md/v8 draft reader meta

Plan: Universe Deployment

Ship the enterprise-grade deployment for demarkus. The deliverable is the deployment: a customer's ops team installs a Helm chart, runs a broker, runs an agent, and has a federated demarkus universe their org can use. Any company evaluating demarkus (their internal "POC") installs the same product an established customer runs in production. There is no separate "POC slice" — the slice mentality is rejected. We build it once, right, and customers trial the real thing.

Goal

Deliver a complete, supportable, production-grade Kubernetes deployment package for demarkus, including:

  1. A Helm chart for demarkus-server (one world).
  2. A Helm chart for demarkus-broker (OIDC token issuance + revocation).
  3. A Helm chart for demarkus-agent (hub aggregator, crawl-and-index).
  4. Reference topology examples (Argo CD ApplicationSet, Kustomize overlay).
  5. Backend-agnostic observability — structured slog emission + reference configs for Datadog, OTel Collector, Vector, Fluent Bit, Grafana Alloy.
  6. Customer-facing documentation — installation, security/threat model, operations, upgrade path, per-provider OIDC setup, observability recipes.
  7. A release pipeline producing images and chart releases consumable from GHCR.

The same artifacts power first-customer trial and steady-state operations.

Non-Goals (Phase 7+ territory)

  • Multi-replica worlds with shared storage (RWX / object backend).
  • Cross-cluster universe federation.
  • Operator with a World CRD.
  • Hosted / managed SaaS.
  • Non-markdown content.

Constraints

  • No core protocol/server changes. Period. Per prior precedent (Claude Code plugin, Obsidian plugin, feedback_plugin_scope.md). Observability is achieved by log derivation in a collector, not by adding /metrics or OTel SDK calls to the server. Note: in Slice A we promoted HashToken and the token-mint library to protocol/ (under protocol/auth.go and protocol/token/). These are additive helper relocations consumed by server + CLI + future broker; no wire-protocol or server behavior changed.
  • Capability-based auth model is non-negotiable. The server never learns identity — only labels.
  • Markdown-only scope is non-negotiable.

Decisions (resolved during planning)

  • Health probes use exec, not HTTP. Both charts ship liveness/readiness exec probes that invoke the demarkus CLI fetching /.well-known/agent-manifest.md (always public per /architecture.md). No core change. Same pattern as redis-cli ping / pg_isready.
  • UDP port is a values knob. Default 6309, override via server.udpPort. Documented option: switch to 443 for VPN/middlebox-hostile networks (Cloudflare Warp Zero Trust, corp firewalls that filter non-standard UDP). Protocol default stays 6309.
  • Wildcard TLS via cert-manager DNS-01. Single *.<root> cert covers all worlds + broker hostnames. HTTP-01 cannot work for worlds (no HTTP); DNS-01 is the standard path for QUIC services.
  • DNS topology. One A record per world + one for the broker, all under the same wildcard zone. Each world exposed by Service: LoadBalancer protocol: UDP; broker by standard Ingress (HTTPS).
  • Hub aggregator is the existing demarkus-agent at client/cmd/demarkus-agent/. Already implements crawl + daemon subcommands with TOML config, multi-worker crawl, per-host token auth, per-server + aggregated index publishing. Verified end-to-end on 2026-05-11. Phase 5 is no longer a blocking prerequisite.
  • demarkus-agent lives in client/cmd/, not tools/. It's a protocol client (uses fedcrawl, fetch, tokens, links), not a utility. Memory rule sharpened: client/cmd/ = protocol clients (CLI, TUI, MCP, agent); tools/ = utilities (broker, future sync/ops binaries).
  • Token-mint library lives at protocol/token/, not tools/internal/token/ (revised mid-Slice-A after CodeRabbit review). protocol/ is reachable by both server/ (where demarkus-token CLI lives) and tools/ (where the broker will live), which tools/internal/ was not. protocol/ already owns the HashToken byte-shape contract, so it's the natural home for the on-disk token primitives that must round-trip identically across CLI mints and broker mints.
  • demarkus-token CLI stays at server/cmd/demarkus-token/ for now (rewired to import protocol/token). Moving the binary to tools/ was deferred to a follow-up that splits the release pipeline (server release currently ships demarkus-token; tools has no release cadence yet). Same applies to demarkus-publish — still at server/cmd/demarkus-publish/, deferred because hoisting it requires also hoisting server/internal/store/ out of internal-package scope.
  • Multi-OIDC. Broker speaks generic OIDC, not Google-specific. Provider behind a Verifier interface. Google validated first; Okta, Entra ID, Auth0 follow with config only.
  • Workload Identity (GKE) as a broker option. Values flag annotates the broker ServiceAccount with the GSA mapping. Off by default; on for GKE customers wanting no static SA keys. Not applicable to world servers.
  • Broker is HA. Multi-replica with resourceVersion optimistic concurrency on Secret writes, retry on conflict.
  • Observability is log-derived, backend-agnostic. demarkus already emits structured slog. Charts ship Datadog autodiscovery annotations + reference configs for OTel Collector, Vector, Fluent Bit, Grafana Alloy. Customer's SRE picks the agent/backend. Latency histograms deferred — depends on what current slog includes per request; revisit during trials if needed.
  • Backup/DR is documented, not built. The chart deliberately does not run backup CronJobs. Operations doc covers Velero, VolumeSnapshot, and demarkus-agent sync as DR options.
  • Image hosting: ghcr.io/latebit-io/{demarkus-server,demarkus-broker,demarkus-agent}.
  • Chart registry: OCI charts in GHCR.
  • Cosign signing: deferred to backlog. Half-day CI add when wanted.

Open Questions

  1. First customer trial. Nesto (*.library.nesto.ca) is path-B — trial waits for product. Trial runbook lands at /trials/nesto.md when scoping starts.
  2. CLI relocation to tools/. Follow-up to Slice A. Requires tools/.goreleaser.yml, a release-tools job in the workflow, an install.sh block fetching the tools archive, and updates to plugin scripts + helm image build. Acceptable to defer until the release-pipeline work has its own dedicated PR.

Repository Layout

Reflects state as of Slice B merge (2026-05-11):

deploy/
  helm/
    demarkus-server/         # one-world chart (Phase 6.1) — shipped PR #107
    demarkus-broker/         # OIDC token broker chart (Phase 6.3)
    demarkus-agent/          # crawl/index agent chart (Phase 6.0) — shipped PR #106
  k8s/
    examples/
      applicationset.yaml    # Argo CD ApplicationSet over a worlds: list
      kustomize-overlay/     # Kustomize alternative
  observability/
    datadog/                 # autodiscovery annotations + dashboard JSON
    otel-collector/          # collector config recipes
    vector/                  # vector config recipes
    fluent-bit/              # fluent-bit parser + filter recipes
  scripts/                   # operator helpers (cert pre-check, MTU probe, etc.)

protocol/
  auth.go                    # HashToken — sha256-<hex> contract (Slice A)
  token/                     # Generate, ReadFile, AppendEntry, WriteFile,
                             # FormatEntry, flock helpers (Slice A) +
                             # AppendBytes, RemoveBytes in-memory helpers
                             # (Slice B) for callers that round-trip TOML
                             # through k8s Secrets instead of disk.
                             # Shared by CLI and broker; on-disk path is
                             # atomic temp+rename, fsync data + parent dir
                             # for durability, advisory flock(2) for
                             # cross-process serialization, quoted-key
                             # handling for non-bare TOML labels.

server/cmd/
  demarkus-server/           # the server binary (existing)
  demarkus-token/            # token mint CLI — uses protocol/token
                             # (Slice A: rewired, not moved; tools/
                             # relocation deferred — see Open Questions)
  demarkus-publish/          # publish CLI (existing; deferred move)

client/cmd/
  demarkus-agent/            # protocol client — federation crawler (existing)

tools/
  demarkus-broker/           # broker binary (Slice B) — imports
                             # protocol/token for byte-identical hashes
                             # (Phase 6.2). main.go + internal/broker/
                             # { config, session, oidc, issuer, server,
                             #   labels }. Single-world OIDC mint flow
                             # complete; Slice C adds groups-claim auth,
                             # expiry sweeper, leader election, rotate,
                             # rate limit, SCIM webhook.

Sub-Phases

6.0 — demarkus-agent verified + chart ✓ binary, ✓ chart (merged PR #106)

Binary status (2026-05-11): existing demarkus-agent verified end-to-end against a 3-server smoke test (2 team worlds + 1 hub). Crawls, builds aggregated and per-server indexes, publishes to hub. Two bugs fixed during verification:

  • fedcrawl/crawl.go publishIndex accepted only ok status; first publish returns created, surfaced as a misleading warning despite the publish succeeding. Now accepts both.
  • publishIndex always used expected_version=0 (create-only), causing every re-publish in daemon mode to fail with conflict. Now uses -1 (no check) for idempotent hub re-publish; server's no-op-on-duplicate-content prevents version churn.
  • Makefile did not build demarkus-agent; the agent had to be hand-built. Now builds with make client.

Tests in client/internal/fedcrawl/crawl_test.go cover all three: create + re-publish + status acceptance + per-server / aggregated modes. bash pre-commit.sh clean.

Agent Helm chart shipped in PR #106 at deploy/helm/demarkus-agent/:

  • Deployment (not StatefulSet — agent is stateless modulo state file; recoverable from next crawl).
  • ConfigMap holding the TOML agent config (seeds, hubs, crawl, politeness, schedule).
  • Secret holding per-host tokens for publishing to hub(s).
  • ServiceAccount, no special RBAC needed (no k8s API calls).
  • Exec liveness probe (demarkus-agent version returns 0).
  • No Service (agent is outbound-only).
  • Pod annotations for Datadog autodiscovery; log fields documented for mint/crawl events.

6.1 — demarkus-server Helm chart ✓ (merged PR #107)

deploy/helm/demarkus-server/. Production-grade.

Workload:

  • StatefulSet, 1 replica (multi-replica is Phase 7).
  • volumeClaimTemplates — each world owns its PVC. Never a shared PVC.
  • Container image bundles demarkus-server + demarkus CLI (for exec probes).
  • Exec liveness + readiness probes against /.well-known/agent-manifest.md.
  • Resource requests/limits with sane defaults, overridable.
  • Service type LoadBalancer, protocol: UDP, port from server.udpPort (default 6309). Annotations for cloud-specific LB type (NLB on AWS, etc.).

Secrets (world namespace):

  • <release>-tokens — TOML of SHA-256 hashes. Server-mounted.
    • Persistence across helm upgrade is non-negotiable. Rendered with helm.sh/resource-policy: keep + helm.sh/hook: pre-install and empty data: {} in the template body. Helm creates the Secret on first install and never overwrites it on subsequent upgrades; broker writes via k8s API are preserved.
    • Bootstrap Job (below) seeds the initial admin token on first install only. The broker manages all subsequent entries at runtime via the k8s API.
    • Without this pattern, every helm upgrade would silently wipe every broker-minted token. This is the single most important Helm correctness property in the chart and must be unit-tested.
    • kubelet propagation: Secret data changes propagate to the mounted file with a delay (default ~60s). Server re-reads via SIGHUP (preferred) or fsnotify; verify which during chart implementation.
  • Broker state (raw tokens are never persisted; email→label mappings and issuance metadata) lives in the broker namespace, not the world namespace. See §6.2 for the broker's issuances Secret. The world's tokens Secret holds only hashes and is the sole source of truth the server reads.

Persistence boundary recap: pod restarts, node failures, control-plane restarts → fine. helm upgrade → fine iff resource-policy: keep. kubectl delete namespace → everything gone (operations doc covers Velero / VolumeSnapshot recovery). helm uninstall without keep annotation → tokens Secret gone; with keep, retained.

Auth + TLS:

  • TLS Secret mounted via volumeMounts; cert/key paths via flags.
  • Optional cert-manager Certificate resource (behind a flag) requesting *.<root> from a configured ClusterIssuer.

Bootstrap Job:

  • Mints initial admin token on first install via protocol/token.Generate + protocol/token.AppendEntry (Slice A).
  • helm.sh/hook: pre-install only — does not run on upgrade.
  • Idempotent guard: reads the tokens Secret; skips if admin label already exists.
  • SIGHUPs pod after writing (no-op on first install before pod is up).

Observability:

  • Pod annotations for Datadog autodiscovery.
  • Reference configs for OTel Collector, Vector, Fluent Bit in deploy/observability/.
  • slog output already structured; no chart-side instrumentation needed.

RBAC:

  • Bootstrap Job SA with get/update on the named Secret, get/list/create on pods/exec for SIGHUP. Namespace-scoped Role, not ClusterRole.

Tests:

  • helm-unittest for templates, including a test that asserts resource-policy: keep is present on the tokens Secret and that the rendered Secret has empty data: {}.
  • Kind-based integration test in CI: install chart → exec into pod → verify health → publish via CLI → verify version increments → helm upgrade with a changed value → re-verify token-based auth still works (regression guard against the upgrade-wipe footgun).

6.2 — demarkus-broker binary (tools/demarkus-broker/)

Slice B (single-world OIDC mint flow) is committed on feat-oidc-broker (commit fd18dce, not yet PR'd). Slice C (groups-claim authorization, expiry sweeper, leader election, rotate, rate limit, SCIM webhook) is open.

Role & topology

  • Issuance authority, not a request proxy. Broker sits on the demarkus login path; clients then talk to world servers directly carrying the raw token. World servers stay identity-blind; broker never sees mark:// requests. Capability model preserved.
  • One broker per universe (cluster) by default. Single OIDC client registration, single worlds: list in values, single issuance state Secret. Multiple brokers only when (a) multiple OIDC providers must coexist, (b) hard tenant isolation between orgs sharing a cluster, or (c) Phase-7 geo split. N>1 is supported but not the common case.
  • Broker SA holds a namespace-scoped Role + RoleBinding in each world's namespace, with get/patch limited to that world's tokens Secret only. No ClusterRole. Blast radius bounded to the token Secrets it's explicitly granted.

State

Two distinct Kubernetes Secrets, never merged:

Secret Lives in Contents Reader
<world>-tokens each world's namespace TOML: [tokens.<label>] hash=, paths=, operations=, expires= demarkus-server
<broker>-issuances broker namespace JSON: label → {email, world, paths, operations, issued_at, expires} + secondary index email → [labels] broker only

World servers see only hashes. Email and identity live in broker state alone. Both Secrets follow the §6.1 resource-policy: keep pattern — broker state must also survive helm upgrade of the broker chart.

Labels

Opaque IDs (usr_<random8>; 4 bytes of entropy, collision-retry up to 5 in the issuer). Not email-derived. Reasons:

  • Sanitizing emails (dots, plus signs, IDN) into TOML keys is fragile — though note protocol/token.FormatEntry does handle quoted-key emission for non-bare labels if you ever need it.
  • Token rotation yields a new label each time; opaque IDs reflect that naturally.
  • "Revoke everything for fredrick@x" becomes a broker-state index lookup, not a label string scan.

Per-world authorization (values schema)

worlds:
  - name: team-a
    namespace: team-a
    tokensSecret: team-a-tokens
    allow:
      domains: ["nesto.ca"]
      groups: ["engineering"]      # OIDC `groups` claim — Slice C
    defaultToken:
      paths: ["/team-a/*"]
      operations: ["read", "publish"]
      expiresAfter: 24h

On demarkus login, broker evaluates the OIDC identity against every world in its config and mints one token per world the user qualifies for. Client may request narrower scope; never broader.

Slice B authorization is domain-allowlist only. Groups claim wiring is Slice C.

Token-mint library

Broker imports github.com/latebit/demarkus/protocol/token (Slice A + Slice B helpers). Available primitives:

Slice A — on-disk and pure helpers (used by demarkus-token CLI):

  • token.Generate(label, paths, operations) (Minted, error) — pure, no I/O. Returns Minted{Label, Raw, Entry} with Raw as the one-time secret to hand to the user via the OIDC callback response, and Entry as the hash+capabilities to persist.
  • token.AppendEntry(path, label, *Entry) — adds an entry to a tokens.toml file with atomic temp+rename publishing, fsync data + parent dir for crash durability, advisory flock(2) for cross-process serialization, and rejects duplicate labels via ErrLabelExists.
  • token.WriteFile(path, File) — full-replace with the same atomicity/durability/locking guarantees (used for revoke from CLI).
  • token.ReadFile(path) — decode tokens.toml from disk.
  • token.FormatEntry(label, *Entry) — render a single labeled entry; handles bare vs quoted TOML keys correctly.

Slice B — in-memory helpers (used by the broker, which round-trips TOML through k8s Secrets instead of files):

  • token.AppendBytes(existing []byte, label string, entry *Entry) ([]byte, error) — decode existing, append the new labeled entry, re-encode. Same duplicate-check (ErrLabelExists) and quoted-key handling as AppendEntry, minus the disk-side concerns (no temp+rename, no fsync, no flock — k8s API + resourceVersion provide the equivalent ordering guarantees at a different layer).
  • token.RemoveBytes(existing []byte, label string) ([]byte, error) — decode, drop the named label, re-encode. Used for DELETE /tokens/:label and the expiry sweeper.

Broker writes the returned []byte into the world's tokens Secret via Patch(StrategicMergePatchType), paired with resourceVersion optimistic concurrency and retry-on-conflict (up to 5). protocol.HashToken guarantees byte-identical hashes to what world servers read regardless of mint origin.

Revocation

Three triggers, one cleanup index (the issuances Secret):

  1. User-initiated (demarkus token revoke <label> or rotate):

    • DELETE /tokens/:label carrying OIDC ID token (bearer-authed; no broker session cookie — see "Routes" below).
    • Broker confirms entry.email == claims.email (owner check — users can only revoke their own).
    • Patches the world's tokens Secret via RemoveBytes to drop the label.
    • Drops the entry from issuances Secret.
    • SIGHUPs world server pod(s) so tokens.toml is re-read. (Or relies on server fsnotify if present — verify during implementation.)

  2. Expiry sweeper (Slice C — broker in-process ticker, default every 5 min, leader-elected via Lease so only one replica sweeps):

    • Queries issuances where expires < now.
    • For each: patch the world's tokens Secret + drop from issuances + SIGHUP.

  3. Identity lifecycle (user leaves the org):

    • Default: short-lived tokens. defaultToken.expiresAfter of 24h means stale tokens age out within a day. Next demarkus login reruns OIDC; if the user's IdP account is disabled, login fails. No broker→IdP coupling needed. Same approach as AWS STS / gcloud auth login.
    • Backlog (Slice C): SCIM lifecycle webhook (POST /scim/v2/Users/:id) for enterprise IdPs that push deprovisioning events (Okta, Entra). Optional add when a customer asks.
    • Periodic IdP re-validation rejected — worse trade-off than either above (couples broker to IdP rate limits, slower than SCIM, less simple than short-lived).

Cleanup edge cases

  • Orphan in tokens.toml (admin-minted via legacy demarkus-token CLI): broker never claims it. Admins manage via CLI. Broker API only operates on labels it minted.
  • Orphan in issuances Secret (admin hand-deleted from world tokens Secret): expiry sweeper (Slice C) detects drift on each pass and prunes broker-state entries whose label no longer exists in the world's TOML.
  • In-flight request after revoke: a raw token already authenticated on a live QUIC connection completes its current request. Property of the capability + connection-reuse model. Documented; not fixable without core server changes.
  • Partial mint across multiple worlds (Slice B behavior): if world N+1's Secret patch fails mid-iteration, successful mints from worlds 1..N stay live and the caller gets back the partial result plus the error. Orphans land in those worlds' tokens Secrets until the Slice-C sweeper prunes them. Acceptable for Slice B since realistic single-broker config has one world; revisit when multi-world is the common case.

Routes (HTTP, behind Ingress)

  • GET /auth/login — OIDC redirect entry point. Sets a signed state cookie (HMAC-SHA256 over JSON {nonce, expiresAt}, HttpOnly+Secure+SameSite=Lax, path-scoped to /auth/callback, default 5-minute TTL).
  • GET /auth/callback — OIDC callback. Verifies state cookie (CSRF), exchanges code, mints tokens for every qualifying world, returns JSON {world → raw_token} once. No broker session cookie is issued — the state cookie is the only cookie in the system, and it exists purely to defend the OIDC dance against CSRF.
  • GET /tokens — list caller's tokens from broker state (labels + metadata only, never raw tokens). Bearer-authed with the user's OIDC ID token, not a broker session.
  • DELETE /tokens/:label — revoke caller-owned token. Bearer-authed (OIDC ID token).
  • POST /tokens/:label/rotate — Slice C. Revoke + mint with same scope; returns new raw token.
  • GET /healthz, GET /readyz.

Rationale for skipping a broker session: the CLI is the primary /tokens consumer and already handles token lifetimes; the browser only ever sees the one-time JSON callback response. Adding a session cookie would couple broker state to browser state for no gain. The Verifier interface accordingly exposes VerifyIDToken(ctx, raw) (Claims, error) so handlers can authenticate bearer tokens uniformly.

OIDC providers

Provider behind a Verifier interface (AuthCodeURL, Exchange, VerifyIDToken). Initial implementation wraps coreos/go-oidc/v3 + golang.org/x/oauth2. Follow-ons: Okta, Entra ID, Auth0 — config only. Group-claim availability varies per provider; doc page per provider explains required claim configuration (e.g., Entra needs the groups optional claim enabled; some providers require a userinfo call to fetch groups not in the ID token).

HA

Multi-replica. Issuance writes to k8s Secrets use resourceVersion optimistic concurrency with retry-on-conflict. Sweeper uses a coordination.k8s.io/Lease for leader election so only one replica runs the periodic loop (Slice C).

Slice B test surface (shipped)

  • Mint flow: table-driven, Verifier test double in oidc_test.go, fake k8s clientset, OIDC discovery mocked via httptest. End-to-end coverage at the HTTP layer via httptest.NewServer.
  • State cookie: HMAC verify path, expiry path, malformed-envelope path, short-key rejection.
  • Config: YAML decode with KnownFields(true) rejecting typos; zero-expiresAfter rejected at startup.
  • Issuer: resourceVersion retry-on-conflict, label collision retry, mint ordering (world Secret first, issuance record second — orphan-in-issuances is the documented partial-failure mode).
  • Server: clock decoupling between server and issuer (state-cookie expiry vs token-expiry assertions need different clocks).

87% line coverage on internal/broker/. bash pre-commit.sh clean.

Slice C test surface (open)

  • Sweeper: fake clock + fake clientset; drift-pruning case for orphan-in-issuances.
  • Leader election: two-replica fake-clientset test asserting only the holder runs the loop.
  • Owner check on DELETE: different OIDC subject than the entry → 403.
  • RBAC permission-denied path: broker SA without patch on a world's Secret → mint fails cleanly, not partial state.
  • Rate limit + rotate endpoint coverage.

6.3 — demarkus-broker Helm chart

Multi-replica HA. As previously specified.

6.4 — Universe topology examples

ApplicationSet + Kustomize overlay. As previously specified.

6.5 — Observability recipes

Per-backend configs in deploy/observability/. As previously specified.

6.6 — Documentation suite

/deployment/*.md + per-chart READMEs. As previously specified.

6.7 — Release pipeline

GHCR images + OCI charts. Cosign deferred to backlog. Also folds in the deferred CLI relocation (demarkus-token, demarkus-publish) from server/cmd/ to tools/ — that change requires a new tools/.goreleaser.yml + release-tools workflow job + install.sh block, all of which naturally belong in the release-pipeline slice.

Sequencing

  1. Slice A — token-mint library ✓ merged 2026-05-11 (PR #108). HashToken and the on-disk token primitives live at protocol/auth.go + protocol/token/. Server's demarkus-token CLI rewired to the library; legacy bugs (unquoted TOML keys, non-atomic appends) fixed in passing. Build infra (Makefile, pre-commit.sh) extended to lint tools/.
  2. 6.0 chart ✓ merged 2026-05-11 (PR #106).
  3. 6.1 server chart ✓ merged 2026-05-11 (PR #107).
  4. 6.2 broker binary — Slice B ✓ committed 2026-05-11 (fd18dce on feat-oidc-broker, not yet PR'd). Adds protocol/token.AppendBytes / RemoveBytes in-memory helpers for k8s-Secret-backed callers.
  5. 6.2 broker binary — Slice C — groups-claim authz, expiry sweeper + Lease-based leader election, rotate endpoint, rate limit, SCIM webhook, drift-pruning sweeper test, RBAC-denied test, owner-check 403 test.
  6. 6.3 broker chart — after 6.2 Slice C testable.
  7. 6.4 topology examples.
  8. 6.5 observability recipes.
  9. 6.6 docs — incremental throughout.
  10. 6.7 release pipeline — final; also relocates demarkus-token / demarkus-publish to tools/.

Rough effort: 1–3 weeks of focused work remaining (3 of 4 core code units shipped; broker binary committed; what's left is Slice C, the broker chart, examples, recipes, docs, and the release pipeline).

Backlog (deferred, easy to add later)

  • Cosign signing of images + chart releases. Half-day CI add when wanted.
  • Latency log-enrichment (duration_ms field on request slog lines). Tiny additive change. Defer until trials show it's needed.
  • SCIM lifecycle webhook on the broker (POST /scim/v2/Users/:id) for enterprise IdPs that push deprovisioning events. Adds responsiveness beyond what short-lived tokens give. Slice C.
  • CLI relocation to tools/ (demarkus-token, demarkus-publish). Bundled with 6.7 release-pipeline work — see Open Questions.

Risks

  • Observability via logs ceiling. If customers want signals not derivable from current slog (latency, internal state like version counts or PVC fullness), we hit a wall. Mitigation: log enrichment is a small additive change; internal-state metrics derivable by a cluster-side sidecar that calls LIST periodically.
  • Broker secret-write blast radius. Holds k8s API creds across world namespaces. Mitigated by namespace-scoped Roles (one per world), audit log, optional NetworkPolicy. Per-world Role is non-negotiable — never ClusterRole.
  • OIDC provider coupling. First impl is Google; structure so second provider is a one-day add.
  • Token revocation in-flight latency. SIGHUP reloads tokens.toml, but a request already authenticated on a live QUIC connection completes. Property of model.
  • Helm upgrade wiping tokens. Classic templating footgun: rendering a Secret on every upgrade overwrites broker-minted content. Mitigated by helm.sh/resource-policy: keep + pre-install hook on the world tokens Secret (and the broker issuances Secret). Regression-tested in CI via the kind integration test.
  • Chart proliferation. Three charts + examples + dashboards. Mitigate with shared common-labels templates.
  • Trial scope creep. First customer is path-B. If feedback pulls scope back to "demo slice" mid-build, decide explicitly.

Status

Plan v8, 2026-05-11. Slice A, 6.0 chart, 6.1 chart all merged (PRs #106, #107, #108). 6.2 broker binary Slice B committed (fd18dce on feat-oidc-broker, not yet PR'd) — single-world OIDC mint flow with Verifier-interface OIDC provider, signed state cookie (no broker session), bearer-auth on /tokens and DELETE /tokens/:label, k8s Secret writes via resourceVersion retry, opaque usr_<8 hex> labels with collision-retry, 87% line coverage. Slice B also added protocol/token.AppendBytes / RemoveBytes in-memory helpers — the library that started Slice A on disk now serves both disk callers (CLI) and Secret callers (broker) from the same byte-shape contract. Next: 6.2 Slice C (groups-claim authorization, expiry sweeper + Lease-based leader election, rotate endpoint, rate limit, SCIM webhook, drift-pruning sweeper test, RBAC-denied test, owner-check 403 test) → 6.3 broker chart → examples/recipes/docs → 6.7 release pipeline.

trail
  1. soul.demarkus.io v8