soul.demarkus.io/plans/universe-deployment.md/v16 draft reader meta

Plan: Universe Deployment

Ship the enterprise-grade deployment for demarkus. The deliverable is the deployment: a customer's ops team installs a Helm chart, runs a broker, runs an agent, and has a federated demarkus universe their org can use. Any company evaluating demarkus (their internal "POC") installs the same product an established customer runs in production. There is no separate "POC slice" — the slice mentality is rejected. We build it once, right, and customers trial the real thing.

Goal

Deliver a complete, supportable, production-grade Kubernetes deployment package for demarkus, including:

  1. A Helm chart for demarkus-server (one world).
  2. A Helm chart for demarkus-broker (OIDC token issuance + revocation).
  3. A Helm chart for demarkus-agent (hub aggregator, crawl-and-index).
  4. Reference topology examples (Argo CD ApplicationSet, Kustomize overlay).
  5. Backend-agnostic observability — structured slog emission + reference configs for Datadog, OTel Collector, Vector, Fluent Bit, Grafana Alloy.
  6. Customer-facing documentation — installation, security/threat model, operations, upgrade path, per-provider OIDC setup, observability recipes.
  7. A release pipeline producing images and chart releases consumable from GHCR.

The same artifacts power first-customer trial and steady-state operations.

Non-Goals (Phase 7+ territory)

  • Multi-replica worlds with shared storage (RWX / object backend).
  • Cross-cluster universe federation.
  • Operator with a World CRD.
  • Hosted / managed SaaS.
  • Non-markdown content.

Constraints

  • No core protocol/server changes. Period. Per prior precedent (Claude Code plugin, Obsidian plugin, feedback_plugin_scope.md). Observability is achieved by log derivation in a collector, not by adding /metrics or OTel SDK calls to the server. Note: in Slice A we promoted HashToken and the token-mint library to protocol/ (under protocol/auth.go and protocol/token/). These are additive helper relocations consumed by server + CLI + future broker; no wire-protocol or server behavior changed. Slice C.2 added protocol/token.ParseBytes in the same additive spirit — a read-side helper so the broker's drift sweeper can inspect a world's tokens.toml payload via the map shape rather than substring matching on serialized TOML. §6.7.0 hoisted server/internal/store/ → protocol/store/ in the same additive spirit — the disk-shape of versioned content lives at the protocol layer alongside token/ and the HashToken contract; no API change, four import-path rewrites.
  • Capability-based auth model is non-negotiable. The server never learns identity — only labels.
  • Markdown-only scope is non-negotiable.

Decisions (resolved during planning)

  • Health probes use exec, not HTTP. Both charts ship liveness/readiness exec probes that invoke the demarkus CLI fetching /.well-known/agent-manifest.md (always public per /architecture.md). No core change. Same pattern as redis-cli ping / pg_isready.
  • UDP port is a values knob. Default 6309, override via server.udpPort. Documented option: switch to 443 for VPN/middlebox-hostile networks (Cloudflare Warp Zero Trust, corp firewalls that filter non-standard UDP). Protocol default stays 6309.
  • Wildcard TLS via cert-manager DNS-01. Single *.<root> cert covers all worlds + broker hostnames. HTTP-01 cannot work for worlds (no HTTP); DNS-01 is the standard path for QUIC services.
  • DNS topology. One A record per world + one for the broker, all under the same wildcard zone. Each world exposed by Service: LoadBalancer protocol: UDP; broker by standard Ingress (HTTPS).
  • Hub aggregator is the existing demarkus-agent at client/cmd/demarkus-agent/. Already implements crawl + daemon subcommands with TOML config, multi-worker crawl, per-host token auth, per-server + aggregated index publishing. Verified end-to-end on 2026-05-11. Phase 5 is no longer a blocking prerequisite.
  • demarkus-agent lives in client/cmd/, not tools/. It's a protocol client (uses fedcrawl, fetch, tokens, links), not a utility. Memory rule sharpened: client/cmd/ = protocol clients (CLI, TUI, MCP, agent); tools/ = utilities (broker, future sync/ops binaries).
  • Token-mint library lives at protocol/token/, not tools/internal/token/ (revised mid-Slice-A after CodeRabbit review). protocol/ is reachable by both server/ (where demarkus-token CLI lives) and tools/ (where the broker will live), which tools/internal/ was not. protocol/ already owns the HashToken byte-shape contract, so it's the natural home for the on-disk token primitives that must round-trip identically across CLI mints and broker mints.
  • demarkus-token and demarkus-publish CLIs live at tools/ (§6.7.A merged 2026-05-13 PR #121). demarkus-token had been a clean candidate since Slice A (only imports protocol/token). demarkus-publish followed once §6.7.0 (PR #120) hoisted server/internal/store/ → protocol/store/, removing the server-internal dependency. Both ship as standalone binaries via the (forthcoming) tools/.goreleaser.yml archives; they are NOT bundled into any runtime image.
  • Multi-OIDC. Broker speaks generic OIDC, not Google-specific. Provider behind a Verifier interface. Google validated first; Okta, Entra ID, Auth0 follow with config only.
  • Workload Identity (GKE) as a broker option. Values flag annotates the broker ServiceAccount with the GSA mapping. Off by default; on for GKE customers wanting no static SA keys. Not applicable to world servers.
  • Broker is HA. Multi-replica with resourceVersion optimistic concurrency on Secret writes, retry on conflict.
  • Observability is log-derived, backend-agnostic. demarkus already emits structured slog. Charts ship Datadog autodiscovery annotations + reference configs for OTel Collector, Vector, Fluent Bit, Grafana Alloy. Customer's SRE picks the agent/backend. Latency histograms deferred — depends on what current slog includes per request; revisit during trials if needed.
  • Backup/DR is documented, not built. The chart deliberately does not run backup CronJobs. Operations doc covers Velero, VolumeSnapshot, and demarkus-agent sync as DR options.
  • Image hosting: ghcr.io/latebit-io/{demarkus-server,demarkus-broker,demarkus-agent}. Three separate images, one per service. See §Decisions on per-service split below.
  • Chart registry: OCI charts in GHCR.
  • Cosign signing: deferred to backlog. Half-day CI add when wanted.
  • Groups-claim case sensitivity (revised mid-Slice-C.1 after CodeRabbit back-and-forth). Group names match case-insensitively: AllowConfig.Groups is lowercased+trimmed at config load (same as Domains and Emails) and groupsMatch lowercases the claim's groups before compare. The validated IdP set (Google, Okta, Entra ID, Auth0) enforces case-insensitive group-name uniqueness, so distinct case-variant groups can't exist; a case-sensitive compare would silently fail when an operator writes "Engineering" but the IdP emits "engineering" after upstream normalization. Keycloak with deliberately distinct case-variant groups is out of scope; revisit if a customer asks. Domains and emails get the same lowercase-at-load + plain == treatment for symmetry.
  • Sweeper interval capped at 24h (Slice C.2). The defaultToken.expiresAfter typical is 24h; a sweep interval longer than that would let expired tokens linger past their ExpiresAt for up to a whole sweep cycle, defeating the short-lived-tokens identity-lifecycle model (§Revocation §3). validate() rejects with sweeper.interval must be <= 24h. Lease-election timings (15s lease / 10s renew / 2s retry) intentionally not exposed in YAML — the client-go defaults match our failover budget and adding knobs invites misconfiguration. Revisit if a customer needs sub-15s failover.
  • sweeper.disabled rather than sweeper.enabled (Slice C.2). YAML zero-value (false) is the production-safe default so an operator who omits the block gets the sweeper running. Naming the field positively (Enabled) would have made omission silently disable the janitor — exactly the wrong default for multi-replica deployments where expired tokens accumulate forever without it.
  • Rotate is re-login semantics, not refresh (Slice C.3). POST /tokens/:label/rotate re-runs worldAllows(world.Allow, claims) on every call, so a user removed from the world's allowlist between mint and rotate can NOT extend access via rotation. The bearer ID-token verification at the HTTP layer already proves the IdP identity is still active; the per-world predicate re-check catches in-org permission changes (group removed, domain renamed, email taken off carve-out). Picked over rotation-as-refresh because rotation extends the lifetime — without re-auth, a user whose IdP account stayed live could rotate forever even after losing world-scope authorization.
  • Rotate scope vs. lifetime asymmetry (Slice C.3). Scope (paths + operations) stays frozen to the issuance record on rotate: operator narrowing of DefaultToken.Paths between mint and rotate does NOT shrink the rotated token's reach, because the user shouldn't be surprised by reduced access mid-session — they re-login to pick up new scope. Lifetime resets to now + DefaultToken.ExpiresAfter from the operator's CURRENT config: operator tightening of expiry DOES apply on rotate, because shorter lifetimes are an explicit security-tightening lever and rotation must not bypass it. The asymmetry is intentional and pinned by TestRotateLabelPreservesIssuanceScope + TestRotateLabelLifetimeUpdatesToCurrentConfig.
  • Rotate sequence: mint new, then revoke old (Slice C.3). On revoke failure the new token is returned anyway with a wrapped soft error; the sweeper retires the orphan old label on expiry (or via drift if an operator hand-cleans). Picked over revoke-then-mint because that alternative leaves the user with no token if mint then fails, forcing a full re-login — worse UX than a brief two-valid-tokens window for the same user. Mirrors the Slice B partial-mint convention: (MintResult{Label:""}, err) for hard failures, (MintResult{non-empty}, err) for soft partial success.
  • Middleware-not-handler auth + rate limit (Slice C.4). The three authed /tokens routes go through requireAuth → subjectRateLimit → handler; /auth/login goes through ipRateLimit → authLogin; /auth/callback, /healthz, /readyz stay middleware-free. The old per-handler s.authenticate(w, r) was extracted into the requireAuth middleware that stashes verified Claims on r.Context() via a typed key; handlers read claims via claimsFromCtx. The alternative — keeping auth in handlers and adding rate-limit inside each one — would either double-verify the bearer (one verify for the limit key, one in the handler) or hide the limiter inside each handler, opaque to the route-registration block. Middleware composition keeps the per-route policy visible in Routes().
  • One shared subject bucket across the three /tokens routes (Slice C.4). subjectRateLimit keys on hashSubject(claims.Subject) and the same registry covers GET /tokens, DELETE /tokens/:label, and POST /tokens/:label/rotate. Picked over three per-route buckets so a misbehaving client cannot multiply effective throughput by fanning out (3 routes × 10/min would give 30/min effective rather than the operator-intended 10/min). Pinned by TestRateLimitTokensSharedBucketAcrossRoutes.
  • Reserve() + Cancel() on denial, not Allow() (Slice C.4). Functional equivalence — neither pattern consumes budget on denial — but Reserve().Delay() gives the precise wait time, which we surface as Retry-After with a 1s minimum floor. (Retry-After: 0 reads as "retry immediately" to aggressive clients and would defeat the limiter.) Pinned by TestRateLimitRegistryDenialDoesNotConsumeBudget (50 denials in a tight loop, then 150ms regen window allows the 51st request) and the Retry-After assertions on the integration 429 tests.
  • trustForwardedFor: false default in the binary (Slice C.4). The broker behind an Ingress sees the controller's IP in r.RemoteAddr, so the per-IP limiter on /auth/login collapses into one bucket for every client unless we honor XFF. But trusting XFF when NOT behind a proxy lets an attacker rotate the header to bypass per-IP limits. The binary ships safe; chart-side §6.3.A inverts the default to true (the chart's "behind an Ingress" assumption holds in deployment). TestRateLimitLoginIPIgnoresForwardedForByDefault pins the default-untrusted behavior at the binary level; TestRateLimitLoginIPCrossIPIsolation pins the trust-enabled behavior.
  • rateLimit.disabled not enabled (Slice C.4). Same shape as sweeper.disabled. Zero-value (false) gives the production-safe behavior so an operator who omits the rateLimit: block in their values file still gets the protection. Defaults applied at validate-time: 10/min subject burst 5, 20/min IP burst 5. Field validation skipped entirely when Disabled: true so an operator opting out can leave the per-route knobs empty.
  • Per-replica unbounded registry, by design (Slice C.4). The rateLimitRegistry keeps one *rate.Limiter per key with no TTL eviction and no max-entry cap. CodeRabbit flagged this as a memory-DoS vector; rejected after re-examining the threat model: (a) subjectReg growth is bounded by IdP user count (every key required a successful requireAuth pass, which means a valid IdP-issued token), (b) loginReg growth is bounded by the deployment posture (with trustForwardedFor=false the key is r.RemoteAddr, bounded by clients reaching the listener; with trustForwardedFor=true plus a correctly-configured XFF-stripping Ingress, the key space is bounded by real client IPs), and (c) TTL/LRU eviction makes the property worse under the stated attack: an attacker rotating keys faster than the TTL gets fresh buckets indefinitely, and LRU eviction during attack evicts legitimate users' older buckets first while the attacker's fresh buckets stay in the map. The right Phase-7+ fix lives at a different layer (cluster-shared rate limiter so per-replica eviction can't reset attacker state, or per-IP rate-limit annotations at the ingress controller). Realistic worst-case sizing is ~500KB at ~10k authenticated subjects — not memory-DoS-shaped. Mirrors the C.2 "sweep is unbounded" pushback in shape.
  • OIDC_CLIENT_SECRET env-var override (§6.3.C, additive to §6.2). broker.LoadConfig.applyEnvOverrides() (tools/demarkus-broker/internal/broker/config.go:262) lets the OAuth client secret come from the environment instead of the on-disk config file. Env wins over file when both set; empty env is treated as unset so an accidentally-cleared variable can't blank out a file-supplied value. Production deployments keep the OAuth secret in an externally-managed Kubernetes Secret (External Secrets Operator, Sealed Secrets, Vault) mounted via secretKeyRef, instead of baking it into the chart-rendered config Secret where it would leak into helm release history. This is a one-line escape hatch for the production-secret-ref deployment shape, NOT a §6.2 reopening — §6.2 broker binary complete framing stands. Pinned by TestLoadConfigOIDCClientSecretEnvOverride (4-row table).
  • helm-unittest pinned to v0.6.2 in CI (§6.3.D.1). v1.0+ uses platformHooks in plugin.yaml which needs helm v3.16+; Fritz's local helm is v3.13.2. Pinning v0.6.2 in CI matches the local-dev validation surface so test behavior is identical between developer machines and CI. Bump together when helm itself bumps. Helm pinned to v3.13.2 alongside.
  • test-broker Go CI job folded into §6.3.D.1. The §6.2 broker binary had zero CI coverage at the start of §6.3 chart work — .github/workflows/ci.yml only filtered protocol/server/client, not tools/. Adding helm-unittest plumbing while leaving the binary CI-uncovered would have been incoherent. New test-broker job runs Go test/vet/lint/build for tools/demarkus-broker/, with working-directory: tools/demarkus-broker (narrowed past the broader tools/ scope to avoid lint failures on the package-marker file tools/tools.go). Path filter scoped to tools/demarkus-broker/** + tools/go.mod + tools/go.sum + protocol/** — broker CI does NOT run on unrelated tool changes.
  • Three-branch hybrid template for upgrade-safe Secrets (§6.3.D.2). deploy/helm/demarkus-server/templates/tokens.yaml keys on the live Secret's state: (1) first install → render fresh + keep annotation; (2) legacy upgrade (Secret exists, no keep annotation) → render existing data verbatim + add the annotation, a one-time migration so future upgrades flip to branch 3; (3) race-free no-op (Secret exists + has keep annotation) → emit nothing, helm GC blocked by live annotation, broker stays sole writer. Picked over Option A (always render from lookup) because Option A keeps helm "managing" the Secret on every upgrade, opening a race window between helm template's lookup snapshot and helm apply while the broker is concurrently writing via mutateSecret — apply uses last-writer-wins on fields outside strategic-merge-patch and would clobber concurrent broker mutations. The hybrid's branch 3 eliminates the race entirely by removing helm from the write path post-migration.
  • SIMULATE_LEGACY_NO_KEEP test-mode flag over a separate script (§6.3.D.2). deploy/helm/test-upgrade-wipe.sh exercises both the normal three-branch path and the migration path via one env-flag-controlled script. Two scripts would have drifted in assertion shape; one script keeps the property-under-test identical. The flag strips the annotation from the live Secret between install and upgrade to simulate a pre-D.2 release, then asserts the legacy branch re-applies the annotation while preserving data.
  • test-charts-kind CI job (§6.3.D.2). New 47-line job in .github/workflows/ci.yml; spins up a kind cluster and runs test-upgrade-wipe.sh three times: server normal + server SIMULATE_LEGACY_NO_KEEP + broker normal. Triggers only on charts-path changes. Broker chart doesn't have a legacy state to migrate from (shipped with the keep annotation in §6.3.A) so a broker-legacy entry would test nothing.
  • kubectl jsonpath bracket notation for hyphenated keys (§6.3.D.2 CodeRabbit round 2). {.data['key-with-hyphens']}, NOT {.data.key-with-hyphens}. The dot form happens to work on kubectl's permissive parser but isn't spec-conformant; a stricter version could break it silently. The kind regression script uses the bracket form throughout. Pin this convention for any future jsonpath against k8s data fields.
  • Per-service split container images over a unified single-image bundle (§6.7.A). Three images: ghcr.io/latebit-io/demarkus-server (server + CLI for probes), demarkus-broker (single binary + CA bundle for OIDC), demarkus-agent (single binary + CA bundle for remote mark:// crawl TLS). Initial §6.7.A draft packaged all six binaries into one ghcr.io/latebit-io/demarkus image with each chart picking via command: field; reverted to split because demarkus's shape (multi-component product like Kubernetes/Prometheus — three distinct protocol surfaces) calls for per-service blast-radius isolation. A CVE in broker's OIDC dependency shouldn't fail the server pod's image scan. Cost: 3 CI builds vs 1, ~48MB total across images vs 52.8MB unified. Reversible if a customer requests single-image distribution.
  • Admin CLIs not bundled in runtime images (§6.7.A). demarkus-token and demarkus-publish ship as standalone binaries via tools/.goreleaser.yml (forthcoming §6.7.B). Operators run them locally with port-forward or in dedicated Job pods with the right PVC mount, NOT via kubectl exec into long-running pods. Bundling admin tools into runtime images would leak "admin tooling" into the running pod's blast radius — the tighter abstraction keeps each image scoped to its one primary service binary plus only the runtime support it needs (CLI for server's exec probes, CA bundle for broker/agent's outbound TLS).
  • CA bundle selectively included (§6.7.A). Broker + agent images copy /etc/ssl/certs/ca-certificates.crt from the build stage (broker calls OIDC discovery + token endpoints over HTTPS; agent crawls remote mark:// servers with insecure: false in production). Server image skips it — no outbound TLS today; bundled CLI's exec probes use -insecure for localhost. Adding CA bundle to server "just in case" would leak "this image makes outbound TLS calls" into its abstraction; one-line add later if needed.
  • USER 65532:65532 in all three runtime stages (§6.7.A). Matches each chart's podSecurityContext.runAsUser for defense-in-depth — images run non-root even when deployed outside k8s. The chart's runAsNonRoot: true is the authoritative enforcement; the Dockerfile-level USER ensures the image is also self-contained-secure.
  • go build -C <module> over RUN cd <module> && go build (§6.7.A). Go 1.20+ has -C flag for "change directory before running"; idiomatic, cleaner, and addresses Trivy DS-0013 ("RUN should not be used to change directory"). Auto-creates the -o parent directory in 1.26, so no mkdir -p needed (verified by a clean-container test against CodeRabbit's "Critical" alarm).
  • No SHA256 digest pinning on golang:1.26-alpine (§6.7.A, CodeRabbit nit skipped). Floating the minor tag picks up Go stdlib security patches automatically; pinning would lock all three Dockerfiles to a fixed 1.26.x patch level and require coordinated manual bumps on every Go release. Without renovate/dependabot wired to track base-image digests, pinning trades reproducibility for a maintenance hazard. Revisit when base-image automation lands alongside module-bump automation.

Open Questions

  1. First customer trial. Nesto (*.library.nesto.ca) is path-B — trial waits for product. Trial runbook lands at /trials/nesto.md when scoping starts.
  2. Order of attack confirmed for §6.4–§6.7 remainder (2026-05-13): §6.7 → §6.5 → §6.6 → §6.4. §6.7 split into §6.7.0 (store hoist, merged), §6.7.A (image consolidation + CLI relocation, merged), §6.7.B (release plumbing, next).
  3. Two-week deadline (2026-05-27): §6.7.B release plumbing + §6.5 observability recipes + §6.6 docs + §6.4 topology examples all must land. §6.7.0 + §6.7.A complete in one day signals tight execution is feasible; remaining work is mostly config-and-docs heavy with low engineering surface beyond §6.7.B.
  4. Hardened broker chart symmetry (deferred from §6.3.D.2). The broker chart's secret-issuances.yaml uses the simpler lookup-skip pattern. Defense-in-depth follow-up; not load-bearing. Only act on if Fritz brings it up.

Repository Layout

Reflects state as of §6.7.A merge (2026-05-13):

deploy/
  helm/
    demarkus-server/         # one-world chart (Phase 6.1) — shipped PR #107
      templates/tokens.yaml  # three-branch hybrid; §6.3.D.2 PR #119 added keep
                             # annotation + race-free no-op + legacy-migration
      tests/                 # helm-unittest suites — wired into CI by §6.3.D.1
                             # PR #118; §6.3.D.2 PR #119 added keep+namespace
                             # assertions on tokens_test.yaml
    demarkus-broker/         # OIDC token broker chart (Phase 6.3) — shipped via
                             # PR #115, #116, #117
      tests/                 # helm-unittest suites — shipped by §6.3.D.1 PR #118
    demarkus-agent/          # crawl/index agent chart (Phase 6.0) — shipped PR #106
      tests/                 # helm-unittest suites — pre-existing template-include
                             # bug fixed by §6.7.A PR #121
  test-upgrade-wipe.sh       # generic kind regression: install → kubectl-patch
                             # → upgrade → assert data persists. SIMULATE_LEGACY_NO_KEEP
                             # env flag exercises migration path. (§6.3.D.2 PR #119)
  k8s/
    examples/
      applicationset.yaml    # Argo CD ApplicationSet over a worlds: list (§6.4)
      kustomize-overlay/     # Kustomize alternative (§6.4)
  observability/
    datadog/                 # autodiscovery annotations + dashboard JSON (§6.5)
    otel-collector/          # collector config recipes (§6.5)
    vector/                  # vector config recipes (§6.5)
    fluent-bit/              # fluent-bit parser + filter recipes (§6.5)
  scripts/                   # operator helpers (cert pre-check, MTU probe, etc.)

.dockerignore                # repo-root, applies to all three image builds
                             # (§6.7.A PR #121)

protocol/
  auth.go                    # HashToken — sha256-<hex> contract (Slice A)
  token/                     # Generate, ReadFile, AppendEntry, WriteFile,
                             # FormatEntry, flock helpers (Slice A) +
                             # AppendBytes, RemoveBytes in-memory helpers
                             # (Slice B) + ParseBytes read-side helper
                             # (Slice C.2) for the broker's drift sweep.
  store/                     # versioned content-store primitives, hoisted from
                             # server/internal/store by §6.7.0 PR #120. Used by
                             # demarkus-server (the main consumer), and
                             # demarkus-publish (in tools/).

server/
  Dockerfile                 # multi-stage; bundles demarkus-server + CLI for
                             # exec probes. Build context = repo root.
                             # (§6.7.A PR #121)
  .goreleaser.yml            # binary archives only; docker blocks removed in
                             # §6.7.A — image build moves to per-component
                             # Dockerfiles + (forthcoming) release workflow.
  cmd/
    demarkus-server/         # the server binary (existing)

client/
  cmd/
    demarkus/                # primary CLI
    demarkus-tui/            # interactive TUI
    demarkus-mcp/            # MCP server for AI agents
    demarkus-agent/          # protocol client — federation crawler (existing)
      Dockerfile             # multi-stage; agent + CA bundle for outbound TLS.
                             # (§6.7.A PR #121)

tools/
  demarkus-broker/           # broker binary (Slice B PR #109, Slice C.1 PR #110,
                             # Slice C.2 PR #111, Slice C.3 PR #112, Slice C.4
                             # PR #114). main.go + internal/broker/
                             # { config, session, oidc, issuer, server,
                             #   labels, sweeper, ratelimit }. §6.2 broker
                             # binary complete; SCIM webhook stays in backlog.
    Dockerfile               # multi-stage; broker + CA bundle for OIDC
                             # discovery + token-endpoint HTTPS calls.
                             # (§6.7.A PR #121)
  demarkus-token/            # token mint admin CLI — uses protocol/token.
                             # Moved from server/cmd/ by §6.7.A PR #121.
  demarkus-publish/          # publish CLI — uses protocol/store. Moved from
                             # server/cmd/ by §6.7.A PR #121 (after §6.7.0
                             # hoisted store/ out of server/internal/).

Sub-Phases

6.0 — demarkus-agent verified + chart ✓ (merged PR #106)

Existing demarkus-agent binary verified end-to-end (2 team worlds + 1 hub smoke test) and chart at deploy/helm/demarkus-agent/ shipped. Stateless Deployment; ConfigMap holds TOML agent config; Secret holds per-host tokens; exec liveness probe; outbound-only (no Service). Three publish bugs fixed during verification (publishIndex status acceptance, expected_version=-1 for idempotent re-publish, Makefile make client build).

§6.7.A PR #121 additionally fixed a pre-existing structural bug in deploy/helm/demarkus-agent/tests/deployment_test.yaml — the suite listed only deployment.yaml in templates: but the template includes configmap.yaml + secret.yaml for checksum annotations, causing the 10 agent deployment tests to silently error since the file was written. Same shape as the §6.3.D.1 fix on the server's statefulset suite. Agent now ships per-image (ghcr.io/latebit-io/demarkus-agent) with CA bundle for outbound TLS.

6.1 — demarkus-server Helm chart ✓ (merged PR #107) + §6.3.D.2 ✓ + §6.7.A image fix ✓

deploy/helm/demarkus-server/ — StatefulSet, 1 replica, volumeClaimTemplates (each world owns its PVC, never shared), exec probes against /.well-known/agent-manifest.md, Service type LoadBalancer protocol: UDP. Bootstrap path seeds the initial admin token via protocol/token.Generate + AppendEntry (Slice A). Cert-manager Certificate resource behind a flag. helm-unittest test files at tests/ wired into CI by §6.3.D.1 PR #118 and validated end-to-end against a kind cluster by §6.3.D.2 PR #119.

§6.3.D.2 sidecar landed: the tokens Secret now carries helm.sh/resource-policy: keep via the three-branch hybrid template (first install / legacy upgrade / race-free no-op). PR #119 merged 2026-05-13 (commit 3d3b4bc). test-charts-kind CI job pins the upgrade-wipe property end-to-end.

§6.7.A image fix landed: the chart's image.repository (default ghcr.io/latebit-io/demarkus-server) is now built from server/Dockerfile as a multi-binary image bundling demarkus-server + demarkus (CLI used by exec probes). The §6.1 caveat carried since PR #107 ("probes will fail against the current single-binary image until 6.7 lands") is now empirically closed — kind smoke verified the server pod goes Ready in 19s with the exec probe satisfied by the bundled CLI.

6.2 — demarkus-broker binary (tools/demarkus-broker/) ✓ complete

Slice B (single-world OIDC mint flow) merged as PR #109 (commit 2a6aae6, 2026-05-11). Slice C.1 (groups-claim authorization + AllowEmails carve-out + email canonicalization) merged as PR #110 (commit 343f808, 2026-05-12). Slice C.2 (expiry + drift sweeper with Lease-based leader election, RBAC-denied mint guard) merged as PR #111 (commit cb5f800, 2026-05-12). Slice C.3 (POST /tokens/:label/rotate with re-login semantics + scope-frozen lifetime-reset asymmetry) merged as PR #112 (commit 6ee0d8b, 2026-05-12). Slice C.4 (per-subject + per-IP rate-limit middleware with shared subject buckets across the three /tokens routes, optional leftmost-XFF trust for the IP limiter, Reserve()+Cancel() for denial-doesn't-consume-budget) merged as PR #114 (commit e2933ff, 2026-05-12). §6.2 broker binary complete.

The §6.3.C OIDC_CLIENT_SECRET env-var override (applyEnvOverrides in config.go:262) is a one-line additive surface that does NOT reopen §6.2 — see §Decisions for the framing.

Role & topology

  • Issuance authority, not a request proxy. Broker sits on the demarkus login path; clients then talk to world servers directly carrying the raw token. World servers stay identity-blind; broker never sees mark:// requests. Capability model preserved.
  • One broker per universe (cluster) by default. Single OIDC client registration, single worlds: list in values, single issuance state Secret.
  • Broker SA holds a namespace-scoped Role + RoleBinding in each world's namespace, with get/update limited to that world's tokens Secret only. No ClusterRole. Slice C.2 sweeper adds coordination.k8s.io/leases get/create/update in the broker namespace for leader election; §6.3 chart bundles both into the broker SA.

State

Two distinct Kubernetes Secrets, never merged:

Secret Lives in Contents Reader
<world>-tokens each world's namespace TOML: [tokens.<label>] hash=, paths=, operations=, expires= demarkus-server
<broker>-issuances broker namespace JSON: label → {email, world, paths, operations, issued_at, expires} + secondary index email → [labels] broker only

World servers see only hashes. Email and identity live in broker state alone. Both Secrets carry helm.sh/resource-policy: keep end-to-end: the broker-side issuances Secret since §6.3.A, the world-side tokens Secret since §6.3.D.2.

Plus one cluster-scoped coordination object (Slice C.2):

Resource Lives in Contents Reader
demarkus-broker-sweeper Lease broker namespace holderIdentity, renewTime, leaseDurationSeconds leader-election library only

6.3 — demarkus-broker Helm chart ✓ complete (all five sub-slices merged)

Multi-replica HA at deploy/helm/demarkus-broker/, shipped across five sub-PRs, all merged 2026-05-13:

  • §6.3.A ✓ merged (PR #115, commit c71f499) — chart skeleton + security defaults.
  • §6.3.B ✓ merged (PR #116, commit 175e230) — RBAC + NetworkPolicy + PDB.
  • §6.3.C ✓ merged (PR #117, commit 599f63a) — Ingress + cert-manager Certificate + oidc.existingSecretRef deployment path.
  • §6.3.D.1 ✓ merged (PR #118) — helm-unittest test files + CI plumbing for both charts + folded-in test-broker Go CI job.
  • §6.3.D.2 ✓ merged (PR #119, commit 3d3b4bc) — three-branch hybrid template + kind upgrade-wipe regression.

The chart's image.repository default (ghcr.io/latebit-io/demarkus-broker) is now built from tools/demarkus-broker/Dockerfile (§6.7.A PR #121) — single-binary image, CA bundle for OIDC discovery + token-endpoint HTTPS validation, runAsUser 65532 in the image to match chart values defense-in-depth.

6.4 — Universe topology examples

ApplicationSet + Kustomize overlay at deploy/k8s/examples/. Pending §6.7.B (chart OCI publish).

6.5 — Observability recipes

Per-backend configs in deploy/observability/. Independent of §6.7.

6.6 — Documentation suite

docs/deployment/*.md (Fritz confirmed location 2026-05-13). Per-chart READMEs at deploy/helm/<chart>/README.md (broker README exists from §6.3.C; server + agent need similar treatment).

6.7 — Release pipeline

§6.7.0 ✓ merged (PR #120, commit b210015) — Precursor refactor. Hoisted server/internal/store/ → protocol/store/. Pure namespace move; four import-path rewrites; protocol layer now owns disk-store primitives alongside token/ and HashToken. Cleared the load-bearing dependency that had been deferring demarkus-publish's relocation to tools/.

§6.7.A ✓ merged (PR #121, commit aa9a3c6) — Per-service split container images + CLI relocation. Three images: ghcr.io/latebit-io/{demarkus-server,demarkus-broker,demarkus-agent} built from server/Dockerfile, tools/demarkus-broker/Dockerfile, client/cmd/demarkus-agent/Dockerfile respectively. Each multi-stage; build context = repo root for cross-module access. Server image bundles CLI for exec probes; broker + agent images include CA bundle for outbound TLS. All three runtime stages have USER 65532:65532 matching chart values. demarkus-token + demarkus-publish moved from server/cmd/ to tools/. server/.goreleaser.yml slimmed to binary-only. Makefile has image-server/image-broker/image-agent targets. Kind smoke verified the §6.1 probe caveat closed.

§6.7.B (next) — Release pipeline plumbing. Wire per-image docker build + docker push ghcr.io/latebit-io/<service>:<tag> into .github/workflows/release.yml for all three Dockerfiles. Add tools/.goreleaser.yml for demarkus-broker + demarkus-token + demarkus-publish archive distribution. Add OCI chart publish via helm push oci://ghcr.io/latebit-io/charts/<chart> for all three charts. Bump chart appVersions to align with first OCI release. Cosign signing stays backlogged.

Transition window note: until §6.7.B merges, charts' default image.repository references images that don't yet exist on GHCR. Operators building from main must make image locally or pass --set image.repository=... at install time.

Sequencing

  1. Slice A — token-mint library ✓ merged 2026-05-11 (PR #108).
  2. 6.0 chart ✓ merged 2026-05-11 (PR #106).
  3. 6.1 server chart ✓ merged 2026-05-11 (PR #107). §6.3.D.2 sidecar ✓ merged 2026-05-13. §6.7.A image fix ✓ merged 2026-05-13.
  4. 6.2 broker binary — Slice B ✓ merged 2026-05-11 (PR #109, commit 2a6aae6).
  5. 6.2 broker binary — Slice C ✓ four PRs merged 2026-05-12 (PRs #110, #111, #112, #114).
  6. 6.3 broker chart ✓ five-slice trajectory all merged 2026-05-13 (PRs #115, #116, #117, #118, #119).
  7. 6.7.0 store hoist ✓ merged 2026-05-13 (PR #120, commit b210015).
  8. 6.7.A image consolidation + CLI relocation ✓ merged 2026-05-13 (PR #121, commit aa9a3c6).
  9. 6.7.B release pipeline — NEXT. Per-image docker build/push + tools/.goreleaser.yml + OCI chart publish.
  10. 6.5 observability recipes — parallelizable with §6.7.B.
  11. 6.6 docs — incremental throughout.
  12. 6.4 topology examples — last; depends on §6.7.B release artifacts.

Rough effort: ~2 weeks of focused work remaining for §6.7.B + §6.4 + §6.5 + §6.6 against the 2026-05-27 deadline.

Backlog (deferred, easy to add later)

  • Cosign signing of images + chart releases. Half-day CI add.
  • Latency log-enrichment (duration_ms field on request slog lines). Tiny additive change.
  • SCIM lifecycle webhook on the broker. Optional add when a customer asks.
  • Userinfo-based groups in the broker Verifier. Slice C.1 ships ID-token-only; AllowEmails is the documented workaround.
  • Case-sensitive group matching for Keycloak. Slice C.1 chose case-insensitive for the validated IdP set.
  • Configurable lease timings (sweeper.leaseDuration, renewDeadline, retryPeriod). C.2 hardcodes the client-go defaults.
  • SIGHUP on revoke + sweep + rotate to shrink the revoke-to-effect window. Needs pods/exec RBAC across world namespaces.
  • Sweeper backing-store migration when the issuances Secret hits the ~1MB / ~5000-issuance k8s storage ceiling.
  • Cluster-shared rate limiter (Slice C.4 follow-up).
  • Idle-key GC for the rate-limit registry (Slice C.4 follow-up).
  • Timeouts on the existing test-protocol/test-server/test-client CI jobs. §6.3.D.1 added timeout-minutes: 20 to the new jobs; the older jobs still rely on the GitHub Actions default (360 min). One-line cleanup.
  • Hardened broker chart Secret symmetry. Broker's secret-issuances.yaml uses the simpler lookup-skip pattern (vs. the §6.3.D.2 three-branch hybrid on the server chart). Defense-in-depth follow-up; not load-bearing.
  • Base-image digest pinning (§6.7.A CodeRabbit nit skipped). When renovate/dependabot is wired to bump base-image digests automatically, pin all three Dockerfiles to specific SHAs for reproducibility. Until then, floating golang:1.26-alpine picks up Go security patches automatically.
  • Single-image distribution mode (§6.7.A architectural alternative). If a customer asks for one-image-with-all-binaries (single CVE scan target, single tag to bump across charts), the unified-Dockerfile shape from the §6.7.A draft can be added as an opt-in. The split-images shape stays the default.
  • Server image CA bundle (§6.7.A deferred). If a need arises for kubectl exec debugging or future outbound-TLS server features, copy /etc/ssl/certs/ca-certificates.crt from build stage. One-line add.

Risks

  • Observability via logs ceiling. If customers want signals not derivable from current slog, we hit a wall. Mitigation: log enrichment is a small additive change.
  • Broker secret-write blast radius. Mitigated by namespace-scoped Roles, audit log, optional NetworkPolicy. Per-world Role is non-negotiable. TestMintRBACDeniedNoPartialState validates clean failure.
  • OIDC provider coupling. First impl is Google; structure so second provider is a one-day add.
  • Token revocation in-flight latency. Revocation latency = kubelet Secret propagation + whatever the server does on re-mount. Acceptable for the deployment posture (~60s default kubelet propagation).
  • Rotate transient two-tokens window. Documented; verified by TestRotateTokenSoftPartial.
  • §6.1 chart upgrade-wipe gap. ✓ Resolved by §6.3.D.2 PR #119.
  • §6.1 chart-test plumbing latent failure. ✓ Resolved by §6.3.D.1 PR #118; agent chart equivalent resolved by §6.7.A PR #121.
  • §6.7 multi-binary image hand-off. ✓ Resolved by §6.7.A PR #121. Server image now bundles CLI for exec probes; per-image release plumbing follows in §6.7.B.
  • Release-pipeline image gap (§6.7.A → §6.7.B transition window). Until §6.7.B merges, no release workflow pushes images to GHCR — server/.goreleaser.yml no longer has dockers: blocks. Charts' default image.repository references images not yet on GHCR. Mitigation: operators building from main must make image locally OR pass --set image.repository=.... Window closes when §6.7.B lands. Two-week deadline keeps this short.
  • Issuances Secret scaling wall at the ~1MB / ~5000-issuance k8s storage ceiling. Phase-7+ fix is a different backing store.
  • Chart proliferation. Three charts + examples + dashboards. Mitigate with shared common-labels templates.
  • Trial scope creep. First customer is path-B.
  • Per-replica rate limit ⇒ N× effective rate. Slice C.4 documented; deferred fix in §Backlog.
  • Rate-limit registry growth under misconfigured XFF trust. Documented in C.4 §Decisions and chart README.

Status

Plan v16, 2026-05-13. Slice A, 6.0 chart, 6.1 chart (incl. §6.3.D.2 sidecar + §6.7.A image fix), 6.2 broker binary (Slices B + C.1–C.4), 6.3 broker chart (all five sub-slices), 6.7.0 store hoist, 6.7.A image consolidation + CLI relocation all merged (PRs #106, #107, #108, #109, #110, #111, #112, #114, #115, #116, #117, #118, #119, #120, #121).

Eleven PRs merged on 2026-05-13: #115 (6.3.A) → #116 (6.3.B) → #117 (6.3.C) → #118 (6.3.D.1) → #119 (6.3.D.2) → #120 (6.7.0 store hoist) → #121 (6.7.A image consolidation). Phase 6 is now ~75% complete by sub-phase count (6.0/6.1/6.2/6.3/6.7.0/6.7.A done; 6.4/6.5/6.6/6.7.B remaining).

§6.7.0 + §6.7.A trajectory recap:

  • 6.7.0 (PR #120, b210015): pure refactor. server/internal/store/ → protocol/store/. 5-min mechanical move + 4 import-path rewrites. Earlier handoff prose calling this "non-trivial" was an overstatement; useful calibration for "is this load-bearing" judgments.
  • 6.7.A (PR #121, aa9a3c6): per-service split container images + CLI relocation. Three Dockerfiles co-located with their primary binaries; build context = repo root for cross-module access. Server image bundles CLI for probes; broker + agent get CA bundle for outbound TLS; all three runtime stages USER 65532. Admin CLIs (token, publish) ship as standalone binaries — not in any runtime image. Initial draft was unified single-image; pivoted to split after architectural-fit discussion (demarkus is multi-component, not single-purpose). Three CodeRabbit rounds: USER + CA bundle (valid, fixed); mkdir-for-/out (false alarm, skipped); digest pinning (skipped, replied with rationale). Helm unittest 107/107 + kind smoke verified end-to-end.

Next: §6.7.B release pipeline. Wire per-image docker build + docker push into .github/workflows/release.yml; add tools/.goreleaser.yml for admin CLI + broker binary archives; add OCI chart publish for all three charts. Closes the transition-window risk where charts reference not-yet-published images. Then §6.5 observability recipes (parallelizable), §6.6 docs (incremental), §6.4 topology examples (last). Two-week deadline at 2026-05-27.

trail
  1. soul.demarkus.io v16