soul.demarkus.io/plans/universe-deployment.md/v17 draft reader meta

Plan: Universe Deployment

Ship the enterprise-grade deployment for demarkus. The deliverable is the deployment: a customer's ops team installs a Helm chart, runs a broker, runs an agent, and has a federated demarkus universe their org can use. Any company evaluating demarkus (their internal "POC") installs the same product an established customer runs in production. There is no separate "POC slice" — the slice mentality is rejected. We build it once, right, and customers trial the real thing.

Goal

Deliver a complete, supportable, production-grade Kubernetes deployment package for demarkus, including:

  1. A Helm chart for demarkus-server (one world). ✓
  2. A Helm chart for demarkus-broker (OIDC token issuance + revocation). ✓
  3. A Helm chart for demarkus-agent (hub aggregator, crawl-and-index). ✓
  4. Reference topology examples (Argo CD ApplicationSet, Kustomize overlay).
  5. Backend-agnostic observability — structured slog emission + reference configs for Datadog, OTel Collector, Vector, Fluent Bit, Grafana Alloy.
  6. Customer-facing documentation — installation, security/threat model, operations, upgrade path, per-provider OIDC setup, observability recipes.
  7. A release pipeline producing images and chart releases consumable from GHCR. ✓

The same artifacts power first-customer trial and steady-state operations.

Non-Goals (Phase 7+ territory)

  • Multi-replica worlds with shared storage (RWX / object backend).
  • Cross-cluster universe federation.
  • Operator with a World CRD.
  • Hosted / managed SaaS.
  • Non-markdown content.

Constraints

  • No core protocol/server changes. Period. Per prior precedent (Claude Code plugin, Obsidian plugin, feedback_plugin_scope.md). Observability is achieved by log derivation in a collector, not by adding /metrics or OTel SDK calls to the server. Note: in Slice A we promoted HashToken and the token-mint library to protocol/ (under protocol/auth.go and protocol/token/). These are additive helper relocations consumed by server + CLI + future broker; no wire-protocol or server behavior changed. Slice C.2 added protocol/token.ParseBytes in the same additive spirit — a read-side helper so the broker's drift sweeper can inspect a world's tokens.toml payload via the map shape rather than substring matching on serialized TOML. §6.7.0 hoisted server/internal/store/ → protocol/store/ in the same additive spirit — the disk-shape of versioned content lives at the protocol layer alongside token/ and the HashToken contract; no API change, four import-path rewrites.
  • Capability-based auth model is non-negotiable. The server never learns identity — only labels.
  • Markdown-only scope is non-negotiable.

Decisions (resolved during planning)

  • Health probes use exec, not HTTP. Both charts ship liveness/readiness exec probes that invoke the demarkus CLI fetching /.well-known/agent-manifest.md (always public per /architecture.md). No core change. Same pattern as redis-cli ping / pg_isready.
  • UDP port is a values knob. Default 6309, override via server.udpPort. Documented option: switch to 443 for VPN/middlebox-hostile networks (Cloudflare Warp Zero Trust, corp firewalls that filter non-standard UDP). Protocol default stays 6309.
  • Wildcard TLS via cert-manager DNS-01. Single *.<root> cert covers all worlds + broker hostnames. HTTP-01 cannot work for worlds (no HTTP); DNS-01 is the standard path for QUIC services.
  • DNS topology. One A record per world + one for the broker, all under the same wildcard zone. Each world exposed by Service: LoadBalancer protocol: UDP; broker by standard Ingress (HTTPS).
  • Hub aggregator is the existing demarkus-agent at client/cmd/demarkus-agent/. Already implements crawl + daemon subcommands with TOML config, multi-worker crawl, per-host token auth, per-server + aggregated index publishing. Verified end-to-end on 2026-05-11. Phase 5 is no longer a blocking prerequisite.
  • demarkus-agent lives in client/cmd/, not tools/. It's a protocol client (uses fedcrawl, fetch, tokens, links), not a utility. Memory rule sharpened: client/cmd/ = protocol clients (CLI, TUI, MCP, agent); tools/ = utilities (broker, future sync/ops binaries).
  • Token-mint library lives at protocol/token/, not tools/internal/token/ (revised mid-Slice-A after CodeRabbit review). protocol/ is reachable by both server/ (where demarkus-token CLI lives) and tools/ (where the broker will live), which tools/internal/ was not. protocol/ already owns the HashToken byte-shape contract, so it's the natural home for the on-disk token primitives that must round-trip identically across CLI mints and broker mints.
  • demarkus-token and demarkus-publish CLIs live at tools/ (§6.7.A merged 2026-05-13 PR #121). demarkus-token had been a clean candidate since Slice A (only imports protocol/token). demarkus-publish followed once §6.7.0 (PR #120) hoisted server/internal/store/ → protocol/store/, removing the server-internal dependency. Both ship as standalone binaries via tools/.goreleaser.yml archives (§6.7.B PR #123); they are NOT bundled into any runtime image.
  • Multi-OIDC. Broker speaks generic OIDC, not Google-specific. Provider behind a Verifier interface. Google validated first; Okta, Entra ID, Auth0 follow with config only.
  • Workload Identity (GKE) as a broker option. Values flag annotates the broker ServiceAccount with the GSA mapping. Off by default; on for GKE customers wanting no static SA keys. Not applicable to world servers.
  • Broker is HA. Multi-replica with resourceVersion optimistic concurrency on Secret writes, retry on conflict.
  • Observability is log-derived, backend-agnostic. demarkus already emits structured slog. Charts ship Datadog autodiscovery annotations + reference configs for OTel Collector, Vector, Fluent Bit, Grafana Alloy. Customer's SRE picks the agent/backend. Latency histograms deferred — depends on what current slog includes per request; revisit during trials if needed.
  • Backup/DR is documented, not built. The chart deliberately does not run backup CronJobs. Operations doc covers Velero, VolumeSnapshot, and demarkus-agent sync as DR options.
  • Image hosting: ghcr.io/latebit-io/{demarkus-server,demarkus-broker,demarkus-agent}. Three separate images, one per service. Released by §6.7.B per-module tag flow.
  • Chart registry: oci://ghcr.io/latebit-io/charts/{demarkus-server,demarkus-broker,demarkus-agent}. Three OCI helm charts released by §6.7.B per-module tag flow alongside their images.
  • Cosign signing: deferred to backlog. Half-day CI add when wanted.
  • Groups-claim case sensitivity (revised mid-Slice-C.1 after CodeRabbit back-and-forth). Group names match case-insensitively: AllowConfig.Groups is lowercased+trimmed at config load (same as Domains and Emails) and groupsMatch lowercases the claim's groups before compare. The validated IdP set (Google, Okta, Entra ID, Auth0) enforces case-insensitive group-name uniqueness, so distinct case-variant groups can't exist; a case-sensitive compare would silently fail when an operator writes "Engineering" but the IdP emits "engineering" after upstream normalization. Keycloak with deliberately distinct case-variant groups is out of scope; revisit if a customer asks. Domains and emails get the same lowercase-at-load + plain == treatment for symmetry.
  • Sweeper interval capped at 24h (Slice C.2). The defaultToken.expiresAfter typical is 24h; a sweep interval longer than that would let expired tokens linger past their ExpiresAt for up to a whole sweep cycle, defeating the short-lived-tokens identity-lifecycle model (§Revocation §3). validate() rejects with sweeper.interval must be <= 24h. Lease-election timings (15s lease / 10s renew / 2s retry) intentionally not exposed in YAML — the client-go defaults match our failover budget and adding knobs invites misconfiguration. Revisit if a customer needs sub-15s failover.
  • sweeper.disabled rather than sweeper.enabled (Slice C.2). YAML zero-value (false) is the production-safe default so an operator who omits the block gets the sweeper running. Naming the field positively (Enabled) would have made omission silently disable the janitor — exactly the wrong default for multi-replica deployments where expired tokens accumulate forever without it.
  • Rotate is re-login semantics, not refresh (Slice C.3). POST /tokens/:label/rotate re-runs worldAllows(world.Allow, claims) on every call, so a user removed from the world's allowlist between mint and rotate can NOT extend access via rotation. The bearer ID-token verification at the HTTP layer already proves the IdP identity is still active; the per-world predicate re-check catches in-org permission changes (group removed, domain renamed, email taken off carve-out). Picked over rotation-as-refresh because rotation extends the lifetime — without re-auth, a user whose IdP account stayed live could rotate forever even after losing world-scope authorization.
  • Rotate scope vs. lifetime asymmetry (Slice C.3). Scope (paths + operations) stays frozen to the issuance record on rotate: operator narrowing of DefaultToken.Paths between mint and rotate does NOT shrink the rotated token's reach, because the user shouldn't be surprised by reduced access mid-session — they re-login to pick up new scope. Lifetime resets to now + DefaultToken.ExpiresAfter from the operator's CURRENT config: operator tightening of expiry DOES apply on rotate, because shorter lifetimes are an explicit security-tightening lever and rotation must not bypass it. The asymmetry is intentional and pinned by TestRotateLabelPreservesIssuanceScope + TestRotateLabelLifetimeUpdatesToCurrentConfig.
  • Rotate sequence: mint new, then revoke old (Slice C.3). On revoke failure the new token is returned anyway with a wrapped soft error; the sweeper retires the orphan old label on expiry (or via drift if an operator hand-cleans). Picked over revoke-then-mint because that alternative leaves the user with no token if mint then fails, forcing a full re-login — worse UX than a brief two-valid-tokens window for the same user. Mirrors the Slice B partial-mint convention: (MintResult{Label:""}, err) for hard failures, (MintResult{non-empty}, err) for soft partial success.
  • Middleware-not-handler auth + rate limit (Slice C.4). The three authed /tokens routes go through requireAuth → subjectRateLimit → handler; /auth/login goes through ipRateLimit → authLogin; /auth/callback, /healthz, /readyz stay middleware-free. The old per-handler s.authenticate(w, r) was extracted into the requireAuth middleware that stashes verified Claims on r.Context() via a typed key; handlers read claims via claimsFromCtx. The alternative — keeping auth in handlers and adding rate-limit inside each one — would either double-verify the bearer (one verify for the limit key, one in the handler) or hide the limiter inside each handler, opaque to the route-registration block. Middleware composition keeps the per-route policy visible in Routes().
  • One shared subject bucket across the three /tokens routes (Slice C.4). subjectRateLimit keys on hashSubject(claims.Subject) and the same registry covers GET /tokens, DELETE /tokens/:label, and POST /tokens/:label/rotate. Picked over three per-route buckets so a misbehaving client cannot multiply effective throughput by fanning out (3 routes × 10/min would give 30/min effective rather than the operator-intended 10/min). Pinned by TestRateLimitTokensSharedBucketAcrossRoutes.
  • Reserve() + Cancel() on denial, not Allow() (Slice C.4). Functional equivalence — neither pattern consumes budget on denial — but Reserve().Delay() gives the precise wait time, which we surface as Retry-After with a 1s minimum floor. (Retry-After: 0 reads as "retry immediately" to aggressive clients and would defeat the limiter.) Pinned by TestRateLimitRegistryDenialDoesNotConsumeBudget (50 denials in a tight loop, then 150ms regen window allows the 51st request) and the Retry-After assertions on the integration 429 tests.
  • trustForwardedFor: false default in the binary (Slice C.4). The broker behind an Ingress sees the controller's IP in r.RemoteAddr, so the per-IP limiter on /auth/login collapses into one bucket for every client unless we honor XFF. But trusting XFF when NOT behind a proxy lets an attacker rotate the header to bypass per-IP limits. The binary ships safe; chart-side §6.3.A inverts the default to true (the chart's "behind an Ingress" assumption holds in deployment). TestRateLimitLoginIPIgnoresForwardedForByDefault pins the default-untrusted behavior at the binary level; TestRateLimitLoginIPCrossIPIsolation pins the trust-enabled behavior.
  • rateLimit.disabled not enabled (Slice C.4). Same shape as sweeper.disabled. Zero-value (false) gives the production-safe behavior so an operator who omits the rateLimit: block in their values file still gets the protection. Defaults applied at validate-time: 10/min subject burst 5, 20/min IP burst 5. Field validation skipped entirely when Disabled: true so an operator opting out can leave the per-route knobs empty.
  • Per-replica unbounded registry, by design (Slice C.4). The rateLimitRegistry keeps one *rate.Limiter per key with no TTL eviction and no max-entry cap. CodeRabbit flagged this as a memory-DoS vector; rejected after re-examining the threat model. Realistic worst-case sizing is ~500KB at ~10k authenticated subjects — not memory-DoS-shaped. Cluster-shared limiter is the Phase-7+ fix; documented in §Backlog.
  • OIDC_CLIENT_SECRET env-var override (§6.3.C, additive to §6.2). broker.LoadConfig.applyEnvOverrides() (tools/demarkus-broker/internal/broker/config.go:262) lets the OAuth client secret come from the environment instead of the on-disk config file. Env wins over file when both set; empty env is treated as unset so an accidentally-cleared variable can't blank out a file-supplied value. Production deployments keep the OAuth secret in an externally-managed Kubernetes Secret (External Secrets Operator, Sealed Secrets, Vault) mounted via secretKeyRef, instead of baking it into the chart-rendered config Secret where it would leak into helm release history. This is a one-line escape hatch for the production-secret-ref deployment shape, NOT a §6.2 reopening — §6.2 broker binary complete framing stands. Pinned by TestLoadConfigOIDCClientSecretEnvOverride (4-row table).
  • helm-unittest pinned to v0.6.2 in CI (§6.3.D.1). v1.0+ uses platformHooks in plugin.yaml which needs helm v3.16+; Fritz's local helm is v3.13.2. Pinning v0.6.2 in CI matches the local-dev validation surface so test behavior is identical between developer machines and CI. Bump together when helm itself bumps. Helm pinned to v3.13.2 alongside.
  • test-broker Go CI job folded into §6.3.D.1. Documented above.
  • Three-branch hybrid template for upgrade-safe Secrets (§6.3.D.2). deploy/helm/demarkus-server/templates/tokens.yaml keys on the live Secret's state: (1) first install → render fresh + keep annotation; (2) legacy upgrade → re-render existing data verbatim + add the annotation; (3) race-free no-op → emit nothing, helm GC blocked by live annotation, broker stays sole writer. Picked over Option A (always render) which keeps a race window between helm template's lookup snapshot and helm apply while broker writes via mutateSecret.
  • SIMULATE_LEGACY_NO_KEEP test-mode flag over a separate script (§6.3.D.2). One script with env-flag exercises both the normal three-branch path and the migration path.
  • test-charts-kind CI job (§6.3.D.2). 47-line job in .github/workflows/ci.yml runs test-upgrade-wipe.sh three times: server normal + server SIMULATE_LEGACY_NO_KEEP + broker normal.
  • kubectl jsonpath bracket notation for hyphenated keys (§6.3.D.2 CodeRabbit). {.data['key-with-hyphens']}, NOT {.data.key-with-hyphens}. Spec-conformant bracket form. Pin this convention for any future jsonpath against k8s data fields.
  • Per-service split container images over a unified single-image bundle (§6.7.A). Three images: ghcr.io/latebit-io/demarkus-server (server + CLI for probes), demarkus-broker (single binary + CA bundle for OIDC), demarkus-agent (single binary + CA bundle for remote mark:// crawl TLS). Initial §6.7.A draft packaged all six binaries into one ghcr.io/latebit-io/demarkus image; reverted to split because demarkus's shape (multi-component product) calls for per-service blast-radius isolation. A CVE in broker's OIDC dependency shouldn't fail the server pod's image scan.
  • Admin CLIs not bundled in runtime images (§6.7.A). demarkus-token and demarkus-publish ship as standalone binaries via tools/.goreleaser.yml archives. Operators run them locally with port-forward or in dedicated Job pods, NOT via kubectl exec into long-running pods. Bundling admin tools into runtime images would leak "admin tooling" into the pod's blast radius.
  • CA bundle selectively included (§6.7.A). Broker + agent images copy /etc/ssl/certs/ca-certificates.crt from the build stage. Server image skips it — no outbound TLS today; bundled CLI's exec probes use -insecure for localhost.
  • USER 65532:65532 in all three runtime stages (§6.7.A). Matches each chart's podSecurityContext.runAsUser for defense-in-depth.
  • go build -C <module> over RUN cd <module> && go build (§6.7.A). Go 1.20+ idiom; auto-creates -o parent directory.
  • No SHA256 digest pinning on golang:1.26-alpine (§6.7.A CodeRabbit nit skipped). Floating the minor tag picks up Go stdlib security patches automatically; pinning trades reproducibility for a maintenance hazard without renovate automation. Revisit when base-image automation lands.
  • docker/build-push-action@v6 over re-introducing goreleaser's dockers: block (§6.7.B.1 PR #122). The new multi-stage Dockerfiles compile from source via Go's cross-compilation under buildx, not by COPY-ing goreleaser-built binaries. Wiring image release through build-push-action keeps the Dockerfiles unchanged and portable (anyone can docker build locally without goreleaser-pro). Cost: binaries compile twice on release (once for archives, once inside image build) — Go is fast, ~30s extra per platform, negligible.
  • Chart-version + appVersion pinned 1:1 to module version on release (§6.7.B.3 PR #124). Coupled lifecycle: chart-of-version-X always references image-of-version-X. Independent chart versioning would be useful for chart-only fixes (broken template, NOTES.txt edit), but adds tag-scheme complexity for a use case without evidence yet. Revisit if it materializes.
  • OCI helm auth via shared ~/.docker/config.json (§6.7.B.3). Helm 3.13+ reads the docker config natively for OCI registry auth; the existing docker/login-action step satisfies both image push and chart push. No separate helm registry login step needed.
  • tag_prefix: tools/v + change_path: tools/ (§6.7.B.2 semver-tools job). PaulHatch/semantic-version mirror of server/client semver jobs; tools/ is now a peer module in the release-pipeline taxonomy with independent version bumps.

Open Questions

  1. First customer trial. Nesto (*.library.nesto.ca) is path-B — trial waits for product. Trial runbook lands at /trials/nesto.md when scoping starts.
  2. Two-week deadline (2026-05-27): §6.7 fully landed; §6.5 / §6.6 / §6.4 remain. ~11 days runway. Suggested order: §6.5 → §6.6 → §6.4 (per Fritz's earlier confirmation). §6.5 is independent and parallelizable; §6.6 grows incrementally; §6.4 depends on §6.7.B published artifacts (now available).
  3. First module-release tag firing of the §6.7.B pipeline. Triggered automatically by the next push to main that touches server/, client/, or tools/. Until a first push fires, no image or chart is yet on GHCR despite the pipeline being live. Operators building from main still need make image locally or --set image.repository=... overrides. Resolves on the first feat: or fix: commit to a module path.
  4. Hardened broker chart symmetry (deferred from §6.3.D.2). The broker chart's secret-issuances.yaml uses the simpler lookup-skip pattern. Defense-in-depth follow-up; not load-bearing. Only act on if Fritz brings it up.

Repository Layout

Reflects state as of §6.7.B.3 merge (2026-05-14):

deploy/
  helm/
    demarkus-server/         # one-world chart (Phase 6.1) — shipped PR #107
    demarkus-broker/         # OIDC token broker chart (Phase 6.3)
    demarkus-agent/          # crawl/index agent chart (Phase 6.0) — shipped PR #106
  test-upgrade-wipe.sh       # kind regression: §6.3.D.2 PR #119
  k8s/
    examples/
      applicationset.yaml    # Argo CD ApplicationSet (§6.4 — pending)
      kustomize-overlay/     # Kustomize alternative (§6.4 — pending)
  observability/             # per-backend log-shipper configs (§6.5 — pending)
docs/
  deployment/                # install / security / operations / observability
                             # (§6.6 — pending; location confirmed 2026-05-13)

.dockerignore                # repo-root, applies to all three image builds
                             # (§6.7.A PR #121)

.github/
  workflows/
    release.yml              # full per-module release pipeline:
                             # - server/v*  → goreleaser archives
                             #               + docker buildx push demarkus-server image
                             #               + helm package + push OCI chart
                             # - client/v*  → goreleaser archives
                             #               + docker buildx push demarkus-agent image
                             #               + helm package + push OCI chart
                             # - tools/v*   → goreleaser archives (broker+token+publish)
                             #               + docker buildx push demarkus-broker image
                             #               + helm package + push OCI chart
                             # (§6.7.B PRs #122 + #123 + #124)
    ci.yml                   # PR + push tests; test-broker + test-charts +
                             # test-charts-kind (§6.3.D.1, §6.3.D.2)

protocol/
  auth.go                    # HashToken — sha256-<hex> contract (Slice A)
  token/                     # Generate, ReadFile, AppendEntry, WriteFile,
                             # FormatEntry, flock helpers (Slice A) +
                             # AppendBytes, RemoveBytes in-memory (Slice B) +
                             # ParseBytes (Slice C.2).
  store/                     # versioned content-store primitives, hoisted
                             # from server/internal/store (§6.7.0 PR #120).

server/
  Dockerfile                 # multi-stage; bundles demarkus-server + CLI
                             # (§6.7.A PR #121)
  .goreleaser.yml            # binary archives only
  cmd/
    demarkus-server/

client/
  .goreleaser.yml            # binary archives for demarkus, demarkus-tui,
                             # demarkus-mcp (demarkus-agent ships via Dockerfile
                             # build in release-client)
  cmd/
    demarkus/
    demarkus-tui/
    demarkus-mcp/
    demarkus-agent/
      Dockerfile             # (§6.7.A PR #121)

tools/
  .goreleaser.yml            # binary archives for demarkus-broker,
                             # demarkus-token, demarkus-publish (§6.7.B PR #123)
  demarkus-broker/
    Dockerfile               # (§6.7.A PR #121)
  demarkus-token/            # admin CLI (no Dockerfile — archive-only)
  demarkus-publish/          # admin CLI (no Dockerfile — archive-only)

Sub-Phases

6.0 — demarkus-agent verified + chart ✓ (merged PR #106)

Existing demarkus-agent binary verified end-to-end (2 team worlds + 1 hub smoke test) and chart at deploy/helm/demarkus-agent/ shipped. Stateless Deployment; ConfigMap holds TOML agent config; Secret holds per-host tokens; exec liveness probe; outbound-only (no Service). §6.7.A PR #121 fixed a pre-existing structural bug in the agent deployment test suite (10 silently-erroring tests, same shape as the server-statefulset suite). Agent ships per-image (ghcr.io/latebit-io/demarkus-agent) with CA bundle for outbound TLS, released via client/v* tag flow (§6.7.B PR #122 + #124).

6.1 — demarkus-server Helm chart ✓ (merged PR #107) + §6.3.D.2 ✓ + §6.7.A image fix ✓ + §6.7.B release ✓

deploy/helm/demarkus-server/ — StatefulSet, 1 replica, volumeClaimTemplates, exec probes against /.well-known/agent-manifest.md, Service type LoadBalancer protocol: UDP. Bootstrap path seeds the initial admin token via protocol/token.Generate + AppendEntry. Cert-manager Certificate resource behind a flag. helm-unittest test files at tests/ wired into CI by §6.3.D.1 PR #118 and validated end-to-end against a kind cluster by §6.3.D.2 PR #119.

§6.3.D.2 sidecar: tokens Secret now carries helm.sh/resource-policy: keep via three-branch hybrid template. PR #119 merged 2026-05-13. test-charts-kind CI job pins the upgrade-wipe property end-to-end.

§6.7.A image fix: chart's image.repository (default ghcr.io/latebit-io/demarkus-server) is built from server/Dockerfile as multi-binary image (demarkus-server + demarkus CLI). PR #121. The §6.1 probe caveat is empirically closed (kind smoke verified Ready in 19s).

§6.7.B release plumbing: next push to main touching server/ triggers server/v* tag → goreleaser archives + multi-arch image push to ghcr.io/latebit-io/demarkus-server:<version> + OCI chart push to oci://ghcr.io/latebit-io/charts/demarkus-server:<version>. PRs #122 + #124.

6.2 — demarkus-broker binary (tools/demarkus-broker/) ✓ complete

Slice B (single-world OIDC mint flow) merged as PR #109 (commit 2a6aae6, 2026-05-11). Slice C.1 PR #110. Slice C.2 PR #111. Slice C.3 PR #112. Slice C.4 PR #114. §6.2 broker binary complete.

The §6.3.C OIDC_CLIENT_SECRET env-var override is a one-line additive surface that does NOT reopen §6.2.

6.3 — demarkus-broker Helm chart ✓ complete (all five sub-slices merged) + §6.7.B release ✓

Multi-replica HA at deploy/helm/demarkus-broker/, shipped across five sub-PRs (#115, #116, #117, #118, #119), all merged 2026-05-13. Chart now released via tools/v* tag flow (§6.7.B PR #123 + #124): image push to ghcr.io/latebit-io/demarkus-broker:<version> + OCI chart push to oci://ghcr.io/latebit-io/charts/demarkus-broker:<version>.

6.4 — Universe topology examples (pending)

ApplicationSet + Kustomize overlay at deploy/k8s/examples/. Both reference the chart pull URLs from §6.7.B (now live). Argo CD ApplicationSet template generator iterates a worlds: list and templates one Application per world. Kustomize overlay is the GitOps-without-Argo alternative. Customer's ops team picks one. Verification surface: argocd app create --dry-run or kustomize build against a kind cluster.

6.5 — Observability recipes (pending — independent of any prior phase, can start immediately)

Per-backend configs in deploy/observability/. Slice C.2 added sweeper-side log lines (broker: swept, broker: sweep failed, broker: sweeper observing new leader, broker: sweeper lost leadership); Slice C.3 added rotate-side log lines; Slice C.4 added rate-limit log lines (broker: rate limit exceeded with route, subject or ip, retryAfter fields). Recipes are pure config — no core code changes. Datadog autodiscovery via pod annotations + a dashboard JSON; OTel Collector via a config.yaml; Vector via a [transforms.parse] block; Fluent Bit via parser + filter; Grafana Alloy via discovery.kubernetes + loki.process.

6.6 — Documentation suite (pending — incremental)

docs/deployment/*.md (location confirmed 2026-05-13). Per-chart READMEs at deploy/helm/<chart>/README.md (broker README exists from §6.3.C; server + agent need similar treatment). Install guide (per-provider OIDC setup for Google / Okta / Entra ID / Auth0; chart install order: server → broker → agent; cert-manager + DNS topology). Security/threat model. Operations (upgrade path including the now-tested resource-policy:keep contract, backup/DR via Velero or demarkus-agent sync). Observability recipes (links to §6.5).

6.7 — Release pipeline ✓ complete

§6.7.0 ✓ merged (PR #120, commit b210015) — Precursor refactor. Hoisted server/internal/store/ → protocol/store/. Pure namespace move; four import-path rewrites.

§6.7.A ✓ merged (PR #121, commit aa9a3c6) — Per-service split container images + CLI relocation. Three Dockerfiles co-located with their primary binaries. Server image bundles CLI for probes; broker + agent images include CA bundle for outbound TLS. All three runtime stages USER 65532. Admin CLIs (token, publish) ship as standalone binaries — not bundled in any runtime image.

§6.7.B ✓ complete via three sub-PRs:

  • §6.7.B.1 ✓ merged (PR #122, commit 0e1a4ae, 2026-05-13) — added docker/build-push-action@v6 to existing release-server + release-client jobs. Multi-arch (amd64/arm64/armv7) push to ghcr.io/latebit-io/{demarkus-server,demarkus-agent}:<version> on server/v* / client/v* tag fire.
  • §6.7.B.2 ✓ merged (PR #123, commit afb6155, 2026-05-13) — new tools/.goreleaser.yml with three archive builds (broker + token + publish). New workflow plumbing: tools filter, test-tools, semver-tools, release-tools. release-tools creates tools/v* tag, runs goreleaser, builds + pushes broker image to ghcr.io/latebit-io/demarkus-broker:<version>.
  • §6.7.B.3 ✓ merged (PR #124, commit 9cf6303, 2026-05-14) — azure/setup-helm@v4 + helm package + helm push at the end of each of the three release jobs. Chart version + appVersion pinned 1:1 to module version. Charts push to oci://ghcr.io/latebit-io/charts/{demarkus-server,demarkus-broker,demarkus-agent}. Helm 3.13+ uses the docker login config natively for OCI auth.

§6.7 fully complete. All three runtime images + all three OCI helm charts have a release path tied to their corresponding module-version tag.

Sequencing

  1. Slice A — token-mint library ✓ merged 2026-05-11 (PR #108).
  2. 6.0 chart ✓ merged 2026-05-11 (PR #106).
  3. 6.1 server chart ✓ merged 2026-05-11 (PR #107). §6.3.D.2 ✓. §6.7.A image fix ✓. §6.7.B release ✓.
  4. 6.2 broker binary — Slices B + C.1–C.4 ✓ all merged 2026-05-11–12 (PRs #109, #110, #111, #112, #114).
  5. 6.3 broker chart ✓ all five sub-slices merged 2026-05-13 (PRs #115, #116, #117, #118, #119).
  6. 6.7.0 store hoist ✓ merged 2026-05-13 (PR #120, b210015).
  7. 6.7.A image consolidation + CLI relocation ✓ merged 2026-05-13 (PR #121, aa9a3c6).
  8. 6.7.B release pipeline ✓ three sub-PRs merged 2026-05-13–14:
    • 6.7.B.1 (PR #122, 0e1a4ae) — image release on existing server/v* + client/v* tags.
    • 6.7.B.2 (PR #123, afb6155) — tools/v* tag flow + broker image release.
    • 6.7.B.3 (PR #124, 9cf6303) — OCI chart publish for all three charts.
  9. 6.5 observability recipes — NEXT. Independent of any prior phase. Pure config files.
  10. 6.6 docs — incremental throughout 6.5/6.4 plus dedicated polish pass.
  11. 6.4 topology examples — last; references chart pull URLs from §6.7.B (now live).

Rough effort: ~10 days of focused work remaining for §6.5 + §6.6 + §6.4 against the 2026-05-27 deadline. Phase 6 ~85% complete by sub-phase count (6.0/6.1/6.2/6.3/6.7 done; 6.4/6.5/6.6 remaining).

Backlog (deferred, easy to add later)

  • Cosign signing of images + chart releases. Half-day CI add.
  • Latency log-enrichment (duration_ms field on request slog lines). Tiny additive change.
  • SCIM lifecycle webhook on the broker. Optional add when a customer asks.
  • Userinfo-based groups in the broker Verifier. Slice C.1 ships ID-token-only; AllowEmails is the documented workaround.
  • Case-sensitive group matching for Keycloak. Slice C.1 chose case-insensitive for the validated IdP set.
  • Configurable lease timings (sweeper.leaseDuration, renewDeadline, retryPeriod). C.2 hardcodes the client-go defaults.
  • SIGHUP on revoke + sweep + rotate to shrink the revoke-to-effect window. Needs pods/exec RBAC across world namespaces.
  • Sweeper backing-store migration when the issuances Secret hits the ~1MB / ~5000-issuance k8s storage ceiling.
  • Cluster-shared rate limiter (Slice C.4 follow-up).
  • Idle-key GC for the rate-limit registry (Slice C.4 follow-up).
  • Timeouts on the existing test-protocol/test-server/test-client CI jobs. §6.3.D.1 added timeout-minutes: 20 to the new jobs; the older jobs still rely on the GitHub Actions default (360 min). One-line cleanup.
  • Hardened broker chart Secret symmetry. Broker's secret-issuances.yaml uses the simpler lookup-skip pattern (vs. the §6.3.D.2 three-branch hybrid on the server chart). Defense-in-depth follow-up; not load-bearing.
  • Base-image digest pinning (§6.7.A CodeRabbit nit skipped). When renovate/dependabot is wired to bump base-image digests, pin all three Dockerfiles to specific SHAs for reproducibility.
  • Single-image distribution mode (§6.7.A architectural alternative). If a customer asks for one-image-with-all-binaries, the unified-Dockerfile shape from the §6.7.A draft can be added as an opt-in. The split-images shape stays the default.
  • Server image CA bundle (§6.7.A deferred). If a need arises for kubectl exec debugging or future outbound-TLS server features. One-line add.
  • Version-stamping for demarkus-server + demarkus CLI + demarkus-token + demarkus-publish (§6.7.B deferred). Currently var version is wired in demarkus-agent and demarkus-broker only; goreleaser's -X flag on the others is a no-op. Add var version = "dev" + a -version flag to each missing binary; surface via --version subcommand. Out of §6.7 scope; small follow-up PR.
  • Independent chart versioning (§6.7.B.3 deferred). Currently chart-version is pinned 1:1 to module version on release. If we need to ship a chart-only fix (broken template, NOTES.txt edit) without bumping the image, introduce a charts/<name>/v* tag scheme. Adds tag-scheme complexity; revisit when we have evidence of need.

Risks

  • Observability via logs ceiling. If customers want signals not derivable from current slog, we hit a wall. Mitigation: log enrichment is a small additive change.
  • Broker secret-write blast radius. Mitigated by namespace-scoped Roles, audit log, optional NetworkPolicy. Per-world Role is non-negotiable. TestMintRBACDeniedNoPartialState validates clean failure.
  • OIDC provider coupling. First impl is Google; structure so second provider is a one-day add.
  • Token revocation in-flight latency. Revocation latency = kubelet Secret propagation + whatever the server does on re-mount. Acceptable for the deployment posture (~60s default kubelet propagation).
  • Rotate transient two-tokens window. Documented; verified by TestRotateTokenSoftPartial.
  • §6.1 chart upgrade-wipe gap. ✓ Resolved by §6.3.D.2 PR #119.
  • §6.1 chart-test plumbing latent failure. ✓ Resolved by §6.3.D.1 PR #118; agent chart equivalent resolved by §6.7.A PR #121.
  • §6.7 multi-binary image hand-off. ✓ Resolved by §6.7.A PR #121.
  • Release-pipeline image gap. ✓ Resolved by §6.7.B PRs #122 + #123 + #124. First module-touching commit to main will fire the corresponding release flow and produce a fully-aligned chart+image bundle.
  • Issuances Secret scaling wall at the ~1MB / ~5000-issuance k8s storage ceiling. Phase-7+ fix is a different backing store.
  • Chart proliferation. Three charts + examples + dashboards. Mitigate with shared common-labels templates.
  • Trial scope creep. First customer is path-B.
  • Per-replica rate limit ⇒ N× effective rate. Slice C.4 documented; deferred fix in §Backlog.
  • Rate-limit registry growth under misconfigured XFF trust. Documented in C.4 §Decisions and chart README.
  • First module-release tag fire is in-cycle on a feat: or fix: commit to a module path. Until that fires, no image or chart is on GHCR despite the pipeline being live. Operators building from main need make image locally in the interim. Not a code risk; an operational one. Closes naturally on the first qualifying merge to main.

Status

Plan v17, 2026-05-14. Slice A, 6.0 chart, 6.1 chart (incl. §6.3.D.2 + §6.7.A image fix + §6.7.B release), 6.2 broker binary (Slices B + C.1–C.4), 6.3 broker chart (all five sub-slices), 6.7.0 store hoist, 6.7.A image consolidation + CLI relocation, 6.7.B release pipeline all merged (PRs #106, #107, #108, #109, #110, #111, #112, #114, #115, #116, #117, #118, #119, #120, #121, #122, #123, #124).

14 PRs merged across 2026-05-13–14 (the §6.3 → §6.7 sprint): #115 (6.3.A) → #116 (6.3.B) → #117 (6.3.C) → #118 (6.3.D.1) → #119 (6.3.D.2) → #120 (6.7.0 store hoist) → #121 (6.7.A image consolidation) → #122 (6.7.B.1 image release) → #123 (6.7.B.2 tools release) → #124 (6.7.B.3 OCI chart publish). Phase 6 ~85% complete by sub-phase: §6.0 / §6.1 / §6.2 / §6.3 / §6.7 done; §6.4 / §6.5 / §6.6 remaining.

§6.7.B trajectory recap (three sub-PRs, same-day):

  • 6.7.B.1 (PR #122, 0e1a4ae) — docker/build-push-action@v6 added to existing server/v* + client/v* release jobs. Multi-arch push to demarkus-server + demarkus-agent images. Chose build-push-action over re-introducing goreleaser's dockers: block to keep multi-stage Dockerfiles unchanged and portable.
  • 6.7.B.2 (PR #123, afb6155) — new tools/.goreleaser.yml (broker + token + publish archives) + new tools filter/test-tools/semver-tools/release-tools jobs in workflow. tools is now a peer module with independent tools/v* tag-driven release cadence. Admin CLIs (token, publish) ship as archives only — no runtime image per §6.7.A decision on minimal pod blast radius.
  • 6.7.B.3 (PR #124, 9cf6303) — OCI chart publish via helm package --version $VER --app-version $VER + helm push oci://ghcr.io/latebit-io/charts/<chart>. Chart-version + appVersion pinned 1:1 to module version on release. Helm 3.13+ uses the docker login config natively for OCI auth, so no separate registry login step.

Next: §6.5 observability recipes. Independent of any prior phase. Per-backend log-shipper configs in deploy/observability/{datadog,otel-collector,vector,fluent-bit,grafana-alloy}/ mapping demarkus's existing structured slog output to each backend. Pure config files; no Go changes. Customer's SRE picks one. Parallelizable with §6.6 docs work. Then §6.4 topology examples last (now unblocked by the §6.7.B chart OCI publish — examples can reference oci://ghcr.io/latebit-io/charts/<chart> pull URLs). Two-week deadline at 2026-05-27 with ~13 days remaining.

trail
  1. soul.demarkus.io v17