soul.demarkus.io/plans/universe-deployment.md/v3 draft reader meta

Plan: Universe Deployment

Ship the enterprise-grade deployment for demarkus. The deliverable is the deployment: a customer's ops team installs a Helm chart, runs a broker, runs an agent, and has a federated demarkus universe their org can use. Any company evaluating demarkus (their internal "POC") installs the same product an established customer runs in production. There is no separate "POC slice" — the slice mentality is rejected. We build it once, right, and customers trial the real thing.

Goal

Deliver a complete, supportable, production-grade Kubernetes deployment package for demarkus, including:

  1. A Helm chart for demarkus-server (one world).
  2. A Helm chart for demarkus-broker (OIDC token issuance + revocation).
  3. A Helm chart for demarkus-agent (hub aggregator, crawl-and-index).
  4. Reference topology examples (Argo CD ApplicationSet, Kustomize overlay).
  5. Backend-agnostic observability — structured slog emission + reference configs for Datadog, OTel Collector, Vector, Fluent Bit, Grafana Alloy.
  6. Customer-facing documentation — installation, security/threat model, operations, upgrade path, per-provider OIDC setup, observability recipes.
  7. A release pipeline producing images and chart releases consumable from GHCR.

The same artifacts power first-customer trial and steady-state operations.

Non-Goals (Phase 7+ territory)

  • Multi-replica worlds with shared storage (RWX / object backend).
  • Cross-cluster universe federation.
  • Operator with a World CRD.
  • Hosted / managed SaaS.
  • Non-markdown content.

Constraints

  • No core protocol/server changes. Period. Per the prior precedent (Claude Code plugin, Obsidian plugin, feedback_plugin_scope.md). Observability is achieved by log derivation in a collector, not by adding /metrics or OTel SDK calls to the server.
  • Capability-based auth model is non-negotiable. The server never learns identity — only labels.
  • Markdown-only scope is non-negotiable.

Decisions (resolved during planning)

  • Health probes use exec, not HTTP. Both charts ship liveness/readiness exec probes that invoke the demarkus CLI fetching /.well-known/agent-manifest.md (always public per /architecture.md). No core change. Same pattern as redis-cli ping / pg_isready.
  • UDP port is a values knob. Default 6309, override via server.udpPort. Documented option: switch to 443 for VPN/middlebox-hostile networks (Cloudflare Warp Zero Trust, corp firewalls that filter non-standard UDP). Protocol default stays 6309.
  • Wildcard TLS via cert-manager DNS-01. Single *.<root> cert covers all worlds + broker hostnames. HTTP-01 cannot work for worlds (no HTTP); DNS-01 is the standard path for QUIC services.
  • DNS topology. One A record per world + one for the broker, all under the same wildcard zone. Each world exposed by Service: LoadBalancer protocol: UDP; broker by standard Ingress (HTTPS).
  • Hub aggregator is the real demarkus-agent (Phase 5 binary). Shell-script CronJob rejected as duct-tape; productized form is the agent crawling team worlds and publishing an index to the hub. Phase 5 sequences ahead of Phase 6 hub work.
  • Multi-OIDC. Broker speaks generic OIDC, not Google-specific. Provider behind a Verifier interface. Google validated first; Okta, Entra ID, Auth0 follow with config only.
  • Workload Identity (GKE) as a broker option. Values flag annotates the broker ServiceAccount with the GSA mapping. Off by default; on for GKE customers wanting no static SA keys. Not applicable to world servers.
  • Broker is HA. Multi-replica with resourceVersion optimistic concurrency on Secret writes, retry on conflict.
  • Observability is log-derived, backend-agnostic. demarkus already emits structured slog. The chart ships Datadog autodiscovery annotations + reference configs for OTel Collector, Vector, Fluent Bit, Grafana Alloy. Customer's SRE picks the agent/backend. No demarkus-side metrics emission, no core changes. Audit lines from server (op, path, success, token_label) joined with broker lines (mint/revoke/denied with sub, email, world, label) reconstruct the full identity→action chain in the customer's tooling. Latency histograms deferred — depends on what current slog includes per request; revisit during trials if needed (smallest mitigation is enriching existing slog calls with duration_ms, decision deferred).
  • Backup/DR is documented, not built. The chart deliberately does not run backup CronJobs. Operations doc covers Velero, VolumeSnapshot, and demarkus-agent sync as DR options. Operator picks.
  • Image hosting: ghcr.io/latebit-io/{demarkus-server,demarkus-broker,demarkus-agent}.
  • Chart registry: OCI charts in GHCR.
  • Cosign signing: deferred to backlog. Half-day CI add when wanted.

Open Questions

(None blocking kickoff. The two resolved deferrals — latency log-enrichment and cosign signing — are revisited if/when trials surface a need.)

  1. First customer trial. Nesto (*.library.nesto.ca) is path-B — trial waits for product. Trial runbook lands at /trials/nesto.md when scoping starts.

Repository Layout

deploy/
  helm/
    demarkus-server/         # one-world chart
    demarkus-broker/         # OIDC token broker chart
    demarkus-agent/          # crawl/index agent chart (after Phase 5 agent ships)
  k8s/
    examples/
      applicationset.yaml    # Argo CD ApplicationSet over a worlds: list
      kustomize-overlay/     # Kustomize alternative
  observability/
    datadog/                 # autodiscovery annotations + dashboard JSON
    otel-collector/          # collector config recipes
    vector/                  # vector config recipes
    fluent-bit/              # fluent-bit parser + filter recipes
  scripts/                   # operator helpers (cert pre-check, MTU probe, etc.)

tools/
  demarkus-broker/           # broker binary (monorepo until API stabilizes, then split)
  demarkus-agent/            # agent binary (Phase 5 work)
  internal/token/            # shared mint library, extracted from demarkus-token

Sub-Phases

6.0 — Prerequisite: demarkus-agent core loop (Phase 5)

Before hub aggregation can land, the agent binary must exist. Phase 5 already plans this; what's needed concretely:

  • tools/demarkus-agent crawl — fetch one or more seed servers, build a hash index, publish to a configured hub document.
  • tools/demarkus-agent daemon — same loop on a schedule, with conditional fetch (if-none-match) for politeness.
  • Auth-aware: reads token from DEMARKUS_AUTH per server.
  • No new core primitives — uses existing FETCH, LIST, PUBLISH, VERSIONS.

Acceptance: demarkus-agent crawl --seed mark://team-a:6309 --seed mark://team-b:6309 --hub mark://hub:6309 --hub-path /index.md produces a hub index document linking all crawled paths, on a schedule.

6.1 — demarkus-server Helm chart

deploy/helm/demarkus-server/. Production-grade.

Workload:

  • StatefulSet, 1 replica (multi-replica is Phase 7).
  • volumeClaimTemplates — each world owns its PVC. Never a shared PVC.
  • Container image bundles demarkus-server + demarkus CLI (for exec probes).
  • Exec liveness + readiness probes against /.well-known/agent-manifest.md.
  • Resource requests/limits with sane defaults, overridable.
  • Service type LoadBalancer, protocol: UDP, port from server.udpPort (default 6309). Annotations for cloud-specific LB type (NLB on AWS, etc.).

Secrets:

  • <release>-tokens — TOML, SHA-256 hashes. Server-mounted.
  • <release>-token-values — raw tokens. Broker-only. Kept separate so server never mounts raw secrets.

Auth + TLS:

  • TLS Secret mounted via volumeMounts; cert/key paths via flags.
  • Optional cert-manager Certificate resource (behind a flag) requesting *.<root> from a configured ClusterIssuer.

Bootstrap Job:

  • Mints initial admin token on first install via the shared mint library.
  • Idempotent — skips on re-install if admin label exists.
  • SIGHUPs pod after writing.

Observability:

  • Pod annotations for Datadog autodiscovery (ad.datadoghq.com/...).
  • Reference configs for OTel Collector, Vector, Fluent Bit in deploy/observability/.
  • slog output already structured; no chart-side instrumentation needed.

RBAC:

  • Bootstrap Job SA with get/update on the named Secret, get/list/create on pods/exec for SIGHUP. Namespace-scoped Role, not ClusterRole.

Tests:

  • helm-unittest for templates.
  • Kind-based integration test in CI: install chart → exec into pod → verify health → publish via CLI → verify version increments.

Acceptance:

  • helm install team-a ./deploy/helm/demarkus-server produces a healthy world reachable via the LB hostname.
  • kubectl delete pod team-a-0 — content persists.
  • helm upgrade --set tokens.labels[1].name=writer — adds label, SIGHUPs, no rotation of existing labels.
  • --set readOnly=true — server rejects writes.
  • helm-unittest + integration test pass in CI.

6.2 — demarkus-broker binary (tools/demarkus-broker/)

Full feature set, no skip-DELETE corner-cuts.

Routes:

  • GET /healthz — liveness.
  • GET /login?world=<name> — start OIDC, PKCE, signed state cookie.
  • GET /callback — verify id_token, mint, write Secret, SIGHUP, redirect to install page.
  • GET /me/install — render install page (token + claude.json snippet). One-time view.
  • POST /tokens — programmatic mint via OIDC bearer (CI use).
  • DELETE /tokens/{world}/{label} — revoke. Removes from Secret + SIGHUPs.
  • GET /worlds — list worlds the caller may mint against (derived from OIDC group/email mapping).

Implementation:

  • Go single binary. Refactor tools/demarkus-token generate path into tools/internal/token/ — pure-function library shared by CLI and broker.
  • Provider behind Verifier interface (Verify(token) (sub, email, hd, groups, err)). Google concrete impl first; Okta/Entra follow without code changes.
  • Per-namespace Role/RoleBinding: get/update on named Secrets, get/create on named pods/exec. No ClusterRole.
  • resourceVersion-based optimistic concurrency on Secret writes; retry on conflict (HA-safe).
  • Audit log to stdout: JSON line per action (mint, revoke, denied). No DB. These lines are the primary observability signal — joined with server audit lines via token_label, they reconstruct identity→action.
  • Per-user rate limit on POST /tokens.

Tests:

  • Unit: Verifier mock, mint lib, Secret-write retry.
  • Integration: kind cluster + Dex (lightweight OIDC) to exercise full flow without external Google dependency.

6.3 — demarkus-broker Helm chart

deploy/helm/demarkus-broker/.

  • Deployment, multi-replica (default 2). HA-safe given resourceVersion retry.
  • Service, standard Ingress (HTTPS), optional cert-manager Certificate.
  • ConfigMap: world list, OIDC provider config, group/domain → world mapping, allowed paths/ops per group.
  • Secret: OIDC client secret.
  • ServiceAccount with namespace-scoped Role/RoleBinding per world namespace.
  • Optional Workload Identity annotation (serviceAccount.workloadIdentity.gsa) for GKE.
  • Pod annotations for Datadog autodiscovery.
  • PodDisruptionBudget (minAvailable: 1).
  • Optional NetworkPolicy restricting egress to k8s API + OIDC issuer + world Services.

Acceptance:

  • helm install broker ./deploy/helm/demarkus-broker produces a reachable broker.
  • Sign in via configured OIDC, mint a token, paste snippet, read/write the target world. <2 minutes end-to-end.
  • Revoke via DELETE — next request from that token fails within seconds.
  • Two replicas under concurrent mint load — no lost labels.

6.4 — Universe topology examples (deploy/k8s/examples/)

  • Argo CD ApplicationSet over a worlds: list. Three-world reference universe.
  • Kustomize overlay alternative for non-Argo clusters.
  • README walking an operator from zero to a working universe in <15 minutes.

No new code. Reference manifests + docs.

6.5 — Observability recipes (deploy/observability/)

Backend-agnostic. Customer picks one collector; recipes for the common stacks ship in the repo and are referenced from the operations doc.

  • Datadog: pod annotations + parser config + dashboard JSON. Logs-to-Metrics rules for common counters.
  • OTel Collector: filelogreceiver + JSON parser + transform → OTLP export. Reference values for helm install opentelemetry-collector ....
  • Vector: kubernetes_logs source → remap for JSON parsing → sinks (Datadog, Loki, etc).
  • Fluent Bit: tail input + parser + filter for audit-tagged lines.
  • Grafana Alloy: similar shape, Loki-friendly.

Each recipe answers: how do I count publishes per world per minute? How do I count OIDC denials per hour? How do I get an alert when token mints spike?

No demarkus-side instrumentation. The observability deliverable is configs + dashboards + docs, not code.

6.6 — Documentation suite

Soul:

  • /deployment/installation.md — chart values reference, step-by-step install.
  • /deployment/security.md — threat model, RBAC requirements, attack surface, audit log conventions, capability-auth review.
  • /deployment/operations.md — backup/DR options, observability wiring, upgrade procedure, troubleshooting.
  • /deployment/multi-oidc.md — provider-by-provider setup (Google, Okta, Entra ID).
  • /deployment/observability.md — how to wire each collector, what the key metrics are, sample dashboards.
  • /trials/<customer>.md — per-customer trial runbook. Nesto's lands first.

Repo:

  • Each chart has its own README.md with values reference + quick start.

6.7 — Release pipeline

  • GoReleaser extended to push container images for demarkus-server, demarkus-broker, demarkus-agent to ghcr.io/latebit-io/*.
  • Helm chart releases on tag — OCI charts to GHCR (oci://ghcr.io/latebit-io/charts/*).
  • Cosign signing — deferred to backlog. Half-day add later.
  • CI: helm-lint + helm-unittest + kind-based integration tests on every PR touching deploy/ or the broker/agent binaries.

Sequencing

Dependency chain matters. Don't start hub aggregation or broker chart before their prerequisites.

  1. Phase 5 agent (6.0 prerequisite) — must land first.
  2. 6.1 server chart — parallel with broker binary work; does not block on agent.
  3. 6.2 broker binary — parallel with chart; needs token-mint lib refactor first.
  4. 6.3 broker chart — after 6.2 is testable end-to-end.
  5. 6.4 topology examples — small, after 6.1 + 6.3 stable.
  6. 6.5 observability recipes — incremental, lands as charts stabilize.
  7. 6.6 docs — incremental, each sub-phase contributes its piece.
  8. 6.7 release pipeline — final hardening; signed images come in the cosign-deferred follow-up.

Rough effort: 4–8 weeks of focused work.

Backlog (deferred, easy to add later)

  • Cosign signing of images + chart releases. Half-day CI add when wanted.
  • Latency log-enrichment (duration_ms field on request slog lines). Tiny additive change. Defer until trials show it's needed.

Risks

  • Observability via logs ceiling. If customers want signals not derivable from current slog (latency, internal state like version counts or PVC fullness), we hit a wall. Mitigation: log enrichment is a small additive change; internal-state metrics can be derived by a cluster-side sidecar that calls LIST periodically. Both deferred until a customer surfaces the need.
  • Broker secret-write blast radius. Holds k8s API creds across world namespaces. Mitigated by namespace-scoped Roles, audit log, optional NetworkPolicy, and a sealed-secrets/external-secrets recipe in the operations doc.
  • OIDC provider coupling. First impl is Google; structure so the second provider is a one-day add. Late abstraction = rewrite risk.
  • Token revocation latency. SIGHUP reloads tokens.toml, but in-flight requests with the revoked token complete. Property of the model. Document.
  • Chart proliferation. Three charts + topology examples + dashboards. Risk of values drift. Mitigate with shared common-labels templates and CI-validated values schemas.
  • Trial scope creep. First customer (nesto, path-B). If feedback pulls scope back toward "demo slice" mid-build, decide explicitly — don't drift.

Status

Plan v3, 2026-05-11. All planning-time decisions resolved; observability landed as log-derived backend-agnostic; cosign deferred to backlog. Ready to start work: Phase 5 agent (6.0) first, then demarkus-server chart (6.1) and demarkus-broker binary (6.2) in parallel.

trail
  1. soul.demarkus.io v3