soul.demarkus.io/plans/universe-deployment.md/v2 draft reader meta

Plan: Universe Deployment

Ship the enterprise-grade deployment for demarkus. The deliverable is the deployment: a customer's ops team installs a Helm chart, runs a broker, runs an agent, and has a federated demarkus universe their org can use. Any company evaluating demarkus (their internal "POC") installs the same product an established customer runs in production. There is no separate "POC slice" — the slice mentality is rejected. We build it once, right, and customers trial the real thing.

Goal

Deliver a complete, supportable, production-grade Kubernetes deployment package for demarkus, including:

  1. A Helm chart for demarkus-server (one world).
  2. A Helm chart for demarkus-broker (OIDC token issuance + revocation).
  3. A Helm chart for demarkus-agent (hub aggregator, crawl-and-index).
  4. Reference topology examples (Argo CD ApplicationSet, Kustomize overlay).
  5. Observability — Prometheus metrics endpoints, ServiceMonitor CRDs, reference Grafana dashboards.
  6. Customer-facing documentation — installation, security/threat model, operations, upgrade path, per-provider OIDC setup.
  7. A release pipeline producing signed images and chart releases consumable from a known registry.

The same artifacts power first-customer trial and steady-state operations.

Non-Goals (Phase 7+ territory)

  • Multi-replica worlds with shared storage (RWX / object backend).
  • Cross-cluster universe federation.
  • Operator with a World CRD.
  • Hosted / managed SaaS.
  • Non-markdown content.

Constraints

  • No core protocol/server changes unless explicitly discussed. Per the prior precedent (Claude Code plugin, Obsidian plugin, feedback_plugin_scope.md). One known exception to discuss before landing:
    • Prometheus /metrics endpoint on server + broker. Required for enterprise observability; minimal scope; needs explicit go-ahead. See open question 1.
  • Capability-based auth model is non-negotiable. The server never learns identity — only labels.
  • Markdown-only scope is non-negotiable.

Decisions (resolved during planning)

  • Health probes use exec, not HTTP. Both charts ship liveness/readiness exec probes that invoke the demarkus CLI fetching /.well-known/agent-manifest.md (always public per /architecture.md). No core change. Same pattern as redis-cli ping or pg_isready.
  • UDP port is a values knob. Default 6309, override via server.udpPort. Documented option: switch to 443 for VPN/middlebox-hostile networks (Cloudflare Warp Zero Trust, corp firewalls that filter non-standard UDP). Protocol default stays 6309; the chart just makes the flip cheap.
  • Wildcard TLS via cert-manager DNS-01. Single *.<root> cert covers all worlds + broker hostnames. HTTP-01 cannot work for worlds (no HTTP); DNS-01 is the standard path for QUIC services.
  • DNS topology. One A record per world + one for the broker, all under the same wildcard zone. Each world exposed by Service: LoadBalancer protocol: UDP; broker by standard Ingress (HTTPS).
  • Hub aggregator is the real demarkus-agent (Phase 5 binary). Shell-script CronJob was a duct-tape demo idea; the productized form is the agent crawling each team world and publishing an index to the hub. This sequences Phase 5 ahead of Phase 6 hub work.
  • Multi-OIDC. Broker speaks generic OIDC, not Google-specific. Provider behind a Verifier interface. Google validated first; Okta, Entra ID, Auth0 follow with config only.
  • Workload Identity (GKE) as a broker option. Values flag annotates the broker ServiceAccount with the GSA mapping. Off by default; on for GKE customers wanting no static SA keys. Not applicable to world servers (no GCP API calls).
  • Broker is HA. Multi-replica with resourceVersion optimistic concurrency on every Secret write, retry on conflict. Standard Ingress in front.
  • Observability. Prometheus /metrics on server + broker (core change pending). ServiceMonitor CRDs in both charts behind a metrics.enabled values flag. Reference Grafana dashboards ship in deploy/grafana/.
  • Backup/DR is documented, not built. The chart deliberately does not run backup CronJobs. Operations doc covers Velero, VolumeSnapshot, and demarkus-agent sync as DR options. Operator picks.

Open Questions

  1. Metrics core change. Add /metrics HTTP endpoint (Prometheus text format) to demarkus-server and demarkus-broker. ~50 lines per binary. Per the no-core-changes rule this needs explicit go-ahead before landing. Without it, enterprise SRE has no clean way to alert.
  2. Image hosting. GHCR under ghcr.io/latebit-io/* — confirm.
  3. Chart registry. OCI charts in GHCR (modern idiom) or a static-site chart repo at charts.demarkus.io (wider compatibility)?
  4. Release signing. Cosign-sign images and chart releases? Recommended; ~half day.
  5. First customer trial. Nesto (*.library.nesto.ca) is path-B — trial waits for product. Trial runbook lands at /trials/nesto.md once scoping starts.

Repository Layout

deploy/
  helm/
    demarkus-server/         # one-world chart
    demarkus-broker/         # OIDC token broker chart
    demarkus-agent/          # crawl/index agent chart (after Phase 5 agent ships)
  k8s/
    examples/
      applicationset.yaml    # Argo CD ApplicationSet over a worlds: list
      kustomize-overlay/     # Kustomize alternative
  grafana/
    server-dashboard.json
    broker-dashboard.json
  scripts/                   # operator helpers (cert pre-check, MTU probe, etc.)

tools/
  demarkus-broker/           # broker binary (monorepo until API stabilizes, then split)
  demarkus-agent/            # agent binary (Phase 5 work)
  internal/token/            # shared mint library, extracted from demarkus-token

Broker stays in monorepo until API stabilizes — same rationale as the Obsidian plugin pre-split: coupled iteration on the chart Secret schema is faster in one repo.

Sub-Phases

6.0 — Prerequisite: demarkus-agent core loop (Phase 5)

Before hub aggregation can land, the agent binary must exist. Phase 5 already plans this; what's needed concretely:

  • tools/demarkus-agent crawl — fetch one or more seed servers, build a hash index, publish to a configured hub document.
  • tools/demarkus-agent daemon — same loop on a schedule, with conditional fetch (if-none-match) for politeness.
  • Auth-aware: reads token from DEMARKUS_AUTH per server.
  • No new core primitives — uses existing FETCH, LIST, PUBLISH, VERSIONS.

Acceptance: demarkus-agent crawl --seed mark://team-a:6309 --seed mark://team-b:6309 --hub mark://hub:6309 --hub-path /index.md produces a hub index document linking all crawled paths, on a schedule.

6.1 — demarkus-server Helm chart

deploy/helm/demarkus-server/. Production-grade.

Workload:

  • StatefulSet, 1 replica (multi-replica is Phase 7).
  • volumeClaimTemplates — each world owns its PVC. Never a shared PVC.
  • Container image bundles demarkus-server + demarkus CLI (for exec probes).
  • Exec liveness + readiness probes against /.well-known/agent-manifest.md.
  • Resource requests/limits with sane defaults, overridable.
  • Service type LoadBalancer, protocol: UDP, port from server.udpPort (default 6309). Annotations for cloud-specific LB type (NLB on AWS, etc.).

Secrets:

  • <release>-tokens — TOML, SHA-256 hashes. Server-mounted.
  • <release>-token-values — raw tokens. Broker-only. Kept separate so server never mounts raw secrets.

Auth + TLS:

  • TLS Secret mounted via volumeMounts; cert/key paths via flags.
  • Optional cert-manager Certificate resource (behind a flag) requesting *.<root> from a configured ClusterIssuer.

Bootstrap Job:

  • Mints initial admin token on first install via the shared mint library.
  • Idempotent — skips on re-install if admin label exists.
  • SIGHUPs pod after writing.

Observability:

  • ServiceMonitor template behind metrics.enabled (requires server /metrics).
  • slog output structured for cluster log aggregation; audit-log filtering documented.

RBAC:

  • Bootstrap Job SA with get/update on the named Secret, get/list/create on pods/exec for SIGHUP. Namespace-scoped Role, not ClusterRole.

Tests:

  • helm-unittest for templates.
  • Kind-based integration test in CI: install chart → exec into pod → verify health → publish via CLI → verify version increments.

Acceptance:

  • helm install team-a ./deploy/helm/demarkus-server produces a healthy world reachable via the LB hostname.
  • kubectl delete pod team-a-0 — content persists.
  • helm upgrade --set tokens.labels[1].name=writer — adds label, SIGHUPs, no rotation of existing labels.
  • --set readOnly=true — server rejects writes.
  • helm-unittest + integration test pass in CI.

6.2 — demarkus-broker binary (tools/demarkus-broker/)

Full feature set, no skip-DELETE corner-cuts.

Routes:

  • GET /healthz — liveness.
  • GET /metrics — Prometheus (requires core change).
  • GET /login?world=<name> — start OIDC, PKCE, signed state cookie.
  • GET /callback — verify id_token, mint, write Secret, SIGHUP, redirect to install page.
  • GET /me/install — render install page (token + claude.json snippet). One-time view.
  • POST /tokens — programmatic mint via OIDC bearer (CI use).
  • DELETE /tokens/{world}/{label} — revoke. Removes from Secret + SIGHUPs.
  • GET /worlds — list worlds the caller may mint against (derived from OIDC group/email mapping).

Implementation:

  • Go single binary. Refactor tools/demarkus-token generate path into tools/internal/token/ — pure-function library shared by CLI and broker.
  • Provider behind Verifier interface (Verify(token) (sub, email, hd, groups, err)). Google concrete impl first; Okta/Entra follow without code changes.
  • Per-namespace Role/RoleBinding: get/update on named Secrets, get/create on named pods/exec. No ClusterRole.
  • resourceVersion-based optimistic concurrency on Secret writes; retry on conflict (HA-safe).
  • Audit log to stdout: JSON line per action (mint, revoke, denied). No DB.
  • Per-user rate limit on POST /tokens.

Tests:

  • Unit: Verifier mock, mint lib, Secret-write retry.
  • Integration: kind cluster + Dex (lightweight OIDC) to exercise full flow without external Google dependency.

6.3 — demarkus-broker Helm chart

deploy/helm/demarkus-broker/.

  • Deployment, multi-replica (default 2). HA-safe given resourceVersion retry.
  • Service, standard Ingress (HTTPS), optional cert-manager Certificate.
  • ConfigMap: world list, OIDC provider config, group/domain → world mapping, allowed paths/ops per group.
  • Secret: OIDC client secret.
  • ServiceAccount with namespace-scoped Role/RoleBinding per world namespace.
  • Optional Workload Identity annotation (serviceAccount.workloadIdentity.gsa) for GKE.
  • ServiceMonitor behind metrics.enabled.
  • PodDisruptionBudget (minAvailable: 1).
  • Optional NetworkPolicy restricting egress to k8s API + OIDC issuer + world Services.

Acceptance:

  • helm install broker ./deploy/helm/demarkus-broker produces a reachable broker.
  • Sign in via configured OIDC, mint a token, paste snippet, read/write the target world. <2 minutes end-to-end.
  • Revoke via DELETE — next request from that token fails within seconds.
  • Two replicas under concurrent mint load — no lost labels.

6.4 — Universe topology examples (deploy/k8s/examples/)

  • Argo CD ApplicationSet over a worlds: list. Three-world reference universe.
  • Kustomize overlay alternative for non-Argo clusters.
  • README walking an operator from zero to a working universe in <15 minutes.

No new code. Reference manifests + docs.

6.5 — Observability

Conditional on the metrics-endpoint core change being approved (open question 1).

  • Server metrics: request count by verb, latency histogram, audit counts (publish/append/archive success/failure), version-count gauges.
  • Broker metrics: mint/revoke counts by world, OIDC verification failures, Secret-write retries, in-flight requests.
  • ServiceMonitor templates in both charts.
  • Reference Grafana dashboards in deploy/grafana/ (JSON, importable).
  • Operations doc covers wiring into kube-prometheus, GKE Managed Prometheus, etc.

6.6 — Documentation suite

Soul:

  • /deployment/installation.md — chart values reference, step-by-step install.
  • /deployment/security.md — threat model, RBAC requirements, attack surface, audit log conventions, capability-auth review.
  • /deployment/operations.md — backup/DR options, observability wiring, upgrade procedure, troubleshooting.
  • /deployment/multi-oidc.md — provider-by-provider setup (Google, Okta, Entra ID).
  • /trials/<customer>.md — per-customer trial runbook. Nesto's lands first.

Repo:

  • Each chart has its own README.md with values reference + quick start.

6.7 — Release pipeline

  • GoReleaser extended to push container images for demarkus-server, demarkus-broker, demarkus-agent to GHCR (confirm).
  • Helm chart releases on tag — OCI charts to GHCR or static-site repo (open question 3).
  • Cosign signing (open question 4).
  • CI: helm-lint + helm-unittest + kind-based integration tests on every PR touching deploy/ or the broker/agent binaries.

Sequencing

Dependency chain matters. Don't start hub aggregation or broker chart before their prerequisites.

  1. Phase 5 agent (6.0 prerequisite) — must land first.
  2. Metrics core-change decision — gate item. Without buy-in, 6.5 is hollow.
  3. 6.1 server chart — parallel with broker binary work.
  4. 6.2 broker binary — parallel with chart; needs token-mint lib refactor first.
  5. 6.3 broker chart — after 6.2 is testable end-to-end.
  6. 6.5 observability — wires into both charts; lands once metrics endpoints exist.
  7. 6.4 topology examples — small, after 6.1 + 6.3 stable.
  8. 6.6 docs — incremental, each sub-phase contributes its piece.
  9. 6.7 release pipeline — final hardening; signed and tagged artifacts.

Rough effort: 4–8 weeks of focused work depending on how fast open questions resolve.

Risks

  • Metrics core change refused. Observability collapses to log-tail dashboards. Enterprises will push back; expect this to surface in customer security reviews.
  • Broker secret-write blast radius. Holds k8s API creds across world namespaces. Mitigated by namespace-scoped Roles, audit log, optional NetworkPolicy, and a sealed-secrets/external-secrets recipe in the operations doc.
  • OIDC provider coupling. First impl is Google; structure so the second provider is a one-day add. Late abstraction = rewrite risk.
  • Token revocation latency. SIGHUP reloads tokens.toml, but in-flight requests with the revoked token complete. Property of the model. Document.
  • Chart proliferation. Three charts + topology examples + dashboards. Risk of values drift. Mitigate with shared common-labels templates and CI-validated values schemas.
  • Trial scope creep. First customer (nesto, path-B). If feedback pulls scope back toward "demo slice" mid-build, decide explicitly — don't drift.

Status

Plan rewritten 2026-05-11 after framing shift: deliverable is the deployment, not a slice. Awaiting decisions on metrics core change, image hosting, chart registry, and signing before kickoff. /plans/poc-deployment.md is superseded and archived.

trail
  1. soul.demarkus.io v2