soul.demarkus.io/plans/universe-deployment.md/v4 draft reader meta

Plan: Universe Deployment

Ship the enterprise-grade deployment for demarkus. The deliverable is the deployment: a customer's ops team installs a Helm chart, runs a broker, runs an agent, and has a federated demarkus universe their org can use. Any company evaluating demarkus (their internal "POC") installs the same product an established customer runs in production. There is no separate "POC slice" — the slice mentality is rejected. We build it once, right, and customers trial the real thing.

Goal

Deliver a complete, supportable, production-grade Kubernetes deployment package for demarkus, including:

  1. A Helm chart for demarkus-server (one world).
  2. A Helm chart for demarkus-broker (OIDC token issuance + revocation).
  3. A Helm chart for demarkus-agent (hub aggregator, crawl-and-index).
  4. Reference topology examples (Argo CD ApplicationSet, Kustomize overlay).
  5. Backend-agnostic observability — structured slog emission + reference configs for Datadog, OTel Collector, Vector, Fluent Bit, Grafana Alloy.
  6. Customer-facing documentation — installation, security/threat model, operations, upgrade path, per-provider OIDC setup, observability recipes.
  7. A release pipeline producing images and chart releases consumable from GHCR.

The same artifacts power first-customer trial and steady-state operations.

Non-Goals (Phase 7+ territory)

  • Multi-replica worlds with shared storage (RWX / object backend).
  • Cross-cluster universe federation.
  • Operator with a World CRD.
  • Hosted / managed SaaS.
  • Non-markdown content.

Constraints

  • No core protocol/server changes. Period. Per prior precedent (Claude Code plugin, Obsidian plugin, feedback_plugin_scope.md). Observability is achieved by log derivation in a collector, not by adding /metrics or OTel SDK calls to the server.
  • Capability-based auth model is non-negotiable. The server never learns identity — only labels.
  • Markdown-only scope is non-negotiable.

Decisions (resolved during planning)

  • Health probes use exec, not HTTP. Both charts ship liveness/readiness exec probes that invoke the demarkus CLI fetching /.well-known/agent-manifest.md (always public per /architecture.md). No core change. Same pattern as redis-cli ping / pg_isready.
  • UDP port is a values knob. Default 6309, override via server.udpPort. Documented option: switch to 443 for VPN/middlebox-hostile networks (Cloudflare Warp Zero Trust, corp firewalls that filter non-standard UDP). Protocol default stays 6309.
  • Wildcard TLS via cert-manager DNS-01. Single *.<root> cert covers all worlds + broker hostnames. HTTP-01 cannot work for worlds (no HTTP); DNS-01 is the standard path for QUIC services.
  • DNS topology. One A record per world + one for the broker, all under the same wildcard zone. Each world exposed by Service: LoadBalancer protocol: UDP; broker by standard Ingress (HTTPS).
  • Hub aggregator is the existing demarkus-agent at client/cmd/demarkus-agent/. Already implements crawl + daemon subcommands with TOML config, multi-worker crawl, per-host token auth, per-server + aggregated index publishing. Verified end-to-end on 2026-05-11. Phase 5 is no longer a blocking prerequisite.
  • demarkus-agent lives in client/cmd/, not tools/. It's a protocol client (uses fedcrawl, fetch, tokens, links), not a utility. Memory rule sharpened: client/cmd/ = protocol clients (CLI, TUI, MCP, agent); tools/ = utilities (token, publish, sync, broker).
  • Multi-OIDC. Broker speaks generic OIDC, not Google-specific. Provider behind a Verifier interface. Google validated first; Okta, Entra ID, Auth0 follow with config only.
  • Workload Identity (GKE) as a broker option. Values flag annotates the broker ServiceAccount with the GSA mapping. Off by default; on for GKE customers wanting no static SA keys. Not applicable to world servers.
  • Broker is HA. Multi-replica with resourceVersion optimistic concurrency on Secret writes, retry on conflict.
  • Observability is log-derived, backend-agnostic. demarkus already emits structured slog. Charts ship Datadog autodiscovery annotations + reference configs for OTel Collector, Vector, Fluent Bit, Grafana Alloy. Customer's SRE picks the agent/backend. Latency histograms deferred — depends on what current slog includes per request; revisit during trials if needed.
  • Backup/DR is documented, not built. The chart deliberately does not run backup CronJobs. Operations doc covers Velero, VolumeSnapshot, and demarkus-agent sync as DR options.
  • Image hosting: ghcr.io/latebit-io/{demarkus-server,demarkus-broker,demarkus-agent}.
  • Chart registry: OCI charts in GHCR.
  • Cosign signing: deferred to backlog. Half-day CI add when wanted.

Open Questions

  1. First customer trial. Nesto (*.library.nesto.ca) is path-B — trial waits for product. Trial runbook lands at /trials/nesto.md when scoping starts.

Repository Layout

deploy/
  helm/
    demarkus-server/         # one-world chart
    demarkus-broker/         # OIDC token broker chart
    demarkus-agent/          # crawl/index agent chart
  k8s/
    examples/
      applicationset.yaml    # Argo CD ApplicationSet over a worlds: list
      kustomize-overlay/     # Kustomize alternative
  observability/
    datadog/                 # autodiscovery annotations + dashboard JSON
    otel-collector/          # collector config recipes
    vector/                  # vector config recipes
    fluent-bit/              # fluent-bit parser + filter recipes
  scripts/                   # operator helpers (cert pre-check, MTU probe, etc.)

client/cmd/
  demarkus-agent/            # protocol client — federation crawler (existing)

tools/
  demarkus-broker/           # broker binary (monorepo until API stabilizes)
  internal/token/            # shared mint library, extracted from demarkus-token

Sub-Phases

6.0 — demarkus-agent verified + chart ✓ binary, ⏳ chart

Binary status (2026-05-11): existing demarkus-agent verified end-to-end against a 3-server smoke test (2 team worlds + 1 hub). Crawls, builds aggregated and per-server indexes, publishes to hub. Two bugs fixed during verification:

  • fedcrawl/crawl.go publishIndex accepted only ok status; first publish returns created, surfaced as a misleading warning despite the publish succeeding. Now accepts both.
  • publishIndex always used expected_version=0 (create-only), causing every re-publish in daemon mode to fail with conflict. Now uses -1 (no check) for idempotent hub re-publish; server's no-op-on-duplicate-content prevents version churn.
  • Makefile did not build demarkus-agent; the agent had to be hand-built. Now builds with make client.

Tests in client/internal/fedcrawl/crawl_test.go cover all three: create + re-publish + status acceptance + per-server / aggregated modes. bash pre-commit.sh clean.

Remaining 6.0 work — agent Helm chart at deploy/helm/demarkus-agent/:

  • Deployment (not StatefulSet — agent is stateless modulo state file; recoverable from next crawl).
  • ConfigMap holding the TOML agent config (seeds, hubs, crawl, politeness, schedule).
  • Secret holding per-host tokens for publishing to hub(s).
  • ServiceAccount, no special RBAC needed (no k8s API calls).
  • Exec liveness probe (demarkus-agent version returns 0).
  • No Service (agent is outbound-only).
  • Pod annotations for Datadog autodiscovery; log fields documented for mint/crawl events.

Acceptance: helm install agent ./deploy/helm/demarkus-agent --set config.seeds[0]=mark://team-a:6309 ... produces a pod that crawls on schedule and publishes to the configured hub. Pod restart resumes from state PVC (optional; can also be ephemeral).

6.1 — demarkus-server Helm chart

deploy/helm/demarkus-server/. Production-grade.

Workload:

  • StatefulSet, 1 replica (multi-replica is Phase 7).
  • volumeClaimTemplates — each world owns its PVC. Never a shared PVC.
  • Container image bundles demarkus-server + demarkus CLI (for exec probes).
  • Exec liveness + readiness probes against /.well-known/agent-manifest.md.
  • Resource requests/limits with sane defaults, overridable.
  • Service type LoadBalancer, protocol: UDP, port from server.udpPort (default 6309). Annotations for cloud-specific LB type (NLB on AWS, etc.).

Secrets:

  • <release>-tokens — TOML, SHA-256 hashes. Server-mounted.
  • <release>-token-values — raw tokens. Broker-only. Kept separate so server never mounts raw secrets.

Auth + TLS:

  • TLS Secret mounted via volumeMounts; cert/key paths via flags.
  • Optional cert-manager Certificate resource (behind a flag) requesting *.<root> from a configured ClusterIssuer.

Bootstrap Job:

  • Mints initial admin token on first install via the shared mint library.
  • Idempotent — skips on re-install if admin label exists.
  • SIGHUPs pod after writing.

Observability:

  • Pod annotations for Datadog autodiscovery.
  • Reference configs for OTel Collector, Vector, Fluent Bit in deploy/observability/.
  • slog output already structured; no chart-side instrumentation needed.

RBAC:

  • Bootstrap Job SA with get/update on the named Secret, get/list/create on pods/exec for SIGHUP. Namespace-scoped Role, not ClusterRole.

Tests:

  • helm-unittest for templates.
  • Kind-based integration test in CI: install chart → exec into pod → verify health → publish via CLI → verify version increments.

6.2 — demarkus-broker binary (tools/demarkus-broker/)

Full feature set, no skip-DELETE corner-cuts. Routes, implementation, tests as previously specified.

6.3 — demarkus-broker Helm chart

Multi-replica HA. As previously specified.

6.4 — Universe topology examples

ApplicationSet + Kustomize overlay. As previously specified.

6.5 — Observability recipes

Per-backend configs in deploy/observability/. As previously specified.

6.6 — Documentation suite

/deployment/*.md + per-chart READMEs. As previously specified.

6.7 — Release pipeline

GHCR images + OCI charts. Cosign deferred to backlog.

Sequencing

  1. 6.0 chart — quick win since binary is done.
  2. 6.1 server chart — main piece.
  3. 6.2 broker binary — parallel with 6.1.
  4. 6.3 broker chart — after 6.2 testable.
  5. 6.4 topology examples.
  6. 6.5 observability recipes.
  7. 6.6 docs — incremental throughout.
  8. 6.7 release pipeline — final.

Rough effort: 2–4 weeks of focused work (down from 4–8 in v3 — agent binary work eliminated).

Backlog (deferred, easy to add later)

  • Cosign signing of images + chart releases. Half-day CI add when wanted.
  • Latency log-enrichment (duration_ms field on request slog lines). Tiny additive change. Defer until trials show it's needed.

Risks

  • Observability via logs ceiling. If customers want signals not derivable from current slog (latency, internal state like version counts or PVC fullness), we hit a wall. Mitigation: log enrichment is a small additive change; internal-state metrics derivable by a cluster-side sidecar that calls LIST periodically.
  • Broker secret-write blast radius. Holds k8s API creds across world namespaces. Mitigated by namespace-scoped Roles, audit log, optional NetworkPolicy.
  • OIDC provider coupling. First impl is Google; structure so second provider is a one-day add.
  • Token revocation latency. SIGHUP reloads tokens.toml, but in-flight requests with revoked token complete. Property of model.
  • Chart proliferation. Three charts + examples + dashboards. Mitigate with shared common-labels templates.
  • Trial scope creep. First customer is path-B. If feedback pulls scope back to "demo slice" mid-build, decide explicitly.

Status

Plan v4, 2026-05-11. Phase 5 prerequisite cleared — demarkus-agent verified and patched. 6.0 reduced to chart-only. Ready to start: 6.0 agent chart, then 6.1 server chart. See /journal.md for verification details.

trail
  1. soul.demarkus.io v4