soul.demarkus.io/plans/universe-deployment.md/v6 draft reader meta

Plan: Universe Deployment

Ship the enterprise-grade deployment for demarkus. The deliverable is the deployment: a customer's ops team installs a Helm chart, runs a broker, runs an agent, and has a federated demarkus universe their org can use. Any company evaluating demarkus (their internal "POC") installs the same product an established customer runs in production. There is no separate "POC slice" — the slice mentality is rejected. We build it once, right, and customers trial the real thing.

Goal

Deliver a complete, supportable, production-grade Kubernetes deployment package for demarkus, including:

  1. A Helm chart for demarkus-server (one world).
  2. A Helm chart for demarkus-broker (OIDC token issuance + revocation).
  3. A Helm chart for demarkus-agent (hub aggregator, crawl-and-index).
  4. Reference topology examples (Argo CD ApplicationSet, Kustomize overlay).
  5. Backend-agnostic observability — structured slog emission + reference configs for Datadog, OTel Collector, Vector, Fluent Bit, Grafana Alloy.
  6. Customer-facing documentation — installation, security/threat model, operations, upgrade path, per-provider OIDC setup, observability recipes.
  7. A release pipeline producing images and chart releases consumable from GHCR.

The same artifacts power first-customer trial and steady-state operations.

Non-Goals (Phase 7+ territory)

  • Multi-replica worlds with shared storage (RWX / object backend).
  • Cross-cluster universe federation.
  • Operator with a World CRD.
  • Hosted / managed SaaS.
  • Non-markdown content.

Constraints

  • No core protocol/server changes. Period. Per prior precedent (Claude Code plugin, Obsidian plugin, feedback_plugin_scope.md). Observability is achieved by log derivation in a collector, not by adding /metrics or OTel SDK calls to the server. Note: lifting HashToken from server/internal/auth/ to protocol/ (Slice A) is an additive helper relocation, not a behavioral change, and is considered within the spirit of this rule.
  • Capability-based auth model is non-negotiable. The server never learns identity — only labels.
  • Markdown-only scope is non-negotiable.

Decisions (resolved during planning)

  • Health probes use exec, not HTTP. Both charts ship liveness/readiness exec probes that invoke the demarkus CLI fetching /.well-known/agent-manifest.md (always public per /architecture.md). No core change. Same pattern as redis-cli ping / pg_isready.
  • UDP port is a values knob. Default 6309, override via server.udpPort. Documented option: switch to 443 for VPN/middlebox-hostile networks (Cloudflare Warp Zero Trust, corp firewalls that filter non-standard UDP). Protocol default stays 6309.
  • Wildcard TLS via cert-manager DNS-01. Single *.<root> cert covers all worlds + broker hostnames. HTTP-01 cannot work for worlds (no HTTP); DNS-01 is the standard path for QUIC services.
  • DNS topology. One A record per world + one for the broker, all under the same wildcard zone. Each world exposed by Service: LoadBalancer protocol: UDP; broker by standard Ingress (HTTPS).
  • Hub aggregator is the existing demarkus-agent at client/cmd/demarkus-agent/. Already implements crawl + daemon subcommands with TOML config, multi-worker crawl, per-host token auth, per-server + aggregated index publishing. Verified end-to-end on 2026-05-11. Phase 5 is no longer a blocking prerequisite.
  • demarkus-agent lives in client/cmd/, not tools/. It's a protocol client (uses fedcrawl, fetch, tokens, links), not a utility. Memory rule sharpened: client/cmd/ = protocol clients (CLI, TUI, MCP, agent); tools/ = utilities (token, publish, sync, broker).
  • Multi-OIDC. Broker speaks generic OIDC, not Google-specific. Provider behind a Verifier interface. Google validated first; Okta, Entra ID, Auth0 follow with config only.
  • Workload Identity (GKE) as a broker option. Values flag annotates the broker ServiceAccount with the GSA mapping. Off by default; on for GKE customers wanting no static SA keys. Not applicable to world servers.
  • Broker is HA. Multi-replica with resourceVersion optimistic concurrency on Secret writes, retry on conflict.
  • Observability is log-derived, backend-agnostic. demarkus already emits structured slog. Charts ship Datadog autodiscovery annotations + reference configs for OTel Collector, Vector, Fluent Bit, Grafana Alloy. Customer's SRE picks the agent/backend. Latency histograms deferred — depends on what current slog includes per request; revisit during trials if needed.
  • Backup/DR is documented, not built. The chart deliberately does not run backup CronJobs. Operations doc covers Velero, VolumeSnapshot, and demarkus-agent sync as DR options.
  • Image hosting: ghcr.io/latebit-io/{demarkus-server,demarkus-broker,demarkus-agent}.
  • Chart registry: OCI charts in GHCR.
  • Cosign signing: deferred to backlog. Half-day CI add when wanted.

Open Questions

  1. First customer trial. Nesto (*.library.nesto.ca) is path-B — trial waits for product. Trial runbook lands at /trials/nesto.md when scoping starts.

Repository Layout

deploy/
  helm/
    demarkus-server/         # one-world chart
    demarkus-broker/         # OIDC token broker chart
    demarkus-agent/          # crawl/index agent chart
  k8s/
    examples/
      applicationset.yaml    # Argo CD ApplicationSet over a worlds: list
      kustomize-overlay/     # Kustomize alternative
  observability/
    datadog/                 # autodiscovery annotations + dashboard JSON
    otel-collector/          # collector config recipes
    vector/                  # vector config recipes
    fluent-bit/              # fluent-bit parser + filter recipes
  scripts/                   # operator helpers (cert pre-check, MTU probe, etc.)

client/cmd/
  demarkus-agent/            # protocol client — federation crawler (existing)

tools/
  demarkus-token/            # token mint CLI (moved from server/cmd/ in Slice A)
  demarkus-publish/          # publish CLI (moved from server/cmd/ in Slice A)
  demarkus-broker/           # broker binary (monorepo until API stabilizes)
  internal/token/            # shared mint library, extracted from demarkus-token

protocol/
  auth.go                    # HashToken (promoted from server/internal/auth/ in Slice A)

Sub-Phases

6.0 — demarkus-agent verified + chart ✓ binary, ⏳ chart

Binary status (2026-05-11): existing demarkus-agent verified end-to-end against a 3-server smoke test (2 team worlds + 1 hub). Crawls, builds aggregated and per-server indexes, publishes to hub. Two bugs fixed during verification:

  • fedcrawl/crawl.go publishIndex accepted only ok status; first publish returns created, surfaced as a misleading warning despite the publish succeeding. Now accepts both.
  • publishIndex always used expected_version=0 (create-only), causing every re-publish in daemon mode to fail with conflict. Now uses -1 (no check) for idempotent hub re-publish; server's no-op-on-duplicate-content prevents version churn.
  • Makefile did not build demarkus-agent; the agent had to be hand-built. Now builds with make client.

Tests in client/internal/fedcrawl/crawl_test.go cover all three: create + re-publish + status acceptance + per-server / aggregated modes. bash pre-commit.sh clean.

Remaining 6.0 work — agent Helm chart at deploy/helm/demarkus-agent/:

  • Deployment (not StatefulSet — agent is stateless modulo state file; recoverable from next crawl).
  • ConfigMap holding the TOML agent config (seeds, hubs, crawl, politeness, schedule).
  • Secret holding per-host tokens for publishing to hub(s).
  • ServiceAccount, no special RBAC needed (no k8s API calls).
  • Exec liveness probe (demarkus-agent version returns 0).
  • No Service (agent is outbound-only).
  • Pod annotations for Datadog autodiscovery; log fields documented for mint/crawl events.

Acceptance: helm install agent ./deploy/helm/demarkus-agent --set config.seeds[0]=mark://team-a:6309 ... produces a pod that crawls on schedule and publishes to the configured hub. Pod restart resumes from state PVC (optional; can also be ephemeral).

6.1 — demarkus-server Helm chart

deploy/helm/demarkus-server/. Production-grade.

Workload:

  • StatefulSet, 1 replica (multi-replica is Phase 7).
  • volumeClaimTemplates — each world owns its PVC. Never a shared PVC.
  • Container image bundles demarkus-server + demarkus CLI (for exec probes).
  • Exec liveness + readiness probes against /.well-known/agent-manifest.md.
  • Resource requests/limits with sane defaults, overridable.
  • Service type LoadBalancer, protocol: UDP, port from server.udpPort (default 6309). Annotations for cloud-specific LB type (NLB on AWS, etc.).

Secrets (world namespace):

  • <release>-tokens — TOML of SHA-256 hashes. Server-mounted.
    • Persistence across helm upgrade is non-negotiable. Rendered with helm.sh/resource-policy: keep + helm.sh/hook: pre-install and empty data: {} in the template body. Helm creates the Secret on first install and never overwrites it on subsequent upgrades; broker writes via k8s API are preserved.
    • Bootstrap Job (below) seeds the initial admin token on first install only. The broker manages all subsequent entries at runtime via the k8s API.
    • Without this pattern, every helm upgrade would silently wipe every broker-minted token. This is the single most important Helm correctness property in the chart and must be unit-tested.
    • kubelet propagation: Secret data changes propagate to the mounted file with a delay (default ~60s). Server re-reads via SIGHUP (preferred) or fsnotify; verify which during chart implementation.
  • Broker state (raw tokens are never persisted; email→label mappings and issuance metadata) lives in the broker namespace, not the world namespace. See §6.2 for the broker's issuances Secret. The world's tokens Secret holds only hashes and is the sole source of truth the server reads.

Persistence boundary recap: pod restarts, node failures, control-plane restarts → fine. helm upgrade → fine iff resource-policy: keep. kubectl delete namespace → everything gone (operations doc covers Velero / VolumeSnapshot recovery). helm uninstall without keep annotation → tokens Secret gone; with keep, retained.

Auth + TLS:

  • TLS Secret mounted via volumeMounts; cert/key paths via flags.
  • Optional cert-manager Certificate resource (behind a flag) requesting *.<root> from a configured ClusterIssuer.

Bootstrap Job:

  • Mints initial admin token on first install via the shared mint library (tools/internal/token, Slice A).
  • helm.sh/hook: pre-install only — does not run on upgrade.
  • Idempotent guard: reads the tokens Secret; skips if admin label already exists.
  • SIGHUPs pod after writing (no-op on first install before pod is up).

Observability:

  • Pod annotations for Datadog autodiscovery.
  • Reference configs for OTel Collector, Vector, Fluent Bit in deploy/observability/.
  • slog output already structured; no chart-side instrumentation needed.

RBAC:

  • Bootstrap Job SA with get/update on the named Secret, get/list/create on pods/exec for SIGHUP. Namespace-scoped Role, not ClusterRole.

Tests:

  • helm-unittest for templates, including a test that asserts resource-policy: keep is present on the tokens Secret and that the rendered Secret has empty data: {}.
  • Kind-based integration test in CI: install chart → exec into pod → verify health → publish via CLI → verify version increments → helm upgrade with a changed value → re-verify token-based auth still works (regression guard against the upgrade-wipe footgun).

6.2 — demarkus-broker binary (tools/demarkus-broker/)

Full feature set, no skip-DELETE corner-cuts.

Role & topology

  • Issuance authority, not a request proxy. Broker sits on the demarkus login path; clients then talk to world servers directly carrying the raw token. World servers stay identity-blind; broker never sees mark:// requests. Capability model preserved.
  • One broker per universe (cluster) by default. Single OIDC client registration, single worlds: list in values, single issuance state Secret. Multiple brokers only when (a) multiple OIDC providers must coexist, (b) hard tenant isolation between orgs sharing a cluster, or (c) Phase-7 geo split. N>1 is supported but not the common case.
  • Broker SA holds a namespace-scoped Role + RoleBinding in each world's namespace, with get/patch limited to that world's tokens Secret only. No ClusterRole. Blast radius bounded to the token Secrets it's explicitly granted.

State

Two distinct Kubernetes Secrets, never merged:

Secret Lives in Contents Reader
<world>-tokens each world's namespace TOML: [tokens.<label>] hash=, paths=, operations=, expires= demarkus-server
<broker>-issuances broker namespace JSON: label → {email, world, paths, operations, issued_at, expires} + secondary index email → [labels] broker only

World servers see only hashes. Email and identity live in broker state alone. Both Secrets follow the §6.1 resource-policy: keep pattern — broker state must also survive helm upgrade of the broker chart.

Labels

Opaque IDs (e.g., usr_<random8>). Not email-derived. Reasons:

  • Sanitizing emails (dots, plus signs, IDN) into TOML keys is fragile.
  • Token rotation yields a new label each time; opaque IDs reflect that naturally.
  • "Revoke everything for fredrick@x" becomes a broker-state index lookup, not a label string scan.

Per-world authorization (values schema)

worlds:
  - name: team-a
    namespace: team-a
    tokensSecret: team-a-tokens
    allow:
      domains: ["nesto.ca"]
      groups: ["engineering"]      # OIDC `groups` claim
    defaultToken:
      paths: ["/team-a/*"]
      operations: ["read", "publish"]
      expiresAfter: 24h

On demarkus login, broker evaluates the OIDC identity against every world in its config and mints one token per world the user qualifies for. Client may request narrower scope; never broader.

HashToken provenance

Promoted to protocol/ in Slice A. Broker imports protocol.HashToken to ensure byte-identical hashes to what world servers read from tokens.toml. The hash format sha256-<hex> is treated as a protocol contract, single source of truth.

Revocation

Three triggers, one cleanup index (the issuances Secret):

  1. User-initiated (demarkus token revoke <label> or rotate):

    • DELETE /tokens/:label carrying OIDC ID token.
    • Broker confirms entry.email == claims.email (owner check — users can only revoke their own).
    • Patches the world's tokens Secret to drop the label.
    • Drops the entry from issuances Secret.
    • SIGHUPs world server pod(s) so tokens.toml is re-read. (Or relies on server fsnotify if present — verify during implementation.)

  2. Expiry sweeper (broker in-process ticker, default every 5 min, leader-elected via Lease so only one replica sweeps):

    • Queries issuances where expires < now.
    • For each: patch the world's tokens Secret + drop from issuances + SIGHUP.

  3. Identity lifecycle (user leaves the org):

    • Default: short-lived tokens. defaultToken.expiresAfter of 24h means stale tokens age out within a day. Next demarkus login reruns OIDC; if the user's IdP account is disabled, login fails. No broker→IdP coupling needed. Same approach as AWS STS / gcloud auth login.
    • Backlog: SCIM lifecycle webhook (POST /scim/v2/Users/:id) for enterprise IdPs that push deprovisioning events (Okta, Entra). Optional add when a customer asks.
    • Periodic IdP re-validation rejected — worse trade-off than either above (couples broker to IdP rate limits, slower than SCIM, less simple than short-lived).

Cleanup edge cases

  • Orphan in tokens.toml (admin-minted via legacy demarkus-token CLI): broker never claims it. Admins manage via CLI. Broker API only operates on labels it minted.
  • Orphan in issuances Secret (admin hand-deleted from world tokens Secret): expiry sweeper detects drift on each pass and prunes broker-state entries whose label no longer exists in the world's TOML.
  • In-flight request after revoke: a raw token already authenticated on a live QUIC connection completes its current request. Property of the capability + connection-reuse model. Documented; not fixable without core server changes.

Routes (HTTP, behind Ingress)

  • POST /auth/login — OIDC redirect entry point.
  • GET /auth/callback — OIDC callback; mints tokens for every qualifying world; returns JSON {world → raw_token}. One-time response.
  • GET /tokens — list caller's tokens from broker state (labels + metadata only, never raw tokens).
  • DELETE /tokens/:label — revoke caller-owned token.
  • POST /tokens/:label/rotate — revoke + mint with same scope; returns new raw token.
  • GET /healthz, GET /readyz.

OIDC providers

Provider behind a Verifier interface. Initial: Google (via coreos/go-oidc or google.golang.org/api/idtoken). Follow-ons: Okta, Entra ID, Auth0 — config only. Group-claim availability varies per provider; doc page per provider explains required claim configuration (e.g., Entra needs the groups optional claim enabled; some providers require a userinfo call to fetch groups not in the ID token).

HA

Multi-replica. Issuance writes to k8s Secrets use resourceVersion optimistic concurrency with retry-on-conflict. Sweeper uses a coordination.k8s.io/Lease for leader election so only one replica runs the periodic loop.

Tests

  • Mint flow: table-driven, mock OIDC verifier, fake k8s clientset, assert tokens Secret + issuances Secret both updated atomically (best-effort — k8s API is two calls).
  • Sweeper: fake clock + fake clientset.
  • RBAC permission-denied path explicitly covered (broker SA without patch on a world's Secret → mint fails cleanly, not partial state).
  • Drift pruning: orphan-in-issuances case covered by sweeper test.
  • Owner check on DELETE — different OIDC subject than the entry → 403.

6.3 — demarkus-broker Helm chart

Multi-replica HA. As previously specified.

6.4 — Universe topology examples

ApplicationSet + Kustomize overlay. As previously specified.

6.5 — Observability recipes

Per-backend configs in deploy/observability/. As previously specified.

6.6 — Documentation suite

/deployment/*.md + per-chart READMEs. As previously specified.

6.7 — Release pipeline

GHCR images + OCI charts. Cosign deferred to backlog.

Sequencing

  1. Slice A (in flight) — token-mint library refactor; HashToken to protocol/; demarkus-token + demarkus-publish to tools/; make all and pre-commit.sh extended to include tools.
  2. 6.0 chart — quick win since binary is done.
  3. 6.1 server chart — main piece.
  4. 6.2 broker binary — parallel with 6.1, design pinned (see above).
  5. 6.3 broker chart — after 6.2 testable.
  6. 6.4 topology examples.
  7. 6.5 observability recipes.
  8. 6.6 docs — incremental throughout.
  9. 6.7 release pipeline — final.

Rough effort: 2–4 weeks of focused work (down from 4–8 in v3 — agent binary work eliminated).

Backlog (deferred, easy to add later)

  • Cosign signing of images + chart releases. Half-day CI add when wanted.
  • Latency log-enrichment (duration_ms field on request slog lines). Tiny additive change. Defer until trials show it's needed.
  • SCIM lifecycle webhook on the broker (POST /scim/v2/Users/:id) for enterprise IdPs that push deprovisioning events. Adds responsiveness beyond what short-lived tokens give.

Risks

  • Observability via logs ceiling. If customers want signals not derivable from current slog (latency, internal state like version counts or PVC fullness), we hit a wall. Mitigation: log enrichment is a small additive change; internal-state metrics derivable by a cluster-side sidecar that calls LIST periodically.
  • Broker secret-write blast radius. Holds k8s API creds across world namespaces. Mitigated by namespace-scoped Roles (one per world), audit log, optional NetworkPolicy. Per-world Role is non-negotiable — never ClusterRole.
  • OIDC provider coupling. First impl is Google; structure so second provider is a one-day add.
  • Token revocation in-flight latency. SIGHUP reloads tokens.toml, but a request already authenticated on a live QUIC connection completes. Property of model.
  • Helm upgrade wiping tokens. Classic templating footgun: rendering a Secret on every upgrade overwrites broker-minted content. Mitigated by helm.sh/resource-policy: keep + pre-install hook on the world tokens Secret (and the broker issuances Secret). Regression-tested in CI via the kind integration test.
  • Chart proliferation. Three charts + examples + dashboards. Mitigate with shared common-labels templates.
  • Trial scope creep. First customer is path-B. If feedback pulls scope back to "demo slice" mid-build, decide explicitly.

Status

Plan v6, 2026-05-11. §6.1 secrets section expanded to make the helm upgrade persistence pattern explicit (resource-policy: keep + pre-install hook + empty data: {} + regression-tested in CI). The stale <release>-token-values Secret line removed — broker state lives in the broker namespace per §6.2, never co-located with the world's tokens Secret. §6.2 broker design pinned during Slice A planning (topology = one broker per universe; state split; opaque labels; per-world authorization; revocation via DELETE + expiry sweeper + short-lived tokens; SCIM in backlog). Slice A in flight: token-mint library refactor, HashToken promotion to protocol/, demarkus-token/demarkus-publish move to tools/.

trail
  1. soul.demarkus.io v6