soul.demarkus.io/plans/universe-deployment.md/v5 draft reader meta

Plan: Universe Deployment

Ship the enterprise-grade deployment for demarkus. The deliverable is the deployment: a customer's ops team installs a Helm chart, runs a broker, runs an agent, and has a federated demarkus universe their org can use. Any company evaluating demarkus (their internal "POC") installs the same product an established customer runs in production. There is no separate "POC slice" — the slice mentality is rejected. We build it once, right, and customers trial the real thing.

Goal

Deliver a complete, supportable, production-grade Kubernetes deployment package for demarkus, including:

  1. A Helm chart for demarkus-server (one world).
  2. A Helm chart for demarkus-broker (OIDC token issuance + revocation).
  3. A Helm chart for demarkus-agent (hub aggregator, crawl-and-index).
  4. Reference topology examples (Argo CD ApplicationSet, Kustomize overlay).
  5. Backend-agnostic observability — structured slog emission + reference configs for Datadog, OTel Collector, Vector, Fluent Bit, Grafana Alloy.
  6. Customer-facing documentation — installation, security/threat model, operations, upgrade path, per-provider OIDC setup, observability recipes.
  7. A release pipeline producing images and chart releases consumable from GHCR.

The same artifacts power first-customer trial and steady-state operations.

Non-Goals (Phase 7+ territory)

  • Multi-replica worlds with shared storage (RWX / object backend).
  • Cross-cluster universe federation.
  • Operator with a World CRD.
  • Hosted / managed SaaS.
  • Non-markdown content.

Constraints

  • No core protocol/server changes. Period. Per prior precedent (Claude Code plugin, Obsidian plugin, feedback_plugin_scope.md). Observability is achieved by log derivation in a collector, not by adding /metrics or OTel SDK calls to the server. Note: lifting HashToken from server/internal/auth/ to protocol/ (Slice A) is an additive helper relocation, not a behavioral change, and is considered within the spirit of this rule.
  • Capability-based auth model is non-negotiable. The server never learns identity — only labels.
  • Markdown-only scope is non-negotiable.

Decisions (resolved during planning)

  • Health probes use exec, not HTTP. Both charts ship liveness/readiness exec probes that invoke the demarkus CLI fetching /.well-known/agent-manifest.md (always public per /architecture.md). No core change. Same pattern as redis-cli ping / pg_isready.
  • UDP port is a values knob. Default 6309, override via server.udpPort. Documented option: switch to 443 for VPN/middlebox-hostile networks (Cloudflare Warp Zero Trust, corp firewalls that filter non-standard UDP). Protocol default stays 6309.
  • Wildcard TLS via cert-manager DNS-01. Single *.<root> cert covers all worlds + broker hostnames. HTTP-01 cannot work for worlds (no HTTP); DNS-01 is the standard path for QUIC services.
  • DNS topology. One A record per world + one for the broker, all under the same wildcard zone. Each world exposed by Service: LoadBalancer protocol: UDP; broker by standard Ingress (HTTPS).
  • Hub aggregator is the existing demarkus-agent at client/cmd/demarkus-agent/. Already implements crawl + daemon subcommands with TOML config, multi-worker crawl, per-host token auth, per-server + aggregated index publishing. Verified end-to-end on 2026-05-11. Phase 5 is no longer a blocking prerequisite.
  • demarkus-agent lives in client/cmd/, not tools/. It's a protocol client (uses fedcrawl, fetch, tokens, links), not a utility. Memory rule sharpened: client/cmd/ = protocol clients (CLI, TUI, MCP, agent); tools/ = utilities (token, publish, sync, broker).
  • Multi-OIDC. Broker speaks generic OIDC, not Google-specific. Provider behind a Verifier interface. Google validated first; Okta, Entra ID, Auth0 follow with config only.
  • Workload Identity (GKE) as a broker option. Values flag annotates the broker ServiceAccount with the GSA mapping. Off by default; on for GKE customers wanting no static SA keys. Not applicable to world servers.
  • Broker is HA. Multi-replica with resourceVersion optimistic concurrency on Secret writes, retry on conflict.
  • Observability is log-derived, backend-agnostic. demarkus already emits structured slog. Charts ship Datadog autodiscovery annotations + reference configs for OTel Collector, Vector, Fluent Bit, Grafana Alloy. Customer's SRE picks the agent/backend. Latency histograms deferred — depends on what current slog includes per request; revisit during trials if needed.
  • Backup/DR is documented, not built. The chart deliberately does not run backup CronJobs. Operations doc covers Velero, VolumeSnapshot, and demarkus-agent sync as DR options.
  • Image hosting: ghcr.io/latebit-io/{demarkus-server,demarkus-broker,demarkus-agent}.
  • Chart registry: OCI charts in GHCR.
  • Cosign signing: deferred to backlog. Half-day CI add when wanted.

Open Questions

  1. First customer trial. Nesto (*.library.nesto.ca) is path-B — trial waits for product. Trial runbook lands at /trials/nesto.md when scoping starts.

Repository Layout

deploy/
  helm/
    demarkus-server/         # one-world chart
    demarkus-broker/         # OIDC token broker chart
    demarkus-agent/          # crawl/index agent chart
  k8s/
    examples/
      applicationset.yaml    # Argo CD ApplicationSet over a worlds: list
      kustomize-overlay/     # Kustomize alternative
  observability/
    datadog/                 # autodiscovery annotations + dashboard JSON
    otel-collector/          # collector config recipes
    vector/                  # vector config recipes
    fluent-bit/              # fluent-bit parser + filter recipes
  scripts/                   # operator helpers (cert pre-check, MTU probe, etc.)

client/cmd/
  demarkus-agent/            # protocol client — federation crawler (existing)

tools/
  demarkus-token/            # token mint CLI (moved from server/cmd/ in Slice A)
  demarkus-publish/          # publish CLI (moved from server/cmd/ in Slice A)
  demarkus-broker/           # broker binary (monorepo until API stabilizes)
  internal/token/            # shared mint library, extracted from demarkus-token

protocol/
  auth.go                    # HashToken (promoted from server/internal/auth/ in Slice A)

Sub-Phases

6.0 — demarkus-agent verified + chart ✓ binary, ⏳ chart

Binary status (2026-05-11): existing demarkus-agent verified end-to-end against a 3-server smoke test (2 team worlds + 1 hub). Crawls, builds aggregated and per-server indexes, publishes to hub. Two bugs fixed during verification:

  • fedcrawl/crawl.go publishIndex accepted only ok status; first publish returns created, surfaced as a misleading warning despite the publish succeeding. Now accepts both.
  • publishIndex always used expected_version=0 (create-only), causing every re-publish in daemon mode to fail with conflict. Now uses -1 (no check) for idempotent hub re-publish; server's no-op-on-duplicate-content prevents version churn.
  • Makefile did not build demarkus-agent; the agent had to be hand-built. Now builds with make client.

Tests in client/internal/fedcrawl/crawl_test.go cover all three: create + re-publish + status acceptance + per-server / aggregated modes. bash pre-commit.sh clean.

Remaining 6.0 work — agent Helm chart at deploy/helm/demarkus-agent/:

  • Deployment (not StatefulSet — agent is stateless modulo state file; recoverable from next crawl).
  • ConfigMap holding the TOML agent config (seeds, hubs, crawl, politeness, schedule).
  • Secret holding per-host tokens for publishing to hub(s).
  • ServiceAccount, no special RBAC needed (no k8s API calls).
  • Exec liveness probe (demarkus-agent version returns 0).
  • No Service (agent is outbound-only).
  • Pod annotations for Datadog autodiscovery; log fields documented for mint/crawl events.

Acceptance: helm install agent ./deploy/helm/demarkus-agent --set config.seeds[0]=mark://team-a:6309 ... produces a pod that crawls on schedule and publishes to the configured hub. Pod restart resumes from state PVC (optional; can also be ephemeral).

6.1 — demarkus-server Helm chart

deploy/helm/demarkus-server/. Production-grade.

Workload:

  • StatefulSet, 1 replica (multi-replica is Phase 7).
  • volumeClaimTemplates — each world owns its PVC. Never a shared PVC.
  • Container image bundles demarkus-server + demarkus CLI (for exec probes).
  • Exec liveness + readiness probes against /.well-known/agent-manifest.md.
  • Resource requests/limits with sane defaults, overridable.
  • Service type LoadBalancer, protocol: UDP, port from server.udpPort (default 6309). Annotations for cloud-specific LB type (NLB on AWS, etc.).

Secrets:

  • <release>-tokens — TOML, SHA-256 hashes. Server-mounted.
  • <release>-token-values — raw tokens. Broker-only. Kept separate so server never mounts raw secrets.

Auth + TLS:

  • TLS Secret mounted via volumeMounts; cert/key paths via flags.
  • Optional cert-manager Certificate resource (behind a flag) requesting *.<root> from a configured ClusterIssuer.

Bootstrap Job:

  • Mints initial admin token on first install via the shared mint library.
  • Idempotent — skips on re-install if admin label exists.
  • SIGHUPs pod after writing.

Observability:

  • Pod annotations for Datadog autodiscovery.
  • Reference configs for OTel Collector, Vector, Fluent Bit in deploy/observability/.
  • slog output already structured; no chart-side instrumentation needed.

RBAC:

  • Bootstrap Job SA with get/update on the named Secret, get/list/create on pods/exec for SIGHUP. Namespace-scoped Role, not ClusterRole.

Tests:

  • helm-unittest for templates.
  • Kind-based integration test in CI: install chart → exec into pod → verify health → publish via CLI → verify version increments.

6.2 — demarkus-broker binary (tools/demarkus-broker/)

Full feature set, no skip-DELETE corner-cuts.

Role & topology

  • Issuance authority, not a request proxy. Broker sits on the demarkus login path; clients then talk to world servers directly carrying the raw token. World servers stay identity-blind; broker never sees mark:// requests. Capability model preserved.
  • One broker per universe (cluster) by default. Single OIDC client registration, single worlds: list in values, single issuance state Secret. Multiple brokers only when (a) multiple OIDC providers must coexist, (b) hard tenant isolation between orgs sharing a cluster, or (c) Phase-7 geo split. N>1 is supported but not the common case.
  • Broker SA holds a namespace-scoped Role + RoleBinding in each world's namespace, with get/patch limited to that world's tokens Secret only. No ClusterRole. Blast radius bounded to the token Secrets it's explicitly granted.

State

Two distinct Kubernetes Secrets, never merged:

Secret Lives in Contents Reader
<world>-tokens each world's namespace TOML: [tokens.<label>] hash=, paths=, operations=, expires= demarkus-server
<broker>-issuances broker namespace JSON: label → {email, world, paths, operations, issued_at, expires} + secondary index email → [labels] broker only

World servers see only hashes. Email and identity live in broker state alone.

Labels

Opaque IDs (e.g., usr_<random8>). Not email-derived. Reasons:

  • Sanitizing emails (dots, plus signs, IDN) into TOML keys is fragile.
  • Token rotation yields a new label each time; opaque IDs reflect that naturally.
  • "Revoke everything for fredrick@x" becomes a broker-state index lookup, not a label string scan.

Per-world authorization (values schema)

worlds:
  - name: team-a
    namespace: team-a
    tokensSecret: team-a-tokens
    allow:
      domains: ["nesto.ca"]
      groups: ["engineering"]      # OIDC `groups` claim
    defaultToken:
      paths: ["/team-a/*"]
      operations: ["read", "publish"]
      expiresAfter: 24h

On demarkus login, broker evaluates the OIDC identity against every world in its config and mints one token per world the user qualifies for. Client may request narrower scope; never broader.

HashToken provenance

Promoted to protocol/ in Slice A. Broker imports protocol.HashToken to ensure byte-identical hashes to what world servers read from tokens.toml. The hash format sha256-<hex> is treated as a protocol contract, single source of truth.

Revocation

Three triggers, one cleanup index (the issuances Secret):

  1. User-initiated (demarkus token revoke <label> or rotate):

    • DELETE /tokens/:label carrying OIDC ID token.
    • Broker confirms entry.email == claims.email (owner check — users can only revoke their own).
    • Patches the world's tokens Secret to drop the label.
    • Drops the entry from issuances Secret.
    • SIGHUPs world server pod(s) so tokens.toml is re-read. (Or relies on server fsnotify if present — verify during implementation.)

  2. Expiry sweeper (broker in-process ticker, default every 5 min, leader-elected via Lease so only one replica sweeps):

    • Queries issuances where expires < now.
    • For each: patch the world's tokens Secret + drop from issuances + SIGHUP.

  3. Identity lifecycle (user leaves the org):

    • Default: short-lived tokens. defaultToken.expiresAfter of 24h means stale tokens age out within a day. Next demarkus login reruns OIDC; if the user's IdP account is disabled, login fails. No broker→IdP coupling needed. Same approach as AWS STS / gcloud auth login.
    • Backlog: SCIM lifecycle webhook (POST /scim/v2/Users/:id) for enterprise IdPs that push deprovisioning events (Okta, Entra). Optional add when a customer asks.
    • Periodic IdP re-validation rejected — worse trade-off than either above (couples broker to IdP rate limits, slower than SCIM, less simple than short-lived).

Cleanup edge cases

  • Orphan in tokens.toml (admin-minted via legacy demarkus-token CLI): broker never claims it. Admins manage via CLI. Broker API only operates on labels it minted.
  • Orphan in issuances Secret (admin hand-deleted from world tokens Secret): expiry sweeper detects drift on each pass and prunes broker-state entries whose label no longer exists in the world's TOML.
  • In-flight request after revoke: a raw token already authenticated on a live QUIC connection completes its current request. Property of the capability + connection-reuse model. Documented; not fixable without core server changes.

Routes (HTTP, behind Ingress)

  • POST /auth/login — OIDC redirect entry point.
  • GET /auth/callback — OIDC callback; mints tokens for every qualifying world; returns JSON {world → raw_token}. One-time response.
  • GET /tokens — list caller's tokens from broker state (labels + metadata only, never raw tokens).
  • DELETE /tokens/:label — revoke caller-owned token.
  • POST /tokens/:label/rotate — revoke + mint with same scope; returns new raw token.
  • GET /healthz, GET /readyz.

OIDC providers

Provider behind a Verifier interface. Initial: Google (via coreos/go-oidc or google.golang.org/api/idtoken). Follow-ons: Okta, Entra ID, Auth0 — config only. Group-claim availability varies per provider; doc page per provider explains required claim configuration (e.g., Entra needs the groups optional claim enabled; some providers require a userinfo call to fetch groups not in the ID token).

HA

Multi-replica. Issuance writes to k8s Secrets use resourceVersion optimistic concurrency with retry-on-conflict. Sweeper uses a coordination.k8s.io/Lease for leader election so only one replica runs the periodic loop.

Tests

  • Mint flow: table-driven, mock OIDC verifier, fake k8s clientset, assert tokens Secret + issuances Secret both updated atomically (best-effort — k8s API is two calls).
  • Sweeper: fake clock + fake clientset.
  • RBAC permission-denied path explicitly covered (broker SA without patch on a world's Secret → mint fails cleanly, not partial state).
  • Drift pruning: orphan-in-issuances case covered by sweeper test.
  • Owner check on DELETE — different OIDC subject than the entry → 403.

6.3 — demarkus-broker Helm chart

Multi-replica HA. As previously specified.

6.4 — Universe topology examples

ApplicationSet + Kustomize overlay. As previously specified.

6.5 — Observability recipes

Per-backend configs in deploy/observability/. As previously specified.

6.6 — Documentation suite

/deployment/*.md + per-chart READMEs. As previously specified.

6.7 — Release pipeline

GHCR images + OCI charts. Cosign deferred to backlog.

Sequencing

  1. Slice A (in flight) — token-mint library refactor; HashToken to protocol/; demarkus-token + demarkus-publish to tools/; make all and pre-commit.sh extended to include tools.
  2. 6.0 chart — quick win since binary is done.
  3. 6.1 server chart — main piece.
  4. 6.2 broker binary — parallel with 6.1, design pinned (see above).
  5. 6.3 broker chart — after 6.2 testable.
  6. 6.4 topology examples.
  7. 6.5 observability recipes.
  8. 6.6 docs — incremental throughout.
  9. 6.7 release pipeline — final.

Rough effort: 2–4 weeks of focused work (down from 4–8 in v3 — agent binary work eliminated).

Backlog (deferred, easy to add later)

  • Cosign signing of images + chart releases. Half-day CI add when wanted.
  • Latency log-enrichment (duration_ms field on request slog lines). Tiny additive change. Defer until trials show it's needed.
  • SCIM lifecycle webhook on the broker (POST /scim/v2/Users/:id) for enterprise IdPs that push deprovisioning events. Adds responsiveness beyond what short-lived tokens give.

Risks

  • Observability via logs ceiling. If customers want signals not derivable from current slog (latency, internal state like version counts or PVC fullness), we hit a wall. Mitigation: log enrichment is a small additive change; internal-state metrics derivable by a cluster-side sidecar that calls LIST periodically.
  • Broker secret-write blast radius. Holds k8s API creds across world namespaces. Mitigated by namespace-scoped Roles (one per world), audit log, optional NetworkPolicy. Per-world Role is non-negotiable — never ClusterRole.
  • OIDC provider coupling. First impl is Google; structure so second provider is a one-day add.
  • Token revocation in-flight latency. SIGHUP reloads tokens.toml, but a request already authenticated on a live QUIC connection completes. Property of model.
  • Chart proliferation. Three charts + examples + dashboards. Mitigate with shared common-labels templates.
  • Trial scope creep. First customer is path-B. If feedback pulls scope back to "demo slice" mid-build, decide explicitly.

Status

Plan v5, 2026-05-11. §6.2 expanded with broker design pinned during Slice A planning (topology = one broker per universe; state split between world tokens Secret and broker issuances Secret; opaque labels with email mapping in broker state only; per-world authorization in values; revocation via user-initiated DELETE + expiry sweeper + short-lived tokens as the lifecycle answer; SCIM in backlog). Phase 5 prerequisite cleared. Slice A in flight: token-mint library refactor, HashToken promotion to protocol/, demarkus-token/demarkus-publish move to tools/.

trail
  1. soul.demarkus.io v5