# Plan: Universe Deployment

Ship the enterprise-grade deployment for demarkus. The deliverable *is* the deployment: a customer's ops team installs a Helm chart, runs a broker, runs an agent, and has a federated demarkus universe their org can use. Any company evaluating demarkus (their internal "POC") installs the same product an established customer runs in production. There is no separate "POC slice" — the slice mentality is rejected. We build it once, right, and customers trial the real thing.

## Goal

Deliver a complete, supportable, production-grade Kubernetes deployment package for demarkus, including:

1. A Helm chart for `demarkus-server` (one world).
2. A Helm chart for `demarkus-broker` (OIDC token issuance + revocation).
3. A Helm chart for `demarkus-agent` (hub aggregator, crawl-and-index).
4. Reference topology examples (Argo CD `ApplicationSet`, Kustomize overlay).
5. Backend-agnostic observability — structured slog emission + reference configs for Datadog, OTel Collector, Vector, Fluent Bit, Grafana Alloy.
6. Customer-facing documentation — installation, security/threat model, operations, upgrade path, per-provider OIDC setup, observability recipes.
7. A release pipeline producing images and chart releases consumable from GHCR.

The same artifacts power first-customer trial and steady-state operations.

## Non-Goals (Phase 7+ territory)

- Multi-replica worlds with shared storage (RWX / object backend).
- Cross-cluster universe federation.
- Operator with a `World` CRD.
- Hosted / managed SaaS.
- Non-markdown content.

## Constraints

- **No core protocol/server changes.** Period. Per prior precedent (Claude Code plugin, Obsidian plugin, `feedback_plugin_scope.md`). Observability is achieved by log derivation in a collector, not by adding `/metrics` or OTel SDK calls to the server.
- Capability-based auth model is non-negotiable. The server never learns identity — only labels.
- Markdown-only scope is non-negotiable.

## Decisions (resolved during planning)

- **Health probes use `exec`, not HTTP.** Both charts ship liveness/readiness `exec` probes that invoke the `demarkus` CLI fetching `/.well-known/agent-manifest.md` (always public per `/architecture.md`). No core change. Same pattern as `redis-cli ping` / `pg_isready`.
- **UDP port is a values knob.** Default `6309`, override via `server.udpPort`. Documented option: switch to `443` for VPN/middlebox-hostile networks (Cloudflare Warp Zero Trust, corp firewalls that filter non-standard UDP). Protocol default stays `6309`.
- **Wildcard TLS via cert-manager DNS-01.** Single `*.<root>` cert covers all worlds + broker hostnames. HTTP-01 cannot work for worlds (no HTTP); DNS-01 is the standard path for QUIC services.
- **DNS topology.** One A record per world + one for the broker, all under the same wildcard zone. Each world exposed by `Service: LoadBalancer` `protocol: UDP`; broker by standard Ingress (HTTPS).
- **Hub aggregator is the existing `demarkus-agent`** at `client/cmd/demarkus-agent/`. Already implements `crawl` + `daemon` subcommands with TOML config, multi-worker crawl, per-host token auth, per-server + aggregated index publishing. Verified end-to-end on 2026-05-11. **Phase 5 is no longer a blocking prerequisite.**
- **`demarkus-agent` lives in `client/cmd/`, not `tools/`.** It's a protocol client (uses `fedcrawl`, `fetch`, `tokens`, `links`), not a utility. Memory rule sharpened: `client/cmd/` = protocol clients (CLI, TUI, MCP, agent); `tools/` = utilities (token, publish, sync, broker).
- **Multi-OIDC.** Broker speaks generic OIDC, not Google-specific. Provider behind a `Verifier` interface. Google validated first; Okta, Entra ID, Auth0 follow with config only.
- **Workload Identity (GKE) as a broker option.** Values flag annotates the broker ServiceAccount with the GSA mapping. Off by default; on for GKE customers wanting no static SA keys. Not applicable to world servers.
- **Broker is HA.** Multi-replica with `resourceVersion` optimistic concurrency on Secret writes, retry on conflict.
- **Observability is log-derived, backend-agnostic.** demarkus already emits structured slog. Charts ship Datadog autodiscovery annotations + reference configs for OTel Collector, Vector, Fluent Bit, Grafana Alloy. Customer's SRE picks the agent/backend. Latency histograms deferred — depends on what current slog includes per request; revisit during trials if needed.
- **Backup/DR is documented, not built.** The chart deliberately does not run backup CronJobs. Operations doc covers Velero, VolumeSnapshot, and `demarkus-agent sync` as DR options.
- **Image hosting:** `ghcr.io/latebit-io/{demarkus-server,demarkus-broker,demarkus-agent}`.
- **Chart registry:** OCI charts in GHCR.
- **Cosign signing:** deferred to backlog. Half-day CI add when wanted.

## Open Questions

1. **First customer trial.** Nesto (`*.library.nesto.ca`) is path-B — trial waits for product. Trial runbook lands at `/trials/nesto.md` when scoping starts.

## Repository Layout

```
deploy/
  helm/
    demarkus-server/         # one-world chart
    demarkus-broker/         # OIDC token broker chart
    demarkus-agent/          # crawl/index agent chart
  k8s/
    examples/
      applicationset.yaml    # Argo CD ApplicationSet over a worlds: list
      kustomize-overlay/     # Kustomize alternative
  observability/
    datadog/                 # autodiscovery annotations + dashboard JSON
    otel-collector/          # collector config recipes
    vector/                  # vector config recipes
    fluent-bit/              # fluent-bit parser + filter recipes
  scripts/                   # operator helpers (cert pre-check, MTU probe, etc.)

client/cmd/
  demarkus-agent/            # protocol client — federation crawler (existing)

tools/
  demarkus-broker/           # broker binary (monorepo until API stabilizes)
  internal/token/            # shared mint library, extracted from demarkus-token
```

## Sub-Phases

### 6.0 — `demarkus-agent` verified + chart ✓ binary, ⏳ chart

**Binary status (2026-05-11):** existing `demarkus-agent` verified end-to-end against a 3-server smoke test (2 team worlds + 1 hub). Crawls, builds aggregated and per-server indexes, publishes to hub. Two bugs fixed during verification:

- `fedcrawl/crawl.go publishIndex` accepted only `ok` status; first publish returns `created`, surfaced as a misleading warning despite the publish succeeding. Now accepts both.
- `publishIndex` always used `expected_version=0` (create-only), causing every re-publish in daemon mode to fail with `conflict`. Now uses `-1` (no check) for idempotent hub re-publish; server's no-op-on-duplicate-content prevents version churn.
- `Makefile` did not build `demarkus-agent`; the agent had to be hand-built. Now builds with `make client`.

Tests in `client/internal/fedcrawl/crawl_test.go` cover all three: create + re-publish + status acceptance + per-server / aggregated modes. `bash pre-commit.sh` clean.

**Remaining 6.0 work — agent Helm chart** at `deploy/helm/demarkus-agent/`:
- `Deployment` (not StatefulSet — agent is stateless modulo state file; recoverable from next crawl).
- ConfigMap holding the TOML agent config (`seeds`, `hubs`, `crawl`, `politeness`, `schedule`).
- Secret holding per-host tokens for publishing to hub(s).
- ServiceAccount, no special RBAC needed (no k8s API calls).
- Exec liveness probe (`demarkus-agent version` returns 0).
- No `Service` (agent is outbound-only).
- Pod annotations for Datadog autodiscovery; log fields documented for `mint`/`crawl` events.

Acceptance: `helm install agent ./deploy/helm/demarkus-agent --set config.seeds[0]=mark://team-a:6309 ...` produces a pod that crawls on schedule and publishes to the configured hub. Pod restart resumes from state PVC (optional; can also be ephemeral).

### 6.1 — `demarkus-server` Helm chart

`deploy/helm/demarkus-server/`. Production-grade.

Workload:
- `StatefulSet`, 1 replica (multi-replica is Phase 7).
- `volumeClaimTemplates` — each world owns its PVC. Never a shared PVC.
- Container image bundles `demarkus-server` + `demarkus` CLI (for exec probes).
- Exec liveness + readiness probes against `/.well-known/agent-manifest.md`.
- Resource requests/limits with sane defaults, overridable.
- `Service` type `LoadBalancer`, `protocol: UDP`, port from `server.udpPort` (default 6309). Annotations for cloud-specific LB type (NLB on AWS, etc.).

Secrets:
- `<release>-tokens` — TOML, SHA-256 hashes. Server-mounted.
- `<release>-token-values` — raw tokens. Broker-only. Kept separate so server never mounts raw secrets.

Auth + TLS:
- TLS Secret mounted via `volumeMounts`; cert/key paths via flags.
- Optional `cert-manager` `Certificate` resource (behind a flag) requesting `*.<root>` from a configured `ClusterIssuer`.

Bootstrap Job:
- Mints initial `admin` token on first install via the shared mint library.
- Idempotent — skips on re-install if `admin` label exists.
- SIGHUPs pod after writing.

Observability:
- Pod annotations for Datadog autodiscovery.
- Reference configs for OTel Collector, Vector, Fluent Bit in `deploy/observability/`.
- slog output already structured; no chart-side instrumentation needed.

RBAC:
- Bootstrap Job SA with `get/update` on the named Secret, `get/list/create` on `pods/exec` for SIGHUP. Namespace-scoped `Role`, not `ClusterRole`.

Tests:
- `helm-unittest` for templates.
- Kind-based integration test in CI: install chart → exec into pod → verify health → publish via CLI → verify version increments.

### 6.2 — `demarkus-broker` binary (`tools/demarkus-broker/`)

Full feature set, no skip-DELETE corner-cuts. Routes, implementation, tests as previously specified.

### 6.3 — `demarkus-broker` Helm chart

Multi-replica HA. As previously specified.

### 6.4 — Universe topology examples

ApplicationSet + Kustomize overlay. As previously specified.

### 6.5 — Observability recipes

Per-backend configs in `deploy/observability/`. As previously specified.

### 6.6 — Documentation suite

`/deployment/*.md` + per-chart READMEs. As previously specified.

### 6.7 — Release pipeline

GHCR images + OCI charts. Cosign deferred to backlog.

## Sequencing

1. **6.0 chart** — quick win since binary is done.
2. **6.1 server chart** — main piece.
3. **6.2 broker binary** — parallel with 6.1.
4. **6.3 broker chart** — after 6.2 testable.
5. **6.4 topology examples**.
6. **6.5 observability recipes**.
7. **6.6 docs** — incremental throughout.
8. **6.7 release pipeline** — final.

Rough effort: 2–4 weeks of focused work (down from 4–8 in v3 — agent binary work eliminated).

## Backlog (deferred, easy to add later)

- **Cosign signing** of images + chart releases. Half-day CI add when wanted.
- **Latency log-enrichment** (`duration_ms` field on request slog lines). Tiny additive change. Defer until trials show it's needed.

## Risks

- **Observability via logs ceiling.** If customers want signals not derivable from current slog (latency, internal state like version counts or PVC fullness), we hit a wall. Mitigation: log enrichment is a small additive change; internal-state metrics derivable by a cluster-side sidecar that calls `LIST` periodically.
- **Broker secret-write blast radius.** Holds k8s API creds across world namespaces. Mitigated by namespace-scoped `Role`s, audit log, optional `NetworkPolicy`.
- **OIDC provider coupling.** First impl is Google; structure so second provider is a one-day add.
- **Token revocation latency.** SIGHUP reloads `tokens.toml`, but in-flight requests with revoked token complete. Property of model.
- **Chart proliferation.** Three charts + examples + dashboards. Mitigate with shared common-labels templates.
- **Trial scope creep.** First customer is path-B. If feedback pulls scope back to "demo slice" mid-build, decide explicitly.

## Status

Plan v4, 2026-05-11. Phase 5 prerequisite cleared — `demarkus-agent` verified and patched. 6.0 reduced to chart-only. Ready to start: 6.0 agent chart, then 6.1 server chart. See [/journal.md](/journal.md) for verification details.