# Plan: Universe Deployment

Ship the enterprise-grade deployment for demarkus. The deliverable *is* the deployment: a customer's ops team installs a Helm chart, runs a broker, runs an agent, and has a federated demarkus universe their org can use. Any company evaluating demarkus (their internal "POC") installs the same product an established customer runs in production. There is no separate "POC slice" — the slice mentality is rejected. We build it once, right, and customers trial the real thing.

## Goal

Deliver a complete, supportable, production-grade Kubernetes deployment package for demarkus, including:

1. A Helm chart for `demarkus-server` (one world).
2. A Helm chart for `demarkus-broker` (OIDC token issuance + revocation).
3. A Helm chart for `demarkus-agent` (hub aggregator, crawl-and-index).
4. Reference topology examples (Argo CD `ApplicationSet`, Kustomize overlay).
5. Observability — Prometheus metrics endpoints, `ServiceMonitor` CRDs, reference Grafana dashboards.
6. Customer-facing documentation — installation, security/threat model, operations, upgrade path, per-provider OIDC setup.
7. A release pipeline producing signed images and chart releases consumable from a known registry.

The same artifacts power first-customer trial and steady-state operations.

## Non-Goals (Phase 7+ territory)

- Multi-replica worlds with shared storage (RWX / object backend).
- Cross-cluster universe federation.
- Operator with a `World` CRD.
- Hosted / managed SaaS.
- Non-markdown content.

## Constraints

- **No core protocol/server changes** unless explicitly discussed. Per the prior precedent (Claude Code plugin, Obsidian plugin, `feedback_plugin_scope.md`). One known exception to discuss before landing:
  - Prometheus `/metrics` endpoint on server + broker. Required for enterprise observability; minimal scope; needs explicit go-ahead. See open question 1.
- Capability-based auth model is non-negotiable. The server never learns identity — only labels.
- Markdown-only scope is non-negotiable.

## Decisions (resolved during planning)

- **Health probes use `exec`, not HTTP.** Both charts ship liveness/readiness `exec` probes that invoke the `demarkus` CLI fetching `/.well-known/agent-manifest.md` (always public per `/architecture.md`). No core change. Same pattern as `redis-cli ping` or `pg_isready`.
- **UDP port is a values knob.** Default `6309`, override via `server.udpPort`. Documented option: switch to `443` for VPN/middlebox-hostile networks (Cloudflare Warp Zero Trust, corp firewalls that filter non-standard UDP). Protocol default stays `6309`; the chart just makes the flip cheap.
- **Wildcard TLS via cert-manager DNS-01.** Single `*.<root>` cert covers all worlds + broker hostnames. HTTP-01 cannot work for worlds (no HTTP); DNS-01 is the standard path for QUIC services.
- **DNS topology.** One A record per world + one for the broker, all under the same wildcard zone. Each world exposed by `Service: LoadBalancer` `protocol: UDP`; broker by standard Ingress (HTTPS).
- **Hub aggregator is the real `demarkus-agent`** (Phase 5 binary). Shell-script CronJob was a duct-tape demo idea; the productized form is the agent crawling each team world and publishing an index to the hub. This sequences Phase 5 ahead of Phase 6 hub work.
- **Multi-OIDC.** Broker speaks generic OIDC, not Google-specific. Provider behind a `Verifier` interface. Google validated first; Okta, Entra ID, Auth0 follow with config only.
- **Workload Identity (GKE) as a broker option.** Values flag annotates the broker ServiceAccount with the GSA mapping. Off by default; on for GKE customers wanting no static SA keys. Not applicable to world servers (no GCP API calls).
- **Broker is HA.** Multi-replica with `resourceVersion` optimistic concurrency on every Secret write, retry on conflict. Standard Ingress in front.
- **Observability.** Prometheus `/metrics` on server + broker (core change pending). `ServiceMonitor` CRDs in both charts behind a `metrics.enabled` values flag. Reference Grafana dashboards ship in `deploy/grafana/`.
- **Backup/DR is documented, not built.** The chart deliberately does not run backup CronJobs. Operations doc covers Velero, VolumeSnapshot, and `demarkus-agent sync` as DR options. Operator picks.

## Open Questions

1. **Metrics core change.** Add `/metrics` HTTP endpoint (Prometheus text format) to `demarkus-server` and `demarkus-broker`. ~50 lines per binary. Per the no-core-changes rule this needs explicit go-ahead before landing. Without it, enterprise SRE has no clean way to alert.
2. **Image hosting.** GHCR under `ghcr.io/latebit-io/*` — confirm.
3. **Chart registry.** OCI charts in GHCR (modern idiom) or a static-site chart repo at `charts.demarkus.io` (wider compatibility)?
4. **Release signing.** Cosign-sign images and chart releases? Recommended; ~half day.
5. **First customer trial.** Nesto (`*.library.nesto.ca`) is path-B — trial waits for product. Trial runbook lands at `/trials/nesto.md` once scoping starts.

## Repository Layout

```
deploy/
  helm/
    demarkus-server/         # one-world chart
    demarkus-broker/         # OIDC token broker chart
    demarkus-agent/          # crawl/index agent chart (after Phase 5 agent ships)
  k8s/
    examples/
      applicationset.yaml    # Argo CD ApplicationSet over a worlds: list
      kustomize-overlay/     # Kustomize alternative
  grafana/
    server-dashboard.json
    broker-dashboard.json
  scripts/                   # operator helpers (cert pre-check, MTU probe, etc.)

tools/
  demarkus-broker/           # broker binary (monorepo until API stabilizes, then split)
  demarkus-agent/            # agent binary (Phase 5 work)
  internal/token/            # shared mint library, extracted from demarkus-token
```

Broker stays in monorepo until API stabilizes — same rationale as the Obsidian plugin pre-split: coupled iteration on the chart Secret schema is faster in one repo.

## Sub-Phases

### 6.0 — Prerequisite: `demarkus-agent` core loop (Phase 5)

Before hub aggregation can land, the agent binary must exist. Phase 5 already plans this; what's needed concretely:

- `tools/demarkus-agent crawl` — fetch one or more seed servers, build a hash index, publish to a configured hub document.
- `tools/demarkus-agent daemon` — same loop on a schedule, with conditional fetch (`if-none-match`) for politeness.
- Auth-aware: reads token from `DEMARKUS_AUTH` per server.
- No new core primitives — uses existing FETCH, LIST, PUBLISH, VERSIONS.

Acceptance: `demarkus-agent crawl --seed mark://team-a:6309 --seed mark://team-b:6309 --hub mark://hub:6309 --hub-path /index.md` produces a hub index document linking all crawled paths, on a schedule.

### 6.1 — `demarkus-server` Helm chart

`deploy/helm/demarkus-server/`. Production-grade.

Workload:
- `StatefulSet`, 1 replica (multi-replica is Phase 7).
- `volumeClaimTemplates` — each world owns its PVC. Never a shared PVC.
- Container image bundles `demarkus-server` + `demarkus` CLI (for exec probes).
- Exec liveness + readiness probes against `/.well-known/agent-manifest.md`.
- Resource requests/limits with sane defaults, overridable.
- `Service` type `LoadBalancer`, `protocol: UDP`, port from `server.udpPort` (default 6309). Annotations for cloud-specific LB type (NLB on AWS, etc.).

Secrets:
- `<release>-tokens` — TOML, SHA-256 hashes. Server-mounted.
- `<release>-token-values` — raw tokens. Broker-only. Kept separate so server never mounts raw secrets.

Auth + TLS:
- TLS Secret mounted via `volumeMounts`; cert/key paths via flags.
- Optional `cert-manager` `Certificate` resource (behind a flag) requesting `*.<root>` from a configured `ClusterIssuer`.

Bootstrap Job:
- Mints initial `admin` token on first install via the shared mint library.
- Idempotent — skips on re-install if `admin` label exists.
- SIGHUPs pod after writing.

Observability:
- `ServiceMonitor` template behind `metrics.enabled` (requires server `/metrics`).
- slog output structured for cluster log aggregation; audit-log filtering documented.

RBAC:
- Bootstrap Job SA with `get/update` on the named Secret, `get/list/create` on `pods/exec` for SIGHUP. Namespace-scoped `Role`, not `ClusterRole`.

Tests:
- `helm-unittest` for templates.
- Kind-based integration test in CI: install chart → exec into pod → verify health → publish via CLI → verify version increments.

Acceptance:
- `helm install team-a ./deploy/helm/demarkus-server` produces a healthy world reachable via the LB hostname.
- `kubectl delete pod team-a-0` — content persists.
- `helm upgrade --set tokens.labels[1].name=writer` — adds label, SIGHUPs, no rotation of existing labels.
- `--set readOnly=true` — server rejects writes.
- helm-unittest + integration test pass in CI.

### 6.2 — `demarkus-broker` binary (`tools/demarkus-broker/`)

Full feature set, no skip-DELETE corner-cuts.

Routes:
- `GET /healthz` — liveness.
- `GET /metrics` — Prometheus (requires core change).
- `GET /login?world=<name>` — start OIDC, PKCE, signed state cookie.
- `GET /callback` — verify `id_token`, mint, write Secret, SIGHUP, redirect to install page.
- `GET /me/install` — render install page (token + claude.json snippet). One-time view.
- `POST /tokens` — programmatic mint via OIDC bearer (CI use).
- `DELETE /tokens/{world}/{label}` — revoke. Removes from Secret + SIGHUPs.
- `GET /worlds` — list worlds the caller may mint against (derived from OIDC group/email mapping).

Implementation:
- Go single binary. Refactor `tools/demarkus-token` generate path into `tools/internal/token/` — pure-function library shared by CLI and broker.
- Provider behind `Verifier` interface (`Verify(token) (sub, email, hd, groups, err)`). Google concrete impl first; Okta/Entra follow without code changes.
- Per-namespace `Role`/`RoleBinding`: `get/update` on named `Secrets`, `get/create` on named `pods/exec`. No `ClusterRole`.
- `resourceVersion`-based optimistic concurrency on Secret writes; retry on conflict (HA-safe).
- Audit log to stdout: JSON line per action (`mint`, `revoke`, `denied`). No DB.
- Per-user rate limit on `POST /tokens`.

Tests:
- Unit: Verifier mock, mint lib, Secret-write retry.
- Integration: kind cluster + Dex (lightweight OIDC) to exercise full flow without external Google dependency.

### 6.3 — `demarkus-broker` Helm chart

`deploy/helm/demarkus-broker/`.

- `Deployment`, multi-replica (default 2). HA-safe given resourceVersion retry.
- `Service`, standard `Ingress` (HTTPS), optional cert-manager `Certificate`.
- ConfigMap: world list, OIDC provider config, group/domain → world mapping, allowed paths/ops per group.
- Secret: OIDC client secret.
- ServiceAccount with namespace-scoped `Role`/`RoleBinding` per world namespace.
- Optional Workload Identity annotation (`serviceAccount.workloadIdentity.gsa`) for GKE.
- `ServiceMonitor` behind `metrics.enabled`.
- `PodDisruptionBudget` (minAvailable: 1).
- Optional `NetworkPolicy` restricting egress to k8s API + OIDC issuer + world Services.

Acceptance:
- `helm install broker ./deploy/helm/demarkus-broker` produces a reachable broker.
- Sign in via configured OIDC, mint a token, paste snippet, read/write the target world. <2 minutes end-to-end.
- Revoke via `DELETE` — next request from that token fails within seconds.
- Two replicas under concurrent mint load — no lost labels.

### 6.4 — Universe topology examples (`deploy/k8s/examples/`)

- Argo CD `ApplicationSet` over a `worlds:` list. Three-world reference universe.
- Kustomize overlay alternative for non-Argo clusters.
- README walking an operator from zero to a working universe in <15 minutes.

No new code. Reference manifests + docs.

### 6.5 — Observability

Conditional on the metrics-endpoint core change being approved (open question 1).

- Server metrics: request count by verb, latency histogram, audit counts (publish/append/archive success/failure), version-count gauges.
- Broker metrics: mint/revoke counts by world, OIDC verification failures, Secret-write retries, in-flight requests.
- `ServiceMonitor` templates in both charts.
- Reference Grafana dashboards in `deploy/grafana/` (JSON, importable).
- Operations doc covers wiring into kube-prometheus, GKE Managed Prometheus, etc.

### 6.6 — Documentation suite

Soul:
- `/deployment/installation.md` — chart values reference, step-by-step install.
- `/deployment/security.md` — threat model, RBAC requirements, attack surface, audit log conventions, capability-auth review.
- `/deployment/operations.md` — backup/DR options, observability wiring, upgrade procedure, troubleshooting.
- `/deployment/multi-oidc.md` — provider-by-provider setup (Google, Okta, Entra ID).
- `/trials/<customer>.md` — per-customer trial runbook. Nesto's lands first.

Repo:
- Each chart has its own `README.md` with values reference + quick start.

### 6.7 — Release pipeline

- GoReleaser extended to push container images for `demarkus-server`, `demarkus-broker`, `demarkus-agent` to GHCR (confirm).
- Helm chart releases on tag — OCI charts to GHCR or static-site repo (open question 3).
- Cosign signing (open question 4).
- CI: helm-lint + helm-unittest + kind-based integration tests on every PR touching `deploy/` or the broker/agent binaries.

## Sequencing

Dependency chain matters. Don't start hub aggregation or broker chart before their prerequisites.

1. **Phase 5 agent** (6.0 prerequisite) — must land first.
2. **Metrics core-change decision** — gate item. Without buy-in, 6.5 is hollow.
3. **6.1 server chart** — parallel with broker binary work.
4. **6.2 broker binary** — parallel with chart; needs token-mint lib refactor first.
5. **6.3 broker chart** — after 6.2 is testable end-to-end.
6. **6.5 observability** — wires into both charts; lands once metrics endpoints exist.
7. **6.4 topology examples** — small, after 6.1 + 6.3 stable.
8. **6.6 docs** — incremental, each sub-phase contributes its piece.
9. **6.7 release pipeline** — final hardening; signed and tagged artifacts.

Rough effort: 4–8 weeks of focused work depending on how fast open questions resolve.

## Risks

- **Metrics core change refused.** Observability collapses to log-tail dashboards. Enterprises will push back; expect this to surface in customer security reviews.
- **Broker secret-write blast radius.** Holds k8s API creds across world namespaces. Mitigated by namespace-scoped `Role`s, audit log, optional `NetworkPolicy`, and a sealed-secrets/external-secrets recipe in the operations doc.
- **OIDC provider coupling.** First impl is Google; structure so the second provider is a one-day add. Late abstraction = rewrite risk.
- **Token revocation latency.** SIGHUP reloads `tokens.toml`, but in-flight requests with the revoked token complete. Property of the model. Document.
- **Chart proliferation.** Three charts + topology examples + dashboards. Risk of values drift. Mitigate with shared common-labels templates and CI-validated values schemas.
- **Trial scope creep.** First customer (nesto, path-B). If feedback pulls scope back toward "demo slice" mid-build, decide explicitly — don't drift.

## Status

Plan rewritten 2026-05-11 after framing shift: deliverable is the deployment, not a slice. Awaiting decisions on metrics core change, image hosting, chart registry, and signing before kickoff. `/plans/poc-deployment.md` is superseded and archived.