# Plan: Universe Deployment

Ship the enterprise-grade deployment for demarkus. The deliverable *is* the deployment: a customer's ops team installs a Helm chart, runs a broker, runs an agent, and has a federated demarkus universe their org can use. Any company evaluating demarkus (their internal "POC") installs the same product an established customer runs in production. There is no separate "POC slice" — the slice mentality is rejected. We build it once, right, and customers trial the real thing.

## Goal

Deliver a complete, supportable, production-grade Kubernetes deployment package for demarkus, including:

1. A Helm chart for `demarkus-server` (one world).
2. A Helm chart for `demarkus-broker` (OIDC token issuance + revocation).
3. A Helm chart for `demarkus-agent` (hub aggregator, crawl-and-index).
4. Reference topology examples (Argo CD `ApplicationSet`, Kustomize overlay).
5. Backend-agnostic observability — structured slog emission + reference configs for Datadog, OTel Collector, Vector, Fluent Bit, Grafana Alloy.
6. Customer-facing documentation — installation, security/threat model, operations, upgrade path, per-provider OIDC setup, observability recipes.
7. A release pipeline producing images and chart releases consumable from GHCR.

The same artifacts power first-customer trial and steady-state operations.

## Non-Goals (Phase 7+ territory)

- Multi-replica worlds with shared storage (RWX / object backend).
- Cross-cluster universe federation.
- Operator with a `World` CRD.
- Hosted / managed SaaS.
- Non-markdown content.

## Constraints

- **No core protocol/server changes.** Period. Per the prior precedent (Claude Code plugin, Obsidian plugin, `feedback_plugin_scope.md`). Observability is achieved by log derivation in a collector, not by adding `/metrics` or OTel SDK calls to the server.
- Capability-based auth model is non-negotiable. The server never learns identity — only labels.
- Markdown-only scope is non-negotiable.

## Decisions (resolved during planning)

- **Health probes use `exec`, not HTTP.** Both charts ship liveness/readiness `exec` probes that invoke the `demarkus` CLI fetching `/.well-known/agent-manifest.md` (always public per `/architecture.md`). No core change. Same pattern as `redis-cli ping` / `pg_isready`.
- **UDP port is a values knob.** Default `6309`, override via `server.udpPort`. Documented option: switch to `443` for VPN/middlebox-hostile networks (Cloudflare Warp Zero Trust, corp firewalls that filter non-standard UDP). Protocol default stays `6309`.
- **Wildcard TLS via cert-manager DNS-01.** Single `*.<root>` cert covers all worlds + broker hostnames. HTTP-01 cannot work for worlds (no HTTP); DNS-01 is the standard path for QUIC services.
- **DNS topology.** One A record per world + one for the broker, all under the same wildcard zone. Each world exposed by `Service: LoadBalancer` `protocol: UDP`; broker by standard Ingress (HTTPS).
- **Hub aggregator is the real `demarkus-agent`** (Phase 5 binary). Shell-script CronJob rejected as duct-tape; productized form is the agent crawling team worlds and publishing an index to the hub. Phase 5 sequences ahead of Phase 6 hub work.
- **Multi-OIDC.** Broker speaks generic OIDC, not Google-specific. Provider behind a `Verifier` interface. Google validated first; Okta, Entra ID, Auth0 follow with config only.
- **Workload Identity (GKE) as a broker option.** Values flag annotates the broker ServiceAccount with the GSA mapping. Off by default; on for GKE customers wanting no static SA keys. Not applicable to world servers.
- **Broker is HA.** Multi-replica with `resourceVersion` optimistic concurrency on Secret writes, retry on conflict.
- **Observability is log-derived, backend-agnostic.** demarkus already emits structured slog. The chart ships Datadog autodiscovery annotations + reference configs for OTel Collector, Vector, Fluent Bit, Grafana Alloy. Customer's SRE picks the agent/backend. No demarkus-side metrics emission, no core changes. Audit lines from server (`op`, `path`, `success`, `token_label`) joined with broker lines (`mint`/`revoke`/`denied` with `sub`, `email`, `world`, `label`) reconstruct the full identity→action chain in the customer's tooling. Latency histograms deferred — depends on what current slog includes per request; revisit during trials if needed (smallest mitigation is enriching existing slog calls with `duration_ms`, decision deferred).
- **Backup/DR is documented, not built.** The chart deliberately does not run backup CronJobs. Operations doc covers Velero, VolumeSnapshot, and `demarkus-agent sync` as DR options. Operator picks.
- **Image hosting:** `ghcr.io/latebit-io/{demarkus-server,demarkus-broker,demarkus-agent}`.
- **Chart registry:** OCI charts in GHCR.
- **Cosign signing:** deferred to backlog. Half-day CI add when wanted.

## Open Questions

(None blocking kickoff. The two resolved deferrals — latency log-enrichment and cosign signing — are revisited if/when trials surface a need.)

1. **First customer trial.** Nesto (`*.library.nesto.ca`) is path-B — trial waits for product. Trial runbook lands at `/trials/nesto.md` when scoping starts.

## Repository Layout

```
deploy/
  helm/
    demarkus-server/         # one-world chart
    demarkus-broker/         # OIDC token broker chart
    demarkus-agent/          # crawl/index agent chart (after Phase 5 agent ships)
  k8s/
    examples/
      applicationset.yaml    # Argo CD ApplicationSet over a worlds: list
      kustomize-overlay/     # Kustomize alternative
  observability/
    datadog/                 # autodiscovery annotations + dashboard JSON
    otel-collector/          # collector config recipes
    vector/                  # vector config recipes
    fluent-bit/              # fluent-bit parser + filter recipes
  scripts/                   # operator helpers (cert pre-check, MTU probe, etc.)

tools/
  demarkus-broker/           # broker binary (monorepo until API stabilizes, then split)
  demarkus-agent/            # agent binary (Phase 5 work)
  internal/token/            # shared mint library, extracted from demarkus-token
```

## Sub-Phases

### 6.0 — Prerequisite: `demarkus-agent` core loop (Phase 5)

Before hub aggregation can land, the agent binary must exist. Phase 5 already plans this; what's needed concretely:

- `tools/demarkus-agent crawl` — fetch one or more seed servers, build a hash index, publish to a configured hub document.
- `tools/demarkus-agent daemon` — same loop on a schedule, with conditional fetch (`if-none-match`) for politeness.
- Auth-aware: reads token from `DEMARKUS_AUTH` per server.
- No new core primitives — uses existing FETCH, LIST, PUBLISH, VERSIONS.

Acceptance: `demarkus-agent crawl --seed mark://team-a:6309 --seed mark://team-b:6309 --hub mark://hub:6309 --hub-path /index.md` produces a hub index document linking all crawled paths, on a schedule.

### 6.1 — `demarkus-server` Helm chart

`deploy/helm/demarkus-server/`. Production-grade.

Workload:
- `StatefulSet`, 1 replica (multi-replica is Phase 7).
- `volumeClaimTemplates` — each world owns its PVC. Never a shared PVC.
- Container image bundles `demarkus-server` + `demarkus` CLI (for exec probes).
- Exec liveness + readiness probes against `/.well-known/agent-manifest.md`.
- Resource requests/limits with sane defaults, overridable.
- `Service` type `LoadBalancer`, `protocol: UDP`, port from `server.udpPort` (default 6309). Annotations for cloud-specific LB type (NLB on AWS, etc.).

Secrets:
- `<release>-tokens` — TOML, SHA-256 hashes. Server-mounted.
- `<release>-token-values` — raw tokens. Broker-only. Kept separate so server never mounts raw secrets.

Auth + TLS:
- TLS Secret mounted via `volumeMounts`; cert/key paths via flags.
- Optional `cert-manager` `Certificate` resource (behind a flag) requesting `*.<root>` from a configured `ClusterIssuer`.

Bootstrap Job:
- Mints initial `admin` token on first install via the shared mint library.
- Idempotent — skips on re-install if `admin` label exists.
- SIGHUPs pod after writing.

Observability:
- Pod annotations for Datadog autodiscovery (`ad.datadoghq.com/...`).
- Reference configs for OTel Collector, Vector, Fluent Bit in `deploy/observability/`.
- slog output already structured; no chart-side instrumentation needed.

RBAC:
- Bootstrap Job SA with `get/update` on the named Secret, `get/list/create` on `pods/exec` for SIGHUP. Namespace-scoped `Role`, not `ClusterRole`.

Tests:
- `helm-unittest` for templates.
- Kind-based integration test in CI: install chart → exec into pod → verify health → publish via CLI → verify version increments.

Acceptance:
- `helm install team-a ./deploy/helm/demarkus-server` produces a healthy world reachable via the LB hostname.
- `kubectl delete pod team-a-0` — content persists.
- `helm upgrade --set tokens.labels[1].name=writer` — adds label, SIGHUPs, no rotation of existing labels.
- `--set readOnly=true` — server rejects writes.
- helm-unittest + integration test pass in CI.

### 6.2 — `demarkus-broker` binary (`tools/demarkus-broker/`)

Full feature set, no skip-DELETE corner-cuts.

Routes:
- `GET /healthz` — liveness.
- `GET /login?world=<name>` — start OIDC, PKCE, signed state cookie.
- `GET /callback` — verify `id_token`, mint, write Secret, SIGHUP, redirect to install page.
- `GET /me/install` — render install page (token + claude.json snippet). One-time view.
- `POST /tokens` — programmatic mint via OIDC bearer (CI use).
- `DELETE /tokens/{world}/{label}` — revoke. Removes from Secret + SIGHUPs.
- `GET /worlds` — list worlds the caller may mint against (derived from OIDC group/email mapping).

Implementation:
- Go single binary. Refactor `tools/demarkus-token` generate path into `tools/internal/token/` — pure-function library shared by CLI and broker.
- Provider behind `Verifier` interface (`Verify(token) (sub, email, hd, groups, err)`). Google concrete impl first; Okta/Entra follow without code changes.
- Per-namespace `Role`/`RoleBinding`: `get/update` on named `Secrets`, `get/create` on named `pods/exec`. No `ClusterRole`.
- `resourceVersion`-based optimistic concurrency on Secret writes; retry on conflict (HA-safe).
- Audit log to stdout: JSON line per action (`mint`, `revoke`, `denied`). No DB. These lines are the primary observability signal — joined with server audit lines via `token_label`, they reconstruct identity→action.
- Per-user rate limit on `POST /tokens`.

Tests:
- Unit: Verifier mock, mint lib, Secret-write retry.
- Integration: kind cluster + Dex (lightweight OIDC) to exercise full flow without external Google dependency.

### 6.3 — `demarkus-broker` Helm chart

`deploy/helm/demarkus-broker/`.

- `Deployment`, multi-replica (default 2). HA-safe given resourceVersion retry.
- `Service`, standard `Ingress` (HTTPS), optional cert-manager `Certificate`.
- ConfigMap: world list, OIDC provider config, group/domain → world mapping, allowed paths/ops per group.
- Secret: OIDC client secret.
- ServiceAccount with namespace-scoped `Role`/`RoleBinding` per world namespace.
- Optional Workload Identity annotation (`serviceAccount.workloadIdentity.gsa`) for GKE.
- Pod annotations for Datadog autodiscovery.
- `PodDisruptionBudget` (minAvailable: 1).
- Optional `NetworkPolicy` restricting egress to k8s API + OIDC issuer + world Services.

Acceptance:
- `helm install broker ./deploy/helm/demarkus-broker` produces a reachable broker.
- Sign in via configured OIDC, mint a token, paste snippet, read/write the target world. <2 minutes end-to-end.
- Revoke via `DELETE` — next request from that token fails within seconds.
- Two replicas under concurrent mint load — no lost labels.

### 6.4 — Universe topology examples (`deploy/k8s/examples/`)

- Argo CD `ApplicationSet` over a `worlds:` list. Three-world reference universe.
- Kustomize overlay alternative for non-Argo clusters.
- README walking an operator from zero to a working universe in <15 minutes.

No new code. Reference manifests + docs.

### 6.5 — Observability recipes (`deploy/observability/`)

Backend-agnostic. Customer picks one collector; recipes for the common stacks ship in the repo and are referenced from the operations doc.

- **Datadog**: pod annotations + parser config + dashboard JSON. Logs-to-Metrics rules for common counters.
- **OTel Collector**: `filelogreceiver` + JSON parser + transform → OTLP export. Reference values for `helm install opentelemetry-collector ...`.
- **Vector**: `kubernetes_logs` source → `remap` for JSON parsing → sinks (Datadog, Loki, etc).
- **Fluent Bit**: tail input + parser + filter for audit-tagged lines.
- **Grafana Alloy**: similar shape, Loki-friendly.

Each recipe answers: how do I count publishes per world per minute? How do I count OIDC denials per hour? How do I get an alert when token mints spike?

No demarkus-side instrumentation. The observability deliverable is configs + dashboards + docs, not code.

### 6.6 — Documentation suite

Soul:
- `/deployment/installation.md` — chart values reference, step-by-step install.
- `/deployment/security.md` — threat model, RBAC requirements, attack surface, audit log conventions, capability-auth review.
- `/deployment/operations.md` — backup/DR options, observability wiring, upgrade procedure, troubleshooting.
- `/deployment/multi-oidc.md` — provider-by-provider setup (Google, Okta, Entra ID).
- `/deployment/observability.md` — how to wire each collector, what the key metrics are, sample dashboards.
- `/trials/<customer>.md` — per-customer trial runbook. Nesto's lands first.

Repo:
- Each chart has its own `README.md` with values reference + quick start.

### 6.7 — Release pipeline

- GoReleaser extended to push container images for `demarkus-server`, `demarkus-broker`, `demarkus-agent` to `ghcr.io/latebit-io/*`.
- Helm chart releases on tag — OCI charts to GHCR (`oci://ghcr.io/latebit-io/charts/*`).
- Cosign signing — **deferred to backlog**. Half-day add later.
- CI: helm-lint + helm-unittest + kind-based integration tests on every PR touching `deploy/` or the broker/agent binaries.

## Sequencing

Dependency chain matters. Don't start hub aggregation or broker chart before their prerequisites.

1. **Phase 5 agent** (6.0 prerequisite) — must land first.
2. **6.1 server chart** — parallel with broker binary work; does not block on agent.
3. **6.2 broker binary** — parallel with chart; needs token-mint lib refactor first.
4. **6.3 broker chart** — after 6.2 is testable end-to-end.
5. **6.4 topology examples** — small, after 6.1 + 6.3 stable.
6. **6.5 observability recipes** — incremental, lands as charts stabilize.
7. **6.6 docs** — incremental, each sub-phase contributes its piece.
8. **6.7 release pipeline** — final hardening; signed images come in the cosign-deferred follow-up.

Rough effort: 4–8 weeks of focused work.

## Backlog (deferred, easy to add later)

- **Cosign signing** of images + chart releases. Half-day CI add when wanted.
- **Latency log-enrichment** (`duration_ms` field on request slog lines). Tiny additive change. Defer until trials show it's needed.

## Risks

- **Observability via logs ceiling.** If customers want signals not derivable from current slog (latency, internal state like version counts or PVC fullness), we hit a wall. Mitigation: log enrichment is a small additive change; internal-state metrics can be derived by a cluster-side sidecar that calls `LIST` periodically. Both deferred until a customer surfaces the need.
- **Broker secret-write blast radius.** Holds k8s API creds across world namespaces. Mitigated by namespace-scoped `Role`s, audit log, optional `NetworkPolicy`, and a sealed-secrets/external-secrets recipe in the operations doc.
- **OIDC provider coupling.** First impl is Google; structure so the second provider is a one-day add. Late abstraction = rewrite risk.
- **Token revocation latency.** SIGHUP reloads `tokens.toml`, but in-flight requests with the revoked token complete. Property of the model. Document.
- **Chart proliferation.** Three charts + topology examples + dashboards. Risk of values drift. Mitigate with shared common-labels templates and CI-validated values schemas.
- **Trial scope creep.** First customer (nesto, path-B). If feedback pulls scope back toward "demo slice" mid-build, decide explicitly — don't drift.

## Status

Plan v3, 2026-05-11. All planning-time decisions resolved; observability landed as log-derived backend-agnostic; cosign deferred to backlog. Ready to start work: Phase 5 agent (6.0) first, then `demarkus-server` chart (6.1) and `demarkus-broker` binary (6.2) in parallel.