# Plan: Universe Deployment

Ship the enterprise-grade deployment for demarkus. The deliverable *is* the deployment: a customer's ops team installs a Helm chart, runs a broker, runs an agent, and has a federated demarkus universe their org can use. Any company evaluating demarkus (their internal "POC") installs the same product an established customer runs in production. There is no separate "POC slice" — the slice mentality is rejected. We build it once, right, and customers trial the real thing.

## Goal

Deliver a complete, supportable, production-grade Kubernetes deployment package for demarkus, including:

1. A Helm chart for `demarkus-server` (one world).
2. A Helm chart for `demarkus-broker` (OIDC token issuance + revocation).
3. A Helm chart for `demarkus-agent` (hub aggregator, crawl-and-index).
4. Reference topology examples (Argo CD `ApplicationSet`, Kustomize overlay).
5. Backend-agnostic observability — structured slog emission + reference configs for Datadog, OTel Collector, Vector, Fluent Bit, Grafana Alloy.
6. Customer-facing documentation — installation, security/threat model, operations, upgrade path, per-provider OIDC setup, observability recipes.
7. A release pipeline producing images and chart releases consumable from GHCR.

The same artifacts power first-customer trial and steady-state operations.

## Non-Goals (Phase 7+ territory)

- Multi-replica worlds with shared storage (RWX / object backend).
- Cross-cluster universe federation.
- Operator with a `World` CRD.
- Hosted / managed SaaS.
- Non-markdown content.

## Constraints

- **No core protocol/server changes.** Period. Per prior precedent (Claude Code plugin, Obsidian plugin, `feedback_plugin_scope.md`). Observability is achieved by log derivation in a collector, not by adding `/metrics` or OTel SDK calls to the server. Note: lifting `HashToken` from `server/internal/auth/` to `protocol/` (Slice A) is an additive helper relocation, not a behavioral change, and is considered within the spirit of this rule.
- Capability-based auth model is non-negotiable. The server never learns identity — only labels.
- Markdown-only scope is non-negotiable.

## Decisions (resolved during planning)

- **Health probes use `exec`, not HTTP.** Both charts ship liveness/readiness `exec` probes that invoke the `demarkus` CLI fetching `/.well-known/agent-manifest.md` (always public per `/architecture.md`). No core change. Same pattern as `redis-cli ping` / `pg_isready`.
- **UDP port is a values knob.** Default `6309`, override via `server.udpPort`. Documented option: switch to `443` for VPN/middlebox-hostile networks (Cloudflare Warp Zero Trust, corp firewalls that filter non-standard UDP). Protocol default stays `6309`.
- **Wildcard TLS via cert-manager DNS-01.** Single `*.<root>` cert covers all worlds + broker hostnames. HTTP-01 cannot work for worlds (no HTTP); DNS-01 is the standard path for QUIC services.
- **DNS topology.** One A record per world + one for the broker, all under the same wildcard zone. Each world exposed by `Service: LoadBalancer` `protocol: UDP`; broker by standard Ingress (HTTPS).
- **Hub aggregator is the existing `demarkus-agent`** at `client/cmd/demarkus-agent/`. Already implements `crawl` + `daemon` subcommands with TOML config, multi-worker crawl, per-host token auth, per-server + aggregated index publishing. Verified end-to-end on 2026-05-11. **Phase 5 is no longer a blocking prerequisite.**
- **`demarkus-agent` lives in `client/cmd/`, not `tools/`.** It's a protocol client (uses `fedcrawl`, `fetch`, `tokens`, `links`), not a utility. Memory rule sharpened: `client/cmd/` = protocol clients (CLI, TUI, MCP, agent); `tools/` = utilities (token, publish, sync, broker).
- **Multi-OIDC.** Broker speaks generic OIDC, not Google-specific. Provider behind a `Verifier` interface. Google validated first; Okta, Entra ID, Auth0 follow with config only.
- **Workload Identity (GKE) as a broker option.** Values flag annotates the broker ServiceAccount with the GSA mapping. Off by default; on for GKE customers wanting no static SA keys. Not applicable to world servers.
- **Broker is HA.** Multi-replica with `resourceVersion` optimistic concurrency on Secret writes, retry on conflict.
- **Observability is log-derived, backend-agnostic.** demarkus already emits structured slog. Charts ship Datadog autodiscovery annotations + reference configs for OTel Collector, Vector, Fluent Bit, Grafana Alloy. Customer's SRE picks the agent/backend. Latency histograms deferred — depends on what current slog includes per request; revisit during trials if needed.
- **Backup/DR is documented, not built.** The chart deliberately does not run backup CronJobs. Operations doc covers Velero, VolumeSnapshot, and `demarkus-agent sync` as DR options.
- **Image hosting:** `ghcr.io/latebit-io/{demarkus-server,demarkus-broker,demarkus-agent}`.
- **Chart registry:** OCI charts in GHCR.
- **Cosign signing:** deferred to backlog. Half-day CI add when wanted.

## Open Questions

1. **First customer trial.** Nesto (`*.library.nesto.ca`) is path-B — trial waits for product. Trial runbook lands at `/trials/nesto.md` when scoping starts.

## Repository Layout

```
deploy/
  helm/
    demarkus-server/         # one-world chart
    demarkus-broker/         # OIDC token broker chart
    demarkus-agent/          # crawl/index agent chart
  k8s/
    examples/
      applicationset.yaml    # Argo CD ApplicationSet over a worlds: list
      kustomize-overlay/     # Kustomize alternative
  observability/
    datadog/                 # autodiscovery annotations + dashboard JSON
    otel-collector/          # collector config recipes
    vector/                  # vector config recipes
    fluent-bit/              # fluent-bit parser + filter recipes
  scripts/                   # operator helpers (cert pre-check, MTU probe, etc.)

client/cmd/
  demarkus-agent/            # protocol client — federation crawler (existing)

tools/
  demarkus-token/            # token mint CLI (moved from server/cmd/ in Slice A)
  demarkus-publish/          # publish CLI (moved from server/cmd/ in Slice A)
  demarkus-broker/           # broker binary (monorepo until API stabilizes)
  internal/token/            # shared mint library, extracted from demarkus-token

protocol/
  auth.go                    # HashToken (promoted from server/internal/auth/ in Slice A)
```

## Sub-Phases

### 6.0 — `demarkus-agent` verified + chart ✓ binary, ⏳ chart

**Binary status (2026-05-11):** existing `demarkus-agent` verified end-to-end against a 3-server smoke test (2 team worlds + 1 hub). Crawls, builds aggregated and per-server indexes, publishes to hub. Two bugs fixed during verification:

- `fedcrawl/crawl.go publishIndex` accepted only `ok` status; first publish returns `created`, surfaced as a misleading warning despite the publish succeeding. Now accepts both.
- `publishIndex` always used `expected_version=0` (create-only), causing every re-publish in daemon mode to fail with `conflict`. Now uses `-1` (no check) for idempotent hub re-publish; server's no-op-on-duplicate-content prevents version churn.
- `Makefile` did not build `demarkus-agent`; the agent had to be hand-built. Now builds with `make client`.

Tests in `client/internal/fedcrawl/crawl_test.go` cover all three: create + re-publish + status acceptance + per-server / aggregated modes. `bash pre-commit.sh` clean.

**Remaining 6.0 work — agent Helm chart** at `deploy/helm/demarkus-agent/`:
- `Deployment` (not StatefulSet — agent is stateless modulo state file; recoverable from next crawl).
- ConfigMap holding the TOML agent config (`seeds`, `hubs`, `crawl`, `politeness`, `schedule`).
- Secret holding per-host tokens for publishing to hub(s).
- ServiceAccount, no special RBAC needed (no k8s API calls).
- Exec liveness probe (`demarkus-agent version` returns 0).
- No `Service` (agent is outbound-only).
- Pod annotations for Datadog autodiscovery; log fields documented for `mint`/`crawl` events.

Acceptance: `helm install agent ./deploy/helm/demarkus-agent --set config.seeds[0]=mark://team-a:6309 ...` produces a pod that crawls on schedule and publishes to the configured hub. Pod restart resumes from state PVC (optional; can also be ephemeral).

### 6.1 — `demarkus-server` Helm chart

`deploy/helm/demarkus-server/`. Production-grade.

Workload:
- `StatefulSet`, 1 replica (multi-replica is Phase 7).
- `volumeClaimTemplates` — each world owns its PVC. Never a shared PVC.
- Container image bundles `demarkus-server` + `demarkus` CLI (for exec probes).
- Exec liveness + readiness probes against `/.well-known/agent-manifest.md`.
- Resource requests/limits with sane defaults, overridable.
- `Service` type `LoadBalancer`, `protocol: UDP`, port from `server.udpPort` (default 6309). Annotations for cloud-specific LB type (NLB on AWS, etc.).

Secrets:
- `<release>-tokens` — TOML, SHA-256 hashes. Server-mounted.
- `<release>-token-values` — raw tokens. Broker-only. Kept separate so server never mounts raw secrets.

Auth + TLS:
- TLS Secret mounted via `volumeMounts`; cert/key paths via flags.
- Optional `cert-manager` `Certificate` resource (behind a flag) requesting `*.<root>` from a configured `ClusterIssuer`.

Bootstrap Job:
- Mints initial `admin` token on first install via the shared mint library.
- Idempotent — skips on re-install if `admin` label exists.
- SIGHUPs pod after writing.

Observability:
- Pod annotations for Datadog autodiscovery.
- Reference configs for OTel Collector, Vector, Fluent Bit in `deploy/observability/`.
- slog output already structured; no chart-side instrumentation needed.

RBAC:
- Bootstrap Job SA with `get/update` on the named Secret, `get/list/create` on `pods/exec` for SIGHUP. Namespace-scoped `Role`, not `ClusterRole`.

Tests:
- `helm-unittest` for templates.
- Kind-based integration test in CI: install chart → exec into pod → verify health → publish via CLI → verify version increments.

### 6.2 — `demarkus-broker` binary (`tools/demarkus-broker/`)

Full feature set, no skip-DELETE corner-cuts.

#### Role & topology

- **Issuance authority, not a request proxy.** Broker sits on the `demarkus login` path; clients then talk to world servers directly carrying the raw token. World servers stay identity-blind; broker never sees `mark://` requests. Capability model preserved.
- **One broker per universe (cluster) by default.** Single OIDC client registration, single `worlds:` list in values, single issuance state Secret. Multiple brokers only when (a) multiple OIDC providers must coexist, (b) hard tenant isolation between orgs sharing a cluster, or (c) Phase-7 geo split. N>1 is supported but not the common case.
- Broker SA holds a namespace-scoped `Role` + `RoleBinding` in **each world's namespace**, with `get/patch` limited to **that world's `tokens` Secret only**. No `ClusterRole`. Blast radius bounded to the token Secrets it's explicitly granted.

#### State

Two distinct Kubernetes Secrets, never merged:

| Secret | Lives in | Contents | Reader |
|---|---|---|---|
| `<world>-tokens` | each world's namespace | TOML: `[tokens.<label>] hash=, paths=, operations=, expires=` | `demarkus-server` |
| `<broker>-issuances` | broker namespace | JSON: `label → {email, world, paths, operations, issued_at, expires}` + secondary index `email → [labels]` | broker only |

World servers see only hashes. Email and identity live in broker state alone.

#### Labels

Opaque IDs (e.g., `usr_<random8>`). **Not** email-derived. Reasons:

- Sanitizing emails (dots, plus signs, IDN) into TOML keys is fragile.
- Token rotation yields a new label each time; opaque IDs reflect that naturally.
- "Revoke everything for `fredrick@x`" becomes a broker-state index lookup, not a label string scan.

#### Per-world authorization (values schema)

```yaml
worlds:
  - name: team-a
    namespace: team-a
    tokensSecret: team-a-tokens
    allow:
      domains: ["nesto.ca"]
      groups: ["engineering"]      # OIDC `groups` claim
    defaultToken:
      paths: ["/team-a/*"]
      operations: ["read", "publish"]
      expiresAfter: 24h
```

On `demarkus login`, broker evaluates the OIDC identity against every world in its config and mints one token per world the user qualifies for. Client may request narrower scope; never broader.

#### HashToken provenance

Promoted to `protocol/` in Slice A. Broker imports `protocol.HashToken` to ensure byte-identical hashes to what world servers read from `tokens.toml`. The hash format `sha256-<hex>` is treated as a protocol contract, single source of truth.

#### Revocation

Three triggers, one cleanup index (the issuances Secret):

1. **User-initiated** (`demarkus token revoke <label>` or `rotate`):
   - `DELETE /tokens/:label` carrying OIDC ID token.
   - Broker confirms `entry.email == claims.email` (owner check — users can only revoke their own).
   - Patches the world's `tokens` Secret to drop the label.
   - Drops the entry from issuances Secret.
   - SIGHUPs world server pod(s) so `tokens.toml` is re-read. (Or relies on server fsnotify if present — verify during implementation.)

2. **Expiry sweeper** (broker in-process ticker, default every 5 min, leader-elected via `Lease` so only one replica sweeps):
   - Queries issuances where `expires < now`.
   - For each: patch the world's `tokens` Secret + drop from issuances + SIGHUP.

3. **Identity lifecycle** (user leaves the org):
   - **Default: short-lived tokens.** `defaultToken.expiresAfter` of 24h means stale tokens age out within a day. Next `demarkus login` reruns OIDC; if the user's IdP account is disabled, login fails. No broker→IdP coupling needed. Same approach as AWS STS / `gcloud auth login`.
   - **Backlog: SCIM lifecycle webhook** (`POST /scim/v2/Users/:id`) for enterprise IdPs that push deprovisioning events (Okta, Entra). Optional add when a customer asks.
   - **Periodic IdP re-validation rejected** — worse trade-off than either above (couples broker to IdP rate limits, slower than SCIM, less simple than short-lived).

#### Cleanup edge cases

- **Orphan in `tokens.toml`** (admin-minted via legacy `demarkus-token` CLI): broker never claims it. Admins manage via CLI. Broker API only operates on labels it minted.
- **Orphan in issuances Secret** (admin hand-deleted from world tokens Secret): expiry sweeper detects drift on each pass and prunes broker-state entries whose label no longer exists in the world's TOML.
- **In-flight request after revoke**: a raw token already authenticated on a live QUIC connection completes its current request. Property of the capability + connection-reuse model. Documented; not fixable without core server changes.

#### Routes (HTTP, behind Ingress)

- `POST /auth/login` — OIDC redirect entry point.
- `GET /auth/callback` — OIDC callback; mints tokens for every qualifying world; returns JSON `{world → raw_token}`. One-time response.
- `GET /tokens` — list caller's tokens from broker state (labels + metadata only, never raw tokens).
- `DELETE /tokens/:label` — revoke caller-owned token.
- `POST /tokens/:label/rotate` — revoke + mint with same scope; returns new raw token.
- `GET /healthz`, `GET /readyz`.

#### OIDC providers

Provider behind a `Verifier` interface. Initial: Google (via `coreos/go-oidc` or `google.golang.org/api/idtoken`). Follow-ons: Okta, Entra ID, Auth0 — config only. Group-claim availability varies per provider; doc page per provider explains required claim configuration (e.g., Entra needs the `groups` optional claim enabled; some providers require a userinfo call to fetch groups not in the ID token).

#### HA

Multi-replica. Issuance writes to k8s Secrets use `resourceVersion` optimistic concurrency with retry-on-conflict. Sweeper uses a `coordination.k8s.io/Lease` for leader election so only one replica runs the periodic loop.

#### Tests

- Mint flow: table-driven, mock OIDC verifier, fake k8s clientset, assert `tokens` Secret + issuances Secret both updated atomically (best-effort — k8s API is two calls).
- Sweeper: fake clock + fake clientset.
- RBAC permission-denied path explicitly covered (broker SA without `patch` on a world's Secret → mint fails cleanly, not partial state).
- Drift pruning: orphan-in-issuances case covered by sweeper test.
- Owner check on `DELETE` — different OIDC subject than the entry → 403.

### 6.3 — `demarkus-broker` Helm chart

Multi-replica HA. As previously specified.

### 6.4 — Universe topology examples

ApplicationSet + Kustomize overlay. As previously specified.

### 6.5 — Observability recipes

Per-backend configs in `deploy/observability/`. As previously specified.

### 6.6 — Documentation suite

`/deployment/*.md` + per-chart READMEs. As previously specified.

### 6.7 — Release pipeline

GHCR images + OCI charts. Cosign deferred to backlog.

## Sequencing

1. **Slice A (in flight)** — token-mint library refactor; `HashToken` to `protocol/`; `demarkus-token` + `demarkus-publish` to `tools/`; `make all` and `pre-commit.sh` extended to include `tools`.
2. **6.0 chart** — quick win since binary is done.
3. **6.1 server chart** — main piece.
4. **6.2 broker binary** — parallel with 6.1, design pinned (see above).
5. **6.3 broker chart** — after 6.2 testable.
6. **6.4 topology examples**.
7. **6.5 observability recipes**.
8. **6.6 docs** — incremental throughout.
9. **6.7 release pipeline** — final.

Rough effort: 2–4 weeks of focused work (down from 4–8 in v3 — agent binary work eliminated).

## Backlog (deferred, easy to add later)

- **Cosign signing** of images + chart releases. Half-day CI add when wanted.
- **Latency log-enrichment** (`duration_ms` field on request slog lines). Tiny additive change. Defer until trials show it's needed.
- **SCIM lifecycle webhook** on the broker (`POST /scim/v2/Users/:id`) for enterprise IdPs that push deprovisioning events. Adds responsiveness beyond what short-lived tokens give.

## Risks

- **Observability via logs ceiling.** If customers want signals not derivable from current slog (latency, internal state like version counts or PVC fullness), we hit a wall. Mitigation: log enrichment is a small additive change; internal-state metrics derivable by a cluster-side sidecar that calls `LIST` periodically.
- **Broker secret-write blast radius.** Holds k8s API creds across world namespaces. Mitigated by namespace-scoped `Role`s (one per world), audit log, optional `NetworkPolicy`. Per-world `Role` is non-negotiable — never `ClusterRole`.
- **OIDC provider coupling.** First impl is Google; structure so second provider is a one-day add.
- **Token revocation in-flight latency.** SIGHUP reloads `tokens.toml`, but a request already authenticated on a live QUIC connection completes. Property of model.
- **Chart proliferation.** Three charts + examples + dashboards. Mitigate with shared common-labels templates.
- **Trial scope creep.** First customer is path-B. If feedback pulls scope back to "demo slice" mid-build, decide explicitly.

## Status

Plan v5, 2026-05-11. §6.2 expanded with broker design pinned during Slice A planning (topology = one broker per universe; state split between world `tokens` Secret and broker `issuances` Secret; opaque labels with email mapping in broker state only; per-world authorization in values; revocation via user-initiated DELETE + expiry sweeper + short-lived tokens as the lifecycle answer; SCIM in backlog). Phase 5 prerequisite cleared. Slice A in flight: token-mint library refactor, `HashToken` promotion to `protocol/`, `demarkus-token`/`demarkus-publish` move to `tools/`.