# Plan: Universe Deployment (Phase 6)

Deploy a demarkus universe — many worlds, optional hubs, optional souls — onto Kubernetes with declarative GitOps and self-service token distribution.

Phase 5 (the demarkus agent, read auth, security hardening) gave a single hardened server. Phase 6 makes it cheap to stand up *N* of them and to onboard real users without hand-distributing tokens.

## Goals

1. **One Helm chart** that deploys a single demarkus world, parameterized for read-only mode, read-auth, storage class, resources, and image tag.
2. **One topology pattern** (Argo CD `ApplicationSet` over a `worlds:` list) that instantiates the chart N times to materialize a universe.
3. **One token broker** that mints scoped tokens via OIDC login and emits a one-shot install script that wires the user's local MCP config.
4. **Zero changes to protocol or core server.** Phase 6 is packaging and lifecycle, nothing else. If a need for a new core primitive surfaces, stop and discuss before landing.

## Non-Goals

- Operator pattern with a `World` CRD. Argo CD ApplicationSet covers the lifecycle without a controller. Revisit only if per-world dynamic config from a UI becomes a real requirement.
- Multi-cluster federation. A universe is a set of worlds in one cluster. Cross-cluster sync is a Phase 5 (`demarkus-agent sync`) concern.
- Hosted SaaS. The chart and broker are for self-hosted clusters. A managed offering is a separate product question.
- Non-markdown content. Server scope is unchanged.

## Repository Layout

**Chart and reference manifests live in the monorepo.** They are tightly coupled to server flags, env vars, paths, and the `demarkus-token` binary. Splitting them out invites silent drift.

```
deploy/
  helm/
    demarkus-server/      # the chart
      Chart.yaml
      values.yaml
      templates/
        statefulset.yaml
        service.yaml
        bootstrap-job.yaml
        secret-tokens.yaml
        configmap-readauth.yaml
        rbac.yaml
  k8s/
    examples/
      applicationset.yaml # Argo CD ApplicationSet over a worlds: list
      kustomize-overlay/  # hand-rolled alternative
```

**Broker starts in `tools/demarkus-broker/` for the prototype.** Coupled iteration on the broker API and the chart's Secret schema is much faster in one repo. Split to `latebit-io/demarkus-broker` once the API stabilizes (post-6.3) for the same reasons we split the Obsidian plugin: independent release cadence, separate issue tracker, isolated threat model.

## Sub-Phases

### 6.1 — Helm Chart (`deploy/helm/demarkus-server/`)

A single chart that deploys one world.

**Workload shape:**
- `StatefulSet` with `replicas: 1` (sharing a content root across replicas is a Phase 7 problem; for now one server per world).
- `volumeClaimTemplates` so each instance owns its own PVC. No shared `PersistentVolumeClaim` resource — that has been the source of the PVC pain.
- `Service` of type `ClusterIP` exposing UDP/6309. `LoadBalancer` is opt-in via values.

**Values surface (minimal):**
```yaml
image:
  repository: ghcr.io/latebit-io/demarkus-server
  tag: ""              # defaults to chart appVersion
storage:
  className: ""        # required, fail fast if empty
  size: 5Gi
  accessMode: ReadWriteOnce
readOnly: false        # sets DEMARKUS_READ_ONLY
readAuth:
  enabled: false
  paths: []            # path globs that require read tokens
tokens:
  bootstrap: true      # run the Job; false if you manage tokens externally
  labels:              # initial token labels to mint
    - name: admin
      paths: ["/**"]
      ops: ["publish", "read"]
resources: {}          # standard k8s shape
tls:
  certSecret: ""       # opt-in mount
```

**Bootstrap Job:**
- Runs `demarkus-token generate` once per entry in `tokens.labels`, writes the resulting `tokens.toml` into a `Secret` named `<release>-tokens`, and writes the *raw tokens* into a separate `Secret` named `<release>-token-values` for the broker to read and revoke. Hashes vs raw values are kept in different Secrets so the server only ever mounts the hashes.
- Idempotent: skips entries whose label already appears in `<release>-tokens`. Re-running the Job after `helm upgrade` adds new labels without rotating existing ones.
- Sends `SIGHUP` to the StatefulSet pod (via `kubectl rollout restart` fallback) so the server reloads tokens without a full restart. The fallback is fine because the Job is rare (chart install, label add).

**RBAC:**
- ServiceAccount for the bootstrap Job with `get/list/create/patch` on `Secret` and `pods/exec` for the SIGHUP path, scoped to the release namespace only.

**Acceptance:**
- `helm install demo ./deploy/helm/demarkus-server --set storage.className=standard` produces a healthy world reachable via the Service.
- `kubectl exec` into the pod and `mark_fetch /` returns the seeded content (or empty list — chart does not seed content; that is a separate concern).
- Re-running `helm upgrade --set tokens.labels[1].name=writer` adds a new token label and SIGHUPs without rotating `admin`.
- `--set readOnly=true` makes the pod reject `PUBLISH/APPEND/ARCHIVE` with `not-permitted`.

### 6.2 — Universe Topology (`deploy/k8s/examples/`)

Reference Argo CD `ApplicationSet` that iterates a `worlds:` list and instantiates the chart per entry.

```yaml
# pseudo
generators:
  - list:
      elements:
        - { name: team-a, paths: "/**", readAuth: false }
        - { name: team-b, paths: "/private/**", readAuth: true }
        - { name: hub,    paths: "/**", readAuth: false, role: hub }
template:
  metadata: { name: '{{name}}' }
  spec:
    source:
      repoURL: https://github.com/latebit-io/demarkus
      path: deploy/helm/demarkus-server
      helm:
        valuesObject:
          tokens:
            labels: ...
          readAuth: ...
```

Also ship a Kustomize overlay (`deploy/k8s/examples/kustomize-overlay/`) for clusters without Argo CD. Same chart, instantiated per directory.

The `role: hub` flag does not change the chart — a hub is just a world that the `demarkus-agent` (Phase 5) crawls and publishes indexes to. The role label exists only as documentation/selector convenience.

**Acceptance:**
- Apply the ApplicationSet to an Argo CD instance pointing at the demarkus repo. Three worlds materialize as three Argo CD `Application` resources. Deleting an entry from the list removes the world.
- README walks an operator from zero to a working three-world universe in under 15 minutes.

### 6.3 — Token Broker (`tools/demarkus-broker/`, prototype)

Stateless HTTP service. Auth via OIDC (configurable provider). Persistence is K8s Secrets — no DB.

**Endpoints (minimal):**
```
POST /worlds/{name}/tokens
  body: { paths: [...], ops: [...], label: "alice-laptop" }
  → mints token, calls demarkus-token, writes hash to <world>-tokens Secret,
    SIGHUPs the world, returns { token, install_url }

DELETE /tokens/{label}
  → removes label from <world>-tokens, SIGHUPs

GET /me/install?world=<name>&label=<label>
  → returns a one-shot shell script (text/plain). Single-use, expires in 5 min.
    The script writes ~/.config/demarkus/auth and patches ~/.claude.json.

GET /worlds
  → list of worlds the authenticated user is allowed to mint against,
    derived from OIDC group → world mapping in broker config.
```

**Implementation notes:**
- Go service, single binary, reuses `tools/demarkus-token` as a library (refactor `demarkus-token`'s generate path into `tools/internal/token/` so both the CLI and broker call the same function).
- Audit log to stdout: `{ts, sub, action, world, label, ok}`. No separate audit DB; cluster log aggregation handles retention.
- Per-user rate limit (in-memory token bucket) on `POST /tokens` to prevent runaway minting.
- Scope minted token paths/ops to the OIDC subject's group mapping. A user in `team-a` can only mint tokens for `mark://team-a/*`. Group → world mapping lives in broker config.

**Acceptance:**
- `curl -H "Authorization: Bearer <oidc>" -d '{"label":"alice","paths":["/**"],"ops":["read"]}' broker/worlds/team-a/tokens` returns a token that successfully reads `team-a` and is rejected on `team-b`.
- Revocation (`DELETE`) takes effect within one SIGHUP — the next read with the revoked token returns `not-permitted`.

### 6.4 — User Install Flow

The broker's `GET /me/install` returns a small shell script. Goals: idempotent, transparent (user can `cat` before piping), zero new dependencies on the user's machine.

**Script behavior:**
1. Writes `~/.config/demarkus/auth` (mode 0600) with the token. Existing auth file is preserved as `auth.bak`.
2. Reads `~/.claude.json` (or creates it), adds an entry under `mcpServers.demarkus-<world>` pointing to the demarkus-mcp binary with the right `--server` URL and `DEMARKUS_AUTH` env injection. Leaves other MCP servers untouched.
3. Prints the next-step instruction: restart Claude Code, run `/soul`.

**Why not `claude mcp add`:** the CLI works but assumes Claude Code is installed and on PATH. A direct JSON patch is more portable and doesn't depend on the CLI's surface staying stable. The script can fall back to `claude mcp add` if it detects the CLI.

**Acceptance:**
- Fresh machine: `curl -sSL <broker>/me/install?world=team-a | sh` → restart Claude Code → `/soul` lists `team-a` documents.
- Re-running the install script does not duplicate the MCP entry.
- Removing the entry from `claude.json` and re-running re-adds it.

### 6.5 — Documentation (soul)

- **`/deployment.md`** (new): chart values reference, broker setup, OIDC configuration, RBAC, troubleshooting (PVC stuck pending, tokens not reloading, Argo sync drift). Linked from `/index.md` under a new "Deployment" section.
- **`/plans/universe-deployment.md`** (this doc): the working plan, updated as sub-phases complete.
- **`/journal.md`**: an entry per sub-phase landing with what shipped and what surprised us.

## Risks and Open Questions

- **Bootstrap Job RBAC.** The Job needs Secret write + pod exec in its own namespace. Some clusters disallow exec from Jobs by org policy. Fallback: `kubectl rollout restart` instead of `pod exec` for SIGHUP. Restart is heavier but uses only Deployment-level permissions.
- **Helm chart vs operator.** ApplicationSet handles the static "spin up N worlds from a list" case. Dynamic per-world config (e.g., a UI that adds read-auth paths) would push toward an operator. Defer until that requirement is real.
- **Broker secret-write blast radius.** The broker holds K8s API credentials capable of writing token Secrets across all world namespaces. Mitigations: scope its ServiceAccount to a specific list of namespaces (one per world) via `Role` not `ClusterRole`; emit audit log on every Secret write; consider sealed-secrets or external-secrets for the long-term.
- **OIDC provider coupling.** First implementation will likely target one provider (GitHub or Auth0). Keep the provider behind an interface (`Verifier` with `Verify(token) (sub, groups, err)`) so adding a second provider is mechanical, not a rewrite.
- **Token revocation timing.** SIGHUP reloads `tokens.toml` but in-flight requests with the revoked token complete normally. Acceptable for a non-realtime revocation model; document it explicitly.
- **PVC migration / backup.** The chart deliberately does not handle backup. Operator chooses: Velero, volume snapshots, or `demarkus-agent sync` (Phase 5) to a replica world. Document the options, do not bake one in.

## Sequencing

1. **6.1 — chart** lands first. Single world, real cluster (kind or k3d in CI), automated test that installs and reads.
2. **6.2 — ApplicationSet** lands second. Three-world reference universe in a CI cluster.
3. **6.3 — broker prototype** lands in `tools/demarkus-broker/`. Validates the API shape against the chart.
4. **6.4 — install script** lands with the broker.
5. **6.5 — docs** are updated incrementally as each sub-phase merges, not in a single doc PR at the end.
6. **Repo split** for the broker happens *after* 6.4, once the API has been exercised end-to-end.

## What's Explicitly Deferred to Phase 7+

- Multi-replica worlds with shared storage (RWX PVCs or object-store backend).
- Cross-cluster universe federation.
- Operator with a `World` CRD.
- A web UI on top of the broker (currently CLI-only via the install script).
- Managed/SaaS hosting.