Plan: Universe Deployment (Phase 6)

Deploy a demarkus universe — many worlds, optional hubs, optional souls — onto Kubernetes with declarative GitOps and self-service token distribution.

Phase 5 (the demarkus agent, read auth, security hardening) gave a single hardened server. Phase 6 makes it cheap to stand up N of them and to onboard real users without hand-distributing tokens.

Goals

1. One Helm chart that deploys a single demarkus world, parameterized for read-only mode, read-auth, storage class, resources, and image tag.
2. One topology pattern (Argo CD ApplicationSet over a worlds: list) that instantiates the chart N times to materialize a universe.
3. One token broker that mints scoped tokens via OIDC login and emits a one-shot install script that wires the user's local MCP config.
4. Zero changes to protocol or core server. Phase 6 is packaging and lifecycle, nothing else. If a need for a new core primitive surfaces, stop and discuss before landing.

Non-Goals

• Operator pattern with a World CRD. Argo CD ApplicationSet covers the lifecycle without a controller. Revisit only if per-world dynamic config from a UI becomes a real requirement.
• Multi-cluster federation. A universe is a set of worlds in one cluster. Cross-cluster sync is a Phase 5 (demarkus-agent sync) concern.
• Hosted SaaS. The chart and broker are for self-hosted clusters. A managed offering is a separate product question.
• Non-markdown content. Server scope is unchanged.

Repository Layout

Chart and reference manifests live in the monorepo. They are tightly coupled to server flags, env vars, paths, and the demarkus-token binary. Splitting them out invites silent drift.

deploy/
  helm/
    demarkus-server/      # the chart
      Chart.yaml
      values.yaml
      templates/
        statefulset.yaml
        service.yaml
        bootstrap-job.yaml
        secret-tokens.yaml
        configmap-readauth.yaml
        rbac.yaml
  k8s/
    examples/
      applicationset.yaml # Argo CD ApplicationSet over a worlds: list
      kustomize-overlay/  # hand-rolled alternative

Broker starts in tools/demarkus-broker/ for the prototype. Coupled iteration on the broker API and the chart's Secret schema is much faster in one repo. Split to latebit-io/demarkus-broker once the API stabilizes (post-6.3) for the same reasons we split the Obsidian plugin: independent release cadence, separate issue tracker, isolated threat model.

Sub-Phases

6.1 — Helm Chart (deploy/helm/demarkus-server/)

A single chart that deploys one world.

Workload shape:

• StatefulSet with replicas: 1 (sharing a content root across replicas is a Phase 7 problem; for now one server per world).
• volumeClaimTemplates so each instance owns its own PVC. No shared PersistentVolumeClaim resource — that has been the source of the PVC pain.
• Service of type ClusterIP exposing UDP/6309. LoadBalancer is opt-in via values.

Values surface (minimal):

image:
  repository: ghcr.io/latebit-io/demarkus-server
  tag: ""              # defaults to chart appVersion
storage:
  className: ""        # required, fail fast if empty
  size: 5Gi
  accessMode: ReadWriteOnce
readOnly: false        # sets DEMARKUS_READ_ONLY
readAuth:
  enabled: false
  paths: []            # path globs that require read tokens
tokens:
  bootstrap: true      # run the Job; false if you manage tokens externally
  labels:              # initial token labels to mint
    - name: admin
      paths: ["/**"]
      ops: ["publish", "read"]
resources: {}          # standard k8s shape
tls:
  certSecret: ""       # opt-in mount

Bootstrap Job:

• Runs demarkus-token generate once per entry in tokens.labels, writes the resulting tokens.toml into a Secret named <release>-tokens, and writes the raw tokens into a separate Secret named <release>-token-values for the broker to read and revoke. Hashes vs raw values are kept in different Secrets so the server only ever mounts the hashes.
• Idempotent: skips entries whose label already appears in <release>-tokens. Re-running the Job after helm upgrade adds new labels without rotating existing ones.
• Sends SIGHUP to the StatefulSet pod (via kubectl rollout restart fallback) so the server reloads tokens without a full restart. The fallback is fine because the Job is rare (chart install, label add).

RBAC:

• ServiceAccount for the bootstrap Job with get/list/create/patch on Secret and pods/exec for the SIGHUP path, scoped to the release namespace only.

Acceptance:

• helm install demo ./deploy/helm/demarkus-server --set storage.className=standard produces a healthy world reachable via the Service.
• kubectl exec into the pod and mark_fetch / returns the seeded content (or empty list — chart does not seed content; that is a separate concern).
• Re-running helm upgrade --set tokens.labels[1].name=writer adds a new token label and SIGHUPs without rotating admin.
• --set readOnly=true makes the pod reject PUBLISH/APPEND/ARCHIVE with not-permitted.

6.2 — Universe Topology (deploy/k8s/examples/)

Reference Argo CD ApplicationSet that iterates a worlds: list and instantiates the chart per entry.

# pseudo
generators:
  - list:
      elements:
        - { name: team-a, paths: "/**", readAuth: false }
        - { name: team-b, paths: "/private/**", readAuth: true }
        - { name: hub,    paths: "/**", readAuth: false, role: hub }
template:
  metadata: { name: '{{name}}' }
  spec:
    source:
      repoURL: https://github.com/latebit-io/demarkus
      path: deploy/helm/demarkus-server
      helm:
        valuesObject:
          tokens:
            labels: ...
          readAuth: ...

Also ship a Kustomize overlay (deploy/k8s/examples/kustomize-overlay/) for clusters without Argo CD. Same chart, instantiated per directory.

The role: hub flag does not change the chart — a hub is just a world that the demarkus-agent (Phase 5) crawls and publishes indexes to. The role label exists only as documentation/selector convenience.

Acceptance:

• Apply the ApplicationSet to an Argo CD instance pointing at the demarkus repo. Three worlds materialize as three Argo CD Application resources. Deleting an entry from the list removes the world.
• README walks an operator from zero to a working three-world universe in under 15 minutes.

6.3 — Token Broker (tools/demarkus-broker/, prototype)

Stateless HTTP service. Auth via OIDC (configurable provider). Persistence is K8s Secrets — no DB.

Endpoints (minimal):

POST /worlds/{name}/tokens
  body: { paths: [...], ops: [...], label: "alice-laptop" }
  → mints token, calls demarkus-token, writes hash to <world>-tokens Secret,
    SIGHUPs the world, returns { token, install_url }

DELETE /tokens/{label}
  → removes label from <world>-tokens, SIGHUPs

GET /me/install?world=<name>&label=<label>
  → returns a one-shot shell script (text/plain). Single-use, expires in 5 min.
    The script writes ~/.config/demarkus/auth and patches ~/.claude.json.

GET /worlds
  → list of worlds the authenticated user is allowed to mint against,
    derived from OIDC group → world mapping in broker config.

Implementation notes:

• Go service, single binary, reuses tools/demarkus-token as a library (refactor demarkus-token's generate path into tools/internal/token/ so both the CLI and broker call the same function).
• Audit log to stdout: {ts, sub, action, world, label, ok}. No separate audit DB; cluster log aggregation handles retention.
• Per-user rate limit (in-memory token bucket) on POST /tokens to prevent runaway minting.
• Scope minted token paths/ops to the OIDC subject's group mapping. A user in team-a can only mint tokens for mark://team-a/*. Group → world mapping lives in broker config.

Acceptance:

• curl -H "Authorization: Bearer <oidc>" -d '{"label":"alice","paths":["/**"],"ops":["read"]}' broker/worlds/team-a/tokens returns a token that successfully reads team-a and is rejected on team-b.
• Revocation (DELETE) takes effect within one SIGHUP — the next read with the revoked token returns not-permitted.

6.4 — User Install Flow

The broker's GET /me/install returns a small shell script. Goals: idempotent, transparent (user can cat before piping), zero new dependencies on the user's machine.

Script behavior:

1. Writes ~/.config/demarkus/auth (mode 0600) with the token. Existing auth file is preserved as auth.bak.
2. Reads ~/.claude.json (or creates it), adds an entry under mcpServers.demarkus-<world> pointing to the demarkus-mcp binary with the right --server URL and DEMARKUS_AUTH env injection. Leaves other MCP servers untouched.
3. Prints the next-step instruction: restart Claude Code, run /soul.

Why not claude mcp add: the CLI works but assumes Claude Code is installed and on PATH. A direct JSON patch is more portable and doesn't depend on the CLI's surface staying stable. The script can fall back to claude mcp add if it detects the CLI.

Acceptance:

• Fresh machine: curl -sSL <broker>/me/install?world=team-a | sh → restart Claude Code → /soul lists team-a documents.
• Re-running the install script does not duplicate the MCP entry.
• Removing the entry from claude.json and re-running re-adds it.

6.5 — Documentation (soul)

• /deployment.md (new): chart values reference, broker setup, OIDC configuration, RBAC, troubleshooting (PVC stuck pending, tokens not reloading, Argo sync drift). Linked from /index.md under a new "Deployment" section.
• /plans/universe-deployment.md (this doc): the working plan, updated as sub-phases complete.
• /journal.md: an entry per sub-phase landing with what shipped and what surprised us.

Risks and Open Questions

• Bootstrap Job RBAC. The Job needs Secret write + pod exec in its own namespace. Some clusters disallow exec from Jobs by org policy. Fallback: kubectl rollout restart instead of pod exec for SIGHUP. Restart is heavier but uses only Deployment-level permissions.
• Helm chart vs operator. ApplicationSet handles the static "spin up N worlds from a list" case. Dynamic per-world config (e.g., a UI that adds read-auth paths) would push toward an operator. Defer until that requirement is real.
• Broker secret-write blast radius. The broker holds K8s API credentials capable of writing token Secrets across all world namespaces. Mitigations: scope its ServiceAccount to a specific list of namespaces (one per world) via Role not ClusterRole; emit audit log on every Secret write; consider sealed-secrets or external-secrets for the long-term.
• OIDC provider coupling. First implementation will likely target one provider (GitHub or Auth0). Keep the provider behind an interface (Verifier with Verify(token) (sub, groups, err)) so adding a second provider is mechanical, not a rewrite.
• Token revocation timing. SIGHUP reloads tokens.toml but in-flight requests with the revoked token complete normally. Acceptable for a non-realtime revocation model; document it explicitly.
• PVC migration / backup. The chart deliberately does not handle backup. Operator chooses: Velero, volume snapshots, or demarkus-agent sync (Phase 5) to a replica world. Document the options, do not bake one in.

Sequencing

1. 6.1 — chart lands first. Single world, real cluster (kind or k3d in CI), automated test that installs and reads.
2. 6.2 — ApplicationSet lands second. Three-world reference universe in a CI cluster.
3. 6.3 — broker prototype lands in tools/demarkus-broker/. Validates the API shape against the chart.
4. 6.4 — install script lands with the broker.
5. 6.5 — docs are updated incrementally as each sub-phase merges, not in a single doc PR at the end.
6. Repo split for the broker happens after 6.4, once the API has been exercised end-to-end.

What's Explicitly Deferred to Phase 7+

• Multi-replica worlds with shared storage (RWX PVCs or object-store backend).
• Cross-cluster universe federation.
• Operator with a World CRD.
• A web UI on top of the broker (currently CLI-only via the install script).
• Managed/SaaS hosting.