# Knowledge System — GKE Reference Deployment

## Goal

Stand up a real, public, working demarkus knowledge system on GKE via GitOps. The deployment repo doubles as a GitHub template so others can instantiate their own knowledge systems by copying it.

**Deployment repo:** [`latebit-io/demarkus-knowledge-system-deploy`](https://github.com/latebit-io/demarkus-knowledge-system-deploy)
**Hostname:** `knowledge.demarkus.io`
**Visibility:** public, marked as GitHub template repository

This is a deployment-layer plan. No core/protocol/server changes. The demarkus binaries and helm charts (`ghcr.io/latebit-io/charts`) are consumed as-is.

## Locked Decisions

| Topic | Choice | Why |
|---|---|---|
| IaC | **OpenTofu** (MPL 2.0) | Genuinely open source, Linux Foundation; HashiCorp's BSL ruled out Terraform |
| Cluster | **GKE Standard** | Need flexibility for mutating webhooks + custom node configs; Autopilot constrains both |
| Region | **`northamerica-northeast2`** (Toronto), single region | Operator preference for Canadian data residency; all resources (state bucket, project, cluster, DNS) co-located |
| Secrets store | **OpenBao** (MPL 2.0) | OSS fork of Vault 1.14; KV v2, k8s auth, GCP KMS auto-unseal, ESO + bank-vaults integration all work |
| Secret injection | **bank-vaults mutating webhook** | Operator preference; rewrites pod env from `vault:` refs at admission, no secrets in etcd |
| GitOps | **ArgoCD with ApplicationSet** | Already proven in `deploy/k8s/examples/applicationset.yaml`; chart published to `ghcr.io/latebit-io/charts` |
| Ingress | **GKE-native NEG + Google-managed cert** (tentative) | Simplest path; revisit if cert-manager + Let's Encrypt is needed for portability |
| DNS | **Cloud DNS** + external-dns operator | Records register themselves from Ingress annotations |
| Backups | **restic CronJob → GCS bucket** | OSS, encrypted, deduplicated; weakens any case for bucket-as-primary storage |
| OIDC for broker | **Google login** (Google Workspace / Google OAuth) | Day-one choice; broker supports generic OIDC so swap is cheap later |
| Project provisioning | **OpenTofu creates the GCP project** | Self-contained bootstrap; assumes org-level perms |
| State backend | **GCS bucket with object versioning** in a separate bootstrap project (`latebit-tofu-bootstrap`) | Keeps state physically outside the blast radius of the tofu-managed project; chicken/egg solved by hand-creating bucket once |
| CI | **GitHub Actions: `tofu plan` on PR, `tofu apply` on merge** | Standard; OIDC federation to GCP, no long-lived service account keys |
| License/topology choices | **Public, template repo, one repo holds both `envs/example/` and `envs/prod/`** | YAGNI on splitting until drift becomes a real problem |

## Repo Layout

```
demarkus-knowledge-system-deploy/
├── README.md                       — what this is, quick-start, template-instantiation guide
├── .github/workflows/
│   ├── tofu-plan.yml               — on PR: fmt-check, validate, plan with comment
│   └── tofu-apply.yml              — on merge to main: apply per env
├── tofu/
│   ├── modules/
│   │   ├── project/                — GCP project + APIs + billing
│   │   ├── network/                — VPC, subnets, Cloud NAT, firewall
│   │   ├── gke/                    — cluster, node pools, Workload Identity
│   │   ├── dns/                    — Cloud DNS zone + DNSSEC
│   │   ├── backups/                — GCS bucket for restic + lifecycle policy
│   │   └── argocd-bootstrap/       — helm install argocd + apply root ApplicationSet
│   └── envs/
│       ├── example/                — placeholder values; template users copy this
│       │   ├── backend.tf
│       │   ├── main.tf
│       │   └── terraform.tfvars.example
│       └── prod/                   — the real knowledge.demarkus.io deployment
│           ├── backend.tf
│           ├── main.tf
│           └── terraform.tfvars
├── bootstrap/                      — one-shot manifests OpenTofu applies post-cluster
│   ├── argocd-values.yaml          — values for the argocd helm chart
│   └── root-appset.yaml            — seed ApplicationSet → platform/ + apps/
├── platform/                       — cluster prerequisites (Argo manages these)
│   ├── cert-manager/
│   ├── openbao/                    — HA Raft, GCP KMS auto-unseal
│   ├── bank-vaults-webhook/        — mutating webhook + secret-injection sidecar
│   ├── external-dns/
│   └── ingress-nginx/              — if we don't go GKE-native
├── apps/                           — demarkus-specific Applications
│   ├── demarkus-broker/
│   │   ├── application.yaml
│   │   └── values-prod.yaml
│   └── demarkus-worlds/
│       ├── applicationset.yaml     — derived from monorepo example
│       └── values-base.yaml
└── docs/
    ├── instantiate.md              — template-user guide
    ├── runbook.md                  — bootstrap-from-zero, DR, rotate secrets
    └── architecture.md             — what's deployed where, sync waves, dependency order
```

## Phases

Order matters. Each phase produces something testable.

### Phase 1 — Bootstrap state + project
- Hand-create one GCS bucket for OpenTofu state in a separate bootstrap project (`latebit-tofu-bootstrap`), in `northamerica-northeast2`, object versioning on. Chicken/egg breaks here.
- `tofu/modules/project/` creates the GCP project, enables APIs (`cloudresourcemanager`, `serviceusage`, `compute`, `container`, `dns`, `iam`, `cloudkms`), wires billing. No `secretmanager` — OpenBao is the secrets store.
- `tofu/envs/prod/` consumes the module with real values.
- Outcome: `gcloud projects describe <project_id>` returns a live project.

### Phase 2 — Network + DNS
- `tofu/modules/network/` — VPC with private nodes, Cloud NAT for egress, firewall for cluster CIDR.
- `tofu/modules/dns/` — Cloud DNS managed zone for `demarkus.io` (or a delegated subdomain `knowledge.demarkus.io`), DNSSEC on.
- Outcome: zone exists, NS records propagate.

### Phase 3 — GKE cluster
- `tofu/modules/gke/` — regional cluster in `northamerica-northeast2`, private nodes, Workload Identity on, node pool with autoscaling.
- Outcome: `kubectl get nodes` works against the cluster.

### Phase 4 — ArgoCD bootstrap
- `tofu/modules/argocd-bootstrap/` uses helm provider to install ArgoCD into the new cluster.
- Same module applies `bootstrap/root-appset.yaml` via `kubectl_manifest`.
- Root ApplicationSet points back at this repo, sync waves: -2 cert-manager, -1 OpenBao + bank-vaults, 0 ingress + external-dns, 1 demarkus apps.
- Outcome: ArgoCD UI reachable via port-forward; root app reads the repo.

### Phase 5 — Platform layer
- `platform/cert-manager/` — Application + ClusterIssuer (Google-managed certs preferred; Let's Encrypt as backup).
- `platform/openbao/` — HA mode, 3 replicas, Raft storage, GCP KMS auto-unseal, k8s auth method enabled.
- `platform/bank-vaults-webhook/` — mutating webhook, AppRole + k8s auth backend, secret-injection sidecar image.
- `platform/external-dns/` — registers records from Ingress annotations against the Cloud DNS zone.
- `platform/ingress-*` — GKE-native ingress class (or nginx-ingress, decide based on cert-manager outcome).
- Outcome: `https://argocd.knowledge.demarkus.io` returns a valid cert.

### Phase 6 — Seed OpenBao secrets
- Manually (one-time per env): unseal OpenBao with GCP KMS, init root token, enable `kv-v2` at `secret/`, store:
  - `secret/broker/oidc-client` — Google OAuth client ID + secret
  - `secret/broker/jwks-signing-key` — broker's signing key (or auto-generate at startup; decide)
  - `secret/worlds/<name>/admin-token` — per-world admin token (server chart bootstrap Job currently generates this; pivot to consume from OpenBao)
- Outcome: `bao kv get secret/broker/oidc-client` returns the values.

### Phase 7 — Demarkus broker + first world
- `apps/demarkus-broker/` Application using the broker helm chart from `ghcr.io/latebit-io/charts`.
  - Pod annotated for bank-vaults webhook: `vault.security.banzaicloud.io/vault-addr: "https://openbao.openbao.svc:8200"` + env vars as `vault:secret/data/broker/oidc-client#client_secret`.
  - Ingress: `knowledge.demarkus.io` (broker HTTPS gateway), `argocd.knowledge.demarkus.io`, `openbao.knowledge.demarkus.io` (admin only).
- `apps/demarkus-worlds/applicationset.yaml` — start with one world (`world-a`).
- Outcome: `curl https://knowledge.demarkus.io/.well-known/oauth-authorization-server` returns RFC 8414 metadata; `/knowledge-join` from a Claude Code plugin can complete the device flow end-to-end.

### Phase 8 — GitHub Actions
- `.github/workflows/tofu-plan.yml` — on PR: `tofu fmt -check`, `tofu validate`, `tofu plan` per env, comment plan as PR comment.
- `.github/workflows/tofu-apply.yml` — on merge to main: `tofu apply -auto-approve` per env. Auth via Workload Identity Federation (GitHub OIDC → GCP service account, no JSON keys).
- Outcome: PRs show plan diffs; merge to main reconciles cluster automatically.

### Phase 9 — Backups
- `apps/backups/` — restic CronJob, mounts world PVCs read-only, pushes to GCS bucket created in Phase 1.
- Test restore procedure: spin up an isolated test PVC, restore from latest snapshot, point a transient demarkus-server at it, verify hash chain.
- Outcome: documented restore drill that succeeds end-to-end.

### Phase 10 — Public-template polish
- README rewrites: what this is, why public, how to instantiate the template, what it costs to run (~$X/month for the baseline).
- `docs/instantiate.md` — step-by-step for someone forking the template.
- GitHub repo settings: mark as Template repository.
- Announcement post (blog at `demarkus.io/blog`).

## Open Items

These don't block Phase 1 but need answers before the phase that needs them:

- **Ingress choice (blocks Phase 5).** GKE-native NEG ingress + Google-managed cert is simplest. cert-manager + Let's Encrypt is more portable and works with any ingress. Lean GKE-native unless portability is a real requirement.
- **Multi-env (affects Phase 1 forward).** `prod` only to start, or `dev` + `prod`? Recommendation: `prod` only. Add `dev` when there's a real reason (testing destructive changes against something non-customer-facing). Saves ~50% of the recurring cost during bootstrap.
- **Number of worlds initially (blocks Phase 7).** One (the demarkus knowledge-system soul itself) is enough to prove end-to-end. Add more on demand.
- **OpenBao HA scale (Phase 5).** 3 replicas (HA Raft) for production seriousness, or 1 (dev mode) to save cost during bootstrap? Recommendation: start with 1, upgrade to 3 before going live.
- **broker JWKS signing key (Phase 6).** Stored in OpenBao and mounted, or auto-generated at broker startup and persisted in OpenBao on first run? Check broker code before deciding.

## Reference

- Monorepo helm charts: `deploy/helm/demarkus-server`, `deploy/helm/demarkus-broker`, `deploy/helm/demarkus-agent`
- ApplicationSet pattern: `deploy/k8s/examples/applicationset.yaml`
- kind harness for local proof: `deploy/kind/up.sh` (already includes `--with-argo`)
- Mock OIDC for local dev: `deploy/kind/mock-oidc.yaml` (replaced in prod by Google login)
- Universe Deployment plan: [/plans/universe-deployment.md](/plans/universe-deployment.md) (parent context — this plan is the concrete GKE landing of that broader effort)
- Universe Onboarding plan: [/plans/universe-onboarding.md](/plans/universe-onboarding.md) (end-user flow that depends on this deployment being live)

## Non-Goals

Explicitly out of scope for this plan; do not let them grow it:

- **Bucket-backed primary storage** for demarkus-server (S3 backend). Deferred — discussed and rejected for now in favor of PVC + restic backups. Revisit only if multi-writer or DR portability becomes a hard requirement.
- **Multi-cluster / multi-region active-active.** Single region (`northamerica-northeast2`, Toronto), single cluster.
- **Custom OIDC IdP.** Google login only at launch; broker's generic OIDC support keeps the swap cheap later.
- **Migrating existing soul deployments** onto this GKE cluster. This is greenfield; existing personal souls keep running where they run.


## Addendum — Phase 9 implementation note (2026-05-29)

Phase 9 shipped as **CSI VolumeSnapshots**, not the restic CronJob the Locked
Decisions table named. Rationale uncovered during build:

- demarkus-server 0.17.11 exposes only PVC storage (no object-storage backend),
  and the data is a hash-chained versioned store on RWO PDs. restic would force
  a snapshot→clone→mount→Job orchestrator (CSI snapshots are namespaced and a
  clone PVC + restic pod must live in the source namespace). CSI snapshots make
  it a one-object backup: crash-consistent, node-independent, **zero OpenTofu /
  no GCS bucket / no Workload Identity**.
- World namespaces are labeled `demarkus.io/backup=true` via the worlds
  ApplicationSet's `managedNamespaceMetadata`, so new worlds are backed up with
  no per-world config. Implementation: `apps/backups/`,
  `docs/runbook-backup-restore.md`.

Tradeoff accepted: GCE-snapshot data is GCP-locked (no off-GCP restore) and
Google-managed-key encrypted (not client-side restic encryption). Consistent
with the stack's existing all-in-on-GCP posture.

**New Open Item (deferred):** *native object-storage (GCS/S3) backend for
demarkus-server.* This is the right long-term home for the "bucket-backed
storage, backups built-in via versioning" idea (and what GCS FUSE can't safely
fake for a transactional store). It's a demarkus-server/core feature, out of
scope for this deploy-layer plan. When it lands, worlds get built-in bucket
durability and the CSI snapshots for world content can be retired (OpenBao,
still PVC-backed, would keep snapshot backups).


## Addendum — Phase 10 status + announcement deferred (2026-05-29)

Phase 10 docs shipped: README rewritten (live endpoint, GitOps + sync-wave
architecture, cost baseline, repo layout, runbook index) and `docs/instantiate.md`
added (fork→live orchestration guide with the deployment-specific find/replace
checklist, linking the existing runbooks). Repo marked as a **GitHub template**
(description + topics set). Architecture folded into the README; no separate
`docs/architecture.md`.

**Announcement blog post (`demarkus.io/blog`): DEFERRED — intentionally held.**
Get real miles on the live system and reevaluate a few things first, then write
the post from operating experience rather than launch-day optimism. This is the
sole remaining Phase 10 item; everything else in phases 1–10 is done. Revisit
after a soak period.


## Status — COMPLETE except deferred launch blog (verified 2026-05-31)

Corrects the stale `/index.md` pointer ("Phase 1 next"). **Phases 1-10 are done.** Verified against the live deploy repo `latebit-io/demarkus-knowledge-system-deploy` (public, `isTemplate`, pushed 2026-05-30):

- **Tofu (P1-P3):** `project`/`network`/`dns`/`gke`/`argocd-bootstrap` modules (+ `platform-iam`, `billing-budget`); `envs/prod/` populated.
- **ArgoCD (P4):** `bootstrap/root-appset.yaml` + values.
- **Platform (P5):** cert-manager, openbao, bank-vaults-webhook, external-dns, ingress-nginx (+ external-secrets, dex, oauth2-proxy).
- **OpenBao seed (P6):** `scripts/seed-openbao.sh` + runbook.
- **Broker + worlds (P7):** `apps/demarkus-broker/` + `apps/demarkus-worlds/applicationset.yaml`.
- **CI (P8):** `tofu-plan.yml`/`tofu-apply.yml`; a `tofu-apply` on main succeeded 2026-05-29.
- **Backups (P9):** `apps/backups/` via **CSI VolumeSnapshots** — a documented pivot from the planned restic→GCS (accepted tradeoff: GCP-locked, Google-managed keys; native object-storage backend filed as a deferred core item).
- **Polish (P10):** README rewrite, `docs/instantiate.md`, template flag set.

**Live corroboration:** `https://knowledge.demarkus.io/.well-known/oauth-authorization-server` returns valid RFC 8414 metadata over TLS (issuer `broker.knowledge.demarkus.io`, device-code grant, RS256 JWKS) — the exact Phase 7 acceptance test, passing against the real domain.

Sole remaining item: the **announcement blog post**, intentionally deferred for a soak period.