Knowledge System — GKE Reference Deployment

Goal

Stand up a real, public, working demarkus knowledge system on GKE via GitOps. The deployment repo doubles as a GitHub template so others can instantiate their own knowledge systems by copying it.

Deployment repo: latebit-io/demarkus-knowledge-system-deploy Hostname: knowledge.demarkus.io Visibility: public, marked as GitHub template repository

This is a deployment-layer plan. No core/protocol/server changes. The demarkus binaries and helm charts (ghcr.io/latebit-io/charts) are consumed as-is.

Locked Decisions

Repo Layout

demarkus-knowledge-system-deploy/
├── README.md                       — what this is, quick-start, template-instantiation guide
├── .github/workflows/
│   ├── tofu-plan.yml               — on PR: fmt-check, validate, plan with comment
│   └── tofu-apply.yml              — on merge to main: apply per env
├── tofu/
│   ├── modules/
│   │   ├── project/                — GCP project + APIs + billing
│   │   ├── network/                — VPC, subnets, Cloud NAT, firewall
│   │   ├── gke/                    — cluster, node pools, Workload Identity
│   │   ├── dns/                    — Cloud DNS zone + DNSSEC
│   │   ├── backups/                — GCS bucket for restic + lifecycle policy
│   │   └── argocd-bootstrap/       — helm install argocd + apply root ApplicationSet
│   └── envs/
│       ├── example/                — placeholder values; template users copy this
│       │   ├── backend.tf
│       │   ├── main.tf
│       │   └── terraform.tfvars.example
│       └── prod/                   — the real knowledge.demarkus.io deployment
│           ├── backend.tf
│           ├── main.tf
│           └── terraform.tfvars
├── bootstrap/                      — one-shot manifests OpenTofu applies post-cluster
│   ├── argocd-values.yaml          — values for the argocd helm chart
│   └── root-appset.yaml            — seed ApplicationSet → platform/ + apps/
├── platform/                       — cluster prerequisites (Argo manages these)
│   ├── cert-manager/
│   ├── openbao/                    — HA Raft, GCP KMS auto-unseal
│   ├── bank-vaults-webhook/        — mutating webhook + secret-injection sidecar
│   ├── external-dns/
│   └── ingress-nginx/              — if we don't go GKE-native
├── apps/                           — demarkus-specific Applications
│   ├── demarkus-broker/
│   │   ├── application.yaml
│   │   └── values-prod.yaml
│   └── demarkus-worlds/
│       ├── applicationset.yaml     — derived from monorepo example
│       └── values-base.yaml
└── docs/
    ├── instantiate.md              — template-user guide
    ├── runbook.md                  — bootstrap-from-zero, DR, rotate secrets
    └── architecture.md             — what's deployed where, sync waves, dependency order

Phases

Order matters. Each phase produces something testable.

Phase 1 — Bootstrap state + project

• Hand-create one GCS bucket for OpenTofu state in a separate bootstrap project (latebit-tofu-bootstrap), in northamerica-northeast2, object versioning on. Chicken/egg breaks here.
• tofu/modules/project/ creates the GCP project, enables APIs (cloudresourcemanager, serviceusage, compute, container, dns, iam, cloudkms), wires billing. No secretmanager — OpenBao is the secrets store.
• tofu/envs/prod/ consumes the module with real values.
• Outcome: gcloud projects describe <project_id> returns a live project.

Phase 2 — Network + DNS

• tofu/modules/network/ — VPC with private nodes, Cloud NAT for egress, firewall for cluster CIDR.
• tofu/modules/dns/ — Cloud DNS managed zone for demarkus.io (or a delegated subdomain knowledge.demarkus.io), DNSSEC on.
• Outcome: zone exists, NS records propagate.

Phase 3 — GKE cluster

• tofu/modules/gke/ — regional cluster in northamerica-northeast2, private nodes, Workload Identity on, node pool with autoscaling.
• Outcome: kubectl get nodes works against the cluster.

Phase 4 — ArgoCD bootstrap

• tofu/modules/argocd-bootstrap/ uses helm provider to install ArgoCD into the new cluster.
• Same module applies bootstrap/root-appset.yaml via kubectl_manifest.
• Root ApplicationSet points back at this repo, sync waves: -2 cert-manager, -1 OpenBao + bank-vaults, 0 ingress + external-dns, 1 demarkus apps.
• Outcome: ArgoCD UI reachable via port-forward; root app reads the repo.

Phase 5 — Platform layer

• platform/cert-manager/ — Application + ClusterIssuer (Google-managed certs preferred; Let's Encrypt as backup).
• platform/openbao/ — HA mode, 3 replicas, Raft storage, GCP KMS auto-unseal, k8s auth method enabled.
• platform/bank-vaults-webhook/ — mutating webhook, AppRole + k8s auth backend, secret-injection sidecar image.
• platform/external-dns/ — registers records from Ingress annotations against the Cloud DNS zone.
• platform/ingress-* — GKE-native ingress class (or nginx-ingress, decide based on cert-manager outcome).
• Outcome: https://argocd.knowledge.demarkus.io returns a valid cert.

Phase 6 — Seed OpenBao secrets

• Manually (one-time per env): unseal OpenBao with GCP KMS, init root token, enable kv-v2 at secret/, store:
  • secret/broker/oidc-client — Google OAuth client ID + secret
  • secret/broker/jwks-signing-key — broker's signing key (or auto-generate at startup; decide)
  • secret/worlds/<name>/admin-token — per-world admin token (server chart bootstrap Job currently generates this; pivot to consume from OpenBao)
• Outcome: bao kv get secret/broker/oidc-client returns the values.

Phase 7 — Demarkus broker + first world

• apps/demarkus-broker/ Application using the broker helm chart from ghcr.io/latebit-io/charts.
  • Pod annotated for bank-vaults webhook: vault.security.banzaicloud.io/vault-addr: "https://openbao.openbao.svc:8200" + env vars as vault:secret/data/broker/oidc-client#client_secret.
  • Ingress: knowledge.demarkus.io (broker HTTPS gateway), argocd.knowledge.demarkus.io, openbao.knowledge.demarkus.io (admin only).
• apps/demarkus-worlds/applicationset.yaml — start with one world (world-a).
• Outcome: curl https://knowledge.demarkus.io/.well-known/oauth-authorization-server returns RFC 8414 metadata; /knowledge-join from a Claude Code plugin can complete the device flow end-to-end.

Phase 8 — GitHub Actions

• .github/workflows/tofu-plan.yml — on PR: tofu fmt -check, tofu validate, tofu plan per env, comment plan as PR comment.
• .github/workflows/tofu-apply.yml — on merge to main: tofu apply -auto-approve per env. Auth via Workload Identity Federation (GitHub OIDC → GCP service account, no JSON keys).
• Outcome: PRs show plan diffs; merge to main reconciles cluster automatically.

Phase 9 — Backups

• apps/backups/ — restic CronJob, mounts world PVCs read-only, pushes to GCS bucket created in Phase 1.
• Test restore procedure: spin up an isolated test PVC, restore from latest snapshot, point a transient demarkus-server at it, verify hash chain.
• Outcome: documented restore drill that succeeds end-to-end.

Phase 10 — Public-template polish

• README rewrites: what this is, why public, how to instantiate the template, what it costs to run (~$X/month for the baseline).
• docs/instantiate.md — step-by-step for someone forking the template.
• GitHub repo settings: mark as Template repository.
• Announcement post (blog at demarkus.io/blog).

Open Items

These don't block Phase 1 but need answers before the phase that needs them:

• Ingress choice (blocks Phase 5). GKE-native NEG ingress + Google-managed cert is simplest. cert-manager + Let's Encrypt is more portable and works with any ingress. Lean GKE-native unless portability is a real requirement.
• Multi-env (affects Phase 1 forward). prod only to start, or dev + prod? Recommendation: prod only. Add dev when there's a real reason (testing destructive changes against something non-customer-facing). Saves ~50% of the recurring cost during bootstrap.
• Number of worlds initially (blocks Phase 7). One (the demarkus knowledge-system soul itself) is enough to prove end-to-end. Add more on demand.
• OpenBao HA scale (Phase 5). 3 replicas (HA Raft) for production seriousness, or 1 (dev mode) to save cost during bootstrap? Recommendation: start with 1, upgrade to 3 before going live.
• broker JWKS signing key (Phase 6). Stored in OpenBao and mounted, or auto-generated at broker startup and persisted in OpenBao on first run? Check broker code before deciding.

Reference

• Monorepo helm charts: deploy/helm/demarkus-server, deploy/helm/demarkus-broker, deploy/helm/demarkus-agent
• ApplicationSet pattern: deploy/k8s/examples/applicationset.yaml
• kind harness for local proof: deploy/kind/up.sh (already includes --with-argo)
• Mock OIDC for local dev: deploy/kind/mock-oidc.yaml (replaced in prod by Google login)
• Universe Deployment plan: /plans/universe-deployment.md (parent context — this plan is the concrete GKE landing of that broader effort)
• Universe Onboarding plan: /plans/universe-onboarding.md (end-user flow that depends on this deployment being live)

Non-Goals

Explicitly out of scope for this plan; do not let them grow it:

• Bucket-backed primary storage for demarkus-server (S3 backend). Deferred — discussed and rejected for now in favor of PVC + restic backups. Revisit only if multi-writer or DR portability becomes a hard requirement.
• Multi-cluster / multi-region active-active. Single region (northamerica-northeast2, Toronto), single cluster.
• Custom OIDC IdP. Google login only at launch; broker's generic OIDC support keeps the swap cheap later.
• Migrating existing soul deployments onto this GKE cluster. This is greenfield; existing personal souls keep running where they run.

Addendum — Phase 9 implementation note (2026-05-29)

Phase 9 shipped as CSI VolumeSnapshots, not the restic CronJob the Locked Decisions table named. Rationale uncovered during build:

• demarkus-server 0.17.11 exposes only PVC storage (no object-storage backend), and the data is a hash-chained versioned store on RWO PDs. restic would force a snapshot→clone→mount→Job orchestrator (CSI snapshots are namespaced and a clone PVC + restic pod must live in the source namespace). CSI snapshots make it a one-object backup: crash-consistent, node-independent, zero OpenTofu / no GCS bucket / no Workload Identity.
• World namespaces are labeled demarkus.io/backup=true via the worlds ApplicationSet's managedNamespaceMetadata, so new worlds are backed up with no per-world config. Implementation: apps/backups/, docs/runbook-backup-restore.md.

Tradeoff accepted: GCE-snapshot data is GCP-locked (no off-GCP restore) and Google-managed-key encrypted (not client-side restic encryption). Consistent with the stack's existing all-in-on-GCP posture.

New Open Item (deferred): native object-storage (GCS/S3) backend for demarkus-server. This is the right long-term home for the "bucket-backed storage, backups built-in via versioning" idea (and what GCS FUSE can't safely fake for a transactional store). It's a demarkus-server/core feature, out of scope for this deploy-layer plan. When it lands, worlds get built-in bucket durability and the CSI snapshots for world content can be retired (OpenBao, still PVC-backed, would keep snapshot backups).

Addendum — Phase 10 status + announcement deferred (2026-05-29)

Phase 10 docs shipped: README rewritten (live endpoint, GitOps + sync-wave architecture, cost baseline, repo layout, runbook index) and docs/instantiate.md added (fork→live orchestration guide with the deployment-specific find/replace checklist, linking the existing runbooks). Repo marked as a GitHub template (description + topics set). Architecture folded into the README; no separate docs/architecture.md.

Announcement blog post (demarkus.io/blog): DEFERRED — intentionally held. Get real miles on the live system and reevaluate a few things first, then write the post from operating experience rather than launch-day optimism. This is the sole remaining Phase 10 item; everything else in phases 1–10 is done. Revisit after a soak period.

Status — COMPLETE except deferred launch blog (verified 2026-05-31)

Corrects the stale /index.md pointer ("Phase 1 next"). Phases 1-10 are done. Verified against the live deploy repo latebit-io/demarkus-knowledge-system-deploy (public, isTemplate, pushed 2026-05-30):

• Tofu (P1-P3): project/network/dns/gke/argocd-bootstrap modules (+ platform-iam, billing-budget); envs/prod/ populated.
• ArgoCD (P4): bootstrap/root-appset.yaml + values.
• Platform (P5): cert-manager, openbao, bank-vaults-webhook, external-dns, ingress-nginx (+ external-secrets, dex, oauth2-proxy).
• OpenBao seed (P6): scripts/seed-openbao.sh + runbook.
• Broker + worlds (P7): apps/demarkus-broker/ + apps/demarkus-worlds/applicationset.yaml.
• CI (P8): tofu-plan.yml/tofu-apply.yml; a tofu-apply on main succeeded 2026-05-29.
• Backups (P9): apps/backups/ via CSI VolumeSnapshots — a documented pivot from the planned restic→GCS (accepted tradeoff: GCP-locked, Google-managed keys; native object-storage backend filed as a deferred core item).
• Polish (P10): README rewrite, docs/instantiate.md, template flag set.

Live corroboration: https://knowledge.demarkus.io/.well-known/oauth-authorization-server returns valid RFC 8414 metadata over TLS (issuer broker.knowledge.demarkus.io, device-code grant, RS256 JWKS) — the exact Phase 7 acceptance test, passing against the real domain.

Sole remaining item: the announcement blog post, intentionally deferred for a soak period.