soul.demarkus.io/plans/universe-deployment.md/v18 draft reader meta

Plan: Universe Deployment

Ship the enterprise-grade deployment for demarkus. The deliverable is the deployment: a customer's ops team installs a Helm chart, runs a broker, runs an agent, and has a federated demarkus universe their org can use. Any company evaluating demarkus (their internal "POC") installs the same product an established customer runs in production. There is no separate "POC slice" — the slice mentality is rejected. We build it once, right, and customers trial the real thing.

Goal

Deliver a complete, supportable, production-grade Kubernetes deployment package for demarkus, including:

  1. A Helm chart for demarkus-server (one world). ✓
  2. A Helm chart for demarkus-broker (OIDC token issuance + revocation). ✓
  3. A Helm chart for demarkus-agent (hub aggregator, crawl-and-index). ✓
  4. Reference topology examples (Argo CD ApplicationSet, Kustomize overlay).
  5. Backend-agnostic observability — structured slog emission + reference docs for ingestion. ✓ (skinny — schema doc + per-service JSON-by-default; per-backend recipe files deliberately not shipped)
  6. Customer-facing documentation — installation, security/threat model, operations, upgrade path, per-provider OIDC setup, observability recipes.
  7. A release pipeline producing images and chart releases consumable from GHCR. ✓

The same artifacts power first-customer trial and steady-state operations.

Non-Goals (Phase 7+ territory)

  • Multi-replica worlds with shared storage (RWX / object backend).
  • Cross-cluster universe federation.
  • Operator with a World CRD.
  • Hosted / managed SaaS.
  • Non-markdown content.

Constraints

  • No core protocol/server changes. Period. Per prior precedent (Claude Code plugin, Obsidian plugin, feedback_plugin_scope.md). Observability is achieved by log derivation in a collector, not by adding /metrics or OTel SDK calls to the server. Note: in Slice A we promoted HashToken and the token-mint library to protocol/ (under protocol/auth.go and protocol/token/). These are additive helper relocations consumed by server + CLI + future broker; no wire-protocol or server behavior changed. Slice C.2 added protocol/token.ParseBytes in the same additive spirit — a read-side helper so the broker's drift sweeper can inspect a world's tokens.toml payload via the map shape rather than substring matching on serialized TOML. §6.7.0 hoisted server/internal/store/ → protocol/store/ in the same additive spirit — the disk-shape of versioned content lives at the protocol layer alongside token/ and the HashToken contract; no API change, four import-path rewrites.
  • Capability-based auth model is non-negotiable. The server never learns identity — only labels.
  • Markdown-only scope is non-negotiable.

Decisions (resolved during planning)

[Unchanged decisions retained — see plan v17 for full list. New decisions below.]

  • §6.5 ships the schema doc, not per-backend recipes (Fritz cost/value pushback, 2026-05-13). Original §6.5 scope was 5 per-backend log-shipper config dirs (Datadog / OTel Collector / Vector / Fluent Bit / Grafana Alloy), ~700-1000 lines, ~10 hours. Fritz questioned whether observability recipes advance the knowledge universe at all — honest answer: they're operator ergonomics, not product value. Worse, the recipes would be aspirational (no real backends to verify against) and would drift as backend config languages evolve. Switched to the schema-doc-only approach: docs/deployment/observability.md documents the slog field shape demarkus actually emits (one row per significant msg across server/broker/agent, key fields, what it means, operator queries), plus pointers to each backend's own JSON-log ingestion docs. The schema doc stays correct because it's the actual code path; the per-backend pointers stay correct because we delegate to the moving target instead of copying it. ~1 hour total vs ~10 hours. Pattern: when you can't verify a thing end-to-end, ship the stable contract (the schema) and reference the moving-target thing (backend docs) by link.
  • Agent slog migration, strict env-var validation only on the new surface (§6.5 skinny, PR #125). client/cmd/demarkus-agent/main.go converted from stdlib log to log/slog so all three runtime services emit structured JSON when DEMARKUS_LOG_FORMAT=json. Strict validation (typo → os.Exit(2)) on the agent's env var because it's brand-new in this PR; nobody has it set on a deployed agent, so flipping from lenient to strict has no compat surface. Server's DEMARKUS_LOG_FORMAT stays lenient with the silent-fallback behavior it's shipped with since first wiring — existing server deployments that have a typo set would break on upgrade if we changed it to strict-error. logging.New library function stays lenient too; the existing TestNew "unknown format defaults to text" assertion keeps passing. Different env-var histories deserve different treatment even when the name is shared.
  • Server + agent charts default DEMARKUS_LOG_FORMAT: json (§6.5 skinny). Chart-side override of the binary's text default so production deployments emit machine-parseable JSON without operator intervention. Binary defaults stay text (local-dev human-readable). Broker is hardcoded JSON in tools/demarkus-broker/main.go; doesn't expose an override (was that way before §6.5; not changed).

Open Questions

  1. First customer trial. Nesto (*.library.nesto.ca) is path-B — trial waits for product. Trial runbook lands at /trials/nesto.md when scoping starts.
  2. Two-week deadline (2026-05-27): §6.7 and §6.5 (skinny) landed; §6.4 and §6.6 remain on the original Phase 6 trajectory. ~13 days runway. But Fritz's pushback on §6.5 raised a broader question: are §6.4 (topology examples) and parts of §6.6 (operations docs) also "ops polish that doesn't advance the knowledge universe"? Open question for Fritz at session resume: keep grinding Phase 6 polish, or pivot to a knowledge-universe-shaped initiative (/plans/conflict-merge.md, /plans/claude-code-plugin.md, obsidian plugin work, agent/federation features, TUI/CLI ergonomics)?
  3. First module-release tag firing of the §6.7.B pipeline. Triggered automatically by the next push to main that touches server/, client/, or tools/. PR #125 touched client/ and deploy/helm/, so the next push to main following that should fire client/v* and produce the first GHCR images + OCI chart bundle for the demarkus-agent slice. The tools/v* and server/v* fires wait on their respective module paths receiving a feat:/fix: commit.
  4. Hardened broker chart symmetry (deferred from §6.3.D.2). The broker chart's secret-issuances.yaml uses the simpler lookup-skip pattern. Defense-in-depth follow-up; not load-bearing. Only act on if Fritz brings it up.

Repository Layout

Reflects state as of §6.5 skinny merge (2026-05-13):

deploy/
  helm/
    demarkus-server/         # one-world chart (Phase 6.1) — values.yaml has
                             # server.logFormat: json default (§6.5 PR #125)
    demarkus-broker/         # OIDC token broker chart (Phase 6.3)
    demarkus-agent/          # crawl/index agent chart (Phase 6.0) — values.yaml
                             # has logFormat: json default (§6.5 PR #125)
  test-upgrade-wipe.sh       # kind regression: §6.3.D.2 PR #119
  k8s/
    examples/
      applicationset.yaml    # Argo CD ApplicationSet (§6.4 — pending)
      kustomize-overlay/     # Kustomize alternative (§6.4 — pending)
  observability/             # DELIBERATELY EMPTY. Per-backend recipe files
                             # were scoped but skipped — see §6.5 §Decisions.
                             # Operator-facing schema doc lives at
                             # docs/deployment/observability.md instead.

docs/
  deployment/
    observability.md         # slog field schema for server/broker/agent
                             # + operator query patterns + backend ingestion
                             # pointers (§6.5 PR #125)
    [pending §6.6 docs]      # install / security / operations / per-chart READMEs

.dockerignore                # repo-root, applies to all three image builds
                             # (§6.7.A PR #121)

.github/
  workflows/
    release.yml              # full per-module release pipeline:
                             # - server/v*  → goreleaser archives
                             #               + docker buildx push demarkus-server image
                             #               + helm package + push OCI chart
                             # - client/v*  → goreleaser archives
                             #               + docker buildx push demarkus-agent image
                             #               + helm package + push OCI chart
                             # - tools/v*   → goreleaser archives (broker+token+publish)
                             #               + docker buildx push demarkus-broker image
                             #               + helm package + push OCI chart
                             # (§6.7.B PRs #122 + #123 + #124)
    ci.yml                   # PR + push tests; test-broker + test-charts +
                             # test-charts-kind (§6.3.D.1, §6.3.D.2)

protocol/
  auth.go                    # HashToken — sha256-<hex> contract (Slice A)
  token/                     # Generate, ReadFile, AppendEntry, WriteFile,
                             # FormatEntry, flock helpers (Slice A) +
                             # AppendBytes, RemoveBytes in-memory (Slice B) +
                             # ParseBytes (Slice C.2).
  store/                     # versioned content-store primitives, hoisted
                             # from server/internal/store (§6.7.0 PR #120).

server/
  Dockerfile                 # multi-stage; bundles demarkus-server + CLI
                             # (§6.7.A PR #121)
  .goreleaser.yml            # binary archives only
  cmd/
    demarkus-server/         # uses logging.New (lenient on DEMARKUS_LOG_FORMAT)

client/
  .goreleaser.yml            # binary archives for demarkus, demarkus-tui,
                             # demarkus-mcp (demarkus-agent ships via Dockerfile
                             # build in release-client)
  cmd/
    demarkus/
    demarkus-tui/
    demarkus-mcp/
    demarkus-agent/          # slog with strict DEMARKUS_LOG_FORMAT validation
                             # (§6.5 PR #125)
      Dockerfile             # (§6.7.A PR #121)

tools/
  .goreleaser.yml            # binary archives for demarkus-broker,
                             # demarkus-token, demarkus-publish (§6.7.B PR #123)
  demarkus-broker/
    Dockerfile               # (§6.7.A PR #121)
  demarkus-token/            # admin CLI (no Dockerfile — archive-only)
  demarkus-publish/          # admin CLI (no Dockerfile — archive-only)

Sub-Phases

6.0 — demarkus-agent verified + chart ✓ (merged PR #106) + §6.5 slog migration ✓

Existing demarkus-agent binary verified end-to-end (2 team worlds + 1 hub smoke test) and chart at deploy/helm/demarkus-agent/ shipped. Stateless Deployment; ConfigMap holds TOML agent config; Secret holds per-host tokens; exec liveness probe; outbound-only (no Service). §6.7.A PR #121 fixed a pre-existing structural bug in the agent deployment test suite. Agent ships per-image (ghcr.io/latebit-io/demarkus-agent) with CA bundle for outbound TLS, released via client/v* tag flow (§6.7.B PR #122 + #124). §6.5 PR #125 converted agent from stdlib log to log/slog with strict DEMARKUS_LOG_FORMAT validation — agent now emits structured JSON consistent with server and broker when running in k8s.

6.1 — demarkus-server Helm chart ✓ (merged PR #107) + §6.3.D.2 ✓ + §6.7.A image fix ✓ + §6.7.B release ✓ + §6.5 chart default ✓

deploy/helm/demarkus-server/ — StatefulSet, 1 replica, volumeClaimTemplates, exec probes against /.well-known/agent-manifest.md, Service type LoadBalancer protocol: UDP. helm-unittest test files at tests/ wired into CI by §6.3.D.1 PR #118 and validated end-to-end against a kind cluster by §6.3.D.2 PR #119. §6.5 PR #125 added server.logFormat: json chart default so production deployments get structured JSON without operator intervention; binary's standalone default stays text (lenient, unchanged for backwards compat).

6.2 — demarkus-broker binary (tools/demarkus-broker/) ✓ complete

[Unchanged.]

6.3 — demarkus-broker Helm chart ✓ complete (all five sub-slices merged) + §6.7.B release ✓

[Unchanged. Broker has emitted JSON by default since it shipped.]

6.4 — Universe topology examples (pending — possibly defer per Fritz pushback)

ApplicationSet + Kustomize overlay at deploy/k8s/examples/. Both reference the chart pull URLs from §6.7.B (now live). Argo CD ApplicationSet template generator iterates a worlds: list and templates one Application per world. Kustomize overlay is the GitOps-without-Argo alternative. Customer's ops team picks one. Status note (post-§6.5): Fritz pushed back on §6.5 as ops polish that doesn't advance the knowledge universe; the same critique applies to §6.4. Open question whether to ship a skinny version (single ApplicationSet example without dashboards), defer entirely until a customer asks for their specific GitOps shape, or pivot to a knowledge-universe-shaped initiative.

6.5 — Observability ✓ (skinny — schema doc + per-service JSON-by-default; per-backend recipes deferred)

§6.5 (skinny) ✓ merged 2026-05-13 (PR #125, commit 1643806). Three pieces:

  • client/cmd/demarkus-agent/main.go converted from stdlib log to log/slog. Strict DEMARKUS_LOG_FORMAT validation (typo → exit 2) since the agent's env-var contract is brand-new in this PR.
  • Server + agent helm charts default DEMARKUS_LOG_FORMAT: json in the rendered Deployment env block. Binary defaults stay text (local-dev human-readable); chart override is what production sees.
  • docs/deployment/observability.md (136 lines) — single-doc reference with field schema for server/broker/agent (one row per significant msg), operator query patterns ("count by verb", "mint rate", "sweeper health"), backend ingestion pointers (Datadog Agent / OTel filelog receiver / Vector / Fluent Bit / Grafana Alloy), and a "what's NOT in the logs" section (no raw subjects/emails — subject hashing pinned; no tokens, no bodies, no /metrics endpoint).

Per-backend recipe files deliberately not shipped. Original §6.5 scope called for deploy/observability/{datadog,otel-collector,vector,fluent-bit,grafana-alloy}/ directories with primary config files + READMEs. Skipped because (a) the recipes would be aspirational (no real backends to verify against), (b) they'd drift as backend config languages evolve, (c) Fritz pushback identified them as ops ergonomics that don't advance the knowledge universe. Schema doc + backend doc-pointers stay correct because we delegate to the moving target. ~1 hour of work vs ~10 hours for the full scope.

6.6 — Documentation suite (pending — incremental; possibly defer parts per Fritz pushback)

docs/deployment/*.md (location confirmed 2026-05-13). Per-chart READMEs at deploy/helm/<chart>/README.md (broker README exists from §6.3.C; server + agent need similar treatment). Install guide (per-provider OIDC setup for Google / Okta / Entra ID / Auth0; chart install order: server → broker → agent; cert-manager + DNS topology). Security/threat model. Operations (upgrade path including the now-tested resource-policy:keep contract, backup/DR via Velero or demarkus-agent sync). Observability now links to docs/deployment/observability.md (§6.5 skinny landed). Status note: the "what is demarkus / how does federation work" product docs advance the knowledge universe; the "Velero backup runbook / OIDC per-provider step-by-step" operations docs are ops polish. Open question on slicing; can be split between knowledge-universe-shaped docs (keep) and operations runbooks (defer per same critique as §6.5/§6.4).

6.7 — Release pipeline ✓ complete

[Unchanged.]

Sequencing

  1. Slice A — token-mint library ✓ merged 2026-05-11 (PR #108).
  2. 6.0 chart ✓ merged 2026-05-11 (PR #106). §6.5 slog migration ✓ merged 2026-05-13 (PR #125).
  3. 6.1 server chart ✓ merged 2026-05-11 (PR #107). §6.3.D.2 ✓. §6.7.A ✓. §6.7.B ✓. §6.5 chart default ✓.
  4. 6.2 broker binary — Slices B + C.1–C.4 ✓ all merged 2026-05-11–12.
  5. 6.3 broker chart ✓ all five sub-slices merged 2026-05-13.
  6. 6.7.0 store hoist ✓ merged 2026-05-13 (PR #120).
  7. 6.7.A image consolidation + CLI relocation ✓ merged 2026-05-13 (PR #121).
  8. 6.7.B release pipeline ✓ three sub-PRs merged 2026-05-13–14 (PRs #122, #123, #124).
  9. 6.5 observability ✓ skinny version merged 2026-05-13 (PR #125, commit 1643806).
  10. 6.4 topology examples — pending; Fritz-pushback evaluation still open.
  11. 6.6 docs — pending; partial scope (product docs) vs ops runbooks split still open.

Rough effort: ~5-7 days of focused work remaining for whatever subset of §6.4 + §6.6 lands against the 2026-05-27 deadline. Phase 6 ~90% complete by sub-phase count (6.0/6.1/6.2/6.3/6.5/6.7 done; 6.4/6.6 remaining and partly under reframe).

Backlog (deferred, easy to add later)

[Unchanged from v17.]

  • Per-backend observability recipe files (§6.5 skinny deferred this). If a specific customer trial surfaces a request for a paved-path vector.yaml or otel-collector.yaml, ship it on-demand for their specific stack. Maintenance cost is real (configs drift with backend versions); only worth doing when the use case is concrete.

Risks

[Unchanged from v17.]

  • Schema doc drift. The slog field schema documented in docs/deployment/observability.md will drift if log lines change without the doc being updated. Mitigation: every PR that adds/changes a slog call should update the schema doc. Low-risk for the established log surface (request, archive, audit events on server; mint/revoke/rotate/sweep on broker; crawl on agent); higher risk if a future feature adds new log lines.

Status

Plan v18, 2026-05-13. Phase 6 ~90% complete. Slice A, §6.0 chart + slog, §6.1 chart (D.2 + 6.7.A + 6.7.B + 6.5 chart default), §6.2 broker binary, §6.3 broker chart (5 sub-slices), §6.7.0 store hoist, §6.7.A image consolidation + CLI relocation, §6.7.B release pipeline (3 sub-PRs), §6.5 observability (skinny) all merged. 15 PRs merged 2026-05-13–14 (#115 through #125).

§6.5 skinny trajectory:

  • Original scope was 5 per-backend log-shipper recipe dirs (~10h, ~1000 lines). Fritz cost/value pushback: "what's the benefit of observability to the knowledge universe?"
  • Reframed scope is 3 pieces: agent slog migration, server+agent chart JSON default, single schema doc. ~1h, ~14 lines code + 136 lines docs.
  • Net product effect: all three runtime services emit identical structured JSON in production; operators have an authoritative field schema reference; no aspirational backend YAMLs sitting in the repo to maintain.

Open question for Fritz at session resume: keep grinding Phase 6 polish (§6.4 + §6.6), or pivot to a knowledge-universe-shaped initiative? §6.4 (topology examples) is operator-facing ops polish — same critique as §6.5 originally. §6.6 (docs) splits — product docs (security model, federation explanation, per-provider OIDC) advance the knowledge universe; operations runbooks (Velero, upgrade procedures) are ops polish. Alternatives that move the knowledge universe directly: /plans/conflict-merge.md (content-loss reduction in mark_publish), /plans/claude-code-plugin.md (distribution win), Obsidian plugin polish (standalone repo, end-user surface), agent/federation feature work, TUI/CLI ergonomics.

trail
  1. soul.demarkus.io v18