2026-06-12 — demarkus-library in-cluster deploy

Added the Universe Library to the cluster (PR #70, merged to main):

• apps/demarkus-library/applicationset.yaml — chart demarkus-library 0.1.0 from ghcr.io/latebit-io/charts, sync-wave 2, broker transport, ingress library.<domain> (letsencrypt-prod), home world root, replica 1 (in-memory sessions).
• apps/demarkus-library/external-secret.yaml — ESO library-oauth-client from OpenBao secret/library/oauth-client (property client_secret), reusing the cluster-scoped openbao ClusterSecretStore.
• deployment.yaml — library-web redirectURI swapped from local-e2e library.dev.test:8443 to https://library.knowledge.demarkus.io/auth/callback.
• docs/runbook-library.md — manual OpenBao steps: widen external-secrets policy with secret/data/library/*, seed the plaintext client secret (sha256 must equal deployment.yaml's clientSecretHash).

Key gotcha: the library's broker.url must be the MCP gateway origin (https://knowledge.demarkus.io, i.e. https://<domain>), NOT broker.<domain> — RFC 8414 metadata is served only on the gateway host (verified live; broker.<domain>/.well-known/oauth-authorization-server 404s), and the metadata routes authorize/token to broker.<domain> itself.

Validated: helm template render with interpolated prod values (env/secretKeyRef/ingress/Certificate correct), scripts/smoke-broker-web-clients.sh green, image+chart 0.1.0 confirmed on ghcr.

OpenBao seed + secret rotation (post-merge)

Original library-web plaintext was unrecoverable locally (hash-scanned shell history, Claude transcripts, ~/latebit — zero matches against the registered clientSecretHash), so took the runbook's rotation path instead of waiting on the password manager:

• Widened external-secrets policy with secret/data/library/* (now 4 paths).
• Generated fresh secret, seeded secret/library/oauth-client v1.
• Updated deployment.yaml clientSecretHash to e44adb5a… on branch chore/rotate-library-web-secret (awaiting user commit/merge — until the broker re-renders, login 401s invalid_client).
• ESO synced, pod Running 1/1, TLS cert Ready, https://library.knowledge.demarkus.io/ → 302 /login, /health 200.

Security note for cleanup: the OpenBao root token is in plain shell history (~/.zsh_history, export BAO_TOKEN=… — one live with root policy). Runbook says password-manager only; should be rotated/scrubbed.

Side finding — local soul (demarkus-memory plugin, localhost:6309) is broken: server binary predates LOOKUP verb, and plugin token (sha256-d5002dbb…) isn't in the server's tokens.toml (expects sha256-a905094e…); writes failing since ~June 10. The repo .mcp.json demarkus-soul server (soul.demarkus.io) is the working write path.