kind bench of the merged knowledge binary

First load and memory runs of demarkus-knowledge 0.51.0 (knowledge server, broker and both MCP gateways in one process) on a local kind cluster, 2026-10-01. Audience: demarkus maintainers. Scope is an early read, not capacity numbers for GKE; the plan is Merged binary load test, the pre-merge production picture is the GKE memory baseline.

Setup

• Cluster demarkus-bench from deploy/kind/up.sh --with-knowledge --with-mcp-smoke with the released image ghcr.io/latebit-io/demarkus-knowledge:0.51.0 (32 smoke checks passed), then tools/benchmarks/kind/values-bench.yaml: 3 replicas, 25m CPU and 256Mi requested, 1 CPU and 1Gi limit as in production, GODEBUG=gctrace=1, and a bootstrapped publish token.
• One world, world-default, in fake-gcs-server 1.52.2 (default filesystem backend). Mock OIDC for logins. Rate limits off.
• Host: Apple silicon, Docker VM with 8 CPUs and 12 GiB, one kind node shared by the load generator, fake-gcs-server and the three pods.
• Load from tools/demarkus-loadtest in an in-cluster pod; memory from the pods' cgroups every 5 s (working set as kubelet computes it); live heap from gctrace. Harness and raw output on branch bench/merged-binary-kind (tools/benchmarks/kind/).

Reads

Zero errors in about 65,000 requests. The three pods together used 0.12 to 0.37 cores and 65 to 118 MiB; no pod passed 51 MiB.

• The ceiling is fake-gcs-server, not the server. Throughput flattens at about 160 req/s from 4 workers on while the pods idle at a fraction of a core; more workers only queue.
• Every read costs two bucket round trips. OpenReadView stats the world's head object (refreshSnapshot) before each read, then reads the object, so even a LOOKUP answered from the in-memory catalog waits on the bucket. That is the 20 ms floor here and sets the floor on GCS too. The server's own read capacity needs the GKE scratch pool.
• Handshakes are cheap. A fresh connection per request raised CPU from 0.23 to 0.30 cores at the same rate, about 0.5 ms of server CPU per QUIC handshake.
• The MCP gateway adds nothing visible over direct QUIC: identical latency at one worker.
• Connections are cheap in memory: 256 open connections took the three pods from about 65 to 117 MiB in total.

Writes

• Commit pacing caps a world at about 0.67 writes/s. bucketstore waits defaultCommitInterval (1.5 s) between head-object commits, to stay under GCS's per-object update limit. One sequential writer measured exactly 1.51 s per publish when back to back; 4 concurrent writers queued to 6 s each; a 4-writer burst of 200 publishes lost 49 to the 10 s request timeout. Spaced below the pacing, a write takes about 60 ms.
• Soak, 30 minutes: one writer at 0.5 writes/s (about 18 times production's agent) over 20 paths with retention 20, so every write after minute 13 pruned. 899 writes, 0 errors, p50 58 ms, p99 86 ms. The writer pod's working set stayed between 25 and 54 MiB; live heap went from 7 MB to 9 MB by minute 15 and stayed there. The other two pods held 24 to 36 MiB. The writer memory fix holds under write load; before it, kind grew 25 to 357 MiB over 630 writes.
• All writes landed on one pod, as in production.

MCP session churn

5 new sessions a second for 40 minutes, each running initialize, initialized and one mark_fetch and never sending DELETE: 11,999 sessions, 0 errors, p50 10 ms for the three calls. Kube-proxy balances per TCP connection and the client reuses connections, so every session landed on one pod.

• About 9.6 KB of live heap per session, matching the earlier harness on mcp-go.
• The idle TTL bounds it. Each sweep removed exactly the sessions it should (1,906 then 4,500), and live heap fell with them.
• Sessions live 30 to 45 minutes, not 30. mcp-go sweeps every TTL/2, so steady-state live sessions are the new-session rate times 30 to 45 minutes. At 5 a second that is up to 13,500 sessions and about 130 MB of live heap on one pod; working set peaked at 263 MiB. Production's 40 to 99 sessions an hour is under 1 MB. One subject at the production limit of 120 calls a minute can hold about 2,700 sessions (about 27 MB); many subjects multiply that, and 1 GiB is reached around 15 to 20 new sessions a second sustained.
• Log spam past 4,096 live sessions. The fetch dedup store caps at 4,096 sessions and logs a WARN on every eviction: 5,307 lines in 20 minutes, one per new session. It only triggers above about 1.5 new sessions a second, but under a burst it is one line per session.
• kubelet rotated the pod log every 10 MB (about 30 minutes at this rate, mostly the per-request INFO line).

Findings to act on

1. Read latency is bound by two bucket round trips per request; measure on GKE with real GCS before choosing whether a short head-stat cache is worth its staleness.
2. A world takes at most about 0.67 writes/s; concurrent writers to one world queue and time out past six or so. Fine for the agent, worth knowing for bursty multi-agent writes.
3. Session memory is bounded by the TTL but scales with the session rate, with a 45 minute worst case; a shorter TTL or a session cap would tighten it.
4. Rate-limit the dedup cap WARN (log once per interval with a count).

Reproduce

tools/benchmarks/kind/README.md on branch bench/merged-binary-kind: cluster, overlay, load generator pod, seed, scenarios.sh reads tools writes churn, summarize.py.