# kind bench of the merged knowledge binary

First load and memory runs of `demarkus-knowledge` 0.51.0 (knowledge server, broker and both MCP gateways in one process) on a local kind cluster, 2026-10-01. Audience: demarkus maintainers. Scope is an early read, not capacity numbers for GKE; the plan is [Merged binary load test](/plans/merged-binary-load-test.md), the pre-merge production picture is the [GKE memory baseline](/research/gke-memory-baseline.md).

## Setup

- Cluster `demarkus-bench` from `deploy/kind/up.sh --with-knowledge --with-mcp-smoke` with the released image `ghcr.io/latebit-io/demarkus-knowledge:0.51.0` (32 smoke checks passed), then `tools/benchmarks/kind/values-bench.yaml`: 3 replicas, 25m CPU and 256Mi requested, 1 CPU and 1Gi limit as in production, `GODEBUG=gctrace=1`, and a bootstrapped publish token.
- One world, `world-default`, in fake-gcs-server 1.52.2 (default filesystem backend). Mock OIDC for logins. Rate limits off.
- Host: Apple silicon, Docker VM with 8 CPUs and 12 GiB, one kind node shared by the load generator, fake-gcs-server and the three pods.
- Load from `tools/demarkus-loadtest` in an in-cluster pod; memory from the pods' cgroups every 5 s (working set as kubelet computes it); live heap from gctrace. Harness and raw output on branch `bench/merged-binary-kind` (`tools/benchmarks/kind/`).

## Reads

| Step | Workers | req/s | p50 | p99 |
|---|---|---|---|---|
| QUIC FETCH, pooled connections | 1 | 53 | 20 ms | 26 ms |
| | 4 | 160 | 21 ms | 45 ms |
| | 16 | 159 | 100 ms | 166 ms |
| | 64 | 158 | 423 ms | 486 ms |
| | 256 | 151 | 1.77 s | 1.79 s |
| QUIC FETCH, fresh connection per request | 16 | 157 | 23 ms | 323 ms |
| | 64 | 159 | 343 ms | 1.16 s |
| QUIC LOOKUP | 16 | 159 | 101 ms | 124 ms |
| | 64 | 158 | 501 ms | 506 ms |
| QUIC LIST | 16 | 159 | 80 ms | 185 ms |
| MCP `mark_fetch` | 1 | 53 | 20 ms | 26 ms |
| | 8 | 160 | 60 ms | 85 ms |
| | 32 | 160 | 260 ms | 268 ms |
| MCP `mark_lookup_all` (one world) | 1 | 53 | 20 ms | 27 ms |
| | 8 | 160 | 40 ms | 104 ms |
| | 32 | 160 | 240 ms | 287 ms |

Zero errors in about 65,000 requests. The three pods together used 0.12 to 0.37 cores and 65 to 118 MiB; no pod passed 51 MiB.

- **The ceiling is fake-gcs-server, not the server.** Throughput flattens at about 160 req/s from 4 workers on while the pods idle at a fraction of a core; more workers only queue.
- **Every read costs two bucket round trips.** `OpenReadView` stats the world's head object (`refreshSnapshot`) before each read, then reads the object, so even a LOOKUP answered from the in-memory catalog waits on the bucket. That is the 20 ms floor here and sets the floor on GCS too. The server's own read capacity needs the GKE scratch pool.
- **Handshakes are cheap.** A fresh connection per request raised CPU from 0.23 to 0.30 cores at the same rate, about 0.5 ms of server CPU per QUIC handshake.
- **The MCP gateway adds nothing visible** over direct QUIC: identical latency at one worker.
- **Connections are cheap in memory**: 256 open connections took the three pods from about 65 to 117 MiB in total.

## Writes

- **Commit pacing caps a world at about 0.67 writes/s.** `bucketstore` waits `defaultCommitInterval` (1.5 s) between head-object commits, to stay under GCS's per-object update limit. One sequential writer measured exactly 1.51 s per publish when back to back; 4 concurrent writers queued to 6 s each; a 4-writer burst of 200 publishes lost 49 to the 10 s request timeout. Spaced below the pacing, a write takes about 60 ms.
- **Soak, 30 minutes:** one writer at 0.5 writes/s (about 18 times production's agent) over 20 paths with retention 20, so every write after minute 13 pruned. 899 writes, 0 errors, p50 58 ms, p99 86 ms. The writer pod's working set stayed between 25 and 54 MiB; live heap went from 7 MB to 9 MB by minute 15 and stayed there. The other two pods held 24 to 36 MiB. The [writer memory fix](/plans/knowledge-writer-memory.md) holds under write load; before it, kind grew 25 to 357 MiB over 630 writes.
- All writes landed on one pod, as in production.

## MCP session churn

5 new sessions a second for 40 minutes, each running initialize, initialized and one `mark_fetch` and never sending DELETE: 11,999 sessions, 0 errors, p50 10 ms for the three calls. Kube-proxy balances per TCP connection and the client reuses connections, so every session landed on one pod.

| Time | Live heap | Working set | Sweeps |
|---|---|---|---|
| start | 7 MB | 56 MiB | 4 |
| +10 min | 42 MB | 103 MiB | |
| +20 min | 72 MB | 141 MiB | |
| +30 min | 96 MB | 199 MiB | 47 |
| 14:36 sweep, under load | 112 then 99 MB | 230 to 258 MiB | 1,953 |
| load ended +40 min | 104 MB | 237 MiB | |
| 14:51 sweep, idle | 107 then 64 MB | 195 then 156 MiB | 6,453 |

- **About 9.6 KB of live heap per session**, matching the earlier harness on mcp-go.
- **The idle TTL bounds it.** Each sweep removed exactly the sessions it should (1,906 then 4,500), and live heap fell with them.
- **Sessions live 30 to 45 minutes, not 30.** mcp-go sweeps every TTL/2, so steady-state live sessions are the new-session rate times 30 to 45 minutes. At 5 a second that is up to 13,500 sessions and about 130 MB of live heap on one pod; working set peaked at 263 MiB. Production's 40 to 99 sessions an hour is under 1 MB. One subject at the production limit of 120 calls a minute can hold about 2,700 sessions (about 27 MB); many subjects multiply that, and 1 GiB is reached around 15 to 20 new sessions a second sustained.
- **Log spam past 4,096 live sessions.** The fetch dedup store caps at 4,096 sessions and logs a WARN on every eviction: 5,307 lines in 20 minutes, one per new session. It only triggers above about 1.5 new sessions a second, but under a burst it is one line per session.
- kubelet rotated the pod log every 10 MB (about 30 minutes at this rate, mostly the per-request INFO line).

## Findings to act on

1. Read latency is bound by two bucket round trips per request; measure on GKE with real GCS before choosing whether a short head-stat cache is worth its staleness.
2. A world takes at most about 0.67 writes/s; concurrent writers to one world queue and time out past six or so. Fine for the agent, worth knowing for bursty multi-agent writes.
3. Session memory is bounded by the TTL but scales with the session rate, with a 45 minute worst case; a shorter TTL or a session cap would tighten it.
4. Rate-limit the dedup cap WARN (log once per interval with a count).

## Reproduce

`tools/benchmarks/kind/README.md` on branch `bench/merged-binary-kind`: cluster, overlay, load generator pod, seed, `scenarios.sh reads tools writes churn`, `summarize.py`.