kind bench of the merged knowledge binary

First load and memory runs of demarkus-knowledge 0.51.0 (knowledge server, broker and both MCP gateways in one process) on a local kind cluster, 2026-10-01. Audience: demarkus maintainers. Scope is an early read, not capacity numbers for GKE; the plan is Merged binary load test, the pre-merge production picture is the GKE memory baseline.

Setup

• Cluster demarkus-bench from deploy/kind/up.sh --with-knowledge --with-mcp-smoke with the released image ghcr.io/latebit-io/demarkus-knowledge:0.51.0 (32 smoke checks passed), then tools/benchmarks/kind/values-bench.yaml: 3 replicas, 25m CPU and 256Mi requested, 1 CPU and 1Gi limit as in production, GODEBUG=gctrace=1, and a bootstrapped publish token.
• One world, world-default, in fake-gcs-server 1.52.2 (default filesystem backend). Mock OIDC for logins. Rate limits off.
• Host: Apple silicon, Docker VM with 8 CPUs and 12 GiB, one kind node shared by the load generator, fake-gcs-server and the three pods.
• Load from tools/demarkus-loadtest in an in-cluster pod; memory from the pods' cgroups every 5 s (working set as kubelet computes it); live heap from gctrace. Harness and raw output on branch bench/merged-binary-kind (tools/benchmarks/kind/).

Reads

Zero errors in about 65,000 requests. The three pods together used 0.12 to 0.37 cores and 65 to 118 MiB; no pod passed 51 MiB.

• The ceiling is fake-gcs-server, not the server. Throughput flattens at about 160 req/s from 4 workers on while the pods idle at a fraction of a core; more workers only queue.
• Every read costs two bucket round trips. OpenReadView stats the world's head object (refreshSnapshot) before each read, then reads the object, so even a LOOKUP answered from the in-memory catalog waits on the bucket. That is the 20 ms floor here and sets the floor on GCS too. The server's own read capacity needs the GKE scratch pool.
• Handshakes are cheap. A fresh connection per request raised CPU from 0.23 to 0.30 cores at the same rate, about 0.5 ms of server CPU per QUIC handshake.
• The MCP gateway adds nothing visible over direct QUIC: identical latency at one worker.
• Connections are cheap in memory: 256 open connections took the three pods from about 65 to 117 MiB in total.

Writes

• Commit pacing caps a world at about 0.67 writes/s. bucketstore waits defaultCommitInterval (1.5 s) between head-object commits, to stay under GCS's per-object update limit. One sequential writer measured exactly 1.51 s per publish when back to back; 4 concurrent writers queued to 6 s each; a 4-writer burst of 200 publishes lost 49 to the 10 s request timeout. Spaced below the pacing, a write takes about 60 ms.
• Soak, 30 minutes: one writer at 0.5 writes/s (about 18 times production's agent) over 20 paths with retention 20, so every write after minute 13 pruned. 899 writes, 0 errors, p50 58 ms, p99 86 ms. The writer pod's working set stayed between 25 and 54 MiB; live heap went from 7 MB to 9 MB by minute 15 and stayed there. The other two pods held 24 to 36 MiB. The writer memory fix holds under write load; before it, kind grew 25 to 357 MiB over 630 writes.
• All writes landed on one pod, as in production.

MCP session churn

5 new sessions a second for 40 minutes, each running initialize, initialized and one mark_fetch and never sending DELETE: 11,999 sessions, 0 errors, p50 10 ms for the three calls. Kube-proxy balances per TCP connection and the client reuses connections, so every session landed on one pod.

• About 9.6 KB of live heap per session, matching the earlier harness on mcp-go.
• The idle TTL bounds it. Each sweep removed exactly the sessions it should (1,906 then 4,500), and live heap fell with them.
• Sessions live 30 to 45 minutes, not 30. mcp-go sweeps every TTL/2, so steady-state live sessions are the new-session rate times 30 to 45 minutes. At 5 a second that is up to 13,500 sessions and about 130 MB of live heap on one pod; working set peaked at 263 MiB. Production's 40 to 99 sessions an hour is under 1 MB. One subject at the production limit of 120 calls a minute can hold about 2,700 sessions (about 27 MB); many subjects multiply that, and 1 GiB is reached around 15 to 20 new sessions a second sustained.
• Log spam past 4,096 live sessions. The fetch dedup store caps at 4,096 sessions and logs a WARN on every eviction: 5,307 lines in 20 minutes, one per new session. It only triggers above about 1.5 new sessions a second, but under a burst it is one line per session.
• kubelet rotated the pod log every 10 MB (about 30 minutes at this rate, mostly the per-request INFO line).

Findings to act on

1. Read latency is bound by two bucket round trips per request; measure on GKE with real GCS before choosing whether a short head-stat cache is worth its staleness.
2. A world takes at most about 0.67 writes/s; concurrent writers to one world queue and time out past six or so. Fine for the agent, worth knowing for bursty multi-agent writes.
3. Session memory is bounded by the TTL but scales with the session rate, with a 45 minute worst case; a shorter TTL or a session cap would tighten it.
4. Rate-limit the dedup cap WARN (log once per interval with a count).

Reproduce

tools/benchmarks/kind/README.md on branch bench/merged-binary-kind: cluster, overlay, load generator pod, seed, scenarios.sh reads tools writes churn, summarize.py.

Correction: the read ceiling is the rate limiter, not the bucket (2026-10-01)

The Reads section above is wrong about the cause. fake-gcs-server answers a head-object metadata read or an object read in under 1 ms, and its request log shows a whole FETCH taking about 1 to 2 ms of bucket time. The 20 ms per request at one worker is exactly 1/50 s: worldruntime.admit paces every request with the per-IP token bucket (DEMARKUS_RATE_LIMIT, default 50 req/s, burst 100), one bucket per world per pod, and it waits instead of refusing. One worker on one connection gets about 53 req/s; three pods give about 160. In-process MCP calls pass localAddr{}, so every MCP caller on a pod shares one 50 req/s bucket per world, by design (it mirrors the old broker pod's single IP). The read numbers above therefore measure the limiter; the server's real read capacity is still unmeasured, and the next run sets DEMARKUS_RATE_LIMIT=0.

The bucket calls per request, from fake-gcs-server's log:

• FETCH: four calls, a metadata read of head.json (refreshSnapshot), then the manifest, the history chunk and the blob, each content-addressed by sha256 and hash-verified.
• LOOKUP and LIST: the head.json metadata read only; the catalog and directory come from the in-memory snapshot.

Why the head check runs on every read: replicas share one bucket and do not coordinate, and the snapshot is cached per head generation. Checking the generation before each request gives read-your-writes across replicas: a write committed through one pod is visible on the next read at any pod. Peer hints (best-effort UDP) and the 5 s poll only serve watchers, so they cannot guarantee that. Why the object reads: the snapshot keeps only path to manifest references, so memory stays bounded for large worlds; histories and bodies are read on demand and verified against their hashes.