Bucket commit log: a 20 minute gate on real GCS

The commit-log store (main at 06568432, #538) run from GKE against a real GCS bucket with three replicas, a 100,000-document seed and about 20 minutes of write phases, to get the numbers the fake could not give: compaction speed, adopter cost, p99 under compaction, cold start, three writers. Audience: demarkus maintainers. Run 2026-10-04, about $5. Verdict: nothing lost (343,641 acknowledged versions read back, 0 gaps or breaks), one writer sustains 365 writes/s and the compactor keeps pace at 400 writes/s at this size; three replicas writing one world collapse to 94 writes/s with p50 5 s, which is the finding that needs work before phase 7; cold start and the 250 ms p99 need more parallelism and a node of their own to judge. Plan: the GCS gate; the local run it follows: local 1M scale test. Raw results under the ignored tools/benchmarks/artifacts/scale1m-2026-10-03/gcs-gate20/ (phase outputs, verify, server logs, counter samples, manifests); the replicas' per-call event files could not be copied out of the distroless pods.

Setup

• Cluster demarkus (northamerica-northeast2-a), a tainted scratch pool of one n2-standard-4 (4 vCPU, 16 GB), namespace gate: three replica pods A, B, C (scale-server, the real knowledgeserver.Open with a counting bucket wrapper, 1 vCPU requested, 3 to 4 GiB, GOMEMLIMIT 3.5 GiB, 16 shard workers), peer hints between them over QUIC, and a load pod running scale-load on the same node. Images from a scratch Artifact Registry repository; GCS through Workload Identity on a scratch service account.
• Bucket scale1m-gate-49722, Standard, same region, uniform access, soft delete off.
• The harness build shortened the checkpoint cadence to 2,048 slots or 2 minutes so the test yielded a dozen steps; the step bound (32,768 documents) and every other constant are the release's.
• Documents: agent-shaped markdown of about 4 KB with tags and metadata, blind publishes; updates hit random documents. The world grew from 100,000 to about 245,000 documents, so shards went from 512 to 1,024.

Results

No 429 or 5xx from GCS at any point; the stalls are not throttling. Slot wins over the whole run: A 19,471, B 879, C 727.

Compaction. The first step after genesis was written by all three replicas (checkpoint zero is exempt from the defer rule; A and C lost the checkpoint create), then B alone wrote the other twelve steps, 24k to 33k documents each, about every two minutes, with the lag at each landing under 10k at 200 writes/s and zero at 400 writes/s while one replica wrote; 23k to 35k while B wrote and compacted in the three-way phases. B's bucket operations over the run: 299k history block creates, 397k history block reads, 248k blob reads for segments, 8.7k shard creates, 1.2k segment parts: about 3.2 bucket operations per changed document, two of them reads. One sampled step (1,656 documents at 1,024 shards): history blocks in 6 s at about 280 creates and 200 reads per second, 832 shards in 6 s (about 140 shards of 200 KB a second, 28 MB/s), segments 10 s. The steps of 29k documents under 400 writes/s took at most 113 s including their trigger wait. After the final burst no step ran: the compactor has no timer, so a world that goes quiet keeps its last slots (here 1,500) for the next cold start to replay, by design.

Stalls. Open 200 had three windows where p50 jumped to 2 s, open 400 two windows of 20 s where throughput fell to 93 and 250 writes/s and 16 slot creates passed the 10 s request deadline. They line up with B's steps and with A adopting them (reading every shard, replaying the base). Pod CPU stayed at 0.2 to 1.5 cores and the node under 51%, so this is not the fake's lock and not the bucket; it is A's own work during adoption plus shared CPU on a 4 vCPU node that also ran the compactor and the load generator. The 250 ms p99 gate at 200 writes/s is therefore not judged here: between stalls the windows ran at the offered rate with p50 170 to 180 ms.

Three replicas writing is the result that matters. Fairness held (A 7,188, B 5,536, C 5,088 acknowledged) but every replica answered at p50 5 s and the world took 94 writes/s against 259 to 365 with one writer. Each slot create races three ways; a loser reads the winner's slot, catches up, rebuilds and may hold its next create up to a second for a hint, all at 40 to 50 ms a call on GCS. The local two-replica runs were fair and slow in the same way (86 writes/s closed loop), which the fake's lock was blamed for; it is the design under real latency. The compactor also lived on B during these phases, which cost B.

Projection to 1,000,000 documents

• Compaction capacity, from the measured rates at 16 workers: a 32,768-document step is about 120 s of history blocks (creates plus the reads of the blocks being extended) plus 30 to 60 s of shards (4,096 or 8,192 at 28 MB/s) plus segments once per window, so 150 to 180 s per step, about 180 to 220 documents a second. Break-even near 200 writes/s; at a sustained 400 writes/s the lag grows. The knob is the compactor's worker count: 32 to 48 doubles or triples the history phase, at 560 to 840 bucket operations a second next to the writers' 440 at 400 writes/s, which is at the bucket's envelope of about 1,000. The segment reads (0.8 blob reads per changed document) are the other lever.
• Cold start: 1,033 shards in about 5 s and 1,500 slots in about 4 s at 16 workers project to 20 s for 4,096 shards and 40 s for 8,192, plus up to 20 s for a 10 minute backlog of 8,192 slots. The 15 s gate needs the cold start to read with 64 workers, as the phase 0 spike did (4,096 shards in 7.6 s), not 16.
• Heap: 655 MiB idle at 245k documents is 2.7 KB a document, consistent with the 2 GiB measured locally at 1M; A's 1,412 MiB after the write phases is the compactor base's divergence and version records before adoption, which the 10 minute cadence makes several hundred MB at 1M.
• Three writers: no projection; it is a design result, not a scale one.

What to do before phase 7

1. Several replicas writing one world. Measure two and three replicas at a fixed offered rate below saturation, with the compactor pinned away from the writers, and the hold time and hint path instrumented; then decide between a shorter hold, a leader-writer per world above the store (rejected in ADR 0036 as a layering violation, to be reweighed against 94 writes/s), or accepting that a world has one writing replica in practice. This is the gate's one failure.
2. Cold start with 64 shard workers, and a gate run at 1M on a node where the compactor and the writer do not share CPU, for the 250 ms p99.
3. Compactor workers as a constant to size from the capacity numbers above, with segments' blob reads counted.
4. The first step after genesis is written by every replica, since checkpoint zero is exempt from the defer rule; three copies of one 32k step is tolerable once per world, but a world created and seeded at speed pays it.

Cost

About 0.7 M Class A operations (writers' blobs and slots, the compactor's history blocks, shards and segments), about 1 M Class B, one n2-standard-4 for about 1.5 hours, negligible storage with soft delete off: about $5 at list prices, to be confirmed in billing under the scale1m-gate labels.