Bucket commit log spike: real GCS, phase 0

Phase 0 of the bucket commit log plan: the log protocol simulated against real GCS from GKE pods, measured against the plan's gate. Audience: demarkus maintainers. Run 2026-10-02. Verdict: the gate passes, with two design changes (stage two batches ahead; size the shard count to the world) and a latency gate restated in milliseconds.

Setup

• Harness: a throwaway Go binary (demarkus-logspike) with setup, run and teardown scripts, built on the benchmarks worktree and deleted after the run; it was never committed. It simulated slots, blobs, checkpoints and replicas with synthetic objects using the same GCS settings as server/blob/gcs (JSON API client, SDK retries off, single-request uploads with CRC32C, DoesNotExist creates). It never ran bucketstore, so member builds and index work cost nothing here.
• Raw evidence: 49 JSON reports under tools/benchmarks/artifacts/logspike-2026-10-02/ (ignored).
• Cluster demarkus (zonal, northamerica-northeast2-a), a tainted scratch pool of 2 n2-standard-8 nodes, pods limited to 4 vCPU and 10 GiB. Scratch buckets in northamerica-northeast2, Standard, soft delete off. All removed afterwards.
• Shapes: 2 KB bodies, 1.2 KB slot entries, 850 byte shard entries, 3 KB of section text per document, 4096 shards. Batches form from whatever is queued, with no timer.
• To fit the time, the last third of the runs went in three lanes on three separate buckets at once; lanes shared nodes, so cold start and gate timings may carry some CPU contention.

Gate

10 minutes open-loop at 250 writes/s on a 1,000,000-document world, batch limit 16, two batches staged ahead, with a replica following and reading every changed body, and a compactor writing a full 1M checkpoint every minute.

c and read costs

Under load (16 parallel blob creates plus a slot create) creates run 45 to 52 ms at p50, about 60 ms at p90. A path read pays one tip check, about 14 ms.

Slot chain

Closed loop, one store, 2 minute runs; every run verified with 0 lost writes.

• With one batch staged ahead, the critical path is staging, not the slot: the next batch is ready only when the slowest of its 16 parallel blob creates finishes, about the p94 create latency (~60 ms) against a slot create of ~45 ms. A local dry run with injected latency reproduced this exactly.
• Two batches ahead takes staging off the critical path: batch 16 reaches about 355/s, close to the batch limit divided by c. Three ahead adds nothing.
• Closed-loop latency is mostly queueing (writers divided by throughput).

Latency without a queue

Open loop, one store, batch 16, two ahead, 1 minute runs, 0 lost.

A write waits for batch formation, its blob create, the slot already in flight, and its own slot create: about 3.5c at p50 and 5c at p99, not the 1c to 3c the plan projected.

Several stores racing

Closed loop, 64 writers per store, batch 16, one ahead, 1 minute runs; the third store of the 3-store runs shared a pod with the second. Every run verified against every store's acks: 0 lost, 0 gaps, 0 chain breaks.

• Total throughput holds at the single-store level; stores trade slots, they do not add contention cost.
• Without yield, one store starves the others. A 30 ms yield after a win, while another store's ID was seen in the last 3 s, evens throughput and cuts the worst p99 from 6.5 s to 3 s, for about 10% of throughput.
• About half of all slot attempts lose with 2 or 3 stores; each loss costs one create and one read.

Bucket rate and the head baseline

• Ramp of open-loop creates in 30 s steps to 1200/s, on a bucket warmed by a checkpoint write: no 429 and no 5xx at any step. Latency is the signal instead: p50 stays at 38 to 41 ms through 800/s and doubles to about 91 ms at 1000/s and above. A cold bucket may throttle sooner; it was not tested.
• Checkpoint writes reached 1123 creates/s on a fresh bucket without a 429.
• Today's head CAS, paced at 1.5 s: exactly 0.67 commits/s. Three unpaced writers on one head: 1.2 commits/s, with 552 of 692 attempts throttled (429) and 68 conflicts.

Checkpoints and cold start

• At 100,000 documents the 4096 shards are about 21 KB each, so cold start is bound by object count (4096 gets, 64 at a time) rather than bytes, and misses the 3 s bound.
• During the gate, 11 of 60 ten-second windows had a writer p99 above 230 ms; only one of them overlapped a compactor run, so the 60 s compactor shows no clear effect on write latency.

Replica apply and body catch-up

During the gate the follower applied every slot (14,591 slots, 250 entries/s) by reading the next name until not found: slot get p50 20 ms, apply lag p99 166 ms. With 16 workers reading every changed body, body lag p99 was 191 ms with no backlog, well inside the 1 s search bound.

What this changes in the plan

• c: 40 ms p50 and 50 ms p99 at low concurrency, 45 to 52 ms under load. The plan's 40 ms case holds.
• Stage two batches ahead. Pipelining with one speculative batch caps batch 16 near 260 to 290/s; two ahead reaches about 355/s.
• Batch limit: batch 32 with one ahead reaches 470 to 510/s. Letting the limit grow to 32 when the queue is deep is worth it.
• Latency gate in milliseconds: single store at 200 writes/s, p99 at most 250 ms (measured 200 ms). Under contention between stores at saturation, p99 is in seconds and is set by queueing and fairness, not c.
• Yield: keep the rule; 30 ms of jitter with a 3 s recency window.
• Checkpoint interval: 60 s is cheap at 1M (full rewrite in about 10 s).
• Shard count: scale it with the world (target about 200 KB per shard), or load shards with far more parallelism; a fixed 4096 makes small worlds slow to start.
• Section index at 1M: a first body search answers from the catalog for about 20 s while 3 GB of segments load, and those sections would stay resident at about 3 GB per world per replica. Section segments need to stay off-heap or be queried in the bucket; this belongs with the catalog memory wall.
• blob.Store.List needs a start offset for cold start to list the log from a checkpoint; the spike used the SDK directly.

Caveats

• Synthetic objects: no member builds, topology checks or catalog updates, so real per-batch CPU is not included.
• Only the gate ran 10 minutes; the matrix ran 2 minutes, lanes 1 minute, the ramp 30 s per step.
• Contention runs were closed loop only; an open-loop race at a fixed rate was not run.
• Batch 16 with three ahead at 256 writers was not run.
• Buckets were warm from earlier runs; a cold bucket's ramp is untested.