# Bucket commit log: local 1,000,000-document scale test

The commit-log bucket store ([ADR 0036](/adr/0036-bucket-store-commit-log.md), branch `feat/bucket-commit-log`, PR 538) driven to the plan's baseline of one million documents on a laptop against a patched fake GCS, with two replicas on one bucket. Audience: demarkus maintainers. Run 2026-10-03 and 2026-10-04. Verdict: **no write was lost in any run (3.1 million acknowledged versions read back across the two runs), the compaction defect the first run found is fixed and verified, and the fake's single lock bounds every latency and stall figure here**, so the throughput and cold-start numbers are lower bounds for GCS, not gates. Part of [the bucket commit log plan](/plans/bucket-commit-log.md); the fixes are recorded in the ADR's [2026-10-04 amendment](/adr/0036-bucket-store-commit-log.md#amendment-compaction-in-bounded-steps-and-content-objects-are-never-freshened-2026-10-04) and the day's [journal](/journal/2026-10-04.md). Raw evidence under the ignored `tools/benchmarks/artifacts/scale1m-2026-10-03/` (`run1m/`, `catchup/`, `run1m-v2/`, `dry100k-v2/`, `harness/`).

## Setup

- Server: the real `knowledgeserver.Open` in a throwaway `scale-server` main (counting bucket wrapper, heap sampler, debug endpoint), two replicas A and B on one bucket with peer hints, QUIC on 7309 and 7319, request timeout 10 s. Load: a throwaway `scale-load` driver over QUIC (seed, open and closed loop writes, log walk and read-back, lookups, body search).
- Bucket: fsouza/fake-gcs-server 1.56.1 with a local patch (prefix-rooted listing walk, name offset, no fsync on create), filesystem backend in OrbStack, HTTPS with HTTP/2. The fake serializes everything under one lock, so a large upload stalls every other call; 1.52.2 (deploy/kind) has no list pagination and cannot serve this branch at all.
- Documents: agent-shaped markdown with tags and metadata, about 4 KB, published blind; updates hit random documents (spread writes).
- Three builds of the branch, all with the turn-taking and blind-publish fixes of 2026-10-03: run 1 (2026-10-03) before the compaction fix; the catch-up and run 2's seed with bounded steps and content creates; run 2's remaining phases with the defer rule and the 8,192-slot cadence. Run 2 was not reseeded between builds.

## The defect and the fixes

Run 1 stopped on the stop condition: from 21:35 every checkpoint run on both replicas hit the 5 minute bound, 16 in a row, no checkpoint landed for 37 minutes at about 200 writes/s, and a fresh replica replayed 24,781 slots in 33 to 41 s. The events show the cause: during the seed at 468 writes/s successive checkpoints covered 20k, then 88k sequences and took 20 s, then 5.7 min, each folding a history block per document changed since the one before, so every run had more to do than the last; past the bound nothing was kept and the retry started over, paying a create, a read back and, past 7.5 min, a rewrite for every block already there (A: 996,241 present against 248,555 new, 137,679 rewritten).

Fixes, each with tests that fail under a mutation (13 mutations in all), details in the ADR amendment:

1. **Content creates.** History blocks and blobs are never dropped, so a taken name is done: one call, no read back, no freshen.
2. **Bounded steps.** The compactor keeps a base snapshot at the newest checkpoint and, when the served snapshot holds more than 32,768 changed documents, replays the log from the base to the slot boundary that fills a step and checkpoints that; a run loops steps. On the run 1 bucket, a 440,000-sequence backlog was checkpointed in 14 steps in 12 min 20 s, each step 27 to 37 s of writes (33k history blocks, 4,096 shards, 925 to 1,000 MB), heap 2,465 MiB at cold start with the backlog, 1,986 MiB after.
3. **One compactor without a claim.** Run 2's seed showed both replicas writing identical steps in lockstep, since both are triggered by the same slots from the same base to the same deterministic boundary; the open loop at 200 writes/s then acknowledged 93/s with 30,558 timeouts. A store now steps only when the newest checkpoint is its own (its plain create made it), checkpoint zero, or older than the step bound. Afterwards every phase had one compactor; the first step after a quiet period is still written by both, as the rule allows once.
4. **Cadence 8,192 slots or 10 minutes**, from 1,024 or 60 s. With spread writes every step rewrites every shard (800 MB to 1 GB at this size) and every follower reads them back to adopt, so a step a minute was the dominant bucket traffic; the slots already are the O(change) record and 1,024 of them replay in about 0.9 s on GCS. At 200 writes/s the open loop went from 169.8/s with 7,798 timeouts (two 40 s stalls) to 193.5/s with 1,728 (two stalls of 10 to 20 s).

## Results against the goals

**Seed 1M through Publish.** Run 2: 999,805 acknowledged at 388.7/s over 42.9 min, 195 timeouts, p50 121 ms, p99 1,156 ms, p999 4.0 s; 68,430 slots, mean 15.8 entries (60,495 slots of exactly 16, 1,204 of 32), slot create p50 30 ms, p99 200 ms; 29 checkpoints during the seed by two compactors leapfrogging, lag at most 103k sequences, 0 failed runs. Run 1 seeded at 468/s with 16 timeouts and p99 559 ms while its compactor did nothing useful. Shard count grew with the world to 4,096 at the end of the seed and 8,192 once the world passed 1,048,576 documents during the write phases.

**Open loop, one writer with a follower** (run 2, final build): 200/s offered: 193.5/s, 116,135 acknowledged, 1,728 timeouts, p50 191 ms, p99 5.0 s. 250/s: 227.1/s, 136,321 acknowledged, 4,096 timeouts, p50 290 ms, p99 6.5 s. Every timeout sits in a stall of 10 to 20 s where the compactor's shard upload holds the fake's lock; between stalls the windows run at the offered rate with p50 90 to 230 ms. Run 1: 190.5/s and 227.9/s with 1,932 and 5,024 timeouts, its stalls being the compactor's history reads.

**Closed loop, 64 writers on one replica**: 183.5/s, 55,114 acknowledged in 5 min, 128 errors, p50 281 ms, p90 432 ms, p99 1.18 s; windows of 200 to 300/s with three dips at checkpoint steps. Run 1: 224/s, 0 errors.

**Both replicas writing**: open loop at 250/s split: 146/s, 88,734 acknowledged, 16,932 timeouts, p50 4.8 s; the failing minutes are the compactor's shard uploads on B and A's download of the same shards to adopt (534 slow shard reads in a minute while A's slot wins fell to 38). Closed loop, 64 writers split: 71.6/s, 22,189 acknowledged, 224 errors, p50 717 ms, p99 3.2 s. Run 1: 188/s with 1,428 timeouts and 85.6/s with 0 errors, fair at 56k/58k and 12.8k/13.0k. The handoff cost is bound by the fake (a loser's recovery read queues behind the lock); the GCS gate measures it.

**Every acknowledged write found.** Run 2: 106,608 slots, 1,644,823 entries, 0 gaps, 0 chain breaks, 0 duplicate operation IDs or versions; all 1,641,479 acknowledged versions read back, 0 bad, 0 retries, in 26 min 39 s at about 1,000 reads/s. Run 1: 88,725 slots, 1,458,194 entries, all 1,457,706 acknowledged versions read back. Four store IDs wrote run 2's log, one per server start.

**Cold start at 1M.** On the caught-up run 1 bucket (8,192 shards, 1,047 MB): 16.6 s and 16.2 s with no load, 1 slot listed and none replayed, heap 1,905 MiB. On run 2's bucket while A served: 22.4 s and 19.0 s, 8,260 shard reads, heap 1,993 MiB. Gate 15 s: missed by 1.2 to 7 s, bound by the fake's object rate (about 500 reads/s); the shard split at 2^20 documents doubled the objects for the same bytes, and the 1M baseline sits 5% under that boundary. Before the fix, with the 440k backlog: 37.4 s (24,781 slots replayed).

**Checkpoint cost.** One changed document (10 min after the previous write): 1 history block, 1 shard (0.13 MB), the root (1.58 MB at 8,192 shards), 1 checkpoint object, 1 segment part (0.55 MB), written in 0.04 s; the adopter read 1 shard and the root. Spread writes (2,500 documents in 10 s): 2,501 history blocks, 2,174 shards (277 MB), root, 260 segment parts (138 MB), 4.5 s; the other replica ran the same step (first step after a quiet period), lost every create and read the 2,184 shards back (279 MB) to adopt. A full step at this size: about 34k history blocks (24 MB), 8,400 shards (950 to 1,010 MB), 120 to 177 s under write load on the fake, 27 to 37 s with the bucket to itself. Segment windows: a shard's segment is written once per 10 minute window, 984 and 579 labels in the two windows of the catch-up.

**Body search, index heap, eviction.** The heap guard stopped A 15 s into the first body search at 1M: 8,345 MiB in use, 4,341 MiB live (from 2,223 MiB at open), growing about 230 MiB/s of live heap while segments loaded. From the 100k dry run (index 1,447 MiB for 100k documents, catalog fallback for 18.8 s, body lookups p50 140 to 270 ms, eviction 1,755 to 308 MiB after 31 idle minutes) the resident index at 1M projects to about 14.5 GB, over three times body bytes. A world this size cannot hold its section index on an 8 GiB replica; the memory wall the plan names stands.

**Catalog lookups at 1M** (300 each, max 50): `*` p50 150 ms, p99 168 ms; terms `zetgar` 436 / 486 ms, `domain:notes` 472 / 669 ms, `pelurmak` 445 / 486 ms; about 1.6 MB and 5,350 mallocs per lookup including QUIC (644 KB and 2,750 for the rarest term). At 100k the same lookups took 47 to 50 ms, so a term lookup scans and scores the catalog in time linear in the world.

**Heap across write cycles**: four cycles of 2 min at 250/s with 10% creates, 30,000 acknowledged each, 0 errors, p99 548 ms to 2.5 s (a step inside cycle 2). A after each cycle, forced GC: 2,087, 2,155, 2,205, 2,250 MiB; B 2,110 to 2,270 MiB. The growth is the uncheckpointed state since the newest checkpoint (versions, new documents, the compactor base's divergence) and is bounded by the cadence, which four cycles do not outlast. Idle after the write phases, A held 2,675 MiB against 1,991 MiB in run 1 at a minute cadence, and peaked at 4,246 MiB during the seed while both replicas compacted: the base snapshot costs several KB per document changed since the newest checkpoint, not the 1 KB estimated in the ADR, and the cadence trades shard traffic for that heap.

## What carries over to the GCS gate, and what does not

Carries over: the correctness results (nothing lost, the chain, the read-back), the step mechanics and the one-compactor rule, the O(change) one-document checkpoint, the catalog lookup times and allocations (CPU-bound), the heap figures (the snapshot, the base, the body index projection), the shard doubling at 2^20 documents, the duplicated first step after a quiet period.

Does not: every latency and timeout count under compaction, the stalls, the cold start seconds, the step durations and the closed-loop handoff cost. All of them are the fake's single lock serializing 200 KB shard transfers against slot creates, which GCS does not do.

What the GCS gate should measure differently:

- **Compaction capacity against arrival.** Step duration at 200 and 400 writes/s with spread writes, and whether the lag between checkpoints shrinks or grows. The estimate is about 2.5 to 3 min per 33k-document step with 16 workers (history reads and creates, then 4,096 shards), so break-even near 200 writes/s. The knobs if it falls short: compactor workers (writes use about 220 of the bucket's roughly 1,000 ops/s at 200 writes/s), step size, and segments out of the step loop.
- **The adopter's cost**: shards read per adoption (all of them under spread writes, 800 MB to 1 GB) and the adopter's own write latency during it; with N replicas it is N-1 downloads per step.
- **p99 during a step**, which on GCS is bandwidth share rather than a stall, against the plan's 250 ms gate at 200 writes/s.
- **Cold start** on a world just over 2^20 documents (8,192 shards) and one just under, against the 15 s gate, and with a 10 minute backlog of slots to replay.
- **Heap of the base** at the 10 minute cadence, per document changed between checkpoints.
- The kind gate needs fake-gcs 1.56.1 or newer (1.52.2 lists at most 1,000 objects) and HTTP/2 to the emulator, or the port range is exhausted.

## Other findings

- A run that cannot reach the bucket at start now fails instead of running the load against nothing; the first attempt at run 2 did, after the patched image vanished from Docker and the 21 GB volume took 30 minutes to delete.
- `go vet ./...` in the tools module compiled the harness under the ignored benchmarks directory; a `go.mod` there fences it.
- Compactor bursts remain unpaced: about 1,000 creates/s against the GCS bucket envelope while a step writes history blocks, the same as noted after run 1.