# Several replicas writing one world

Why three replicas writing one world ran at 94 writes/s with every write at p50 5 s on real GCS, where one replica ran at 365, and what to do about it. Audience: demarkus maintainers. Proposed 2026-10-04 from the [GCS gate](/research/bucket-commit-log-gcs-gate20.md); blocks phase 7 of the [bucket commit log plan](/plans/bucket-commit-log-gcs-gate.md). Part of [active plans](/index/active-plans.md). Design context: [ADR 0036](/adr/0036-bucket-store-commit-log.md), its group commit section and the turn-taking amendment.

## What was measured

| Writers | Load | Result |
|---|---|---|
| 1 (A), 2 followers | open 400/s | 365/s, p50 287 ms |
| 1 (A) | closed 64 | 259/s, p50 209 ms |
| 3 (A, B, C) | open 300/s | 94/s, p50 5.5 s, fair: 7,188 / 5,536 / 5,088 |
| 3 | closed 64 | 58/s, p50 841 ms |
| 2, local fake | closed 64 | 86/s (two runs); 312/s once in memory with 2 ms calls |

Slot wins over the run: A 19,471, B 879, C 727. CPU on every pod stayed under 1.5 cores; the bucket returned no 429. The compactor lived on B during the three-writer phases. The phase 0 spike (synthetic slots, no committer) saw total throughput stay at the single-store level with 2 or 3 stores and about half the attempts lost; the real committer does far worse.

## The mechanism, as understood

Every writer races the same slot name, so a slot commits only when the writer that creates it has observed the previous one. With one writer that is its own pipeline: two batches staged ahead, slot k+1 created right after k confirms, 32 entries a slot. With R writers, each slot is created by whichever replica observed the tip most recently; the others lose, and a loser's path back to a create is read the winning slot (one GET), list and read anything after it, rebuild its batch against the new tip, then create again, while the winner holds its own next create for up to `handoffWait` (1 s) after a loser's hint until a peer's slot lands. On GCS every one of those calls is 40 to 50 ms, so the log advances about one slot per loser round trip, and the batches are small because each replica's queue drains between its wins. Both effects were seen locally and attributed to the fake's lock; the lock amplified them, but they are the design under real latency.

Hypotheses to confirm or kill, in order:

1. Throughput with R writers is bounded by the loser's recovery time (loss to next create), not by CPU or the bucket: writes/s ≈ entries per slot / recovery time.
2. The hold of up to 1 s adds dead time well past what a GCS recovery needs (about 100 to 150 ms), so a shorter or adaptive hold recovers part of the loss.
3. Entries per slot collapse under contention (each replica's queue drains between wins), so the loss is multiplicative with 1.
4. Peer hints arrive later than a direct read of the next slot name, so the follower that learns of a win by hint is always a round trip behind.
5. A compactor on a writing replica costs that replica wins (B's 879 against C's 727 says little either way).

## Experiments

All in-process first, free and deterministic, then one short GCS lane.

1. **A model in the committer tests**: R stores on one `blob.Memory` behind `latentStore` with GCS-shaped latency (create 45 ms, get 25 ms, list 30 ms), closed loop 64 writers per store and open loop at a fixed offered rate, for R = 1, 2, 3; record writes/s, entries per slot, slot losses, recovery time (loss to next create) and hold time per replica from the committer's events. Repeat with `handoffWait` at 1 s, 250 ms, 100 ms and 0, and with hints off (reads only). This reproduces the 94 writes/s or shows the model misses something; it is also the regression test for whatever is decided. Cost: nothing.
2. **One GCS lane**, 20 minutes, about $2: the gate harness with a 20k-document world, the compactor pinned away from the writers (a fourth replica takes no load), 2 then 3 writers at a fixed offered 100 writes/s (below saturation) and closed loop 64, with the recovery and hold times logged per slot. Confirms the model's constants on real latency.

## Options to decide between

- **Tune the turns.** A hold sized to the measured recovery (about 2 × recovery p50, 200 to 300 ms on GCS) instead of 1 s, and a loser that creates again as soon as it has rebuilt rather than after a full catch-up listing when the winner's slot was the only one. Cheapest; recovers a fraction. Keeps every replica writing.
- **Hand the batch to the winner inside the store.** A loser sends its built batch to the replica that holds the tip over the peer QUIC connection that already carries hints, and that replica commits it in its pipeline; the loser acknowledges when the peer reports the slot. Total throughput becomes the single-writer pipeline's, latency gains one hop, and nothing above the store changes: the world manager still sees `backend.Store` and `Follower`. It changes ADR 0036's "every replica commits for every world" to "every replica accepts writes; the replica at the tip commits them", needs the peer channel to carry batches (sizes, backpressure, a peer that dies mid-handoff falls back to racing), and preconditions still evaluate at the tip (the winner replays the batch on its base before creating, as it does today). This is the likely answer if tuning does not reach about 70% of single-writer throughput.
- **One writing replica per world above the store.** Route PUBLISH to one replica at the ingress. Rejected in ADR 0036 as a layering violation; reweighed only if the in-store handoff proves too complex.
- **Accept it.** Document that a world's throughput is one replica's and that two or more writing replicas cost it. Not acceptable for the product goal of many agents writing one world through any replica.

Decision rule: run experiment 1, read the recovery and hold numbers, try the tune; if R = 3 reaches 70% of R = 1 in the model, confirm on the GCS lane and ship the tune as an ADR amendment. Otherwise design the in-store handoff, amend ADR 0036, and gate it on the same model and lane.

## Invariants

- Each acknowledged mutation is in exactly one slot; acknowledgement follows the slot's confirmation, wherever it was created.
- Preconditions are evaluated at the tip plus the members before them, whichever replica creates the slot.
- A loser's or a handed-off batch is rebuilt, never re-staged; staged blobs are remembered per request.
- Nothing above the store elects writers or routes writes; the backend contract is unchanged.
- The replica conformance suite, including the blind-write races, passes unedited.

## Status

Proposed 2026-10-04. Next step is experiment 1 in `committer_test.go`.

Decided 2026-10-04 with the maintainer: the goal is **steady throughput as replicas are added, between 200 and 365 writes/s per world with any number of writing replicas**, not more throughput per replica. The fix is done in a fresh session, with cost discipline: the in-process model (experiment 1) first and as the regression test, code only once the model reproduces the collapse and shows the fix reaching the goal, and one GCS lane only to confirm a fix that already passes the model. No exploratory cloud runs. Start by reading this plan, the [GCS gate report](/research/bucket-commit-log-gcs-gate20.md), ADR 0036's group commit section and turn-taking amendment, and `commit.go` (`handoffWait`, `ownSlots`, `rebase`, `handOff`).

Status 2026-10-04, model done (branch `fix/several-writers`, uncommitted): experiment 1 reproduced the collapse only once the load matched the gate's (a seeded world, 70% blind updates); writes to new paths read nothing and ran three writers as fast as one. Hypothesis 1 held with a different constant: a loser's recovery was 1.2 s, not 150 ms, because the two batches requeued behind a lost head were rebuilt reading each document from the bucket serially on the committer, so the 1 s hold expired first and a store starved (hypotheses 2 and 3 followed from that; 4 and 5 were not needed). The tune is the fix: requeued requests carry their batch's reads, `handoffWait` is 300 ms. Model, 3 s runs, writes/s closed 64 a store and open 300/s: R = 1 235 and 300; R = 2 199 to 207 and 300; R = 3 221 to 237 and 300, before the fix 115 and 91 with a starved store. Open-loop p50 with three writers is 600 to 660 ms against 200 ms with one: the turn's cost, left as is. The listing-free rebase was tried and rejected. Recorded in [ADR 0036](/adr/0036-bucket-store-commit-log.md#amendment-a-loser-keeps-its-reads-and-the-hold-is-300-ms-2026-10-04-several-writers). Next: the GCS lane (experiment 2), on the maintainer's go.

Status 2026-10-04, lane done, **confirmed**: [the GCS lane report](/research/several-writers-gcs-lane.md). Open 300/s split over three replicas served at 298/s with 0 errors and p50 559 ms (gate: 94/s, 8,861 timeouts, p50 5.5 s); closed 64 over two and three at 167 and 147 writes/s, p50 387 ms (gate: 58/s); recovery 46 to 80 ms at p50 on GCS; nothing lost in 148,540 acknowledged versions. Tails of 3 to 7 s came only from compaction steps and adoptions on writing replicas (the compactor did not stay on D: its first step outlived the peers' 45 s wait, a lane configuration fault). The fix ships as is; phase 7 is unblocked on this question. Scratch resources deleted the same evening; the node pool is the maintainer's.