ADR 0036: The bucket store commits through a create-only log

Status: proposed (2026-10-02). Plan: Bucket store commit log. Evidence: phase 0 spike on real GCS. Amends ADR 0008 and ADR 0015; supersedes ADR 0031. Part of the decisions hub.

Context

ADR 0008 commits every mutation by replacing _demarkus/v1/head.json, the world's only mutable object, and assumed a read-heavy world under one mutation per second. GCS allows about one update per second to one object name, so the bucket store paces head attempts at 1.5 s. The product now needs many users writing one world at once.
Measured, the head is the ceiling: 0.67 commits/s paced on real GCS, and 1.2 commits/s from three unpaced writers with 552 of 692 attempts throttled. On kind, one writer took 1.51 s per publish and a four-writer burst lost 49 of 200 publishes to the 10 s request deadline.
The cap belongs to any design where every commit mutates one object name (CAS, lock, lease, pointer). Creating a new name with create-if-absent has no per-name cap.
The second wall is bytes. A commit rewrites the document's history block and manifest, one of 256 shards and the root. A shard holds a 256th of the world at about 850 bytes per document: about 3.3 MB at 1,000,000 documents, so 250 writes/s would upload, and every replica would decode, about 830 MB/s.
The knowledge server reaches past the backend contract. Its world manager holds the concrete *bucketstore.Store, compares peer hints with its head sequence, schedules its polls, creates the world's genesis and calls its constructor inline, so a second backend could not replace it without changes above the store.

Decision

Commit point

A commit creates _demarkus/v1/log/<first sequence as 16 hex digits>.json with blob.Store.Create (create-if-absent: ifGenerationMatch=0 on GCS, If-None-Match: * on S3). The successful create is the commit, and a slot is never rewritten. This replaces ADR 0008's head replacement as the commit point.
A slot holds one batch of mutations. Per mutation it carries the receipt, the new index entry (path, current version, archive state, body hash, modified time, catalog record, history block references), the new version record and any retention prune range. It also carries the schema, world ID, first sequence, the writing store's ID, the sweep epoch and the hash of the previous slot, so the log is a verifiable chain of contiguous sequences.
A commit writes the content-addressed document blobs and one slot, nothing else: about 1.1 objects and the body plus about 1.2 KB per mutation, whatever the world's size. No root, shard, manifest, history block, change block or head is written on the commit path.
The manifest layer is folded into the index entry, which carries its history block references; there is no per-document manifest object.
Every writer racing from one tip targets the same slot name. A loser reads the winning slot, rebuilds only the members whose path, ancestors, descendants or the world's publish policy that slot touched, and tries the next name. Expected-version, topology, policy and archive semantics are unchanged.
An ambiguous create is decided by reading the slot: absent, retry the identical create; present with this batch's operation IDs, committed; present with others, lost. ADR 0008's evicted-receipt server-error disappears.

Group commit

One committer per world in each store takes whatever is queued, up to a batch limit of 16 that may grow to 32 while the queue stays deep, with no timer. Members build in queue order, each against the tip plus the members before it; a member whose precondition fails gets its normal error alone.
Up to two further batches are built and staged while one slot create is in flight, and slot k+1 is created only after slot k is confirmed. Every member is acknowledged only after its slot is confirmed.
Every replica commits for every world it serves. A store that has seen another store's ID in a slot within the last 3 s waits up to 30 ms of jitter after each win; a store alone in a world never waits.

Reads and freshness

A replica's snapshot is the latest checkpoint it loaded plus every slot applied since. Slots carry full entries, so applying one needs no further reads, and a replica installs its own commits as they succeed.
Reads by path (FETCH, VERSIONS, chain verification) and directory checks are exact: they see every commit acknowledged on any replica before the request. They check the next slot name first, one metadata read of 11 to 14 ms. Directory checks are exact because FETCH asks one before it reads a version path.
Catalog reads (LOOKUP, LIST and hash lookup) may trail another replica's commit by up to one second: a replica serves them from a snapshot confirmed current within the last second, which costs at most one tip check per second per world. This relaxes ADR 0008's immediate LOOKUP and hash freshness. Write preconditions are always evaluated at the tip, so the lag never affects write correctness.
One request still reads one snapshot. A view pins its snapshot at its first read: an exact read pins the tip, a catalog read may pin a snapshot up to one second old.
The bound is a property of the backend contract. It is documented on backend.ReadView and in the SPEC and proven by the conformance suite; handlers do not know which backend serves them.

Checkpoints, world marker and cold start

Checkpoints are derived, not committed. Any store's compactor may write one when the newest lags the tip by 1024 slots or 60 s, after a random delay. A checkpoint at sequence S is a deterministic function of the log up to S, so racing compactors write identical bytes through create-if-absent. Readers verify it against the slot chain, and a missing or failed checkpoint costs only replay.
A checkpoint keeps ADR 0008's content-addressed, verified object model: a root, shards whose entries carry history block references, history blocks and section segments, each written only when it changed and otherwise referenced by hash. Shard count scales with the world at about 200 KB per shard: 4096 shards at 1,000,000 documents.
head.json becomes a frozen world marker in schema 2: world ID and format, created once for a new world or replaced once at migration, never written on the commit path. An old replica refuses it and fails closed.
A new world starts with the marker and checkpoint zero, an empty root at sequence 1. Cold start lists checkpoints/ for the newest, loads it, lists log/ from its sequence and applies the slots. blob.Store.List gains a start offset for this, with no GCS types in it.

Section index

The body-search index (ADR 0012) is never built on the commit path or when a slot is applied. A world builds it on its first body search, from checkpoint section segments plus the bodies changed since; until then body search answers as catalog fallback, which LOOKUP already reports as inconclusive. It is kept current in the background, reported as partial while behind, and dropped when body search goes idle.

Change feed

Slots are the durable change record (ADR 0028), and the hub's backlog reads them through an LRU of recent slots. Slots are retained at least as long as the hub's ring and the federation deriver's checkpointed cursor age, so a resume inside that window never resyncs.
ADR 0031's sealed change blocks are removed: sealChanges, sealedThrough, the block LRU and the changes/ objects. A slot already holds up to 16 changes.

Scope

Bucket only. GCS, and later S3, stays the only state; a metadata database stays out (ADR 0010).
Ordering per world. WATCH cursors and the change feed stay numbered per world (ADR 0025, ADR 0028).
A baseline of 1,000,000 documents per world. Every gate, shard count, checkpoint interval and cold start bound is sized for it, and ADR 0008's supported limit of 100,000 rises tenfold. Write cost depends on the change, never on the document count.
Throughput. At least 200 writes per second per world, and p99 at most 250 ms for one store at 200 writes/s.

Store boundary

Everything above is private to bucketstore. The server sees a world's store only through the backend contract in server/internal/backend and constructs it at one wiring point, so swapping bucketstore for filestore, an S3 store or a database changes one constructor.

What the world manager holds:

backend.Store: OpenReadView, Publish, Append and SetArchived, with the freshness property above on its views.
backend.ChangeSource: the store's change hub, nil without WATCH. The hub counts readers waiting for its next event, so a store can schedule work that only live watchers need; a subscription nobody reads does not count, and callers have nothing to release.
backend.Follower, new, for a store several replicas share. Follow(sequence int64) says a peer replica committed through sequence: the store reads ahead of its own schedule unless it already serves that sequence, and a store whose reads are always current ignores it. It never blocks.
io.Closer, which stops whatever the store runs in the background, ends its watches and refuses later writes with backend.ErrClosed.
One store factory in the world manager's config. It takes the world's config, a logger and a commit hook, func(sequence int64), which the store calls after each of this replica's own commits. The server's wiring in run.go is the only code that names bucketstore or a blob store.

Peer hints carry only a world ID and a sequence, never a slot name, checkpoint or object key. The world manager hands an incoming hint to that world's Follower and does nothing else with it.

Private to bucketstore: the blob store and every object key; the log, slots, checkpoints and the world marker; the committer, batching, staging and the yield rule; the compactor and the collector; when to read the bucket, including the backstop poll for watchers (5 s today, run only while a watcher waits on the hub) and coalescing of peer hints; creating a new world's genesis inside open, refusing a bucket that holds foreign objects and leaving read-only worlds untouched; migration; building and evicting the section index.

The refactor that sets this boundary lands before any log code, on today's head CAS store, and changes no behavior. Today's leaks are all in server/knowledgeserver/worlds.go: worldEntry.store is a *bucketstore.Store; Hint compares HeadSequence(); pollChanges, changePollInterval and the kick channel schedule Poll; ensureGenesis calls bucketstore.EnsureWorld; and bucketstore.Open is called inline with its options and Committed hook. A world manager test serves a world on a store that is not a bucket store, through the factory.

Consequences

Measured on real GCS from GKE on 2026-10-02 unless noted.

Throughput. Batch 16 with two batches staged ahead sustained 352 to 355 writes/s; batch 32 with one ahead, 470 to 512. One batch ahead capped batch 16 at 256 to 291, because the next batch waited on the slowest of its parallel blob creates; three ahead added nothing.
Gate. 250 writes/s for 10 minutes on a 1,000,000-document world, with a following replica and a compactor: 150,012 writes, none lost, no gaps or chain breaks, no 429 or 5xx, p50 143 ms, p99 196 ms, p999 319 ms.
Latency. Without a queue, one store: p99 200 ms at 200 writes/s and 306 ms at 300. A write waits for batch formation, its blob, the slot in flight and its own slot, about 3.5 creates at p50 and 5 at p99. A small create costs 40 ms at p50 and 50 ms at p99 alone, 45 to 52 ms at p50 under load.
Several stores. With 2 or 3 stores racing one world, total throughput stays at the single-store level and about half the slot attempts lose. Without the yield one of three stores got 51 writes/s against another's 151, with p99 6.5 s; with it the stores evened out (95, 99 and 78 writes/s) and the worst p99 fell to 3 s, for about 10% of throughput. Under saturated contention p99 is in seconds and set by queueing.
Bucket. A warmed bucket took 1200 creates/s without a 429, and latency doubled from about 1000/s; at about 1.1 objects per write that puts one world's ceiling near 900 writes/s. A cold bucket was not tested.
Cold start. At 1,000,000 documents: 7.6 s (10.3 s on a contended node), index heap 1.04 GiB, and 1024 slots replayed in 0.9 s. A full checkpoint (863 MB) took 6 to 13 s, and one a minute showed no clear effect on write latency. At 100,000 documents a fixed 4096 shards took 4.2 s, bound by object count, which is why shard count scales with the world.
Freshness. A replica following by reads applied 250 writes/s with apply lag p99 166 ms, and body catch-up with 16 workers stayed under 200 ms at p99, both well inside one second. Catalog reads on another replica may now trail a write by up to a second, and the SPEC says so; requests that reach the replica that committed see it at once.
Memory. A first body search at 1,000,000 documents answers as catalog fallback for about 20 s while 3 GB of section segments load, and resident sections are about 3 GB per active world per replica. That catalog and section memory wall is out of scope and needs its own plan before several baseline worlds share a process.
Migration. A new replica checkpoints a schema 1 head and replaces it once with the marker. Cursors older than the migration sequence get one resync, which triggers a federation rebuild per written world unless ADR 0031's blocks stay readable until the deriver passes it. Rollback is open: a tool that writes a schema 1 head from a fresh checkpoint, or a one-way migration.
Hash lookup stays in the one second class until ADR 0035 is decided; if it is accepted, the body-hash map leaves snapshots and checkpoints.
Garbage collection follows reachability from retained checkpoints, retained slots, backup pins and imports. The sweep epoch is a log entry, which keeps ADR 0008's fencing.
A new backend implements backend.Store, ChangeSource if it supports WATCH and Follower if replicas share it, passes the conformance suites, and is wired by one constructor.

Alternatives considered

Group commit on head.json. Still one replace per batch at 1.5 s, a hot object whose 429s delay the whole batch, and a receipt window that must grow with the batch.
Slots that point to a rewritten root and shards. Removes the head cap but keeps a per-commit write that grows with the world: about 3.3 MB per write at 1,000,000 documents.
A two-level shard tree rewritten per commit. Tens of KB per write, but an extra object per write and a cold start that reads 65,536 leaves.
A per-world writer elected above the store, with writes forwarded to it. Puts the bucket store's concurrency shape into server routing and needs pod-to-pod transport that a database store would not. Rejected as a layering violation.
An in-memory head shared between replicas and flushed every 2 s. Loses acknowledged writes on a crash unless acknowledgement waits for the flush, which is group commit on the head; making it safe needs quorum replication.
Per-document conditional writes, or per-version create-only objects. No world sequence for WATCH and the feed, topology races between creates, and a second commit point beside the log that a crash can split from it.
Section entries carried in slots. A section entry is about as large as the body, and every replica would hold sections for every world whether or not anyone searches it.
A metadata database beside the bucket. Out by the bucket-only decision.
Libraries. glassdb, objwal, SlateDB and delta-go were surveyed and none fits (alpha, single writer, one rewritten manifest, or no GCS). Delta Lake's log plus checkpoints and SlateDB's put-if-absent commit are design references.
Peer hints as a hub catch-up (Hub.CatchUpTo, weighed in review 2026-10-02). It would drop backend.Follower, but hints also keep catalog reads inside the one second bound whether or not WATCH is on, and the hub would then schedule store reads.
The freshness class chosen per verb when the view opens (weighed in review 2026-10-02). It would make the class explicit, but handlers would have to know about it; classing reads by method, with directory checks exact, keeps them unaware.

Amendment: collection without a log barrier (2026-10-02)

Decided with the maintainer while building phase 4. It replaces "The sweep epoch is a log entry, which keeps ADR 0008's fencing" and the sweep epoch named among a slot's fields; slots carry no epoch.

Why. A barrier in the log fences only slot writers. Checkpoints are written outside the log, so a compactor in flight across a sweep could name an object the sweep deletes, and the writer and its peers would rebase on a broken checkpoint. The barrier would also stall every write for the whole deletion phase.
Generation fence. A sweep deletes an object only when no kept checkpoint or retained slot reaches it and it was older than a one hour grace when the sweep began, and each delete is conditioned on the generation the sweep listed. A writer or compactor whose create finds its bytes already stored, older than half the grace, rewrites them in place first. The new generation makes a racing delete fail, and a delete that already won makes the create write the object again. This fences writers and compactors alike and never stalls a write.
What is kept. Every checkpoint written in the last hour and at least the newest three; everything they reach; every slot after the oldest of them, with the blobs those slots' versions name; slots before it for 24 hours, for the change feed and the federation deriver's cursors. Backup pins and imports join the roots once they exist; neither does yet.
Who sweeps. Each store tries hourly, and the first to create that hour's create-only claim object sweeps. Each store also catches up and rebases on the newest checkpoint every 20 minutes, so none rests on a collected one. A store whose snapshot has gone unconfirmed for 12 hours (half the slot retention) reloads from the newest checkpoint before trusting a missing slot as the log's tip.
Backends. The fence needs a delete conditioned on the listed version. GCS has it (ifGenerationMatch); an S3 backend must confirm conditional deletes before it lands.

Amendment: collection deferred (2026-10-02, later)

The maintainer deferred content collection after building it, since today's worlds grow by a couple of gigabytes a month. Until it is needed, a compactor that writes a checkpoint deletes the checkpoint objects past the newest three that are more than an hour old, each delete conditioned on the generation it listed, so a cold start's listing stays short. Nothing else is deleted: superseded shards, roots, history blocks, blobs of pruned versions and slots stay, as ADR 0008 accepted. A replica resting on a deleted checkpoint keeps reading, because everything that checkpoint reached is still stored.

The generation fence above is the decided design for when collection is built. The full collector (sweep, mark, the rewrite of reused objects, slot retention, the stale replica reload and the maintenance loop) was written and tested, and was set aside as a patch rather than merged.

Amendment: old shards and roots are collected (2026-10-02, latest)

Shards and roots are the bulk of what checkpoints leave behind: changing one entry rewrites its whole shard, about 200 KB. Nothing reads one once its checkpoint is dropped, so they are collected after all. History blocks, blobs and slots still are not.

Kept checkpoints. The newest three, and any whose successor was written within the last hour. A checkpoint therefore goes only after it stopped being the newest more than an hour ago, by which time any compactor building on it has finished (each run is capped at 5 minutes).
What goes with a dropped checkpoint. Its shards and root that no kept checkpoint uses, then the checkpoint object last. A drop cut short leaves the checkpoint object, and the next drop finds its shards and root through it. A missing root means its shards already went.
The fence. Shards and roots are content-addressed, so a new checkpoint whose shard returned to an earlier state reuses the old object. A shard or root is deleted only when it was older than an hour when the drop began, and only at the generation the drop read. A create that reuses stored bytes older than 30 minutes rewrites them first, so a pending delete fails. This is the generation fence from the first amendment, limited to shards and roots.

Amendment: catalog fallback and body segments (2026-10-02, phase 5)

Decided with the maintainer before phase 5 was built. It replaces "reported as partial while behind" in the section index decision above, and the bucket store consequence of ADR 0012, which indexed every body at open and on refresh.

Fallback, not partial. A body search waits up to one second (backend.SearchFreshness) for a section index that covers exactly the snapshot its view pinned. Without one in time the store refuses with backend.ErrBodyMatchUnavailable, and the handler answers from the catalog on the same view, echoing match: catalog, which clients already treat as inconclusive. Nothing new goes on the wire. The SPEC now says a server with body match may answer a body request in catalog mode while its index is built or behind, and must not echo match: body for an index that misses part of the state the request reads.
One version per snapshot. The index keeps an immutable version for each installed snapshot, retained for the request timeout, so a body search never mixes sections from another snapshot. More than 1024 queued installs collapse into one comparison with the served snapshot.
Body segments keyed by shard. A segment holds the bodies of one checkpoint shard's live documents at _demarkus/v1/segments/<label>/<shard hash>/<part>.json, in parts of at most 2 MiB encoded because objects are capped at 4 MiB; part 0 names the part count and is written last. It holds bodies rather than serialized sections: about 1.33 times body bytes against 2.34, measured on the soul corpus, and it never goes stale when the tokenizer is retuned. The root names no segment, so checkpoint bytes stay a function of the log; a missing segment only costs blob reads. A reader uses a body only when it hashes to the document's body hash.
Writers. The compactor writes the segment of each rewritten shard, carrying unchanged bodies from the previous checkpoint's segment and reading each changed body once; unchanged bodies with no segment to carry them come from a budget of 4096 reads per checkpoint. A replica whose build read a whole unchanged shard from blobs writes that shard's segment. A world from before segments gains them through its first body searches and later checkpoints.
Collection. A dropped shard's segment goes with it, under the same fence. A segment written within the grace for a shard being dropped stays behind; that needs a replica building from a checkpoint more than an hour superseded.
Defaults. A world's index is evicted after 30 minutes without a body search, and catch-up reads with the store's 16 shard workers.

Revised the same day after review: the compactor writes segments after the checkpoint object, so optional work can never fail a checkpoint, and a drop collects segments with one sweep of segments/ instead of per shard: a part goes once its shard is in no kept checkpoint and it is past the grace, at its listed generation. The sweep also takes parts a cut-short write or a late repair left behind, so the leak noted above is gone.

Amendment: the checkpoint grace is configurable (2026-10-03)

At the maintainer's request the grace in the collection amendments above drops from one hour to 15 minutes by default, and a world can set it as bucket.checkpointGrace (Helm passes it through). One value now serves both rules: a checkpoint stays while its successor is younger than the grace, and a shard, root or segment part is deleted only once older than the grace at the drop's start. The age past which a create rewrites stored bytes it reuses is fixed at 7.5 minutes, so replicas configured with different graces stay fenced, and the store refuses a grace under 15 minutes: a compactor that reused bytes just younger than that age needs the rest of the grace to finish its run, which is capped at 5 minutes. A shorter grace also makes each drop cheaper, since it reads fewer kept roots and lists fewer segments.

Amendment: no migration (2026-10-03)

The maintainer dropped migration: no world outside our own tests uses the bucket store, so existing schema 1 worlds are emptied at rollout and start again from genesis. The in-place conversion, the head replace race, the rollback tool and the federation rebuild at migration are not built. A schema 1 head stays refused at open.

Amendment: the change feed reaches past the ring, and slots are collected (2026-10-03)

Decided with the maintainer in phase 6. The hub read its backlog only for a cursor within one ring (4096 events) of the head, so the federation deriver, whose saved cursor is up to a minute old, rebuilt a world (about 14 hours at 1,000,000 documents) whenever more than 4096 changes had passed, whatever the slots retained. At 400 writes per second that is every deriver restart.

Reach, not ring. A resume older than the hub's ring reads the slots when the cursor is at most 16,384 slots behind the head, which bounds one resume to that many reads (about $0.007). A lag of at most 16,384 changes passes without a listing, one over 16,384 × 32 fails without one, and between them the slot names are counted. That is about 10 to 20 minutes of writes at 400 writes per second, and hours at low rates.
Paged. A hub subscription reads the backlog a ring's worth at a time and keeps reading slots until it reaches the ring, so a slow reader is not lapped while it catches up. The fan-out writes a catch-up one 64 KB batch at a time and joins its own ring wherever that ring holds what follows; coalescing applies per batch, as SPEC 6.8's "at any moment" allows.
Slots are collected. After writing a checkpoint, the compactor deletes, oldest first, the slots the oldest kept checkpoint covers that were 24 hours old when the drop began, each at its listed generation; the listing stops at the first slot it keeps. A cursor before the retained slots resyncs.
Stale replicas reload. A replica whose snapshot has gone unconfirmed for 12 hours reloads from the newest checkpoint before it treats a missing slot as the log's tip; otherwise it would serve a stale world and commit into a collected slot's name, forking the log. Its section index resyncs against the reloaded snapshot. The argument rests on real object ages: a slot created after a confirmation is younger than the time since it.
Slot cache. The backlog's cache of slot events empties after 30 minutes without a backlog read.

This replaces "slots stay" in the earlier amendments. History blocks and the blobs of pruned versions still stay.

Amendment: full-feature scan (2026-10-03, later)

A review of the whole branch before rollout tightened four points; none changes the format.

No schema 1 code at all. The named refusal of a pre-commit-log head is gone with the rest of the old format. Such a head now fails the marker's strict decoding as any foreign object does, so a world missed by the wipe still fails closed. The peer hint line is commit <world> <sequence>; the head token kept for mixed-version replicas is gone.
A slot is never rewritten. Slot creates skip the freshen rewrite that content-addressed objects use, so equal bytes under a slot's name are always its own create.
The stale reload is shared and bounded like a compaction. It runs detached from the request that noticed the staleness, loads the newest checkpoint it found, and every refresh waits on it up to its own deadline; Close waits for it.
The compactor writes only on its own base. A snapshot that failed to rebase on the newest checkpoint is refused rather than written over shards a drop may have taken, and an adoption older than the served snapshot's checkpoint is ignored.

Amendment: segment windows, install ordering, an unwatched hub (2026-10-03, last)

Decided with the maintainer after the full-feature scan.

Segments are a windowed cache. A segment is keyed by shard label (the layout's bits and the shard's index) and a 10-minute wall-clock window, segments/<bits>-<label>/<window>/<part>.json, schema 2, no longer by shard hash. A checkpoint writes a rewritten shard's segment at most once per window, carrying bodies from the label's newest segment by body hash (a split shard from its parent's); a later checkpoint in the window leaves it. A builder lists segments once, reads each label's newest, and reads from blobs what it lacks or holds at an older hash, at most about a window's changes. Parts are created if absent: a racer's part serves as well, since every body is verified on read. Checkpoint bytes are unchanged, so racing compactors still agree. A checkpoint sweeps parts a newer window superseded, and those of another layout's bits, once past the grace. At 400 writes per second over a million documents this cuts segment traffic from about 5 GB each way a minute to the same once per window.
Installs never wait on the bucket. refreshMu still lets one probe of the log read the bucket at a time; a separate install lock orders installs and is never held across a bucket call, so a commit confirming its slot never waits for a probe. A catch-up a commit overtook starts again from the newer snapshot.
An unwatched hub holds no ring. A hub with a backlog drops its ring when its last subscription closes and resumes page from the slots until one opens; subscriptions are closed by their holders.
Reads take one history block. A read of one version, and every write's base, reads the block that holds it, never the whole history; retention is computed from the first and current versions.
SPEC 6.8: an event's op names the operation that made the change on every replica; the allowance to report an append as publish is gone.

Amendment: every format starts at schema 1 (2026-10-03)

With no migration and every bucket emptied at rollout, nothing will read an object of the old head-CAS store or of this branch's first segment layout. The marker, slots and checkpoints (logSchema), roots and shards (foldedSchema) and segments (segmentSchema) are all schema 1, as history objects already were. Where the sections above say "schema 2", read schema 1. An old head left in a bucket the wipe missed still fails to open: its fields fail the marker's strict decoding.

Found by the dry run of the local 1,000,000-document scale test, and decided by the maintainer: "do the right fix for the product". It replaces the yield in Group commit above (30 ms of jitter after each win while a peer's slot was seen within 3 s).

Why the yield failed. A loser must read the winning slot, catch up, rebuild and replay before its next create, which takes longer than 30 ms of jitter (one GET alone is 18 to 37 ms on GCS), so a winner with staged batches kept every name. A starving store writes nothing, so the winner also stopped yielding 3 s after the loser's last win. Locally, one of two replicas won no slot for 30 s and more and 160 of its writes timed out; with the window raised to 60 s, the winner yielded after every win and the loser still won nothing for a minute.
Turns. A store that loses a slot reports the winner's last sequence through its Committed hook, so its peers get the usual hint. A store that receives a hint inside one of its eight newest slots, or the one it is creating, knows a peer lost that slot and is retrying: after its next win it holds its next create until a peer's slot is installed, or for at most one second, then rebuilds the held batch on the peer's slot instead of losing its create. Each hint holds at most once, and a hint older than the hold, or already answered by a peer's slot, holds nothing. A lone store, or one whose peers do not lose to it, never waits.
Rebuild only after a handoff. Rebuilding every staged batch that the served snapshot had passed livelocked: while a peer committed continuously, the head was stale again by the time it was staged, so the store never created, never lost, and so never hinted. The held batch is rebuilt once, after its hold; otherwise a stale create loses and its loss is the signal.
Contract. Committed and backend.Follower carry every sequence a replica learns the store committed through, its own or one it lost to; the wire stays commit <world> <sequence>. Without peer hints writes stay correct, but a replica that keeps losing to a busy peer waits until that peer's queue drains, so the configuration reference says more than one replica needs peers.
Cost. With 2 ms calls, two closed-loop stores in memory committed about 365 writes each in 1.5 s, where without turns one committed about 1,100 and the other none: fairness costs total throughput under saturation. On the local fake, a loser's recovery takes 165 to 218 ms at p50 against 3 ms slot creates (its reads queue behind the fake's global lock), so contention there is bound by the fake; on GCS the recovery is about one read and a rebuild. The phase 7 gate measures it.
Blind publish. A publish without an expected version no longer pins the version it first saw: a rebuild after a lost race writes on the new current version, as SPEC 6.4 says and as one store's batch always did. This supersedes ADR 0008's "a target-document change returns the normal Mark conflict" for writes without an expected version. The replica conformance writers race blind writes too, and every backend must acknowledge each one.

Revised the same day, after the fake GCS runs: a store remembers its 256 newest slots, not eight. A peer with a short queue pipelines small slots (about 70 a second locally), so a loser's hint often named a slot more than eight back, the winner ignored it, and the backlogged replica got about two slots a second and timed out. With 256 (over a second of slots at any rate), an open loop of 250 writes/s split over two replicas acknowledged all 30,000 writes, 15,000 on each, p99 3.8 s, and a closed loop of 64 writers ran 312 writes/s with no error.

Amendment: compaction in bounded steps, and content objects are never freshened (2026-10-04)

Found by the local 1,000,000-document scale test (journal). Once a checkpoint run outgrew its 5 minute bound, no checkpoint landed for 37 minutes at about 200 writes/s: a run folded a history block for every document changed since the newest checkpoint, kept nothing on timeout, and every retry started over, each present block costing a create, a full read back and, past the freshen age, a rewrite. Replay and in-memory version records grew without bound, against the plan's invariant. Fixes, as the maintainer proposed:

Content objects skip the fence. History blocks and blobs are never dropped (content collection is deferred), so a create that finds their content-addressed name taken is done: one call, no read back, no freshen. Readers verify every block and blob by hash. An ambiguous create is settled by a metadata read. Shards, roots and checkpoints keep the generation fence of the collection amendments, since drops take them. Should content collection ever land, the fence returns to those kinds with it.
Bounded steps. A checkpoint folds at most 32,768 documents. The compactor keeps a base: the snapshot at the newest checkpoint, rebased on it, sharing every unchanged node with the served snapshot (the loaded checkpoint at open, the written or adopted one after). When the served snapshot holds more changed documents than one step, the compactor replays the log from the base to the first slot boundary at which the step touches 32,768 distinct paths and checkpoints that snapshot; a run loops steps until the served snapshot is checkpointed, each step under its own 5 minute bound, so progress accumulates and every step lands. The boundary depends only on the log and the newest checkpoint, so two compactors cut the same step and write the same bytes. A replica that adopts a peer's checkpoint replays its base to it, reading the slots between, so it can step too; a base that cannot follow is dropped and the next own checkpoint sets one, and until then that replica writes the whole backlog as before.
Adopting ends a run. A compactor that finds a peer's newer checkpoint adopts it and stops, so with two replicas the one that stepped first tends to keep stepping while the other adopts; the recheck before each step keeps duplicate steps rare, and a duplicate now costs one create per present object. The single-compactor claim across replicas stays undecided.
Memory. The base costs the divergence since the newest checkpoint: old document states and copied tree nodes for every document changed since, about 1 KB each, bounded by one checkpoint interval when the compactor keeps up and by the backlog when it does not, shrinking with every step. On the 1,000,000-document world with a 440,000-sequence backlog it was 385 MiB at cold start. Guarded by TestSteppedCheckpointsLeaveNoResidue and TestCheckpointedStoreHeapTracksLiveData.

Cold start now derives the served snapshot from the loaded checkpoint instead of applying slots into it, and a replay reads slots 256 at a time so a bounded step stops without reading a whole page. Not changed: the step size is a constant, and the compactor's bursts stay unpaced (about 1,000 creates/s against the GCS bucket envelope), both for the GCS gate to size.

Revised the same day, after the rerun's seed: "adopting ends a run" did not keep one compactor. Both replicas install the same slots, so both triggers fire within the 10 s jitter from the same base and, the boundary being deterministic, write the same step at the same time; the recheck between steps never saw the other's checkpoint because a step takes 60 to 300 s. The two leapfrogged for 25 minutes of the seed, then wrote identical steps in lockstep, the loser reading every present shard back; on the fake the doubled uploads stalled slot creates (open loop at 200 writes/s: 93/s acknowledged, 30,558 timeouts). Decided with the maintainer, with no claim object: a store writes a step only when the newest checkpoint is its own, is checkpoint zero, or is older than the step bound (5 minutes, judged by one metadata read of the checkpoint object); otherwise it adopts and ends its run. A checkpoint is a store's own only when its plain create made it, so of two compactors racing one step exactly one goes on. One compactor in steady state; a dead one is replaced within the bound, during which the lag grows by at most the bound's writes. Tests: TestFollowerDefersToAFreshCompactor, TestRacedCheckpointIsNotOwn, and the race in TestStepBoundaryIsDeterministic.

Revised again the same day, with the maintainer: the checkpoint cadence is 8,192 slots or 10 minutes, not 1,024 slots or 60 s. The minute cadence dates from the spike, when a full checkpoint cost 6 to 13 s; with spread writes over a million documents every step rewrites all 4,096 shards, about 800 MB, and every follower reads them back to adopt, so a step a minute at 200 writes/s was about 13 MB/s of uploads per world plus the same per follower (on the fake, a 40 s stall of slot creates per step; open loop at 200 writes/s acknowledged 170/s with 7,798 timeouts, all inside two stalls). A checkpoint exists only to bound replay and in-memory version records, and the slots already are the O(change) deltas: 1,024 slots replay in about 0.9 s on GCS and a version record is about 150 bytes, so ten minutes of writes at 200/s cost about 7 s more cold start and about 18 MB of heap, against a tenfold cut in shard traffic. The step bound and the defer rule are unchanged. A tiered index that merges deltas into shards rarely would make checkpoints O(change) too; that is a format change, left for after the GCS gate measures this cadence.

Also revised 2026-10-04, from PR 538 review: "installs never wait on the bucket" now holds on a reload too. A stale or diverged reload replayed slots under the install lock, so a commit confirming its slot waited for the replay, up to the 5 minute reload bound. The replay now reads outside the lock, buffers the applied slots' events, and installs only if no commit moved the served snapshot past its tip; if one did, it replays on and tries again. Test: TestCommitConfirmsWhileAReloadReplays.

Amendment: a loser keeps its reads, and the hold is 300 ms (2026-10-04, several writers)

Found by the in-process model of several replicas writing one world (plan), after the GCS gate ran three writers at 94 writes/s with p50 5 s against 365 with one. Branch fix/several-writers.

The defect. A lost slot rebuilt the head on the winner's slot and requeued the two batches staged behind it (speculative batches are requeued, not rebased). Their warm-ups were spent at their first build, so the committer rebuilt them reading each updated document from the bucket, one at a time, before it could see the head's stage event and create again: about 1.2 s at 22 updates a batch and 25 ms a read, against a hold of 1 s. The hold expired before the loser came back, the winner created again, the loser lost again, and one replica starved. Writes to new paths read nothing, so a model of creates alone ran three writers as fast as one and missed it; the gate's load was 70% blind updates of random documents.
Fix. A requeued request carries what its batch read (batchObjects.asWarmup), and the batch that takes it adopts those reads once. A loser's recovery is then the winner's read, one listing and a rebuild in memory: 120 to 175 ms in the model.
Hold. handoffWait is 300 ms, about twice the measured recovery. 1 s and 300 ms gave the same throughput in the model; 300 ms wastes less on a stale hint, 100 ms and no hold lost more slots or starved a store.
Not taken. A loser creating again on the winner's slot alone, without the listing: no throughput gain, more losses, p50 equal or worse.
Unchanged. Every replica still commits for every world; the in-store handoff of a loser's batch to the replica at the tip was not needed. Fairness still costs latency: an open loop at 300 writes/s split over three replicas answers at p50 600 to 660 ms against 200 ms on one, since a write waits for its replica's turn.
Model and regression test. TestSeveralWritersModel in committer_test.go: stores on one blob.Memory behind GCS-shaped latency (create 45 ms, get 25 ms, list 30 ms, a tenth of jitter), a 2,000-document seed, 30% creates and 70% blind updates, closed loop 64 writers a store and open loop 300 writes/s, R = 1, 2, 3, each store's slot creates probed. Before the fix R = 3 ran at 91 writes/s closed with one store starved and p50 6.4 s open; after, 211 to 237 writes/s against 219 to 235 alone. The test holds two and three writers within 70% of one, every store within half its share, recovery p50 under 500 ms, the open loop served whole at p50 under 1.5 s, and every acknowledged write in exactly one slot; TestDiscardedBatchesKeepTheirReads holds the read count across a lost slot. Each fails on its mutation. One GCS lane confirms the constants.

Confirmed on real GCS the same day (lane report): three replicas served an open loop of 300 writes/s whole (298/s, 0 errors, p50 559 ms) against the gate's 94/s with 8,861 timeouts; closed loop 64 split over three 147/s at p50 387 ms against 58/s; a loser's recovery 46 to 80 ms at p50; 148,540 acknowledged versions read back, 0 bad. The remaining latency tails are compaction steps and adoptions on writing replicas, as the gate had already seen with one writer.

Amendment: the probe is a flight (2026-10-05, reads and freshness)

Planned in Refresh as a shared flight from the full-feature scan's "the committer waits on refreshMu while read probes do bucket I/O". Branch fix/refresh-flight.

The defect. Store.refresh took a mutex and probed the bucket under it, so every caller that needed the tip waited on whoever held it, with no bound of its own: reads by path, the committer's head refresh, a loser's catch-up after a lost slot, the follower's poll and the stale reload. One bucket call at a GCS tail held them all for its duration, about a second with blob.Hedged.
A shared flight. One probe reads the bucket for a world at a time. A caller that arrives while one is in flight waits on its done channel or its own context, whichever ends first, then takes the served snapshot when it was confirmed after the caller asked and otherwise starts the next probe. The asked rule is unchanged: a missing next slot confirms the tip as of the probe's start, a listing as of its own start, so a caller that asked after that time probes again. The flight is a channel that dies with its probe; no state outlives it.
A loser catches up outside the flight. It holds the winner's slot already and reads only the listing after it, installing under installMu with the existing restart check. If the served snapshot moved past the winner's slot meanwhile, the rest is an ordinary refresh through the flight. A loser's recovery never waits behind a reader's probe, at the cost of a bounded duplicate read next to a concurrent probe.
The reload installs through start's own loop, which replays past any slot a concurrent probe or commit installed before it installs, so the mutex it took around that install is gone too.
Unchanged. installMu and its rule: a commit confirming its slot never waits on a probe. Nothing above the store; backend.Store and backend.Follower are untouched.
Tests (refresh_test.go): a read by path fails on its own 200 ms deadline while another caller's probe is stalled on a held Get; a store whose create lost during such a stall creates again with at most four bucket calls between the lost create and the next; two refreshes share one probe when its listing confirmed the tip after the second ask, and the second probes again when a missing slot confirmed it before. The first two fail on the mutex. The replica and change conformance suites and TestSeveralWritersModel are unedited.

Revised after review the same day (PR 540): a probe that fails on the bucket answers every refresh waiting on its flight with that error, so a persistent failure costs one probe per burst; only a probe that ended because its starter's context did leaves the waiters to probe again. A loser whose served snapshot already passed the winner's slot, installed by a concurrent probe, rebuilds on that snapshot and never joins the flight; the earlier draft sent it through refresh, which could wait on a stalled probe up to the request timeout and fail the batch.

Amendment: a compaction role, and adoption off the install lock (2026-10-05, checkpoints)

Planned in The compactor off the writers from the GCS gate's list and the several-writers lane, whose tails were compaction steps and adoptions on writing replicas. Branch feat/compactor-role: steps 1, 2 and 4a of the plan. The deploy topology (step 3) and the GCS lane (step 4b) are still open.

A role per replica. bucketstore.Options.Compaction is Eager (the default and today's behavior: trigger at 8,192 slots or 10 minutes, step, defer to a peer's checkpoint fresher than one step bound, never to its own, keep a base) or Backstop. A backstop runs the same trigger but steps only when the newest checkpoint, whoever wrote it, checkpoint zero and its own included, is older than backstopDelay: the cadence plus two step bounds, 20 minutes, which covers an eager replica's trigger, its step and one more. It keeps no base (start, keepBase and advanceBase skip it), so when it does step it writes the backlog whole, and it adopts peers' checkpoints as before. One compactor in steady state, the first step after genesis written once while an eager replica lives, a dead eager replica replaced within about 20 minutes of writes.
The role is a server setting. compaction.role: eager | backstop at the top level of the knowledge server's YAML, default eager, validated at load; run.go hands it to every world's store. A config field rather than an environment variable because the strict YAML is the server's one configuration surface (nothing else is read from the environment), a typo is refused with the field's name, and a chart renders a second ConfigMap for the compactor Deployment as easily as a second env entry. Not hot-reloaded, like the listeners and peers.
Adoption off the install lock. installAdoption derives and rebases the served snapshot outside installMu, then takes the lock and installs only if the served snapshot is still the one it rebased; if a commit installed meanwhile it rebases again on the new one, as start replays on. The third attempt rebases under the lock, since a restart repeats the whole rebase and a writer committing faster than one would never finish. Measured: a 32,768-entry rebase onto 32,768 documents takes 34 ms (209 ms under the race detector) against a slot every 50 to 200 ms on a loaded writer, so the first attempt usually lands. Batches built before the adoption still rebase as they install, once each, as before.
Unchanged. The defer rule for eager replicas, the step bound, the cadence, checkpoint collection; backend.Store, backend.Follower and everything above the store but the role's wiring.
Tests (role_test.go): a backstop next to an eager store writes nothing, the first step after genesis included, and still nothing once the eager store's checkpoint is past the step bound; alone, with store.now past the delay, it writes the backlog in one checkpoint; a backstop's own trigger writes nothing; a backstop keeps no base across 50 adoptions of production-shaped writes, memtest.Retained outside the bucket under two agent bodies; a commit confirms its slot while an adoption's rebase is held by a test hook (Store.holdRebase), and the adoption restarts on the snapshot the commit installed. None builds on main; the lock test times out with the rebase under the lock. The replica and change conformance suites and TestSeveralWritersModel are unedited.
Model (TestSeveralWritersModelWithACompactor): three writers at 300 writes/s open loop for 20 s over GCS-shaped latency, a checkpointed seed aged past the step bound, a fourth store that takes no writes, checkpoints every 64 slots; writers eager (main's arrangement) against writers backstop, three runs each. Writers' checkpoint creates: 5 to 7 per run eager, 0 backstop. The fourth store's steps: 1 to 3 eager (it defers once a writer's checkpoint lands), 3 backstop, 5.0 to 6.4 s each. Writer heap at the end: 5.2 to 5.3 MB eager, 4.1 to 4.6 MB backstop, the base's share. Write p99 2.0 to 5.3 s eager, 1.9 to 8.6 s backstop; the writers' longest windows without a won slot 1.0 to 2.4 s in both: run-to-run noise larger than any difference. The model's latency is sleep, not CPU, and its adoptions read a few shards, so it cannot show the lane's effect; the p99 claim waits for the GCS lane.

Open, for the maintainer:

A backstop without a base writes the backlog whole, so a takeover after a long eager outage with a backlog past what one step bound can fold (about 30,000 changed documents at the gate's rates) fails on the bound and retries from nothing until an eager replica returns. A base built on demand from the newest checkpoint at takeover would restore bounded steps for one shard read per takeover.
Several backstops with the eager replica dead step together: their triggers fire on the same slots and the delay check passes for all, so each writes the same step (identical bytes, one create wins) every 20 minutes until an eager replica returns. A larger backstop jitter, within a step bound, would thin it at the cost of a later takeover.

Amendment: a writable world builds its section index from open (2026-10-21, near-duplicate gate)

The near-duplicate gate (near-duplicate gate) judges every PUBLISH and APPEND against the section index and answers from the newest version built rather than waiting for one covering the snapshot. Under the lazy rule above a pod's first write after open went unchecked, and on production with three pods behind the gateway the first three writes to a world passed. So a writable world now starts the build at open, off the commit path as before; a read-only replica keeps the lazy rule, since it never writes. The gate's stale read waits up to two seconds for a cold index (one activated with nothing built yet, at open or after idle eviction), then lets the write through unchecked. Idle eviction stays, with writes counting as use. Startup reads the checkpoint's body segments plus the bodies changed since, the same work the first body search did.

Revised the same day after review: the warm-up belongs to the gate, not to a store option. writepolicy.Enforce warms any store that offers WarmSections (the bucket store does, as a no-op when read-only); the library's own default stays lazy. The cold wait is the existing freshness bound (backend.SearchFreshness, one second), the same loop await uses, not a new number; a warm index answers at once with no timer or waiter accounting.

Graph: several-writers