ADR 0036: The bucket store commits through a create-only log

Status: proposed (2026-10-02). Plan: Bucket store commit log. Evidence: phase 0 spike on real GCS. Amends ADR 0008 and ADR 0015; supersedes ADR 0031. Part of the decisions hub.

Context

ADR 0008 commits every mutation by replacing _demarkus/v1/head.json, the world's only mutable object, and assumed a read-heavy world under one mutation per second. GCS allows about one update per second to one object name, so the bucket store paces head attempts at 1.5 s. The product now needs many users writing one world at once.
Measured, the head is the ceiling: 0.67 commits/s paced on real GCS, and 1.2 commits/s from three unpaced writers with 552 of 692 attempts throttled. On kind, one writer took 1.51 s per publish and a four-writer burst lost 49 of 200 publishes to the 10 s request deadline.
The cap belongs to any design where every commit mutates one object name (CAS, lock, lease, pointer). Creating a new name with create-if-absent has no per-name cap.
The second wall is bytes. A commit rewrites the document's history block and manifest, one of 256 shards and the root. A shard holds a 256th of the world at about 850 bytes per document: about 3.3 MB at 1,000,000 documents, so 250 writes/s would upload, and every replica would decode, about 830 MB/s.
The knowledge server reaches past the backend contract. Its world manager holds the concrete *bucketstore.Store, compares peer hints with its head sequence, schedules its polls, creates the world's genesis and calls its constructor inline, so a second backend could not replace it without changes above the store.

Decision

Commit point

A commit creates _demarkus/v1/log/<first sequence as 16 hex digits>.json with blob.Store.Create (create-if-absent: ifGenerationMatch=0 on GCS, If-None-Match: * on S3). The successful create is the commit, and a slot is never rewritten. This replaces ADR 0008's head replacement as the commit point.
A slot holds one batch of mutations. Per mutation it carries the receipt, the new index entry (path, current version, archive state, body hash, modified time, catalog record, history block references), the new version record and any retention prune range. It also carries the schema, world ID, first sequence, the writing store's ID, the sweep epoch and the hash of the previous slot, so the log is a verifiable chain of contiguous sequences.
A commit writes the content-addressed document blobs and one slot, nothing else: about 1.1 objects and the body plus about 1.2 KB per mutation, whatever the world's size. No root, shard, manifest, history block, change block or head is written on the commit path.
The manifest layer is folded into the index entry, which carries its history block references; there is no per-document manifest object.
Every writer racing from one tip targets the same slot name. A loser reads the winning slot, rebuilds only the members whose path, ancestors, descendants or the world's publish policy that slot touched, and tries the next name. Expected-version, topology, policy and archive semantics are unchanged.
An ambiguous create is decided by reading the slot: absent, retry the identical create; present with this batch's operation IDs, committed; present with others, lost. ADR 0008's evicted-receipt server-error disappears.

Group commit

One committer per world in each store takes whatever is queued, up to a batch limit of 16 that may grow to 32 while the queue stays deep, with no timer. Members build in queue order, each against the tip plus the members before it; a member whose precondition fails gets its normal error alone.
Up to two further batches are built and staged while one slot create is in flight, and slot k+1 is created only after slot k is confirmed. Every member is acknowledged only after its slot is confirmed.
Every replica commits for every world it serves. A store that has seen another store's ID in a slot within the last 3 s waits up to 30 ms of jitter after each win; a store alone in a world never waits.

Reads and freshness

A replica's snapshot is the latest checkpoint it loaded plus every slot applied since. Slots carry full entries, so applying one needs no further reads, and a replica installs its own commits as they succeed.
Reads by path (FETCH, VERSIONS, chain verification) and directory checks are exact: they see every commit acknowledged on any replica before the request. They check the next slot name first, one metadata read of 11 to 14 ms. Directory checks are exact because FETCH asks one before it reads a version path.
Catalog reads (LOOKUP, LIST and hash lookup) may trail another replica's commit by up to one second: a replica serves them from a snapshot confirmed current within the last second, which costs at most one tip check per second per world. This relaxes ADR 0008's immediate LOOKUP and hash freshness. Write preconditions are always evaluated at the tip, so the lag never affects write correctness.
One request still reads one snapshot. A view pins its snapshot at its first read: an exact read pins the tip, a catalog read may pin a snapshot up to one second old.
The bound is a property of the backend contract. It is documented on backend.ReadView and in the SPEC and proven by the conformance suite; handlers do not know which backend serves them.

Checkpoints, world marker and cold start

Checkpoints are derived, not committed. Any store's compactor may write one when the newest lags the tip by 1024 slots or 60 s, after a random delay. A checkpoint at sequence S is a deterministic function of the log up to S, so racing compactors write identical bytes through create-if-absent. Readers verify it against the slot chain, and a missing or failed checkpoint costs only replay.
A checkpoint keeps ADR 0008's content-addressed, verified object model: a root, shards whose entries carry history block references, history blocks and section segments, each written only when it changed and otherwise referenced by hash. Shard count scales with the world at about 200 KB per shard: 4096 shards at 1,000,000 documents.
head.json becomes a frozen world marker in schema 2: world ID and format, created once for a new world or replaced once at migration, never written on the commit path. An old replica refuses it and fails closed.
A new world starts with the marker and checkpoint zero, an empty root at sequence 1. Cold start lists checkpoints/ for the newest, loads it, lists log/ from its sequence and applies the slots. blob.Store.List gains a start offset for this, with no GCS types in it.

Section index

The body-search index (ADR 0012) is never built on the commit path or when a slot is applied. A world builds it on its first body search, from checkpoint section segments plus the bodies changed since; until then body search answers as catalog fallback, which LOOKUP already reports as inconclusive. It is kept current in the background, reported as partial while behind, and dropped when body search goes idle.

Change feed

Slots are the durable change record (ADR 0028), and the hub's backlog reads them through an LRU of recent slots. Slots are retained at least as long as the hub's ring and the federation deriver's checkpointed cursor age, so a resume inside that window never resyncs.
ADR 0031's sealed change blocks are removed: sealChanges, sealedThrough, the block LRU and the changes/ objects. A slot already holds up to 16 changes.

Scope

Bucket only. GCS, and later S3, stays the only state; a metadata database stays out (ADR 0010).
Ordering per world. WATCH cursors and the change feed stay numbered per world (ADR 0025, ADR 0028).
A baseline of 1,000,000 documents per world. Every gate, shard count, checkpoint interval and cold start bound is sized for it, and ADR 0008's supported limit of 100,000 rises tenfold. Write cost depends on the change, never on the document count.
Throughput. At least 200 writes per second per world, and p99 at most 250 ms for one store at 200 writes/s.

Store boundary

Everything above is private to bucketstore. The server sees a world's store only through the backend contract in server/internal/backend and constructs it at one wiring point, so swapping bucketstore for filestore, an S3 store or a database changes one constructor.

What the world manager holds:

backend.Store: OpenReadView, Publish, Append and SetArchived, with the freshness property above on its views.
backend.ChangeSource: the store's change hub, nil without WATCH. The hub counts readers waiting for its next event, so a store can schedule work that only live watchers need; a subscription nobody reads does not count, and callers have nothing to release.
backend.Follower, new, for a store several replicas share. Follow(sequence int64) says a peer replica committed through sequence: the store reads ahead of its own schedule unless it already serves that sequence, and a store whose reads are always current ignores it. It never blocks.
io.Closer, which stops whatever the store runs in the background, ends its watches and refuses later writes with backend.ErrClosed.
One store factory in the world manager's config. It takes the world's config, a logger and a commit hook, func(sequence int64), which the store calls after each of this replica's own commits. The server's wiring in run.go is the only code that names bucketstore or a blob store.

Peer hints carry only a world ID and a sequence, never a slot name, checkpoint or object key. The world manager hands an incoming hint to that world's Follower and does nothing else with it.

Private to bucketstore: the blob store and every object key; the log, slots, checkpoints and the world marker; the committer, batching, staging and the yield rule; the compactor and the collector; when to read the bucket, including the backstop poll for watchers (5 s today, run only while a watcher waits on the hub) and coalescing of peer hints; creating a new world's genesis inside open, refusing a bucket that holds foreign objects and leaving read-only worlds untouched; migration; building and evicting the section index.

The refactor that sets this boundary lands before any log code, on today's head CAS store, and changes no behavior. Today's leaks are all in server/knowledgeserver/worlds.go: worldEntry.store is a *bucketstore.Store; Hint compares HeadSequence(); pollChanges, changePollInterval and the kick channel schedule Poll; ensureGenesis calls bucketstore.EnsureWorld; and bucketstore.Open is called inline with its options and Committed hook. A world manager test serves a world on a store that is not a bucket store, through the factory.

Consequences

Measured on real GCS from GKE on 2026-10-02 unless noted.

Throughput. Batch 16 with two batches staged ahead sustained 352 to 355 writes/s; batch 32 with one ahead, 470 to 512. One batch ahead capped batch 16 at 256 to 291, because the next batch waited on the slowest of its parallel blob creates; three ahead added nothing.
Gate. 250 writes/s for 10 minutes on a 1,000,000-document world, with a following replica and a compactor: 150,012 writes, none lost, no gaps or chain breaks, no 429 or 5xx, p50 143 ms, p99 196 ms, p999 319 ms.
Latency. Without a queue, one store: p99 200 ms at 200 writes/s and 306 ms at 300. A write waits for batch formation, its blob, the slot in flight and its own slot, about 3.5 creates at p50 and 5 at p99. A small create costs 40 ms at p50 and 50 ms at p99 alone, 45 to 52 ms at p50 under load.
Several stores. With 2 or 3 stores racing one world, total throughput stays at the single-store level and about half the slot attempts lose. Without the yield one of three stores got 51 writes/s against another's 151, with p99 6.5 s; with it the stores evened out (95, 99 and 78 writes/s) and the worst p99 fell to 3 s, for about 10% of throughput. Under saturated contention p99 is in seconds and set by queueing.
Bucket. A warmed bucket took 1200 creates/s without a 429, and latency doubled from about 1000/s; at about 1.1 objects per write that puts one world's ceiling near 900 writes/s. A cold bucket was not tested.
Cold start. At 1,000,000 documents: 7.6 s (10.3 s on a contended node), index heap 1.04 GiB, and 1024 slots replayed in 0.9 s. A full checkpoint (863 MB) took 6 to 13 s, and one a minute showed no clear effect on write latency. At 100,000 documents a fixed 4096 shards took 4.2 s, bound by object count, which is why shard count scales with the world.
Freshness. A replica following by reads applied 250 writes/s with apply lag p99 166 ms, and body catch-up with 16 workers stayed under 200 ms at p99, both well inside one second. Catalog reads on another replica may now trail a write by up to a second, and the SPEC says so; requests that reach the replica that committed see it at once.
Memory. A first body search at 1,000,000 documents answers as catalog fallback for about 20 s while 3 GB of section segments load, and resident sections are about 3 GB per active world per replica. That catalog and section memory wall is out of scope and needs its own plan before several baseline worlds share a process.
Migration. A new replica checkpoints a schema 1 head and replaces it once with the marker. Cursors older than the migration sequence get one resync, which triggers a federation rebuild per written world unless ADR 0031's blocks stay readable until the deriver passes it. Rollback is open: a tool that writes a schema 1 head from a fresh checkpoint, or a one-way migration.
Hash lookup stays in the one second class until ADR 0035 is decided; if it is accepted, the body-hash map leaves snapshots and checkpoints.
Garbage collection follows reachability from retained checkpoints, retained slots, backup pins and imports. The sweep epoch is a log entry, which keeps ADR 0008's fencing.
A new backend implements backend.Store, ChangeSource if it supports WATCH and Follower if replicas share it, passes the conformance suites, and is wired by one constructor.

Alternatives considered

Group commit on head.json. Still one replace per batch at 1.5 s, a hot object whose 429s delay the whole batch, and a receipt window that must grow with the batch.
Slots that point to a rewritten root and shards. Removes the head cap but keeps a per-commit write that grows with the world: about 3.3 MB per write at 1,000,000 documents.
A two-level shard tree rewritten per commit. Tens of KB per write, but an extra object per write and a cold start that reads 65,536 leaves.
A per-world writer elected above the store, with writes forwarded to it. Puts the bucket store's concurrency shape into server routing and needs pod-to-pod transport that a database store would not. Rejected as a layering violation.
An in-memory head shared between replicas and flushed every 2 s. Loses acknowledged writes on a crash unless acknowledgement waits for the flush, which is group commit on the head; making it safe needs quorum replication.
Per-document conditional writes, or per-version create-only objects. No world sequence for WATCH and the feed, topology races between creates, and a second commit point beside the log that a crash can split from it.
Section entries carried in slots. A section entry is about as large as the body, and every replica would hold sections for every world whether or not anyone searches it.
A metadata database beside the bucket. Out by the bucket-only decision.
Libraries. glassdb, objwal, SlateDB and delta-go were surveyed and none fits (alpha, single writer, one rewritten manifest, or no GCS). Delta Lake's log plus checkpoints and SlateDB's put-if-absent commit are design references.
Peer hints as a hub catch-up (Hub.CatchUpTo, weighed in review 2026-10-02). It would drop backend.Follower, but hints also keep catalog reads inside the one second bound whether or not WATCH is on, and the hub would then schedule store reads.
The freshness class chosen per verb when the view opens (weighed in review 2026-10-02). It would make the class explicit, but handlers would have to know about it; classing reads by method, with directory checks exact, keeps them unaware.

Amendment: collection without a log barrier (2026-10-02)

Decided with the maintainer while building phase 4. It replaces "The sweep epoch is a log entry, which keeps ADR 0008's fencing" and the sweep epoch named among a slot's fields; slots carry no epoch.

Why. A barrier in the log fences only slot writers. Checkpoints are written outside the log, so a compactor in flight across a sweep could name an object the sweep deletes, and the writer and its peers would rebase on a broken checkpoint. The barrier would also stall every write for the whole deletion phase.
Generation fence. A sweep deletes an object only when no kept checkpoint or retained slot reaches it and it was older than a one hour grace when the sweep began, and each delete is conditioned on the generation the sweep listed. A writer or compactor whose create finds its bytes already stored, older than half the grace, rewrites them in place first. The new generation makes a racing delete fail, and a delete that already won makes the create write the object again. This fences writers and compactors alike and never stalls a write.
What is kept. Every checkpoint written in the last hour and at least the newest three; everything they reach; every slot after the oldest of them, with the blobs those slots' versions name; slots before it for 24 hours, for the change feed and the federation deriver's cursors. Backup pins and imports join the roots once they exist; neither does yet.
Who sweeps. Each store tries hourly, and the first to create that hour's create-only claim object sweeps. Each store also catches up and rebases on the newest checkpoint every 20 minutes, so none rests on a collected one. A store whose snapshot has gone unconfirmed for 12 hours (half the slot retention) reloads from the newest checkpoint before trusting a missing slot as the log's tip.
Backends. The fence needs a delete conditioned on the listed version. GCS has it (ifGenerationMatch); an S3 backend must confirm conditional deletes before it lands.

Amendment: collection deferred (2026-10-02, later)

The maintainer deferred content collection after building it, since today's worlds grow by a couple of gigabytes a month. Until it is needed, a compactor that writes a checkpoint deletes the checkpoint objects past the newest three that are more than an hour old, each delete conditioned on the generation it listed, so a cold start's listing stays short. Nothing else is deleted: superseded shards, roots, history blocks, blobs of pruned versions and slots stay, as ADR 0008 accepted. A replica resting on a deleted checkpoint keeps reading, because everything that checkpoint reached is still stored.

The generation fence above is the decided design for when collection is built. The full collector (sweep, mark, the rewrite of reused objects, slot retention, the stale replica reload and the maintenance loop) was written and tested, and was set aside as a patch rather than merged.

Amendment: old shards and roots are collected (2026-10-02, latest)

Shards and roots are the bulk of what checkpoints leave behind: changing one entry rewrites its whole shard, about 200 KB. Nothing reads one once its checkpoint is dropped, so they are collected after all. History blocks, blobs and slots still are not.

Kept checkpoints. The newest three, and any whose successor was written within the last hour. A checkpoint therefore goes only after it stopped being the newest more than an hour ago, by which time any compactor building on it has finished (each run is capped at 5 minutes).
What goes with a dropped checkpoint. Its shards and root that no kept checkpoint uses, then the checkpoint object last. A drop cut short leaves the checkpoint object, and the next drop finds its shards and root through it. A missing root means its shards already went.
The fence. Shards and roots are content-addressed, so a new checkpoint whose shard returned to an earlier state reuses the old object. A shard or root is deleted only when it was older than an hour when the drop began, and only at the generation the drop read. A create that reuses stored bytes older than 30 minutes rewrites them first, so a pending delete fails. This is the generation fence from the first amendment, limited to shards and roots.

Amendment: catalog fallback and body segments (2026-10-02, phase 5)

Decided with the maintainer before phase 5 was built. It replaces "reported as partial while behind" in the section index decision above, and the bucket store consequence of ADR 0012, which indexed every body at open and on refresh.

Fallback, not partial. A body search waits up to one second (backend.SearchFreshness) for a section index that covers exactly the snapshot its view pinned. Without one in time the store refuses with backend.ErrBodyMatchUnavailable, and the handler answers from the catalog on the same view, echoing match: catalog, which clients already treat as inconclusive. Nothing new goes on the wire. The SPEC now says a server with body match may answer a body request in catalog mode while its index is built or behind, and must not echo match: body for an index that misses part of the state the request reads.
One version per snapshot. The index keeps an immutable version for each installed snapshot, retained for the request timeout, so a body search never mixes sections from another snapshot. More than 1024 queued installs collapse into one comparison with the served snapshot.
Body segments keyed by shard. A segment holds the bodies of one checkpoint shard's live documents at _demarkus/v1/segments/<label>/<shard hash>/<part>.json, in parts of at most 2 MiB encoded because objects are capped at 4 MiB; part 0 names the part count and is written last. It holds bodies rather than serialized sections: about 1.33 times body bytes against 2.34, measured on the soul corpus, and it never goes stale when the tokenizer is retuned. The root names no segment, so checkpoint bytes stay a function of the log; a missing segment only costs blob reads. A reader uses a body only when it hashes to the document's body hash.
Writers. The compactor writes the segment of each rewritten shard, carrying unchanged bodies from the previous checkpoint's segment and reading each changed body once; unchanged bodies with no segment to carry them come from a budget of 4096 reads per checkpoint. A replica whose build read a whole unchanged shard from blobs writes that shard's segment. A world from before segments gains them through its first body searches and later checkpoints.
Collection. A dropped shard's segment goes with it, under the same fence. A segment written within the grace for a shard being dropped stays behind; that needs a replica building from a checkpoint more than an hour superseded.
Defaults. A world's index is evicted after 30 minutes without a body search, and catch-up reads with the store's 16 shard workers.

Revised the same day after review: the compactor writes segments after the checkpoint object, so optional work can never fail a checkpoint, and a drop collects segments with one sweep of segments/ instead of per shard: a part goes once its shard is in no kept checkpoint and it is past the grace, at its listed generation. The sweep also takes parts a cut-short write or a late repair left behind, so the leak noted above is gone.