ADR 0036: The bucket store commits through a create-only log

Status: proposed (2026-10-02). Plan: Bucket store commit log. Evidence: phase 0 spike on real GCS. Amends ADR 0008 and ADR 0015; supersedes ADR 0031. Part of the decisions hub.

Context

• ADR 0008 commits every mutation by replacing _demarkus/v1/head.json, the world's only mutable object, and assumed a read-heavy world under one mutation per second. GCS allows about one update per second to one object name, so the bucket store paces head attempts at 1.5 s. The product now needs many users writing one world at once.
• Measured, the head is the ceiling: 0.67 commits/s paced on real GCS, and 1.2 commits/s from three unpaced writers with 552 of 692 attempts throttled. On kind, one writer took 1.51 s per publish and a four-writer burst lost 49 of 200 publishes to the 10 s request deadline.
• The cap belongs to any design where every commit mutates one object name (CAS, lock, lease, pointer). Creating a new name with create-if-absent has no per-name cap.
• The second wall is bytes. A commit rewrites the document's history block and manifest, one of 256 shards and the root. A shard holds a 256th of the world at about 850 bytes per document: about 3.3 MB at 1,000,000 documents, so 250 writes/s would upload, and every replica would decode, about 830 MB/s.
• The knowledge server reaches past the backend contract. Its world manager holds the concrete *bucketstore.Store, compares peer hints with its head sequence, schedules its polls, creates the world's genesis and calls its constructor inline, so a second backend could not replace it without changes above the store.

Decision

Commit point

• A commit creates _demarkus/v1/log/<first sequence as 16 hex digits>.json with blob.Store.Create (create-if-absent: ifGenerationMatch=0 on GCS, If-None-Match: * on S3). The successful create is the commit, and a slot is never rewritten. This replaces ADR 0008's head replacement as the commit point.
• A slot holds one batch of mutations. Per mutation it carries the receipt, the new index entry (path, current version, archive state, body hash, modified time, catalog record, history block references), the new version record and any retention prune range. It also carries the schema, world ID, first sequence, the writing store's ID, the sweep epoch and the hash of the previous slot, so the log is a verifiable chain of contiguous sequences.
• A commit writes the content-addressed document blobs and one slot, nothing else: about 1.1 objects and the body plus about 1.2 KB per mutation, whatever the world's size. No root, shard, manifest, history block, change block or head is written on the commit path.
• The manifest layer is folded into the index entry, which carries its history block references; there is no per-document manifest object.
• Every writer racing from one tip targets the same slot name. A loser reads the winning slot, rebuilds only the members whose path, ancestors, descendants or the world's publish policy that slot touched, and tries the next name. Expected-version, topology, policy and archive semantics are unchanged.
• An ambiguous create is decided by reading the slot: absent, retry the identical create; present with this batch's operation IDs, committed; present with others, lost. ADR 0008's evicted-receipt server-error disappears.

Group commit

• One committer per world in each store takes whatever is queued, up to a batch limit of 16 that may grow to 32 while the queue stays deep, with no timer. Members build in queue order, each against the tip plus the members before it; a member whose precondition fails gets its normal error alone.
• Up to two further batches are built and staged while one slot create is in flight, and slot k+1 is created only after slot k is confirmed. Every member is acknowledged only after its slot is confirmed.
• Every replica commits for every world it serves. A store that has seen another store's ID in a slot within the last 3 s waits up to 30 ms of jitter after each win; a store alone in a world never waits.

Reads and freshness

• A replica's snapshot is the latest checkpoint it loaded plus every slot applied since. Slots carry full entries, so applying one needs no further reads, and a replica installs its own commits as they succeed.
• Reads by path (FETCH, VERSIONS, chain verification) and directory checks are exact: they see every commit acknowledged on any replica before the request. They check the next slot name first, one metadata read of 11 to 14 ms. Directory checks are exact because FETCH asks one before it reads a version path.
• Catalog reads (LOOKUP, LIST and hash lookup) may trail another replica's commit by up to one second: a replica serves them from a snapshot confirmed current within the last second, which costs at most one tip check per second per world. This relaxes ADR 0008's immediate LOOKUP and hash freshness. Write preconditions are always evaluated at the tip, so the lag never affects write correctness.
• One request still reads one snapshot. A view pins its snapshot at its first read: an exact read pins the tip, a catalog read may pin a snapshot up to one second old.
• The bound is a property of the backend contract. It is documented on backend.ReadView and in the SPEC and proven by the conformance suite; handlers do not know which backend serves them.

Checkpoints, world marker and cold start

• Checkpoints are derived, not committed. Any store's compactor may write one when the newest lags the tip by 1024 slots or 60 s, after a random delay. A checkpoint at sequence S is a deterministic function of the log up to S, so racing compactors write identical bytes through create-if-absent. Readers verify it against the slot chain, and a missing or failed checkpoint costs only replay.
• A checkpoint keeps ADR 0008's content-addressed, verified object model: a root, shards whose entries carry history block references, history blocks and section segments, each written only when it changed and otherwise referenced by hash. Shard count scales with the world at about 200 KB per shard: 4096 shards at 1,000,000 documents.
• head.json becomes a frozen world marker in schema 2: world ID and format, created once for a new world or replaced once at migration, never written on the commit path. An old replica refuses it and fails closed.
• A new world starts with the marker and checkpoint zero, an empty root at sequence 1. Cold start lists checkpoints/ for the newest, loads it, lists log/ from its sequence and applies the slots. blob.Store.List gains a start offset for this, with no GCS types in it.

Section index

The body-search index (ADR 0012) is never built on the commit path or when a slot is applied. A world builds it on its first body search, from checkpoint section segments plus the bodies changed since; until then body search answers as catalog fallback, which LOOKUP already reports as inconclusive. It is kept current in the background, reported as partial while behind, and dropped when body search goes idle.

Change feed

• Slots are the durable change record (ADR 0028), and the hub's backlog reads them through an LRU of recent slots. Slots are retained at least as long as the hub's ring and the federation deriver's checkpointed cursor age, so a resume inside that window never resyncs.
• ADR 0031's sealed change blocks are removed: sealChanges, sealedThrough, the block LRU and the changes/ objects. A slot already holds up to 16 changes.

Scope

• Bucket only. GCS, and later S3, stays the only state; a metadata database stays out (ADR 0010).
• Ordering per world. WATCH cursors and the change feed stay numbered per world (ADR 0025, ADR 0028).
• A baseline of 1,000,000 documents per world. Every gate, shard count, checkpoint interval and cold start bound is sized for it, and ADR 0008's supported limit of 100,000 rises tenfold. Write cost depends on the change, never on the document count.
• Throughput. At least 200 writes per second per world, and p99 at most 250 ms for one store at 200 writes/s.

Store boundary

Everything above is private to bucketstore. The server sees a world's store only through the backend contract in server/internal/backend and constructs it at one wiring point, so swapping bucketstore for filestore, an S3 store or a database changes one constructor.

What the world manager holds:

• backend.Store: OpenReadView, Publish, Append and SetArchived, with the freshness property above on its views.
• backend.ChangeSource: the store's change hub, nil without WATCH. The hub counts readers waiting for its next event, so a store can schedule work that only live watchers need; a subscription nobody reads does not count, and callers have nothing to release.
• backend.Follower, new, for a store several replicas share. Follow(sequence int64) says a peer replica committed through sequence: the store reads ahead of its own schedule unless it already serves that sequence, and a store whose reads are always current ignores it. It never blocks.
• io.Closer, which stops whatever the store runs in the background, ends its watches and refuses later writes with backend.ErrClosed.
• One store factory in the world manager's config. It takes the world's config, a logger and a commit hook, func(sequence int64), which the store calls after each of this replica's own commits. The server's wiring in run.go is the only code that names bucketstore or a blob store.

Peer hints carry only a world ID and a sequence, never a slot name, checkpoint or object key. The world manager hands an incoming hint to that world's Follower and does nothing else with it.

Private to bucketstore: the blob store and every object key; the log, slots, checkpoints and the world marker; the committer, batching, staging and the yield rule; the compactor and the collector; when to read the bucket, including the backstop poll for watchers (5 s today, run only while a watcher waits on the hub) and coalescing of peer hints; creating a new world's genesis inside open, refusing a bucket that holds foreign objects and leaving read-only worlds untouched; migration; building and evicting the section index.

The refactor that sets this boundary lands before any log code, on today's head CAS store, and changes no behavior. Today's leaks are all in server/knowledgeserver/worlds.go: worldEntry.store is a *bucketstore.Store; Hint compares HeadSequence(); pollChanges, changePollInterval and the kick channel schedule Poll; ensureGenesis calls bucketstore.EnsureWorld; and bucketstore.Open is called inline with its options and Committed hook. A world manager test serves a world on a store that is not a bucket store, through the factory.

Consequences

Measured on real GCS from GKE on 2026-10-02 unless noted.

• Throughput. Batch 16 with two batches staged ahead sustained 352 to 355 writes/s; batch 32 with one ahead, 470 to 512. One batch ahead capped batch 16 at 256 to 291, because the next batch waited on the slowest of its parallel blob creates; three ahead added nothing.
• Gate. 250 writes/s for 10 minutes on a 1,000,000-document world, with a following replica and a compactor: 150,012 writes, none lost, no gaps or chain breaks, no 429 or 5xx, p50 143 ms, p99 196 ms, p999 319 ms.
• Latency. Without a queue, one store: p99 200 ms at 200 writes/s and 306 ms at 300. A write waits for batch formation, its blob, the slot in flight and its own slot, about 3.5 creates at p50 and 5 at p99. A small create costs 40 ms at p50 and 50 ms at p99 alone, 45 to 52 ms at p50 under load.
• Several stores. With 2 or 3 stores racing one world, total throughput stays at the single-store level and about half the slot attempts lose. Without the yield one of three stores got 51 writes/s against another's 151, with p99 6.5 s; with it the stores evened out (95, 99 and 78 writes/s) and the worst p99 fell to 3 s, for about 10% of throughput. Under saturated contention p99 is in seconds and set by queueing.
• Bucket. A warmed bucket took 1200 creates/s without a 429, and latency doubled from about 1000/s; at about 1.1 objects per write that puts one world's ceiling near 900 writes/s. A cold bucket was not tested.
• Cold start. At 1,000,000 documents: 7.6 s (10.3 s on a contended node), index heap 1.04 GiB, and 1024 slots replayed in 0.9 s. A full checkpoint (863 MB) took 6 to 13 s, and one a minute showed no clear effect on write latency. At 100,000 documents a fixed 4096 shards took 4.2 s, bound by object count, which is why shard count scales with the world.
• Freshness. A replica following by reads applied 250 writes/s with apply lag p99 166 ms, and body catch-up with 16 workers stayed under 200 ms at p99, both well inside one second. Catalog reads on another replica may now trail a write by up to a second, and the SPEC says so; requests that reach the replica that committed see it at once.
• Memory. A first body search at 1,000,000 documents answers as catalog fallback for about 20 s while 3 GB of section segments load, and resident sections are about 3 GB per active world per replica. That catalog and section memory wall is out of scope and needs its own plan before several baseline worlds share a process.
• Migration. A new replica checkpoints a schema 1 head and replaces it once with the marker. Cursors older than the migration sequence get one resync, which triggers a federation rebuild per written world unless ADR 0031's blocks stay readable until the deriver passes it. Rollback is open: a tool that writes a schema 1 head from a fresh checkpoint, or a one-way migration.
• Hash lookup stays in the one second class until ADR 0035 is decided; if it is accepted, the body-hash map leaves snapshots and checkpoints.
• Garbage collection follows reachability from retained checkpoints, retained slots, backup pins and imports. The sweep epoch is a log entry, which keeps ADR 0008's fencing.
• A new backend implements backend.Store, ChangeSource if it supports WATCH and Follower if replicas share it, passes the conformance suites, and is wired by one constructor.

Alternatives considered

• Group commit on head.json. Still one replace per batch at 1.5 s, a hot object whose 429s delay the whole batch, and a receipt window that must grow with the batch.
• Slots that point to a rewritten root and shards. Removes the head cap but keeps a per-commit write that grows with the world: about 3.3 MB per write at 1,000,000 documents.
• A two-level shard tree rewritten per commit. Tens of KB per write, but an extra object per write and a cold start that reads 65,536 leaves.
• A per-world writer elected above the store, with writes forwarded to it. Puts the bucket store's concurrency shape into server routing and needs pod-to-pod transport that a database store would not. Rejected as a layering violation.
• An in-memory head shared between replicas and flushed every 2 s. Loses acknowledged writes on a crash unless acknowledgement waits for the flush, which is group commit on the head; making it safe needs quorum replication.
• Per-document conditional writes, or per-version create-only objects. No world sequence for WATCH and the feed, topology races between creates, and a second commit point beside the log that a crash can split from it.
• Section entries carried in slots. A section entry is about as large as the body, and every replica would hold sections for every world whether or not anyone searches it.
• A metadata database beside the bucket. Out by the bucket-only decision.
• Libraries. glassdb, objwal, SlateDB and delta-go were surveyed and none fits (alpha, single writer, one rewritten manifest, or no GCS). Delta Lake's log plus checkpoints and SlateDB's put-if-absent commit are design references.
• Peer hints as a hub catch-up (Hub.CatchUpTo, weighed in review 2026-10-02). It would drop backend.Follower, but hints also keep catalog reads inside the one second bound whether or not WATCH is on, and the hub would then schedule store reads.
• The freshness class chosen per verb when the view opens (weighed in review 2026-10-02). It would make the class explicit, but handlers would have to know about it; classing reads by method, with directory checks exact, keeps them unaware.