Bucket store commit log: create-only commits instead of one head CAS

A plan to take one world from about 0.67 writes per second to hundreds, at enterprise scale with 1,000,000 documents per world as the baseline, by committing batches of mutations as create-only log slots that carry their own index changes, and materializing the index in background checkpoints instead of rewriting it on every commit. Search reads may lag by up to one second. Everything stays behind the backend contract: nothing above the store changes how it routes, elects or schedules. Audience: demarkus maintainers. Proposed 2026-10-01, deferred until the incremental federation graph ships. Builds on ADR 0008 and replaces its commit point, its per-commit root and shard rewrite, its manifest layer and its immediate search freshness; supersedes ADR 0031. Part of the Knowledge server track.

Problem

Every mutation in a world ends with one conditional replace of _demarkus/v1/head.json, the world's only mutable object. GCS limits updates to one object name to about one per second, so bucketstore serializes commits behind commitToken and paces head attempts at defaultCommitInterval (1.5 s). The ceiling is about 0.67 commits per second per world, whatever the paths.

Measured on kind (writes): one sequential writer took exactly 1.51 s per publish; 4 concurrent writers queued to 6 s each; a 4-writer burst of 200 publishes lost 49 to the 10 s request timeout. The product target is many users writing one world concurrently, so this is a blocker, not a tuning issue.

ADR 0008 stated the assumption (read-heavy, under one mutation per second per world) and measured it, but it was never revisited when concurrent multi-user writing became the goal.

The constraint is the single mutable object name, not conditional writes. Any design where every commit mutates one object (CAS, lock, lease, pointer) inherits the same cap. Creating a new object name with create-if-absent has no per-name cap. Without the world-level ordering, each document would be its own object and capped at about one write per second on its own, which is harmless; the shared head is what turns that per-object limit into a per-world one.

The second limit appears at enterprise scale. Every commit today rewrites the document's history block and manifest, one of 256 shards and the root. A shard holds a world's documents divided by 256 at about 850 bytes each: about 330 KB at 100,000 documents and about 3.3 MB at 1,000,000. At 250 writes per second that is about 80 MB/s, or about 830 MB/s, uploaded by the writer and decoded again by every replica. Removing the head ceiling alone moves the wall to bytes.

Decisions

Taken 2026-10-01 and 2026-10-02 with the maintainer.

Bucket only. GCS (and later S3) stays the only state. A database for metadata, such as Cloud SQL Postgres beside GCS, is out; ADR 0010 stands.
Ordering stays per world. WATCH cursors and the change feed are numbered per world (ADR 0025, ADR 0028), and the federation graph is derived from that sequence. The world is the narrowest scope that can own it.
Search may lag by up to one second. LOOKUP, LIST and hash lookup on one replica may trail another replica's commit by at most 1 s. FETCH, VERSIONS and chain verification by path stay exact. Every write evaluates its preconditions at the tip, so the lag never affects write correctness. This relaxes ADR 0008's immediate LOOKUP and hash freshness.
Group commit on a create-only log. Batches of mutations per slot, pipelined, with a throughput target of at least 200 writes per second per world.
Fold the manifest into the index entry. The entry carries its history block references; no per-document manifest object.
Enterprise scale. Write cost must scale with the change, not the world: a commit writes the new bytes and one log slot, never a structure whose size grows with the document count.
Baseline: 1,000,000 documents per world. Every gate, shard count, checkpoint interval and cold start bound is sized for it; ADR 0008's supported limit of 100,000 rises tenfold.
head.json is frozen (2026-10-02). It becomes a world marker written once, at world creation or migration, and never on the commit path; there is no tip hint to keep current.
No change blocks (2026-10-02). Slots are the change feed; ADR 0031's sealed blocks and their sealing code are removed.
Bounded by the store abstraction (2026-10-02). Write concurrency control and background maintenance live inside bucketstore, behind the backend contract (ADR 0015). Every replica commits; nothing above the store elects a writer, forwards writes between replicas, or schedules store maintenance, so a different backend, such as a database, drops in without changes above it.

Concurrency model stays the same

Expected-version semantics already hold today: bucketstore checks ExpectedVersion per document and a writer that loses the head race revalidates against the new snapshot, so writers to different documents never see a conflict and writers to one document get the normal Mark conflict. The world-level ordering exists for what one document cannot carry: the change sequence, path topology (/a.md against /a.md/b.md), and one snapshot per request. The file store has the same single ordering point (filestore.go mu), it just costs microseconds on local disk. This plan keeps the model and makes the ordering step cheap, batched and independent of world size.

Alternatives considered

Group commit on head.json alone. Batching onto the head object still pays 1.5 s per batch, keeps a hot object whose 429s delay a whole batch, needs the 16-receipt window to scale with batch size, and skips ADR 0031's blocks of 8 sequences.
Log slots that point to a rewritten root and shards (this plan's first version). Fixes the head ceiling, but every commit still rewrites a shard whose size grows with the world: about 3.3 MB per write at 1,000,000 documents.
A two-level shard tree rewritten per commit. Root to 256 inner nodes to 65,536 leaves cuts bytes to tens of KB per write, but adds an object per write, keeps per-commit rewrites, and makes a cold start read 65,536 leaves.
A per-world committer elected above the store, with writes forwarded to it. Avoids slot races between replicas, but puts the bucket store's concurrency shape into server routing and needs pod-to-pod transport, and a database-backed store would need none of it. Rejected 2026-10-02 as a layering violation; contention is handled inside the store instead.
In-memory head shared between replicas, flushed every 2 s. Acknowledging before the flush loses up to 2 s of acknowledged writes when the holder crashes, and a deposed holder acknowledges writes its flush then loses. Making it safe means quorum replication (Raft), which makes GCS a snapshot store and does not survive all pods restarting together. Acknowledging after the flush is group commit on the head.
Database for metadata. Out by the bucket-only decision.
Per-document conditional writes alone. Correct for versions, but there is no world sequence for WATCH and the feed, two creates on different objects can break topology, and a commit followed by a separate feed entry can crash between the two and never be indexed.
Per-version create-only slots (docs/<path hash>/v/<N>.json, create-if-absent, so the expected version maps straight onto the create). Weighed 2026-10-01; it does not help.
- As a commit point beside the log it is a second commit point. A crash between the version create and the log slot leaves a version that path reads see but the feed and search never do, and a version name that blocks the document's next write.
- As the only commit point for updates, with the log sequenced afterwards, it needs orphan discovery (listing the bucket, or a notification service outside the bucket) and lets feed order drift from commit order.
- As a staged object it cannot be contended: a batch whose log slot loses would already hold the version name the winner needs. Anything staged before the log commit must be content-addressed.
- It adds an object per write.
- The log already gives same-document writers the same conflict answer: at build time inside a batch, and on rebase across pods.
One head per shard or subtree. A parent and child path hash to different shards, and the change sequence is per world.
Libraries (surveyed 2026-10-01). None fits: mbrt/glassdb is alpha, GCS only, has no change feed or global index; JayJamieson/objwal is single writer and orders through one rewritten manifest; SlateDB is Rust with Go bindings, single writer with fencing, a key-value model; rivian/delta-go is Parquet tables, alpha, S3 with DynamoDB locking, no GCS. Useful as design references: Delta Lake's log of actions plus periodic checkpoints and its logical conflict check on retry, SlateDB's put-if-absent commit and writer fencing.

Design

The shape is a log-structured store: the log is the source of truth and carries every change in full, and checkpoints are derived, verifiable materializations that only make reads and cold starts cheaper.

Log slots are the commit point

A commit creates _demarkus/v1/log/<first sequence as 16 hex digits>.json with blob.Store.Create (create-if-absent; ifGenerationMatch=0 on GCS, If-None-Match: * on S3). One slot holds one batch of mutations. For each mutation the slot carries the receipt, the new index entry (path, current version, archive state, body hash, modified time, catalog entry), the new version record (version, blob reference, body hash, modified time), and any retention prune range. The slot also carries schema, world ID, first sequence, the writing replica's store ID, the sweep epoch, and the hash of the previous slot's entry so readers verify the chain. The next slot's name is the tip entry's last sequence plus one, so every writer racing from the same tip targets the same name. The successful create is the commit.

A commit writes the new document blobs and one slot, nothing else: about 1.1 objects and the body plus about 1.2 KB per mutation, whatever the world's size. No root, shard, manifest, history block, change block or head is written on the commit path.

Group commit

One committer per world inside each replica's store replaces commitToken. Requests enqueue their mutation builders and wait for their outcome.
When the previous slot is confirmed, the committer takes whatever is queued, up to the batch limit (16 to start), with no timer. An idle world commits a lone write at once; under load batches fill on their own.
Members are built in queue order on a chained snapshot: member i sees the tip plus members before it. A member whose expected version, topology, policy or archive precondition fails gets its normal error and is dropped; the rest of the batch continues.
The batch's blobs are staged in parallel; they are content-addressed, so a lost race reuses them.
Every member of a batch is acknowledged only after its slot is confirmed.

Pipelining

While slot k's create is in flight, the committer builds the next batch on slot k's candidate and stages its blobs. Slot k+1 is created only after slot k is confirmed. If slot k is lost, the speculative batch is rebuilt at the new tip; its blobs stay valid. Per-slot time is about one create round trip.

Several replicas

Every replica's store commits for every world it serves; requests are served where they arrive. The slot race is the only arbitration, and it lives entirely in the store.
A lost slot costs one read of the winning slot and a retry on the next name. The loser's batch keeps its blobs and rebuilds only members the winning slot touched.
Fairness: a pipelining winner starts its next attempt one slot read ahead of the losers, so it could keep winning under sustained contention. A store that has seen another store's ID in a slot within the last few seconds waits a short jitter after each win before its next attempt. A store alone in a world never sees another ID and never waits.
Phase 0 measures per-replica latency with 2 and 3 replicas racing. If contention still costs too much, any further coordination stays inside bucketstore.

Lost slots and rebase

If the slot exists, read the slots up to the new tip; they carry their entries, so this costs no further reads. A member whose path, ancestors or descendants, or the world's publish policy changed in those slots is rebuilt so its checks answer as today, which may now return a conflict. The others are carried as they are, and the batch tries the next slot name. Rebase never restages anything. Attempts are bounded by the request deadline; there is no fixed pacing on the commit path.

Ambiguous outcomes

On a timeout or lost response, read the slot. Absent: retry the identical create. Present with this batch's operation IDs: committed. Present with other IDs: lost, rebase. A slot is never rewritten, so the outcome is always decidable; the evicted-receipt server-error path of ADR 0008 disappears.

Snapshot and read path

A replica's snapshot is the latest checkpoint it loaded plus every slot applied since. Slots carry full entries, so applying one needs no further reads.
Path reads (FETCH, VERSIONS, chain verification) check the next slot name (metadata only) before serving; new slots are read and applied. GCS and S3 both give strong read-after-write and strong not-found. FETCH of the current version is then the tip check plus the blob: two bucket calls instead of today's four.
Search reads (LOOKUP, LIST, hash lookup) serve the snapshot if it was confirmed current within the last second, and check the tip first otherwise. That bounds the lag at 1 s and costs at most one tip check per second per replica, however heavy the search traffic.
Versions. Version records since the latest checkpoint are held in memory from the slots; older versions come from the checkpoint's history blocks.
A replica installs its own commits as they succeed, so a writer reading through the replica it wrote to sees its writes at once.
Optionally, the hub's fan-out pushes new slot names to the other replicas so they usually catch up well inside the bound.
One request still pins one snapshot.

Checkpoints

The compactor is part of the store and runs in every replica; there is no leader and no Kubernetes Lease. A checkpoint at sequence S is fully determined by the log up to S and the format (canonical JSON, sorted entries, content-addressed objects), so two compactors that both checkpoint S write identical bytes and create-if-absent makes the race harmless.
To avoid duplicate work, a store compacts only when the newest checkpoint lags the tip by N slots or T seconds (starting points: 1024 slots or 60 s), waits a random delay first, and checks again before writing.
It writes the index in the ADR 0008 shape with the manifest folded in: the root, the shards, each entry carrying its history block references, and new history blocks for documents that gained versions. Then it creates _demarkus/v1/checkpoints/<sequence as 16 hex digits>.json naming the sequence it covers and its root.
Only shards and history blocks that changed since the previous checkpoint are written; unchanged ones are referenced by hash.
A checkpoint is derived, not committed. Readers verify it against the slot chain; a missing, late or failed checkpoint only means a longer replay. Correctness never depends on the compactor.
Shard count is sized for the baseline: 4096 shards hold about 250 entries each at 1,000,000 documents, about 200 KB.

World marker and cold start

head.json becomes a frozen world marker in the schema 2 format: world ID and format version, nothing that changes. It is created with create-if-absent for a new world and replaced exactly once for a migrated one, and never written again. It stays because an old replica that found no head would take the world for a new one and initialize a second genesis; a schema 2 marker makes it fail closed instead.
A new world starts with the marker and checkpoint zero, an empty root at sequence 1; the first slot is sequence 2.
Cold start lists checkpoints/ for the newest, loads it, then lists log/ from its sequence with a start offset (strongly consistent on GCS and S3) and applies the slots to the tip. Old checkpoints are collected, so the first list stays short.

Change feed

Slots are the durable change record, so the hub's backlog reads them directly, with an LRU of recent slots. The slot holding sequence q starts within one batch limit of names at or below q, so a resume lists from there.
Slots are kept at least as long as the hub's ring covers, which keeps ADR 0031's guarantee: a cursor resumes across a replica restart up to the ring. A cursor older than the retained slots gets resync.
ADR 0031's sealed blocks and their code (sealChanges, sealedThrough, the block LRU, changes/ objects) are removed. A slot already holds up to 16 changes, so a second batching layer adds nothing.

Garbage collection

Reachability is the latest checkpoints kept for rollback, every slot after the oldest of them, the feed's slot retention, backup pins and active imports. The collector's sweep epoch becomes a log entry like any other commit; a batch whose slot create loses to a barrier entry sees the active epoch, waits for the clearing entry, and re-stages its blobs, which preserves ADR 0008's fencing. Like the compactor, the collector runs inside the store and is safe to start on several replicas.

Rates

Projected 2026-10-01 for the 1,000,000-document baseline, not measured. c is one small-object create on GCS from a GKE pod. Real GCS reads in the cluster take about 20 ms per call (probe); writes have never been timed on real GCS, so c is assumed to be 40 to 80 ms. With commits writing only blobs and slots, the slot chain sets the rate: batch limit divided by c.

Commit mode	c = 40 ms	c = 80 ms	Bytes per write	Write latency
Today, head CAS with 1.5 s pacing	0.67/s	0.67/s	body plus shard, history block, manifest, root	queues into seconds
Log with checkpoints, batch 16, pipelined	about 400/s	about 200/s	body plus about 1.2 KB	about 1c to 3c
Same, batch 32	about 800/s	about 400/s	body plus about 1.2 KB	about 1c to 3c

Bucket bound. About 1.1 objects per write. GCS starts a bucket at about 1000 object writes per second and ramps up gradually, so about 900 writes per second before scaling; the slot chain binds first.
Replica refresh. A replica reads only slots: about 20 KB per batch of 16, about 0.5 MB/s at 400 writes per second.
Compactor. At 400 writes per second over 60 s, about 24,000 changed entries touch nearly every one of 4096 shards: about 800 MB per checkpoint, about 13 MB/s, off the request path. A longer interval trades checkpoint bytes for replay length.
Cold start. About 850 MB of index in 4096 shard reads plus up to one interval of slots: a few seconds of transfer and decode, to be measured.
Queue bound. The 10 s request deadline caps the queue at about ten seconds of throughput before timeouts.
Several replicas. With R replicas writing one world, the slot rate stays about the same and each replica wins about 1/R of the slots, so a replica's write latency grows toward R slot times under sustained contention. Total throughput holds as long as each winning batch is full.

Migration

A new replica that finds a schema 1 head.json at sequence S creates checkpoints/<S> naming the head's root (deterministic bytes, so concurrent replicas converge through create-if-absent). Schema 2 readers accept unfolded shard entries with manifest references in a checkpoint.
It then CAS-replaces head.json with the frozen schema 2 marker, conditioned on the generation it read. If an old replica committed in between, the replace fails; the new replica rereads the head, checkpoints the new sequence and tries again. A checkpoint left behind at an older sequence is valid state and is simply not the newest.
The first write after the marker creates slot S+1. The compactor's first checkpoint writes the folded format and the larger shard count.
No change blocks are sealed. Cursors at or after S resume; cursors older than S get one resync.
Old replicas refuse a schema 2 head (unknown fields are refused) and fail closed with server-error, so they cannot commit during a rolling deploy. No split brain.
Rollback needs a tool that writes a schema 1 head from a fresh checkpoint at the log tip in the old format; decide whether to ship it or document the migration as one way.

Invariants

A slot is created once and never rewritten. The next slot is only attempted after the current tip was observed, so the log has no gaps; each entry names its predecessor's hash, and sequences run contiguously across slots.
head.json is written once per world, at creation or migration, and never on the commit path.
The log alone determines world state: replaying slots from any valid checkpoint gives the same snapshot, and a world with no checkpoint after checkpoint zero is still correct.
A checkpoint is a deterministic function of the log up to its sequence; any number of compactors may race on it.
A batch's mutations are applied in queue order; each member's preconditions are evaluated against the tip plus the members before it, and one member's failure never fails another.
Each acknowledged mutation is in exactly one slot; acknowledgement follows only a successful create or a reread that finds its operation ID in the slot.
A speculative batch never commits ahead of its predecessor slot.
Everything written before the slot commit is content-addressed, so a losing batch never holds a name a winner needs.
A commit's write cost depends on the change, never on the world's document count.
Write concurrency control and background maintenance live behind the backend contract; nothing above the store elects writers, routes writes by replica, or schedules store work.
Slots are retained at least as long as the hub's ring covers; a resume within that window never resyncs.
Every write's preconditions are evaluated at the tip; path reads are exact; search reads serve a snapshot confirmed current within 1 s.
One request reads one snapshot.
Expected-version, topology, policy and archive semantics are unchanged; the single-store conformance suite passes without edits, and cross-replica freshness tests move to the 1 s bound for search.
New long-lived state (the committer queue, version records since the checkpoint, the slot LRU) is bounded by the checkpoint interval or an explicit cap, evicts, and ships with a memtest.Retained test on production-shaped data.

Phases

Spike on the benchmarks branch against real GCS from a GKE pod. Measure c (create p50 and p99), slot chain rate with batches of 16 and 32, pipelined, under 64 and 256 concurrent writers; 2 and 3 replicas racing on one world, with per-replica p99; 429 and 5xx behaviour near 1000 object writes per second; checkpoint write cost and cold start for synthetic worlds of 100,000 and 1,000,000 documents; replica apply rate. Current head CAS as baseline. Gate: at least 200 writes per second per world sustained for 10 minutes on a 1,000,000-document world, no lost writes, 429s absorbed by bounded retry, p99 latency within about 3c plus queueing, cold start within a bound set before the run. Record in /research/.
ADR (next free number). Replaces ADR 0008's commit point, per-commit root and shard rewrite, manifest layer, immediate search freshness and 100,000-document limit, and supersedes ADR 0031; keeps ADR 0008's content-addressed, verified object model for checkpoints and records the bucket-only, 1 s, throughput, fold, enterprise-scale, 1,000,000-document, frozen head, no-blocks and store-abstraction decisions.
Log commit and read path in bucketstore behind the schema 2 format: slots carrying entries and version records, snapshot as checkpoint plus applied slots, the frozen world marker, checkpoint zero for new and migrated worlds, cold start by listing, parallel blob staging. Tests: slot race, ambiguous create, rebase without conflict, same-path conflict, parent and child topology race, policy change during a race, sweep barrier, chain verification, path reads exact across replicas, search lag bounded at 1 s, replay from checkpoint zero equals the live snapshot, an old replica refusing the schema 2 marker.
Group commit, pipelining and several replicas, all inside bucketstore: the per-world committer in each store, chained member builds, per-member failure, speculative batches, rebase of a whole batch, queue bounds against the request deadline, the store ID in each slot and the yield-after-win rule. No writer election, forwarding or routing above the store. Tests: 2 and 3 stores sharing one bucket and committing concurrently lose no writes and preserve every conflict, a pipelining winner yields once another store is seen, a lone store never yields, and the conformance suite passes against each store.
Checkpoints and the manifest fold: the compactor inside the store on every replica, lag-triggered with a random delay, folded entries with history block references, history block materialization, incremental shard writes, 4096 shards, garbage collection reachability and old checkpoint collection. Tests: a checkpoint equals replay, two compactors racing on one sequence write identical bytes, a crashed or skipped checkpoint costs only replay, retention pruning across a 256-version block boundary, archive and unarchive, export parity, snapshot memory under memtest.Retained.
Change feed over slots: the hub backlog reads slots, slot retention at least the ring, removal of ADR 0031's blocks and sealing code. Tests: resume across a restart within the ring, resync past retention, resync for a pre-migration cursor.
Migration and the rollback decision; re-run the write scenarios on kind and on a scratch GKE world of 1,000,000 documents with 3 replicas: 64 concurrent writers sustained at 200 writes per second must lose none. Include an old replica committing during the migration's head replace.

Next wall, out of scope

Every replica holds the whole catalog in memory: about 1.36 KB of heap per document (ADR 0008 measured 133 MiB per 100,000), so about 1.3 GiB per replica for one world at the 1,000,000-document baseline. With that baseline, several large worlds in one process need a catalog that is not fully resident, such as immutable search segments in the bucket with a local cache. That is a separate plan, needed before more than one baseline-sized world shares a process; this one keeps write cost independent of world size, not memory.

Root's head shows generated graph and index documents republished every crawl cycle (/graph.md at v8718, /index/bruno:6309.md at v8491 while bruno is at sequence 3). fedcrawl writes time.Now() into bodies (Exported:, Indexed:), which defeats the unchanged-body skip in bucketstore/write.go. Root only takes graph writes, so this blocks no one, but it is wasted commits and version bloat. Check whether the incremental federation graph removes it before fixing it separately.

Open questions

Remaining enterprise targets: sustained and peak writes per world, worlds per process.
The p99 write latency gate in milliseconds and the cold start bound, once phase 0 measures c.
Checkpoint interval at the baseline.
Whether a hierarchical namespace bucket starts at a higher write rate, and whether to pre-warm buckets.
Whether the batch limit should grow past 16 when the queue is deep.
The yield jitter and how long another store's ID counts as recent, once phase 0 measures contention.
Slot retention beyond the hub's ring, and how many old checkpoints to keep for rollback.
One-way migration or a shipped rollback tool.
Callers of hash lookup: confirm none needs path-read exactness before it joins the 1 s class.
When an S3 backend lands; the design needs only create-if-absent, strong read and not-found, and a strongly consistent list, which S3 has.

Status

Proposed 2026-10-01, decisions recorded 2026-10-01 and 2026-10-02. Not started; picked up after the incremental federation graph work on feat/incremental-federation is done.

Bucket store commit log: create-only commits instead of one head CAS

Problem

ADR 0008 stated the assumption (read-heavy, under one mutation per second per world) and measured it, but it was never revisited when concurrent multi-user writing became the goal.

Decisions

Taken 2026-10-01 and 2026-10-02 with the maintainer.

Bucket only. GCS (and later S3) stays the only state. A database for metadata, such as Cloud SQL Postgres beside GCS, is out; ADR 0010 stands.
Ordering stays per world. WATCH cursors and the change feed are numbered per world (ADR 0025, ADR 0028), and the federation graph is derived from that sequence. The world is the narrowest scope that can own it.
Search may lag by up to one second. LOOKUP, LIST and hash lookup on one replica may trail another replica's commit by at most 1 s. FETCH, VERSIONS and chain verification by path stay exact. Every write evaluates its preconditions at the tip, so the lag never affects write correctness. This relaxes ADR 0008's immediate LOOKUP and hash freshness.
Group commit on a create-only log. Batches of mutations per slot, pipelined, with a throughput target of at least 200 writes per second per world.
Fold the manifest into the index entry. The entry carries its history block references; no per-document manifest object.
Enterprise scale. Write cost must scale with the change, not the world: a commit writes the new bytes and one log slot, never a structure whose size grows with the document count.
Baseline: 1,000,000 documents per world. Every gate, shard count, checkpoint interval and cold start bound is sized for it; ADR 0008's supported limit of 100,000 rises tenfold.
head.json is frozen (2026-10-02). It becomes a world marker written once, at world creation or migration, and never on the commit path; there is no tip hint to keep current.
No change blocks (2026-10-02). Slots are the change feed; ADR 0031's sealed blocks and their sealing code are removed.
Bounded by the store abstraction (2026-10-02). Write concurrency control and background maintenance live inside bucketstore, behind the backend contract (ADR 0015). Every replica commits; nothing above the store elects a writer, forwards writes between replicas, or schedules store maintenance, so a different backend, such as a database, drops in without changes above it.

Concurrency model stays the same

Alternatives considered

Group commit on head.json alone. Batching onto the head object still pays 1.5 s per batch, keeps a hot object whose 429s delay a whole batch, needs the 16-receipt window to scale with batch size, and skips ADR 0031's blocks of 8 sequences.
Log slots that point to a rewritten root and shards (this plan's first version). Fixes the head ceiling, but every commit still rewrites a shard whose size grows with the world: about 3.3 MB per write at 1,000,000 documents.
A two-level shard tree rewritten per commit. Root to 256 inner nodes to 65,536 leaves cuts bytes to tens of KB per write, but adds an object per write, keeps per-commit rewrites, and makes a cold start read 65,536 leaves.
A per-world committer elected above the store, with writes forwarded to it. Avoids slot races between replicas, but puts the bucket store's concurrency shape into server routing and needs pod-to-pod transport, and a database-backed store would need none of it. Rejected 2026-10-02 as a layering violation; contention is handled inside the store instead.
In-memory head shared between replicas, flushed every 2 s. Acknowledging before the flush loses up to 2 s of acknowledged writes when the holder crashes, and a deposed holder acknowledges writes its flush then loses. Making it safe means quorum replication (Raft), which makes GCS a snapshot store and does not survive all pods restarting together. Acknowledging after the flush is group commit on the head.
Database for metadata. Out by the bucket-only decision.
Per-document conditional writes alone. Correct for versions, but there is no world sequence for WATCH and the feed, two creates on different objects can break topology, and a commit followed by a separate feed entry can crash between the two and never be indexed.
Per-version create-only slots (docs/<path hash>/v/<N>.json, create-if-absent, so the expected version maps straight onto the create). Weighed 2026-10-01; it does not help.
- As a commit point beside the log it is a second commit point. A crash between the version create and the log slot leaves a version that path reads see but the feed and search never do, and a version name that blocks the document's next write.
- As the only commit point for updates, with the log sequenced afterwards, it needs orphan discovery (listing the bucket, or a notification service outside the bucket) and lets feed order drift from commit order.
- As a staged object it cannot be contended: a batch whose log slot loses would already hold the version name the winner needs. Anything staged before the log commit must be content-addressed.
- It adds an object per write.
- The log already gives same-document writers the same conflict answer: at build time inside a batch, and on rebase across pods.
One head per shard or subtree. A parent and child path hash to different shards, and the change sequence is per world.
Libraries (surveyed 2026-10-01). None fits: mbrt/glassdb is alpha, GCS only, has no change feed or global index; JayJamieson/objwal is single writer and orders through one rewritten manifest; SlateDB is Rust with Go bindings, single writer with fencing, a key-value model; rivian/delta-go is Parquet tables, alpha, S3 with DynamoDB locking, no GCS. Useful as design references: Delta Lake's log of actions plus periodic checkpoints and its logical conflict check on retry, SlateDB's put-if-absent commit and writer fencing.

Design

Log slots are the commit point

Group commit

One committer per world inside each replica's store replaces commitToken. Requests enqueue their mutation builders and wait for their outcome.
When the previous slot is confirmed, the committer takes whatever is queued, up to the batch limit (16 to start), with no timer. An idle world commits a lone write at once; under load batches fill on their own.
Members are built in queue order on a chained snapshot: member i sees the tip plus members before it. A member whose expected version, topology, policy or archive precondition fails gets its normal error and is dropped; the rest of the batch continues.
The batch's blobs are staged in parallel; they are content-addressed, so a lost race reuses them.
Every member of a batch is acknowledged only after its slot is confirmed.

Pipelining

Several replicas

Every replica's store commits for every world it serves; requests are served where they arrive. The slot race is the only arbitration, and it lives entirely in the store.
A lost slot costs one read of the winning slot and a retry on the next name. The loser's batch keeps its blobs and rebuilds only members the winning slot touched.
Fairness: a pipelining winner starts its next attempt one slot read ahead of the losers, so it could keep winning under sustained contention. A store that has seen another store's ID in a slot within the last few seconds waits a short jitter after each win before its next attempt. A store alone in a world never sees another ID and never waits.
Phase 0 measures per-replica latency with 2 and 3 replicas racing. If contention still costs too much, any further coordination stays inside bucketstore.

Lost slots and rebase

Ambiguous outcomes

Snapshot and read path

A replica's snapshot is the latest checkpoint it loaded plus every slot applied since. Slots carry full entries, so applying one needs no further reads.
Path reads (FETCH, VERSIONS, chain verification) check the next slot name (metadata only) before serving; new slots are read and applied. GCS and S3 both give strong read-after-write and strong not-found. FETCH of the current version is then the tip check plus the blob: two bucket calls instead of today's four.
Search reads (LOOKUP, LIST, hash lookup) serve the snapshot if it was confirmed current within the last second, and check the tip first otherwise. That bounds the lag at 1 s and costs at most one tip check per second per replica, however heavy the search traffic.
Versions. Version records since the latest checkpoint are held in memory from the slots; older versions come from the checkpoint's history blocks.
A replica installs its own commits as they succeed, so a writer reading through the replica it wrote to sees its writes at once.
Optionally, the hub's fan-out pushes new slot names to the other replicas so they usually catch up well inside the bound.
One request still pins one snapshot.

Checkpoints

The compactor is part of the store and runs in every replica; there is no leader and no Kubernetes Lease. A checkpoint at sequence S is fully determined by the log up to S and the format (canonical JSON, sorted entries, content-addressed objects), so two compactors that both checkpoint S write identical bytes and create-if-absent makes the race harmless.
To avoid duplicate work, a store compacts only when the newest checkpoint lags the tip by N slots or T seconds (starting points: 1024 slots or 60 s), waits a random delay first, and checks again before writing.
It writes the index in the ADR 0008 shape with the manifest folded in: the root, the shards, each entry carrying its history block references, and new history blocks for documents that gained versions. Then it creates _demarkus/v1/checkpoints/<sequence as 16 hex digits>.json naming the sequence it covers and its root.
Only shards and history blocks that changed since the previous checkpoint are written; unchanged ones are referenced by hash.
A checkpoint is derived, not committed. Readers verify it against the slot chain; a missing, late or failed checkpoint only means a longer replay. Correctness never depends on the compactor.
Shard count is sized for the baseline: 4096 shards hold about 250 entries each at 1,000,000 documents, about 200 KB.

World marker and cold start

head.json becomes a frozen world marker in the schema 2 format: world ID and format version, nothing that changes. It is created with create-if-absent for a new world and replaced exactly once for a migrated one, and never written again. It stays because an old replica that found no head would take the world for a new one and initialize a second genesis; a schema 2 marker makes it fail closed instead.
A new world starts with the marker and checkpoint zero, an empty root at sequence 1; the first slot is sequence 2.
Cold start lists checkpoints/ for the newest, loads it, then lists log/ from its sequence with a start offset (strongly consistent on GCS and S3) and applies the slots to the tip. Old checkpoints are collected, so the first list stays short.

Change feed

Slots are the durable change record, so the hub's backlog reads them directly, with an LRU of recent slots. The slot holding sequence q starts within one batch limit of names at or below q, so a resume lists from there.
Slots are kept at least as long as the hub's ring covers, which keeps ADR 0031's guarantee: a cursor resumes across a replica restart up to the ring. A cursor older than the retained slots gets resync.
ADR 0031's sealed blocks and their code (sealChanges, sealedThrough, the block LRU, changes/ objects) are removed. A slot already holds up to 16 changes, so a second batching layer adds nothing.

Garbage collection

Rates

Commit mode	c = 40 ms	c = 80 ms	Bytes per write	Write latency
Today, head CAS with 1.5 s pacing	0.67/s	0.67/s	body plus shard, history block, manifest, root	queues into seconds
Log with checkpoints, batch 16, pipelined	about 400/s	about 200/s	body plus about 1.2 KB	about 1c to 3c
Same, batch 32	about 800/s	about 400/s	body plus about 1.2 KB	about 1c to 3c

Bucket bound. About 1.1 objects per write. GCS starts a bucket at about 1000 object writes per second and ramps up gradually, so about 900 writes per second before scaling; the slot chain binds first.
Replica refresh. A replica reads only slots: about 20 KB per batch of 16, about 0.5 MB/s at 400 writes per second.
Compactor. At 400 writes per second over 60 s, about 24,000 changed entries touch nearly every one of 4096 shards: about 800 MB per checkpoint, about 13 MB/s, off the request path. A longer interval trades checkpoint bytes for replay length.
Cold start. About 850 MB of index in 4096 shard reads plus up to one interval of slots: a few seconds of transfer and decode, to be measured.
Queue bound. The 10 s request deadline caps the queue at about ten seconds of throughput before timeouts.
Several replicas. With R replicas writing one world, the slot rate stays about the same and each replica wins about 1/R of the slots, so a replica's write latency grows toward R slot times under sustained contention. Total throughput holds as long as each winning batch is full.

Migration

A new replica that finds a schema 1 head.json at sequence S creates checkpoints/<S> naming the head's root (deterministic bytes, so concurrent replicas converge through create-if-absent). Schema 2 readers accept unfolded shard entries with manifest references in a checkpoint.
It then CAS-replaces head.json with the frozen schema 2 marker, conditioned on the generation it read. If an old replica committed in between, the replace fails; the new replica rereads the head, checkpoints the new sequence and tries again. A checkpoint left behind at an older sequence is valid state and is simply not the newest.
The first write after the marker creates slot S+1. The compactor's first checkpoint writes the folded format and the larger shard count.
No change blocks are sealed. Cursors at or after S resume; cursors older than S get one resync.
Old replicas refuse a schema 2 head (unknown fields are refused) and fail closed with server-error, so they cannot commit during a rolling deploy. No split brain.
Rollback needs a tool that writes a schema 1 head from a fresh checkpoint at the log tip in the old format; decide whether to ship it or document the migration as one way.

Invariants

A slot is created once and never rewritten. The next slot is only attempted after the current tip was observed, so the log has no gaps; each entry names its predecessor's hash, and sequences run contiguously across slots.
head.json is written once per world, at creation or migration, and never on the commit path.
The log alone determines world state: replaying slots from any valid checkpoint gives the same snapshot, and a world with no checkpoint after checkpoint zero is still correct.
A checkpoint is a deterministic function of the log up to its sequence; any number of compactors may race on it.
A batch's mutations are applied in queue order; each member's preconditions are evaluated against the tip plus the members before it, and one member's failure never fails another.
Each acknowledged mutation is in exactly one slot; acknowledgement follows only a successful create or a reread that finds its operation ID in the slot.
A speculative batch never commits ahead of its predecessor slot.
Everything written before the slot commit is content-addressed, so a losing batch never holds a name a winner needs.
A commit's write cost depends on the change, never on the world's document count.
Write concurrency control and background maintenance live behind the backend contract; nothing above the store elects writers, routes writes by replica, or schedules store work.
Slots are retained at least as long as the hub's ring covers; a resume within that window never resyncs.
Every write's preconditions are evaluated at the tip; path reads are exact; search reads serve a snapshot confirmed current within 1 s.
One request reads one snapshot.
Expected-version, topology, policy and archive semantics are unchanged; the single-store conformance suite passes without edits, and cross-replica freshness tests move to the 1 s bound for search.
New long-lived state (the committer queue, version records since the checkpoint, the slot LRU) is bounded by the checkpoint interval or an explicit cap, evicts, and ships with a memtest.Retained test on production-shaped data.

Phases

Spike on the benchmarks branch against real GCS from a GKE pod. Measure c (create p50 and p99), slot chain rate with batches of 16 and 32, pipelined, under 64 and 256 concurrent writers; 2 and 3 replicas racing on one world, with per-replica p99; 429 and 5xx behaviour near 1000 object writes per second; checkpoint write cost and cold start for synthetic worlds of 100,000 and 1,000,000 documents; replica apply rate. Current head CAS as baseline. Gate: at least 200 writes per second per world sustained for 10 minutes on a 1,000,000-document world, no lost writes, 429s absorbed by bounded retry, p99 latency within about 3c plus queueing, cold start within a bound set before the run. Record in /research/.
ADR (next free number). Replaces ADR 0008's commit point, per-commit root and shard rewrite, manifest layer, immediate search freshness and 100,000-document limit, and supersedes ADR 0031; keeps ADR 0008's content-addressed, verified object model for checkpoints and records the bucket-only, 1 s, throughput, fold, enterprise-scale, 1,000,000-document, frozen head, no-blocks and store-abstraction decisions.
Log commit and read path in bucketstore behind the schema 2 format: slots carrying entries and version records, snapshot as checkpoint plus applied slots, the frozen world marker, checkpoint zero for new and migrated worlds, cold start by listing, parallel blob staging. Tests: slot race, ambiguous create, rebase without conflict, same-path conflict, parent and child topology race, policy change during a race, sweep barrier, chain verification, path reads exact across replicas, search lag bounded at 1 s, replay from checkpoint zero equals the live snapshot, an old replica refusing the schema 2 marker.
Group commit, pipelining and several replicas, all inside bucketstore: the per-world committer in each store, chained member builds, per-member failure, speculative batches, rebase of a whole batch, queue bounds against the request deadline, the store ID in each slot and the yield-after-win rule. No writer election, forwarding or routing above the store. Tests: 2 and 3 stores sharing one bucket and committing concurrently lose no writes and preserve every conflict, a pipelining winner yields once another store is seen, a lone store never yields, and the conformance suite passes against each store.
Checkpoints and the manifest fold: the compactor inside the store on every replica, lag-triggered with a random delay, folded entries with history block references, history block materialization, incremental shard writes, 4096 shards, garbage collection reachability and old checkpoint collection. Tests: a checkpoint equals replay, two compactors racing on one sequence write identical bytes, a crashed or skipped checkpoint costs only replay, retention pruning across a 256-version block boundary, archive and unarchive, export parity, snapshot memory under memtest.Retained.
Change feed over slots: the hub backlog reads slots, slot retention at least the ring, removal of ADR 0031's blocks and sealing code. Tests: resume across a restart within the ring, resync past retention, resync for a pre-migration cursor.
Migration and the rollback decision; re-run the write scenarios on kind and on a scratch GKE world of 1,000,000 documents with 3 replicas: 64 concurrent writers sustained at 200 writes per second must lose none. Include an old replica committing during the migration's head replace.

Next wall, out of scope

Open questions

Remaining enterprise targets: sustained and peak writes per world, worlds per process.
The p99 write latency gate in milliseconds and the cold start bound, once phase 0 measures c.
Checkpoint interval at the baseline.
Whether a hierarchical namespace bucket starts at a higher write rate, and whether to pre-warm buckets.
Whether the batch limit should grow past 16 when the queue is deep.
The yield jitter and how long another store's ID counts as recent, once phase 0 measures contention.
Slot retention beyond the hub's ring, and how many old checkpoints to keep for rollback.
One-way migration or a shipped rollback tool.
Callers of hash lookup: confirm none needs path-read exactness before it joins the 1 s class.
When an S3 backend lands; the design needs only create-if-absent, strong read and not-found, and a strongly consistent list, which S3 has.

Status

Proposed 2026-10-01, decisions recorded 2026-10-01 and 2026-10-02. Not started; picked up after the incremental federation graph work on feat/incremental-federation is done.

Bucket store commit log: create-only commits instead of one head CAS

Problem

Decisions

Concurrency model stays the same

Alternatives considered

Design

Log slots are the commit point

Group commit

Pipelining

Several replicas

Lost slots and rebase

Ambiguous outcomes

Snapshot and read path

Checkpoints

World marker and cold start

Change feed

Garbage collection

Rates

Migration

Invariants

Phases

Next wall, out of scope

Related, separate

Open questions

Status