# Bucket store commit log: create-only commits instead of one head CAS A plan to take one world from about 0.67 to about 200 to 300 writes per second by committing batches of mutations as new create-only log objects instead of conditional overwrites of `head.json`, with search reads allowed to lag by up to one second. Audience: demarkus maintainers. Proposed 2026-10-01, deferred until the [incremental federation graph](/plans/agent-in-process.md) ships. Builds on [ADR 0008](/adr/0008-gcs-root-cas-storage.md) and replaces its commit point, its immediate search freshness and its manifest layer; would supersede the sealing in [ADR 0031](/adr/0031-bucket-store-seals-change-blocks.md). Part of the [Knowledge server](/index/knowledge-server.md) track. ## Problem Every mutation in a world ends with one conditional replace of `_demarkus/v1/head.json`, the world's only mutable object. GCS limits updates to one object name to about one per second, so `bucketstore` serializes commits behind `commitToken` and paces head attempts at `defaultCommitInterval` (1.5 s). The ceiling is about 0.67 commits per second per world, whatever the paths. Measured on kind ([writes](/research/kind-bench-merged-binary.md#writes)): one sequential writer took exactly 1.51 s per publish; 4 concurrent writers queued to 6 s each; a 4-writer burst of 200 publishes lost 49 to the 10 s request timeout. The product target is many users writing one world concurrently, so this is a blocker, not a tuning issue. ADR 0008 stated the assumption (read-heavy, under one mutation per second per world) and measured it, but it was never revisited when concurrent multi-user writing became the goal. The constraint is the single mutable object name, not conditional writes. Any design where every commit mutates one object (CAS, lock, lease, pointer) inherits the same cap. Creating a new object name with create-if-absent has no per-name cap. Without the world-level ordering, each document would be its own object and capped at about one write per second on its own, which is harmless; the shared head is what turns that per-object limit into a per-world one. ## Decisions Taken 2026-10-01 with the maintainer. - **Bucket only.** GCS (and later S3) stays the only state. A database for metadata, such as Cloud SQL Postgres beside GCS, is out; [ADR 0010](/adr/0010-remove-postgres-backend.md) stands. - **Ordering stays per world.** WATCH cursors and the change feed are numbered per world ([ADR 0025](/adr/0025-watch-verb.md), [ADR 0028](/adr/0028-stores-own-a-durable-change-sequence.md)), and the federation graph is derived from that sequence. The world is the narrowest scope that can own it. - **Search may lag by up to one second.** LOOKUP, LIST and hash lookup on one replica may trail another replica's commit by at most 1 s. FETCH, VERSIONS and chain verification by path stay exact. Every write evaluates its preconditions at the tip, so the lag never affects write correctness. This relaxes ADR 0008's immediate LOOKUP and hash freshness. - **Target: log plus group commit of 16 mutations per slot.** About 200 to 300 writes per second per world, limited by the bucket's write rate, with write latency of about one to three object-create round trips. - **Fold the manifest into the shard entry.** Cuts a write from about 4 objects to about 3, lifting the bucket-bound ceiling to about 320 writes per second (phase 4). ## Concurrency model stays the same Expected-version semantics already hold today: `bucketstore` checks `ExpectedVersion` per document and a writer that loses the head race revalidates against the new snapshot, so writers to different documents never see a conflict and writers to one document get the normal Mark conflict. The world-level ordering exists for what one document cannot carry: the change sequence, path topology (`/a.md` against `/a.md/b.md`), and one snapshot per request. The file store has the same single ordering point (`filestore.go` `mu`), it just costs microseconds on local disk. This plan keeps the model and makes the ordering step cheap and batched. ## Alternatives considered - **Group commit on `head.json` alone.** Batching onto the head object still pays 1.5 s per batch, keeps a hot object whose 429s delay a whole batch, needs the 16-receipt window to scale with batch size, and skips ADR 0031's blocks of 8 sequences. Group commit is part of this plan, on top of the log instead. - **In-memory head shared between replicas, flushed every 2 s.** Acknowledging before the flush loses up to 2 s of acknowledged writes when the holder crashes, and a deposed holder acknowledges writes its flush then loses. Making it safe means quorum replication (Raft), which makes GCS a snapshot store and does not survive all pods restarting together. Acknowledging after the flush is group commit on the head. - **Database for metadata.** Out by the bucket-only decision. - **Per-document conditional writes alone.** Correct for versions, but there is no world sequence for WATCH and the feed, two creates on different objects can break topology, and a commit followed by a separate feed entry can crash between the two and never be indexed. - **Per-version create-only slots** (`docs//v/.json`, create-if-absent, so the expected version maps straight onto the create). Weighed 2026-10-01; it does not help. - As a commit point beside the log it is a second commit point. A crash between the version create and the log slot leaves a version that path reads see but the feed and search never do, and a version name that blocks the document's next write. - As the only commit point for updates, with the log sequenced afterwards, it needs orphan discovery (listing the bucket, or a notification service outside the bucket) and lets feed order drift from commit order. - As a staged object it cannot be contended: a batch whose log slot loses would already hold the version name the winner needs. Anything staged before the log commit must be content-addressed, which history blocks and manifests already are. - It adds an object per write, about 4 to 5, which lowers the bucket-bound ceiling from about 245 to about 195 writes per second, under the target. - The log already gives same-document writers the same conflict answer: at build time inside a batch, and on rebase across pods. - **One head per shard or subtree.** A parent and child path hash to different shards, and the change sequence is per world. - **Libraries (surveyed 2026-10-01).** None fits: `mbrt/glassdb` is alpha, GCS only, has no change feed or global index; `JayJamieson/objwal` is single writer and orders through one rewritten manifest; SlateDB is Rust with Go bindings, single writer with fencing, a key-value model; `rivian/delta-go` is Parquet tables, alpha, S3 with DynamoDB locking, no GCS. Useful as design references only: Delta Lake's put-if-absent log and logical conflict check on retry, SlateDB's put-if-absent commit and writer fencing. ## Design ### Log slots are the commit point A commit creates `_demarkus/v1/log/.json` with `blob.Store.Create` (create-if-absent; `ifGenerationMatch=0` on GCS, `If-None-Match: *` on S3). One slot holds one batch of 1 to 16 mutations. The entry carries schema, world ID, first sequence, root reference, one receipt per mutation (sequences contiguous from the first), the sweep epoch, and the hash of the previous slot's entry so readers verify the chain. The next slot's name is always the tip entry's last sequence plus one, so every writer racing from the same tip targets the same name. The successful create is the commit; nothing else is mutable on the commit path. ### Group commit - One committer per world replaces `commitToken`. Requests enqueue their mutation builders and wait for their outcome. - When the previous slot is confirmed, the committer takes whatever is queued, up to 16, with no timer. An idle world commits a lone write at once; under load batches fill on their own. - Members are built in queue order on a chained snapshot: member i sees the tip plus members before it. A member whose expected version, topology, policy or archive precondition fails gets its normal error and is dropped; the rest of the batch continues. - Staging writes each member's blob, history block and manifest (no manifest after phase 4), plus the batch's changed shards and one root, all in parallel. Today's loop in `commit.go` stages one object at a time. - Every member of a batch is acknowledged only after its slot is confirmed. ### Pipelining While slot k's create is in flight, the committer builds and stages the next batch on slot k's candidate. Slot k+1 is created only after slot k is confirmed. If slot k is lost, the speculative batch's staged objects are left for garbage collection and the batch is rebuilt at the new tip. Per-slot time drops from about two create round trips to about one. ### Lost slots and rebase If the slot exists, load the slots up to the new tip. A member whose path, ancestors or descendants, or the world's publish policy changed in those slots is rebuilt so its checks answer as today, which may now return a conflict. The others keep their per-document objects; only the changed shards and the root are rebuilt over the new tip, and the batch tries the next slot name. Attempts are bounded by the request deadline; there is no fixed pacing on the commit path. ### Ambiguous outcomes On a timeout or lost response, read the slot. Absent: retry the identical create. Present with this batch's operation IDs: committed. Present with other IDs: lost, rebase. A slot is never rewritten, so the outcome is always decidable; the evicted-receipt `server-error` path of ADR 0008 disappears. ### Read path - **Path reads** (FETCH, VERSIONS, chain verification) check the next slot name (metadata only) before serving, as today's head validation does. Not found means the snapshot is current; GCS and S3 both give strong read-after-write and strong not-found. Found means reading forward until not found, then loading the changed shards. - **Search reads** (LOOKUP, LIST, hash lookup) serve the replica's snapshot if it was confirmed current within the last second, and check the tip first otherwise. That bounds the lag at 1 s and costs at most one tip check per second per replica, however heavy the search traffic. - A replica installs its own commits as they succeed, so a writer reading through the replica it wrote to sees its writes at once. - Optionally, the hub's fan-out pushes new slot names to the other replicas so they usually catch up well inside the bound. - One request still pins one snapshot. ### Tip hint `head.json` stays as a non-authoritative hint (last slot name and root), written best effort off the commit path at most once per pacing interval. Slot names are not contiguous, so cold start lists `log/` from the hint with a start offset (strongly consistent on GCS and S3) and follows the chain to the tip. A missing or stale hint costs a longer list, never correctness. ### Change feed Log entries are the durable change record, so the hub's backlog reads slots directly, with an LRU of recent entries. The slot holding sequence q starts within the 16 names at or below q, so a resume lists from q minus 15. ADR 0031's sealed blocks become a compaction format: ranges of old slots are sealed into blocks so old slots can be collected while the feed stays continuous. ### Garbage collection barrier The collector's sweep epoch becomes a log entry like any other commit. A batch whose slot create loses to a barrier entry sees the active epoch, waits for the clearing entry, and re-stages, which preserves ADR 0008's fencing. ### Manifest folded into the shard entry Today a document's manifest holds its current version, archive state and history block references, and the shard entry already repeats the first two. After the fold the shard entry carries the history block references itself and the per-document manifest object goes away. - A write creates blob and history block (plus the batch's shards, root and slot): about 3 objects per mutation instead of about 4. - FETCH reads head check, history block and blob: three bucket calls instead of four, about 20 ms less on real GCS. - Archive and unarchive write only a shard and a root. - Entries stay about the same size for documents under 256 versions, since one history reference replaces one manifest reference; each further 256 retained versions adds one reference. - Every walker of the object graph (reads, validation, export, backup pins, garbage collection marking) follows shard to history directly. ### Rates Projected 2026-10-01, not measured. c is one small-object create on GCS from a GKE pod. Real GCS reads in the cluster take about 20 ms per call ([probe](/research/gke-knowledge-mcp-probe.md)); writes have never been timed on real GCS, so c is assumed to be 40 to 80 ms. | Commit mode | c = 40 ms | c = 80 ms | Write latency | |---|---|---|---| | Today, head CAS with 1.5 s pacing | 0.67/s | 0.67/s | queues into seconds | | Log, stage then slot | about 12/s | about 6/s | about 2c idle | | Log, parallel staging and pipelining | about 25/s | about 12/s | about 2c idle | | Log, group commit of 16, pipelined (target) | about 250 to 300/s, bucket bound | about 200/s | about 1c to 3c | | Target plus manifest fold | about 320/s, bucket bound | about 200/s | about 1c to 3c | - **Bucket bound.** A batch of 16 creates 48 per-document objects (blob, history block, manifest), up to 16 shards, one root and one slot: about 4 objects per mutation, about 3 after the fold. GCS starts a bucket at about 1000 object writes per second and ramps up gradually, so the target sits at the bucket's starting capacity until it scales. - **Shard bytes.** Every mutation rewrites its whole shard. At 100,000 documents a shard is about 330 KB, so 250 writes per second upload about 80 MB/s from the writer and every replica decodes about as much. Small worlds are unaffected; large worlds may hit bandwidth and CPU before the bucket bound (open question below). - **Queue bound.** The 10 s request deadline caps the queue at about ten seconds of throughput before timeouts. - **Writer pods.** Today all writes land on one pod. With several writer pods every lost slot costs one batch round trip, so throughput drops below the table unless one pod commits per world. - **Name hotspots.** Slot names are sequential but run at only about 15 per second; blob, history and manifest names are hashes and spread evenly. ## Migration - A new replica that finds a schema 1 `head.json` and no log first seals every complete ADR 0031 block the head covers (idempotent, deterministic bytes), then creates the first slot, named after the first sequence not covered by a complete block and holding the head's remaining receipts and root. Both steps derive from the head's bytes alone, so concurrent replicas converge through create-if-absent. Earlier sequences stay readable from the sealed blocks. - It then CAS-replaces `head.json` with a schema 2 hint. - Shards are not rewritten at migration. Schema 2 readers accept a shard entry with either a manifest reference or folded history references; every write produces the folded form, so entries convert as their documents are next written. - Heads are decoded with unknown fields refused, so an old replica reading a schema 2 head fails closed with `server-error` and cannot commit during a rolling deploy. No split brain. - Rollback needs a tool that writes a schema 1 head from the log tip and unfolds converted entries; decide whether to ship it or document the migration as one way. ## Invariants - A slot is created once and never rewritten. The next slot is only attempted after the current tip was observed, so the log has no gaps; each entry names its predecessor's hash, and sequences run contiguously across slots. - A slot holds 1 to 16 mutations, applied in queue order; each member's preconditions are evaluated against the tip plus the members before it, and one member's failure never fails another. - Each acknowledged mutation is in exactly one slot; acknowledgement follows only a successful create or a reread that finds its operation ID in the slot. - A speculative batch never commits ahead of its predecessor slot. - Everything staged before the log commit is content-addressed, so a losing batch never holds a name a winner needs. - Every write's preconditions are evaluated at the tip; path reads are exact; search reads serve a snapshot confirmed current within 1 s. - One request reads one snapshot. - Expected-version, topology, policy and archive semantics are unchanged; the single-store conformance suite passes without edits, and cross-replica freshness tests move to the 1 s bound for search. - New long-lived state (the committer queue, the slot LRU, any rebase cache) is bounded, evicts, and ships with a `memtest.Retained` test on production-shaped data. ## Phases 0. **Spike on the `benchmarks` branch against real GCS from a GKE pod.** Measure c (create p50 and p99), parallel staging, serial and pipelined slot rates, group commit at 16 per slot under 64 and 256 concurrent writers with 4 and with 3 objects per mutation, two pods racing, and 429 and 5xx behaviour as a bucket approaches 1000 object writes per second; current head CAS as baseline. Gate: at least 200 writes per second per world sustained for 10 minutes, no lost writes, 429s absorbed by bounded retry, p99 latency within about 3c plus queueing. Record in `/research/`. 1. **ADR (next free number).** Replaces ADR 0008's commit point, its immediate search freshness and its manifest layer, and ADR 0031's sealing; keeps ADR 0008's immutable root, shard, history and blob model and records the bucket-only, 1 s, throughput and fold decisions. 2. **Log commit and read path** in `bucketstore` behind the schema 2 format, with parallel staging: slot race, ambiguous create, rebase without conflict, same-path conflict, parent and child topology race, policy change during a race, sweep barrier, chain verification, path reads exact across replicas, search lag bounded at 1 s. 3. **Group commit and pipelining:** the per-world committer, chained member builds, per-member failure, speculative staging, rebase of a whole batch, queue bounds against the request deadline. 4. **Fold the manifest into the shard entry,** inside the schema 2 format so worlds migrate once. Shard entries carry history block references; writes stop creating manifests; readers accept both entry forms; every graph walker (reads, validation, export, backup pins, garbage collection marking) follows shard to history. Tests: retention pruning across a 256-version block boundary, archive and unarchive with no new history, chain verification and export parity on both entry forms, a mixed shard during migration, snapshot memory under `memtest.Retained` on production-shaped data. 5. **Change feed over slots**, compaction into blocks, retention for old slots. 6. **Migration** and the rollback decision; re-run the write scenarios on kind and on a scratch GKE world: 64 concurrent writers sustained at 200 writes per second must lose none. ## Related, separate Root's head shows generated graph and index documents republished every crawl cycle (`/graph.md` at v8718, `/index/bruno:6309.md` at v8491 while bruno is at sequence 3). `fedcrawl` writes `time.Now()` into bodies (`Exported:`, `Indexed:`), which defeats the unchanged-body skip in `bucketstore/write.go`. Root only takes graph writes, so this blocks no one, but it is wasted commits and version bloat. Check whether the incremental federation graph removes it before fixing it separately. ## Open questions - The p99 write latency gate in milliseconds, once phase 0 measures c. - Whether a hierarchical namespace bucket starts at a higher write rate, and whether to pre-warm buckets for the target. - Whether the batch size should grow past 16 when the queue is deep. - Shard bytes at scale: whether large worlds need a two-level shard tree (root to 256 inner nodes to leaves) so a write rewrites a few small objects instead of one 330 KB shard. - Retention for log slots and the compaction cadence. - One-way migration or a shipped rollback tool. - Callers of hash lookup: confirm none needs path-read exactness before it joins the 1 s class. - When an S3 backend lands; the design needs only create-if-absent, strong read and not-found, and a strongly consistent list, which S3 has. ## Status Proposed 2026-10-01, decisions recorded the same day. Not started; picked up after the incremental federation graph work on `feat/incremental-federation` is done.