Bucket store commit log: create-only commits instead of one head CAS

A plan to take one world from about 0.67 writes per second to hundreds, at enterprise scale with 1,000,000 documents per world as the baseline, by committing batches of mutations as create-only log slots that carry their own index changes, and materializing the index in background checkpoints instead of rewriting it on every commit. Search reads may lag by up to one second, and the body-search section index is built lazily, off the write path. Everything stays behind the backend contract: nothing above the store changes how it routes, elects or schedules. Audience: demarkus maintainers. Proposed 2026-10-01; phase 0 measured it on real GCS on 2026-10-02 (spike); the decision is recorded in ADR 0036 (proposed). Builds on ADR 0008 and replaces its commit point, its per-commit root and shard rewrite, its manifest layer and its immediate search freshness; supersedes ADR 0031. Part of the Knowledge server track.

Problem

Every mutation in a world ends with one conditional replace of _demarkus/v1/head.json, the world's only mutable object. GCS limits updates to one object name to about one per second, so bucketstore serializes commits behind commitToken and paces head attempts at defaultCommitInterval (1.5 s). The ceiling is about 0.67 commits per second per world, whatever the paths.

Measured on kind (writes): one sequential writer took exactly 1.51 s per publish; 4 concurrent writers queued to 6 s each; a 4-writer burst of 200 publishes lost 49 to the 10 s request timeout. On real GCS (spike) the paced head gave exactly 0.67 commits/s, and three unpaced writers on one head got 1.2 commits/s with 552 of 692 attempts throttled. The product target is many users writing one world concurrently, so this is a blocker, not a tuning issue.

ADR 0008 stated the assumption (read-heavy, under one mutation per second per world) and measured it, but it was never revisited when concurrent multi-user writing became the goal.

The constraint is the single mutable object name, not conditional writes. Any design where every commit mutates one object (CAS, lock, lease, pointer) inherits the same cap. Creating a new object name with create-if-absent has no per-name cap. Without the world-level ordering, each document would be its own object and capped at about one write per second on its own, which is harmless; the shared head is what turns that per-object limit into a per-world one.

The second limit appears at enterprise scale. Every commit today rewrites the document's history block and manifest, one of 256 shards and the root. A shard holds a world's documents divided by 256 at about 850 bytes each: about 330 KB at 100,000 documents and about 3.3 MB at 1,000,000. At 250 writes per second that is about 80 MB/s, or about 830 MB/s, uploaded by the writer and decoded again by every replica. Removing the head ceiling alone moves the wall to bytes.

Decisions

Taken 2026-10-01 and 2026-10-02 with the maintainer.

• Bucket only. GCS (and later S3) stays the only state. A database for metadata, such as Cloud SQL Postgres beside GCS, is out; ADR 0010 stands.
• Ordering stays per world. WATCH cursors and the change feed are numbered per world (ADR 0025, ADR 0028), and the federation graph is derived from that sequence. The world is the narrowest scope that can own it.
• Search may lag by up to one second. LOOKUP, LIST and hash lookup on one replica may trail another replica's commit by at most 1 s. FETCH, VERSIONS and chain verification by path stay exact. Every write evaluates its preconditions at the tip, so the lag never affects write correctness. This relaxes ADR 0008's immediate LOOKUP and hash freshness.
• Group commit on a create-only log. Batches of mutations per slot, pipelined, with a throughput target of at least 200 writes per second per world.
• Fold the manifest into the index entry. The entry carries its history block references; no per-document manifest object.
• Enterprise scale. Write cost must scale with the change, not the world: a commit writes the new bytes and one log slot, never a structure whose size grows with the document count.
• Baseline: 1,000,000 documents per world. Every gate, shard count, checkpoint interval and cold start bound is sized for it; ADR 0008's supported limit of 100,000 rises tenfold.
• head.json is frozen (2026-10-02). It becomes a world marker written once, at world creation or migration, and never on the commit path; there is no tip hint to keep current.
• No change blocks (2026-10-02). Slots are the change feed; ADR 0031's sealed blocks and their sealing code are removed.
• Bounded by the store abstraction (2026-10-02). Write concurrency control and background maintenance live inside bucketstore, behind the backend contract (ADR 0015). Every replica commits; nothing above the store elects a writer, forwards writes between replicas, or schedules store maintenance, so a different backend, such as a database, drops in without changes above it.
• Swappable store, with today's leaks fixed first (2026-10-02). The server sees a world's store only through the backend contract and constructs it at one wiring point, so swapping bucketstore for filestore, an S3 store or a database changes one constructor and nothing else. No bucket concept (head, slot, log, checkpoint, poll, genesis) crosses the contract. The knowledge server already reaches past it, so a refactor that changes no behavior removes those leaks before any log code lands; see Store boundary.
• Lazy section index (2026-10-02). The body-search index (ADR 0012) is never built on the commit path or when a slot is applied. A world builds it only once body search is used there, keeps it current in the background, and drops it when body search goes idle.

Concurrency model stays the same

Expected-version semantics already hold today: bucketstore checks ExpectedVersion per document and a writer that loses the head race revalidates against the new snapshot, so writers to different documents never see a conflict and writers to one document get the normal Mark conflict. The world-level ordering exists for what one document cannot carry: the change sequence, path topology (/a.md against /a.md/b.md), and one snapshot per request. The file store has the same single ordering point (filestore.go mu), it just costs microseconds on local disk. This plan keeps the model and makes the ordering step cheap, batched and independent of world size.

Alternatives considered

• Group commit on head.json alone. Batching onto the head object still pays 1.5 s per batch, keeps a hot object whose 429s delay a whole batch, needs the 16-receipt window to scale with batch size, and skips ADR 0031's blocks of 8 sequences.
• Log slots that point to a rewritten root and shards (this plan's first version). Fixes the head ceiling, but every commit still rewrites a shard whose size grows with the world: about 3.3 MB per write at 1,000,000 documents.
• A two-level shard tree rewritten per commit. Root to 256 inner nodes to 65,536 leaves cuts bytes to tens of KB per write, but adds an object per write, keeps per-commit rewrites, and makes a cold start read 65,536 leaves.
• A per-world committer elected above the store, with writes forwarded to it. Avoids slot races between replicas, but puts the bucket store's concurrency shape into server routing and needs pod-to-pod transport, and a database-backed store would need none of it. Rejected 2026-10-02 as a layering violation; contention is handled inside the store instead.
• In-memory head shared between replicas, flushed every 2 s. Acknowledging before the flush loses up to 2 s of acknowledged writes when the holder crashes, and a deposed holder acknowledges writes its flush then loses. Making it safe means quorum replication (Raft), which makes GCS a snapshot store and does not survive all pods restarting together. Acknowledging after the flush is group commit on the head.
• Database for metadata. Out by the bucket-only decision.
• Per-document conditional writes alone. Correct for versions, but there is no world sequence for WATCH and the feed, two creates on different objects can break topology, and a commit followed by a separate feed entry can crash between the two and never be indexed.
• Per-version create-only slots (docs/<path hash>/v/<N>.json, create-if-absent, so the expected version maps straight onto the create). Weighed 2026-10-01; it does not help.
  • As a commit point beside the log it is a second commit point. A crash between the version create and the log slot leaves a version that path reads see but the feed and search never do, and a version name that blocks the document's next write.
  • As the only commit point for updates, with the log sequenced afterwards, it needs orphan discovery (listing the bucket, or a notification service outside the bucket) and lets feed order drift from commit order.
  • As a staged object it cannot be contended: a batch whose log slot loses would already hold the version name the winner needs. Anything staged before the log commit must be content-addressed.
  • It adds an object per write.
  • The log already gives same-document writers the same conflict answer: at build time inside a batch, and on rebase across pods.
• Section entries carried in slots. The writer has the body, so it could index sections once and ship them in the slot. Rejected 2026-10-02: a section entry is about as large as the body, so this is inlining bodies under another name, and every replica would hold sections for every world whether or not anyone searches bodies there.
• One head per shard or subtree. A parent and child path hash to different shards, and the change sequence is per world.
• Libraries (surveyed 2026-10-01). None fits: mbrt/glassdb is alpha, GCS only, has no change feed or global index; JayJamieson/objwal is single writer and orders through one rewritten manifest; SlateDB is Rust with Go bindings, single writer with fencing, a key-value model; rivian/delta-go is Parquet tables, alpha, S3 with DynamoDB locking, no GCS. Useful as design references: Delta Lake's log of actions plus periodic checkpoints and its logical conflict check on retry, SlateDB's put-if-absent commit and writer fencing.

Design

The shape is a log-structured store: the log is the source of truth and carries every change in full, and checkpoints are derived, verifiable materializations that only make reads and cold starts cheaper.

Store boundary

The server sees a world's store only through the backend contract in server/internal/backend: Store for writes and for reads through a view, ChangeSource for the change hub, and Follower, a small replica follow contract this plan adds (ADR 0036): Follow(sequence) says a peer replica committed through that sequence, and the store decides whether to read. Everything else is private to bucketstore: the log, slots, checkpoints, the world marker, the committer, the compactor, the collector, when to read the bucket, and how a new world is created.

Today the knowledge server reaches past the contract in server/knowledgeserver/worlds.go:

• worldEntry holds the concrete *bucketstore.Store instead of a contract type.
• Hint compares a peer's sequence with HeadSequence(), a head concept.
• pollChanges, its 5 s changePollInterval and the kick channel decide when the store calls Poll, so the store's refresh schedule lives above it.
• ensureGenesis calls bucketstore.EnsureWorld, creating the world outside the store's open.
• The world manager calls bucketstore.Open inline, with its options and the Committed hook.

The fix opens phase 2, on today's head CAS store and before any log code; it is committed on its own and changes no behavior:

• The world manager holds contract types only. Peer hints carry a world ID and a sequence, never a slot name.
• The poll loop and its timer move into bucketstore, which decides when to read the bucket. The store gates its watcher poll on its own change hub, which gains a subscriber count, instead of on the runtime's watch count.
• Genesis folds into the store's open, honoring read-only worlds, behind one store factory in the world manager's config, so only the server's wiring imports bucketstore.
• A world manager test opens and serves a world on a store that is not a bucket store, through that factory.
• Other packages above the store (handlers, worldruntime, writepolicy, changefeed, peerhint, knowledgetest, the knowledge broker) are swept for the same kind of leak, and any found are fixed in the same refactor.

Changes beneath the contract are allowed: the start offset on blob.Store.List belongs to the blob contract the store depends on, and it carries no GCS types. The 1 s search freshness is a property of the backend contract, documented there and in the SPEC and tested in the conformance suite, so no handler special-cases a backend.

Log slots are the commit point

A commit creates _demarkus/v1/log/<first sequence as 16 hex digits>.json with blob.Store.Create (create-if-absent; ifGenerationMatch=0 on GCS, If-None-Match: * on S3). One slot holds one batch of mutations. For each mutation the slot carries the receipt, the new index entry (path, current version, archive state, body hash, modified time, catalog entry), the new version record (version, blob reference, body hash, modified time), and any retention prune range. The slot also carries schema, world ID, first sequence, the writing replica's store ID, the sweep epoch, and the hash of the previous slot's entry so readers verify the chain. The next slot's name is the tip entry's last sequence plus one, so every writer racing from the same tip targets the same name. The successful create is the commit.

A commit writes the new document blobs and one slot, nothing else: about 1.1 objects and the body plus about 1.2 KB per mutation, whatever the world's size. No root, shard, manifest, history block, change block or head is written on the commit path.

Group commit

• One committer per world inside each replica's store replaces commitToken. Requests enqueue their mutation builders and wait for their outcome.
• When there is room in the pipeline, the committer takes whatever is queued, up to the batch limit, with no timer. An idle world commits a lone write at once; under load batches fill on their own. The limit starts at 16 and may grow to 32 while the queue stays deep (batch 32 measured 470 to 510 writes/s).
• Members are built in queue order on a chained snapshot: member i sees the tip plus members before it. A member whose expected version, topology, policy or archive precondition fails gets its normal error and is dropped; the rest of the batch continues.
• The batch's blobs are staged in parallel; they are content-addressed, so a lost race reuses them.
• Every member of a batch is acknowledged only after its slot is confirmed.

Pipelining

While slot k's create is in flight, the committer builds and stages up to two further batches on slot k's candidate. Slot k+1 is created only after slot k is confirmed. If slot k is lost, the speculative batches are rebuilt at the new tip; their blobs stay valid. Two batches ahead is measured, not a guess: with one, the next batch waits on the slowest of its parallel blob creates (about the p94 create latency), which capped batch 16 at 256 to 291 writes/s on real GCS; with two, staging leaves the critical path and batch 16 reached about 355 writes/s; three added nothing. Per-slot time is then about one create round trip.

Several replicas

• Every replica's store commits for every world it serves; requests are served where they arrive. The slot race is the only arbitration, and it lives entirely in the store.
• A lost slot costs one read of the winning slot and a retry on the next name. The loser's batch keeps its blobs and rebuilds only members the winning slot touched.
• Fairness: a pipelining winner starts its next attempt one slot read ahead of the losers, so it keeps winning under sustained contention; measured without a rule, one of three stores got 51 writes/s against another's 151, with a p99 of 6.5 s. A store that has seen another store's ID in a slot within the last 3 s waits up to 30 ms of jitter after each win before its next attempt; measured, that evens the stores out and cuts the worst p99 to about 3 s for about 10% of throughput. A store alone in a world never sees another ID and never waits.
• With 2 or 3 stores racing, total throughput stayed at the single-store level and about half of all slot attempts lost. If contention costs too much in production, any further coordination stays inside bucketstore.

Lost slots and rebase

If the slot exists, read the slots up to the new tip; they carry their entries, so this costs no further reads. A member whose path, ancestors or descendants, or the world's publish policy changed in those slots is rebuilt so its checks answer as today, which may now return a conflict. The others are carried as they are, and the batch tries the next slot name. Rebase never restages anything. Attempts are bounded by the request deadline; there is no fixed pacing on the commit path.

Ambiguous outcomes

On a timeout or lost response, read the slot. Absent: retry the identical create. Present with this batch's operation IDs: committed. Present with other IDs: lost, rebase. A slot is never rewritten, so the outcome is always decidable; the evicted-receipt server-error path of ADR 0008 disappears.

Snapshot and read path

• A replica's snapshot is the latest checkpoint it loaded plus every slot applied since. Slots carry full entries, so applying one needs no further reads.
• Path reads (FETCH, VERSIONS, chain verification) check the next slot name (metadata only, measured 11 to 14 ms) before serving; new slots are read and applied. GCS and S3 both give strong read-after-write and strong not-found. FETCH of the current version is then the tip check plus the blob: two bucket calls instead of today's four.
• Search reads (LOOKUP, LIST, hash lookup) serve the snapshot if it was confirmed current within the last second, and check the tip first otherwise. That bounds the lag at 1 s and costs at most one tip check per second per replica, however heavy the search traffic.
• Versions. Version records since the latest checkpoint are held in memory from the slots; older versions come from the checkpoint's history blocks.
• A replica installs its own commits as they succeed, so a writer reading through the replica it wrote to sees its writes at once.
• Optionally, peer hints push each new commit's world ID and sequence to the other replicas, never a slot name, so they usually catch up well inside the bound. Without them, a replica following by reads applied 250 writes/s with a lag p99 of 166 ms.
• One request still pins one snapshot. A view pins it at its first read: a path read pins the tip, a search read may pin one confirmed within the last second.

Section index for body search

Today the section index is rebuilt inside every commit (indexSections walks every path, sections.go:20), and a cold start reads every document body to build it: about 1,000,000 bucket reads at the baseline. It is also the heaviest per-replica memory after the catalog; the writer memory leak fixed in Knowledge writer memory growth was in its vocabulary.

• Off the write path. No commit and no slot apply computes sections. Catalog LOOKUP is unaffected.
• Built on first use. A world has no section index until a LOOKUP with match=body arrives there. That query starts the build and is answered from the catalog, reported as catalog fallback, which the LOOKUP contract already defines as inconclusive. Body search answers from sections once the build is done.
• Built from checkpoints, not bodies. The compactor writes section segments per shard into each checkpoint, indexing only documents whose body changed since the previous checkpoint; it reads each changed body once, so no replica does. A replica builds a world's index by loading segments, then indexes the documents changed since that checkpoint. At the baseline the segments are about 3 GB and took about 21 s to load, so the catalog fallback window at 1,000,000 documents is tens of seconds.
• Kept current in the background. After slots apply, a bounded worker reads changed bodies and updates their sections. Body-search lag is that queue's backlog; measured at 250 writes/s with 16 workers it stayed under 200 ms at p99. A body search that finds the queue behind reports the result as partial.
• Dropped when idle. A world's section index is evicted after a period with no body search, and rebuilt from the next checkpoint on demand. Memory for sections follows body-search use, not world count; a resident index for a baseline world is about 3 GB of text before indexing, which is part of the next wall below.
• Each replica reads each changed body at most once, only for worlds where body search is active: at the target rate, up to a few hundred small reads per second per replica, against a bucket that starts at about 5000 reads per second.

Checkpoints

• The compactor is part of the store and runs in every replica; there is no leader and no Kubernetes Lease. A checkpoint at sequence S is fully determined by the log up to S and the format (canonical JSON, sorted entries, content-addressed objects), so two compactors that both checkpoint S write identical bytes and create-if-absent makes the race harmless.
• To avoid duplicate work, a store compacts only when the newest checkpoint lags the tip by N slots or T seconds (1024 slots or 60 s), waits a random delay first, and checks again before writing. At the baseline a full checkpoint (863 MB) took 6 to 13 s, and writing one every 60 s showed no clear effect on write latency.
• It writes the index in the ADR 0008 shape with the manifest folded in: the root, the shards, each entry carrying its history block references, new history blocks for documents that gained versions, and section segments for documents whose body changed. Then it creates _demarkus/v1/checkpoints/<sequence as 16 hex digits>.json naming the sequence it covers and its root.
• Only shards, history blocks and section segments that changed since the previous checkpoint are written; unchanged ones are referenced by hash.
• A checkpoint is derived, not committed. Readers verify it against the slot chain; a missing, late or failed checkpoint only means a longer replay. Correctness never depends on the compactor.
• Shard count scales with the world, targeting about 200 KB per shard: 4096 shards at the 1,000,000-document baseline. A fixed 4096 made a 100,000-document cold start take 4.2 s, bound by the number of small shard reads rather than bytes.

World marker and cold start

• head.json becomes a frozen world marker in the schema 2 format: world ID and format version, nothing that changes. It is created with create-if-absent for a new world and replaced exactly once for a migrated one, and never written again. It stays because an old replica that found no head would take the world for a new one and initialize a second genesis; a schema 2 marker makes it fail closed instead.
• A new world starts with the marker and checkpoint zero, an empty root at sequence 1; the first slot is sequence 2.
• Cold start lists checkpoints/ for the newest, loads it, then lists log/ from its sequence with a start offset (strongly consistent on GCS and S3) and applies the slots to the tip. blob.Store.List needs a start offset for this. No document bodies are read; the section index waits for the first body search. Old checkpoints are collected, so the first list stays short.
• Measured at the baseline: 7.6 s on an uncontended node (10.3 s contended), 1024 slots replayed in 0.9 s, index heap 1.04 GiB. Bound: 15 s at 1,000,000 documents.

Change feed

• Slots are the durable change record, so the hub's backlog reads them directly, with an LRU of recent slots. The slot holding sequence q starts within one batch limit of names at or below q, so a resume lists from there.
• Slots are kept at least as long as the hub's ring covers, which keeps ADR 0031's guarantee: a cursor resumes across a replica restart up to the ring. A cursor older than the retained slots gets resync.
• Retention must also cover the federation deriver (incremental federation, shipped in #536). It resumes each world from the cursor in its last hub checkpoint; a cursor past retention gets resync, and the deriver rebuilds that world by reading every document at readsPerSecond = 20 (about 14 hours at 1,000,000 documents). Slot retention is at least the deriver's checkpoint interval plus the deriver outage the deployment tolerates.
• ADR 0031's sealed blocks and their code (sealChanges, sealedThrough, the block LRU, changes/ objects) are removed. A slot already holds up to 16 changes, so a second batching layer adds nothing.

Garbage collection

Reachability is the latest checkpoints kept for rollback, every slot after the oldest of them, the feed's slot retention, backup pins and active imports. The collector's sweep epoch becomes a log entry like any other commit; a batch whose slot create loses to a barrier entry sees the active epoch, waits for the clearing entry, and re-stages its blobs, which preserves ADR 0008's fencing. Like the compactor, the collector runs inside the store and is safe to start on several replicas.

Rates

Projected 2026-10-01, measured 2026-10-02 on real GCS from GKE (spike). c is one small-object create: 40 ms at p50 and 50 ms at p99 alone, 45 to 52 ms at p50 under a committer's load.

• Write latency. Without a queue, one store, batch 16, two ahead: p50 147 ms and p99 200 ms at 200 writes/s; p99 306 ms at 300 writes/s. That is about 3.5c at p50 and 5c at p99, not the projected 1c to 3c: a write waits for batch formation, its blob create, the slot in flight and its own slot create.
• Gate. 250 writes/s for 10 minutes on a 1,000,000-document world with a following replica and a compactor: 150,012 writes, none lost, no 429 or 5xx, p99 196 ms.
• Bucket bound. About 1.1 objects per write. A warmed bucket took 1200 creates/s without a 429; latency doubled from about 1000/s. A cold bucket was not tested.
• Replica refresh. A replica reads only slots: about 20 KB per batch of 16, about 0.5 MB/s at 400 writes per second.
• Compactor. At 400 writes per second over 60 s, about 24,000 changed entries touch nearly every one of 4096 shards, so a checkpoint is close to a full rewrite: 863 MB in 6 to 13 s at the baseline.
• Queue bound. The 10 s request deadline caps the queue at about ten seconds of throughput before timeouts.
• Several replicas. With R replicas writing one world, total throughput stays at the single-store level and each replica wins about 1/R of the slots; with the yield rule each replica's p99 under saturation was 1.6 s with 2 and 2.2 to 3 s with 3.

Migration

• A new replica that finds a schema 1 head.json at sequence S creates checkpoints/<S> naming the head's root (deterministic bytes, so concurrent replicas converge through create-if-absent). Schema 2 readers accept unfolded shard entries with manifest references in a checkpoint, and a checkpoint without section segments, whose first body search reads bodies as today.
• It then CAS-replaces head.json with the frozen schema 2 marker, conditioned on the generation it read. If an old replica committed in between, the replace fails; the new replica rereads the head, checkpoints the new sequence and tries again. A checkpoint left behind at an older sequence is valid state and is simply not the newest.
• The first write after the marker creates slot S+1. The compactor's first checkpoint writes the folded format, the world-sized shard count and section segments.
• No change blocks are sealed. Cursors at or after S resume; cursors older than S get one resync.
• The federation deriver's checkpointed cursors are older than S for every world written since its last checkpoint, so migration triggers one federation rebuild per such world. Cheap at today's world sizes; keeping ADR 0031's blocks readable (read only) until the deriver has checkpointed past S avoids it.
• Old replicas refuse a schema 2 head (unknown fields are refused) and fail closed with server-error, so they cannot commit during a rolling deploy. No split brain.
• Rollback needs a tool that writes a schema 1 head from a fresh checkpoint at the log tip in the old format; decide whether to ship it or document the migration as one way.

Invariants

• A slot is created once and never rewritten. The next slot is only attempted after the current tip was observed, so the log has no gaps; each entry names its predecessor's hash, and sequences run contiguously across slots.
• head.json is written once per world, at creation or migration, and never on the commit path.
• The log alone determines world state: replaying slots from any valid checkpoint gives the same snapshot, and a world with no checkpoint after checkpoint zero is still correct.
• A checkpoint is a deterministic function of the log up to its sequence; any number of compactors may race on it.
• A batch's mutations are applied in queue order; each member's preconditions are evaluated against the tip plus the members before it, and one member's failure never fails another.
• Each acknowledged mutation is in exactly one slot; acknowledgement follows only a successful create or a reread that finds its operation ID in the slot.
• A speculative batch never commits ahead of its predecessor slot.
• Everything written before the slot commit is content-addressed, so a losing batch never holds a name a winner needs.
• A commit's write cost depends on the change, never on the world's document count.
• Section indexing never runs on the commit path or when a slot is applied; a world's section index is resident only while body search is in use there.
• A body search never claims completeness it lacks: a missing or lagging section index is reported as catalog fallback or partial.
• Write concurrency control and background maintenance live behind the backend contract; nothing above the store elects writers, routes writes by replica, or schedules store work.
• The server holds a world's store only as backend contract types and constructs it at one wiring point; no bucket concept crosses the contract, so another store drops in by changing that constructor.
• Slots are retained at least as long as the hub's ring covers and the federation deriver's checkpointed cursor age; a resume within that window never resyncs.
• Every write's preconditions are evaluated at the tip; path reads are exact; search reads serve a snapshot confirmed current within 1 s.
• One request reads one snapshot.
• Expected-version, topology, policy and archive semantics are unchanged; the single-store conformance suite passes without edits, and cross-replica freshness tests move to the 1 s bound for search.
• New long-lived state (the committer queue, version records since the checkpoint, the slot LRU, resident section indexes) is bounded by the checkpoint interval, idle eviction or an explicit cap, evicts, and ships with a memtest.Retained test on production-shaped data.

Phases

1. Spike against real GCS from a GKE pod. Done 2026-10-02: the gate passed (250 writes/s for 10 minutes on a 1,000,000-document world, no lost writes, no 429, p99 196 ms; cold start 7.6 s at 1M). Results and the harness in the spike report.
2. ADR. Written 2026-10-02 as ADR 0036 (proposed). Replaces ADR 0008's commit point, per-commit root and shard rewrite, manifest layer, immediate search freshness and 100,000-document limit, and supersedes ADR 0031; keeps ADR 0008's content-addressed, verified object model for checkpoints and records the bucket-only, 1 s, throughput, fold, enterprise-scale, 1,000,000-document, frozen head, no-blocks, store-abstraction, swappable store and lazy section index decisions, with the phase 0 numbers as evidence. Its store boundary section names the replica follow contract (backend.Follower) and lists what stays private to bucketstore.
3. Log commit and read path in bucketstore behind the schema 2 format. It starts with the store boundary refactor on today's store, committed on its own before any log code: the world manager on contract types only, the poll loop inside the store, genesis inside open, one store factory, and a world served on a store that is not a bucket store. Then slots carrying entries and version records, snapshot as checkpoint plus applied slots, the frozen world marker, checkpoint zero for new and migrated worlds, cold start by listing (a start offset on blob.Store.List), parallel blob staging, and no section work in commit or apply. Tests: slot race, ambiguous create, rebase without conflict, same-path conflict, parent and child topology race, policy change during a race, sweep barrier, chain verification, path reads exact across replicas, search lag bounded at 1 s, replay from checkpoint zero equals the live snapshot, an old replica refusing the schema 2 marker, and the single-store conformance suite passing against both filestore and bucketstore.
4. Group commit, pipelining and several replicas, all inside bucketstore: the per-world committer in each store, chained member builds, per-member failure, two speculative batches staged ahead, a batch limit of 16 growing to 32 under a deep queue, rebase of a whole batch, queue bounds against the request deadline, the store ID in each slot and the yield-after-win rule (30 ms jitter, 3 s recency). No writer election, forwarding or routing above the store. Tests: 2 and 3 stores sharing one bucket and committing concurrently lose no writes and preserve every conflict, a pipelining winner yields once another store is seen, a lone store never yields, and the conformance suite passes against each store. Latency gate: one store at 200 writes/s, p99 at most 250 ms.
5. Checkpoints and the manifest fold: the compactor inside the store on every replica, lag-triggered with a random delay, folded entries with history block references, history block materialization, section segments for changed bodies, incremental shard writes, a shard count sized to the world (about 200 KB per shard), garbage collection reachability and old checkpoint collection. Tests: a checkpoint equals replay, two compactors racing on one sequence write identical bytes, a crashed or skipped checkpoint costs only replay, retention pruning across a 256-version block boundary, archive and unarchive, export parity, snapshot memory under memtest.Retained, cold start within 15 s at 1,000,000 documents.
6. Lazy section index: build on the first body search from checkpoint segments, catalog fallback reported while building, bounded background catch-up after slot apply, partial results while behind, idle eviction. Tests: no section work before the first body search, the first body search reports catalog fallback, results match an eager index once caught up, a lagging queue reports partial, an idle world releases its sections under memtest.Retained, a pre-migration checkpoint without segments builds from bodies.
7. Change feed over slots: the hub backlog reads slots, slot retention covering the ring and the federation deriver's cursor age, removal of ADR 0031's blocks and sealing code. Tests: resume across a restart within the ring, resync past retention, resync for a pre-migration cursor.
8. Migration and the rollback decision; re-run the write scenarios on kind and on a scratch GKE world of 1,000,000 documents with 3 replicas: 64 concurrent writers sustained at 200 writes per second must lose none. Include an old replica committing during the migration's head replace.

Next wall, out of scope

Every replica holds the whole catalog in memory: about 1.04 GiB of index heap at the 1,000,000-document baseline in the spike, and ADR 0008 measured 133 MiB per 100,000 for the full catalog. A resident section index for a baseline world adds about 3 GB of section text before indexing. The lazy section index keeps that to worlds where body search is in use, but one active baseline world already needs several GiB per replica. With that baseline, several large worlds in one process need a catalog and section index that are not fully resident, such as immutable search segments in the bucket with a local cache. That is a separate plan, needed before more than one baseline-sized world shares a process; this one keeps write cost independent of world size, not memory.

The federation deriver caps a world at maxSources = 50_000 rows (knowledge/internal/federation/federation.go); past it the world's graph checkpoint is incomplete. A 1,000,000-document world needs that cap raised and its shard sizing checked, in the federation track.

Related, separate

Root's head showed generated graph and index documents republished every crawl cycle (/graph.md at v8718, /index/bruno:6309.md at v8491 while bruno was at sequence 3). The in-process federation graph (#536) replaces the crawler in the knowledge system and skips a graph export whose content hash is unchanged (federation/export.go). The standalone demarkus-agent still stamps time.Now() into bodies (fedcrawl/crawl.go, Exported: and Indexed:), which defeats the unchanged-body skip; fix it only if it keeps publishing to a bucket store.

Open questions

• Remaining enterprise targets: sustained and peak writes per world, worlds per process.
• Whether a cold bucket throttles near 1000 creates/s before GCS scales it, whether a hierarchical namespace bucket starts higher, and whether to pre-warm buckets.
• Write latency under contention at a fixed rate below saturation; phase 0 measured contention only at saturation.
• The idle period before a world's section index is evicted, and the catch-up worker count.
• Slot retention beyond the hub's ring and the federation deriver's cursor age, and how many old checkpoints to keep for rollback.
• The federation deriver outage to tolerate before a world rebuilds (about 14 hours of reads at 1,000,000 documents and 20 reads per second), which sets slot retention.
• At migration: accept one federation rebuild per written world, or keep ADR 0031's blocks readable until the deriver checkpoints past the migration sequence.
• Raising the federation deriver's maxSources = 50_000 cap to the 1,000,000-document baseline.
• One-way migration or a shipped rollback tool.
• Hash lookup: ADR 0035 (proposed) removes it. If accepted, hash lookup leaves the 1 s class and the body-hash map leaves snapshots and checkpoints; until then its only callers, FETCH by hash and mark_resolve, tolerate the 1 s lag.
• When an S3 backend lands; the design needs only create-if-absent, strong read and not-found, and a strongly consistent list, which S3 has.

Status

Proposed 2026-10-01, decisions recorded 2026-10-01 and 2026-10-02. Unblocked 2026-10-02: the incremental federation graph shipped in #536. Phase 0 done 2026-10-02: the gate passed on real GCS, and the measurements added two batches staged ahead, a batch limit growing to 32, the 30 ms yield, a world-sized shard count and a 250 ms p99 latency gate. The swappable store decision on 2026-10-02 adds a store boundary refactor at the start of phase 2, which removes the knowledge server's reach into bucketstore. Phase 1 done 2026-10-02: ADR 0036 (proposed), naming backend.Follower as the replica follow contract. Next is the store boundary refactor that opens phase 2, on branch feat/bucket-commit-log.

Store boundary refactor done 2026-10-02 on feat/bucket-commit-log, uncommitted and awaiting review: backend.Follower, changefeed.Hub.Subscribers with Subscription.Close, the follow loop and genesis inside bucketstore.Open with a Close, one store factory in server/knowledgeserver/run.go (the only non-test importer of bucketstore), and a world manager test that serves a world on filestore. The sweep found two more leaks, fixed in the same change: hint routing read world.Bucket.WorldID (now WorldConfig.ID()), and peerhint documented hints as head moves (comments only; the head wire token stays for replicas mid-deploy). Next is committing it, then phase 2's log code.

Phase 2 done 2026-10-02 on feat/bucket-commit-log, uncommitted: schema 2 world marker, checkpoint zero at genesis, one slot per commit with parallel blob staging and read-back of unknown outcomes, cold start by listing (blob.Store.List gained startAfter), tip probes for reads by path and a one second confirmation for catalog reads (backend.SearchFreshness, the SPEC's freshness rule, storetest.RunReplicaConformance), and the hub fed from slots with a slot backlog in place of ADR 0031's sealed blocks. Decisions taken while building it, with the maintainer:

• The feature rolls out whole (2026-10-02): nothing merges before phase 7, so phase 2 refuses schema 1 worlds by name and migration stays in phase 7.
• Snapshots are copy-on-write trees (google/btree, 2026-10-02): rebuilding the snapshot on every slot would cost time proportional to the world, which the spike never measured; paths, per-directory child counts and the body-hash index are trees a slot derives in time proportional to its change.
• The catalog scans an index the store supplies (catalog.Index and catalog.Search, 2026-10-02), so bucketstore keeps no second catalog.
• Sections stay eager until phase 5: a replica indexes the bodies a slot changes, so the body-match conformance cases pass unedited until phase 5 moves indexing off the write path and changes them.
• A view pins its snapshot at its first read, by the read's class; handlers are unchanged.
• The sweep barrier test waits for phase 4, where the collector that writes barrier entries lands.

Until phase 4, version records since checkpoint zero stay in memory and a cold start replays every slot, both growing with the log. Next is phase 3, group commit.

Phase 3 done 2026-10-02 on feat/bucket-commit-log, uncommitted, except the latency gate on real GCS, which waits for approval of the run. Group commit, pipelining and several replicas live in bucketstore (commit.go, batch.go); nothing above the store changed. Decisions taken while building it:

• The committer runs on demand. The first queued write starts it and it exits once the queue and pipeline are empty, so an idle world holds no goroutine; Close waits for it, lets a slot already being created finish and refuses everything else with backend.ErrClosed.
• A batch refreshes only when the pipeline is empty. Batches behind it build on the previous batch's candidate; a refused write still runs its precondition once on a single store.
• Answers wait for the state they were judged on. A refusal judged on a confirmed base alone goes out at once; one judged against earlier members, or on an unconfirmed base, goes out when that state exists, so no refusal is ever answered from a state that never existed.
• Rebase tracks what each member read. A member records its path, ancestors, whether its path is a directory, and every path its precondition read through the attempt view, as the snapshot's state pointers; a listing or catalog read marks it broad. After a lost slot, a member whose reads still hold is kept as built and its precondition is not run again; the rest are rebuilt. The store never learns where the publish policy lives.
• A slot is replayed on its base before it is created, so a slot that would not apply is never written, whatever the rebase logic got wrong.
• Reads during a build go through the batch's objects: the unstaged objects of earlier batches and a cache of what the batch read, with each existing member's history and tip prefetched in parallel, so members of one batch do not read the bucket one by one.
• Speculative batches are requeued, not rebased, when the batch before them loses or fails; staged keys are remembered per request, so nothing is created twice.
• The Committed hook runs once per slot with its last sequence, from the last member's request.

Next is the phase 3 latency gate on real GCS (one store, 200 writes/s, p99 at most 250 ms), then phase 4.

Phase 3 latency gate run 2026-10-02 on real GCS (report): failed, p99 326 and 336 ms against 250 ms at 200 writes/s, with no write lost and every acknowledged write verified. GC, CPU throttling and GCS throttling are ruled out. The tail is GCS calls over 200 ms on the serial slot path (one create at a time, staging waits for the slowest blob), over a baseline per-second p99 of about 210 ms at 92% of one-create-at-a-time capacity. Candidate fixes: hedge GCS calls past about p95, and start the prefetch read at enqueue. Phase 3 stays open until the gate passes.

Phase 3 latency gate passed 2026-10-02 (run 3 in the gate report): p99 203 ms at 200 writes/s on real GCS, p999 244 ms, no error, every acknowledged write verified. It took two changes, both uncommitted on feat/bucket-commit-log: blob.Hedged, which sends a second Get or Create once the first passes the recent p95 of its kind (about 5% more GCS operations), turned on by bucketstore.Open because its creates reconcile by reading the name back; and a warm-up read of each write's document started when it is queued. Next is phase 4.

Phase 4 step A (checkpoints) done 2026-10-02 on feat/bucket-commit-log, uncommitted; step B (collection) waits for review. Decisions taken with the maintainer before building it:

• Section segments move to phase 5, with the loader that reads them. Phase 4 checkpoints carry none, which phase 5's "checkpoint without segments builds from bodies" case already covers.
• The 1,000,000-document cold start gate runs after phase 5. Sections stay eager until then, so a cold start still reads every live body and cannot meet 15 s; phase 4 proves checkpoint load and replay locally.
• Folded format, schema 2 root and shards. The shard count is the smallest power of two that keeps each shard to 256 documents (4096 at 1,000,000, one for an empty world), and a path lies in the shard its path hash's top bits name; the root records shard_bits and must match its document count. An entry carries history: [{first, last, hash}], key and path hash derived. History objects are unchanged and reused by hash. Checkpoint zero is now one empty folded shard, so genesis writes 4 objects instead of 259.
• Replicas rebase on a peer's checkpoint by reading only the shards whose reference changed, then drop the versions the checkpoint holds from memory.

What was built (checkpoint.go): a compactor in every store, started when the slots it installed since its newest checkpoint reach 1024 or the first of them is 60 s old, after a random wait of up to 10 s, and a recheck that adopts a peer's newer checkpoint instead of writing. There is no timer: a world gone quiet keeps its last slots until the next write. A checkpoint writes history blocks only for documents that changed, shards only where something changed (all of them when the shard count changes or the base is schema 1), the root, and last the checkpoint object; a shard is reused when every document in it still equals its checkpoint entry, judged by content, so a snapshot that missed an adoption only costs memory. The writer and peers adopt a checkpoint by rebasing the served snapshot, and a snapshot built before it (a batch in the pipeline, a pending slot) rebases as it is installed, so pathState identity changes once per checkpoint. Read-only stores adopt but never write. Cold start still bounds the checkpoint load by the request timeout; revisit with the 1,000,000-document gate.

Also: blob.Hedged now starts with its hedge budget banked, since a new world no longer reads 256 shards at open to earn it, and its first stalled create waited out the request deadline.

Next is step B: reachability, old checkpoint collection and the sweep epoch with its barrier test.

Phase 4 step B (collection) done 2026-10-02 on feat/bucket-commit-log, uncommitted; the design is recorded as an amendment to ADR 0036. Decisions taken with the maintainer, all four as proposed:

• A generation fence, not a log barrier. The plan's barrier fenced slot writers only, since checkpoints are written outside the log, and stalled every write while it deleted. A sweep deletes objects that nothing reaches and that are older than one hour, each conditioned on its listed generation. A create that reuses stored bytes older than 30 minutes rewrites them first.
• Kept checkpoints: every one from the last hour plus the newest three. Each store catches up and rebases every 20 minutes.
• Slot retention: 24 hours before the oldest kept checkpoint. A cursor past it resyncs.
• Cadence: each store tries hourly, and a create-only claim per hour (_demarkus/v1/sweeps/<hour>.json) lets one sweep.

What was built (collect.go): a maintenance goroutine per store, first tick 20 to 25 minutes after open. The sweep marks the kept checkpoints (roots, shards, manifests, history blocks to blobs) and every slot after the oldest kept one. It then lists each content-addressed prefix and deletes in order: expired checkpoints first, then unreached objects past the grace, slots past retention, and claims. Listings stream page by page, and the mark keeps raw 32-byte hashes. A replica whose snapshot has gone unconfirmed for 12 hours reloads from the newest checkpoint; without that, a collected next slot would read as the log's tip, and its next commit would re-create a collected slot name. ADR 0008's collector, pins and imports were never implemented, so pins and imports are not roots yet.

Cost to measure with the 1,000,000-document gate: every sweep reads every kept shard and history block, about 1,000,000 reads and about 150 MB of hashes at the baseline, once an hour per world. Superseded checkpoint objects stay about two hours, about 100 GB at a sustained 400 writes per second. If either is too much, the mark can become incremental.

Phase 4 is done apart from its 1,000,000-document cold start gate, which runs after phase 5.

Phase 4 step B revised 2026-10-02, later: at today's volume, about a couple of gigabytes a month, the maintainer deferred content collection. Step B is now old checkpoint collection only. After writing a checkpoint, the compactor deletes checkpoint objects past the newest three that are more than an hour old, each at its listed generation, so cold start lists few. All content and slots stay. The full collector and its tests are kept as a patch, and the design stands in the ADR 0036 amendments. Revisit when storage growth matters; at sustained hundreds of writes a second it is tens of gigabytes a day per world. Backup pins and imports are moot until then.

Phase 4 step B, final 2026-10-02: after writing a checkpoint, the compactor drops old checkpoints together with the shards and roots only they used, which is the bulk of the growth. The newest three stay, and so does any checkpoint whose successor is under an hour old. Deletion is fenced: objects are deleted only past one hour old, at the generation read, and a create that reuses bytes older than 30 minutes rewrites them first. History blocks, blobs of pruned versions and slots still accumulate, which is small next to the shards. Design in the ADR 0036 amendments.

Phase 5 (lazy section index) done 2026-10-02 on feat/bucket-commit-log, uncommitted. Decisions taken with the maintainer, all four as proposed and recorded as an ADR 0036 amendment; they replace "partial" wherever the sections above use it, and answer the open question on the idle period and worker count:

• Fallback, not partial. A body search waits up to 1 s for an index version that covers exactly its pinned snapshot, then refuses with backend.ErrBodyMatchUnavailable; the handler reruns the lookup in catalog mode on the same view and echoes match: catalog. The SPEC gained one sentence allowing that answer while the index is built or behind. No client change: fetch.AnsweredFromCatalog, the MCP note and mark_lookup_all already treat it as inconclusive.
• Body segments keyed by shard. Segments hold bodies, not serialized sections, at segments/<label>/<shard hash>/<part>.json, in parts under 2 MiB since production objects are capped at 4 MiB. The root names none, so checkpoint bytes are unchanged.
• Defaults: eviction after 30 minutes without a body search; catch-up with the 16 shard workers.
• Conformance: body lookups in the suites retry a catalog fallback for up to 5 s (storetest.SendSettled, settledLookup), and a new handler case, BodyFallbackEchoesCatalog, checks the echo and the catalog rows over a store that always refuses body mode.

What was built: sections.go (the index: one immutable version per installed snapshot, retained for the request timeout; a queue of installs fed by install, which does no section work; a worker started by the first body search that builds from segments and blobs, catches up behind installs, backs off on failure and exits when idle; 1024 queued installs collapse into one comparison with the served snapshot), segment.go (format, parts, verification, the compactor's carry with a backfill budget of 4096 reads per checkpoint, the builder's repair of unchanged shards, deletion with a dropped shard). Eager indexing is gone from load, catchUpLocked, slot apply and the committer, and pathState no longer holds sections.

Measured locally: IndexSections runs at 17 MB/s per core on the soul corpus, whose resident index is 2.48 times body bytes; on agent graph bodies the resident index is 1.42 times body bytes, and eviction releases all of it. At the baseline a build reads about 4 GB of segments and indexes about 3 GB of text, so the first body search on a 1,000,000-document world falls back for about 45 s on 4 vCPU. Each new test was checked against a mutation of the guarantee it covers.

Next is the 1,000,000-document gate on GCS (cold start within 15 s, plus checkpoint, segment and drop costs), proposed with its cost before anything is created, then phase 6.

Phase 5 revised after review the same day: segments are written after the checkpoint object, and a drop sweeps segments/ once, collecting orphans too (ADR 0036 amendment). For the 1,000,000-document gate: a build without segments at that size would take longer than the 5 minute section timeout and start over each time; repaired segments carry over between attempts, but a world migrated in phase 7 may need a longer bound.

2026-10-03: the checkpoint grace is now 15 minutes by default, configurable per world as bucket.checkpointGrace with 15 minutes as the minimum (ADR 0036 amendment). At the baseline and 400 writes/s that keeps about 15 checkpoints instead of 60 and about 70 GB of superseded shards and segments instead of about 290 GB. Also from PR 538 review: an evicted section worker now closes its done channel in the same locked step, so a new worker never overlaps it.

No migration (decided 2026-10-03 with the maintainer). Nobody but us runs a bucket store, and the production knowledge system holds only test data, so phase 7 builds no conversion and no rollback tool. At rollout the production buckets are emptied (or replaced) and every world starts from genesis; a schema 1 head is already refused at open, so a world missed by the wipe fails closed instead of opening. That also settles three open questions: migration is one-way, there is no federation rebuild to accept, and no pre-migration cursor exists to resync. Phase 7 is now the rollout (wipe, deploy, reseed what the tests need) and the end-to-end gates on kind and on a scratch 1,000,000-document world with 3 replicas. Emptying production buckets is asked for at rollout, not before.

Phase 6 (change feed over slots) done 2026-10-03 on feat/bucket-commit-log, uncommitted. The verification found the hub read its backlog only within one ring (4096 events) of the head, so slot retention did nothing for the federation deriver: with its cursor saved about once a minute, any deriver restart more than 4096 changes behind rebuilt the world. Decisions taken with the maintainer, recorded as an ADR 0036 amendment:

• The feed reaches 16,384 slots back, read a page at a time; a slow reader keeps reading slots until it reaches the ring. This needed server/internal/fanout too (approved): its catch-up held the whole backlog in memory and joined its ring where it stood at attach.
• Slots are collected after 24 hours once the oldest kept checkpoint covers them, with the stale replica reload at 12 hours, both from the set-aside collector.
• The slot cache empties after 30 minutes idle.

What was built: changefeed.Backlog.Reaches and paged subscriptions (TryNext takes a context); fanout's batched catch-up, dynamic join and a resume cursor that never passes an unwritten event; bucketstore reach counting, dropSlots after each checkpoint's drop, reloadIfBehind with a section resync, the cache's idle eviction. The change conformance suite is unedited: the bucket site's window is now its reach, and a second run over stores that checkpoint after every slot, stamped an hour old, makes resumes cross real checkpoints and dropped ones. Leftovers removed: headSequence (now logSequence), stale comments in batch.go, run.go and a fanout test fake; ADR 0025 and 0028 carry notes. Every new test fails on its mutation.

This answers two open questions: slot retention is 24 hours, and the deriver outage tolerated before a rebuild is the reach, about 10 to 20 minutes at 400 writes per second and hours at low rates. Two limits remain outside the store: the deriver applies a resume at 20 reads per second, so ten minutes at 400 writes per second takes hours to apply even after coalescing (the pacer is an open question in the federation plan); and it never saves a cursor for the hub world while only its own graph writes change it, so a restart after 16,384 such slots rebuilds the hub world, which is cheap.

Next is the 1,000,000-document gate on GCS, once its run and cost are approved, then phase 7's rollout.

Full-feature scan 2026-10-03, uncommitted on feat/bucket-commit-log: five reviews (old-format vestiges, architecture and concurrency, memory, two on simplicity) over main...HEAD, every finding checked against the code. make test, bash pre-commit.sh and -race pass, and each guarantee added this session fails a test when broken (21 mutations).

• Removed: schema 1 refusal and its tests, the head peer hint token, head-CAS and 1.5 s pacing wording in docs and the chart README, "world head" wording (now "world marker").
• Fixed: slots never freshened; the stale reload shared, detached and bounded; the compactor refuses a snapshot that missed its rebase; warm-ups end with the store and drop the batch they read through; written metadata no longer pins stored frontmatter (944 bytes a document before); peer adoption reads a shard at a time and keeps bases only; adoption entries released once unused; section versions pruned on idle wakes; blob.Memory stamps wall time, which removed the test-only youngListing.
• Simplified: one fetch-and-verify path, one catalog record builder, checkpoint.go split into collect.go and adopt.go, four test knobs off Options, dead fields and wrappers gone.

Left for a decision or the 1,000,000-document gate: the committer waits on refreshMu while read probes do bucket I/O; each checkpoint hashes and groups every document (about 190 ms and 229 MB allocated per million) and rewrites a dirty shard's whole segment; catalog lookups scan and sort every match (about 1.15 GB for a broad lookup at a million); every world keeps a hub ring for life; reads decode the whole checkpointed history to use its tip; a long catch-up holds decoded slots until it installs; SPEC 6.8 still lets a server report an append as publish.

The seven open items from the scan are fixed (2026-10-03), uncommitted, with the segment design chosen by the maintainer and recorded in an ADR 0036 amendment: commits confirm under an install lock no probe holds; a checkpoint hashes only the shards it writes (60 bytes per further document against 188); segments are a 10-minute windowed cache, written once per window per shard; catalog lookups keep the best Max results in a heap, score ASCII entries without allocating and stop soon after their request ends; an unwatched hub drops its ring; a read takes one history block; a catch-up keeps each slot's events only; SPEC 6.8 no longer lets a server report an append as publish. make test, bash pre-commit.sh and -race pass; 23 guarantee mutations each fail a test.

What remains is the 1,000,000-document gate on GCS, now measuring the windowed segments and the lookup heap too, then phase 7.

Local 1,000,000-document scale test done 2026-10-04 (report) on a patched fake GCS, two replicas, in two runs. Run 1 stopped on compaction that could not catch up; four fixes followed, uncommitted on feat/bucket-commit-log and recorded as the 2026-10-04 ADR 0036 amendment with its two revisions: content creates without read back or freshen, bounded compaction steps from a kept base snapshot, one compactor per world without a claim (defer to a peer's fresh checkpoint), and a checkpoint cadence of 8,192 slots or 10 minutes. Verified: a 440k-sequence backlog checkpointed in 14 steps in 12 minutes; 3.1 million acknowledged versions read back across the runs with no gap, break or duplicate; open loop 193.5/s at 200 offered and 227/s at 250 on the fake, every timeout inside a compaction stall of the fake's lock; one-document checkpoint 0.04 s; catalog lookups at 1M 150 ms for * and 440 to 470 ms for a term; cold start 16 to 22 s against the 15 s gate, bound by the fake's object rate and doubled objects past 2^20 documents; the body index projects to about 14.5 GB at 1M (heap guard tripped). Open for the GCS gate: compaction capacity against arrival (estimated break-even near 200 writes/s with 16 workers), the adopter's per-step shard download, the base snapshot's heap at the 10 minute cadence (several KB per changed document), and the first step after a quiet period being written by both replicas. Phase 7 follows the GCS gate.

The catalog and section memory wall named above has its own plan since 2026-10-04: The memory wall, which starts after the GCS gate.

Merged to main 2026-10-04 as 06568432 (#538, "GCS write optimizations"), CI green, with the compaction fixes, the review fix and the scale test's findings. Still ahead: the GCS gate on a scratch 1,000,000-document world with 3 replicas (compaction capacity, adopter cost, cold start, p99 during steps), then phase 7's rollout (wipe production buckets, deploy, reseed, kind gate on fake-gcs 1.56.1 or newer), then ADR 0036 moves to accepted. The memory wall follows the gate.

The GCS gate has its own plan with its cost: Bucket commit log: the GCS gate at 1,000,000 documents, proposed 2026-10-04, about $35 per full run, a $100 budget to approve, one working day on a scratch pool of two n2-standard-8 and a scratch bucket with soft delete off.

GCS gate, tier B, run 2026-10-04 (report): correctness and single-writer throughput hold on real GCS; compaction keeps pace at 400 writes/s at 245k documents and projects to break-even near 200 writes/s at 1M with 16 workers; three replicas writing one world ran at 94 writes/s with p50 5 s, which blocks phase 7 until the several-writers question is measured and decided (shorter hold, a leader writer above the store, or one writing replica per world in practice). Cold start needs 64 shard workers to meet 15 s at 1M.

The several-writers result has its own plan: Several replicas writing one world, proposed 2026-10-04: an in-process model with GCS-shaped latency first, then one short GCS lane; tune the turns if that reaches 70% of one writer's throughput, else hand losers' batches to the replica at the tip inside the store.

Several writers, model done 2026-10-04 on fix/several-writers (uncommitted): the collapse was a loser's recovery of 1.2 s, the batches requeued behind a lost head rebuilt reading the bucket serially, so the 1 s hold expired and a replica starved. Requeued requests now carry their batch's reads and the hold is 300 ms; in the model three writers run at 221 to 237 writes/s against 235 alone, where before they ran at 91 with a starved store. Recorded as an ADR 0036 amendment; details in the several-writers plan. One GCS lane confirms it before phase 7.

Several writers confirmed on GCS 2026-10-04 (lane report): three writers at 298/s offered 300, 0 errors, p50 559 ms, against the gate's 94/s; nothing lost. Phase 7 is unblocked on this question. Still on the list before phase 7: cold start with 64 shard workers, the first checkpoint step written by every replica (and a compactor off the writers for the p99), compactor workers sized from the capacity numbers.

Planned 2026-10-05, for a session after PR 539 merges: Refresh as a shared flight, the scan's "committer waits on refreshMu while read probes do bucket I/O" item. Latency isolation, not throughput; about half a day.

Several-writers fix merged 2026-10-05 as b7a61d73 (#539). Next for phase 7: Refresh as a shared flight, then cold start with 64 shard workers and the compactor off the writers.

Refresh flight done 2026-10-05 on fix/refresh-flight, uncommitted and awaiting review: refreshMu replaced by a shared probe flight bounded by each caller's context, and a loser's catch-up outside it (plan, ADR 0036 amendment). The scan's refreshMu item is closed. Next for phase 7: cold start with 64 shard workers and the compactor off the writers.