Bucket store commit log: create-only commits instead of one head CAS

A plan to lift the per-world write ceiling of the GCS bucket store by committing each mutation as a new create-only log object instead of a conditional overwrite of head.json, and to let search reads lag by up to one second. Audience: demarkus maintainers. Proposed 2026-10-01, deferred until the incremental federation graph ships. Builds on ADR 0008 and replaces its commit point and its immediate search freshness; would supersede the sealing in ADR 0031. Part of the Knowledge server track.

Problem

Every mutation in a world ends with one conditional replace of _demarkus/v1/head.json, the world's only mutable object. GCS limits updates to one object name to about one per second, so bucketstore serializes commits behind commitToken and paces head attempts at defaultCommitInterval (1.5 s). The ceiling is about 0.67 commits per second per world, whatever the paths.

Measured on kind (writes): one sequential writer took exactly 1.51 s per publish; 4 concurrent writers queued to 6 s each; a 4-writer burst of 200 publishes lost 49 to the 10 s request timeout. Below the pacing, a write takes about 60 ms. The product target is many users writing one world concurrently, so this is a blocker, not a tuning issue.

ADR 0008 stated the assumption (read-heavy, under one mutation per second per world) and measured it, but it was never revisited when concurrent multi-user writing became the goal.

The constraint is the single mutable object name, not conditional writes. Any design where every commit mutates one object (CAS, lock, lease, pointer) inherits the same cap. Creating a new object name with create-if-absent has no per-name cap.

Decisions

Taken 2026-10-01 with the maintainer.

Bucket only. GCS (and later S3) stays the only state. A database for metadata, such as Cloud SQL Postgres beside GCS, is out; ADR 0010 stands.
Ordering stays per world. WATCH cursors and the change feed are numbered per world (ADR 0025, ADR 0028), and the federation graph is derived from that sequence. The world is the narrowest scope that can own it.
Search may lag by up to one second. LOOKUP, LIST and hash lookup on one replica may trail another replica's commit by at most 1 s. FETCH, VERSIONS and chain verification by path stay exact. Every write evaluates its preconditions at the tip, so the lag never affects write correctness. This relaxes ADR 0008's immediate LOOKUP and hash freshness.

Concurrency model stays the same

Expected-version semantics already hold today: bucketstore checks ExpectedVersion per document and a writer that loses the head race revalidates against the new snapshot, so writers to different documents never see a conflict and writers to one document get the normal Mark conflict. The world-level ordering exists for what one document cannot carry: the change sequence, path topology (/a.md against /a.md/b.md), and one snapshot per request. The file store has the same single ordering point (filestore.go mu), it just costs microseconds on local disk. This plan keeps the model and makes the ordering step cheap.

Alternatives considered

Group commit on head.json. Coalesce queued mutations into one head replace, committing at once when idle and batching only under load. Throughput becomes batch size per 1.5 s and the change is smaller, but writes under load take 1.5 to 3 s, the head stays a hot object whose 429s delay a whole batch, several writer pods race whole batches, the 16-receipt window must scale with batch size, and ADR 0031's blocks of 8 sequences would be skipped by one large batch. Kept only as an optional later phase on top of the log.
In-memory head shared between replicas, flushed every 2 s. Acknowledging before the flush loses up to 2 s of acknowledged writes when the holder crashes, and a deposed holder acknowledges writes its flush then loses. Making it safe means quorum replication (Raft), which makes GCS a snapshot store and does not survive all pods restarting together. Acknowledging after the flush is the group commit above.
Database for metadata. Out by the bucket-only decision.
Per-document conditional writes alone. Correct for versions, but there is no world sequence for WATCH and the feed, two creates on different objects can break topology, and a commit followed by a separate feed entry can crash between the two and never be indexed.
One head per shard or subtree. A parent and child path hash to different shards, and the change sequence is per world.
Libraries (surveyed 2026-10-01). None fits: mbrt/glassdb is alpha, GCS only, has no change feed or global index; JayJamieson/objwal is single writer and orders through one rewritten manifest; SlateDB is Rust with Go bindings, single writer with fencing, a key-value model; rivian/delta-go is Parquet tables, alpha, S3 with DynamoDB locking, no GCS. Useful as design references only: Delta Lake's put-if-absent log and logical conflict check on retry, SlateDB's put-if-absent commit and writer fencing.

Design

Log slots are the commit point

A commit creates _demarkus/v1/log/<sequence as 16 hex digits>.json with blob.Store.Create (create-if-absent; ifGenerationMatch=0 on GCS, If-None-Match: * on S3). The entry carries what the head carries today: schema, world ID, sequence, root reference, the operation receipt, the sweep epoch, plus the hash of the previous slot's entry so readers verify the chain. The successful create is the commit; nothing else is mutable on the commit path.

Write path

Pin the snapshot at the tip T. Writes always build at the tip, never on a lagging search snapshot.
Canonicalize, authorize, evaluate policy, topology and protocol preconditions against it, as today.
Stage blob, history, manifest, changed shard and root as immutable objects, as today.
Create log/T+1. Success installs the candidate on this replica and acknowledges.
If the slot exists, load the slots up to the new tip. If any of them changed the target path, an ancestor or descendant (topology), or the world's publish policy, re-run the full build so expected-version, topology and policy checks answer as today. Otherwise rebase: keep blob, history and manifest, rebuild only the changed shard and the root over the new tip, and try the next slot.
Bound attempts by the request deadline; no fixed pacing on the commit path.

Because the topology check runs at the tip inside the slot race, no path rule or protocol change is needed for it.

Ambiguous outcomes

On a timeout or lost response, read log/T+1. Absent: retry the identical create. Present with this operation's ID: committed. Present with another ID: lost, rebase. A slot is never rewritten, so the outcome is always decidable; the evicted-receipt server-error path of ADR 0008 disappears.

Read path

Path reads (FETCH, VERSIONS, chain verification) check log/<tip+1> (metadata only) before serving, as today's head validation does. Not found means the snapshot is current; GCS and S3 both give strong read-after-write and strong not-found. Found means reading forward until not found, then loading the changed shards.
Search reads (LOOKUP, LIST, hash lookup) serve the replica's snapshot if it was confirmed current within the last second, and check the tip first otherwise. That bounds the lag at 1 s and costs at most one tip check per second per replica, however heavy the search traffic.
A replica installs its own commits as they succeed, so a writer reading through the replica it wrote to sees its writes at once.
Optionally, the hub's fan-out pushes new slot numbers to the other replicas so they usually catch up well inside the bound.
One request still pins one snapshot.

Tip hint

head.json stays as a non-authoritative hint (sequence and root), written best effort off the commit path at most once per pacing interval. Cold start reads the hint and probes forward (exponential, then binary search) to the first missing slot. A missing or stale hint costs probes, never correctness.

Change feed

Log entries are the durable change record, so the hub's backlog reads slots directly (range read with an LRU of recent entries). ADR 0031's sealed blocks become a compaction format: ranges of old slots are sealed into blocks so old slots can be collected while the feed stays continuous.

Garbage collection barrier

The collector's sweep epoch becomes a log entry like any other commit. A writer whose slot create loses to a barrier entry sees the active epoch, waits for the clearing entry, and re-stages, which preserves ADR 0008's fencing.

Rates

At about 60 ms per commit, one world should reach roughly 10 to 16 serial commits per second before any batching (unmeasured on real GCS; see phase 0). Sequential slot names only hotspot GCS above roughly a thousand writes per second per bucket. Today all writes land on one pod; with several writer pods the slot race costs one round trip, not 1.5 s. Read cost: path reads as today, search reads at most one tip check per second per replica.

Migration

A new replica that finds a schema 1 head.json and no log creates log/<head sequence> from the head's exact fields (deterministic bytes, so concurrent replicas converge through create-if-absent), then CAS-replaces head.json with a schema 2 hint.
Heads are decoded with unknown fields refused, so an old replica reading a schema 2 head fails closed with server-error and cannot commit during a rolling deploy. No split brain.
Rollback needs a tool that writes a schema 1 head from the log tip; decide whether to ship it or document the migration as one way.

Invariants

A slot is created once and never rewritten; slot N+1 is only attempted after slot N was observed, so the log has no gaps, and each entry names its predecessor's hash.
Each acknowledged mutation is in exactly one slot; acknowledgement follows only a successful create or a reread that finds this operation's ID in the slot.
Every write's preconditions are evaluated at the tip; path reads are exact; search reads serve a snapshot confirmed current within 1 s.
One request reads one snapshot.
Expected-version, topology, policy and archive semantics are unchanged; the single-store conformance suite passes without edits, and cross-replica freshness tests move to the 1 s bound for search.
New long-lived state (the slot LRU, any rebase cache) is bounded, evicts, and ships with a memtest.Retained test on production-shaped data.

Phases

Spike on the benchmarks branch against real GCS. Slot-create commit latency p50 and p99, sequential rate, 4, 16 and 64 concurrent writers on one pod, two pods racing, 429 and 5xx counts, with the current head CAS as baseline. Record in /research/. Gate targets to set before running (open question below).
ADR (next free number). Replaces ADR 0008's commit point and its immediate search freshness, and ADR 0031's sealing; keeps ADR 0008's immutable root, shard, manifest, history and blob model and records the bucket-only and 1 s decisions.
Commit and read path in bucketstore behind the schema 2 format, with tests: slot race, ambiguous create, rebase without conflict, same-path conflict, parent and child topology race, policy change during a race, sweep barrier, chain verification, path reads exact across replicas, search lag bounded at 1 s.
Change feed over slots, compaction into blocks, retention for old slots.
Migration and the rollback decision; re-run the kind write scenarios: the 4-writer burst of 200 must lose none.
Group commit, only if phase 0 or 4 falls short: coalesce the commitToken queue into one slot holding several receipts.

Root's head shows generated graph and index documents republished every crawl cycle (/graph.md at v8718, /index/bruno:6309.md at v8491 while bruno is at sequence 3). fedcrawl writes time.Now() into bodies (Exported:, Indexed:), which defeats the unchanged-body skip in bucketstore/write.go. Root only takes graph writes, so this blocks no one, but it is wasted commits and version bloat. Check whether the incremental federation graph removes it before fixing it separately.

Open questions

Target write throughput and p99 latency per world for the phase 0 gate.
Retention for log slots and the compaction cadence.
One-way migration or a shipped rollback tool.
Callers of hash lookup: confirm none needs path-read exactness before it joins the 1 s class.
When an S3 backend lands; the design needs only create-if-absent and strong read and not-found, which S3 has since 2024.

Status

Proposed 2026-10-01, decisions recorded the same day. Not started; picked up after the incremental federation graph work on feat/incremental-federation is done.

Bucket store commit log: create-only commits instead of one head CAS

Problem

ADR 0008 stated the assumption (read-heavy, under one mutation per second per world) and measured it, but it was never revisited when concurrent multi-user writing became the goal.

Decisions

Taken 2026-10-01 with the maintainer.

Bucket only. GCS (and later S3) stays the only state. A database for metadata, such as Cloud SQL Postgres beside GCS, is out; ADR 0010 stands.
Ordering stays per world. WATCH cursors and the change feed are numbered per world (ADR 0025, ADR 0028), and the federation graph is derived from that sequence. The world is the narrowest scope that can own it.
Search may lag by up to one second. LOOKUP, LIST and hash lookup on one replica may trail another replica's commit by at most 1 s. FETCH, VERSIONS and chain verification by path stay exact. Every write evaluates its preconditions at the tip, so the lag never affects write correctness. This relaxes ADR 0008's immediate LOOKUP and hash freshness.

Concurrency model stays the same

Alternatives considered

Group commit on head.json. Coalesce queued mutations into one head replace, committing at once when idle and batching only under load. Throughput becomes batch size per 1.5 s and the change is smaller, but writes under load take 1.5 to 3 s, the head stays a hot object whose 429s delay a whole batch, several writer pods race whole batches, the 16-receipt window must scale with batch size, and ADR 0031's blocks of 8 sequences would be skipped by one large batch. Kept only as an optional later phase on top of the log.
In-memory head shared between replicas, flushed every 2 s. Acknowledging before the flush loses up to 2 s of acknowledged writes when the holder crashes, and a deposed holder acknowledges writes its flush then loses. Making it safe means quorum replication (Raft), which makes GCS a snapshot store and does not survive all pods restarting together. Acknowledging after the flush is the group commit above.
Database for metadata. Out by the bucket-only decision.
Per-document conditional writes alone. Correct for versions, but there is no world sequence for WATCH and the feed, two creates on different objects can break topology, and a commit followed by a separate feed entry can crash between the two and never be indexed.
One head per shard or subtree. A parent and child path hash to different shards, and the change sequence is per world.
Libraries (surveyed 2026-10-01). None fits: mbrt/glassdb is alpha, GCS only, has no change feed or global index; JayJamieson/objwal is single writer and orders through one rewritten manifest; SlateDB is Rust with Go bindings, single writer with fencing, a key-value model; rivian/delta-go is Parquet tables, alpha, S3 with DynamoDB locking, no GCS. Useful as design references only: Delta Lake's put-if-absent log and logical conflict check on retry, SlateDB's put-if-absent commit and writer fencing.

Design

Log slots are the commit point

Write path

Pin the snapshot at the tip T. Writes always build at the tip, never on a lagging search snapshot.
Canonicalize, authorize, evaluate policy, topology and protocol preconditions against it, as today.
Stage blob, history, manifest, changed shard and root as immutable objects, as today.
Create log/T+1. Success installs the candidate on this replica and acknowledges.
If the slot exists, load the slots up to the new tip. If any of them changed the target path, an ancestor or descendant (topology), or the world's publish policy, re-run the full build so expected-version, topology and policy checks answer as today. Otherwise rebase: keep blob, history and manifest, rebuild only the changed shard and the root over the new tip, and try the next slot.
Bound attempts by the request deadline; no fixed pacing on the commit path.

Because the topology check runs at the tip inside the slot race, no path rule or protocol change is needed for it.

Ambiguous outcomes

Read path

Path reads (FETCH, VERSIONS, chain verification) check log/<tip+1> (metadata only) before serving, as today's head validation does. Not found means the snapshot is current; GCS and S3 both give strong read-after-write and strong not-found. Found means reading forward until not found, then loading the changed shards.
Search reads (LOOKUP, LIST, hash lookup) serve the replica's snapshot if it was confirmed current within the last second, and check the tip first otherwise. That bounds the lag at 1 s and costs at most one tip check per second per replica, however heavy the search traffic.
A replica installs its own commits as they succeed, so a writer reading through the replica it wrote to sees its writes at once.
Optionally, the hub's fan-out pushes new slot numbers to the other replicas so they usually catch up well inside the bound.
One request still pins one snapshot.

Tip hint

Change feed

Garbage collection barrier

Rates

Migration

A new replica that finds a schema 1 head.json and no log creates log/<head sequence> from the head's exact fields (deterministic bytes, so concurrent replicas converge through create-if-absent), then CAS-replaces head.json with a schema 2 hint.
Heads are decoded with unknown fields refused, so an old replica reading a schema 2 head fails closed with server-error and cannot commit during a rolling deploy. No split brain.
Rollback needs a tool that writes a schema 1 head from the log tip; decide whether to ship it or document the migration as one way.

Invariants

A slot is created once and never rewritten; slot N+1 is only attempted after slot N was observed, so the log has no gaps, and each entry names its predecessor's hash.
Each acknowledged mutation is in exactly one slot; acknowledgement follows only a successful create or a reread that finds this operation's ID in the slot.
Every write's preconditions are evaluated at the tip; path reads are exact; search reads serve a snapshot confirmed current within 1 s.
One request reads one snapshot.
Expected-version, topology, policy and archive semantics are unchanged; the single-store conformance suite passes without edits, and cross-replica freshness tests move to the 1 s bound for search.
New long-lived state (the slot LRU, any rebase cache) is bounded, evicts, and ships with a memtest.Retained test on production-shaped data.

Phases

Spike on the benchmarks branch against real GCS. Slot-create commit latency p50 and p99, sequential rate, 4, 16 and 64 concurrent writers on one pod, two pods racing, 429 and 5xx counts, with the current head CAS as baseline. Record in /research/. Gate targets to set before running (open question below).
ADR (next free number). Replaces ADR 0008's commit point and its immediate search freshness, and ADR 0031's sealing; keeps ADR 0008's immutable root, shard, manifest, history and blob model and records the bucket-only and 1 s decisions.
Commit and read path in bucketstore behind the schema 2 format, with tests: slot race, ambiguous create, rebase without conflict, same-path conflict, parent and child topology race, policy change during a race, sweep barrier, chain verification, path reads exact across replicas, search lag bounded at 1 s.
Change feed over slots, compaction into blocks, retention for old slots.
Migration and the rollback decision; re-run the kind write scenarios: the 4-writer burst of 200 must lose none.
Group commit, only if phase 0 or 4 falls short: coalesce the commitToken queue into one slot holding several receipts.

Open questions

Target write throughput and p99 latency per world for the phase 0 gate.
Retention for log slots and the compaction cadence.
One-way migration or a shipped rollback tool.
Callers of hash lookup: confirm none needs path-read exactness before it joins the 1 s class.
When an S3 backend lands; the design needs only create-if-absent and strong read and not-found, which S3 has since 2024.

Status

Proposed 2026-10-01, decisions recorded the same day. Not started; picked up after the incremental federation graph work on feat/incremental-federation is done.

Bucket store commit log: create-only commits instead of one head CAS

Problem

Decisions

Concurrency model stays the same

Alternatives considered

Design

Log slots are the commit point

Write path

Ambiguous outcomes

Read path

Tip hint

Change feed

Garbage collection barrier

Rates

Migration

Invariants

Phases

Related, separate

Open questions

Status