ADR 0008: GCS world state commits through one root CAS

Status: proposed (2026-08-21). Acceptance depends on store conformance, multi-replica correctness, and measured sizing at 100,000 documents in each of three worlds.

Context

The multi-world knowledge server needs durable storage shared by two or more replicas. GCS provides strongly consistent object reads and generation preconditions, but it does not provide transactions across object names.

Per-document conditional writes are insufficient. Concurrent creation of /a.md and /a.md/b.md can otherwise violate the document-versus-directory invariant. Periodically rebuilt per-pod hash and LOOKUP indexes can also return stale negative answers after another replica commits a write. Both outcomes violate backend parity.

The initial workload is read-heavy: up to 100,000 documents and less than one committed mutation per second per world. That permits one world-level linearization point if reads can reuse immutable state efficiently.

Decision

One bucket and one commit point per world

Each world uses its own GCS bucket and has an immutable random world ID. The bucket name is deployment configuration, not world identity. Moving or restoring a world preserves its world ID.

_demarkus/v1/head.json is the only mutable object and the sole linearization point for every world mutation. It records the schema version, world ID, monotonic sequence, immutable root key and hash, and a bounded set of recent operation receipts. Every replacement uses the previously observed GCS object generation as a precondition.

All referenced state is immutable:

_demarkus/v1/head.json
_demarkus/v1/roots/<root-hash>.json
_demarkus/v1/index/<00..ff>/<shard-hash>.json
_demarkus/v1/docs/<path-hash>/manifests/<manifest-hash>.json
_demarkus/v1/history/<history-hash>.json
_demarkus/v1/blobs/<stored-bytes-hash>
_demarkus/v1/pins/<backup-id>.json

A root references 256 namespace and catalog shards selected by the first byte of the canonical path hash. Each entry contains the canonical path, manifest reference, current version, archive state, current body hash, modified time, and complete LOOKUP catalog entry. Archived documents continue to reserve namespace topology but are absent from live body-hash and catalog indexes.

Document manifests reference retained history chunks. History entries contain the version number, exact stored-version blob reference, body hash, and modified time. Raw document paths are never authoritative object names.

Immutable data has deterministic identities

Immutable JSON uses UTF-8, no insignificant whitespace, fixed schema field order, lexicographically sorted map keys, schema-defined array order, and no floating-point values. Its identity is lowercase hexadecimal SHA-256 of those exact bytes. Readers recompute the hash and verify both the object key and every parent reference before accepting an object.

Blob objects contain the exact bytes produced by store.SerializeVersion and are create-only. Their key is the SHA-256 of those bytes. A protocol ETag is the same raw stored-byte SHA-256 rendered without the sha256- prefix. GCS generation, MD5, CRC32C, root hashes, manifest hashes, and body hashes are never used as protocol ETags.

Every request uses one validated snapshot

Each protocol request performs a strong read or conditional validation of head.json. There is no time-based freshness window. An unchanged generation reuses cached immutable state. A changed generation loads the new root and only the shards whose hashes changed, rebuilds derived indexes, and atomically installs a new snapshot.

One request pins one root snapshot for path lookup, FETCH, LIST, VERSIONS, chain verification, hash resolution, LOOKUP, and publish-policy evaluation. A failure to validate the current head returns server-error; replicas never serve a known-stale snapshot during an outage.

Writes stage immutable state, then replace the head

Every mutation follows this sequence:

1. Read and pin the current head and root snapshot.
2. Canonicalize and authorize the path against the selected world.
3. Evaluate policy and all protocol and namespace preconditions against that snapshot.
4. Create the blob, history, manifest, changed shard, and root as immutable objects.
5. Replace head.json with an if-generation-match precondition.

A successful head replacement commits the entire candidate. A failed replacement exposes none of it. The loser reloads the head and revalidates all conditions before rebasing; a target-document change returns the normal Mark conflict. Staged objects that lost a race remain unreachable until garbage collection.

Each mutation has an operation UUID. Candidate heads retain a bounded receipt set containing operation UUIDs and results. After a timeout or lost response, a writer rereads the head: a receipt proves success, a newer head without the receipt proves the old generation cannot commit, and an unchanged head permits retry of the identical candidate. If reconciliation cannot read the head before the deadline, the server returns server-error and logs the operation UUID.

Migration, backup, and reclamation follow the root graph

Bulk import creates all immutable state and publishes one initial root and head, rather than committing once per document. Verification compares exact stored bytes, versions, chains, archive state, catalog state, and namespace topology.

Backups pin an immutable root under _demarkus/v1/pins/ before copying or retaining reachable objects. Garbage collection marks from the current root, retained prior roots, active import roots, and backup pins. It sweeps only after confirming the head generation is unchanged. Logical retention takes effect in the committed manifest immediately; physical deletion occurs later and never changes protocol-visible history.

Isolation boundary

One bucket per world provides independent lifecycle, restore, accounting, and accidental-deletion boundaries. Per-world tokens, policies, caches, limits, and logs remain separate in memory. The process, workload identity, resource limits, readiness, and crash domain are shared; this design is logical isolation within one enterprise trust boundary, not hostile multi-tenancy.

Consequences

• Every visible mutation is globally ordered per world, including unrelated paths and policy changes.
• Two replicas sharing only GCS observe the same head on the next request without sleeps, polling intervals, or replica-local negative-cache staleness.
• Concurrent document and descendant creation cannot both commit.
• Read cost includes one strong head validation per request; changed roots load only changed shards.
• One object name limits sustained commits to roughly one per second per world. The GCS implementation must reconcile ambiguous outcomes and apply bounded retry with jitter; deployments must remain inside the measured write envelope.
• Failed writes may leave unreachable immutable objects, so safe garbage collection is required.
• The default filesystem server remains free of the GCS SDK. The knowledge server is a separate binary and image, consistent with ADR 0006.

Sizing validation

Before production implementation proceeds, a real-GCS spike must measure three 100,000-document worlds: cold load, warm head validation, LOOKUP latency, memory, write throttling, conditional-write retries, and cleanup cost.

GCS is the required backend behind the same server storage interfaces as the filesystem store. Measurements establish resource requests, cache shape, concurrency, retry policy, and the supported write envelope. An unacceptable measurement requires optimizing the GCS design or narrowing its documented capacity, not adding a Postgres coordinator. Path topology, immediate hash and LOOKUP freshness, policy atomicity, and one-request snapshot consistency remain non-negotiable.

Alternatives rejected

• Per-document CAS. It cannot atomically enforce cross-path topology or update all derived indexes.
• Periodic catalog refresh. It returns stale LOOKUP and hash misses across replicas.
• One shared bucket with world prefixes. It weakens lifecycle and restore boundaries without reducing the accepted process-level IAM blast radius.
• gcsfuse. Filesystem emulation does not supply the required transaction and snapshot semantics.
• Postgres as a coordinator. It would split one storage backend across two systems and make GCS subordinate to a database. The GCS store implements the existing server storage contract directly.
• Serve cached state when head validation fails. This violates the selected immediate-consistency contract and can return false not-found results.