# Multi-world Knowledge Server

Audience: demarkus contributors and enterprise deployment operators familiar with the Mark Protocol, brokered knowledge systems, and Kubernetes.

Build a bucket-backed `demarkus-knowledge-server` that hosts multiple logically isolated worlds in one replicated process while every caller continues to observe ordinary standalone world-server behavior.

Status: planned 2026-08-21. Implementation has not started.

## Goal

A single knowledge-server Deployment hosts `world-a`, `world-b`, and `world-c` behind one UDP port. The broker, agent, and direct clients address each world independently. Each world keeps separate documents, versions, hashes, catalog, capability tokens, broker access rules, publish policy, runtime limits, logs, backup, and restore boundary.

The target reduces enterprise deployment complexity and enables high availability without changing the Mark request format or duplicating the existing server implementation.

## Locked Decisions

| Area | Decision |
|---|---|
| Runtime | One process per pod, hosting every configured world |
| Availability | Two or more identical replicas behind one UDP Service |
| Routing | Distinct DNS authorities and TLS SNI on shared UDP port 6309 |
| Storage | GCS first, one bucket per world |
| Isolation | Logical isolation inside one enterprise trust boundary |
| Lifecycle | Static strict config with rolling updates for world additions and removals |
| Policy | Versioned per-world policy document, enforced by plugins and server |
| Scale | Prove 100,000 documents and less than one committed write per second per world |
| Scope | Replace world-server workloads only; broker and agent remain separate |

## Explicit Isolation Boundary

The knowledge server preserves request, data, authorization, policy, and operational state boundaries. It does not claim hostile-tenant isolation.

| Preserved per world | Shared by design |
|---|---|
| GCS bucket and storage snapshot | Process and address space |
| Namespace, versions, hashes, and catalog | Pod service account with access to all configured buckets |
| Capability-token store | UDP listener and listener-level QUIC limits |
| Broker read and write ACLs | Pod readiness and crash domain |
| Publish policy | Hard CPU and memory limits |
| Application rate limiter and request semaphore | TLS infrastructure |
| Read-only state, caches, and audit fields | Deployment rollout |

A process compromise, panic, memory leak, or exhausted file-descriptor limit can affect every world. A world requiring hard process, IAM, Secret, or resource isolation must remain a standalone deployment.

## Compatibility Contract

The Mark wire request remains `{Verb, Path, Metadata, Body}`. No world field, path prefix, token-derived selector, or new ALPN is added.

Direct clients continue to call URLs such as `mark://world-a.example.com/docs/a.md`. Broker tool URLs remain `mark://world-a/docs/a.md`. World identity remains the logical authority, not the shared dial address, consistent with [ADR 0005](/adr/0005-node-identity-default-port.md).

The TLS client sends the world authority as SNI. The knowledge server selects one world during the handshake and pins that QUIC connection to the selected runtime. Empty, unknown, wildcard, trailing-dot, and IP-literal SNI values fail closed. A connection can never switch worlds between streams.

The L4 load balancer passes QUIC and TLS through without terminating either protocol. Every pod contains every configured world, so any Service endpoint can accept any valid world authority.

## Broker View

The broker continues to model each world as a separate `WorldConfig` and keep one client pool per world.

```text
mark://world-a/doc -> world-a DNS alias -> shared IP:6309 -> SNI world-a -> runtime A
mark://world-b/doc -> world-b DNS alias -> shared IP:6309 -> SNI world-b -> runtime B
mark://world-c/doc -> world-c DNS alias -> shared IP:6309 -> SNI world-c -> runtime C
```

The preferred first deployment gives each world a unique internal DNS alias that resolves to the shared Service. Existing clients then derive the correct SNI from the dial hostname without a wire change. The same IP and port may be shared; the hostname must remain distinct.

If an environment cannot provide per-world internal aliases, the client transport gains an explicit endpoint shape that separates logical authority, dial address, and TLS server name. Connection-pool keys must include the dial address and server name together.

`mark_worlds` continues to list each logical world independently. Graph, token, cache, and crawler identity must use the logical authority, never the shared Service address.

## Code Reuse

The implementation is multiple instances of one extracted server runtime, not a fork of the server.

```text
demarkus-server           -> OpenWorld(single config)
demarkus-knowledge-server -> OpenWorld(config A/B/C) plus SNI registry
```

The shared runtime reuses the protocol parser and response writer, handlers, capability authorization, content validation, hash-chain format, lookup interfaces, logging, rate limiting, token reload, migration contract, and conformance tests.

New code is limited to the GCS blob driver and bucket store, request snapshots, world runtime lifecycle, SNI registry, multi-world config, aggregate health, and deployment packaging. Changes to verb behavior remain shared so standalone and knowledge servers cannot drift.

The GCS dependency follows [ADR 0006](/adr/0006-postgres-backend-is-an-optional-build.md): the default `demarkus-server` binary must not link the GCS SDK.

## Runtime Model

Each `WorldRuntime` owns:

- Name and exact SNI authorities.
- Bucket store and immutable snapshot cache.
- Lookup catalog view and body-hash index view.
- Atomic token store and token-file watcher.
- Parsed publish policy.
- Handler instance.
- Read-only state.
- Per-world rate limiter and concurrent-request semaphore.
- Logger pre-bound with world and authority fields.
- Close and drain lifecycle.

The process owns:

- One QUIC listener.
- Listener-level stream and idle limits.
- Immutable SNI-to-runtime registry.
- TLS certificate selection and reload.
- Internal liveness and aggregate readiness server.
- Global emergency concurrency and memory guards.
- Graceful shutdown and signal handling.

Startup validates and opens every world before serving traffic. Any invalid world prevents readiness. Configuration is immutable for the process lifetime; adding or removing a world uses a rolling update. Token and certificate contents retain last-known-good hot reload behavior.

## Knowledge Server Configuration

Use one strict YAML file with unknown-field rejection and an explicit schema version. Do not expand the single-world environment-variable surface into indexed per-world variables.

```yaml
version: 1

listen:
  address: ":6309"
  maxIncomingStreams: 128
  idleTimeout: 30s

health:
  address: ":8081"

tls:
  certFile: /run/demarkus/tls/tls.crt
  keyFile: /run/demarkus/tls/tls.key

worlds:
  - name: world-a
    authorities:
      - world-a.example.com
      - world-a.knowledge.svc.cluster.local
    bucket:
      url: gs://deployment-world-a
      worldID: 52b471f7-8d38-4c89-b44a-6f4f8b1a4f48
    auth:
      tokensFile: /run/demarkus/world-a/tokens.toml
    policy:
      path: /.well-known/demarkus/policy.md
    readOnly: false
    limits:
      maxConcurrentRequests: 32
      requestTimeout: 10s
      requestsPerSecond: 50
      burst: 100
```

Validation rejects duplicate world names, normalized authorities, bucket identities, token files, or token hashes across worlds. It also rejects certificate SAN gaps, invalid bucket URLs, missing policy documents, unsupported limits, and any world whose bucket marker does not match its configured world ID.

## Automatic World Provisioning

Adding a world should be one declarative change to the deployment repository's canonical `worlds[]` list. The deployment layer, not the server process, performs provisioning.

The generated workflow creates:

- A globally unique GCS bucket with the standard protection settings.
- Bucket-scoped access for the shared knowledge-server workload identity.
- A unique immutable world ID and bucket marker.
- A per-world capability-token Secret.
- An initial `/.well-known/demarkus/policy.md` document.
- Internal and public DNS aliases covered by the configured certificate.
- Knowledge-server, broker, and agent configuration entries.
- A rolling knowledge-server update before the new alias receives traffic.

The knowledge server never creates or deletes buckets at runtime. GCS bucket names are globally scarce, IAM propagation is not immediate, and storage lifecycle changes do not belong in the request path. The server validates provisioned resources and fails closed on mismatch.

## GCS Storage Decision

One bucket per world is required for the first release. Prefixes in one shared bucket weaken lifecycle, restore, accidental-deletion, accounting, and operator boundaries without reducing the process-level IAM blast radius that has already been accepted.

The earlier [bucket backend plan](/plans/bucket-store.md) is superseded by this plan. Its reusable findings remain valid: native SDK access, conditional writes, write-once stored-version bytes, byte-identical migration, and no gcsfuse. Two accepted limitations are no longer acceptable:

- Concurrent document and descendant creation could violate path topology.
- Per-pod 30-second catalog refresh could return stale LOOKUP and hash misses.

Backend parity requires immediate cross-replica protocol behavior and the same document-versus-directory invariants as the file and Postgres stores.

## GCS Object Layout

```text
_demarkus/v1/head.json
_demarkus/v1/roots/<root-hash>.json
_demarkus/v1/index/<00..ff>/<shard-hash>.json
_demarkus/v1/docs/<path-hash>/manifests/<manifest-hash>.json
_demarkus/v1/history/<history-hash>.json
_demarkus/v1/blobs/<stored-bytes-hash>
_demarkus/v1/pins/<backup-id>.json
```

`head.json` is the only mutable object. It contains the schema version, immutable world ID, monotonic sequence, root key, root hash, and recent operation receipts. GCS generation preconditions protect every replacement.

A root is immutable and references 256 immutable namespace/catalog shards. A shard is selected by the first byte of the canonical path hash and contains current entries for those paths. An entry carries the canonical path, manifest reference, current version, archive state, current body hash, modified time, and complete lookup catalog entry.

A document manifest is immutable and references retained history chunks. History chunks contain version number, raw stored-byte blob key, body hash, and modified time. Blobs contain exact `store.SerializeVersion` bytes and are create-only. Raw document paths never become authoritative GCS object keys, avoiding the 1024-byte object-name limit.

The root snapshot also derives the live path trie, hash-to-path map, and lookup catalog. Archived documents continue to reserve namespace topology but do not appear in live hash or lookup results.

## Read Protocol

Every protocol request performs a strong authenticated read or conditional validation of `head.json`. There is no time-based freshness window.

If the head generation is unchanged, the request reuses cached immutable state. If it changed, the server fetches the new root, compares shard hashes, fetches changed shards only, rebuilds derived indexes, and atomically installs the snapshot.

One request pins one root snapshot. FETCH, LIST, IsDir, VERSIONS, chain verification, hash resolution, LOOKUP, and policy evaluation cannot mix two commits. Negative caches are scoped to one root hash.

If GCS cannot validate the head, the request returns `server-error`. Serving a stale snapshot during an outage would violate the selected immediate-consistency contract.

## Write Protocol

The mutable head generation is the sole linearization point for every world mutation.

1. Strongly read the current head and immutable root snapshot.
2. Canonicalize the path before authorization, topology checks, or storage lookup.
3. Authorize the selected world's capability token.
4. Load and evaluate the selected world's policy from the same snapshot.
5. Revalidate archive, deduplication, expected-version, metadata, retention, and document-versus-directory rules.
6. Create the stored-byte blob, history chunk, manifest, changed index shard, and candidate root as immutable objects.
7. Replace `head.json` with the prior generation as a precondition.
8. Return success only after the CAS succeeds or reconciliation proves the operation receipt committed.

A concurrent write may stage unreachable immutable objects. Only the winner's root becomes visible. A losing writer reloads the head and revalidates. An unrelated path change may rebase and retry; a target-document change returns the normal protocol conflict.

Concurrent `/a.md` and `/a.md/b.md` creation cannot both commit. The first head CAS wins; the loser reloads the namespace and fails the topology check.

GCS limits replacement of one object name to approximately one write per second. The supported first-release envelope is therefore less than one committed mutation per second per world. Bursts receive bounded retry with jitter. If enterprise workloads need sustained higher write rates, Postgres or a different coordinator is required; consistency will not be weakened to hide the limit.

## Ambiguous Write Outcomes

Every internal mutation receives one operation UUID. Candidate roots retain a bounded recent receipt set containing the operation UUID and result.

After a timeout or lost CAS response, the writer rereads the head. A matching receipt proves success. A newer head without the receipt proves the old generation can no longer commit and permits rebase. An unchanged head permits retrying the identical candidate CAS.

If the head cannot be read before the request deadline, return `server-error` and log the operation UUID. Never claim success from local state alone.

Client-level exactly-once replay across a new request remains impossible without a protocol idempotency key. Existing expected-version requirements prevent blind APPEND replay.

## Publish Policy

Each world carries a versioned `/.well-known/demarkus/policy.md`, following today's knowledge-system model. New deployments seed every world from the current root policy, after which policies may diverge.

Plugins continue to mirror and enforce policy before dispatch for immediate user guidance. They must key cached policy by knowledge system and target world. The root policy is a compatibility fallback only while old worlds are backfilled.

The knowledge server parses and enforces the same deterministic policy core so non-plugin clients cannot bypass it. The parser and evaluator must be shared code, not duplicated implementations.

Server enforcement covers required tags, required tag axes, required metadata fields, importance range, and any other mechanically defined policy keys. `block` rejects without mutation. `warn` permits the mutation and emits response metadata plus an audit event. Noninteractive `ask` fails closed until a protocol-safe approval mechanism exists.

A policy-document update must parse successfully before commit. The new policy and its root generation become active atomically. A write rebased after a head conflict must reevaluate against the new policy before retrying.

Style judgment, secrets and PII detection, usefulness, and human curation remain agent responsibilities unless the policy format gains explicit deterministic settings for them.

## Capability Tokens and Broker ACLs

Capability tokens remain world-local. Each runtime loads a distinct token Secret, and every handler callback closes over only that runtime's atomic token store. Duplicate token hashes across worlds fail startup to catch accidental credential reuse.

Broker identity authorization becomes explicit per world and per operation:

```yaml
access:
  read:
    mode: allowlist
    allow:
      groups: [knowledge-readers]
  write:
    mode: allowlist
    allow:
      groups: [knowledge-writers]
```

Supported modes are `allAuthenticated`, `allowlist`, and `none`. Write access implies read access. `mark_worlds`, installation responses, tools, prompts, resources, graph operations, and federation operations filter inaccessible worlds before dispatch.

Private worlds receive separate broker-held read and publish capability tokens. Broker reads no longer assume an empty token, and helper reads for merge or append do not misuse a publish-only token. Direct access remains governed by server capabilities because the server intentionally knows no user identity.

Authorization uses one centralized broker facade so no tool can bypass the read or write predicate. OIDC hosted-domain checks must apply identically to management and MCP bearer middleware.

## Runtime Limits

Per-world limits cover application request rate, burst, concurrent handlers, request timeout, body and metadata tightening, read-only mode, and bucket-cache budget.

QUIC idle timeout, handshake timeout, connection flow windows, and maximum incoming streams are listener-level settings because world selection occurs during the TLS handshake. The configuration must not expose per-world knobs that cannot be enforced.

Per-world limits are logical fairness controls. They are not cgroup or Kubernetes resource guarantees. A global emergency limiter protects the process from aggregate exhaustion.

## TLS and Certificates

The first release uses one wildcard or multi-SAN certificate covering every configured internal and public authority. The SNI registry still maps exact hostnames; wildcard routing entries are forbidden.

Certificate startup and reload validate the time window and hostname coverage for every configured authority. A malformed or incomplete reload retains the previous certificate and logs the failure. Dev self-signed mode is not allowed for production multi-world configuration.

Private internal DNS names require an internal CA or split-horizon public authorities. Broker and agent verification must use a CA bundle; `insecureSkipVerify` remains development-only.

## Health and High Availability

A private TCP management listener exposes:

- `/livez`: process and QUIC listener are alive; no GCS dependency.
- `/readyz`: TLS, every world runtime, initial head snapshot, token store, and policy are valid.

The management port is never exposed by the public Service. Aggregate readiness is intentional because every Service endpoint must serve every world. One broken runtime removes the pod from all world traffic; this is part of the accepted shared failure domain.

The Helm chart deploys at least two replicas, a PodDisruptionBudget, topology spread across nodes and zones, graceful termination, and one UDP Service. NetworkPolicy permits broker and agent UDP 6309 egress to the knowledge-server namespace and denies direct pod access from other namespaces by default.

Kubernetes L4 Services do not provide QUIC connection-ID routing across pod migration. A migrated connection may fail and redial. Existing client retry behavior covers availability, but seamless connection migration is not promised.

## Bucket Protection and Backup

Buckets are pre-created with uniform bucket-level access, public-access prevention, `force_destroy=false`, deletion protection in infrastructure code, and seven-day soft delete. Application versioning, immutable roots, and soft delete make GCS Object Versioning unnecessary initially.

The knowledge-server workload identity receives object access on every configured world bucket and no project-wide storage role. This scopes accidental infrastructure grants but does not create an in-process IAM boundary.

Backup pins one committed root, copies every reachable immutable object to an independent bucket or project, verifies hashes, then writes the destination backup manifest last. A root pointer in the source bucket is a snapshot, not an independent backup.

GC performs mark and sweep from the current root, retained prior roots, active migration roots, and backup pins. It deletes only unmarked objects older than the grace period using generation-matched deletes. Any incomplete mark, list, or read aborts the sweep. Logical retention takes effect at commit; physical deletion remains deferred.

## Migration

The existing `store.Migrator` contract remains the byte-equality foundation. Bucket migration adds a bulk-import session because importing 100,000 documents through one head CAS per document would take more than a day at the supported write rate.

A bulk import requires an empty destination bucket with no head. It stages all stored-byte blobs, histories, manifests, and 256 index shards, verifies them, writes one initial root, then creates `head.json` once with create-only preconditions. A failed import leaves no visible world and may be retried after GC.

File-to-bucket, Postgres-to-bucket, bucket-to-file, and bucket-to-Postgres round trips must preserve document paths, version numbers, exact stored bytes, modified timestamps to protocol precision, archive state, retention suffixes, hashes, and lookup metadata.

## Implementation Sequence

Each numbered step is an independently reviewable change. The conformance suite and `bash pre-commit.sh` stay green after every step.

1. Write repo ADRs for SNI virtual worlds and GCS root-CAS storage; amend TLS and authority rules in `docs/SPEC.md`.
2. Run a real-GCS feasibility spike at 100,000 documents and three worlds; measure cold load, warm head validation, LOOKUP latency, memory, write throttling, and retry behavior.
3. Fix prerequisites: raw stored-byte ETags, canonical-path authorization, directory index authorization, LIST filtering, and error-reporting hash lookups.
4. Add request-scoped store views so all handler operations use one committed snapshot.
5. Extract the current policy parser and evaluator into one shared package; preserve plugin behavior and add optional server enforcement.
6. Add generation-aware blob interfaces, deterministic in-memory driver, GCS driver, and driver conformance tests.
7. Implement immutable roots, 256 index shards, manifests, history chunks, blobs, snapshot cache, write CAS, and failure injection in `server/internal/bucketstore/`.
8. Add bulk migration, verification, reverse migration, backup pins, and GC to the existing migration surface and server tooling.
9. Extract reusable QUIC serving and `WorldRuntime` lifecycle from `server/cmd/demarkus-server`; keep existing single-world flags and behavior unchanged.
10. Add strict multi-world config, SNI router, aggregate health, and `server/cmd/demarkus-knowledge-server` as a bucket-only build.
11. Add explicit broker read and write ACLs, separate read and publish capabilities, centralized dispatch authorization, policy-aware world filtering, and UDP NetworkPolicy.
12. Add per-world policy discovery to knowledge plugins and logical-authority versus dial-address support to broker and agent where DNS aliases are unavailable.
13. Add the knowledge-server build target, separate image, release artifacts, CI matrix, and dedicated `deploy/helm/demarkus-knowledge-server/` chart.
14. Extend the deployment repository's canonical `worlds[]` flow to provision buckets, IAM, Secrets, policies, DNS, broker entries, agent seeds, and rollout ordering.
15. Migrate a non-root pilot world, run the complete HA and restore drills, then migrate remaining worlds with root last.

## Verification Gates

The feature is not complete until every gate passes:

- Existing `storetest.RunConformance` and `RunLookupConformance` pass unchanged.
- Two independent bucket-store instances share only GCS and observe every mutation immediately without sleeps or polling.
- Same path, body hash, and version number in different worlds remain independent.
- A token accepted by one world is rejected by every other world.
- Unknown or absent SNI cannot reach a default runtime.
- Every stream on one QUIC connection remains pinned to one world.
- Concurrent same-path, unrelated-path, archive/write, retention/write, and document/descendant races preserve all invariants.
- Failure injection before every staged object and before, during, and after head CAS produces either one complete commit or no visible mutation.
- GCS 412, 429, timeout, and 5xx handling never returns false success or false not-found.
- Policy updates and concurrent writes use one atomic policy snapshot.
- Broker tools, resources, prompts, graph operations, and `mark_worlds` deny inaccessible worlds before dispatch.
- Three 100,000-document worlds stay inside the approved memory and latency budgets from the feasibility spike.
- A two-replica end-to-end test covers broker dispatch, direct SNI calls, agent crawl, pod kill during write, rolling update, backup, restore, and rollback.
- `make test`, real-GCS gated tests, race tests, Helm tests, the multi-world bucket end-to-end script, and `bash pre-commit.sh` pass.

## Deployment Rollout

1. Provision per-world buckets, world IDs, workload identity grants, token Secrets, policy documents, certificate, and DNS aliases.
2. Deploy the knowledge server on a canary Service and keep broker and agent routed to old worlds.
3. Freeze old writers, stop agent publication, and snapshot source storage.
4. Bulk-migrate worlds in parallel and verify byte-identical exports, chain validity, catalog contents, and policy documents.
5. Start the new server read-only and validate every SNI route directly and through the broker.
6. Switch broker internal aliases and agent seeds, enable new writes, then move public DNS aliases to the shared endpoint.
7. Stop old servers to force pooled clients to redial; DNS TTL alone does not move existing QUIC connections.
8. Retain old storage and snapshots through the rollback and restore-verification window.

## Rollback

Before the first new write, rollback reverses broker, agent, and DNS routing to the old read-only worlds.

After the new server accepts writes, rollback freezes it and bulk-migrates each bucket into a fresh empty file or Postgres store. The original source storage is stale and must never be reactivated directly.

## Go or No-Go Conditions

Proceed from the feasibility spike only if pure GCS can meet the agreed 100,000-document read, memory, and less-than-one-write-per-second envelope while validating the head on every request.

Stop and use Postgres as the coordinator if the global head limit, snapshot memory, cold-load time, or strong-read latency is unacceptable. Do not weaken path topology, LOOKUP freshness, hash freshness, or policy atomicity to make bucket storage appear viable.

## Related Documents

- [Architecture](/architecture.md)
- [Superseded bucket backend plan](/plans/bucket-store.md)
- [Store parity](/plans/store-parity.md)
- [Universe deployment](/plans/universe-deployment.md)
- [Knowledge system GKE deployment](/plans/knowledge-system-gke-deploy.md)
- [ADR 0005: node identity](/adr/0005-node-identity-default-port.md)
- [ADR 0006: optional Postgres build](/adr/0006-postgres-backend-is-an-optional-build.md)