# File worlds in the knowledge server

First move of step 7 of [Memory and knowledge layering](/plans/memory-knowledge-layering.md), under the [delivery](/plans/memory-knowledge-layering/delivery.md) bar, and the "self-host without GCS" item of [The engine, not the wire](/plans/engine-not-wire.md#4-fortify-the-knowledge-tier). For demarkus maintainers: how `knowledgeserver` opens a world from a directory with the file store the soul already runs, what the config looks like, what a file world cannot do, the tests, and the two PRs. Drafted 2026-10-19 from main at `bb24d871` (server 0.70.0). The shape it serves is [ADR 0037](/adr/0037-contract-is-the-protocol-two-doors.md) decision 6: one runtime, two thin mains; this is the half that gives the knowledge path a file backend, so the thin `local` main can follow.

## Today

| | Soul (`cmd/demarkus-server`) | Knowledge (`knowledgeserver`) |
|---|---|---|
| Store | `store.New(dir)`, `BuildIndex`, `catalog`, `filestore.Open` with a change ring: `openFileStore` in `store.go`, with `buildCatalog` and `logPartialWalk` in `main.go` | `bucketstore.Open` over a `blob.Store`; the only `storeFactory`, `bucketStores` in `run.go` |
| World identity | none (one world at the root) | `Bucket.WorldID`, the key peer hints carry and the WATCH cursor epoch |
| Config | `config.Config` (`-store file`, `ContentDir`) | `knowledgeconfig.WorldConfig` with a required `bucket:` section; `validateWorlds` requires `bucket.url` and `bucket.worldID` on every world |
| GCS client | none | `Server.open` builds `storage.NewClient` whenever `Options.Buckets` is nil, before any world opens |
| Runtime | `worldruntime.New` wired by hand in `main.go` | `worldManager.serve`: policy seed, style guide on the floor world, `writepolicy.Enforce`, `worldruntime.New` |

`filestore.Store` already satisfies `worldStore` (`backend.Store`, `backend.ChangeSource`, `io.Closer`) and passes the same storetest conformance, lookup, handler and differential suites `bucketstore` does. Nothing in the runtime knows which store it holds. The gap is wiring and config.

## The change

1. **One file-store constructor.** `filestore.OpenDir(dir string, opts Options) (*Store, error)`: `store.New`, `BuildIndex` (a partial walk logged per entry, a broken walk an error), the catalog, `Open` with the change ring. `openFileStore`, `buildCatalog` and `logPartialWalk` leave `cmd/demarkus-server` for `filestore`; the soul calls `OpenDir`. Pure move, no behavior change; the soul's on-disk layout is unchanged by construction, since the knowledge path will call the same function.

2. **A world has a `dir` or a `bucket`.** `WorldConfig` gains `Dir string yaml:"dir"`. Exactly one of `dir` and `bucket.url` is set; a world with both or neither is refused by name. A file world's `dir` is cleaned and absolute; two worlds cannot share one (same rule as duplicate bucket URLs). `bucket.worldID` is not required on a file world and `ID()` returns `""` for one. A file world is one replica: a config with `peers` and a file world is refused, because a directory cannot be shared across replicas and the hint it would carry has no epoch. `readOnly`, `auth`, `policy`, `limits` and `authorities` apply as today.

3. **The factory dispatches on the world.** `storeFactory` in `run.go` becomes `worldStores(objects, role)`: a world with `Dir` returns `filestore.OpenDir(world.Dir, filestore.Options{ChangeRing: changefeed.DefaultRingSize, Logger: hooks.logger})`; otherwise the bucket path as today. A writable file world's directory is created on open (`MkdirAll`); a read-only one must exist. `hooks.committed` is not wired for a file world: no peers, nothing to hint.

4. **The GCS client is lazy.** `Server.open` builds `storage.NewClient` on the first bucket world, not at startup, so a server whose worlds are all directories runs with no credentials and no network. `Options.Buckets` keeps its test seam.

5. **Limits.** `limits.maxDocuments` is enforced by `bucketstore` only; `filestore` has no count. For a file world the field is accepted and logged as not enforced, with a follow-up to enforce it in `filestore` or in `worldruntime` for both. Not a blocker: the soul has never had the limit.

Nothing else moves: policy seed, floor merge, style guide, token files, SNI routing, the gateway's in-process exchange and WATCH all run against the store interface and gain file worlds for free.

Config, the whole of it for a self-hosted system:

```yaml
version: 1
listen: { address: ":6309" }
tls: { certFile: /etc/demarkus/tls.crt, keyFile: /etc/demarkus/tls.key }
worlds:
  - name: team
    authorities: [team.example.com]
    dir: /var/lib/demarkus/worlds/team
    auth: { tokensFile: /etc/demarkus/team-tokens.toml }
floorWorld: team
```

## What it does not do

- **Broker state on disk.** `demarkus-knowledge` (worlds plus the in-process broker) still keeps its registry, OAuth state and tenant provisioning in a GCS state bucket, and provisions tenants as bucket worlds. A self-hosted knowledge system with SSO needs that state off GCS too; it is the next item under "self-host without GCS", its own design. This step's proof is `demarkus-knowledge-server` alone.
- **Dynamic file worlds.** The `worldsFile` seam and the broker's provisioner stay bucket-only. A file world is operator config.
- **Replicated file worlds.** Refused by validation, above.
- **The thin main.** `demarkus-server` keeps its own wiring in this step. It becomes a `local` main over `knowledgeserver` in the step after, once a file world is proven; that is where the layer guard learns the forbidden imports.

## Tests that fail without each change

| Change | Test |
|---|---|
| 1 | `filestore`: `TestOpenDirServesAnExistingSoul` opens the upgrade capture's content directory (`testdata/upgrade/home/.demarkus/content`) and reads it, the same documents the soul server test reads; `cmd/demarkus-server` tests unchanged and green (`TestSoulWritesAreGated`, the upgrade capture) |
| 2 | `knowledgeconfig`: a world with `dir` and no bucket loads; both set, neither set, a relative `dir`, two worlds on one `dir`, and `peers` with a file world are refused by name; `ID()` is empty for a file world; the chart-rendered fixture is untouched (the chart knows only buckets) |
| 3 | `knowledgeserver`: `TestFileWorldServes` opens a server from a config with one file world under `t.TempDir()`, publishes through `Exchange`, reopens the server on the same directory and fetches the document and its version; a WATCH cursor taken before the restart resumes after it; the policy document is seeded on first open; a read-only file world on a missing directory fails the open by name |
| 3 | `knowledgeserver`: `TestFileWorldReadsASoul` opens the upgrade capture's content directory as a world and fetches a document the soul wrote; the proof that the merged binary reads a soul's files without migration |
| 4 | `knowledgeserver`: the file-world tests pass with `Options.Buckets` nil and no GCS credentials in the environment, which today fails in `Server.open` |
| 5 | `knowledgeconfig`: `maxDocuments` on a file world loads; `knowledgeserver` logs it as not enforced once at open |
| all | storetest suites unchanged; gateway goldens unchanged (no surface change); `scripts/check-chart-fixtures.sh` unchanged |

## Release

Two PRs, both in the server module, each its own release (a pure move or a behavior change, never both):

1. `refactor(server): file store opener lives in filestore`. The move. Server tag, pin bump, no user-visible change; the plugin's managed server upgrades through the launcher as always.
2. `feat(server): knowledge worlds open from a directory`. Changes 2 to 5 with their tests. Server tag, pin bump. No prompt, tool, instruction or chart change; nothing for the plugins.

Ride-along on PR 2, its own commit: `TestSeveralWritersModel`'s 70 percent closed-loop ratio becomes a log line; the starvation, slot-loss, recovery and p50 assertions stay (the flake recorded in [debugging](/debugging.md#flaky-bucketstore-testseveralwritersmodel-throughput-ratio-on-ci-runners-2026-10-19)).

## Skew

| Pair | Effect |
|---|---|
| Old config, new server | a `bucket:` world is unchanged; nothing to migrate |
| New config with `dir`, old server | refused at load: `bucket.url` required. A config is deployed with its server, so this is the only mixed pair and it fails loud |
| Soul directory, knowledge server | readable as a file world (the capture test); the soul keeps writing it with `demarkus-server` until the thin main lands. Both write through `filestore`, so a handover either way is a config change, not a migration |
| Chart | unchanged; the chart renders bucket worlds only until a self-host values path exists |

## Risks

- `ID()` empty for a file world reaches two places: the committed hook (not wired) and `Hint` (compares IDs, then requires `backend.Follower`, which `filestore` is not). A peer hint with an empty world ID is already malformed at the parser. Covered by the validation that refuses peers with file worlds.
- A file world and the soul server on the same directory at once would corrupt the change journal. Not prevented by code today for two soul servers either; the lock is the launcher's. Documented, and the thin-main step removes the second writer.
- The lazy GCS client moves a startup failure (no credentials) to the first bucket world's open; in static mode that is still fatal at startup, in dynamic mode it goes to pending with a logged reason. Same contract as a bucket that cannot be reached.

## Progress

- Drafted 2026-10-19. Waiting on Fritz to approve the shape before any branch.
- Approved by Fritz 2026-10-19, one feature branch for the whole feature (move and behavior together, his call): `feat/file-worlds` from `bb24d871`. Shipped as designed with four corrections found in the code: the capture's soul directory is `testdata/upgrade/home/.demarkus/soul`, not `content`; a read-only file world needs a store-level guard (`readOnlyStore` in `knowledgeserver` answers `backend.ErrReadOnly`, since the file store has no read-only mode and the policy seed writes straight to the store) and opens without a change journal, so an empty directory stays empty and WATCH is off on it; the WATCH-cursor-across-restart case stays with the file store's own journal tests rather than a server-level test; the config's seen-store maps are one `storeKeys` struct (argument-limit lint). Tests: `TestOpenDirServesAnExistingSoul`, `TestOpenDirRefusesAMissingDirectory`, `TestParseFileWorld`, `TestValidateFileWorld` (five refusals), `TestFileWorldAcceptsLimits`, `TestFileWorldServesAcrossARestart`, `TestFileWorldReadsACapturedSoul`, `TestReadOnlyFileWorldWritesNothing`, `TestOpenServesFileWorldsWithoutGCS` (credentials pointed at a missing file; the server opens and routes). The flake fix rode along. Server module green with race, pre-commit green. Uncommitted for Fritz.