File worlds in the knowledge server

First move of step 7 of Memory and knowledge layering, under the delivery bar, and the "self-host without GCS" item of The engine, not the wire. For demarkus maintainers: how knowledgeserver opens a world from a directory with the file store the soul already runs, what the config looks like, what a file world cannot do, the tests, and the two PRs. Drafted 2026-10-19 from main at bb24d871 (server 0.70.0). The shape it serves is ADR 0037 decision 6: one runtime, two thin mains; this is the half that gives the knowledge path a file backend, so the thin local main can follow.

Today

filestore.Store already satisfies worldStore (backend.Store, backend.ChangeSource, io.Closer) and passes the same storetest conformance, lookup, handler and differential suites bucketstore does. Nothing in the runtime knows which store it holds. The gap is wiring and config.

The change

1. One file-store constructor. filestore.OpenDir(dir string, opts Options) (*Store, error): store.New, BuildIndex (a partial walk logged per entry, a broken walk an error), the catalog, Open with the change ring. openFileStore, buildCatalog and logPartialWalk leave cmd/demarkus-server for filestore; the soul calls OpenDir. Pure move, no behavior change; the soul's on-disk layout is unchanged by construction, since the knowledge path will call the same function.

2. A world has a dir or a bucket. WorldConfig gains Dir string yaml:"dir". Exactly one of dir and bucket.url is set; a world with both or neither is refused by name. A file world's dir is cleaned and absolute; two worlds cannot share one (same rule as duplicate bucket URLs). bucket.worldID is not required on a file world and ID() returns "" for one. A file world is one replica: a config with peers and a file world is refused, because a directory cannot be shared across replicas and the hint it would carry has no epoch. readOnly, auth, policy, limits and authorities apply as today.

3. The factory dispatches on the world. storeFactory in run.go becomes worldStores(objects, role): a world with Dir returns filestore.OpenDir(world.Dir, filestore.Options{ChangeRing: changefeed.DefaultRingSize, Logger: hooks.logger}); otherwise the bucket path as today. A writable file world's directory is created on open (MkdirAll); a read-only one must exist. hooks.committed is not wired for a file world: no peers, nothing to hint.

4. The GCS client is lazy. Server.open builds storage.NewClient on the first bucket world, not at startup, so a server whose worlds are all directories runs with no credentials and no network. Options.Buckets keeps its test seam.

5. Limits. limits.maxDocuments is enforced by bucketstore only; filestore has no count. For a file world the field is accepted and logged as not enforced, with a follow-up to enforce it in filestore or in worldruntime for both. Not a blocker: the soul has never had the limit.

Nothing else moves: policy seed, floor merge, style guide, token files, SNI routing, the gateway's in-process exchange and WATCH all run against the store interface and gain file worlds for free.

Config, the whole of it for a self-hosted system:

version: 1
listen: { address: ":6309" }
tls: { certFile: /etc/demarkus/tls.crt, keyFile: /etc/demarkus/tls.key }
worlds:
  - name: team
    authorities: [team.example.com]
    dir: /var/lib/demarkus/worlds/team
    auth: { tokensFile: /etc/demarkus/team-tokens.toml }
floorWorld: team

What it does not do

• Broker state on disk. demarkus-knowledge (worlds plus the in-process broker) still keeps its registry, OAuth state and tenant provisioning in a GCS state bucket, and provisions tenants as bucket worlds. A self-hosted knowledge system with SSO needs that state off GCS too; it is the next item under "self-host without GCS", its own design. This step's proof is demarkus-knowledge-server alone.
• Dynamic file worlds. The worldsFile seam and the broker's provisioner stay bucket-only. A file world is operator config.
• Replicated file worlds. Refused by validation, above.
• The thin main. demarkus-server keeps its own wiring in this step. It becomes a local main over knowledgeserver in the step after, once a file world is proven; that is where the layer guard learns the forbidden imports.

Tests that fail without each change

Release

Two PRs, both in the server module, each its own release (a pure move or a behavior change, never both):

1. refactor(server): file store opener lives in filestore. The move. Server tag, pin bump, no user-visible change; the plugin's managed server upgrades through the launcher as always.
2. feat(server): knowledge worlds open from a directory. Changes 2 to 5 with their tests. Server tag, pin bump. No prompt, tool, instruction or chart change; nothing for the plugins.

Ride-along on PR 2, its own commit: TestSeveralWritersModel's 70 percent closed-loop ratio becomes a log line; the starvation, slot-loss, recovery and p50 assertions stay (the flake recorded in debugging).

Skew

Risks

• ID() empty for a file world reaches two places: the committed hook (not wired) and Hint (compares IDs, then requires backend.Follower, which filestore is not). A peer hint with an empty world ID is already malformed at the parser. Covered by the validation that refuses peers with file worlds.
• A file world and the soul server on the same directory at once would corrupt the change journal. Not prevented by code today for two soul servers either; the lock is the launcher's. Documented, and the thin-main step removes the second writer.
• The lazy GCS client moves a startup failure (no credentials) to the first bucket world's open; in static mode that is still fatal at startup, in dynamic mode it goes to pending with a logged reason. Same contract as a bucket that cannot be reached.

Progress

• Drafted 2026-10-19. Waiting on Fritz to approve the shape before any branch.
• Approved by Fritz 2026-10-19, one feature branch for the whole feature (move and behavior together, his call): feat/file-worlds from bb24d871. Shipped as designed with four corrections found in the code: the capture's soul directory is testdata/upgrade/home/.demarkus/soul, not content; a read-only file world needs a store-level guard (readOnlyStore in knowledgeserver answers backend.ErrReadOnly, since the file store has no read-only mode and the policy seed writes straight to the store) and opens without a change journal, so an empty directory stays empty and WATCH is off on it; the WATCH-cursor-across-restart case stays with the file store's own journal tests rather than a server-level test; the config's seen-store maps are one storeKeys struct (argument-limit lint). Tests: TestOpenDirServesAnExistingSoul, TestOpenDirRefusesAMissingDirectory, TestParseFileWorld, TestValidateFileWorld (five refusals), TestFileWorldAcceptsLimits, TestFileWorldServesAcrossARestart, TestFileWorldReadsACapturedSoul, TestReadOnlyFileWorldWritesNothing, TestOpenServesFileWorldsWithoutGCS (credentials pointed at a missing file; the server opens and routes). The flake fix rode along. Server module green with race, pre-commit green. Uncommitted for Fritz.
• Simplify pass 2026-10-19 (four reviewers) moved three things to their right depth: filestore.Options owns ReadOnly (no journal, writes answer backend.ErrReadOnly; the soul's --read-only stops rewriting its journal too) and MaxDocuments (a create past the cap is ErrDocumentQuota wrapping backend.ErrQuota, a new version always lands), so the readOnlyStore wrapper and the "not enforced" log went; the lazy GCS client is gcs.Lazy in server/blob/gcs with Open, Opened, Close, so knowledgeserver no longer imports the storage client; OpenDir creates a writable directory itself. Also: the soul's cmd lost its one-entry storeOpeners registry and backend struct (openStore returns the file store, ValidateStoreBackend refuses any other name now); knowledgeconfig shares one canonicalPath rule between dir and policy.file and one storeKeys.claim for every duplicate check; Hint ignores a world with no ID. Tests: testCertFiles takes the DNS name, backendtest.Direct reads the capture, the boundary test uses OpenDir; new TestOpenDirCreatesOrRequiresTheDirectory and TestOpenDirCapsDocuments. Skipped: the double walk at open (index then catalog; pre-existing, a BuildIndex observer would halve startup I/O, follow-up), a shared capture locator across three test packages, dropping TestFileWorldReadsACapturedSoul (it is the design's proof at the world level). Lint ratchet improved by one and was updated. Server module green with race, pre-commit green.
• PR 631 review, CodeRabbit round one: two security findings, both real and fixed. The document cap keyed on expected-version: 0 missed the absent header (-1, an unconditional write) and so let a client create paths past the cap; protocol/store.Indexed now tells a new path from an existing one and the cap applies to every create. Two file worlds could nest and serve each other's files; validateStore refuses an ancestor or descendant (strict within), symlink aliases named as not resolved. Rolled in after approval: operator docs for dir (server guide and config reference), and one walk at open: BuildIndexObserved fills the index and the catalog together, halving startup reads for every soul and file world (the follow-up from the simplify pass).
• Simplify pass on the single-walk round (four reviewers): WalkCurrent deleted (no production caller once the observed build existed; demarkus-library does not import protocol/store), its test retargeted at BuildIndexObserved and renamed; the index is now built aside and swapped in whole, so the observer runs with no lock held and a broken walk keeps the previous index; the partial-walk helper lost its one-valued parameter and the log its derivable catalog count; docs say each thing once and the file-world note sits beside the bucket note. Skipped: filestore.New/Open owning the catalog (eleven callers across test packages, a separate tidy), the body-to-string copies in catalog.Put and the repeated frontmatter split per document (pre-existing, follow-ups), one review claim that peers with a file world is not refused (it is; TestValidateFileWorld/peers).
• Smoke 2026-10-19 on built binaries over real QUIC: scripts/smoke.sh file-world added (the knowledge server on a dir world, credentials pointed at a missing file, the same verbs, policy, watch and restart checks as the GCS mode) at 77 of 77; the soul mode at 105 of 105 after its restart check moved to the new document index built line; the GCS mode against fake-gcs at 86 of 86. The one failure found was the smoke script's own expectation of the removed lookup catalog built log line.
• Merged 2026-10-19 as PR 631 (9154affc) after three CodeRabbit rounds; server 0.71.0, protocol 0.28.0, client 0.63.0, knowledge 0.73.0, tools 0.69.1; pins by PRs 632 and 633, plugin 0.13.201. Done. Next move of step 7: the thin local main.