# Plan: Memory and knowledge layering

Audience: demarkus maintainers. Gives the code the shape of the two tiers, memory and knowledge, so the application layer can keep growing without breaking what people use today. Drafted 2026-10-16 with Fritz; step 1 merged the same day as PR 604 (`7ed84f1d`), step 2 on 2026-10-17 as PR 606 (`83f8729a`), step 3 the same day across PRs 608 to 613 (`4d37dca1`), step 4 on 2026-10-18 and 2026-10-19 across PRs 615 to 624 (`41273924`), step 5 on 2026-10-19 across PRs 625 to 630 (`bb24d871`), step 7's first move the same day as PR 631 (`9154affc`) and its second on 2026-10-20 as PR 634 (`30a6555d`). Positioning, the two doors and the transport freeze are settled in [The engine, not the wire](/plans/engine-not-wire.md) and are not repeated here.

## Premise

Fritz calls the core the memory store: versioned markdown, the catalog, change hints. It is well defined. What sits above it is not. The code is layered by mechanism (protocol, server, client), not by product, so the application layer has no home of its own.

| Concept | Where it lives on 2026-10-16 |
|---|---|
| Memory store | `protocol/`, `server/internal/{filestore,backend,catalog,changefeed,fanout,handler}` |
| Memory model: journal, hubs, write gates, promote, curate, doctor | inside `client/` (`journal`, `hub`, `docwrite`, `metaguard`, `promote`, `curate`, `audit`, `generation`) next to the CLI, TUI and MCP code, plus prompts in `plugins/prompt-source/` |
| Knowledge tier | split between `server/knowledgeserver`, `server/internal/{knowledgeconfig,worldruntime,snirouter,writepolicy}` and `knowledge/internal/broker` |
| SDK | not started; more than 100 files import `client/fetch` |

The memory model exists in code but has no name and no module. Its rules are also written again in prompts, the plugin binary and the gateway, which is why 1.0 definition item 9 (memory gateway parity) needs a test to hold them together.

## Target layers

1. **Memory store.** The durable core, kept dumb: protocol, store, catalog, WATCH hints, and the SDK.
2. **Memory model.** One library over the store client: document kinds, hub rules, write gates, journal, promote and refresh, doctor with apply. No MCP, no CLI.
3. **Tiers.** A soul is the model plus local provisioning. A knowledge system is the same model plus worlds, identity and policy. One model, not two applications.
4. **Surfaces.** MCP tools, plugins, the gateway, CLI, TUI and the library stay thin.

Tiers and the durability layers in [the vision](/vision.md) are two axes: tiers say who owns knowledge, layers say how long it must last.

No protocol change. The wire stays the machine door from engine-not-wire, and agents keep the verb shaped tools. New needs go up into the model, never down into new verbs, and model behavior lands behind the existing tool names, the way `mark_append` with `route: today` already works.

## Contracts that must not break

| Contract | Who depends on it | Guard |
|---|---|---|
| Wire protocol | CLI, TUI, MCP server, broker, demarkus-library, Caztor (FETCH only) | storetest; wire fixtures planned in SDK phase 1 |
| MCP tool names, parameters, instructions, output | every agent session | plugin eval; goldens of `tools/list` and the server instructions for `demarkus-mcp` (with and without `-host`) and both gateway profiles, through `client/mcpfmt/mcpfmttest`; one tool class table in `mcpfmt` sets the read-only hints |
| Plugin commands and hooks | plugin users | plugin eval, pre-commit |
| Stored format (`protocol/storefmt`) | every local soul, the GCS deploy | format and fuzz tests; the upgrade capture under `testdata/upgrade` served by the current server |
| Config and token files | deploy repo, installs | chart-rendered fixtures for the knowledge server and broker configs, checked by `scripts/check-chart-fixtures.sh` in CI; the upgrade capture for the plugin's state and token files |
| Go import paths | demarkus-library (`client/fetch`, `client/memory/graphstore`, `protocol/mdoutline` since step 3) | its `go.mod` pin |

mark-knowledge imports no demarkus Go code. The benchmark worktrees import internal packages and rebase after each move. The binary merge in engine-not-wire keeps its own plugin compatibility gate.

## Gaps, ranked

1. **Maintenance before features.** Apply shipped in step 2 and this soul went from 731 doctor findings to 508; what is left sits in classes apply does not own (size, document shape, style) or needs judgment. The root hub is at 37 links. `/adr/` (44) and `/plans/` (151) are over what one generated hub carries.
2. **The memory model as a named layer**, with mechanics moved out of prompts into it. Named and placed by step 3; the mechanics moved in step 4.
3. **The same write gates in both tiers.** The knowledge system refuses secrets at write; this soul held three `connection-string-password` findings (fixed by hand on 2026-10-16). Closed by step 5: `demarkus-server` runs the same content check and policy gate and refuses a secret since server 0.70.0.
4. **Lifecycle as model rules**: supersede, stale, archive. Overlaps [Embracing staleness](/plans/embracing-staleness.md).
5. **The protocol freeze** after WATCH and ALPN `mark/1` ([ADR 0025](/adr/0025-watch-verb.md)), so the wire stops costing attention.

## Sequence

Each step is additive or behavior neutral.

1. Done, PR 604: guards before code, all on one branch so the release workflow cut one round of releases. Layer import rules in pre-commit, goldens of what the MCP surfaces show an agent (tool lists and server instructions), chart-rendered config fixtures, and the upgrade capture.
2. Done, PR 606 (`83f8729a`, 2026-10-17): doctor apply and generated hubs, built where the code lives now (`client/audit`), and applied to this soul on 2026-10-16 from a branch build (152 writes in two batches, all landed). Projects follow the directory hub rule too. Design, results and the follow-ups left open in [Doctor apply and generated hubs](/plans/memory-knowledge-layering/doctor-apply.md).
3. Done, PRs 608 to 613 (`4d37dca1`, 2026-10-17): the memory model lives in `client/memory/` ([ADR 0043](/adr/0043-memory-model-in-client-memory.md)) and `mdoutline` split ([ADR 0044](/adr/0044-mdoutline-split.md)). One refactor PR removed the model's upward edges, two PRs of pure moves each matched a transform rerun on main, and the eval held at 8 of 8. The model section of the layer map is one line. Design, results, the library and benchmark follow-ups and what is left in [Model carve-out](/plans/memory-knowledge-layering/model-carve-out.md).
4. Done, PRs 615 to 624 (`41273924`, 2026-10-18 and 19): the Cowork plugin retired into one memory plugin, the eval mocks rendered from code so a mechanic case can fail, decision numbers and the refresh inventory behind `mark_list` and `mark_promote`, the plugin binary's host contracts (`verdict=`, binary states, `SKIPPED` rows, `provision if-configured`), and the prompts and hooks rewritten to read them. Eval at 10 of 10; prompt source 44,727 bytes against the 40 KB aim, the rest in Go follow-ups. Design, results and follow-ups in [Prompt mechanics](/plans/memory-knowledge-layering/prompt-mechanics.md).
5. Done, PRs 625 to 630 (`bb24d871`, 2026-10-19): same gates in both tiers, warn then refuse in two rounds the same day ([ADR 0045](/adr/0045-write-gates-run-in-the-store.md)). The soul server wraps its store with the `writepolicy` gate the knowledge server already runs (secrets, personal data, a published policy); a reserved `warning` response key carries server-side warns to the writer in both tiers; the hook mirrors the scan pre-write; no new tool, parameter or knob; the step 4 follow-ups rode along. Server 0.70.0 refuses a secret; plugin 0.13.199. Eval 11 of 11. Design, order, skew table, tests and the live check in [Same gates in both tiers](/plans/memory-knowledge-layering/gates.md). Left open from the ride-alongs: the single health probe (tools, then the hook script a release later) and the `adr:` layout key.
6. SDK as narrowed in engine-not-wire. Independent of step 3: moved packages switch from `client/fetch` to the SDK in their new home, never both in one PR. The layer guard learns the external module first: the map names the SDK module path as an `sdk` layer that imports nothing, `storeclient` and everything above may import it, `store` may not, so pre-commit rather than review enforces [ADR 0026](/adr/0026-client-sdk-mit-server-never-imports.md). Today the checker drops imports from outside the repo. `storeclient` then means the client helpers that stay in core over the SDK (`listwalk`, `links`, `merge`, `generation`, `joinurl`, `token`, the caches), and the map lines for `fetch`, `listing` and `lookuptable` go once core switches over.
7. Knowledge tier consolidation, decided together with the binary merge (ADR 0037 decision 6, amended 2026-10-19). First move done, PR 631 (`9154affc`, 2026-10-19): a knowledge world opens from a directory through `filestore.OpenDir`; `dir` is the alternative to `bucket:`. Second move done, PR 634 (`30a6555d`, 2026-10-20; server 0.72.0, plugin 0.13.203): `demarkus-server` is a thin main over `knowledgeserver` in a local profile (`knowledgeconfig.LocalProfile`: one file world on every server name, a regenerating dev certificate, no health port, no policy seed), the bucket factory and the GCS client out of the runtime into `bucketworlds`, the binary pinned free of cloud packages; the gate passed on Fritz's soul (eval 11 of 11, `/soul-status`). Designs and results in [File worlds](/plans/memory-knowledge-layering/file-worlds.md) and [The thin local main](/plans/memory-knowledge-layering/local-main.md). Next: the third move, `demarkus-knowledge-server` collapsing into `demarkus-knowledge` (the open question is a worlds-only mode of the full binary for the smoke script); then the `broker/core -> mcpfmt` baseline edge is settled here: the tier parses the gateway's surface config, and the tool profile names are MCP vocabulary no layer below `mcpfmt` should carry.

How each step ships, the engineering bar and the no-friction gate: [Delivery](/plans/memory-knowledge-layering/delivery.md). In short, a pure move or a behavior change, never both, and an upgrade the user does not notice.

## Plugins

[One memory plugin](/plans/memory-knowledge-layering/one-memory-plugin.md): done in step 4. `demarkus-cowork` is retired; every host runs the same commands, host writes stay shell, hooks stay optional and provision only a configured setup. Open: the knowledge plugin in Cowork (user-scope MCP reachability untested, its hook still bootstraps in the sandbox, no host partial on that surface).

## What elegant means

| Principle | Test |
|---|---|
| One model, two tiers | same names, gates and commands in soul and knowledge; promote and refresh are the only edge |
| Mechanics in software, judgment in the agent | prompt source bytes go down; [ADR 0042](/adr/0042-soul-entries-are-agent-written.md) holds |
| Memory maintains itself | zero doctor findings in the classes apply owns; hubs stay under the cap without manual edits |
| Few nouns | nothing beyond memory, soul, knowledge system, world |
| Clean boundaries | dependencies point down only, enforced in pre-commit |
| Thin surfaces | one plugin per host format; the parity test of 1.0 item 9 passes |
| Invisible upgrades | the upgrade capture passes at every checkpoint |
| Measured | eval pass rate holds; tokens per correct answer go down |

Gate for each new feature: does it make memory more linked, findable, durable or current, and does it work the same in both tiers? If not, it is a product feature, not the model.

## Open questions

- Multi-world hosting: core or knowledge tier? Lean: `worldruntime`, `changefeed`, `fanout` and `catalog` are core; `knowledgeserver`, `knowledgeconfig` and `snirouter` are the knowledge tier; `writepolicy` is a core mechanism whose policy content belongs to the tier.
- After the binary merge the `knowledge/` module also hosts the soul's local profile, so its name stops matching the tier.
- Is "memory store" the public word engine-not-wire leaves open?

The module or directory question is settled by [ADR 0043](/adr/0043-memory-model-in-client-memory.md): a directory, `client/memory/`.

## Related documents

- [The engine, not the wire](/plans/engine-not-wire.md): positioning, doors, transport freeze
- [Vision](/vision.md): the end state and durability layers
- [Fortify the durable core](/plans/durable-core.md): what the store must keep
- [demarkus-sdk](/plans/demarkus-sdk.md): the client SDK
- [Agent UX](/plans/agent-ux.md): doctor apply
- [Embracing staleness](/plans/embracing-staleness.md): lifecycle
- [demarkus 1.0](/plans/v1.md): item 9, gateway parity
- [Architecture](/architecture.md): module boundaries today
- [Model carve-out](/plans/memory-knowledge-layering/model-carve-out.md): step 3, the move into `client/memory/`
- [Move script](/plans/memory-knowledge-layering/move-script.md): Part of [Model carve-out](/plans/memory-knowledge-layering/model-carve-out.md). The transform that produced the step 3 move PRs (PR 610 and PR 612), kept here because it lived in a
- [Prompt mechanics](/plans/memory-knowledge-layering/prompt-mechanics.md): step 4, mechanics behind tool names, host contracts, the prompts that read them
- [Same gates in both tiers](/plans/memory-knowledge-layering/gates.md): step 5, the write gates per tier, warn then refuse, skew, tests, ride-along follow-ups
- [File worlds in the knowledge server](/plans/memory-knowledge-layering/file-worlds.md): step 7's first move, a knowledge world on a directory, the file backend the thin main needs
- [The thin local main](/plans/memory-knowledge-layering/local-main.md): step 7's second move, demarkus-server as the shared runtime's local profile, the seams, the plugin gate
- [One full binary](/plans/memory-knowledge-layering/one-full-binary.md)