soul.demarkus.io:6309/plans/memory-knowledge-layering.md/v9 active reader meta

Plan: Memory and knowledge layering

Audience: demarkus maintainers. Gives the code the shape of the two tiers, memory and knowledge, so the application layer can keep growing without breaking what people use today. Drafted 2026-10-16 with Fritz; step 1 merged the same day as PR 604 (7ed84f1d). Positioning, the two doors and the transport freeze are settled in The engine, not the wire and are not repeated here.

Premise

Fritz calls the core the memory store: versioned markdown, the catalog, change hints. It is well defined. What sits above it is not. The code is layered by mechanism (protocol, server, client), not by product, so the application layer has no home of its own.

Concept Where it lives on 2026-10-16
Memory store protocol/, server/internal/{filestore,backend,catalog,changefeed,fanout,handler}
Memory model: journal, hubs, write gates, promote, curate, doctor inside client/ (journal, hub, docwrite, metaguard, promote, curate, audit, generation) next to the CLI, TUI and MCP code, plus prompts in plugins/prompt-source/
Knowledge tier split between server/knowledgeserver, server/internal/{knowledgeconfig,worldruntime,snirouter,writepolicy} and knowledge/internal/broker
SDK not started; more than 100 files import client/fetch

The memory model exists in code but has no name and no module. Its rules are also written again in prompts, the plugin binary and the gateway, which is why 1.0 definition item 9 (memory gateway parity) needs a test to hold them together.

Target layers

  1. Memory store. The durable core, kept dumb: protocol, store, catalog, WATCH hints, and the SDK.
  2. Memory model. One library over the store client: document kinds, hub rules, write gates, journal, promote and refresh, doctor with apply. No MCP, no CLI.
  3. Tiers. A soul is the model plus local provisioning. A knowledge system is the same model plus worlds, identity and policy. One model, not two applications.
  4. Surfaces. MCP tools, plugins, the gateway, CLI, TUI and the library stay thin.

Tiers and the durability layers in the vision are two axes: tiers say who owns knowledge, layers say how long it must last.

No protocol change. The wire stays the machine door from engine-not-wire, and agents keep the verb shaped tools. New needs go up into the model, never down into new verbs, and model behavior lands behind the existing tool names, the way mark_append with route: today already works.

Contracts that must not break

Contract Who depends on it Guard
Wire protocol CLI, TUI, MCP server, broker, demarkus-library, Caztor (FETCH only) storetest; wire fixtures planned in SDK phase 1
MCP tool names, parameters, instructions, output every agent session plugin eval; goldens of tools/list and the server instructions for demarkus-mcp (with and without -host) and both gateway profiles, through client/mcpfmt/mcpfmttest; one tool class table in mcpfmt sets the read-only hints
Plugin commands and hooks plugin users plugin eval, pre-commit
Stored format (protocol/storefmt) every local soul, the GCS deploy format and fuzz tests; the upgrade capture under testdata/upgrade served by the current server
Config and token files deploy repo, installs chart-rendered fixtures for the knowledge server and broker configs, checked by scripts/check-chart-fixtures.sh in CI; the upgrade capture for the plugin's state and token files
Go import paths demarkus-library (client/fetch, mdoutline, graphstore) its go.mod pin

mark-knowledge imports no demarkus Go code. The benchmark worktrees import internal packages and rebase after each move. The binary merge in engine-not-wire keeps its own plugin compatibility gate.

Gaps, ranked

  1. Maintenance before features. This soul has 745 doctor findings and apply is unbuilt (agent UX item 3). The root hub holds 47 links against a cap of 40; /journal/ (127) and /plans/ (147) have no generated hubs.
  2. The memory model as a named layer, with mechanics moved out of prompts into it.
  3. The same write gates in both tiers. The knowledge system refuses secrets at write; this soul holds three connection-string-password findings. Check whether the soul write path runs the same content check.
  4. Lifecycle as model rules: supersede, stale, archive. Overlaps Embracing staleness.
  5. The protocol freeze after WATCH and ALPN mark/1 (ADR 0025), so the wire stops costing attention.

Sequence

Each step is additive or behavior neutral.

  1. Done, PR 604: guards before code, all on one branch so the release workflow cut one round of releases. Layer import rules in pre-commit, goldens of what the MCP surfaces show an agent (tool lists and server instructions), chart-rendered config fixtures, and the upgrade capture.
  2. Doctor apply and generated hubs, built where the code lives now (client/audit). In progress on feat/doctor-apply; design in Doctor apply and generated hubs.
  3. Carve out the memory model, one package per PR, pure moves; proof is identical pre-commit and eval results. docwrite splits: write rules toward the SDK, the gate into the model. Moving the profile constants and the tool class table below mcpfmt removes the broker/core baseline entry.
  4. Move prompt mechanics into the model behind existing tool names, one eval case per mechanic. This retires the Cowork plugin (below).
  5. Same gates in both tiers: warn, then refuse.
  6. SDK as narrowed in engine-not-wire. Independent of step 3: moved packages switch from client/fetch to the SDK in their new home, never both in one PR. The layer guard learns the external module first: the map names the SDK module path as an sdk layer that imports nothing, storeclient and everything above may import it, store may not, so pre-commit rather than review enforces ADR 0026. Today the checker drops imports from outside the repo. storeclient then means the client helpers that stay in core over the SDK (listwalk, links, merge, generation, joinurl, token, the caches), and the map lines for fetch, listing and lookuptable go once core switches over.
  7. Knowledge tier consolidation, decided together with the binary merge.

How each step ships, the engineering bar and the no-friction gate: Delivery. In short, a pure move or a behavior change, never both, and an upgrade the user does not notice.

Plugins

One memory plugin: once host reads are MCP tools, demarkus-cowork retires and every host runs the same commands; host writes stay shell, hooks stay optional. Whether Cowork's hooks run on the host decides whether that can happen sooner.

What elegant means

Principle Test
One model, two tiers same names, gates and commands in soul and knowledge; promote and refresh are the only edge
Mechanics in software, judgment in the agent prompt source bytes go down; ADR 0042 holds
Memory maintains itself zero doctor findings in the classes apply owns; hubs stay under the cap without manual edits
Few nouns nothing beyond memory, soul, knowledge system, world
Clean boundaries dependencies point down only, enforced in pre-commit
Thin surfaces one plugin per host format; the parity test of 1.0 item 9 passes
Invisible upgrades the upgrade capture passes at every checkpoint
Measured eval pass rate holds; tokens per correct answer go down

Gate for each new feature: does it make memory more linked, findable, durable or current, and does it work the same in both tiers? If not, it is a product feature, not the model.

Open questions

  • Multi-world hosting: core or knowledge tier? Lean: worldruntime, changefeed, fanout and catalog are core; knowledgeserver, knowledgeconfig and snirouter are the knowledge tier; writepolicy is a core mechanism whose policy content belongs to the tier.
  • The memory model as its own Go module (memory/, lockstep versioned) or a directory inside client/?
  • After the binary merge the knowledge/ module also hosts the soul's local profile, so its name stops matching the tier.
  • Is "memory store" the public word engine-not-wire leaves open?
trail
  1. soul.demarkus.io:6309 v9