soul.demarkus.io:6309/plans/memory-knowledge-layering.md/v24 active reader meta

Plan: Memory and knowledge layering

Audience: demarkus maintainers. Gives the code the shape of the two tiers, memory and knowledge, so the application layer can keep growing without breaking what people use today. Drafted 2026-10-16 with Fritz; step 1 merged the same day as PR 604 (7ed84f1d), step 2 on 2026-10-17 as PR 606 (83f8729a), step 3 the same day across PRs 608 to 613 (4d37dca1), step 4 on 2026-10-18 and 2026-10-19 across PRs 615 to 624 (41273924), and step 5 on 2026-10-19 across PRs 625 to 630 (bb24d871). Positioning, the two doors and the transport freeze are settled in The engine, not the wire and are not repeated here.

Premise

Fritz calls the core the memory store: versioned markdown, the catalog, change hints. It is well defined. What sits above it is not. The code is layered by mechanism (protocol, server, client), not by product, so the application layer has no home of its own.

Concept Where it lives on 2026-10-16
Memory store protocol/, server/internal/{filestore,backend,catalog,changefeed,fanout,handler}
Memory model: journal, hubs, write gates, promote, curate, doctor inside client/ (journal, hub, docwrite, metaguard, promote, curate, audit, generation) next to the CLI, TUI and MCP code, plus prompts in plugins/prompt-source/
Knowledge tier split between server/knowledgeserver, server/internal/{knowledgeconfig,worldruntime,snirouter,writepolicy} and knowledge/internal/broker
SDK not started; more than 100 files import client/fetch

The memory model exists in code but has no name and no module. Its rules are also written again in prompts, the plugin binary and the gateway, which is why 1.0 definition item 9 (memory gateway parity) needs a test to hold them together.

Target layers

  1. Memory store. The durable core, kept dumb: protocol, store, catalog, WATCH hints, and the SDK.
  2. Memory model. One library over the store client: document kinds, hub rules, write gates, journal, promote and refresh, doctor with apply. No MCP, no CLI.
  3. Tiers. A soul is the model plus local provisioning. A knowledge system is the same model plus worlds, identity and policy. One model, not two applications.
  4. Surfaces. MCP tools, plugins, the gateway, CLI, TUI and the library stay thin.

Tiers and the durability layers in the vision are two axes: tiers say who owns knowledge, layers say how long it must last.

No protocol change. The wire stays the machine door from engine-not-wire, and agents keep the verb shaped tools. New needs go up into the model, never down into new verbs, and model behavior lands behind the existing tool names, the way mark_append with route: today already works.

Contracts that must not break

Contract Who depends on it Guard
Wire protocol CLI, TUI, MCP server, broker, demarkus-library, Caztor (FETCH only) storetest; wire fixtures planned in SDK phase 1
MCP tool names, parameters, instructions, output every agent session plugin eval; goldens of tools/list and the server instructions for demarkus-mcp (with and without -host) and both gateway profiles, through client/mcpfmt/mcpfmttest; one tool class table in mcpfmt sets the read-only hints
Plugin commands and hooks plugin users plugin eval, pre-commit
Stored format (protocol/storefmt) every local soul, the GCS deploy format and fuzz tests; the upgrade capture under testdata/upgrade served by the current server
Config and token files deploy repo, installs chart-rendered fixtures for the knowledge server and broker configs, checked by scripts/check-chart-fixtures.sh in CI; the upgrade capture for the plugin's state and token files
Go import paths demarkus-library (client/fetch, client/memory/graphstore, protocol/mdoutline since step 3) its go.mod pin

mark-knowledge imports no demarkus Go code. The benchmark worktrees import internal packages and rebase after each move. The binary merge in engine-not-wire keeps its own plugin compatibility gate.

Gaps, ranked

  1. Maintenance before features. Apply shipped in step 2 and this soul went from 731 doctor findings to 508; what is left sits in classes apply does not own (size, document shape, style) or needs judgment. The root hub is at 37 links. /adr/ (44) and /plans/ (151) are over what one generated hub carries.
  2. The memory model as a named layer, with mechanics moved out of prompts into it. Named and placed by step 3; the mechanics moved in step 4.
  3. The same write gates in both tiers. The knowledge system refuses secrets at write; this soul held three connection-string-password findings (fixed by hand on 2026-10-16). Closed by step 5: demarkus-server runs the same content check and policy gate and refuses a secret since server 0.70.0.
  4. Lifecycle as model rules: supersede, stale, archive. Overlaps Embracing staleness.
  5. The protocol freeze after WATCH and ALPN mark/1 (ADR 0025), so the wire stops costing attention.

Sequence

Each step is additive or behavior neutral.

  1. Done, PR 604: guards before code, all on one branch so the release workflow cut one round of releases. Layer import rules in pre-commit, goldens of what the MCP surfaces show an agent (tool lists and server instructions), chart-rendered config fixtures, and the upgrade capture.
  2. Done, PR 606 (83f8729a, 2026-10-17): doctor apply and generated hubs, built where the code lives now (client/audit), and applied to this soul on 2026-10-16 from a branch build (152 writes in two batches, all landed). Projects follow the directory hub rule too. Design, results and the follow-ups left open in Doctor apply and generated hubs.
  3. Done, PRs 608 to 613 (4d37dca1, 2026-10-17): the memory model lives in client/memory/ (ADR 0043) and mdoutline split (ADR 0044). One refactor PR removed the model's upward edges, two PRs of pure moves each matched a transform rerun on main, and the eval held at 8 of 8. The model section of the layer map is one line. Design, results, the library and benchmark follow-ups and what is left in Model carve-out.
  4. Done, PRs 615 to 624 (41273924, 2026-10-18 and 19): the Cowork plugin retired into one memory plugin, the eval mocks rendered from code so a mechanic case can fail, decision numbers and the refresh inventory behind mark_list and mark_promote, the plugin binary's host contracts (verdict=, binary states, SKIPPED rows, provision if-configured), and the prompts and hooks rewritten to read them. Eval at 10 of 10; prompt source 44,727 bytes against the 40 KB aim, the rest in Go follow-ups. Design, results and follow-ups in Prompt mechanics.
  5. Done, PRs 625 to 630 (bb24d871, 2026-10-19): same gates in both tiers, warn then refuse in two rounds the same day (ADR 0045). The soul server wraps its store with the writepolicy gate the knowledge server already runs (secrets, personal data, a published policy); a reserved warning response key carries server-side warns to the writer in both tiers; the hook mirrors the scan pre-write; no new tool, parameter or knob; the step 4 follow-ups rode along. Server 0.70.0 refuses a secret; plugin 0.13.199. Eval 11 of 11. Design, order, skew table, tests and the live check in Same gates in both tiers. Left open from the ride-alongs: the single health probe (tools, then the hook script a release later) and the adr: layout key.
  6. SDK as narrowed in engine-not-wire. Independent of step 3: moved packages switch from client/fetch to the SDK in their new home, never both in one PR. The layer guard learns the external module first: the map names the SDK module path as an sdk layer that imports nothing, storeclient and everything above may import it, store may not, so pre-commit rather than review enforces ADR 0026. Today the checker drops imports from outside the repo. storeclient then means the client helpers that stay in core over the SDK (listwalk, links, merge, generation, joinurl, token, the caches), and the map lines for fetch, listing and lookuptable go once core switches over.
  7. Knowledge tier consolidation, decided together with the binary merge: the file-store opener in knowledgeserver, then demarkus-server as the thin local main over it (ADR 0037 decision 6, amended 2026-10-19). The broker/core -> mcpfmt baseline edge is settled here: the tier parses the gateway's surface config, and the tool profile names are MCP vocabulary no layer below mcpfmt should carry.

How each step ships, the engineering bar and the no-friction gate: Delivery. In short, a pure move or a behavior change, never both, and an upgrade the user does not notice.

Plugins

One memory plugin: done in step 4. demarkus-cowork is retired; every host runs the same commands, host writes stay shell, hooks stay optional and provision only a configured setup. Open: the knowledge plugin in Cowork (user-scope MCP reachability untested, its hook still bootstraps in the sandbox, no host partial on that surface).

What elegant means

Principle Test
One model, two tiers same names, gates and commands in soul and knowledge; promote and refresh are the only edge
Mechanics in software, judgment in the agent prompt source bytes go down; ADR 0042 holds
Memory maintains itself zero doctor findings in the classes apply owns; hubs stay under the cap without manual edits
Few nouns nothing beyond memory, soul, knowledge system, world
Clean boundaries dependencies point down only, enforced in pre-commit
Thin surfaces one plugin per host format; the parity test of 1.0 item 9 passes
Invisible upgrades the upgrade capture passes at every checkpoint
Measured eval pass rate holds; tokens per correct answer go down

Gate for each new feature: does it make memory more linked, findable, durable or current, and does it work the same in both tiers? If not, it is a product feature, not the model.

Open questions

  • Multi-world hosting: core or knowledge tier? Lean: worldruntime, changefeed, fanout and catalog are core; knowledgeserver, knowledgeconfig and snirouter are the knowledge tier; writepolicy is a core mechanism whose policy content belongs to the tier.
  • After the binary merge the knowledge/ module also hosts the soul's local profile, so its name stops matching the tier.
  • Is "memory store" the public word engine-not-wire leaves open?

The module or directory question is settled by ADR 0043: a directory, client/memory/.

soul.demarkus.io:6309/vision.md draft reader meta

Vision: live, reactive knowledge

The north star for demarkus and the knowledge system: what the end state is, why it is new, how the pieces fit into three layers, and the order of work that gets there. Audience: Fritz and anyone deciding what demarkus builds next. Written 2026-09-25; the plans it links carry the detail.

The end state

Knowledge that knows when it is stale.

demarkus is the shared, live, versioned memory for people and agents. Every document has a stable address, an immutable history, and provenance. Documents link to each other with typed relations. When a document changes, everything that depends on it finds out within a latency budget: the agents that cited it, the summaries built from it, the caches holding it, and the people reading it.

The test for any proposal comes from Information Management as the core principle: does it make knowledge more linked, more findable, more durable? This vision adds a fourth word: more current.

What is new here

Each neighbor has part of it and none has all of it:

System Has Lacks
git History, content addressing Live changes, per-document network addresses, a catalog, agent-native access
Wikis and document tools Human editing, links A protocol, provenance per claim, agent-native access
Vector memory for agents Retrieval Auditability, provenance, human-readable storage
Message buses Live delivery Durable, addressable, versioned content

The combination demarkus offers is provenance plus live change plus typed links. That combination is what makes knowledge reactive.

Reactive knowledge

What becomes possible once changes propagate and links carry meaning:

  • Citations become subscriptions. A document or agent that links to or declares a rel-* relation to a source watches it. Superseding an ADR notifies everything that depends on it.
  • Derived documents track their sources, like a build system. Hubs, summaries, indexes, and graph.md record what they were built from; a change to a source marks them stale or rebuilds them. ADR 0013 (graph source freshness) and the generation package are early forms.
  • The organization's current beliefs can be queried and watched. superseded-by, valid-from and valid-until, and live LOOKUP over decisions answer "what do we believe now, and tell me when it changes."
  • Agents coordinate on the documents themselves. Mailboxes, claims by expected version, and handoffs leave a versioned, attributed record. The conversation is the record.
  • Trust crosses organizations. Client-verifiable history plus content addressing lets one organization rely on another's knowledge without trusting its server.

Three layers

Each layer has one rule. Nothing moves down a layer without an ADR.

Layer Rule Contents
Durable core Stays dumb. Stores, versions, catalogs, and announces changes as hints. Protocol and SPEC, stores, WATCH hints, replica commit hints, protocol version in ALPN
Live layer Changes propagate within a latency budget. QUIC, the QUIC broker with fan-out and content cache, the client SDK, 0-RTT, UDP 443
Reactive layer Links and provenance drive staleness, rebuilding, and coordination, at the edge. client/live, agents and skills, the HTTPS adapter, the library

The durable core test still governs the bottom layer: would this matter if every product on top were thrown away tomorrow? The live and reactive layers are products; they can be rebuilt, the core cannot.

Architecture at the end state

  • Worlds hold knowledge, fenced inside the network on 6309, and send hints only.
  • The QUIC broker on UDP 443 is the one broker that does the work: identity, tenancy, routing, fan-out, content cache, mark and h3 on one listener.
  • The HTTPS adapter on TCP 443 translates for clients that need TCP and HTTP: hosted MCP platforms, UDP-blocked enterprises, OAuth web login. It is deleted once those clients reach QUIC.
  • The client SDK is how every application, CLI, MCP bridge, and the library speak to demarkus.
  • Souls stay local, personal, and promote into the knowledge system.

Details: Lean into QUIC, Realtime demarkus, demarkus-sdk.

Order of work

The 1.0 feature freeze was lifted on 2026-09-25 (1.0 plan). The live layer proceeds alongside the 1.0 workstreams instead of after them.

Track Contents
1.0 stability The 1.0 plan's remaining workstreams: quality gate, one vocabulary, reduced surface, protocol final.
Live, now QUIC measurement baseline on the benchmarks branch; SDK extraction; WATCH and replica push; ALPN mark/1; then the QUIC broker with fan-out, 0-RTT, and UDP 443.
Reactive, next Dependency subscriptions, derived documents that know they are stale, watchable beliefs, client/live in every surface, the HTTPS broker reduced to an adapter.
Later Verifiable federation across organizations, the pure-QUIC profile as a default, the adapter phase-out, level 3 collaboration as a separate room service.

Within the live track the order is fixed by dependencies: SDK before the client side of WATCH, WATCH before the QUIC broker's fan-out, the auth-model ADR before the QUIC broker.

The live track started on 2026-09-27: ADR 0026 (SDK phase 0), realtime PR 1 (client error classification) and PR 2 (ADR 0025, SPEC §6.8, the WATCH codec) are on branch feat/live-track-front awaiting review.

Tensions to settle

  • When the protocol freezes. Settled 2026-09-27 in ADR 0025, proposed: protocol 1.0 freezes after WATCH and ALPN mark/1 land, so the spec is not frozen and extended in the same cycle.
  • Lockstep versions and the SDK. Settled 2026-09-27 in ADR 0026, proposed: the SDK has its own semver in its own repo, v0 until WATCH ships, v1 after; the lockstep (W2) covers the monorepo modules only, and core 1.0 pins an SDK v1.
  • Subscriptions and "no sessions". A watch looks like a session. It stays defensible only while events are hints, subscribers hold no durable server state, and overload degrades to resync. ADR 0025 says so.
  • Surface count. Knowledge broker, memory broker, QUIC broker, several plugins, the library, the TUI. Convergence on the QUIC broker helps; the ecosystem plan's budgets and default-surface rule still apply to everything the new plans expose.
  • Measurement may say no. If the QUIC baseline shows no gain on most hops, the live layer still ships, and a TLS on TCP binding for direct clients becomes the honest answer.

What this vision rules out

  • Full-text or semantic search in the server. It stays a sidecar that reads over LIST and FETCH.
  • Graphs as the primary store. Markdown is primary; graphs are derived.
  • Application features that cannot be expressed as documents and links: forms, sessions beyond watches, RPC endpoints.
  • Chat through the store. Documents are for durable coordination.
trail
  1. soul.demarkus.io:6309 v24
  2. vision
from here → 0013-graph-source-freshness 0025-watch-verb 0026-client-sdk-mit-server-never-imports demarkus-sdk durable-core ecosystem-optimization