2026-06-16 — Knowledge ingestion pipeline design

Designed the knowledge ingestion pipeline end to end in one session. Plan recorded at /plans/knowledge-ingestion.md (v6, design-complete).

What it is

How org knowledge flows into a knowledge destination, and how the demarkus-memory (soul) and demarkus-knowledge plugins compose into one curation pipeline. Core frame: soul = write-ahead / staging tier (cheap, high-volume capture); knowledge = published read-model (curated, authoritative). Promotion from soul to knowledge IS the curation pipeline. The soul is the universal staging tier — external sources (Confluence, Slack, Jira, meetings) land in an ingestion soul and reuse the same gate.

Decisions made (all four forks resolved)

Knowledge destination is curated-only; the soul holds the raw/retrieval tier.
Human-in-the-loop by default; autonomy is a graded knob (human-only / verify-then-auto / auto) capped by the destination's well-known descriptor, not chosen freely by the client. Coupled to model tier.
World partitioning is discovered, not designed — it falls out of the access topology (read and write, token-encoded; broad read, narrow write) plus per-world well-known descriptors declaring team, domain, partition role, autonomy ceiling.
The promote primitive is a detection-gated bridge owned by neither plugin: memory contributes intent/source and stays broker-unaware; knowledge contributes execution (gate, dedup, endpoint write, destination selection). Mutual read-only detection lights it up. Back-stamp is one-directional and also refreshes the soul to the settled version.

Key design moves

Model routing is a cascade: Haiku triages the firehose (~95% discard), the strong model distills only survivors, a human gates authority. Cost tracks volume, concentrated at the cheap stages.
"Knowledge destination" generalized to an endpoint abstraction — a broker-fronted system OR a plain remote demarkus server. The Library reads across all.
Capability tiers gated on mutual detection: memory-only (personal KB, promote dormant), knowledge-only (publisher/consumer, no triggers — triggers are inherently memory-side because they observe the staging tier), memory+knowledge (full bridge).
Coherence edge (knowledge to soul): directional refresh, never a symmetric merge. Link-not-copy stubs, version-stamped cache invalidation by etag/content-hash, local edits re-promote as gated updates via the existing mark_publish conflict-merge.

Build prerequisites before phase 0

Promote primitive; access discovery (writable-worlds surface on the broker + token-grant introspection for plain remote servers); per-world world.md well-known descriptor; the coherence edge.

Scope the promote primitive — the load-bearing prerequisite the others feed into. Dogfood: promote this plan itself, soul to knowledge, as the pipeline's first real run.

2026-06-16 — Knowledge ingestion pipeline design

Designed the knowledge ingestion pipeline end to end in one session. Plan recorded at /plans/knowledge-ingestion.md (v6, design-complete).

What it is

Decisions made (all four forks resolved)

Knowledge destination is curated-only; the soul holds the raw/retrieval tier.
Human-in-the-loop by default; autonomy is a graded knob (human-only / verify-then-auto / auto) capped by the destination's well-known descriptor, not chosen freely by the client. Coupled to model tier.
World partitioning is discovered, not designed — it falls out of the access topology (read and write, token-encoded; broad read, narrow write) plus per-world well-known descriptors declaring team, domain, partition role, autonomy ceiling.
The promote primitive is a detection-gated bridge owned by neither plugin: memory contributes intent/source and stays broker-unaware; knowledge contributes execution (gate, dedup, endpoint write, destination selection). Mutual read-only detection lights it up. Back-stamp is one-directional and also refreshes the soul to the settled version.

Key design moves

Model routing is a cascade: Haiku triages the firehose (~95% discard), the strong model distills only survivors, a human gates authority. Cost tracks volume, concentrated at the cheap stages.
"Knowledge destination" generalized to an endpoint abstraction — a broker-fronted system OR a plain remote demarkus server. The Library reads across all.
Capability tiers gated on mutual detection: memory-only (personal KB, promote dormant), knowledge-only (publisher/consumer, no triggers — triggers are inherently memory-side because they observe the staging tier), memory+knowledge (full bridge).
Coherence edge (knowledge to soul): directional refresh, never a symmetric merge. Link-not-copy stubs, version-stamped cache invalidation by etag/content-hash, local edits re-promote as gated updates via the existing mark_publish conflict-merge.

Build prerequisites before phase 0

Promote primitive; access discovery (writable-worlds surface on the broker + token-grant introspection for plain remote servers); per-world world.md well-known descriptor; the coherence edge.

Scope the promote primitive — the load-bearing prerequisite the others feed into. Dogfood: promote this plan itself, soul to knowledge, as the pipeline's first real run.

2026-06-16 — Knowledge ingestion pipeline design

What it is

Decisions made (all four forks resolved)

Key design moves

Build prerequisites before phase 0

Next

2026-06-16 — Knowledge ingestion pipeline design

What it is

Decisions made (all four forks resolved)

Key design moves

Build prerequisites before phase 0

Next