# 2026-06-16 — Knowledge ingestion pipeline design

Designed the knowledge ingestion pipeline end to end in one session. Plan recorded at [/plans/knowledge-ingestion.md](/plans/knowledge-ingestion.md) (v6, design-complete).

## What it is

How org knowledge flows into a knowledge destination, and how the demarkus-memory (soul) and demarkus-knowledge plugins compose into one curation pipeline. Core frame: soul = write-ahead / staging tier (cheap, high-volume capture); knowledge = published read-model (curated, authoritative). Promotion from soul to knowledge IS the curation pipeline. The soul is the universal staging tier — external sources (Confluence, Slack, Jira, meetings) land in an ingestion soul and reuse the same gate.

## Decisions made (all four forks resolved)

1. Knowledge destination is curated-only; the soul holds the raw/retrieval tier.
2. Human-in-the-loop by default; autonomy is a graded knob (`human-only` / `verify-then-auto` / `auto`) capped by the destination's well-known descriptor, not chosen freely by the client. Coupled to model tier.
3. World partitioning is discovered, not designed — it falls out of the access topology (read and write, token-encoded; broad read, narrow write) plus per-world well-known descriptors declaring team, domain, partition role, autonomy ceiling.
4. The promote primitive is a detection-gated bridge owned by neither plugin: memory contributes intent/source and stays broker-unaware; knowledge contributes execution (gate, dedup, endpoint write, destination selection). Mutual read-only detection lights it up. Back-stamp is one-directional and also refreshes the soul to the settled version.

## Key design moves

- Model routing is a cascade: Haiku triages the firehose (~95% discard), the strong model distills only survivors, a human gates authority. Cost tracks volume, concentrated at the cheap stages.
- "Knowledge destination" generalized to an endpoint abstraction — a broker-fronted system OR a plain remote demarkus server. The Library reads across all.
- Capability tiers gated on mutual detection: memory-only (personal KB, promote dormant), knowledge-only (publisher/consumer, no triggers — triggers are inherently memory-side because they observe the staging tier), memory+knowledge (full bridge).
- Coherence edge (knowledge to soul): directional refresh, never a symmetric merge. Link-not-copy stubs, version-stamped cache invalidation by etag/content-hash, local edits re-promote as gated updates via the existing `mark_publish` conflict-merge.

## Build prerequisites before phase 0

Promote primitive; access discovery (writable-worlds surface on the broker + token-grant introspection for plain remote servers); per-world `world.md` well-known descriptor; the coherence edge.

## Next

Scope the promote primitive — the load-bearing prerequisite the others feed into. Dogfood: promote this plan itself, soul to knowledge, as the pipeline's first real run.