Knowledge Ingestion Pipeline

A narrative and design for how organizational knowledge flows into the demarkus knowledge system, and how the two Claude Code plugins (demarkus-memory / the soul, and demarkus-knowledge / the shared catalog) compose into a single curation pipeline. Status: draft, unstarted. The forks in the Open Decisions section need resolving before any build.

The thesis: soul = write-ahead log, knowledge system = published read-model

The soul and the knowledge system are not two stores that happen to coexist. They are two stages of one refinement pipeline.

• Soul is the capture tier: cheap, continuous, private, high-volume, low bar to write. Agents already record here as they work (journals, debugging lessons, ADRs, patterns). Noisy but rich. It is meant to be a firehose, and that is correct.
• Knowledge system is the curated index: high bar, shared, authoritative, deduped, tagged to the org taxonomy, low-volume relative to the soul.
• The promotion from soul to knowledge system IS the curation pipeline. The model-routing and gate decisions below apply to this internal flow exactly as they apply to external sources.

This is CQRS / write-ahead-log thinking applied to knowledge: append-heavy cheap capture, periodic compaction into a queryable authoritative read-model. We already built the front of the pipeline. It is the soul. What is missing is the promotion lane on top of it.

The unifying move: the soul is the universal staging tier

External sources (Confluence, Slack, Jira, meeting notes) do not land in the knowledge system directly. They land in a soul (an ingestion soul) first. Then the same promotion gate that curates agent-authored soul content also curates externally-ingested content. One gate, many inflows.

INFLOWS                          STAGING (soul tier)        GATE                       CATALOG (knowledge)
agents recording        ───────► personal souls ──┐
Confluence ETL          ───────► ingestion soul ───┼─► curation gate ───────────────► knowledge worlds
Slack/Jira/mtg distill  ───────► ingestion soul ───┘   triage→distill→dedup→tag→        authoritative,
                                                       human approve                     deduped, provenance

Soul is the universal write-buffer. The knowledge system is the published read-model. The gate is built once and reused for every inflow.

The narrative: life of one fact

1. Capture. An agent fixes a gnarly bug and records the root cause to its local soul debugging.md. Immediate, private, cheap. This already works.
2. Accumulate. Days pass; the soul gathers journals, patterns, ADRs. Some trivia, some durable and broadly useful. Uncurated.
3. Trigger. The fact gets recalled repeatedly, or it is a high-signal type (an ADR), or a periodic sweep flags it. It becomes a promotion candidate.
4. Triage (cheap model, firehose volume): durable? broadly useful? not already in the catalog? Most soul content fails here and correctly stays personal.
5. Distill and dedup (capable model, low volume): rewrite for a shared audience, strip personal and local framing, check overlap and conflict against existing catalog docs.
6. Tag (cheap model): assign category: plus org tags from the closed taxonomy.
7. Gate (human): approve. The authority check.
8. Publish to the knowledge world, with provenance linking back to the soul origin (and the external source if any). The soul doc is stamped promoted.
9. Consume. Any agent now finds it via mark_lookup, authoritative and shared. The originating soul links to it rather than keeping a stale fork.

External sources enter the same lifecycle at step 1, with an ETL/distill in place of an agent.

Model routing: cascade, do not pick one model

Curation is not one task. Decompose by difficulty and volume, then route each sub-task to the cheapest model that clears the bar. Cost tracks volume, and volume is concentrated at the cheap stages.

Run the cheap model on the firehose as the triage gate (discards ~95% of Slack/Jira noise cheaply). Run the capable model only on the survivors. Never run the expensive model over every message.

The lever that lets Haiku go further is gate strength. A strong human gate makes Haiku-drafted distillations safe (a human polishes an 80%-good draft). An optional Haiku refute-check (a second cheap call that tests the distillation against its source) pushes strong-model usage lower still. Auto-publish with no gate is the only mode that forces the strong model, and it should not be used into an authoritative store.

The connective tissue the two plugins are missing

Today the plugins compose only by convention ("draft in soul, promote when ready"). There is no mechanism. The build list is this glue.

1. A promote primitive. Mark a soul doc as a candidate; a promote operation re-tags to the shared taxonomy, dedups against the catalog, and publishes with provenance. This is the single missing piece between the plugins.
2. Two-way provenance. The knowledge doc links to its soul/external origin. The soul doc is stamped promoted → mark://world/path so it is not re-promoted and is not duplicated. This enforces the existing "do not copy authoritative knowledge into the soul" rule mechanically.
3. Trigger policy. Manual ("ready for others") plus signal-based (recall-count, doc type) plus a batch sweep (a cron agent scans souls, runs the cascade, queues candidates for approval).
4. Dedup, staged. First dedup against the catalog (cheap, do now). Defer dedup across souls (entity resolution, hard) until the catalog flow is proven.
5. Taxonomy translation. Soul tags are loose and personal; the knowledge system requires a category: axis and policy compliance. Promotion re-tags. Haiku does this (closed-set classification is its sweet spot).

Source taxonomy (external inflows)

The external sources are not the same shape, and they need different handling before they reach the gate.

Confluence is doc-shaped and converts deterministically (own sub-plan; a tools/demarkus-import Go binary using client/fetch.Client.Publish, pure-Go or pandoc converter, two-pass link rewrite, idempotency manifest). Slack/Jira/meetings are conversation-shaped: worthless raw, value locked in threads that distill to a paragraph. They are the reason the distillation stage exists.

Phasing

Sequence by fit, not by which API is easiest.

1. Confluence first. Doc-shaped, deterministic, highest signal. Proves the publish path, tagging, and provenance at lowest risk.
2. Meeting notes. Bounded volume, decision-dense. First real test of the distillation plus human-gate loop.
3. Jira. Outcomes-only extractor. Teaches entity-resolution against the existing Confluence docs.
4. Slack last. Lowest fit, highest volume, hardest privacy story. By now the gate is mature enough to be ruthless.

Underlying all of this: the soul-to-knowledge promotion lane is phase 0, because it is the gate every external source reuses. Build the gate against soul content first (it already exists and is safe to experiment on), then point the external ETLs at it.

Risks and things to watch

• Most soul content must never promote. If more than ~10–20% crosses the gate, the bar is too low and the catalog becomes a soul mirror, destroying the curation value. Ruthlessness is the feature.
• Fork staleness. Promotion forks knowledge. Solve with link-not-copy, a promoted marker, and supersede-propagation, or the soul accumulates stale copies of authoritative docs.
• Authority needs an owner. The human gate per world maps to policy.md's owner:. Decide who approves before wiring triggers.
• Signal-based triggers need telemetry that may not exist yet (recall counts). Start with manual plus doc-type triggers; add signal-based once recall is instrumented.
• ACL to world boundary. Source resources have access controls; worlds have read/write grants. A private channel must not flow into a world everyone reads. This is governance, not code.
• Privacy/PII. Never publish PII or secrets to a shared world (policy already forbids it). Distillation must strip, not merely summarize.

Open decisions (resolve before build)

1. Is the knowledge system the curated tier only, or also the raw retrieval tier? Decided so far: curated-only, to keep it useful. Confirm and record.
2. Human-in-the-loop or auto-publish for distillations? Drives the entire gate design and the safe model tier. Leaning human-in-the-loop at least initially.
3. One world per source, one per team/space, or one per access-domain? Drives the ACL mapping.
4. Where does the promote primitive live — in the demarkus-memory plugin, the demarkus-knowledge plugin, or a new shared component? It is the connective tissue between two currently-independent plugins.

Dogfood

Once the forks are resolved, promote this very plan from the soul to the knowledge system as the first exercise of the pipeline it describes.