Knowledge Ingestion Pipeline

A narrative and design for how organizational knowledge flows into a demarkus knowledge destination, and how the two Claude Code plugins (demarkus-memory / the soul, and demarkus-knowledge / the shared catalog) compose into a single curation pipeline. Status: draft, unstarted. The forks in the Open Decisions section need resolving before any build.

The thesis: soul = write-ahead log, knowledge destination = published read-model

The soul and the knowledge destination are not two stores that happen to coexist. They are two stages of one refinement pipeline.

• Soul is the capture tier: cheap, continuous, private, high-volume, low bar to write. Agents already record here as they work (journals, debugging lessons, ADRs, patterns). Noisy but rich. It is meant to be a firehose, and that is correct.
• Knowledge destination is the curated index: high bar, shared, authoritative, deduped, tagged to the org taxonomy, low-volume relative to the soul.
• The promotion from soul to the knowledge destination IS the curation pipeline. The model-routing and gate decisions below apply to this internal flow exactly as they apply to external sources.

This is CQRS / write-ahead-log thinking applied to knowledge: append-heavy cheap capture, periodic compaction into a queryable authoritative read-model. We already built the front of the pipeline. It is the soul. What is missing is the promotion lane on top of it.

The unifying move: the soul is the universal staging tier

External sources (Confluence, Slack, Jira, meeting notes) do not land in the knowledge destination directly. They land in a soul (an ingestion soul) first. Then the same promotion gate that curates agent-authored soul content also curates externally-ingested content. One gate, many inflows.

INFLOWS                          STAGING (soul tier)        GATE                       CATALOG (knowledge)
agents recording        ───────► personal souls ──┐
Confluence ETL          ───────► ingestion soul ───┼─► curation gate ───────────────► knowledge endpoint
Slack/Jira/mtg distill  ───────► ingestion soul ───┘   triage→distill→dedup→tag→        authoritative,
                                                       human approve                     deduped, provenance

Soul is the universal write-buffer. The knowledge destination is the published read-model. The gate is built once and reused for every inflow.

The bridge: detection-gated capability tiers

Promote is not owned by either plugin. It is the bridge that mutual detection creates between two roles, and it activates only when both roles are present.

Two roles the bridge connects.

1. Memory (demarkus-memory) — the personal knowledge base. Its own backing store is transport-flexible: local or a remote demarkus server. This is the baseline and is fully usable on its own.
2. Knowledge destination (demarkus-knowledge) — the shared/team target. This is not only the broker. It is an abstraction over two transports, and promote must work against either:
   • a broker-fronted knowledge system — multi-world, OIDC/OAuth, the org-wide hub; and
   • a plain remote demarkus server — a single remote world, direct QUIC, no broker.

So "knowledge system / base" means any demarkus write target that is not your personal soul. The promote primitive targets an endpoint, not specifically the broker. That generalization is load-bearing: everything below is written against endpoints, not against the broker.

The unifying abstraction: a demarkus endpoint. An endpoint is "a demarkus server you can read or write," reached either directly (QUIC, single world) or through a broker (HTTPS to QUIC, multi-world system). What varies across endpoints is transport, auth, and single- versus multi-world. Memory, Knowledge, and the Library (the demarkus-library web reading room, which must read across all of them) all speak to endpoints. Promote simply targets one that is not your soul.

Capability tiers, gated on mutual detection. Detection is mutual and read-only: memory recognizes a knowledge plugin, and the knowledge plugin recognizes memory (a conf-peek in both directions, mirroring the one-way peek that exists today). Each adapts to the other's presence.

Triggers are inherently memory-side. The "this seems important — promote it?" prompts (manual, signal-based on recall-count or doc type, and the batch sweep) all observe the staging tier. With no soul there is nothing to observe, so the knowledge-only tier has no triggers by construction: it is deliberate manual authoring, not staged promotion. The cascade exists to lift low-bar soul content up to the high bar; when you author directly into the catalog you are already at the high bar and skip it.

Where the code lives follows the bridge, not one plugin. The operation splits along its natural seam.

• Memory side (intent / source): detect a knowledge endpoint, mark a candidate, read the source doc, apply the back-stamp. Memory stays broker-unaware throughout — it only detects that an endpoint exists and hands off; it never talks to the broker itself.
• Knowledge side (execution): triage, distill, dedup against the catalog, taxonomy translation, the policy gate, the endpoint write, and destination selection (which world, filtered to the caller's writer allow-set — readable is not writable).
• Back-stamp stays one-directional: the knowledge side reports the resulting mark:// back, and memory applies its own promoted stamp. The knowledge plugin never writes the soul.

The batch sweep (below) is orthogonal: it automates the memory-side "mark a candidate" step at volume, and remains a separate agent/daemon rather than plugin code.

The narrative: life of one fact

1. Capture. An agent fixes a gnarly bug and records the root cause to its local soul debugging.md. Immediate, private, cheap. This already works.
2. Accumulate. Days pass; the soul gathers journals, patterns, ADRs. Some trivia, some durable and broadly useful. Uncurated.
3. Trigger. The fact gets recalled repeatedly, or it is a high-signal type (an ADR), or a periodic sweep flags it. It becomes a promotion candidate.
4. Triage (cheap model, firehose volume): durable? broadly useful? not already in the catalog? Most soul content fails here and correctly stays personal.
5. Distill and dedup (capable model, low volume): rewrite for a shared audience, strip personal and local framing, check overlap and conflict against existing catalog docs.
6. Tag (cheap model): assign category: plus org tags from the closed taxonomy.
7. Gate (human): approve. The authority check.
8. Publish to the chosen knowledge endpoint, with provenance linking back to the soul origin (and the external source if any). The soul doc is stamped promoted.
9. Consume. Any agent now finds it via mark_lookup, authoritative and shared. The originating soul links to it rather than keeping a stale fork.

External sources enter the same lifecycle at step 1, with an ETL/distill in place of an agent.

Model routing: cascade, do not pick one model

Curation is not one task. Decompose by difficulty and volume, then route each sub-task to the cheapest model that clears the bar. Cost tracks volume, and volume is concentrated at the cheap stages.

Run the cheap model on the firehose as the triage gate (discards ~95% of Slack/Jira noise cheaply). Run the capable model only on the survivors. Never run the expensive model over every message.

The lever that lets Haiku go further is gate strength. A strong human gate makes Haiku-drafted distillations safe (a human polishes an 80%-good draft). An optional Haiku refute-check (a second cheap call that tests the distillation against its source) pushes strong-model usage lower still. Auto-publish with no gate is the only mode that forces the strong model, and it should not be used into an authoritative store.

The connective tissue the two plugins are missing

Today the plugins compose only by convention ("draft in soul, promote when ready"). There is no mechanism. The build list is this glue.

1. A promote primitive. Mark a soul doc as a candidate; a promote operation re-tags to the shared taxonomy, dedups against the catalog, and publishes with provenance to a chosen endpoint. It is a detection-gated bridge, not owned by either plugin (see "The bridge: detection-gated capability tiers"): memory contributes detect, candidate-mark, source-read, and back-stamp; knowledge contributes the gate, dedup, endpoint write, and destination selection.
2. Two-way provenance. The knowledge doc links to its soul/external origin. The soul doc is stamped promoted → mark://world/path so it is not re-promoted and is not duplicated. This enforces the existing "do not copy authoritative knowledge into the soul" rule mechanically.
3. Trigger policy. Memory-side only (there is no staging tier to watch without a soul). Manual ("ready for others") plus signal-based (recall-count, doc type) plus a batch sweep (a cron agent scans souls, runs the cascade, queues candidates for approval).
4. Dedup, staged. First dedup against the catalog (cheap, do now). Defer dedup across souls (entity resolution, hard) until the catalog flow is proven.
5. Taxonomy translation. Soul tags are loose and personal; the knowledge destination requires a category: axis and policy compliance. Promotion re-tags. Haiku does this (closed-set classification is its sweet spot).
6. Writable-worlds discovery. Destination-selection needs the caller's writer allow-set, but mark_worlds lists readable worlds only (the #189 distinction). Add either a "worlds I can write" discovery surface or an accepted probe-on-promote (attempt the write, handle denial). Prerequisite for routing.

Autonomy: human-in-the-loop by default, capped by the destination

Promotion autonomy is a graded knob, not a local boolean. Authority over a shared authoritative store belongs to its owner, so auto-publish cannot be a purely client-side toggle — otherwise any client flips to auto and floods the catalog.

Model it as two levels:

• Destination ceiling, set by the world owner in the world's well-known descriptor (see "Partitioning"): human-only / verify-then-auto / auto. The org root caps at human-only; a personal remote endpoint or a low-stakes scratch world may allow auto.
• Local preference, applied as min(local preference, destination ceiling). Default is human-in-the-loop.

Autonomy is coupled to model tier (see Model routing): auto requires the strong model plus the refute-check; human-in-the-loop is exactly what makes a cheaper Haiku draft safe. Start everything at human-only; relax per-destination only once the gate and verification are proven.

Partitioning: discovered, not designed

Where knowledge lands is not a partition scheme the pipeline invents. It is already determined by the access topology and by each world's self-description. The pipeline conforms to it; it does not define it. Two layers decide the destination.

1. Access (hard constraint). You can read broadly across a knowledge system but write only to your writer allow-set (readable is not writable — the #189 distinction). Write access alone filters the candidate worlds for any promotion.
2. Well-known hints (soft routing). Each world self-describes via a per-world well-known descriptor, alongside the existing policy.md and template.md — e.g. mark://<world>/.well-known/demarkus/world.md: owning team, subject domain, partition role, and its autonomy ceiling (the same ceiling the Autonomy section needs). Destination-selection reads these to route a candidate to the right world among the ones you can write.

Source of truth is the world owner — each world declares its own scope, self-sovereign. The root hub may aggregate descriptors into a discovery index, but it must be derived (from the per-world descriptors or the federation crawler), never hand-maintained, or the central map drifts from what worlds actually hold.

Destination-selection algorithm. Enumerate writable worlds (needs the writable-worlds primitive above) → read each one's descriptor → either auto-route (match the candidate's subject/team to a world's declared domain) or present a labeled, write-filtered list for the human to pick.

Source taxonomy (external inflows)

The external sources are not the same shape, and they need different handling before they reach the gate.

Confluence is doc-shaped and converts deterministically (own sub-plan; a tools/demarkus-import Go binary using client/fetch.Client.Publish, pure-Go or pandoc converter, two-pass link rewrite, idempotency manifest). Slack/Jira/meetings are conversation-shaped: worthless raw, value locked in threads that distill to a paragraph. They are the reason the distillation stage exists.

Phasing

Sequence by fit, not by which API is easiest.

1. Confluence first. Doc-shaped, deterministic, highest signal. Proves the publish path, tagging, and provenance at lowest risk.
2. Meeting notes. Bounded volume, decision-dense. First real test of the distillation plus human-gate loop.
3. Jira. Outcomes-only extractor. Teaches entity-resolution against the existing Confluence docs.
4. Slack last. Lowest fit, highest volume, hardest privacy story. By now the gate is mature enough to be ruthless.

Underlying all of this: the soul-to-knowledge promotion lane is phase 0, because it is the gate every external source reuses. Build the gate against soul content first (it already exists and is safe to experiment on), then point the external ETLs at it.

Risks and things to watch

• Most soul content must never promote. If more than ~10–20% crosses the gate, the bar is too low and the catalog becomes a soul mirror, destroying the curation value. Ruthlessness is the feature.
• Fork staleness. Promotion forks knowledge. Solve with link-not-copy, a promoted marker, and supersede-propagation, or the soul accumulates stale copies of authoritative docs.
• Authority needs an owner. The human gate per world maps to policy.md's owner:. Decide who approves before wiring triggers.
• Signal-based triggers need telemetry that may not exist yet (recall counts). Start with manual plus doc-type triggers; add signal-based once recall is instrumented.
• ACL to world boundary. Source resources have access controls; worlds have read/write grants. A private channel must not flow into a world everyone reads. This is governance, not code. The per-world descriptor plus writer allow-set is the enforcement surface.
• Privacy/PII. Never publish PII or secrets to a shared world (policy already forbids it). Distillation must strip, not merely summarize.

Open decisions (resolve before build)

1. Is the knowledge destination the curated tier only, or also the raw retrieval tier? Decided: curated-only, to keep it useful. The soul (and external ingestion souls) hold the raw/retrieval tier.
2. Human-in-the-loop or auto-publish for distillations? Decided: human-in-the-loop by default; autonomy is a graded knob capped by the destination, not chosen freely by the client (see "Autonomy"). Destination owner sets a ceiling (human-only / verify-then-auto / auto) in the world's well-known descriptor; local config picks at or below it; default human-only. Coupled to model tier.
3. How are worlds partitioned for ingestion? Decided: partitioning is discovered, not designed (see "Partitioning"). It falls out of the write-access topology (broad read, narrow write) plus per-world well-known descriptors declaring team, domain, partition role, and autonomy ceiling. The pipeline enumerates writable worlds and routes by their declared domain. Prerequisite: a writable-worlds discovery primitive (mark_worlds lists readable only, per #189) or accepted probe-on-promote.
4. Where does the promote primitive live? Resolved: a detection-gated bridge capability, owned by neither plugin unilaterally (see "The bridge"). Memory contributes the intent/source half and stays broker-unaware; the knowledge plugin contributes the execution half (gate, dedup, endpoint write, destination selection). Mutual read-only detection lights it up. The destination is a parameter — a team world, the org system root, or a plain remote demarkus server — filtered to the caller's writer allow-set. The batch sweep remains a separate agent. Remaining sub-decision: the exact back-stamp mechanism (one-directional via a memory-honored marker, preferred) versus a single narrow knowledge→soul provenance write.

Dogfood

Once the forks are resolved, promote this very plan from the soul to the knowledge destination as the first exercise of the pipeline it describes.