Knowledge Ingestion Pipeline

A narrative and design for how organizational knowledge flows into a demarkus knowledge destination, and how the two Claude Code plugins (demarkus-memory / the soul, and demarkus-knowledge / the shared catalog) compose into a single curation pipeline. Status: draft, unstarted. All design forks are resolved (see Decisions); what remains is build prerequisites — the promote primitive, access discovery, and the coherence edge — not open questions.

The thesis: soul = write-ahead log, knowledge destination = published read-model

The soul and the knowledge destination are not two stores that happen to coexist. They are two stages of one refinement pipeline.

• Soul is the capture tier: cheap, continuous, private, high-volume, low bar to write. Agents already record here as they work (journals, debugging lessons, ADRs, patterns). Noisy but rich. It is meant to be a firehose, and that is correct.
• Knowledge destination is the curated index: high bar, shared, authoritative, deduped, tagged to the org taxonomy, low-volume relative to the soul.
• The promotion from soul to the knowledge destination IS the curation pipeline. The model-routing and gate decisions below apply to this internal flow exactly as they apply to external sources.

This is CQRS / write-ahead-log thinking applied to knowledge: append-heavy cheap capture, periodic compaction into a queryable authoritative read-model. We already built the front of the pipeline. It is the soul. What is missing is the promotion lane on top of it.

The unifying move: the soul is the universal staging tier

External sources (Confluence, Slack, Jira, meeting notes) do not land in the knowledge destination directly. They land in a soul (an ingestion soul) first. Then the same promotion gate that curates agent-authored soul content also curates externally-ingested content. One gate, many inflows.

INFLOWS                          STAGING (soul tier)        GATE                       CATALOG (knowledge)
agents recording        ───────► personal souls ──┐
Confluence ETL          ───────► ingestion soul ───┼─► curation gate ───────────────► knowledge endpoint
Slack/Jira/mtg distill  ───────► ingestion soul ───┘   triage→distill→dedup→tag→        authoritative,
                                                       human approve                     deduped, provenance

Soul is the universal write-buffer. The knowledge destination is the published read-model. The gate is built once and reused for every inflow. The reverse edge — keeping the soul fresh as the published copy evolves — is the Coherence section.

The bridge: detection-gated capability tiers

Promote is not owned by either plugin. It is the bridge that mutual detection creates between two roles, and it activates only when both roles are present.

Two roles the bridge connects.

1. Memory (demarkus-memory) — the personal knowledge base. Its own backing store is transport-flexible: local or a remote demarkus server. This is the baseline and is fully usable on its own.
2. Knowledge destination (demarkus-knowledge) — the shared/team target. This is not only the broker. It is an abstraction over two transports, and promote must work against either:
   • a broker-fronted knowledge system — multi-world, OIDC/OAuth, the org-wide hub; and
   • a plain remote demarkus server — a single remote world, direct QUIC, no broker.

So "knowledge system / base" means any demarkus write target that is not your personal soul. The promote primitive targets an endpoint, not specifically the broker. That generalization is load-bearing: everything below is written against endpoints, not against the broker.

The unifying abstraction: a demarkus endpoint. An endpoint is "a demarkus server you can read or write," reached either directly (QUIC, single world) or through a broker (HTTPS to QUIC, multi-world system). What varies across endpoints is transport, auth, and single- versus multi-world. Memory, Knowledge, and the Library (the demarkus-library web reading room, which must read across all of them) all speak to endpoints. Promote simply targets one that is not your soul.

Capability tiers, gated on mutual detection. Detection is mutual and read-only: memory recognizes a knowledge plugin, and the knowledge plugin recognizes memory (a conf-peek in both directions, mirroring the one-way peek that exists today). Each adapts to the other's presence.

Triggers are inherently memory-side. The "this seems important — promote it?" prompts (manual, signal-based on recall-count or doc type, and the batch sweep) all observe the staging tier. With no soul there is nothing to observe, so the knowledge-only tier has no triggers by construction: it is deliberate manual authoring, not staged promotion. The cascade exists to lift low-bar soul content up to the high bar; when you author directly into the catalog you are already at the high bar and skip it.

Where the code lives follows the bridge, not one plugin. The operation splits along its natural seam.

• Memory side (intent / source): detect a knowledge endpoint, mark a candidate, read the source doc, apply the back-stamp. Memory stays broker-unaware throughout — it only detects that an endpoint exists and hands off; it never talks to the broker itself.
• Knowledge side (execution): triage, distill, dedup against the catalog, taxonomy translation, the policy gate, the endpoint write, and destination selection (which world, filtered to the caller's writer allow-set — readable is not writable).
• Back-stamp stays one-directional: the knowledge side reports the resulting mark:// back, and memory applies its own promoted stamp and refresh. The knowledge plugin never writes the soul.

The batch sweep (below) is orthogonal: it automates the memory-side "mark a candidate" step at volume, and remains a separate agent/daemon rather than plugin code.

The narrative: life of one fact

1. Capture. An agent fixes a gnarly bug and records the root cause to its local soul debugging.md. Immediate, private, cheap. This already works.
2. Accumulate. Days pass; the soul gathers journals, patterns, ADRs. Some trivia, some durable and broadly useful. Uncurated.
3. Trigger. The fact gets recalled repeatedly, or it is a high-signal type (an ADR), or a periodic sweep flags it. It becomes a promotion candidate.
4. Triage (cheap model, firehose volume): durable? broadly useful? not already in the catalog? Most soul content fails here and correctly stays personal.
5. Distill and dedup (capable model, low volume): rewrite for a shared audience, strip personal and local framing, check overlap and conflict against existing catalog docs.
6. Tag (cheap model): assign category: plus org tags from the closed taxonomy.
7. Gate (human): approve. The authority check.
8. Publish to the chosen knowledge endpoint, with provenance linking back to the soul origin (and the external source if any). The soul doc is stamped promoted and refreshed to the settled version.
9. Consume. Any agent now finds it via mark_lookup, authoritative and shared. The originating soul links to it rather than keeping a stale fork; the Coherence edge keeps that link live as the published copy evolves.

External sources enter the same lifecycle at step 1, with an ETL/distill in place of an agent.

Model routing: cascade, do not pick one model

Curation is not one task. Decompose by difficulty and volume, then route each sub-task to the cheapest model that clears the bar. Cost tracks volume, and volume is concentrated at the cheap stages.

Run the cheap model on the firehose as the triage gate (discards ~95% of Slack/Jira noise cheaply). Run the capable model only on the survivors. Never run the expensive model over every message.

The lever that lets Haiku go further is gate strength. A strong human gate makes Haiku-drafted distillations safe (a human polishes an 80%-good draft). An optional Haiku refute-check (a second cheap call that tests the distillation against its source) pushes strong-model usage lower still. Auto-publish with no gate is the only mode that forces the strong model, and it should not be used into an authoritative store.

The connective tissue the two plugins are missing

Today the plugins compose only by convention ("draft in soul, promote when ready"). There is no mechanism. The build list is this glue.

1. A promote primitive. Mark a soul doc as a candidate; a promote operation re-tags to the shared taxonomy, dedups against the catalog, and publishes with provenance to a chosen endpoint. It is a detection-gated bridge, not owned by either plugin (see "The bridge: detection-gated capability tiers"): memory contributes detect, candidate-mark, source-read, and back-stamp; knowledge contributes the gate, dedup, endpoint write, and destination selection.
2. Two-way provenance. The knowledge doc links to its soul/external origin. The soul doc is stamped promoted → mark://world/path so it is not re-promoted and is not duplicated. This enforces the existing "do not copy authoritative knowledge into the soul" rule mechanically.
3. Trigger policy. Memory-side only (there is no staging tier to watch without a soul). Manual ("ready for others") plus signal-based (recall-count, doc type) plus a batch sweep (a cron agent scans souls, runs the cascade, queues candidates for approval).
4. Dedup, staged. First dedup against the catalog (cheap, do now). Defer dedup across souls (entity resolution, hard) until the catalog flow is proven.
5. Taxonomy translation. Soul tags are loose and personal; the knowledge destination requires a category: axis and policy compliance. Promotion re-tags. Haiku does this (closed-set classification is its sweet spot).
6. Access discovery (read and write). Destination-selection needs the caller's {readable, writable} sets, but mark_worlds lists readable worlds only (the #189 distinction) and says nothing about write. Discovery is topology-dependent: against a brokered system the broker knows your identity and can enumerate both sets (extend mark_worlds with a writable surface); against a plain remote server there is no directory — the token you send IS your access, so discovery is introspecting that token's operation/path grants. Both resolve to the same {readable, writable} the router needs. Prerequisite for routing.
7. The coherence edge. The downward counterpart to promote — keeping the soul fresh as the published copy evolves. See "Coherence".

Autonomy: human-in-the-loop by default, capped by the destination

Promotion autonomy is a graded knob, not a local boolean. Authority over a shared authoritative store belongs to its owner, so auto-publish cannot be a purely client-side toggle — otherwise any client flips to auto and floods the catalog.

Model it as two levels:

• Destination ceiling, set by the world owner in the world's well-known descriptor (see "Partitioning"): human-only / verify-then-auto / auto. The org root caps at human-only; a personal remote endpoint or a low-stakes scratch world may allow auto.
• Local preference, applied as min(local preference, destination ceiling). Default is human-in-the-loop.

Autonomy is coupled to model tier (see Model routing): auto requires the strong model plus the refute-check; human-in-the-loop is exactly what makes a cheaper Haiku draft safe. Start everything at human-only; relax per-destination only once the gate and verification are proven.

Partitioning: discovered, not designed

Where knowledge lands is not a partition scheme the pipeline invents. It is already determined by the access topology and by each world's self-description. The pipeline conforms to it; it does not define it. Two layers decide the destination.

1. Access (hard constraint). Access is read and write, both token-encoded. You can typically read broadly across a knowledge system but write only to your writer allow-set (readable is not writable — the #189 distinction). Write access filters the candidate worlds for any promotion; read access governs what the Library and lookups can see. How you discover your access is topology-dependent — ask the broker (identity-based) or introspect your token's grants (capability-based, on a plain remote server); see access discovery in the connective tissue.
2. Well-known hints (soft routing). Each world self-describes via a per-world well-known descriptor, alongside the existing policy.md and template.md — e.g. mark://<world>/.well-known/demarkus/world.md: owning team, subject domain, partition role, and its autonomy ceiling (the same ceiling the Autonomy section needs). Destination-selection reads these to route a candidate to the right world among the ones you can write.

Source of truth is the world owner — each world declares its own scope, self-sovereign. The root hub may aggregate descriptors into a discovery index, but it must be derived (from the per-world descriptors or the federation crawler), never hand-maintained, or the central map drifts from what worlds actually hold.

Destination-selection algorithm. Enumerate writable worlds (needs access discovery above) → read each one's descriptor → either auto-route (match the candidate's subject/team to a world's declared domain) or present a labeled, write-filtered list for the human to pick.

Coherence: keeping the soul fresh after promotion

Promotion is an upward edge (soul → knowledge). It needs a downward counterpart, or the soul's promoted docs rot while the authoritative copy moves on. The rule that makes this safe: reconciliation is directional, never a symmetric merge. Knowledge is the base of truth; the soul refreshes from it; soul changes re-enter as gated updates. A naive two-way merge that let local soul edits silently land in the authoritative doc would corrupt the authority model.

Three mechanisms, each leaning on primitives demarkus already has.

1. Default: link-not-copy, so there is nothing to merge. Once promoted, the soul doc becomes a stub — title plus promoted → mark://world/path — and the body of truth lives in knowledge. Recall resolves the link and fetches fresh. The identity-case staleness class disappears because the soul defers instead of holding a rival copy.
2. If the soul keeps a cached snapshot (offline, speed): invalidate by version, do not merge. Stamp the cache with the promoted version and etag/content-hash (promoted → mark://world/path@v3). On pull or recall, compare the stamped version to the live one; if they differ the cache is stale, so refresh the body down — knowledge wins. Deterministic, no model; demarkus versions, etags, and content-hashes already exist for exactly this.
3. Local edits after promotion re-promote as a gated update, not a silent overwrite. If the soul refines its copy, that edit becomes a new promotion candidate — an update to the existing knowledge doc — back through the gate. The actual content reconciliation uses mark_publish's existing conflict-aware merge (expected_version plus diff3 with conflict markers). The soul proposes; the gate disposes.

The back-stamp does double duty here: it marks promoted (anti-re-promote) and performs the first downward refresh, syncing the soul stub to the settled post-gate version rather than the pre-promotion draft.

Related-but-distinct staleness is the hard case, deferred. A separate soul note that merely overlaps a refined knowledge doc (not the same doc) needs semantic relatedness to catch. Defer it; when added, lean on the link graph first (mark_backlinks / mark_graph — if the soul note links the knowledge doc, a version bump propagates a "your linked source moved" notice) before reaching for embeddings or an LLM similarity pass. This is the dedup problem in reverse and should not block phase 0.

Source taxonomy (external inflows)

The external sources are not the same shape, and they need different handling before they reach the gate.

Confluence is doc-shaped and converts deterministically (own sub-plan; a tools/demarkus-import Go binary using client/fetch.Client.Publish, pure-Go or pandoc converter, two-pass link rewrite, idempotency manifest). Slack/Jira/meetings are conversation-shaped: worthless raw, value locked in threads that distill to a paragraph. They are the reason the distillation stage exists.

Phasing

Sequence by fit, not by which API is easiest.

1. Confluence first. Doc-shaped, deterministic, highest signal. Proves the publish path, tagging, and provenance at lowest risk.
2. Meeting notes. Bounded volume, decision-dense. First real test of the distillation plus human-gate loop.
3. Jira. Outcomes-only extractor. Teaches entity-resolution against the existing Confluence docs.
4. Slack last. Lowest fit, highest volume, hardest privacy story. By now the gate is mature enough to be ruthless.

Underlying all of this: the soul-to-knowledge promotion lane is phase 0, because it is the gate every external source reuses. Build the gate against soul content first (it already exists and is safe to experiment on), then point the external ETLs at it.

Risks and things to watch

• Most soul content must never promote. If more than ~10–20% crosses the gate, the bar is too low and the catalog becomes a soul mirror, destroying the curation value. Ruthlessness is the feature.
• Fork staleness. Promotion forks knowledge. Solve with link-not-copy, a promoted marker, and version-stamped cache invalidation; the downward refresh edge is specced in "Coherence". Without it the soul accumulates stale copies of authoritative docs.
• Merge directionality. The coherence edge must be directional — knowledge authoritative, soul refreshes from it, soul edits re-enter as gated updates. A symmetric merge that silently lands local edits in the authoritative doc corrupts authority. Guard this explicitly in the implementation.
• Authority needs an owner. The human gate per world maps to policy.md's owner:. Decide who approves before wiring triggers.
• Signal-based triggers need telemetry that may not exist yet (recall counts). Start with manual plus doc-type triggers; add signal-based once recall is instrumented.
• ACL to world boundary. Source resources have access controls; worlds have read/write grants. A private channel must not flow into a world everyone reads. This is governance, not code. The per-world descriptor plus writer allow-set is the enforcement surface.
• Privacy/PII. Never publish PII or secrets to a shared world (policy already forbids it). Distillation must strip, not merely summarize.

Decisions (all resolved)

1. Is the knowledge destination the curated tier only, or also the raw retrieval tier? Curated-only, to keep it useful. The soul (and external ingestion souls) hold the raw/retrieval tier.
2. Human-in-the-loop or auto-publish for distillations? Human-in-the-loop by default; autonomy is a graded knob capped by the destination, not chosen freely by the client (see "Autonomy"). Destination owner sets a ceiling (human-only / verify-then-auto / auto) in the world's well-known descriptor; local config picks at or below it; default human-only. Coupled to model tier.
3. How are worlds partitioned for ingestion? Discovered, not designed (see "Partitioning"). It falls out of the access topology (read and write, token-encoded; broad read, narrow write) plus per-world well-known descriptors declaring team, domain, partition role, and autonomy ceiling. The pipeline enumerates writable worlds and routes by their declared domain.
4. Where does the promote primitive live? A detection-gated bridge capability, owned by neither plugin unilaterally (see "The bridge"). Memory contributes the intent/source half and stays broker-unaware; the knowledge plugin contributes the execution half (gate, dedup, endpoint write, destination selection). Mutual read-only detection lights it up. The destination is a parameter — a team world, the org system root, or a plain remote demarkus server — filtered to the caller's writer allow-set. The batch sweep remains a separate agent. Back-stamp: one-directional — the knowledge side reports the published mark://, and memory writes the promoted stamp and the first downward refresh onto its own soul doc (syncing it to the settled post-gate version, see "Coherence"); the knowledge plugin never writes the soul, preserving the v0.5.0 partition.

Build prerequisites (surfaced by the decisions)

Not design forks — concrete things that must exist before phase 0.

• The promote primitive (the bridge): candidate-mark, source-read, hand-off, execution cascade, one-directional back-stamp.
• Access discovery (read and write): a writable-worlds surface on the broker, plus token-grant introspection for plain remote servers (connective tissue #6).
• The per-world well-known descriptor (world.md): team, domain, partition role, autonomy ceiling — the routing and autonomy substrate.
• The coherence edge (knowledge → soul): link-not-copy stubs, version-stamped cache invalidation by etag/content-hash, and re-promotion of local edits as gated updates via mark_publish conflict-merge (see "Coherence").

Dogfood

With the forks resolved, promote this very plan from the soul to the knowledge destination as the first exercise of the pipeline it describes.