2026-06-17 — Knowledge ingestion phase 0: prerequisites (brokered half)

Picked up the knowledge-ingestion epic. Phase 0 = the soul→knowledge promotion lane. Fritz chose prerequisites-first over a minimal manual cascade slice, so this session built the routing/autonomy substrate the promote primitive will sit on, not the cascade itself.

Done

A1 — mark_worlds writable column (broker, Go). The writer set already existed in code (authorizedWorlds / worldAllows(&w.Allow, claims), used by /auth/callback + /me/install) but was never exposed through read-discovery — #189 deliberately made mark_worlds list readable worlds only. Added a 4th writable: yes/no column computed per row from the caller's claims, so a single call yields {readable, writable}. Append-only column → library table parser unaffected. tools/demarkus-broker/internal/broker/mcp_tools_worlds.go (+yesNo), mcp_tools_list.go (tool description), mcp_tools_worlds_test.go. Full broker suite + pre-commit.sh green.

B — world.md per-world descriptor (spec + seed). Example at plugins/claude-code-knowledge/examples/knowledge-system/world.md + README wiring. Structured core: descriptor_version, team, domain, partition_role (hub|team|project|scratch), autonomy_ceiling (human-only|verify-then-auto|auto). Per-world well-known (mark://<world>/.well-known/demarkus/world.md), distinct from root-global policy/template. No-descriptor worlds still work (writable-but-unlabeled, pick-list only, default human-only). Reader is plain mark_fetch.

Key facts learned about the code

• Plugins are pure bash/awk hooks + markdown slash-commands + skills — no Go on the promote path. The cascade (triage→distill→dedup→tag→gate→publish→back-stamp) is judgment-heavy model work → belongs in a slash command + skill driving MCP tools, matching the "agent as librarian / intelligence at the edge" invariant. Bash only for detection + back-stamp glue.
• Detection is one-way today: knowledge reads ~/.demarkus/plugin-memory.conf (soul_is_configured) to spot a sibling soul; memory does NOT detect knowledge. Mutual detection (plan's bridge) = a memory-side peek at ~/.demarkus/knowledge-systems (the joined-slug registry written by /knowledge-join). Small, not yet built.
• Broker authz: worldAllows is the writer predicate (per-world Allow: Emails carve-out, else Domains∧Groups); readableWorlds = every configured world (SSO org gate only). The two are kept deliberately separate so "all logins read, writes allow-listed" is expressible.

Deferred

• A2 — plain-remote token-grant introspection (a demarkus-server surface reporting a token's {operations, path-globs}). Live target is brokered, so A1 unblocks routing now; A2 after the brokered path is proven.

Next

Remaining phase-0 prerequisites/work: the promote primitive (the cascade, as /promote command + skill, detection-gated) and the coherence edge (link-not-copy stub / version-stamped back-stamp). A1+B are now in place to route and cap autonomy. Dogfood still stands: promote /plans/knowledge-ingestion.md itself as the first run.

Recorded progress in the plan under "## Build progress".

Session 2 — the promote primitive

Built the promote primitive (phase 0's core) on top of A1+B. Detection-gated bridge, split across both plugins, model-driven.

New files / changes:

• plugins/claude-code/scripts/lib.sh — KNOWLEDGE_REGISTRY const + knowledge_endpoints() / knowledge_present() (reverse-peek the knowledge registry; mutual detection; broker-unaware).
• plugins/claude-code/scripts/detect-knowledge.sh — detection gate (mirrors detect-soul.sh).
• plugins/claude-code/commands/promote.md — /promote <soul-path>: gate → read source → already-promoted check → knowledge cascade → back-stamp (stub | marker-only). One-directional back-stamp; directional reconciliation.
• plugins/claude-code-knowledge/skills/knowledge-promote/SKILL.md — execution cascade (triage→distill→dedup→tag→route→gate→publish). First skill in the knowledge plugin (auto-discovered, no manifest entry needed).
• plugins/claude-code/tests/detect-knowledge_test.sh — 5 tests, green. Full plugin regression green (memory + knowledge).

Gotchas captured:

• Skills + commands auto-discover from their dirs — no plugin.json entry (memory's manifest has no skills key yet soul-memory works).
• KNOWLEDGE_REGISTRY is readonly, bound to HOME at source time → can't unit-test the function with a swapped HOME in one shell; test the script in fresh subprocesses with controlled HOME instead (what detect-knowledge_test.sh does).
• shellcheck SC1091 (can't follow lib.sh) is info-level and identical on the established detect-soul.sh; no CI shellcheck config in .github. SC2317 in lib.sh:9 is pre-existing.

Next: coherence edge active half (version-stamped invalidation + re-promotion-as-gated-update), then dogfood /promote /plans/knowledge-ingestion.md. Triggers beyond manual (signal/batch) and A2 still open. Plugin version bumps left for Fritz.

Session 3 — coherence edge (folded into the promote branch)

Active downward half of the coherence edge. All plugin work, on branch feat/knowledge-promote-primitive.

Changes:

• commands/promote.md — standardized back-stamp marker to a single greppable line promoted: mark://<world>/<path>@v<N> (both stub + marker-only modes); back-stamp now adds the promoted metadata tag. Step-3 already-promoted check updated to the new format + framed as the upward leg.
• commands/soul-refresh.md (new) — downward sync. Gate → find promoted docs (tag=promoted) → parse @vN → compare to live knowledge version → directional refresh: stub auto-refreshes down, marker-only surfaces for pull-down vs re-promote-upward. Dangling target surfaced not deleted. Related-but-distinct staleness deferred.
• context/session-guidance.md — one line noting /promote + /soul-refresh activate only with a joined knowledge system.

Design notes:

• Hooks can't reach the broker, so the staleness check + refresh must be an agent-driven command, not a PostToolUse/recall hook. The plan called this out.
• Reconciliation is strictly directional. /soul-refresh never two-way merges; local edits go up via /promote (which owns the gate + the cascade's update path), keeping the v0.5.0 plugin partition + the authority model intact.
• No new bash → no new tests needed; full plugin regression still green (memory + knowledge).

Phase-0 prerequisites now all but A2 done. Next candidates: signal/batch triggers, or the dogfood promote of the plan itself. Plugin version bumps (memory + knowledge) still pending Fritz's commit.

Session 4 — dogfood run (the pipeline on itself)

Ran /promote /plans/knowledge-ingestion.md end-to-end against the live brokered knowledge system (knowledge.demarkus.io). Executed the cascade by hand (the just-built command/skill aren't loaded in this session) using soul + knowledge MCP tools, stopping at the human gate before the write.

Result: published distilled copy → mark://world-a/test/knowledge-ingestion-pipeline.md@v1; soul plan back-stamped marker-only (v10, promoted tag added, Dogfood marked done). Both sides verified via mark_lookup. Mechanism works end-to-end: triage → distill → dedup → tag → gate → publish → back-stamp → round-trip discoverable.

Two non-obvious observations (feed back into the work):

1. The live prod broker predates #191 — mark_worlds returned the old 3-column shape (no writable). So the access-discovery payoff isn't live until the broker is redeployed from main. Routing fell back to the human gate. Action: deploy the broker to pick up #191 before the writable column matters in practice.
2. The knowledge system has no demarkus-project world — only root (hub) and world-a ("Demo / test world — nib subtree + e2e test docs"). So demarkus's own design knowledge has no natural home; the dogfood landed honestly as a test/reference doc in world-a (which explicitly holds e2e test docs — and this is an e2e test). A real demarkus knowledge home would need a world added to the deploy repo's worlds[]. This is exactly the "Partitioning: discovered, not designed" point biting in practice — the catalog's shape constrains where promotion can land.

Distillation transforms applied (for reference, what "distill for shared audience" meant here): dropped the "draft, unstarted" status line; collapsed the three personal build-progress session logs into one shared-audience "Implementation status" (shipped vs deferred); added a provenance line to the soul origin; kept the design narrative + decisions + prerequisites intact. No secrets/PII present. Tag category:reference satisfied the block-strictness require_tags: category gate on the first try.

Session 5 — promotion triggers

Moved promotion past manual-only. Built the two memory-side trigger pieces the plan prescribes for now (signal-based/recall-count deferred — no telemetry; autonomous batch sweep is a separate agent).

• hooks/promote-nudge.sh — PostToolUse on mark_publish. Discreet "/promote?" nudge, gated on: local-soul scope + knowledge_present + ADR path (*/adr/*.md) + no promoted: mark:// marker in body. Registered as a 2nd PostToolUse hook beside publish-gate. 7 tests green.
• commands/promote-scan.md — on-demand candidate sweep: excludes tag=promoted, surfaces high-signal types (ADR/architecture/high-importance), ranked shortlist → user runs /promote. Never auto-promotes.
• memory 0.6.0 → 0.7.0; session-guidance + marketplace description updated.

Gotcha reused: HOME-bound readonly vars → test the hook via fresh subprocesses with controlled HOME + payload on stdin (same pattern as detect-knowledge_test). All plugin suites green; shellcheck clean (only the standard SC1091 source-follow info, same as every sibling hook).

Branch feat/promotion-triggers, uncommitted. Next candidates: A2 (plain-remote access discovery) or Confluence (phase 1, first external inflow).

Session 6 — A2: plain-remote access discovery

Closed the last phase-0 prerequisite. Plain remote demarkus server (no broker): the token IS the access, so discovery = reflect the presented token's grants.

• No new verb (7-verb set is final) and no new MCP tool: a synthesized well-known doc /.well-known/demarkus/access.md, served via FETCH, computed per-request from the token, dispatched before read-auth, never gated (reflects only the caller's own grants → no leak).
• Server: auth.GrantsFor (anonymous on empty/unknown/expired, returns copies) + handler/access.go (buildAccessManifest pure + handleAccessManifest, etag/content-hash, if-none-match) + dispatch at top of handleFetch. Tests in auth_test + new access_test (incl. stored-file-doesn't-shadow, anonymous-leaks-nothing). Server suite + pre-commit green.
• Client: knowledge-promote skill destination-select now branches brokered (mark_worlds writable) vs plain-remote (fetch access.md).
• docs/SPEC.md §11.8 updated.

Gotchas: ParseResponse returns a value not a pointer; store pkg is protocol/store (not server/internal/store). auth_test is package auth so I set unexported expiresAt/now directly for the expiry case.

Phase-0 prerequisites all done. Branch feat/plain-remote-access-discovery (about to create), uncommitted. Server version auto-bumps via CI; plugin SERVER_VERSION pin to follow once released.

Related documents

• Knowledge ingestion plan: phase 0 prerequisites and promote primitive built here
• Claude Code plugin plan: memory and knowledge plugins carrying promote and soul-refresh
• Knowledge system deploy: broker redeploy needed for the writable column