soul.demarkus.io:6309/journal/2026-06-17.md/v9 draft reader meta

2026-06-17 — Knowledge ingestion phase 0: prerequisites (brokered half)

Picked up the knowledge-ingestion epic. Phase 0 = the soul→knowledge promotion lane. Fritz chose prerequisites-first over a minimal manual cascade slice, so this session built the routing/autonomy substrate the promote primitive will sit on, not the cascade itself.

Done

A1 — mark_worlds writable column (broker, Go). The writer set already existed in code (authorizedWorlds / worldAllows(&w.Allow, claims), used by /auth/callback + /me/install) but was never exposed through read-discovery — #189 deliberately made mark_worlds list readable worlds only. Added a 4th writable: yes/no column computed per row from the caller's claims, so a single call yields {readable, writable}. Append-only column → library table parser unaffected. tools/demarkus-broker/internal/broker/mcp_tools_worlds.go (+yesNo), mcp_tools_list.go (tool description), mcp_tools_worlds_test.go. Full broker suite + pre-commit.sh green.

B — world.md per-world descriptor (spec + seed). Example at plugins/claude-code-knowledge/examples/knowledge-system/world.md + README wiring. Structured core: descriptor_version, team, domain, partition_role (hub|team|project|scratch), autonomy_ceiling (human-only|verify-then-auto|auto). Per-world well-known (mark://<world>/.well-known/demarkus/world.md), distinct from root-global policy/template. No-descriptor worlds still work (writable-but-unlabeled, pick-list only, default human-only). Reader is plain mark_fetch.

Key facts learned about the code

  • Plugins are pure bash/awk hooks + markdown slash-commands + skills — no Go on the promote path. The cascade (triage→distill→dedup→tag→gate→publish→back-stamp) is judgment-heavy model work → belongs in a slash command + skill driving MCP tools, matching the "agent as librarian / intelligence at the edge" invariant. Bash only for detection + back-stamp glue.
  • Detection is one-way today: knowledge reads ~/.demarkus/plugin-memory.conf (soul_is_configured) to spot a sibling soul; memory does NOT detect knowledge. Mutual detection (plan's bridge) = a memory-side peek at ~/.demarkus/knowledge-systems (the joined-slug registry written by /knowledge-join). Small, not yet built.
  • Broker authz: worldAllows is the writer predicate (per-world Allow: Emails carve-out, else Domains∧Groups); readableWorlds = every configured world (SSO org gate only). The two are kept deliberately separate so "all logins read, writes allow-listed" is expressible.

Deferred

  • A2 — plain-remote token-grant introspection (a demarkus-server surface reporting a token's {operations, path-globs}). Live target is brokered, so A1 unblocks routing now; A2 after the brokered path is proven.

Next

Remaining phase-0 prerequisites/work: the promote primitive (the cascade, as /promote command + skill, detection-gated) and the coherence edge (link-not-copy stub / version-stamped back-stamp). A1+B are now in place to route and cap autonomy. Dogfood still stands: promote /plans/knowledge-ingestion.md itself as the first run.

Recorded progress in the plan under "## Build progress".

Session 2 — the promote primitive

Built the promote primitive (phase 0's core) on top of A1+B. Detection-gated bridge, split across both plugins, model-driven.

New files / changes:

  • plugins/claude-code/scripts/lib.sh — KNOWLEDGE_REGISTRY const + knowledge_endpoints() / knowledge_present() (reverse-peek the knowledge registry; mutual detection; broker-unaware).
  • plugins/claude-code/scripts/detect-knowledge.sh — detection gate (mirrors detect-soul.sh).
  • plugins/claude-code/commands/promote.md — /promote <soul-path>: gate → read source → already-promoted check → knowledge cascade → back-stamp (stub | marker-only). One-directional back-stamp; directional reconciliation.
  • plugins/claude-code-knowledge/skills/knowledge-promote/SKILL.md — execution cascade (triage→distill→dedup→tag→route→gate→publish). First skill in the knowledge plugin (auto-discovered, no manifest entry needed).
  • plugins/claude-code/tests/detect-knowledge_test.sh — 5 tests, green. Full plugin regression green (memory + knowledge).

Gotchas captured:

  • Skills + commands auto-discover from their dirs — no plugin.json entry (memory's manifest has no skills key yet soul-memory works).
  • KNOWLEDGE_REGISTRY is readonly, bound to HOME at source time → can't unit-test the function with a swapped HOME in one shell; test the script in fresh subprocesses with controlled HOME instead (what detect-knowledge_test.sh does).
  • shellcheck SC1091 (can't follow lib.sh) is info-level and identical on the established detect-soul.sh; no CI shellcheck config in .github. SC2317 in lib.sh:9 is pre-existing.

Next: coherence edge active half (version-stamped invalidation + re-promotion-as-gated-update), then dogfood /promote /plans/knowledge-ingestion.md. Triggers beyond manual (signal/batch) and A2 still open. Plugin version bumps left for Fritz.

Session 3 — coherence edge (folded into the promote branch)

Active downward half of the coherence edge. All plugin work, on branch feat/knowledge-promote-primitive.

Changes:

  • commands/promote.md — standardized back-stamp marker to a single greppable line promoted: mark://<world>/<path>@v<N> (both stub + marker-only modes); back-stamp now adds the promoted metadata tag. Step-3 already-promoted check updated to the new format + framed as the upward leg.
  • commands/soul-refresh.md (new) — downward sync. Gate → find promoted docs (tag=promoted) → parse @vN → compare to live knowledge version → directional refresh: stub auto-refreshes down, marker-only surfaces for pull-down vs re-promote-upward. Dangling target surfaced not deleted. Related-but-distinct staleness deferred.
  • context/session-guidance.md — one line noting /promote + /soul-refresh activate only with a joined knowledge system.

Design notes:

  • Hooks can't reach the broker, so the staleness check + refresh must be an agent-driven command, not a PostToolUse/recall hook. The plan called this out.
  • Reconciliation is strictly directional. /soul-refresh never two-way merges; local edits go up via /promote (which owns the gate + the cascade's update path), keeping the v0.5.0 plugin partition + the authority model intact.
  • No new bash → no new tests needed; full plugin regression still green (memory + knowledge).

Phase-0 prerequisites now all but A2 done. Next candidates: signal/batch triggers, or the dogfood promote of the plan itself. Plugin version bumps (memory + knowledge) still pending Fritz's commit.

Session 4 — dogfood run (the pipeline on itself)

Ran /promote /plans/knowledge-ingestion.md end-to-end against the live brokered knowledge system (knowledge.demarkus.io). Executed the cascade by hand (the just-built command/skill aren't loaded in this session) using soul + knowledge MCP tools, stopping at the human gate before the write.

Result: published distilled copy → mark://world-a/test/knowledge-ingestion-pipeline.md@v1; soul plan back-stamped marker-only (v10, promoted tag added, Dogfood marked done). Both sides verified via mark_lookup. Mechanism works end-to-end: triage → distill → dedup → tag → gate → publish → back-stamp → round-trip discoverable.

Two non-obvious observations (feed back into the work):

  1. The live prod broker predates #191 — mark_worlds returned the old 3-column shape (no writable). So the access-discovery payoff isn't live until the broker is redeployed from main. Routing fell back to the human gate. Action: deploy the broker to pick up #191 before the writable column matters in practice.
  2. The knowledge system has no demarkus-project world — only root (hub) and world-a ("Demo / test world — nib subtree + e2e test docs"). So demarkus's own design knowledge has no natural home; the dogfood landed honestly as a test/reference doc in world-a (which explicitly holds e2e test docs — and this is an e2e test). A real demarkus knowledge home would need a world added to the deploy repo's worlds[]. This is exactly the "Partitioning: discovered, not designed" point biting in practice — the catalog's shape constrains where promotion can land.

Distillation transforms applied (for reference, what "distill for shared audience" meant here): dropped the "draft, unstarted" status line; collapsed the three personal build-progress session logs into one shared-audience "Implementation status" (shipped vs deferred); added a provenance line to the soul origin; kept the design narrative + decisions + prerequisites intact. No secrets/PII present. Tag category:reference satisfied the block-strictness require_tags: category gate on the first try.

Session 5 — promotion triggers

Moved promotion past manual-only. Built the two memory-side trigger pieces the plan prescribes for now (signal-based/recall-count deferred — no telemetry; autonomous batch sweep is a separate agent).

  • hooks/promote-nudge.sh — PostToolUse on mark_publish. Discreet "/promote?" nudge, gated on: local-soul scope + knowledge_present + ADR path (*/adr/*.md) + no promoted: mark:// marker in body. Registered as a 2nd PostToolUse hook beside publish-gate. 7 tests green.
  • commands/promote-scan.md — on-demand candidate sweep: excludes tag=promoted, surfaces high-signal types (ADR/architecture/high-importance), ranked shortlist → user runs /promote. Never auto-promotes.
  • memory 0.6.0 → 0.7.0; session-guidance + marketplace description updated.

Gotcha reused: HOME-bound readonly vars → test the hook via fresh subprocesses with controlled HOME + payload on stdin (same pattern as detect-knowledge_test). All plugin suites green; shellcheck clean (only the standard SC1091 source-follow info, same as every sibling hook).

Branch feat/promotion-triggers, uncommitted. Next candidates: A2 (plain-remote access discovery) or Confluence (phase 1, first external inflow).

Session 6 — A2: plain-remote access discovery

Closed the last phase-0 prerequisite. Plain remote demarkus server (no broker): the token IS the access, so discovery = reflect the presented token's grants.

  • No new verb (7-verb set is final) and no new MCP tool: a synthesized well-known doc /.well-known/demarkus/access.md, served via FETCH, computed per-request from the token, dispatched before read-auth, never gated (reflects only the caller's own grants → no leak).
  • Server: auth.GrantsFor (anonymous on empty/unknown/expired, returns copies) + handler/access.go (buildAccessManifest pure + handleAccessManifest, etag/content-hash, if-none-match) + dispatch at top of handleFetch. Tests in auth_test + new access_test (incl. stored-file-doesn't-shadow, anonymous-leaks-nothing). Server suite + pre-commit green.
  • Client: knowledge-promote skill destination-select now branches brokered (mark_worlds writable) vs plain-remote (fetch access.md).
  • docs/SPEC.md §11.8 updated.

Gotchas: ParseResponse returns a value not a pointer; store pkg is protocol/store (not server/internal/store). auth_test is package auth so I set unexported expiresAt/now directly for the expiry case.

Phase-0 prerequisites all done. Branch feat/plain-remote-access-discovery (about to create), uncommitted. Server version auto-bumps via CI; plugin SERVER_VERSION pin to follow once released.

Related documents

soul.demarkus.io:6309/plans/claude-code-plugin.md complete reader meta

Plan: Claude Code Plugin

Make demarkus usable in Claude Code with a single marketplace install. No manual server setup, no token generation, no .mcp.json hand editing. The plugin reuses the existing demarkus server/client/protocol surface — no core code changes are required to ship it.

Status

Shipped. demarkus-memory plugin v0.1.1 is live on the marketplace; plugins/claude-code/ ships hooks (postinstall.sh, session-start.sh), slash commands (/soul, /soul-journal, /soul-status, /soul-init, /soul-context, /soul-memory), the memory skill, the seed index.md, and the binary-download pipeline. ~/.demarkus/plugin-memory.conf records the operator's chosen MODE / SOUL_DIR / PORT. The plugin's lazy-spawned demarkus-server is the memory backend for every demarkus development session.

Goal

After /plugin marketplace add latebit-io/demarkus and /plugin install demarkus-memory, the user has a working memory agent. mark_fetch, mark_publish, mark_append, and the rest of the MCP tool set are available. A local demarkus-server backs them. Memory persists across sessions. The TUI and browser can hit the same server on the same machine.

Non-Goals

  • Cross-machine sync of the local soul. Users who want that run a networked server and reconfigure the plugin.
  • Windows native. WSL only, matching the rest of demarkus.
  • Embedded (in-process) MCP mode. We keep server and client as separate processes; the plugin just automates the spin-up.
  • Remote marketplace hosting. The plugin lives in the monorepo and is published via a marketplace.json at the repo root.
  • Core code changes. The plugin is built entirely on existing primitives; any need for a new flag or endpoint must be discussed and justified before landing.

User Flow

  1. /plugin marketplace add latebit-io/demarkus
  2. /plugin install demarkus-memory@demarkus
  3. Postinstall downloads platform-specific binaries to ${CLAUDE_PLUGIN_ROOT}/bin/
  4. On the next session, a SessionStart hook ensures the server is up and the token exists
  5. The agent has the full MCP tool set pointed at mark://localhost:<port>

No config files to touch, no prompts, no decisions forced on the user.

Architecture

Process Lifecycle: Lazy Spawn

The SessionStart hook:

  • Checks for an existing server with pgrep -f "demarkus-server.*\.demarkus/soul"; if running, noop
  • Otherwise spawns it detached: ${CLAUDE_PLUGIN_ROOT}/bin/demarkus-server -root ~/.demarkus/soul -port $PORT -log-file ~/.demarkus/soul/.log & followed by disown
  • Server persists across sessions; the next session reuses it

No service manager (launchd, systemd user unit) in v1. If the process model proves fragile later, promote to a user service.

Content Root

~/.demarkus/soul/ is the default content directory. Created on first run. All versioning, graph data, tokens, and logs live under it.

Port Handling

Default port: 6310. On every SessionStart, use primitives the server already exposes:

  1. Try FETCH /health against mark://127.0.0.1:6310
  2. If it succeeds (QUIC handshake completes with ALPN "mark" and /health returns OK), treat the port as "a demarkus server we can use" and reuse it. Any other service would fail the ALPN negotiation, so there is no meaningful risk of mistaking a non-demarkus process for ours.
  3. If the handshake fails or the port is unreachable, pick the next free port in a small range, spawn our server there, and write the chosen port to ~/.demarkus/soul/.port
  4. The MCP client reads the port from .port on startup (falls back to 6310 if absent)

No server-side changes needed — /health and ALPN "mark" are already in the protocol.

Auth

On first run the hook generates a token via demarkus-token generate, scoped /* with ops publish,archive. The existing CLI writes the raw token to stdout and context messages to stderr, so the hook captures it with shell redirection and restricts permissions:

demarkus-token generate -label claude-code-plugin -paths "/*" -ops "publish,archive" \
  -tokens ~/.demarkus/soul/tokens.toml \
  2>/dev/null > ~/.demarkus/soul/.token
chmod 600 ~/.demarkus/soul/.token

The MCP client picks up the token via the existing DEMARKUS_AUTH environment variable, which tokens.Resolve already honors. The session-start.sh hook exports it before Claude Code launches the MCP process:

export DEMARKUS_AUTH="$(cat ~/.demarkus/soul/.token)"

No server-side or client-side changes needed — tokens.Resolve, demarkus-token generate, and the DEMARKUS_AUTH env var are all already shipped.

First-Run Seeding

The plugin ships a seed/index.md template. The SessionStart hook copies it to ~/.demarkus/soul/index.md only when the content root is empty. The agent expands from there.

Minimal seed:

# My Soul

Personal memory for Claude Code agents.

## Sections

- [Journal](/journal.md) — session notes
- [Notes](/notes.md) — anything worth remembering

Plugin Layout

plugins/claude-code/
  .claude-plugin/plugin.json
  hooks/
    postinstall.sh      # download + verify binaries
    session-start.sh    # ensure server, ensure token, export DEMARKUS_AUTH, seed index
  .mcp.json             # demarkus-mcp -host mark://localhost:<port>
  commands/
    soul.md             # /soul → display /index.md
    soul-journal.md     # /soul-journal <entry> → mark_append /journal.md
  skills/
    memory/SKILL.md     # triggers on remember/save/note/recall
  seed/
    index.md

At the monorepo root: .claude-plugin/marketplace.json listing the plugin and pointing at plugins/claude-code/.

Binary Distribution

postinstall.sh:

  1. Detects platform via uname -sm (darwin/arm64, darwin/amd64, linux/amd64, linux/arm64)
  2. Reads the plugin version from plugin.json
  3. Downloads the three binaries from https://github.com/latebit-io/demarkus/releases/download/v<version>/...
  4. Verifies SHA256 checksums against a bundled checksums file
  5. Places binaries in ${CLAUDE_PLUGIN_ROOT}/bin/ and chmod +x them

Binaries required: demarkus-server, demarkus-mcp, demarkus-token. GoReleaser already produces these per platform; the only possible pipeline tweak is ensuring the release assets include a consolidated SHA256SUMS file. Confirm the GoReleaser config before hardcoding download URLs.

Core Code Changes

None. Every piece of plumbing the plugin needs already exists:

  • Token injection: DEMARKUS_AUTH env var (honored by tokens.Resolve)
  • Server identity / port probe: /health endpoint + ALPN "mark" negotiation
  • Writing a generated token to a file: shell redirection of demarkus-token generate stdout + chmod 600

If any future iteration of the plugin surfaces a genuine need for a new core primitive, discuss and justify it before landing. Plugin-driven polish is not a reason to touch core.

Commands and Skills

Slash Commands

  • /soul — mark_fetch /index.md and render
  • /soul-journal <entry> — mark_append /journal.md with the entry, auto-resolving expected_version

Skill: Memory

Triggers on phrases like "remember this", "save to memory", "what do I know about X", "note that", "recall".

  • Read intents → mark_fetch, then mark_backlinks or mark_graph for exploration
  • Write intents → mark_append /notes.md by default, or mark_publish to a topic page when the agent decides the note deserves its own document

Release Pipeline

  • Plugin version tags follow the monorepo convention: claude-code-plugin/v0.1.0
  • Releases are cut from main; the marketplace.json is updated in the same commit
  • Users who already added the marketplace get the update via /plugin marketplace update

Open Questions and Deferred Work

  • Service installer: launchd plist / systemd --user unit. Deferred. Lazy spawn is enough for v1.
  • Binary auto-update: v1 downloads on postinstall only. A later version can verify against latest on SessionStart and prompt.
  • Cross-machine access: document the path for users who want to run a networked server and point the plugin at it, but don't automate the migration.
  • Port collision UX: if the probe can't find a free port in the scanned range, the hook should surface a clear error rather than silently fail.

Phasing

Phase 1: Core plugin (ships usable v1) ✓

  1. ✓ Write postinstall.sh (detect platform, fetch binaries from GitHub releases, verify checksums)
  2. ✓ Write session-start.sh (probe 6310, spawn server if absent, generate token on first run, export DEMARKUS_AUTH, seed index.md)
  3. ✓ Author .mcp.json, plugin.json, seed index.md
  4. ✓ Publish .claude-plugin/marketplace.json at the monorepo root
  5. ✓ Manual end-to-end test on darwin/arm64 and linux/amd64

Phase 2: Agent surface ✓

  1. ✓ Slash commands (/soul, /soul-journal, plus /soul-status, /soul-init, /soul-context, /soul-memory)
  2. ✓ Memory skill
  3. ⚠ Documentation for migrating to a networked server (deferred — no user demand yet)

Phase 3: Polish (deferred)

  1. Optional user-level service installer
  2. Binary update flow on SessionStart
  3. Better port-collision error messaging

Success Criteria

  • ✓ Fresh Claude Code install + marketplace add + plugin install → a working mark_fetch /index.md on first session with zero prompts
  • ✓ Existing demarkus users can disable the plugin-managed server and point MCP at their own server via a documented override (MODE=reuse in ~/.demarkus/plugin-memory.conf)
  • ✓ Plugin works offline on subsequent sessions (no network needed once binaries are cached)
  • ✓ Total install time under 10 seconds on typical broadband
  • ✓ Zero changes to protocol/, server/, or client/ modules outside plugins/claude-code/
trail
  1. soul.demarkus.io:6309 v9
  2. claude-code-plugin
from here → completed-plans ecosystem graph index 2026-04-24 2026-05-22