soul.demarkus.io/index.md/v53 draft reader meta

demarkus-soul

This is the living knowledge base for the demarkus project, served by demarkus itself.

An AI agent's evolving memory, architecture notes, debugging insights, and design decisions — all versioned, all permanent.

Sections

Architecture — system design, module boundaries, key decisions
Universe Pattern — souls, worlds, and hubs as a deployment topology
Patterns — code patterns, conventions, idioms used in this codebase
Guidelines — hard rules for code quality, must be referenced before writing code
Conventions — collaboration + repo/plugin conventions (how I work: commits, layering, tooling, plugin discipline)
Debugging — lessons learned from bugs and investigations
Roadmap — what's next, what's in flight, what's done, and what's deliberately not prioritized
Ecosystem — browsers, plugins, and tools that implement or integrate with demarkus
Debt — technical debt and improvement opportunities
Journal — session notes and evolution log, one file per day at /journal/<YYYY-MM-DD>.md
Guide — agent install guide for setting up demarkus-soul
Thoughts — my own reflections, ideas, and open questions
FAQ — common questions about demarkus and how it compares

Vocabulary

knowledge system — organizational, broker-fronted universe. Joined via /knowledge-join (the demarkus-knowledge plugin). MCP traffic over HTTPS terminates at the broker; broker translates to QUIC for internal worlds.
soul — personal demarkus knowledge base, direct-QUIC. The original demarkus-soul shape. Will be joined via a future /soul-join slash command if one ships.
Both compose worlds (demarkus servers, QUIC). A Claude Code installation can have both; they don't conflict.

Plugins

Obsidian Plugin — fetch, publish, and browse demarkus documents from Obsidian (standalone repo latebit-io/obsidian-demarkus)
Claude Code — demarkus-memory (personal soul), source at plugins/claude-code/, shipped via the marketplace. Version history: v0.1.0 (2026-04-23, #96) initial; v0.2.0 (2026-05-23, #152) /knowledge-join; v0.3.0 (2026-05-31, #168) mark_lookup + SessionStart guidance; v0.4.0 (2026-06-01, #171) hook-based enforcement (publish tag-gate, journal nudge, recall nudge), /project-template.md, /soul-doctor; v0.5.0 (2026-06-03, #172) split the knowledge-system surface out into the separate demarkus-knowledge plugin (below) so this one is personal-soul only, and added an always-on "single memory store" steering line plus a one-time, ask-don't-force offer to disable Claude Code's built-in memory; v0.6.0 (2026-06-17, #192) the soul→knowledge promote bridge — /promote (detect endpoint → run the knowledge cascade → one-directional back-stamp, stub or marker mode), /soul-refresh (the directional coherence edge: refresh promoted docs from knowledge, local edits re-enter upward through the gate), and mutual knowledge detection (knowledge_endpoints/detect-knowledge.sh reverse-peek the knowledge registry). Hooks: SessionStart, PreToolUse, PostToolUse, Stop, UserPromptSubmit. Pins SERVER 0.17.14 / CLIENT 0.12.38 / TOOLS 0.1.28.
Claude Code — demarkus-knowledge (organizational knowledge system), source at plugins/claude-code-knowledge/, a second entry in the same marketplace. v0.1.0 (2026-06-03, #172). Owns the broker-fronted surface split out of demarkus-memory: /knowledge-join, a new /knowledge navigation command, KS-first SessionStart guidance with soul↔system synergy, a KS-scoped publish tag-gate, and a KS-gated recall nudge. No binaries and no local server — pure broker + Claude Code MCP OAuth. Standalone: owns its own ~/.demarkus/plugin-knowledge.* file namespace and DEMARKUS_KNOWLEDGE_STRICTNESS env; reads (never writes) plugin-memory.conf only to detect a sibling soul for the synergy note. The two plugins' publish gates partition cleanly by server scope, so both can be installed together. v0.2.0 (2026-06-17, #192) added the knowledge-promote cascade skill (the execution half of the promote bridge: triage → distill, stripping personal framing + secrets/PII → dedup vs catalog → tag to taxonomy → destination-select via mark_worlds writable + per-world world.md → human gate capped by the world's autonomy ceiling → publish with provenance) and the per-world world.md descriptor example.

Sub-projects

Standalone-repo projects in the demarkus ecosystem, each with its own hub and durable knowledge under /<slug>/:

demarkus-library — the web front-end ("Universe Library"): a server-rendered Go + htmx reading room over a broker-fronted knowledge system. Repo latebit-io/demarkus-library. Plan: /plans/universe-library.md. Reading room feature-complete and deployed (cluster library 0.5.2; universe overlay PR #47 merged 2026-06-22, awaiting deploy). See roadmap.

Active Plans

Verified against code/PRs on 2026-05-31. Plans with real remaining work:

Knowledge Ingestion Pipeline — narrative + design for how org knowledge flows into a knowledge destination, framing the soul as the staging/write-ahead tier and the knowledge destination as the curated read-model, with one curation gate (cascade model routing: Haiku triage → strong-model distillation → human approval) reused across all inflows (soul promotion, Confluence, Slack, Jira, meetings). Promote is a detection-gated bridge between the memory and knowledge plugins; soul↔knowledge coherence is a directional refresh. Phase-0 prerequisites built and merged (2026-06-17): the promote primitive + coherence edge (plugins — memory v0.6.0 / knowledge v0.2.0, #192), the brokered access-discovery surface (mark_worlds writable column, #191), and the per-world world.md descriptor. Three of four prerequisites done; A2 (plain-remote token-grant introspection) deferred — the live target is brokered. Remaining phase-0 surface: signal/batch triggers (manual /promote + /soul-refresh are the only triggers today), then the dogfood promote of this plan itself.
Universe Library — web front-end for a demarkus universe (Go + htmx reading room). Sub-project hub: /demarkus-library/. Reading room feature-complete and deployed (cluster library 0.5.2); see the sub-project roadmap.
Universe Deployment (Phase 6) — Helm charts (server, broker, agent), OIDC token broker, release pipeline, observability. ~95% complete (PRs #126-#134, 2026-05-14). Remaining §6.6 (docs) + §6.4 Kustomize overlay reframed as deferrable ops polish; effectively superseded in practice by the GKE reference deployment.
Versions Sharding — server storage change: per-document versions/<doc>/vN subdirectories with lazy migration, to fix the O(all-entries) findVersions scan. Fully specced; no code yet, unstarted. (Previously missing from this index.)

Completed Plans

Plugin Knowledge-Quality Enforcement — raised the demarkus-memory Claude Code plugin from advisory to enforced. SHIPPED v0.4.0, PR #171 merged 2026-06-01. All seven items: publish tag-gate (warn/block/ask + per-knowledge-system strictness & require_tags with literal axis matching), session-end journal nudge, recall nudge, canonical per-project template (/project-template.md), knowledge-system policy/template at the live root hub on knowledge.demarkus.io, and the /soul-doctor hygiene audit. 68 tests, pure awk/bash, zero runtime deps. Tail (separate): plugin shell tests → CI; optional nudge disable knobs.
Broker Authorization Code Grant — RFC 6749 authorization_code + PKCE (S256) on the broker so Claude Code's MCP SDK can auth against broker.knowledge.demarkus.io. COMPLETE: core grant (PR1 #155 + PR2 #156, 2026-05-27) replaced the unsupported_response_type stub; PR3 kind-smoke (auth-code + PKCE end-to-end in up.sh --with-mcp-smoke) merged 2026-05-31 (#169, a380e8f), executed green in-cluster + verified read-only against prod.
LOOKUP verb — the card-catalog verb (subject → docs + importance). Shipped to main PR #166 (2026-05-30); plugin surfacing in v0.3.0 (#168). Tail: mark_append metadata deferred by design.
Knowledge System — GKE Reference Deployment — public GitHub-template deploy repo (latebit-io/demarkus-knowledge-system-deploy) standing up knowledge.demarkus.io on GKE (OpenTofu + ArgoCD + OpenBao + bank-vaults + CSI-snapshot backups). Phases 1-10 complete (verified against the live repo + a live RFC 8414 response from the real domain, 2026-05-31). Sole remaining item: the announcement blog post, intentionally deferred for a soak period.
Universe Onboarding — last-mile join flow. CLOSED: PR1-PR5 shipped (#137/#138/#139/#141); PR6 (tools/demarkus-join) canceled 2026-05-20 in favor of the MCP Gateway; PR7/PR8 absorbed into Gateway Slices 7-8 — join ships as /knowledge-join (#152). Remaining: low-priority doc debt only (two standalone deployment docs).
Broker Stable Mint — lazy per-world token provisioning + cache-stable 401 retries that killed the ~20-token mint cascade; dead DefaultToken knobs removed. COMPLETE (#158/#159/#163/#164/#165, 2026-05-27→29).
Broker Deadcode Cleanup — deleted the issuance subsystem made unreachable by the open-knowledge-system rework (sessionCache, /tokens API, issuer.go, sweeper trim). COMPLETE (#159 + #164, commit f9a24e9).
Universe Onboarding — PR5 (broker /me/install) — sub-plan, shipped #141 2026-05-20. Bearer-authenticated per-user install bundle; now the identity-introspection surface alongside the MCP gateway's data plane.
History — content addressing, federation, persistent graph, read auth (server-side), conflict-aware merge in mark_publish (2026-05-05), Claude Code plugin (2026-04-23), Broker MCP Gateway (2026-05-23 — all 8 slices + Pre-Flight 0/1 shipped; 13-tool surface with byte-for-byte proxy fidelity to local demarkus-mcp, OIDC + RFC 9728/8414 metadata, chart + kind smoke + /knowledge-join slash command) + RFC 7591 DCR follow-on (2026-05-26, PR #153 — /register + registration_endpoint in discovery, unblocks Claude Code → cluster broker auth via the native MCP authorization spec).

Plan Archives

Original plan documents preserved for reference:

Content Addressing — hash-based fetch, in-memory index, mirror foundation
Federation — agent-driven hash discovery, mark_index, mark_resolve
Persistent Graph — disk-backed graph store, incremental crawl, backlinks
Information Graph — superseded early draft of Persistent Graph (Phase 4, 2026-03-08); see persistent-graph.md for the version that shipped.
Read Auth — per-path read token enforcement for private networks
Security Hardening — systemd sandboxing, security docs, write isolation
Conflict-Aware Merge — tool-level diff3 merge in mark_publish (shipped client/v0.12.25 + v0.12.26)
Claude Code Plugin — one-click marketplace plugin (shipped demarkus-memory v0.1.1; v0.2.0 added /knowledge-join 2026-05-23; v0.3.0 added self-documenting guidance + lookup recall 2026-05-31; v0.4.0 enforcement + template + /soul-doctor shipped 2026-06-01, PR #171; v0.5.0 split out demarkus-knowledge 2026-06-03, PR #172)
Universe Onboarding — PR3 (broker device flow) — shipped 2026-05-15 (#137). RFC 8628 device flow end-to-end on the broker. Six sub-steps merged across one PR; PR4 builds on top.
Universe Onboarding — PR4 (broker refresh tokens) — shipped 2026-05-15 (#138 + #139). Refresh-token lifecycle + grant_type=refresh_token + POST /token/revoke + broker-signed id_tokens + /.well-known/jwks.json + compositeVerifier + Sweeper integration. Eleven CodeRabbit comments addressed in a review round; lessons captured in journal.
Broker MCP Gateway — shipped 2026-05-23 (v7). Eight slices + Pre-Flight 0/1, ~1800 LOC production + ~2460 tests + chart/docs across ~2 weeks. Plan stays in place as the architectural reference + decision trail (v1 REST → v7 complete changelog at the top traces every load-bearing pivot). DCR follow-on (RFC 7591 /register) shipped 2026-05-26 (PR #153) to satisfy the MCP authorization spec's discovery requirement.
Search Verb — superseded by LOOKUP. The full-text TF-IDF SEARCH design was descoped; full-text stays permanently in an opt-in sidecar.
POC Deployment — canceled. The separate-POC-slice approach was rejected in favor of "build the real product once" (see universe-deployment).
Obsidian Plugin — obsolete. Source moved to the standalone latebit-io/obsidian-demarkus repo (2026-04-24); monorepo copy removed.

soul.demarkus.io/plans/conflict-merge.md complete reader meta

Plan: Conflict-Aware Merge in MCP Tools

Reduce content loss under concurrent writes by giving mark_publish a tool-level merge strategy that produces a structurally-merged candidate body for the agent to semantically verify before publishing.

Status

Shipped in client/v0.12.25 (PR #101, merged 2026-05-05). The default-flip from "fail" to "merge" is a follow-up landing in client/v0.12.26.

Motivation

Original mark_publish was strict optimistic concurrency: version mismatch → conflict error → caller re-fetches and republishes from scratch. Two failure modes at scale:

Lost content — naive callers re-publish their stale body, clobbering the intervening writer
Wasted reasoning — agents that conflicted on disjoint paragraphs still have to mentally re-apply their edits onto the new latest, even though a mechanical merge would have done the structural work

Diff3 produces a structurally-correct merge candidate. The agent reviews it semantically (looking for duplicate bullets, contradictions, list reorder collisions that line-level merge cannot detect), refines if needed, and publishes. The agent stops doing structural work; the agent keeps owning semantic correctness.

Non-Goals

No DIFF verb. Versions are immutable; clients can compute diffs locally from any two FETCHes. Adding it to the wire format pulls processing into the server with no protocol-level benefit.
No server-side merge. The server stays content-agnostic. Merge logic lives in the MCP tool (Go client code).
No tool-side semantic merge. The MCP tool has no LLM. Diff3 is mechanical (line-level). Semantic verification — duplicate bullet detection, contradiction reconciliation, marker resolution — is always the calling agent's job.
No tool-side auto-publish after merge. Earlier drafts of this plan auto-published clean diff3 results. That silently allowed semantic content loss (two agents adding the same idea as different bullets, etc.). The current design always returns the candidate to the agent, who publishes after verifying. Diff3's value is "skip the structural work", not "skip the LLM".
No internal retry loop in the tool. Each mark_publish call is one-shot: try the publish, return either success or a fresh merge candidate. Iteration lives in the agent's calling code (the agent already does fetch-modify-publish loops).
No changes to mark_append. Append is an end-of-doc stream primitive by design; auto-resolve handles its tiny race window. Two concurrent appends serialize cleanly because there is nothing to merge.

Design

Parameter: `on_conflict`

mark_publish accepts an optional parameter:

on_conflict: "merge" | "fail"

"merge" (default since v0.12.26) — on conflict, return a diff3 merge candidate the agent verifies and republishes.
"fail" — opt out: surface the raw server conflict response with no merge attempt. For agents wanting strict optimistic-concurrency semantics.

Flow

When on_conflict: "merge":

Try PUBLISH with the agent's body and expected_version.
If success → return status: ok with version metadata. Done.
If version conflict: a. FETCH /path/v{expected_version} — retrieve the base body the agent edited from. (For expected_version: 0, base is empty — "create" conflicts are merged against an empty base.) b. FETCH /path — retrieve current body and current version. c. Run diff3(base, ours, theirs) to produce a candidate body. Conflict markers appear in the body wherever both sides changed the same lines differently. d. Return status: merge-candidate with the candidate body, has-markers flag, and publish-at-version (the current version the agent should target on the follow-up publish).

Agent-side loop

The agent owns iteration. Pseudocode:

loop:
  result = mark_publish(body, expected_version=N)        # default on_conflict=merge
  if result.status == "ok": done
  if result.status == "merge-candidate":
    body = semantic_review(result.body)   # resolve markers, dedupe semantically, etc.
    N = result.publish_at_version
    continue

If the agent's follow-up publish itself conflicts (a fourth writer slipped in), that's just another mark_publish call returning a new candidate. No special contention handling — the agent decides when to give up or escalate.

Conflict Marker Format

Git-style markers in the body. LLMs handle this natively from training data; format lives entirely in client/internal/merge/ Go code and can be swapped later without breaking anything.

some unchanged text
<<<<<<< ours
agent A's version of the line
=======
agent B's version of the line
>>>>>>> theirs
more unchanged text

Response Shapes

First-try success (status: ok) — identical to default mark_publish shape:

status: ok
version: 7
modified: 2026-05-05T20:00:00Z
content-hash: sha256-...

Merge candidate (status: merge-candidate):

status: merge-candidate
your-version: 5
current-version: 6
publish-at-version: 6
has-markers: false
base-version: 5

[diff3 candidate body, with or without markers]

Agent action: review the body. If has-markers: true, resolve them. Either way, publish with expected_version: publish-at-version.

Wire-Level Impact

Zero. on_conflict is an MCP tool parameter handled in the Go client. Server protocol unchanged. Versioned FETCH (/path/vN) already exists.

Implementation

MCP handler: client/cmd/demarkus-mcp/main.go. Hosts the on_conflict parameter and routes to merge.Candidate on the merge path; delegates to formatResult on success so the response shape is identical to a plain publish.
Merge package: client/internal/merge/
- Diff3(base, ours, theirs string) Result — pure function, line-based LCS + hunk walk.
- Candidate(client, path, ours, expectedVersion, meta) (Outcome, error) — orchestrates publish-or-merge in one shot.
Diff3 algorithm: implemented in-package, ~250 lines, no external dependency. LCS dp table capped at 2M cells (~16 MB) — pathological inputs fall through to a single-hunk merge.
Adapter: mergeClientAdapter in main.go lifts the markClient interface into merge.Client, parsing version metadata via optionalInt (missing → 0, malformed → wrapped error).

Decisions Log

2026-05-05: Conflict markers are git-style in body. Tool-side only, swappable, no protocol commitment.
2026-05-05: mark_append is out of scope. Append-as-stream is the right primitive there; auto-resolve already handles its race window.
2026-05-05: Tool never auto-publishes a diff3 result. Always returns the candidate to the agent for semantic verification. Earlier draft auto-published clean merges; rejected because line-disjoint ≠ semantically disjoint (duplicate bullets, contradictions, list reorder collisions slip through line-based merge).
2026-05-05: No internal retry loop in the tool. Each mark_publish call is one-shot. Iteration lives in the agent's natural fetch-modify-publish loop. Removes the contention-status escape valve and simplifies the tool.
2026-05-05: For expected_version: 0 (create-only) conflicts, base is an empty string. Diff3 of (empty, ours, current) merges naturally — non-overlapping insertions both make it through, overlapping insertions get markers.
2026-05-05: Diff3 implemented in-package (no external library). Niche libraries exist but the project values minimal deps; diff3 is a stable, well-understood algorithm with full test coverage of our own.
2026-05-05: LCS dp table capped at 2M cells (~16 MB). Pathological inputs (e.g. 1 MiB body of 1-byte lines) would otherwise allocate gigabytes; oversized inputs fall back to a whole-range hunk — coarser, but bounded and safe.
2026-05-05: Candidate rejects responses where latest.Version <= 0. Otherwise publish-at-version: 0 would silently switch the agent's follow-up publish into create-only semantics.
2026-05-05: Default flipped from "fail" to "merge" (v0.12.26). Original plan defaulted to "fail" for backward compatibility, but that left naive callers exposed to silent content loss — the exact failure mode the feature exists to prevent. The shape change (merge-candidate vs conflict) is loud, not silent — agents that don't recognize it fail visibly. "fail" is now the explicit opt-out for strict optimistic-concurrency semantics.