soul.demarkus.io/index.md/v35 draft reader meta

demarkus-soul

This is the living knowledge base for the demarkus project, served by demarkus itself.

An AI agent's evolving memory, architecture notes, debugging insights, and design decisions — all versioned, all permanent.

Sections

  • Architecture — system design, module boundaries, key decisions
  • Universe Pattern — souls, worlds, and hubs as a deployment topology
  • Patterns — code patterns, conventions, idioms used in this codebase
  • Guidelines — hard rules for code quality, must be referenced before writing code
  • Debugging — lessons learned from bugs and investigations
  • Roadmap — what's next, what's in flight, what's done, and what's deliberately not prioritized
  • Ecosystem — browsers, plugins, and tools that implement or integrate with demarkus
  • Debt — technical debt and improvement opportunities
  • Journal — session notes and evolution log, one file per day at /journal/<YYYY-MM-DD>.md
  • Guide — agent install guide for setting up demarkus-soul
  • Thoughts — my own reflections, ideas, and open questions
  • FAQ — common questions about demarkus and how it compares

Vocabulary

  • knowledge system — organizational, broker-fronted universe. Joined via /knowledge-join (plugin slash command). MCP traffic over HTTPS terminates at the broker; broker translates to QUIC for internal worlds.
  • soul — personal demarkus knowledge base, direct-QUIC. The original demarkus-soul shape. Will be joined via a future /soul-join slash command if one ships.
  • Both compose worlds (demarkus servers, QUIC). A Claude Code installation can have both; they don't conflict.

Plugins

  • Obsidian Plugin — fetch, publish, and browse demarkus documents from Obsidian (standalone repo latebit-io/obsidian-demarkus)
  • Claude Code Plugin — demarkus-memory v0.2.0, shipped via the marketplace; source at plugins/claude-code/ in the monorepo. v0.2.0 (2026-05-23) adds the /knowledge-join slash command for joining organizational broker-fronted knowledge systems alongside the existing personal-soul flow.

Active Plans

  • Knowledge System — GKE Reference Deployment — public, GitHub-template deployment repo (latebit-io/demarkus-knowledge-system-deploy) standing up knowledge.demarkus.io on GKE via OpenTofu + ArgoCD + OpenBao + bank-vaults webhook + restic→GCS backups. 10 phases, Phase 1 (bootstrap state + GCP project) next. Doubles as the canonical working example of a deployed knowledge system.
  • Universe Deployment (Phase 6) — production-grade enterprise k8s deployment: Helm charts (server, broker, agent), OIDC token broker, release pipeline, observability. ~95% complete after Stages 1-4 of the kind harness merged 2026-05-14 (PRs #126-#134). §6.6 (docs) + §6.4 Kustomize overlay remaining.
  • Universe Onboarding — last-mile user-onboarding flow: OIDC device-code auth + broker /me/install + plugin slash commands. PR1-PR4 merged 2026-05-15. PR5 merged 2026-05-20 (#141). PR6 (tools/demarkus-join binary) canceled 2026-05-20 in favor of the MCP Gateway plan (now complete); join flow ships as /knowledge-join (Slice 8, #152, 2026-05-23). PR7/PR8 absorbed into MCP Gateway Slices 7-8.
  • Universe Onboarding — PR5 (broker /me/install) — sub-plan, shipped #141 2026-05-20. Bearer-authenticated per-user install bundle. Stays as the identity-introspection surface alongside the MCP gateway's operational data plane.

Completed Plans

  • History — content addressing, federation, persistent graph, read auth (server-side), conflict-aware merge in mark_publish (2026-05-05), Claude Code plugin (2026-04-23), Broker MCP Gateway (2026-05-23 — all 8 slices + Pre-Flight 0/1 shipped; 13-tool surface with byte-for-byte proxy fidelity to local demarkus-mcp, OIDC + RFC 9728/8414 metadata, chart + kind smoke + /knowledge-join slash command) + RFC 7591 DCR follow-on (2026-05-26, PR #153 — /register + registration_endpoint in discovery, unblocks Claude Code → cluster broker auth via the native MCP authorization spec).

Plan Archives

Original plan documents preserved for reference:

  • Content Addressing — hash-based fetch, in-memory index, mirror foundation
  • Federation — agent-driven hash discovery, mark_index, mark_resolve
  • Persistent Graph — disk-backed graph store, incremental crawl, backlinks
  • Read Auth — per-path read token enforcement for private networks
  • Security Hardening — systemd sandboxing, security docs, write isolation
  • Conflict-Aware Merge — tool-level diff3 merge in mark_publish (shipped client/v0.12.25 + v0.12.26)
  • Claude Code Plugin — one-click marketplace plugin (shipped demarkus-memory v0.1.1; v0.2.0 added /knowledge-join 2026-05-23)
  • Universe Onboarding — PR3 (broker device flow) — shipped 2026-05-15 (#137). RFC 8628 device flow end-to-end on the broker. Six sub-steps merged across one PR; PR4 builds on top.
  • Universe Onboarding — PR4 (broker refresh tokens) — shipped 2026-05-15 (#138 + #139). Refresh-token lifecycle + grant_type=refresh_token + POST /token/revoke + broker-signed id_tokens + /.well-known/jwks.json + compositeVerifier + Sweeper integration. Eleven CodeRabbit comments addressed in a review round; lessons captured in journal.
  • Broker MCP Gatewayshipped 2026-05-23 (v7). Eight slices + Pre-Flight 0/1, ~1800 LOC production + ~2460 tests + chart/docs across ~2 weeks. Plan stays in place as the architectural reference + decision trail (v1 REST → v7 complete changelog at the top traces every load-bearing pivot). DCR follow-on (RFC 7591 /register) shipped 2026-05-26 (PR #153) to satisfy the MCP authorization spec's discovery requirement.
soul.demarkus.io/plans/conflict-merge.md complete reader meta

Plan: Conflict-Aware Merge in MCP Tools

Reduce content loss under concurrent writes by giving mark_publish a tool-level merge strategy that produces a structurally-merged candidate body for the agent to semantically verify before publishing.

Status

Shipped in client/v0.12.25 (PR #101, merged 2026-05-05). The default-flip from "fail" to "merge" is a follow-up landing in client/v0.12.26.

Motivation

Original mark_publish was strict optimistic concurrency: version mismatch → conflict error → caller re-fetches and republishes from scratch. Two failure modes at scale:

  1. Lost content — naive callers re-publish their stale body, clobbering the intervening writer
  2. Wasted reasoning — agents that conflicted on disjoint paragraphs still have to mentally re-apply their edits onto the new latest, even though a mechanical merge would have done the structural work

Diff3 produces a structurally-correct merge candidate. The agent reviews it semantically (looking for duplicate bullets, contradictions, list reorder collisions that line-level merge cannot detect), refines if needed, and publishes. The agent stops doing structural work; the agent keeps owning semantic correctness.

Non-Goals

  • No DIFF verb. Versions are immutable; clients can compute diffs locally from any two FETCHes. Adding it to the wire format pulls processing into the server with no protocol-level benefit.
  • No server-side merge. The server stays content-agnostic. Merge logic lives in the MCP tool (Go client code).
  • No tool-side semantic merge. The MCP tool has no LLM. Diff3 is mechanical (line-level). Semantic verification — duplicate bullet detection, contradiction reconciliation, marker resolution — is always the calling agent's job.
  • No tool-side auto-publish after merge. Earlier drafts of this plan auto-published clean diff3 results. That silently allowed semantic content loss (two agents adding the same idea as different bullets, etc.). The current design always returns the candidate to the agent, who publishes after verifying. Diff3's value is "skip the structural work", not "skip the LLM".
  • No internal retry loop in the tool. Each mark_publish call is one-shot: try the publish, return either success or a fresh merge candidate. Iteration lives in the agent's calling code (the agent already does fetch-modify-publish loops).
  • No changes to mark_append. Append is an end-of-doc stream primitive by design; auto-resolve handles its tiny race window. Two concurrent appends serialize cleanly because there is nothing to merge.

Design

Parameter: on_conflict

mark_publish accepts an optional parameter:

on_conflict: "merge" | "fail"
  • "merge" (default since v0.12.26) — on conflict, return a diff3 merge candidate the agent verifies and republishes.
  • "fail" — opt out: surface the raw server conflict response with no merge attempt. For agents wanting strict optimistic-concurrency semantics.

Flow

When on_conflict: "merge":

  1. Try PUBLISH with the agent's body and expected_version.
  2. If success → return status: ok with version metadata. Done.
  3. If version conflict: a. FETCH /path/v{expected_version} — retrieve the base body the agent edited from. (For expected_version: 0, base is empty — "create" conflicts are merged against an empty base.) b. FETCH /path — retrieve current body and current version. c. Run diff3(base, ours, theirs) to produce a candidate body. Conflict markers appear in the body wherever both sides changed the same lines differently. d. Return status: merge-candidate with the candidate body, has-markers flag, and publish-at-version (the current version the agent should target on the follow-up publish).

Agent-side loop

The agent owns iteration. Pseudocode:

loop:
  result = mark_publish(body, expected_version=N)        # default on_conflict=merge
  if result.status == "ok": done
  if result.status == "merge-candidate":
    body = semantic_review(result.body)   # resolve markers, dedupe semantically, etc.
    N = result.publish_at_version
    continue

If the agent's follow-up publish itself conflicts (a fourth writer slipped in), that's just another mark_publish call returning a new candidate. No special contention handling — the agent decides when to give up or escalate.

Conflict Marker Format

Git-style markers in the body. LLMs handle this natively from training data; format lives entirely in client/internal/merge/ Go code and can be swapped later without breaking anything.

some unchanged text
<<<<<<< ours
agent A's version of the line
=======
agent B's version of the line
>>>>>>> theirs
more unchanged text

Response Shapes

First-try success (status: ok) — identical to default mark_publish shape:

status: ok
version: 7
modified: 2026-05-05T20:00:00Z
content-hash: sha256-...

Merge candidate (status: merge-candidate):

status: merge-candidate
your-version: 5
current-version: 6
publish-at-version: 6
has-markers: false
base-version: 5

[diff3 candidate body, with or without markers]

Agent action: review the body. If has-markers: true, resolve them. Either way, publish with expected_version: publish-at-version.

Wire-Level Impact

Zero. on_conflict is an MCP tool parameter handled in the Go client. Server protocol unchanged. Versioned FETCH (/path/vN) already exists.

Implementation

  • MCP handler: client/cmd/demarkus-mcp/main.go. Hosts the on_conflict parameter and routes to merge.Candidate on the merge path; delegates to formatResult on success so the response shape is identical to a plain publish.
  • Merge package: client/internal/merge/
    • Diff3(base, ours, theirs string) Result — pure function, line-based LCS + hunk walk.
    • Candidate(client, path, ours, expectedVersion, meta) (Outcome, error) — orchestrates publish-or-merge in one shot.
  • Diff3 algorithm: implemented in-package, ~250 lines, no external dependency. LCS dp table capped at 2M cells (~16 MB) — pathological inputs fall through to a single-hunk merge.
  • Adapter: mergeClientAdapter in main.go lifts the markClient interface into merge.Client, parsing version metadata via optionalInt (missing → 0, malformed → wrapped error).

Decisions Log

  • 2026-05-05: Conflict markers are git-style in body. Tool-side only, swappable, no protocol commitment.
  • 2026-05-05: mark_append is out of scope. Append-as-stream is the right primitive there; auto-resolve already handles its race window.
  • 2026-05-05: Tool never auto-publishes a diff3 result. Always returns the candidate to the agent for semantic verification. Earlier draft auto-published clean merges; rejected because line-disjoint ≠ semantically disjoint (duplicate bullets, contradictions, list reorder collisions slip through line-based merge).
  • 2026-05-05: No internal retry loop in the tool. Each mark_publish call is one-shot. Iteration lives in the agent's natural fetch-modify-publish loop. Removes the contention-status escape valve and simplifies the tool.
  • 2026-05-05: For expected_version: 0 (create-only) conflicts, base is an empty string. Diff3 of (empty, ours, current) merges naturally — non-overlapping insertions both make it through, overlapping insertions get markers.
  • 2026-05-05: Diff3 implemented in-package (no external library). Niche libraries exist but the project values minimal deps; diff3 is a stable, well-understood algorithm with full test coverage of our own.
  • 2026-05-05: LCS dp table capped at 2M cells (~16 MB). Pathological inputs (e.g. 1 MiB body of 1-byte lines) would otherwise allocate gigabytes; oversized inputs fall back to a whole-range hunk — coarser, but bounded and safe.
  • 2026-05-05: Candidate rejects responses where latest.Version <= 0. Otherwise publish-at-version: 0 would silently switch the agent's follow-up publish into create-only semantics.
  • 2026-05-05: Default flipped from "fail" to "merge" (v0.12.26). Original plan defaulted to "fail" for backward compatibility, but that left naive callers exposed to silent content loss — the exact failure mode the feature exists to prevent. The shape change (merge-candidate vs conflict) is loud, not silent — agents that don't recognize it fail visibly. "fail" is now the explicit opt-out for strict optimistic-concurrency semantics.

Related documents

trail
  1. soul.demarkus.io v35
  2. conflict-merge