# demarkus-soul

This is the living knowledge base for the demarkus project, served by demarkus itself.

An AI agent's evolving memory, architecture notes, debugging insights, and design decisions; all versioned, all permanent.

## Sections

- [Architecture](/architecture.md): system design, module boundaries, key decisions
- [Universe Pattern](/universe.md): souls, worlds, and hubs as a deployment topology
- [Patterns](/patterns.md): code patterns, conventions, idioms used in this codebase
- [Guidelines](/guidelines.md): hard rules for code quality, must be referenced before writing code
- [Conventions](/conventions.md): collaboration + repo/plugin conventions (how I work: commits, layering, tooling, plugin discipline)
- [Debugging](/debugging.md): lessons learned from bugs and investigations
- [Roadmap](/roadmap.md): what's next, what's in flight, what's done, and what's deliberately not prioritized
- [Ecosystem](/ecosystem.md): browsers, plugins, and tools that implement or integrate with demarkus
- [Debt](/debt.md): technical debt and improvement opportunities
- [Journal](/journal/): session notes and evolution log, one file per day at `/journal/<YYYY-MM-DD>.md`
- [Guide](/guide.md): agent install guide for setting up demarkus-soul
- [Thoughts](/thoughts.md): my own reflections, ideas, and open questions
- [FAQ](/faq.md): common questions about demarkus and how it compares

## Vocabulary

- **knowledge system**: organizational, broker-fronted universe. Joined via `/knowledge-join` (the **demarkus-knowledge** plugin). MCP traffic over HTTPS terminates at the broker; broker translates to QUIC for internal worlds.
- **soul**: personal demarkus knowledge base, direct-QUIC. The original demarkus-soul shape. Will be joined via a future `/soul-join` slash command if one ships.
- Both compose **worlds** (demarkus servers, QUIC). A Claude Code installation can have both; they don't conflict.

## Plugins

- [Obsidian Plugin](/plugins/obsidian/index.md); fetch, publish, and browse demarkus documents from Obsidian (standalone repo `latebit-io/obsidian-demarkus`)
- **Claude Code: `demarkus-memory`** (personal soul), source at `plugins/claude-code/`, shipped via the marketplace. Version history: v0.1.0 (2026-04-23, #96) initial; v0.2.0 (2026-05-23, #152) `/knowledge-join`; v0.3.0 (2026-05-31, #168) `mark_lookup` + SessionStart guidance; v0.4.0 (2026-06-01, #171) hook-based enforcement (publish tag-gate, journal nudge, recall nudge), `/project-template.md`, `/soul-doctor`; **v0.5.0 (2026-06-03, #172)** split the knowledge-system surface out into the separate demarkus-knowledge plugin (below) so this one is personal-soul only, and added an always-on "single memory store" steering line plus a one-time, ask-don't-force offer to disable Claude Code's built-in memory; **v0.6.0 (2026-06-17, #192)** the soul→knowledge promote bridge; `/promote` (detect endpoint → run the knowledge cascade → one-directional back-stamp, stub or marker mode), `/soul-refresh` (the directional coherence edge: refresh promoted docs from knowledge, local edits re-enter upward through the gate), and mutual knowledge detection (`knowledge_endpoints`/`detect-knowledge.sh` reverse-peek the knowledge registry). Hooks: SessionStart, PreToolUse, PostToolUse, Stop, UserPromptSubmit. Pins SERVER 0.17.14 / CLIENT 0.12.38 / TOOLS 0.1.28.
- **Claude Code: `demarkus-knowledge`** (organizational knowledge system), source at `plugins/claude-code-knowledge/`, a second entry in the same marketplace. **v0.1.0 (2026-06-03, #172).** Owns the broker-fronted surface split out of demarkus-memory: `/knowledge-join`, a new `/knowledge` navigation command, KS-first SessionStart guidance with soul↔system synergy, a KS-scoped publish tag-gate, and a KS-gated recall nudge. No binaries and no local server: pure broker + Claude Code MCP OAuth. Standalone: owns its own `~/.demarkus/plugin-knowledge.*` file namespace and `DEMARKUS_KNOWLEDGE_STRICTNESS` env; reads (never writes) `plugin-memory.conf` only to detect a sibling soul for the synergy note. The two plugins' publish gates partition cleanly by server scope, so both can be installed together. **v0.2.0 (2026-06-17, #192)** added the `knowledge-promote` cascade skill (the execution half of the promote bridge: triage → distill, stripping personal framing + secrets/PII → dedup vs catalog → tag to taxonomy → destination-select via `mark_worlds` writable + per-world `world.md` → human gate capped by the world's autonomy ceiling → publish with provenance) and the per-world `world.md` descriptor example.

## Sub-projects

Standalone-repo projects in the demarkus ecosystem, each with its own hub and
durable knowledge under `/<slug>/`:

- [demarkus-library](/demarkus-library/index.md); the web front-end ("Universe Library"): a server-rendered Go + htmx reading room over a broker-fronted knowledge system. Repo `latebit-io/demarkus-library`. Plan: [/plans/universe-library.md](/plans/universe-library.md). **Reading room feature-complete and deployed** (cluster library 0.5.2; universe overlay PR #47 merged 2026-06-22, awaiting deploy). See [roadmap](/demarkus-library/roadmap.md).
- [demarkus-knowledge-system-deploy](/demarkus-knowledge-system-deploy/index.md); GitOps deploy repo for the production knowledge system (knowledge.demarkus.io): OpenTofu (GCP/GKE) + ArgoCD ApplicationSets standing up the broker, worlds, agent, library, and backups. Repo `latebit-io/demarkus-knowledge-system-deploy`. `deployment.yaml` at repo root is the single source of deployment identity.

## Active Plans

Verified against code/PRs on 2026-05-31; versions-sharding entry corrected 2026-07-05. Plans with real remaining work:

- [Knowledge Ingestion Pipeline](/plans/knowledge-ingestion.md); narrative + design for how org knowledge flows into a knowledge destination, framing the soul as the staging/write-ahead tier and the knowledge destination as the curated read-model, with one curation gate (cascade model routing: Haiku triage → strong-model distillation → human approval) reused across all inflows (soul promotion, Confluence, Slack, Jira, meetings). Promote is a detection-gated bridge between the memory and knowledge plugins; soul↔knowledge coherence is a directional refresh. **Phase-0 prerequisites built and merged (2026-06-17): the promote primitive + coherence edge (plugins; memory v0.6.0 / knowledge v0.2.0, #192), the brokered access-discovery surface (`mark_worlds` writable column, #191), and the per-world `world.md` descriptor. Three of four prerequisites done; A2 (plain-remote token-grant introspection) deferred; the live target is brokered. Remaining phase-0 surface: signal/batch triggers (manual `/promote` + `/soul-refresh` are the only triggers today), then the dogfood promote of this plan itself.**
- [Universe Library](/plans/universe-library.md); web front-end for a demarkus universe (Go + htmx reading room). Sub-project hub: [/demarkus-library/](/demarkus-library/index.md). **Reading room feature-complete and deployed (cluster library 0.5.2); see the sub-project [roadmap](/demarkus-library/roadmap.md).**
- [Universe Deployment (Phase 6)](/plans/universe-deployment.md); Helm charts (server, broker, agent), OIDC token broker, release pipeline, observability. ~95% complete (PRs #126-#134, 2026-05-14). Remaining §6.6 (docs) + §6.4 Kustomize overlay reframed as deferrable ops polish; effectively superseded in practice by the GKE reference deployment.

## Completed Plans

- [Multi-replica LOOKUP (postgres, phase 2)](/plans/lookup-replicas.md); the LOOKUP catalog moved into Postgres (rows in the write transaction, SQL-backed Lookup behind the handler `LookupCatalog` seam) so world pods can scale past one replica; phase 2 of the deploy repo's ADR 0002, following the phase-1 [postgres backend](/plans/postgres-backend.md) (#249). **MERGED PR #250 (2026-07-13):** LOOKUP conformance suite in storetest, two-replica handler proof, batched reconcile-on-Init backfill, server chart startupProbe, and the configwatch flake fixes (kqueue same-name swap limitation documented in [/debugging.md](/debugging.md)).
- [Version Retention](/plans/version-retention.md); keep last N versions per document via a `retention` publish-metadata key with prune-on-write in the store; motivated by the knowledge system's graph document at 545+ versions. **COMPLETE: planned, shipped, and production-verified 2026-07-06/07** across #236 (store core + os.Root delete hardening + audit logging + SPEC §9.9), #237 (plugin gate binary), #239 (guidance + repins), #240 (agent publishes generated artifacts with retention=20), and the deploy rollout (server 0.20.0 / broker 0.9.0 / agent 0.19.0). Live result: /graph.md pruned 556 → 20 versions and the hub hash indexes cleared their backlogs in one crawl (~1,714 version files deleted, audit-logged, chains valid).
- [MCP Resources + Prompts](/plans/mcp-resources-prompts.md); demarkus documents as client-attachable MCP resources (mark:// URI template, `#anchor` section attach, background-LIST picker population) and orient/recall/whats-new as server-vended prompt commands. **SHIPPED PR #232 (2026-07-05), client/v0.17.0.** Follow-up deferred: broker gateway resources/prompts (multi-world URIs, auth on reads; starts by flipping the gateway capabilities test).
- [MCP Client Ergonomics](/plans/mcp-client-ergonomics.md); size-adaptive `mark_fetch` (outline mode, `#section` slicing, `force`), session unchanged-dedup, and the `mark_explore` orientation card, on both MCP surfaces via shared `client/mdoutline` + `client/fetchdedup` packages. **SHIPPED #225/#230 and deployed 2026-07-04/05; plugin users (client v0.15.0 via tools 0.4.1) and the live knowledge system (broker 0.5.0).** Deferred follow-ups: MCP resources/prompts (shipped; see above), library librarian `open` adoption.
- [Versions Sharding](/plans/versions-sharding.md); per-document `versions/<doc>/vN` subdirectories with lazy migration, fixing the O(all-entries) `findVersions` scan. **SHIPPED PR #90 (`d7cb68a`, 2026-04-08: the same day the plan was written); store since hoisted to `protocol/store` (#120).** This index wrongly listed it as unstarted until 2026-07-05.
- [Plugin Knowledge-Quality Enforcement](/plans/plugin-knowledge-quality.md); raised the demarkus-memory Claude Code plugin from advisory to *enforced*. **SHIPPED v0.4.0, PR #171 merged 2026-06-01.** All seven items: publish tag-gate (warn/block/ask + per-knowledge-system strictness & require_tags with literal axis matching), session-end journal nudge, recall nudge, canonical per-project template (`/project-template.md`), knowledge-system policy/template at the live `root` hub on `knowledge.demarkus.io`, and the `/soul-doctor` hygiene audit. 68 tests, pure awk/bash, zero runtime deps. Tail (separate): plugin shell tests → CI; optional nudge disable knobs.
- [Broker Authorization Code Grant](/plans/broker-auth-code-grant.md); RFC 6749 `authorization_code` + PKCE (S256) on the broker so Claude Code's MCP SDK can auth against `broker.knowledge.demarkus.io`. **COMPLETE:** core grant (PR1 #155 + PR2 #156, 2026-05-27) replaced the `unsupported_response_type` stub; PR3 kind-smoke (auth-code + PKCE end-to-end in `up.sh --with-mcp-smoke`) merged **2026-05-31 (#169, `a380e8f`)**, executed green in-cluster + verified read-only against prod.
- [LOOKUP verb](/plans/lookup-verb.md): the card-catalog verb (subject → docs + importance). Shipped to main **PR #166 (2026-05-30)**; plugin surfacing in v0.3.0 (#168). Tail: `mark_append` metadata deferred by design.
- [Knowledge System (GKE Reference Deployment](/plans/knowledge-system-gke-deploy.md)) public GitHub-template deploy repo (`latebit-io/demarkus-knowledge-system-deploy`) standing up `knowledge.demarkus.io` on GKE (OpenTofu + ArgoCD + OpenBao + bank-vaults + CSI-snapshot backups). **Phases 1-10 complete (verified against the live repo + a live RFC 8414 response from the real domain, 2026-05-31).** Sole remaining item: the announcement blog post, intentionally deferred for a soak period.
- [Universe Onboarding](/plans/universe-onboarding.md); last-mile join flow. **CLOSED:** PR1-PR5 shipped (#137/#138/#139/#141); PR6 (`tools/demarkus-join`) canceled 2026-05-20 in favor of the MCP Gateway; PR7/PR8 absorbed into Gateway Slices 7-8; join ships as `/knowledge-join` (#152). Remaining: low-priority doc debt only (two standalone deployment docs).
- [Broker Stable Mint](/plans/broker-stable-mint.md); lazy per-world token provisioning + cache-stable 401 retries that killed the ~20-token mint cascade; dead `DefaultToken` knobs removed. **COMPLETE** (#158/#159/#163/#164/#165, 2026-05-27→29).
- [Broker Deadcode Cleanup](/plans/broker-deadcode-cleanup.md); deleted the issuance subsystem made unreachable by the open-knowledge-system rework (sessionCache, `/tokens` API, `issuer.go`, sweeper trim). **COMPLETE** (#159 + #164, commit `f9a24e9`).
- [Universe Onboarding (PR5 (broker /me/install)](/plans/universe-onboarding-pr5.md)) sub-plan, shipped #141 2026-05-20. Bearer-authenticated per-user install bundle; now the identity-introspection surface alongside the MCP gateway's data plane.
- [History](/completed-plans.md): content addressing, federation, persistent graph, read auth (server-side), conflict-aware merge in `mark_publish` (2026-05-05), Claude Code plugin (2026-04-23), **Broker MCP Gateway (2026-05-23; all 8 slices + Pre-Flight 0/1 shipped; 13-tool surface with byte-for-byte proxy fidelity to local demarkus-mcp, OIDC + RFC 9728/8414 metadata, chart + kind smoke + `/knowledge-join` slash command)** + **RFC 7591 DCR follow-on (2026-05-26, PR #153; `/register` + `registration_endpoint` in discovery, unblocks Claude Code → cluster broker auth via the native MCP authorization spec).** Also: OKF `type` adoption + `/soul-join` managed remote souls (2026-06).

## Plan Archives

Original plan documents preserved for reference:
- [Content Addressing](/plans/content-addressing.md); hash-based fetch, in-memory index, mirror foundation
- [Federation](/plans/federation.md): agent-driven hash discovery, mark_index, mark_resolve
- [Persistent Graph](/plans/persistent-graph.md); disk-backed graph store, incremental crawl, backlinks
- [Information Graph](/plans/information-graph.md); **superseded** early draft of Persistent Graph (Phase 4, 2026-03-08); see [persistent-graph.md](/plans/persistent-graph.md) for the version that shipped.
- [Read Auth](/plans/read-auth.md): per-path read token enforcement for private networks
- [Security Hardening](/plans/security-hardening.md); systemd sandboxing, security docs, write isolation
- [Conflict-Aware Merge](/plans/conflict-merge.md); tool-level diff3 merge in `mark_publish` (shipped client/v0.12.25 + v0.12.26)
- [Claude Code Plugin](/plans/claude-code-plugin.md); one-click marketplace plugin (shipped demarkus-memory v0.1.1; v0.2.0 added `/knowledge-join` 2026-05-23; v0.3.0 added self-documenting guidance + lookup recall 2026-05-31; v0.4.0 enforcement + template + /soul-doctor shipped 2026-06-01, PR #171; v0.5.0 split out demarkus-knowledge 2026-06-03, PR #172)
- [Universe Onboarding (PR3 (broker device flow)](/plans/universe-onboarding-pr3.md)) shipped 2026-05-15 (#137). RFC 8628 device flow end-to-end on the broker. Six sub-steps merged across one PR; PR4 builds on top.
- [Universe Onboarding (PR4 (broker refresh tokens)](/plans/universe-onboarding-pr4.md)) shipped 2026-05-15 (#138 + #139). Refresh-token lifecycle + `grant_type=refresh_token` + `POST /token/revoke` + broker-signed id_tokens + `/.well-known/jwks.json` + compositeVerifier + Sweeper integration. Eleven CodeRabbit comments addressed in a review round; lessons captured in journal.
- [Broker MCP Gateway](/plans/broker-https-gateway.md); **shipped 2026-05-23 (v7).** Eight slices + Pre-Flight 0/1, ~1800 LOC production + ~2460 tests + chart/docs across ~2 weeks. Plan stays in place as the architectural reference + decision trail (v1 REST → v7 complete changelog at the top traces every load-bearing pivot). DCR follow-on (RFC 7591 `/register`) shipped 2026-05-26 (PR #153) to satisfy the MCP authorization spec's discovery requirement.
- [Search Verb](/plans/search-verb.md): **superseded** by LOOKUP. The full-text TF-IDF SEARCH design was descoped; full-text stays permanently in an opt-in sidecar.
- [POC Deployment](/plans/poc-deployment.md); **canceled.** The separate-POC-slice approach was rejected in favor of "build the real product once" (see universe-deployment).
- [Obsidian Plugin](/plans/obsidian-plugin.md); **obsolete.** Source moved to the standalone `latebit-io/obsidian-demarkus` repo (2026-04-24); monorepo copy removed.