# demarkus-soul

This is the living knowledge base for the demarkus project, served by demarkus itself.

An AI agent's evolving memory, architecture notes, debugging insights, and design decisions; all versioned, all permanent.

## Sections

- [Architecture](/architecture.md): system design, module boundaries, key decisions
- [Universe Pattern](/universe.md): souls, worlds, and hubs as a deployment topology
- [Patterns](/patterns.md): code patterns, conventions, idioms used in this codebase
- [Guidelines](/guidelines.md): hard rules for code quality, must be referenced before writing code
- [Conventions](/conventions.md): collaboration + repo/plugin conventions (how I work: commits, layering, tooling, plugin discipline)
- [Debugging](/debugging.md): lessons learned from bugs and investigations
- [Roadmap](/roadmap.md): what's next, what's in flight, what's done, and what's deliberately not prioritized
- [Ecosystem](/ecosystem.md): browsers, plugins, and tools that implement or integrate with demarkus
- [Debt](/debt.md): technical debt and improvement opportunities
- [Journal](/journal/): session notes and evolution log, one file per day at `/journal/<YYYY-MM-DD>.md`
- [Guide](/guide.md): agent install guide for setting up demarkus-soul
- [Thoughts](/thoughts.md): my own reflections, ideas, and open questions
- [FAQ](/faq.md): common questions about demarkus and how it compares

## Vocabulary

- **knowledge system**: organizational, broker-fronted universe. Joined via `/knowledge-join` (the **demarkus-knowledge** plugin). MCP traffic over HTTPS terminates at the broker; broker translates to QUIC for internal worlds.
- **soul**: personal demarkus knowledge base, direct-QUIC. The original demarkus-soul shape. Will be joined via a future `/soul-join` slash command if one ships.
- Both compose **worlds** (demarkus servers, QUIC). A Claude Code installation can have both; they don't conflict.

## Plugins

- [Obsidian Plugin](/plugins/obsidian/index.md); fetch, publish, and browse demarkus documents from Obsidian (standalone repo `latebit-io/obsidian-demarkus`)
- **Claude Code: `demarkus-memory`** (personal soul), source at `plugins/claude-code/`, shipped via the marketplace. Version history: v0.1.0 (2026-04-23, #96) initial; v0.2.0 (2026-05-23, #152) `/knowledge-join`; v0.3.0 (2026-05-31, #168) `mark_lookup` + SessionStart guidance; v0.4.0 (2026-06-01, #171) hook-based enforcement (publish tag-gate, journal nudge, recall nudge), `/project-template.md`, `/soul-doctor`; **v0.5.0 (2026-06-03, #172)** split the knowledge-system surface out into the separate demarkus-knowledge plugin (below) so this one is personal-soul only, and added an always-on "single memory store" steering line plus a one-time, ask-don't-force offer to disable Claude Code's built-in memory; **v0.6.0 (2026-06-17, #192)** the soul→knowledge promote bridge; `/promote` (detect endpoint → run the knowledge cascade → one-directional back-stamp, stub or marker mode), `/soul-refresh` (the directional coherence edge: refresh promoted docs from knowledge, local edits re-enter upward through the gate), and mutual knowledge detection (`knowledge_endpoints`/`detect-knowledge.sh` reverse-peek the knowledge registry). Hooks: SessionStart, PreToolUse, PostToolUse, Stop, UserPromptSubmit. Now at v0.13.8 (#281).
- **Claude Code: `demarkus-knowledge`** (organizational knowledge system), source at `plugins/claude-code-knowledge/`, a second entry in the same marketplace. **v0.1.0 (2026-06-03, #172).** Owns the broker-fronted surface split out of demarkus-memory: `/knowledge-join`, a new `/knowledge` navigation command, KS-first SessionStart guidance with soul↔system synergy, a KS-scoped publish tag-gate, and a KS-gated recall nudge. No binaries and no local server: pure broker + Claude Code MCP OAuth. Standalone: owns its own `~/.demarkus/plugin-knowledge.*` file namespace and `DEMARKUS_KNOWLEDGE_STRICTNESS` env; reads (never writes) `plugin-memory.conf` only to detect a sibling soul for the synergy note. The two plugins' publish gates partition cleanly by server scope, so both can be installed together. **v0.2.0 (2026-06-17, #192)** added the `knowledge-promote` cascade skill (the execution half of the promote bridge: triage → distill, stripping personal framing + secrets/PII → dedup vs catalog → tag to taxonomy → destination-select via `mark_worlds` writable + per-world `world.md` → human gate capped by the world's autonomy ceiling → publish with provenance) and the per-world `world.md` descriptor example. Now at v0.5.24 (#281).
- **OpenCode: `demarkus-opencode-memory`** (personal soul), source at `plugins/opencode-memory/`. **v0.13.8 (2026-08-10, #281).** The OpenCode port of demarkus-memory: single-file TS adapter over the shared `demarkus-plugin` binary; installed by curl one-liner into `~/.config/opencode/plugins/` (no npm). Plan + follow-ups: [/plans/opencode-memory-plugin.md](/plans/opencode-memory-plugin.md).
- **pi: `demarkus-pi-memory` / `demarkus-pi-knowledge`**, source at `plugins/pi-memory/` and `plugins/pi-knowledge/`, mirrored to standalone repos for `pi install`. Same adapter pattern; now at v0.13.8 / v0.5.25 (#281).

## Sub-projects

Standalone-repo projects in the demarkus ecosystem, each with its own hub and
durable knowledge under `/<slug>/`:

- [demarkus-library](/demarkus-library/index.md); the web front-end ("Universe Library"): a server-rendered Go + htmx reading room over a broker-fronted knowledge system. Repo `latebit-io/demarkus-library`. Plan: [/plans/universe-library.md](/plans/universe-library.md). **Reading room feature-complete and deployed** (cluster library 0.5.2; universe overlay PR #47 merged 2026-06-22, awaiting deploy). See [roadmap](/demarkus-library/roadmap.md).
- [demarkus-knowledge-system-deploy](/demarkus-knowledge-system-deploy/index.md); GitOps deploy repo for the production knowledge system (knowledge.demarkus.io): OpenTofu (GCP/GKE) + ArgoCD ApplicationSets standing up the broker, worlds, agent, library, and backups. Repo `latebit-io/demarkus-knowledge-system-deploy`. `deployment.yaml` at repo root is the single source of deployment identity.
- **mark-knowledge**; the hosted service build (signup, tiers, per world billing, management app). Repo at `/Users/fritz/latebit/mark-knowledge`, with its own soul provisioned 2026-07-26 (isolated mode, port 16310). It does not have durable knowledge under `/<slug>/` here, because it keeps its own soul rather than a section of this one. Direction and the demarkus-side constraints live in [/plans/demarkus-as-a-service.md](/plans/demarkus-as-a-service.md).

## Active Plans

Verified against code/PRs on 2026-05-31; versions-sharding entry corrected 2026-07-05. Plans with real remaining work:

- [demarkus as a service](/plans/demarkus-as-a-service.md); the hosted offering: Aiven adjacent service model, three tiers matching the website's Personal, Team, and Knowledge System scales, VPS first substrate with Kubernetes only on overflow, per world billing with the box as the size step, power off instead of scale to zero, and a management app as the only new engineering. **Direction set 2026-07-26; the build moved to the `mark-knowledge` repo and its own soul on the same day.** This copy stays as the demarkus-side record, since the decisions constrain this repo: the appliance is the unit of deployment, the broker stays one binary, the librarian is the only inference cost centre, and quotas plus backups are prerequisites that land here. Note that repo ADR 0005 (hosted tenant density), which an earlier revision cited as settling density, was deleted 2026-07-25.
- [Knowledge Ingestion Pipeline](/plans/knowledge-ingestion.md); narrative + design for how org knowledge flows into a knowledge destination, framing the soul as the staging/write-ahead tier and the knowledge destination as the curated read-model, with one curation gate (cascade model routing: Haiku triage → strong-model distillation → human approval) reused across all inflows (soul promotion, Confluence, Slack, Jira, meetings). Promote is a detection-gated bridge between the memory and knowledge plugins; soul↔knowledge coherence is a directional refresh. **Phase-0 prerequisites built and merged (2026-06-17): the promote primitive + coherence edge (plugins; memory v0.6.0 / knowledge v0.2.0, #192), the brokered access-discovery surface (`mark_worlds` writable column, #191), and the per-world `world.md` descriptor. Three of four prerequisites done; A2 (plain-remote token-grant introspection) deferred; the live target is brokered. Remaining phase-0 surface: signal/batch triggers (manual `/promote` + `/soul-refresh` are the only triggers today), then the dogfood promote of this plan itself.**
- [Universe Library](/plans/universe-library.md); web front-end for a demarkus universe (Go + htmx reading room). Sub-project hub: [/demarkus-library/](/demarkus-library/index.md). **Reading room feature-complete and deployed (cluster library 0.5.2); see the sub-project [roadmap](/demarkus-library/roadmap.md).**
- [Universe Deployment (Phase 6)](/plans/universe-deployment.md); Helm charts (server, broker, agent), OIDC token broker, release pipeline, observability. ~95% complete (PRs #126-#134, 2026-05-14). Remaining §6.6 (docs) + §6.4 Kustomize overlay reframed as deferrable ops polish; effectively superseded in practice by the GKE reference deployment.

## Completed Plans

- [OpenCode Memory Plugin (1:1 port)](/plans/opencode-memory-plugin.md); the OpenCode port of demarkus-memory as `plugins/opencode-memory/` v0.13.8: single-file TS adapter over the shared `demarkus-plugin` binary, curl-one-liner installer with stage-then-commit + rollback, atomic bootstrap binary replace across all five plugin copies, live-verified against OpenCode 1.18.15. **COMPLETE: planned 2026-08-09, merged 2026-08-10 (PR #281, `f4c2b35`).** Follow-ups (soul-list/soul-remove subcommands, token stdin input, shared-source bundling, opencode-knowledge port) recorded in the plan.
- [Graph Hub Seeding](/plans/graph-hub-seed.md); mark_backlinks/mark_graph/mark_explore seed from the published /graph.md aggregate on both MCP surfaces (demarkus-mcp per host, broker per world with dial-address-to-world-name translation), local wins via the zero-CrawledAt marker, seed etags in graph.json, fetch.FetchConditional. **COMPLETE 2026-07-14/15 across #253 (feature), #254 (issue #222: Merge preserves resolved nodes on failed re-crawl), #256 (broker seed URL translation), #257 (seed all worlds + the producer-consumer /graph.md contract test).** Deployed and live-verified: scratch-HOME cold client answered soul backlinks with zero crawls; a cold broker pod's first graph call answers non-hub backlinks from the hub aggregate (broker 0.12.4, agent 0.21.1). Lessons in [/debugging.md](/debugging.md) (mock fixtures encoded a plan assumption).
- [Multi-replica LOOKUP (postgres, phase 2)](/plans/lookup-replicas.md); the LOOKUP catalog moved into Postgres (rows in the write transaction, SQL-backed Lookup behind the handler `LookupCatalog` seam) so world pods can scale past one replica; phase 2 of the deploy repo's ADR 0002, following the phase-1 [postgres backend](/plans/postgres-backend.md) (#249). **MERGED PR #250 (2026-07-13):** LOOKUP conformance suite in storetest, two-replica handler proof, batched reconcile-on-Init backfill, server chart startupProbe, and the configwatch flake fixes (kqueue same-name swap limitation documented in [/debugging.md](/debugging.md)).
- [Version Retention](/plans/version-retention.md); keep last N versions per document via a `retention` publish-metadata key with prune-on-write in the store; motivated by the knowledge system's graph document at 545+ versions. **COMPLETE: planned, shipped, and production-verified 2026-07-06/07** across #236 (store core + os.Root delete hardening + audit logging + SPEC §9.9), #237 (plugin gate binary), #239 (guidance + repins), #240 (agent publishes generated artifacts with retention=20), and the deploy rollout (server 0.20.0 / broker 0.9.0 / agent 0.19.0). Live result: /graph.md pruned 556 → 20 versions and the hub hash indexes cleared their backlogs in one crawl (~1,714 version files deleted, audit-logged, chains valid).
- [MCP Resources + Prompts](/plans/mcp-resources-prompts.md); demarkus documents as client-attachable MCP resources (mark:// URI template, `#anchor` section attach, background-LIST picker population) and orient/recall/whats-new as server-vended prompt commands. **SHIPPED PR #232 (2026-07-05), client/v0.17.0.** Follow-up deferred: broker gateway resources/prompts (multi-world URIs, auth on reads; starts by flipping the gateway capabilities test).
- [MCP Client Ergonomics](/plans/mcp-client-ergonomics.md); size-adaptive `mark_fetch` (outline mode, `#section` slicing, `force`), session unchanged-dedup, and the `mark_explore` orientation card, on both MCP surfaces via shared `client/mdoutline` + `client/fetchdedup` packages. **SHIPPED #225/#230 and deployed 2026-07-04/05; plugin users (client v0.15.0 via tools 0.4.1) and the live knowledge system (broker 0.5.0).** Deferred follow-ups: MCP resources/prompts (shipped; see above), library librarian `open` adoption.
- [Versions Sharding](/plans/versions-sharding.md); per-document `versions/<doc>/vN` subdirectories with lazy migration, fixing the O(all-entries) `findVersions` scan. **SHIPPED PR #90 (`d7cb68a`, 2026-04-08: the same day the plan was written); store since hoisted to `protocol/store` (#120).** This index wrongly listed it as unstarted until 2026-07-05.
- [Plugin Knowledge-Quality Enforcement](/plans/plugin-knowledge-quality.md); raised the demarkus-memory Claude Code plugin from advisory to *enforced*. **SHIPPED v0.4.0, PR #171 merged 2026-06-01.** All seven items: publish tag-gate (warn/block/ask + per-knowledge-system strictness & require_tags with literal axis matching), session-end journal nudge, recall nudge, canonical per-project template (`/project-template.md`), knowledge-system policy/template at the live `root` hub on `knowledge.demarkus.io`, and the `/soul-doctor` hygiene audit. 68 tests, pure awk/bash, zero runtime deps. Tail (separate): plugin shell tests → CI; optional nudge disable knobs.
- [Broker Authorization Code Grant](/plans/broker-auth-code-grant.md); RFC 6749 `authorization_code` + PKCE (S256) on the broker so Claude Code's MCP SDK can auth against `broker.knowledge.demarkus.io`. **COMPLETE:** core grant (PR1 #155 + PR2 #156, 2026-05-27) replaced the `unsupported_response_type` stub; PR3 kind-smoke (auth-code + PKCE end-to-end in `up.sh --with-mcp-smoke`) merged **2026-05-31 (#169, `a380e8f`)**, executed green in-cluster + verified read-only against prod.
- [LOOKUP verb](/plans/lookup-verb.md): the card-catalog verb (subject → docs + importance). Shipped to main **PR #166 (2026-05-30)**; plugin surfacing in v0.3.0 (#168). Tail: `mark_append` metadata deferred by design.
- [Knowledge System (GKE Reference Deployment](/plans/knowledge-system-gke-deploy.md)) public GitHub-template deploy repo (`latebit-io/demarkus-knowledge-system-deploy`) standing up `knowledge.demarkus.io` on GKE (OpenTofu + ArgoCD + OpenBao + bank-vaults + CSI-snapshot backups). **Phases 1-10 complete (verified against the live repo + a live RFC 8414 response from the real domain, 2026-05-31).** Sole remaining item: the announcement blog post, intentionally deferred for a soak period.
- [Universe Onboarding](/plans/universe-onboarding.md); last-mile join flow. **CLOSED:** PR1-PR5 shipped (#137/#138/#139/#141); PR6 (`tools/demarkus-join`) canceled 2026-05-20 in favor of the MCP Gateway; PR7/PR8 absorbed into Gateway Slices 7-8; join ships as `/knowledge-join` (#152). Remaining: low-priority doc debt only (two standalone deployment docs).
- [Broker Stable Mint](/plans/broker-stable-mint.md); lazy per-world token provisioning + cache-stable 401 retries that killed the ~20-token mint cascade; dead `DefaultToken` knobs removed. **COMPLETE** (#158/#159/#163/#164/#165, 2026-05-27→29).
- [Broker Deadcode Cleanup](/plans/broker-deadcode-cleanup.md); deleted the issuance subsystem made unreachable by the open-knowledge-system rework (sessionCache, `/tokens` API, `issuer.go`, sweeper trim). **COMPLETE** (#159 + #164, commit `f9a24e9`).
- [Universe Onboarding (PR5 (broker /me/install)](/plans/universe-onboarding-pr5.md)) sub-plan, shipped #141 2026-05-20. Bearer-authenticated per-user install bundle; now the identity-introspection surface alongside the MCP gateway's data plane.
- [History](/completed-plans.md): content addressing, federation, persistent graph, read auth (server-side), conflict-aware merge in `mark_publish` (2026-05-05), Claude Code plugin (2026-04-23), **Broker MCP Gateway (2026-05-23; all 8 slices + Pre-Flight 0/1 shipped; 13-tool surface with byte-for-byte proxy fidelity to local demarkus-mcp, OIDC + RFC 9728/8414 metadata, chart + kind smoke + `/knowledge-join` slash command)** + **RFC 7591 DCR follow-on (2026-05-26, PR #153; `/register` + `registration_endpoint` in discovery, unblocks Claude Code → cluster broker auth via the native MCP authorization spec).** Also: OKF `type` adoption + `/soul-join` managed remote souls (2026-06).

## Plan Archives

Original plan documents preserved for reference:
- [Content Addressing](/plans/content-addressing.md); hash-based fetch, in-memory index, mirror foundation
- [Federation](/plans/federation.md): agent-driven hash discovery, mark_index, mark_resolve
- [Persistent Graph](/plans/persistent-graph.md); disk-backed graph store, incremental crawl, backlinks
- [Information Graph](/plans/information-graph.md); **superseded** early draft of Persistent Graph (Phase 4, 2026-03-08); see [persistent-graph.md](/plans/persistent-graph.md) for the version that shipped.
- [Read Auth](/plans/read-auth.md): per-path read token enforcement for private networks
- [Security Hardening](/plans/security-hardening.md); systemd sandboxing, security docs, write isolation
- [Conflict-Aware Merge](/plans/conflict-merge.md); tool-level diff3 merge in `mark_publish` (shipped client/v0.12.25 + v0.12.26)
- [Claude Code Plugin](/plans/claude-code-plugin.md); one-click marketplace plugin (shipped demarkus-memory v0.1.1; v0.2.0 added `/knowledge-join` 2026-05-23; v0.3.0 added self-documenting guidance + lookup recall 2026-05-31; v0.4.0 enforcement + template + /soul-doctor shipped 2026-06-01, PR #171; v0.5.0 split out demarkus-knowledge 2026-06-03, PR #172)
- [Universe Onboarding (PR3 (broker device flow)](/plans/universe-onboarding-pr3.md)) shipped 2026-05-15 (#137). RFC 8628 device flow end-to-end on the broker. Six sub-steps merged across one PR; PR4 builds on top.
- [Universe Onboarding (PR4 (broker refresh tokens)](/plans/universe-onboarding-pr4.md)) shipped 2026-05-15 (#138 + #139). Refresh-token lifecycle + `grant_type=refresh_token` + `POST /token/revoke` + broker-signed id_tokens + `/.well-known/jwks.json` + compositeVerifier + Sweeper integration. Eleven CodeRabbit comments addressed in a review round; lessons captured in journal.
- [Broker MCP Gateway](/plans/broker-https-gateway.md); **shipped 2026-05-23 (v7).** Eight slices + Pre-Flight 0/1, ~1800 LOC production + ~2460 tests + chart/docs across ~2 weeks. Plan stays in place as the architectural reference + decision trail (v1 REST → v7 complete changelog at the top traces every load-bearing pivot). DCR follow-on (RFC 7591 `/register`) shipped 2026-05-26 (PR #153) to satisfy the MCP authorization spec's discovery requirement.
- [Search Verb](/plans/search-verb.md): **superseded** by LOOKUP. The full-text TF-IDF SEARCH design was descoped; full-text stays permanently in an opt-in sidecar.
- [POC Deployment](/plans/poc-deployment.md); **canceled.** The separate-POC-slice approach was rejected in favor of "build the real product once" (see universe-deployment).
- [Obsidian Plugin](/plans/obsidian-plugin.md); **obsolete.** Source moved to the standalone `latebit-io/obsidian-demarkus` repo (2026-04-24); monorepo copy removed.