# Demarkus / Knowledge System FAQ

For the RFC review session. Sources: demarkus repo, demarkus-knowledge-system-deploy repo. Status: WIP.

## How do you interact with the knowledge system?

Two access paths: MCP over HTTPS for agents, web or CLI for humans.

- Agents: `mark_*` MCP tools against the broker over HTTPS, URLs as `mark://<world>/<path>`.
- Auth: MCP OAuth device flow at the broker.
- Humans: the library web app (OAuth web client of the broker), or `demarkus` CLI / `demarkus-tui` direct to a world over QUIC.
- Navigation anchors on the `root` hub; policy and templates under `mark://root/.well-known/demarkus/`.

## What are a world, a hub, root, and an index?

- **World**: one demarkus server in a knowledge system, addressed by logical name. Own store, tokens, lifecycle; describes itself in `world.md`.
- **Hub**: a server aggregating cross-server state: content-hash indexes from `mark_index` and the `/graph.md` export. `mark_resolve` and backlink seeding read from it.
- **Root**: the guaranteed world every system has, acting as its hub. Holds the global entry point and org conventions under `/.well-known/demarkus/`.
- **Index**: `index.md` is a curated entry-point document, the discovery backstop for what lookup can't surface. The crawler separately publishes machine hash indexes (`/index/<host>.md`) to the hub.

## What automated cues push agents to record memory?

Five hooks in the memory plugin.

- SessionStart: injects the routing table (decisions to `/adr/`, gotchas to `debugging.md`, progress to `journal/`).
- Stop: journal nudge when files changed but nothing was written to the soul.
- PostToolUse on `mark_publish`: promote nudge when a new ADR lands.
- UserPromptSubmit: recall-first reminder on "did we decide" questions.
- Pre/PostToolUse tag-gate: enforces `tags` and `importance` on every write.

## How does content get into the knowledge system?

Through the promote bridge: `/promote <soul-path>` runs the `knowledge-promote` cascade.

- Triage → distill for a shared audience (strip personal framing, secrets, PII).
- Dedup against the catalog → tag to the system taxonomy → route to a writable world.
- Human gate → `mark_publish` with provenance → back-stamp the soul source.
- Direct `mark_publish` to a joined system also works; it passes the same tag-gate and policy.

## How does finding knowledge work?

Primary: `mark_lookup` catalog queries. Secondary: graph traversal and hub pages.

- Lookup matches a query against declared tags and titles; returns an importance-ranked table (path, importance, title, tags).
- Filters: `tag=`, `modified-after=`. Not full-text search; untagged documents are invisible to it.
- From a hit: `mark_explore` to orient, `mark_fetch url#anchor` for the sections needed.
- Backstops: `index.md` hubs, `mark_backlinks` / `mark_graph` for link traversal, `mark_discover` for a server's manifest.

## How is relevance decided?

Deterministic scoring, no embeddings (`server/internal/catalog/catalog.go`).

- Score = count of distinct query terms matching tags (exact, case-insensitive) or title (substring, case-insensitive).
- Sort: score desc, then declared `importance`, then modification time, then path.
- Filters apply before ranking.

## How is importance stamped on a document?

The publisher declares it.

- `metadata.importance` on `mark_publish`, float in [0,1], stored out of band, indexed into the catalog.
- Absent, unparseable, or out-of-range values default to 0.5.
- `mark_append` carries no metadata; the value holds until the next `mark_publish`.

## How does the agent decide importance?

Judgment steered by injected guidance, not computed.

- SessionStart context and the `soul-memory` / `knowledge-promote` skills instruct: reserve 0.8+ for hubs, architecture, key decisions; routine notes lower.
- The server never infers it; the gate only validates the range.

## How does the document graph work?

`mark_graph` crawls outbound `mark://` links (depth default 2, max 5).

- Edges carry provenance: link label, source section anchor, occurrence count.
- Typed relations come from `rel-<predicate>` publisher metadata, e.g. `rel-supersedes`.
- Crawls persist to a graph store that answers `mark_backlinks`; seeded from a published `/graph.md`, local crawls take precedence.
- `mark_graph_publish` republishes the store as a crawlable `/graph.md` (generated doc, default retention 20).

## How is the graph built world to world?

A scheduled agent builds it; the broker reads what that agent publishes.

- The agent is configured with a seed URL per content world and crawls each one.
- It merges every world's edges into one graph and publishes it as `/graph.md` on the hub, alongside the content-hash index.
- Cross-world links are ordinary `mark://{worldName}/{path}` links, so an edge from one world to another is just a link the crawl followed.
- The broker's own graph store is in-memory and pod-scoped. It loads the hub's `/graph.md` on demand, so a fresh pod answers backlinks without crawling.
- A world with no published graph falls back to on-demand `mark_graph` crawls only.

## Is edge information there to guide agents to relevant information?

Yes (ADR 0004).

- Label: what the linker calls the target.
- Anchor: the exact source section, a direct `mark_fetch url#anchor` jump.
- Count: link strength signal.
- `rel-` types: "what superseded this" rather than bare "mentions".
- Principle: the agent owns judgment, the server owns accumulation.

## What does the agent use to decide whether to read a document?

Catalog and graph evidence before bodies.

- The `mark_lookup` row: path, importance, title, tags, modified time.
- Backlink provenance: label, anchor, count, `rel-` type.
- `mark_explore`: outline, links, backlinks, siblings in one call.
- Staged reading: `mark_fetch` returns an outline above 8KB, so the agent targets a `#section` anchor.
- `mark_discover` manifests and hub `index.md` pages set context first.

## Why deploy a knowledge system on k8s?

A knowledge system is distributed: many independent servers behind one broker.

- Each world is its own server with its own store and lifecycle.
- Worlds run and fail independently; one world going down does not take the catalog with it.
- Worlds are added or upgraded without touching the others.
- The whole universe is declared in one place and converges to that declaration.
- Smaller needs: the reference deployment itself recommends one or two plain deploys at a fraction of the cost.

## How does the system keep data fresh?

- Reads dispatch to the world's server; repeat fetches of an unchanged document return an "unchanged" notice.
- Every write creates a new hash-chained version.
- Optimistic concurrency (`expected_version` plus merge-on-conflict) stops stale writes clobbering newer ones.
- A scheduled agent re-crawls every world and republishes the content-hash index and graph on the hub.
- Soul copies of promoted documents can go stale. Refresh runs one way only: `/soul-refresh` copies the knowledge system's version down to the soul, and `/promote` is the only way a soul edit goes back up. There is no two-way sync.

## How do diffs and merges work? When does the agent merge documents?

Two layers.

- Same document, concurrent edits: a conflicting `mark_publish` returns a diff3 merge candidate (base, ours, theirs, git-style markers). The machine merges structure; the agent reviews semantics and republishes.
- Across documents: the promote cascade's dedup step looks up the subject, fetches close matches, and prefers a gated update to an existing doc over a near-duplicate. Conflicts go to the human gate, never a silent overwrite.
- Backstop: `/knowledge-doctor` sweeps find content-hash duplicates across paths and worlds.

## Why does federation matter, and why a server per world or memory?

No central authority or registry.

- Anyone can run a server; content mirrors freely.
- The hash chain lets agents on different mirrors verify they hold identical versions.
- A server per world makes the ownership boundary physical: own store, token file, writer allowlist, lifecycle. Teams evolve independently; blast radius stays contained.
- A soul is the same server at personal scale.
- Cross-server discovery sits on top: the crawler indexes content hashes into `root`; `mark_resolve` fetches by hash across servers.

## What tools does the memory plugin expose?

Fifteen `mark_*` MCP tools.

| Tool | Description |
|------|-------------|
| `mark_fetch` | Fetch a document or `#section`; bodies over 8KB return an outline |
| `mark_list` | List documents and subdirectories; archived hidden by default |
| `mark_explore` | One doc's outline, outbound links, backlinks, and siblings in one call |
| `mark_lookup` | Catalog lookup: importance-ranked matches on tags and titles |
| `mark_publish` | Create or update a document; metadata, optimistic concurrency, diff3 on conflict |
| `mark_append` | Append to an existing document; no metadata, auto-resolves version |
| `mark_archive` | Archive a document; hidden from listings, history preserved |
| `mark_versions` | Version history with hash-chain validation |
| `mark_graph` | Crawl outbound `mark://` links; persists edges to the graph store |
| `mark_backlinks` | What links here, with edge provenance |
| `mark_graph_export` | Export the graph store as publishable markdown |
| `mark_graph_publish` | Export and publish the graph as `/graph.md` (retention default 20) |
| `mark_discover` | Fetch a server's agent manifest |
| `mark_resolve` | Resolve content by SHA-256 hash via a hub index |
| `mark_index` | Crawl a server, publish its content-hash index to a hub |

## What tools does the knowledge plugin expose?

The same 15 MCP tools as the memory plugin (deliberate parity, one vocabulary regardless
of transport), with URLs addressing worlds by logical name instead of host:port. On top
of those:

| Addition | Kind | Description |
|----------|------|-------------|
| `mark_worlds` | MCP tool | Enumerate the system's worlds with each `world.md` descriptor and writability; broker-only, since the local MCP's universe is its one world |
| `/knowledge-join` | command | Validate an org broker URL, register it as an MCP server, wire OAuth device flow, mirror its policy into the local gates |
| `/knowledge` | command | List joined systems and show each `root` hub index |
| `/knowledge-doctor` | command | Read-only hygiene audit: orphans, broken links, untagged and policy-noncompliant docs, ADR gaps, duplicates |
| `knowledge-promote` | skill | The curation cascade that lands a staged document in the catalog; invoked by the memory plugin's `/promote` |

Writes pass a per-world writer allowlist with one shared per-world token; SSO is the org gate.

## How is knowledge formatted and held to a standard?

The standard is itself published on `root` under `.well-known/demarkus/`.

- `policy.md`: strictness, required tag axes such as `category:`, optional required OKF fields.
- `template.md`: per-world layout.
- `style.md`: H1 as name, one-sentence summary under it, unique headings (headings are anchors), no em dashes, no frontmatter fences.
- Joined agents mirror the policy into local write-time gates: tags, axes, importance range, mechanically checkable style rules, at the declared severity (warn / block / ask).
- `/knowledge-doctor` audits the corpus after the fact.

## Why QUIC?

- Encryption is mandatory: TLS 1.3 built in, no plaintext fallback.
- Fast multiplexed streams, one per request.
- No HTTP layer: no cookies, tracking headers, or query strings. Seven text verbs instead.
- The broker fronts worlds over HTTPS, so clients need no direct QUIC access.

## What is the auth and security model?

Capability tokens on the server; SSO at the broker.

- Writes are denied unless a token store is configured.
- Reads are public unless a token grants `read` on a path pattern, which makes matching paths require one (protect `/**` for a private intranet).
- Knowledge system: OIDC SSO is the org gate. Reads open to any authenticated identity; writes pass a per-world writer allowlist, dispatched with one shared long-lived per-world token.
- Policy forbids publishing secrets, credentials, or PII.

## How are versions and integrity handled?

- Every write creates an immutable new version linked by a SHA-256 hash chain.
- `mark_versions` validates the chain; mirrors verify identical content by version and hash.
- Nothing is deleted by default: `mark_archive` hides a document and keeps history.
- The one destructive path is `metadata.retention`, which permanently prunes old versions. It exists for generated documents; clients warn before applying it elsewhere.

## Why no full-text or semantic search?

Deliberate (SPEC §6.7).

- LOOKUP is a per-world catalog over author-declared tags and importance. It never reads bodies; there is no centralized index.
- Full-text and semantic search stay out of core as an opt-in sidecar reading demarkus over LIST/FETCH.
- Keeps the server simple and deterministic; ranking judgment stays in the agent.

## What are the known limitations?

- The graph is not yet optimal: the broker's store is in-memory and pod-scoped, crawls are on-demand, edge semantics are recent (ADR 0004).
- Full-text or semantic search requires an added component; until then recall depends on tagging discipline.
- Behavior at large scale is unproven: catalog size, crawl cost, and cross-world graph growth have not been tested at enterprise corpus sizes.