Architecture

Core Principle: The Agent as the Librarian

The server is a bookshelf. It holds documents and hands them back when asked. That's it. The agent is the librarian. It knows where things are, builds indexes, discovers content across servers, routes around failures, and helps you find what you need.

This is a strict requirement, not a suggestion. The server and protocol must stay dumb and fast. All intelligence (federation, discovery, indexing, routing, caching strategies) lives in the client layer (MCP tools, agents, CLI). When considering where to put new functionality, the answer is almost always "in the agent/client, not the server."

Federation is the proof: zero server changes, two MCP tools, all intelligence at the edge. The server doesn't even know federation exists.

Module Boundaries

• protocol/: Wire format types, parsing, serialization. No network code. The foundation everything else depends on.
• server/: QUIC server. Accepts Mark Protocol requests over QUIC streams, routes to handlers. Depends on protocol.
• client/: CLI, TUI (Bubble Tea), and MCP server. All three share the same protocol client. Depends on protocol.
• tools/: Dev utilities like demarkus-token for generating auth tokens.

Key Design Decisions

Frontmatter as map[string]string

Parsed frontmatter uses map[string]string rather than typed structs. This avoids YAML auto-typing issues (e.g., yes becoming true) and keeps the parsing layer simple. Type conversion happens at the handler level where context is known.

Versioned Store with Symlinks

Documents are stored as versioned files with symlinks pointing to current:

content-root/
  doc.md -> versions/doc.md.v3
  versions/
    doc.md.v1
    doc.md.v2
    doc.md.v3

Each version is immutable once written. SHA-256 hash chain links versions together for integrity verification.

Capability-Based Auth

Tokens grant capabilities, not identities. Server stores SHA-256 hashes in TOML, never raw tokens. Each token is scoped to path globs and specific operations. The server doesn't know or care who you are, only what you're allowed to do.

Read Auth for Private Networks

The auth system supports both "publish" and "read" operations on tokens. Write auth is always enforced: if a tokens file is configured, all writes require a valid token. Read auth is opt-in and per-path: if any token has "read" in its operations with a path pattern matching the request, that path requires read auth. Paths without read tokens remain public.

Key implementation details:

• readPaths []string on TokenStore is pre-computed at load time from tokens with "read" operations, avoiding per-request iteration
• RequiresReadAuth(path) checks both the path as-is and with a trailing slash to prevent bypass via /private vs /private/
• authorizeRead handler helper gates FETCH, LIST, and VERSIONS; LOOKUP filters per result via the shared checkReadAuth
• /.well-known/agent-manifest.md is always public so agents can discover capabilities before authenticating
• Content-addressed fetch by hash resolves to a real path first, then checks auth on that path. Knowing a hash doesn't bypass access control
• Versioned paths (/doc.md/v2) check auth on the base path (/doc.md)
• Hot-reloadable on SIGHUP or when the tokens file changes on disk. Both paths route through the same atomic-swap reload (loadTokenStore); the file-change path uses a debounced parent-directory watch (server/internal/configwatch) so atomic-rename writes and symlink-retarget swaps both trigger a reload. The watcher is generic and platform-agnostic (works where SIGHUP is unavailable, e.g. Windows), and complements the signal path rather than replacing it.

Path Traversal Defense

filepath.Clean + .. check. Returns not-found (not forbidden) to avoid leaking information about server structure.

Status as Text

Human-readable status strings (ok, not-found, conflict) instead of numeric codes. Fits the markdown-native philosophy: everything should be readable.

Size Limits

All protocol-level size constants live in protocol/request.go:

ParseRequest enforces these at the parsing layer using io.LimitReader to prevent unbounded memory allocation. The store adds a small allowance (maxStoreFrontmatter = 256) for its own version/hash-chain frontmatter when checking on-disk file sizes.

Markdown-Only Content Contract

Shipped 2026-07-23 (PR #273). SPEC 2.3 always said content is markdown; nothing enforced it, so a published PNG round-tripped byte-perfect and MCP outline mode injected raw binary as mojibake into agent context. Enforcement is split by layer, each refusing what it owns:

• Handler (protocol surface): PUBLISH/APPEND path must end in lowercase .md and the body must be valid UTF-8 (store.ValidateDocumentContent, sentinels ErrInvalidPath/ErrInvalidContent); violations return bad-request, no version created. Same one-choke pattern as ValidateMeta: called by the handler, backend-agnostic.
• Store (storable-document invariant): store.ValidateBody (UTF-8 only) at both backends' write choke (file store and pgstore), pinned by a RejectsBinaryBody storetest conformance case. The .md rule deliberately stays out of the store: the store is path-agnostic (versioning, hash chain, traversal safety operate on arbitrary paths) and its security tests rely on non-.md paths; a store-level .md gate would fire first and mask what those tests assert.
• MCP render guard (client): mark_fetch, mark_explore, and resource reads return a one-line notice for non-UTF-8 bodies instead of mojibake or a bogus outline; force=true still pulls raw bytes. Shared binaryBody predicate across all three render sites; render-time detection is deliberately decoupled from server storage internals.
• Transport unchanged: raw FETCH stays byte-faithful. Refusing to serve on-disk bytes would make any binary that reached disk permanently uninspectable.

Two invariants worth keeping in mind: valid UTF-8 concatenates to valid UTF-8, so APPEND validates only the fragment; and .MD (uppercase) is rejected, lowercase .md only, because case-insensitive filesystems would collide the two.

Verb Set: Complete

The protocol verb set is finalized at 7 verbs. No more verbs planned.

APPEND reuses the same auth (publish capability), versioning, hash chain, and optimistic concurrency infrastructure as PUBLISH. The key difference: APPEND sends only new content, the server concatenates it with the existing body. Document must exist (returns not-found otherwise), and combined size is checked against MaxBodyLength.

LOOKUP (added 2026-05-30, issue #113) is the catalog verb: given a subject it returns an importance-ranked markdown table of documents whose declared tags or title match; a token-efficient supplement to the index hub. It earns a place in the dumb-server model because it is a mechanical catalog scan, not a search engine: the server keeps an in-memory catalog derived from the store and answers "which documents are filed under this subject," the card-catalog analogue of the librarian metaphor. The earlier note here ("SEARCH was considered and removed; full-text search belongs in an external tool") still holds for full-text / semantic search, which lives permanently in an opt-in sidecar that reads demarkus over LIST/FETCH and builds its own index. LOOKUP is deliberately not that: it never reads document bodies at query time and does no relevance modeling beyond match-count plus author-declared importance. See the LOOKUP Catalog note below.

Wire Format

Request: VERB /path\n followed by optional YAML frontmatter and body. Response: YAML frontmatter (always includes status) + markdown body.

Protocol constants: port 6309, ALPN "mark", scheme mark://.

What I've Internalized

The architecture rewards simplicity. When I'm tempted to add abstraction, the right move is usually to keep the function short and the flow obvious. Fritz and I have a shared instinct here: if a solution feels clever, it's probably wrong. The best code in this project reads like prose.

Audit Logging

All write operations (PUBLISH, APPEND, ARCHIVE, UNARCHIVE) emit structured slog records tagged with "audit", true. Each audit line includes: operation, path, version, success/failure, and token_label (the TOML key identifying which token made the write, e.g., "fritz-laptop").

Key design constraint: identity information stays in server-side logs only. Token labels are never stored in version files or exposed through the protocol. This preserves capability-based auth. Tokens grant what you can do, not who you are. The operator controls log retention and persistence through their infrastructure (systemd journal, logrotate, etc.).

Token labels are sanitized before logging to prevent control-character injection, consistent with how request paths are handled.

Content-Addressed Fetch

FETCH responses include content-hash: sha256-<64hex>, the SHA-256 of the response body (after stripping store frontmatter). This is distinct from etag, which hashes the raw on-disk content including store frontmatter.

Clients can fetch by hash instead of path: FETCH /sha256-<64hex>. The server maintains an in-memory map[string]string (hash to path) protected by sync.RWMutex. The index is:

• Built on startup by walking symlinks (current versions only, skips archived)
• Updated incrementally on Write() and Archive()/unarchive
• Not persisted. The content directory is the source of truth

Key implementation details:

• isHashPath() in handler detects the path pattern (exactly 72 chars: / + sha256- + 64 hex)
• handleFetchByHash() resolves hash to path via the index, then serves normally through serveDocument()
• Hash detection runs before parseVersionPath() in the FETCH dispatch chain
• computeContentHash() hashes the stripped body, not doc.Content (which includes store frontmatter)

Agent-Driven Hash Discovery

Servers don't crawl or discover content. Agents do. The librarian pattern in action:

1. Agent fetches documents from servers, collecting content-hash values
2. Agent builds hash to location mappings
3. Agent publishes the mapping as a document to a hub

When content goes missing, agents check hub indexes for the hash, find alternative servers, and fetch by hash directly. Servers stay simple (serve content, answer hash lookups). Agents own discovery, routing, and indexing. Hubs are just servers hosting index documents, no special protocol support needed.

This is documented in SPEC.md Section 12.1.

LOOKUP Catalog

LOOKUP is answered from an in-memory catalog (server/internal/catalog), the same derived-state pattern as the content hash index:

• One entry per current document: declared tags, importance (float in [0,1], default 0.5), title (declared → first H1 → path basename), modified time, and the publisher metadata map (for filter predicates).
• Built at startup by walking the store (store.WalkCurrent, which shares its traversal with BuildHashIndex), and maintained inline on every write; PUBLISH, APPEND, ARCHIVE, and unarchive. Not persisted; the content directory is the source of truth.
• query matches declared tags + title (term overlap, case-insensitive); filter matches any frontmatter key=value exactly, plus tag= / modified-after / modified-before built-ins. Ranking: match count → importance → recency → path. Importance is a bounded prior, never an override; an unmatched document never surfaces.
• The server interprets exactly three publisher metadata keys for LOOKUP; tags, importance, title; everything else stays opaque and is reachable only through filter.
• Read-auth filtering happens per result in the handler, not the catalog: the catalog holds every document, but results are filtered against the caller's token so protected documents never leak (no path, title, or existence). The catalog stays auth-agnostic.
• Brokered worlds get LOOKUP too: the broker MCP gateway exposes mark_lookup (its 14th tool) and proxies it like the other reads.

Publisher metadata (shipped 2026-05-30, PR #166): tags/importance are settable at publish time; CLI -meta key=value (repeatable) and a metadata object on MCP + broker mark_publish. The agent identity (agent) is applied last so a caller cannot spoof it. Before this, LOOKUP ran title-only with uniform 0.5 importance; now author-declared tags and importance drive matching and ranking. mark_append metadata is deliberately deferred (you tag on publish, not append). See /plans/lookup-verb.md.

Open Knowledge Format (OKF) Compatibility

Shipped 2026-06-22 (PR #195). demarkus aligns its document content model with Google's Open Knowledge Format v0.1. Scoped to two layers, deliberately:

• Document level: compatible. Recognized OKF fields (type, title, description, resource, tags, timestamp) serialize as bare store frontmatter under their OKF names; tags as a YAML flow list. Non-spec publisher keys keep the meta. prefix. Store-operational fields (version, previous-hash, archived) are reserved and enforced by name (a reservedMetaKeys denylist replaced the old "meta. prefix as the integrity boundary" mechanism; see ADR 0002). The in-memory metadata map stays bare-keyed, so only buildVersionFile (write) and extractMetadata (read) changed; catalog/handler/filter were untouched.
• System level: superset. Versioning, hash chain, QUIC, capability auth, and LOOKUP wrap an OKF-compatible document. Not a bundle server: frontmatter is stripped before serving and versions/ is not a bundle tree. Bundle interop is the out-of-band demarkus okf codec (client/internal/okf): validate / import / export, verified byte-identical body round-trip.

Default type on write. The server assigns type: Document (protocol.OKFDefaultType) on PUBLISH and APPEND when none is declared (index.md/log.md exempt), so every served document is a typed OKF concept by construction. ADR 0003.

Metadata caps (supersedes the maxStoreFrontmatter = 256 figure in the Size Limits section above). OKF producer-defined fields drove these up: MaxMetaKeys 10→50, MaxMetaBytes 512→1024, maxStoreFrontmatter 256→2048. MaxMetaBytes is the binding limit (sum of key+value lengths), not key count. Subtlety: tags is validated as a csv string ("a,b") but stored as a longer YAML flow list ([a, b]), so the byte cap is enforced via store.SerializedMetaSize (counts the on-disk serialized length) in BOTH the store validateMeta and the handler size check; one source of truth, so the 2048 frontmatter budget is a provable on-disk bound (worst case ~1542 < 2048).

See SPEC §8.1/§9.4/§13/§14, ADR 0002 (metadata alignment), ADR 0003 (default type).