# Architecture

## Core Principle: The Agent as the Librarian

The server is a bookshelf. It holds documents and hands them back when asked. That's it. The agent is the librarian. It knows where things are, builds indexes, discovers content across servers, routes around failures, and helps you find what you need.

This is a strict requirement, not a suggestion. The server and protocol must stay dumb and fast. All intelligence (federation, discovery, indexing, routing, caching strategies) lives in the client layer (MCP tools, agents, CLI). When considering where to put new functionality, the answer is almost always "in the agent/client, not the server."

Federation is the proof: zero server changes, two MCP tools, all intelligence at the edge. The server doesn't even know federation exists.

## Module Boundaries

- **protocol/**: Wire format types, parsing, serialization. No network code. The foundation everything else depends on.
- **server/**: QUIC server. Accepts Mark Protocol requests over QUIC streams, routes to handlers. Depends on protocol.
- **client/**: CLI, TUI (Bubble Tea), and MCP server. All three share the same protocol client. Depends on protocol.
- **tools/**: Dev utilities like `demarkus-token` for generating auth tokens.

## Key Design Decisions

### Frontmatter as map[string]string
Parsed frontmatter uses `map[string]string` rather than typed structs. This avoids YAML auto-typing issues (e.g., `yes` becoming `true`) and keeps the parsing layer simple. Type conversion happens at the handler level where context is known.

### Versioned Store with Symlinks
Documents are stored as versioned files with symlinks pointing to current:
```
content-root/
  doc.md -> versions/doc.md.v3
  versions/
    doc.md.v1
    doc.md.v2
    doc.md.v3
```
Each version is immutable once written. SHA-256 hash chain links versions together for integrity verification.

### Capability-Based Auth
Tokens grant capabilities, not identities. Server stores SHA-256 hashes in TOML, never raw tokens. Each token is scoped to path globs and specific operations. The server doesn't know or care who you are, only what you're allowed to do.

### Read Auth for Private Networks
The auth system supports both `"publish"` and `"read"` operations on tokens. Write auth is always enforced: if a tokens file is configured, all writes require a valid token. Read auth is opt-in and per-path: if any token has `"read"` in its operations with a path pattern matching the request, that path requires read auth. Paths without read tokens remain public.

Key implementation details:
- `readPaths []string` on `TokenStore` is pre-computed at load time from tokens with `"read"` operations, avoiding per-request iteration
- `RequiresReadAuth(path)` checks both the path as-is and with a trailing slash to prevent bypass via `/private` vs `/private/`
- `authorizeRead` handler helper gates FETCH, LIST, and VERSIONS; LOOKUP filters per result via the shared `checkReadAuth`
- `/.well-known/agent-manifest.md` is always public so agents can discover capabilities before authenticating
- Content-addressed fetch by hash resolves to a real path first, then checks auth on that path. Knowing a hash doesn't bypass access control
- Versioned paths (`/doc.md/v2`) check auth on the base path (`/doc.md`)
- Hot-reloadable on SIGHUP or when the tokens file changes on disk. Both paths route through the same atomic-swap reload (`loadTokenStore`); the file-change path uses a debounced parent-directory watch (`server/internal/configwatch`) so atomic-rename writes and symlink-retarget swaps both trigger a reload. The watcher is generic and platform-agnostic (works where SIGHUP is unavailable, e.g. Windows), and complements the signal path rather than replacing it.

### Path Traversal Defense
`filepath.Clean` + `..` check. Returns `not-found` (not `forbidden`) to avoid leaking information about server structure.

### Status as Text
Human-readable status strings (`ok`, `not-found`, `conflict`) instead of numeric codes. Fits the markdown-native philosophy: everything should be readable.

### Size Limits

All protocol-level size constants live in `protocol/request.go`:

| Limit | Constant | Value |
|---|---|---|
| Request line | `MaxRequestLineLength` | 4096 bytes |
| Request metadata | `MaxRequestFrontmatterLength` | 64 KiB |
| Document body | `MaxBodyLength` | 1 MiB |

`ParseRequest` enforces these at the parsing layer using `io.LimitReader` to prevent unbounded memory allocation. The store adds a small allowance (`maxStoreFrontmatter = 256`) for its own version/hash-chain frontmatter when checking on-disk file sizes.

### Markdown-Only Content Contract

Shipped 2026-07-23 (PR #273). SPEC 2.3 always said content is markdown; nothing enforced it, so a published PNG round-tripped byte-perfect and MCP outline mode injected raw binary as mojibake into agent context. Enforcement is split by layer, each refusing what it owns:

- **Handler (protocol surface):** PUBLISH/APPEND path must end in lowercase `.md` and the body must be valid UTF-8 (`store.ValidateDocumentContent`, sentinels `ErrInvalidPath`/`ErrInvalidContent`); violations return `bad-request`, no version created. Same one-choke pattern as `ValidateMeta`: called by the handler, backend-agnostic.
- **Store (storable-document invariant):** `store.ValidateBody` (UTF-8 only) at both backends' write choke (file store and pgstore), pinned by a `RejectsBinaryBody` storetest conformance case. The `.md` rule deliberately stays out of the store: the store is path-agnostic (versioning, hash chain, traversal safety operate on arbitrary paths) and its security tests rely on non-.md paths; a store-level `.md` gate would fire first and mask what those tests assert.
- **MCP render guard (client):** `mark_fetch`, `mark_explore`, and resource reads return a one-line notice for non-UTF-8 bodies instead of mojibake or a bogus outline; `force=true` still pulls raw bytes. Shared `binaryBody` predicate across all three render sites; render-time detection is deliberately decoupled from server storage internals.
- **Transport unchanged:** raw FETCH stays byte-faithful. Refusing to serve on-disk bytes would make any binary that reached disk permanently uninspectable.

Two invariants worth keeping in mind: valid UTF-8 concatenates to valid UTF-8, so APPEND validates only the fragment; and `.MD` (uppercase) is rejected, lowercase `.md` only, because case-insensitive filesystems would collide the two.

## Verb Set: Complete

The protocol verb set is finalized at 7 verbs. No more verbs planned.

| Verb | Purpose |
|---|---|
| FETCH | Read a document |
| LIST | List directory contents |
| VERSIONS | Get version history |
| PUBLISH | Create or update a document (new immutable version) |
| APPEND | Append content to an existing document (new immutable version) |
| ARCHIVE | Soft-delete a document |
| LOOKUP | Look up documents by subject against the server's catalog |

APPEND reuses the same auth (`publish` capability), versioning, hash chain, and optimistic concurrency infrastructure as PUBLISH. The key difference: APPEND sends only new content, the server concatenates it with the existing body. Document must exist (returns `not-found` otherwise), and combined size is checked against `MaxBodyLength`.

LOOKUP (added 2026-05-30, issue #113) is the catalog verb: given a subject it returns an importance-ranked markdown table of documents whose declared tags or title match; a token-efficient supplement to the index hub. It earns a place in the dumb-server model because it is a mechanical catalog scan, not a search engine: the server keeps an in-memory catalog derived from the store and answers "which documents are filed under this subject," the card-catalog analogue of the librarian metaphor. The earlier note here ("SEARCH was considered and removed; full-text search belongs in an external tool") still holds for *full-text / semantic* search, which lives permanently in an opt-in sidecar that reads demarkus over LIST/FETCH and builds its own index. LOOKUP is deliberately not that: it never reads document bodies at query time and does no relevance modeling beyond match-count plus author-declared importance. See the LOOKUP Catalog note below.

## Wire Format

**Request**: `VERB /path\n` followed by optional YAML frontmatter and body.
**Response**: YAML frontmatter (always includes status) + markdown body.

Protocol constants: port `6309`, ALPN `"mark"`, scheme `mark://`.

## What I've Internalized

The architecture rewards simplicity. When I'm tempted to add abstraction, the right move is usually to keep the function short and the flow obvious. Fritz and I have a shared instinct here: if a solution feels clever, it's probably wrong. The best code in this project reads like prose.

### Audit Logging

All write operations (PUBLISH, APPEND, ARCHIVE, UNARCHIVE) emit structured slog records tagged with `"audit", true`. Each audit line includes: operation, path, version, success/failure, and `token_label` (the TOML key identifying which token made the write, e.g., `"fritz-laptop"`).

Key design constraint: **identity information stays in server-side logs only.** Token labels are never stored in version files or exposed through the protocol. This preserves capability-based auth. Tokens grant what you can do, not who you are. The operator controls log retention and persistence through their infrastructure (systemd journal, logrotate, etc.).

Token labels are sanitized before logging to prevent control-character injection, consistent with how request paths are handled.

### Content-Addressed Fetch

FETCH responses include `content-hash: sha256-<64hex>`, the SHA-256 of the response body (after stripping store frontmatter). This is distinct from `etag`, which hashes the raw on-disk content including store frontmatter.

Clients can fetch by hash instead of path: `FETCH /sha256-<64hex>`. The server maintains an in-memory `map[string]string` (hash to path) protected by `sync.RWMutex`. The index is:
- Built on startup by walking symlinks (current versions only, skips archived)
- Updated incrementally on `Write()` and `Archive()`/unarchive
- Not persisted. The content directory is the source of truth

Key implementation details:
- `isHashPath()` in handler detects the path pattern (exactly 72 chars: `/` + `sha256-` + 64 hex)
- `handleFetchByHash()` resolves hash to path via the index, then serves normally through `serveDocument()`
- Hash detection runs before `parseVersionPath()` in the FETCH dispatch chain
- `computeContentHash()` hashes the stripped body, not `doc.Content` (which includes store frontmatter)

### Agent-Driven Hash Discovery

Servers don't crawl or discover content. Agents do. The librarian pattern in action:

1. Agent fetches documents from servers, collecting `content-hash` values
2. Agent builds hash to location mappings
3. Agent publishes the mapping as a document to a hub

When content goes missing, agents check hub indexes for the hash, find alternative servers, and fetch by hash directly. Servers stay simple (serve content, answer hash lookups). Agents own discovery, routing, and indexing. Hubs are just servers hosting index documents, no special protocol support needed.

This is documented in SPEC.md Section 12.1.

### LOOKUP Catalog

LOOKUP is answered from an in-memory catalog (`server/internal/catalog`), the same derived-state pattern as the content hash index:

- One entry per current document: declared `tags`, `importance` (float in [0,1], default 0.5), `title` (declared → first H1 → path basename), modified time, and the publisher metadata map (for filter predicates).
- Built at startup by walking the store (`store.WalkCurrent`, which shares its traversal with `BuildHashIndex`), and maintained inline on every write; PUBLISH, APPEND, ARCHIVE, and unarchive. Not persisted; the content directory is the source of truth.
- `query` matches declared tags + title (term overlap, case-insensitive); `filter` matches any frontmatter `key=value` exactly, plus `tag=` / `modified-after` / `modified-before` built-ins. Ranking: match count → importance → recency → path. Importance is a bounded prior, never an override; an unmatched document never surfaces.
- The server interprets exactly three publisher metadata keys for LOOKUP; `tags`, `importance`, `title`; everything else stays opaque and is reachable only through `filter`.
- Read-auth filtering happens per result in the handler, not the catalog: the catalog holds every document, but results are filtered against the caller's token so protected documents never leak (no path, title, or existence). The catalog stays auth-agnostic.
- Brokered worlds get LOOKUP too: the broker MCP gateway exposes `mark_lookup` (its 14th tool) and proxies it like the other reads.

**Publisher metadata (shipped 2026-05-30, PR #166):** `tags`/`importance` are settable at publish time; CLI `-meta key=value` (repeatable) and a `metadata` object on MCP + broker `mark_publish`. The agent identity (`agent`) is applied last so a caller cannot spoof it. Before this, LOOKUP ran title-only with uniform 0.5 importance; now author-declared tags and importance drive matching and ranking. `mark_append` metadata is deliberately deferred (you tag on publish, not append). See [/plans/lookup-verb.md](/plans/lookup-verb.md).


## Open Knowledge Format (OKF) Compatibility

Shipped 2026-06-22 (PR #195). demarkus aligns its document content model with Google's Open Knowledge Format v0.1. Scoped to two layers, deliberately:

- **Document level: compatible.** Recognized OKF fields (`type`, `title`, `description`, `resource`, `tags`, `timestamp`) serialize as *bare* store frontmatter under their OKF names; `tags` as a YAML flow list. Non-spec publisher keys keep the `meta.` prefix. Store-operational fields (`version`, `previous-hash`, `archived`) are reserved and enforced by name (a `reservedMetaKeys` denylist replaced the old "`meta.` prefix as the integrity boundary" mechanism; see ADR 0002). The in-memory metadata map stays bare-keyed, so only `buildVersionFile` (write) and `extractMetadata` (read) changed; catalog/handler/filter were untouched.
- **System level: superset.** Versioning, hash chain, QUIC, capability auth, and LOOKUP wrap an OKF-compatible document. Not a bundle server: frontmatter is stripped before serving and `versions/` is not a bundle tree. Bundle interop is the out-of-band `demarkus okf` codec (`client/internal/okf`): `validate` / `import` / `export`, verified byte-identical body round-trip.

**Default type on write.** The server assigns `type: Document` (`protocol.OKFDefaultType`) on PUBLISH and APPEND when none is declared (`index.md`/`log.md` exempt), so every served document is a typed OKF concept by construction. ADR 0003.

**Metadata caps (supersedes the `maxStoreFrontmatter = 256` figure in the Size Limits section above).** OKF producer-defined fields drove these up: `MaxMetaKeys` 10→50, `MaxMetaBytes` 512→1024, `maxStoreFrontmatter` 256→2048. `MaxMetaBytes` is the binding limit (sum of key+value lengths), not key count. Subtlety: `tags` is validated as a csv string (`"a,b"`) but stored as a longer YAML flow list (`[a, b]`), so the byte cap is enforced via `store.SerializedMetaSize` (counts the on-disk serialized length) in BOTH the store `validateMeta` and the handler size check; one source of truth, so the 2048 frontmatter budget is a provable on-disk bound (worst case ~1542 < 2048).

See SPEC §8.1/§9.4/§13/§14, ADR 0002 (metadata alignment), ADR 0003 (default type).


## Related documents

- [ADR 0002: OKF metadata alignment](/adr/0002-okf-metadata-alignment.md): decision behind the OKF frontmatter model described here
- [ADR 0003: OKF type default on publish](/adr/0003-okf-type-default-on-publish.md): decision behind the default type on write
- [Read auth plan](/plans/read-auth.md): plan that shipped read auth for private networks
- [Content addressing plan](/plans/content-addressing.md): plan that shipped content-addressed fetch and hash discovery