# Plan: LOOKUP verb (topic/importance catalog)

Status: draft, pending Fritz review (opened 2026-05-30). Tracks [issue #113](https://github.com/latebit-io/demarkus/issues/113) (filed as "SEARCH"; the verb is named `LOOKUP` — see Naming).

(Supersedes the earlier `/plans/search-verb.md`, which is archived. That doc's history holds the full-text TF-IDF design we descoped.)

## What it is

`LOOKUP` is the **card catalog** for a demarkus world. Given a subject, it returns which documents are about it and how important they are, so the agent can go FETCH the right one without fetching and parsing the whole index hub and guessing. It is a token-efficient **supplement to the index**, not a search engine.

This fits the project's core metaphor exactly. The architecture's first principle is "the agent as the librarian"; LOOKUP is the catalog the librarian consults — look up a subject, get pointed at the shelf, FETCH the book. It is deliberately dumb: match a subject against author-declared catalog fields, order by importance. No relevance modeling, no body reading.

## Naming

Named `LOOKUP`, not `SEARCH`, on purpose. "Search" sets a full-text, relevance-engine expectation this does not meet, and its worst failure (an obvious body match that wasn't tagged returns nothing) reads as broken. "Look up" carries the catalog contract honestly: a catalog only knows what was filed in it, so a miss on an uncatalogued subject is expected behavior, not a bug. LOOKUP also keeps the verb cadence of FETCH / LIST / VERSIONS / PUBLISH / ARCHIVE / APPEND.

**Settled consequence:** full-text / semantic matching never enters core. It lives permanently in an opt-in **sidecar** (separate binary that reads demarkus over LIST/FETCH and builds its own index). A LOOKUP that did full-text would read wrong, and adding a separate SEARCH verb later would break "the verb set is final." Choosing LOOKUP closes that door on purpose. This finalizes the core verb set at 7.

## The verb

`LOOKUP` is a scoped, importance-ranked lookup over a server-maintained **catalog** of declared document metadata. It searches current versions only, excludes archived docs, skips `versions/`. Path scoping works like LIST: `LOOKUP /docs/` scopes to a subtree, `LOOKUP /` covers everything.

**Two axes — keep them separate:**

- `query` (the subject) matches **`tags` + title** by term overlap. This is the "what is it about" axis: topical labels and the human title.
- `filter` matches **any frontmatter `key=value`**, exact. This is the "constrain by attribute" axis: `project`, `type`, `author`, dates, status, etc.

Conflating them was rejected: matching the subject query against all frontmatter values turns bookkeeping attributes (`type: journal`, `author: fritz`) into accidental subject hits and makes results impossible to reason about. Attributes belong on the filter axis. `tags` can serve double duty — fuzzy subject match via `query`, exact membership via `filter: tag=go`.

**Request** (subject + options in metadata, scope in the request line):
```
LOOKUP /docs/
---
query: auth middleware                          # matched against tags + title
filter: project=broker,modified-after=2025-01-01  # optional, exact key=value over any frontmatter
limit: 10
auth: <token>                                   # optional, gates read-auth-protected paths
---
```

- `query` (REQUIRED): subject text, lowercased and split into terms. A doc matches if its tags or title share at least one term. Min 2 chars, max ~256 bytes.
- `filter` (OPTIONAL): comma-separated `key=value` over declared frontmatter, plus built-ins `modified-after` / `modified-before`. Applied before ranking.
- `limit` (OPTIONAL): default 10, hard cap 1000 (matches LIST).
- `auth` (OPTIONAL): token for read-auth-gated paths.

`query`, `filter`, `limit` are control keys (never stored). `matches` is a reserved response key.

**Response** — markdown table (consistent with LIST/VERSIONS, parseable by the `index` table parser). **No body snippets** — bodies cost tokens and defeat the purpose; the title is the hint, the agent FETCHes the chosen doc:
```
---
status: ok
matches: 2
---
# Lookup matches for "auth middleware" in /docs/

| URI | Importance | Title | Tags |
|-----|------------|-------|------|
| mark://soul.demarkus.io/docs/auth-middleware | 0.90 | Auth middleware design | go, auth, middleware |
| mark://soul.demarkus.io/docs/gateway         | 0.50 | Gateway overview       | go |
```

## Declared metadata (set on PUBLISH)

Catalog fields ride the existing PUBLISH path as publisher metadata, so they respect version immutability (change a tag → new version). The server interprets exactly **two** keys; everything else stays opaque and is reachable only through `filter`:

- `tags`: comma-joined string, e.g. `tags: go,auth,middleware`. The subject match target. Multi-valued via comma split + membership. **Interpreted.**
- `importance`: float in [0,1], the ranking weight. Default 0.5 when absent or invalid. The first interpreted key. **Interpreted.**
- `title` (optional): one-line human title shown in results, also part of the subject match. Fallback chain: declared `title` → first `# H1` of the body (extracted cheaply during the startup walk, which already reads the body) → path basename. Not "interpreted" beyond display/match; just read.

A `/guidelines.md` note should ask the authoring agent to set `tags` + `importance` on PUBLISH for soul docs; catalog quality depends on it, and since the agent authors the docs it is enforceable.

## Ranking

No TF-IDF. Deterministic sort by:
1. number of distinct query terms found in tags or title (desc) — tag matches may weight slightly above title matches (refinement, optional in v1),
2. `importance` (desc),
3. modified time (desc),
4. path (asc), stable tiebreaker.

Importance is a **prior, not an override**: a doc only ranks if it matched the subject at all, so importance orders genuine matches and never floats an unrelated doc onto the list. At soul scale (hundreds of docs) this is a linear scan over the in-memory catalog — sub-millisecond, no inverted index.

## Catalog design

A new `server/internal/catalog` package, owned by the server, updated by the handler after successful store writes. The store stays pure storage; the catalog is a server feature.

- In memory: `map[path]entry`, entry = `{tags []string, importance float, title string, modified time}`. One small entry per doc — no token map, no postings.
- **Built at startup** by walking the store, piggybacking the same walk the hash index already does (one pass reads each current body once, feeds the hash index and the catalog, and grabs the first H1 for the title fallback).
- **Maintained inline on writes** — the handler updates the catalog immediately after a successful PUBLISH (add/replace), APPEND (re-read tags/title), ARCHIVE (remove), unarchive (re-add). All four are the consistency surface; missing one drifts the catalog.
- Content directory stays the sole source of truth; the catalog is derived and fully rebuildable from a walk. No persistence — restart re-walks (mirrors the hash index; gob only if startup cost ever proves painful on an Orange Pi).
- Guarded by `sync.RWMutex` (single writer, concurrent readers).

### Out-of-band writes (known edge)

The inline model assumes writes flow through the handler. The hardened **read-only / `demarkus-publish`** deployment writes straight to the on-disk store, bypassing the handler, so the running server's catalog (and the existing hash index) won't see those writes until a rebuild. Pre-existing limitation of the in-memory-index pattern. v1 accepts staleness; clean fix is to hang a rebuild off the existing `configwatch`/SIGHUP reload path. Noted, not solved in v1.

## Security: read-auth filtering

The catalog holds all docs including read-auth-protected ones, but **results are filtered per entry against the caller's token** using the same check as FETCH (`authorizeRead`). Unauthorized matches are dropped silently — no path, no title, no existence signal. Single most important correctness requirement; gets dedicated leakage tests.

## Module touch points

- `protocol/`: `VerbLookup = "LOOKUP"` + `IsValidVerb`; control keys (`query`, `filter`, `limit`), reserved response key `matches`; reserve interpreted `importance` and `tags`.
- `protocol/store/`: expose a current-version walk hook so the catalog builds from the same pass; cheap first-H1 extraction for title; validate `importance` as float in [0,1].
- `server/internal/catalog/`: build-from-walk, tag/importance/title extraction, match + rank, filter eval, RWMutex. Small (no tokenizer-as-inverted-index, no postings, no snippets-from-body).
- `server/internal/handler/`: `handleLookup` + dispatch; parse query/filter/limit; **read-auth filter results**; build markdown response; query validation. Update catalog after PUBLISH/APPEND/ARCHIVE/unarchive.
- `client/fetch/`: `Client.Lookup(host, scope, query, token, opts)`.
- `client/cmd/demarkus/`: `lookup` subcommand.
- `client/cmd/demarkus-mcp/`: `mark_lookup` tool + add to `markClient` interface.
- `docs/SPEC.md`: §6.7 LOOKUP (catalog semantics), metadata tables, `matches` key, verb count 6→7.
- Soul `/architecture.md` + `/roadmap.md`: verb set 6→7; retract the "SEARCH removed" note, framed as a catalog/card-catalog supplement to the index (full-text stays in the sidecar); `/guidelines.md` note to tag + rate importance on PUBLISH.

## Phasing (small, incremental, each tested + pre-commit)

1. **Protocol verb**: `VerbLookup` + `IsValidVerb` + key reservations. Tiny.
2. **catalog package**: build-from-walk, extract tags/importance/title, match (tags+title) + rank, filter. Table tests.
3. **Handler + wiring**: `handleLookup`, dispatch, query validation, markdown response, **read-auth result filtering** (+ leakage tests), catalog updates on PUBLISH/APPEND/ARCHIVE/unarchive.
4. **Client surfaces**: `fetch.Client.Lookup`, `demarkus lookup` CLI, `mark_lookup` MCP tool + interface. Keep CLI/MCP consistent.
5. **Spec + soul**: SPEC §6.7, architecture/roadmap/guidelines updates.

## Out of core (sidecar / client layer, settled)

- **Full-text / semantic matching** — never in core. Opt-in sidecar binary that reads demarkus over LIST/FETCH, builds its own (Bleve/vector) index, exposes its own surface. Core never depends on it.
- **Federated lookup across worlds/hubs** — client/agent layer, mirrors federation over FETCH.
- Match-centered snippets, positional indexing, query DSLs, phrase matching.

## Risks

- Catalog/store drift if any write path (esp. APPEND/ARCHIVE/unarchive) misses an update. Mitigated by build-from-walk being authoritative and rebuildable.
- Out-of-band writes (`demarkus-publish`, direct-to-store sync) skip inline updates; stale until restart/rebuild. Known edge, deferred.
- Read-auth leakage via results. Mitigated by per-result auth filtering + leakage tests.
- **Recall**: a subject neither tagged nor in any title returns nothing. This is the honest catalog contract, not a defect — mitigated by the authoring agent tagging reliably (guideline) and the index/links remaining the fallback. The sidecar is the escape hatch if misses prove common.
- Tagging discipline is the quality lever — if `tags`/`importance` go unset, the catalog degrades to title-only lookup. Acceptable (no worse than the index), but worth measuring.