# Plan: LOOKUP verb (topic/importance catalog) Status: draft, pending Fritz review (opened 2026-05-30). SPEC §6.7 drafted 2026-05-30 (the spec part of Slice 5 is done; soul + guidelines updates and the code slices remain). Slice 1 implemented on branch `feat/lookup-verb` (PR #166). Tracks [issue #113](https://github.com/latebit-io/demarkus/issues/113) (filed as "SEARCH"; the verb is named `LOOKUP` — see Naming). (Supersedes the earlier `/plans/search-verb.md`, which is archived. That doc's history holds the full-text TF-IDF design we descoped.) ## What it is `LOOKUP` is the **card catalog** for a demarkus world. Given a subject, it returns which documents are about it and how important they are, so the agent can go FETCH the right one without fetching and parsing the whole index hub and guessing. It is a token-efficient **supplement to the index**, not a search engine. This fits the project's core metaphor exactly. The architecture's first principle is "the agent as the librarian"; LOOKUP is the catalog the librarian consults — look up a subject, get pointed at the shelf, FETCH the book. It is deliberately dumb: match a subject against author-declared catalog fields, order by importance. No relevance modeling, no body reading. ## Naming Named `LOOKUP`, not `SEARCH`, on purpose. "Search" sets a full-text, relevance-engine expectation this does not meet, and its worst failure (an obvious body match that wasn't tagged returns nothing) reads as broken. "Look up" carries the catalog contract honestly: a catalog only knows what was filed in it, so a miss on an uncatalogued subject is expected behavior, not a bug. LOOKUP also keeps the verb cadence of FETCH / LIST / VERSIONS / PUBLISH / ARCHIVE / APPEND. **Settled consequence:** full-text / semantic matching never enters core. It lives permanently in an opt-in **sidecar** (separate binary that reads demarkus over LIST/FETCH and builds its own index). A LOOKUP that did full-text would read wrong, and adding a separate SEARCH verb later would break "the verb set is final." Choosing LOOKUP closes that door on purpose. This finalizes the core verb set at 7. ## The verb `LOOKUP` is a scoped, importance-ranked lookup over a server-maintained **catalog** of declared document metadata. It searches current versions only, excludes archived docs, skips `versions/`. Path scoping works like LIST: `LOOKUP /docs/` scopes to a subtree, `LOOKUP /` covers everything. **Two axes — keep them separate:** - `query` (the subject) matches **`tags` + title** by term overlap. This is the "what is it about" axis: topical labels and the human title. - `filter` matches **any frontmatter `key=value`**, exact. This is the "constrain by attribute" axis: `project`, `type`, `author`, dates, status, etc. Conflating them was rejected: matching the subject query against all frontmatter values turns bookkeeping attributes (`type: journal`, `author: fritz`) into accidental subject hits and makes results impossible to reason about. Attributes belong on the filter axis. `tags` can serve double duty — fuzzy subject match via `query`, exact membership via `filter: tag=go`. **Request** (subject + options in metadata, scope in the request line): ``` LOOKUP /docs/ --- query: auth middleware # matched against tags + title filter: project=broker,modified-after=2025-01-01 # optional, exact key=value over any frontmatter limit: 10 auth: # optional, gates read-auth-protected paths --- ``` - `query` (REQUIRED): subject text, lowercased and split into terms. A doc matches if its tags or title share at least one term. Min 2 chars, max ~256 bytes. - `filter` (OPTIONAL): comma-separated `key=value` over declared frontmatter, plus built-ins `modified-after` / `modified-before`. Applied before ranking. - `limit` (OPTIONAL): default 10, hard cap 1000 (matches LIST). - `auth` (OPTIONAL): token for read-auth-gated paths. `query`, `filter`, `limit` are LOOKUP request options read directly by `handleLookup` (not added to the global PUBLISH/APPEND control-key set — they are generic words a publisher may legitimately use as document metadata, so reserving them globally would silently drop real metadata). `matches` is a reserved response key (server-owned, like `entries` for LIST). **Response** — markdown table (consistent with LIST/VERSIONS). The `Path` column is **server-relative**; the client composes the full `mark://host/path` from the host it dialed (the server cannot reliably know its own public host, same as LIST/VERSIONS which emit relative links). **No body snippets** — bodies cost tokens and defeat the purpose; the title is the hint, the agent FETCHes the chosen doc: ``` --- status: ok matches: 2 --- # Lookup matches for "auth middleware" in /docs/ | Path | Importance | Title | Tags | |------|------------|-------|------| | /docs/auth-middleware | 0.90 | Auth middleware design | go, auth, middleware | | /docs/gateway | 0.50 | Gateway overview | go | ``` ## Declared metadata (set on PUBLISH) Catalog fields ride the existing PUBLISH path as publisher metadata, so they respect version immutability (change a tag → new version). The server interprets **three** keys — `tags`, `importance`, and `title`; everything else stays opaque and is reachable only through `filter`: - `tags`: comma-joined string, e.g. `tags: go,auth,middleware`. The subject match target. Multi-valued via comma split + membership. **Interpreted** (split + matched). - `importance`: float in [0,1], the ranking weight. Default 0.5 when absent or invalid. **Interpreted** (validated, ranking weight). - `title` (optional): one-line human title shown in results and part of the subject match. Fallback chain: declared `title` → first `# H1` of the body (extracted cheaply during the startup walk, which already reads the body) → path basename. **Interpreted** for display + query matching, with the fallback; unlike `tags`/`importance` it needs no validation or splitting. A `/guidelines.md` note should ask the authoring agent to set `tags` + `importance` on PUBLISH for soul docs; catalog quality depends on it, and since the agent authors the docs it is enforceable. ## Ranking No TF-IDF. Deterministic sort by: 1. number of distinct query terms found in tags or title (desc) — tag matches may weight slightly above title matches (refinement, optional in v1), 2. `importance` (desc), 3. modified time (desc), 4. path (asc), stable tiebreaker. Importance is a **prior, not an override**: a doc only ranks if it matched the subject at all, so importance orders genuine matches and never floats an unrelated doc onto the list. At soul scale (hundreds of docs) this is a linear scan over the in-memory catalog — sub-millisecond, no inverted index. ## Catalog design A new `server/internal/catalog` package, owned by the server, updated by the handler after successful store writes. The store stays pure storage; the catalog is a server feature. - In memory: `map[path]entry`, entry = `{tags []string, importance float, title string, modified time}`. One small entry per doc — no token map, no postings. - **Built at startup** by walking the store, piggybacking the same walk the hash index already does (one pass reads each current body once, feeds the hash index and the catalog, and grabs the first H1 for the title fallback). - **Maintained inline on writes** — the handler updates the catalog immediately after a successful PUBLISH (add/replace), APPEND (re-read tags/title), ARCHIVE (remove), unarchive (re-add). All four are the consistency surface; missing one drifts the catalog. - Content directory stays the sole source of truth; the catalog is derived and fully rebuildable from a walk. No persistence — restart re-walks (mirrors the hash index; gob only if startup cost ever proves painful on an Orange Pi). - Guarded by `sync.RWMutex` (single writer, concurrent readers). ### Out-of-band writes (known edge) The inline model assumes writes flow through the handler. The hardened **read-only / `demarkus-publish`** deployment writes straight to the on-disk store, bypassing the handler, so the running server's catalog (and the existing hash index) won't see those writes until a rebuild. Pre-existing limitation of the in-memory-index pattern. v1 accepts staleness; clean fix is to hang a rebuild off the existing `configwatch`/SIGHUP reload path. Noted, not solved in v1. ## Security: read-auth filtering The catalog holds all docs including read-auth-protected ones, but **results are filtered per entry against the caller's token** using the same check as FETCH (`authorizeRead`). Unauthorized matches are dropped silently — no path, no title, no existence signal. Single most important correctness requirement; gets dedicated leakage tests. ## Module touch points - `protocol/`: `VerbLookup = "LOOKUP"` (done, Slice 1) + add to `IsValidVerb` (with the handler, Slice 3). `matches` reserved response key (done, Slice 1). `query`/`filter`/`limit` are read directly by the handler, not globally reserved. - `protocol/store/`: expose a current-version walk hook so the catalog builds from the same pass; cheap first-H1 extraction for title; validate `importance` as float in [0,1]. - `server/internal/catalog/`: build-from-walk, tag/importance/title extraction, match + rank, filter eval, RWMutex. Small (no tokenizer-as-inverted-index, no postings, no snippets-from-body). - `server/internal/handler/`: `handleLookup` + dispatch (and `IsValidVerb` activation); parse query/filter/limit; **read-auth filter results**; build markdown response; query validation. Update catalog after PUBLISH/APPEND/ARCHIVE/unarchive. - `client/fetch/`: `Client.Lookup(host, scope, query, token, opts)`. - `client/cmd/demarkus/`: `lookup` subcommand. - `client/cmd/demarkus-mcp/`: `mark_lookup` tool + add to `markClient` interface. - `docs/SPEC.md`: §6.7 LOOKUP (catalog semantics), metadata tables, `matches` key, verb count 6→7. **Done 2026-05-30.** - Soul `/architecture.md` + `/roadmap.md`: verb set 6→7; retract the "SEARCH removed" note, framed as a catalog/card-catalog supplement to the index (full-text stays in the sidecar); `/guidelines.md` note to tag + rate importance on PUBLISH. ## Phasing (small, incremental, each tested + pre-commit) 1. **Protocol verb** — DONE (PR #166): `VerbLookup` constant + `matches` reserved key + test. `IsValidVerb` deferred to Slice 3 so the verb is never recognised before it works. 2. **catalog package**: build-from-walk, extract tags/importance/title, match (tags+title) + rank, filter. Table tests. 3. **Handler + wiring**: `handleLookup`, `IsValidVerb` activation, dispatch, query validation, markdown response, **read-auth result filtering** (+ leakage tests), catalog updates on PUBLISH/APPEND/ARCHIVE/unarchive. 4. **Client surfaces**: `fetch.Client.Lookup`, `demarkus lookup` CLI, `mark_lookup` MCP tool + interface. Keep CLI/MCP consistent. 5. **Spec + soul**: SPEC §6.7 (done), architecture/roadmap/guidelines updates. ## Out of core (sidecar / client layer, settled) - **Full-text / semantic matching** — never in core. Opt-in sidecar binary that reads demarkus over LIST/FETCH, builds its own (Bleve/vector) index, exposes its own surface. Core never depends on it. - **Federated lookup across worlds/hubs** — client/agent layer, mirrors federation over FETCH. - Match-centered snippets, positional indexing, query DSLs, phrase matching. ## Risks - Catalog/store drift if any write path (esp. APPEND/ARCHIVE/unarchive) misses an update. Mitigated by build-from-walk being authoritative and rebuildable. - Out-of-band writes (`demarkus-publish`, direct-to-store sync) skip inline updates; stale until restart/rebuild. Known edge, deferred. - Read-auth leakage via results. Mitigated by per-result auth filtering + leakage tests. - **Recall**: a subject neither tagged nor in any title returns nothing. This is the honest catalog contract, not a defect — mitigated by the authoring agent tagging reliably (guideline) and the index/links remaining the fallback. The sidecar is the escape hatch if misses prove common. - Tagging discipline is the quality lever — if `tags`/`importance` go unset, the catalog degrades to title-only lookup. Acceptable (no worse than the index), but worth measuring. --- ## Implementation status (2026-05-30, PR #166) Slices 1-4 + broker parity shipped on branch `feat/lookup-verb`: - **Slice 1** — `VerbLookup` constant, `matches` reserved key. (`IsValidVerb` was deliberately deferred to Slice 3 so the verb was never recognised before it worked.) - **Slice 2** — `server/internal/catalog` package (build-from-walk extraction, tag+title match, rank, filter), fully unit-tested. - **Slice 3** — `IsValidVerb` activation, `handleLookup` + dispatch, read-auth result filtering, markdown table, catalog updates on PUBLISH/APPEND/ARCHIVE/unarchive; `store.WalkCurrent` (shares traversal with `BuildHashIndex`); startup build in `main.go`. Spec §6.7 + metadata tables shipped here too. - **Slice 4** — `fetch.Client.Lookup`, `demarkus lookup` CLI, MCP `mark_lookup`. - **Broker parity** — `mark_lookup` added to the gateway as its 14th tool (`worldDispatcher.Lookup`, `handleMarkLookup`, registration, MCP-API.md + helm README updated). **End-to-end smoke test passed:** real server + CLI over QUIC — publish two docs → `demarkus lookup` returned the ranked table (title-derived), scoped lookups and `matches: 0` worked, and ARCHIVE removed a doc from the catalog. Decision deviations from the original plan, all flagged in review and accepted: - Only `matches` is a reserved key; `query`/`filter`/`limit` are read directly by the handler (not added to the global PUBLISH/APPEND control-key set — they are generic words a publisher might legitimately use as metadata). - Response `Path` column is server-relative (the server can't know its public host); the client composes the full `mark://host/path`. - `title` is the third server-interpreted key (spec/plan both corrected from "two keys" after CodeRabbit caught it). ## Slice 6 — Publish-side metadata (the tags/importance gap) **The gap (found in the smoke test):** no shipping client sets `tags`/`importance` on PUBLISH. The CLI's `requestMain`/`editMain` pass `nil` metadata; MCP `mark_publish` passes only `agentMeta(ctx)` = `{"agent": }`. The server *accepts* publisher metadata via PUBLISH request frontmatter (proven by `TestHandleLookupCatalogUpdates`, whose mock request sets `tags: middleware,go`), but nothing exposes it. So in real use the catalog only ever gets `title` (from the body H1) and the default `importance` 0.5 — the tag matching and importance ranking that justify the verb are unreachable. This means LOOKUP is currently a **title-only** lookup in practice. Useful, but not the designed feature. Closing this is the difference between "find docs whose title contains a word" and "find docs an author filed under this subject, ranked by stated importance." **Scope — expose publisher metadata at every publish surface, reusing the server's existing support (no core change):** 1. **CLI** (`client/cmd/demarkus`): add a repeatable `-meta key=value` flag to PUBLISH (and `edit`?), or dedicated `-tags`/`-importance` flags. Repeatable `-meta` is the general primitive (`tags`, `importance`, and any future publisher key flow through one mechanism); `-tags`/`-importance` could be convenience aliases. Validate `key` against `protocol.IsValidMetaKey`; pass the assembled map into `client.Publish(..., meta)` instead of `nil`. 2. **MCP `mark_publish`** (`client/cmd/demarkus-mcp`): add an optional `metadata` object parameter (or explicit `tags`/`importance` params) merged into the meta map alongside the existing `agent` key. This is the load-bearing one — agents publish via MCP. 3. **Broker `mark_publish`** (`tools/demarkus-broker`): mirror the MCP `mark_publish` parameter for byte-for-byte parity, threading meta through `worldDispatcher.Publish` (which already takes a meta map). 4. **APPEND** carries publisher metadata too (it already accepts a meta map); decide whether `-meta` applies to `demarkus -X APPEND` and `mark_append` for consistency. Note the existing APPEND quirk: an APPEND with no meta drops the prior version's publisher metadata — worth documenting, possibly worth carry-forward, but out of scope unless it bites. 5. **Guidelines** (`/guidelines.md`): once settable, add the authoring note — set `tags` (subject labels) and `importance` (0-1) on PUBLISH for soul docs so LOOKUP ranks them well. (Deferred until the capability exists — an unfulfillable guideline is worse than none.) 6. **Obsidian plugin** — separate repo / cadence; note it should expose the same on publish, but not blocking. **Tests:** CLI publishes `-meta tags=go,auth` → FETCH/LOOKUP shows the tags; MCP `mark_publish` with metadata → meta reaches `client.Publish`; broker parity test. Reuse the existing reserved/invalid-key rejection paths. **Open question for Fritz:** repeatable generic `-meta key=value` vs. dedicated `-tags`/`-importance` flags (and the MCP equivalent — a generic `metadata` object vs. named `tags`/`importance` params). Generic is more future-proof and matches the server's open publisher-metadata model; named is more discoverable for the two keys LOOKUP actually interprets. Leaning generic `-meta` / `metadata` with the two interpreted keys called out in the description. --- ## MERGED to main (2026-05-30, PR #166) All slices (1-6) + broker parity merged. Soul updated: architecture v14, roadmap v37 (LOOKUP = DONE), patterns v11 (LOOKUP authoring note: tag + rate on publish). Plan complete; this doc stays as the design + decision record. Deferred tail: `mark_append` metadata; plugin `knowledge-join.md` tool-count + version-pin bump (next plugin update).