soul.demarkus.io:6309/plans/lookup-verb.md/v6 draft reader meta

Plan: LOOKUP verb (topic/importance catalog)

Status: draft, pending Fritz review (opened 2026-05-30). SPEC §6.7 drafted 2026-05-30 (the spec part of Slice 5 is done; soul + guidelines updates and the code slices remain). Slice 1 implemented on branch feat/lookup-verb (PR #166). Tracks issue #113 (filed as "SEARCH"; the verb is named LOOKUP — see Naming).

(Supersedes the earlier /plans/search-verb.md, which is archived. That doc's history holds the full-text TF-IDF design we descoped.)

What it is

LOOKUP is the card catalog for a demarkus world. Given a subject, it returns which documents are about it and how important they are, so the agent can go FETCH the right one without fetching and parsing the whole index hub and guessing. It is a token-efficient supplement to the index, not a search engine.

This fits the project's core metaphor exactly. The architecture's first principle is "the agent as the librarian"; LOOKUP is the catalog the librarian consults — look up a subject, get pointed at the shelf, FETCH the book. It is deliberately dumb: match a subject against author-declared catalog fields, order by importance. No relevance modeling, no body reading.

Naming

Named LOOKUP, not SEARCH, on purpose. "Search" sets a full-text, relevance-engine expectation this does not meet, and its worst failure (an obvious body match that wasn't tagged returns nothing) reads as broken. "Look up" carries the catalog contract honestly: a catalog only knows what was filed in it, so a miss on an uncatalogued subject is expected behavior, not a bug. LOOKUP also keeps the verb cadence of FETCH / LIST / VERSIONS / PUBLISH / ARCHIVE / APPEND.

Settled consequence: full-text / semantic matching never enters core. It lives permanently in an opt-in sidecar (separate binary that reads demarkus over LIST/FETCH and builds its own index). A LOOKUP that did full-text would read wrong, and adding a separate SEARCH verb later would break "the verb set is final." Choosing LOOKUP closes that door on purpose. This finalizes the core verb set at 7.

The verb

LOOKUP is a scoped, importance-ranked lookup over a server-maintained catalog of declared document metadata. It searches current versions only, excludes archived docs, skips versions/. Path scoping works like LIST: LOOKUP /docs/ scopes to a subtree, LOOKUP / covers everything.

Two axes — keep them separate:

  • query (the subject) matches tags + title by term overlap. This is the "what is it about" axis: topical labels and the human title.
  • filter matches any frontmatter key=value, exact. This is the "constrain by attribute" axis: project, type, author, dates, status, etc.

Conflating them was rejected: matching the subject query against all frontmatter values turns bookkeeping attributes (type: journal, author: fritz) into accidental subject hits and makes results impossible to reason about. Attributes belong on the filter axis. tags can serve double duty — fuzzy subject match via query, exact membership via filter: tag=go.

Request (subject + options in metadata, scope in the request line):

LOOKUP /docs/
---
query: auth middleware                          # matched against tags + title
filter: project=broker,modified-after=2025-01-01  # optional, exact key=value over any frontmatter
limit: 10
auth: <token>                                   # optional, gates read-auth-protected paths
---
  • query (REQUIRED): subject text, lowercased and split into terms. A doc matches if its tags or title share at least one term. Min 2 chars, max ~256 bytes.
  • filter (OPTIONAL): comma-separated key=value over declared frontmatter, plus built-ins modified-after / modified-before. Applied before ranking.
  • limit (OPTIONAL): default 10, hard cap 1000 (matches LIST).
  • auth (OPTIONAL): token for read-auth-gated paths.

query, filter, limit are LOOKUP request options read directly by handleLookup (not added to the global PUBLISH/APPEND control-key set — they are generic words a publisher may legitimately use as document metadata, so reserving them globally would silently drop real metadata). matches is a reserved response key (server-owned, like entries for LIST).

Response — markdown table (consistent with LIST/VERSIONS). The Path column is server-relative; the client composes the full mark://host/path from the host it dialed (the server cannot reliably know its own public host, same as LIST/VERSIONS which emit relative links). No body snippets — bodies cost tokens and defeat the purpose; the title is the hint, the agent FETCHes the chosen doc:

---
status: ok
matches: 2
---
# Lookup matches for "auth middleware" in /docs/

| Path | Importance | Title | Tags |
|------|------------|-------|------|
| /docs/auth-middleware | 0.90 | Auth middleware design | go, auth, middleware |
| /docs/gateway         | 0.50 | Gateway overview       | go |

Declared metadata (set on PUBLISH)

Catalog fields ride the existing PUBLISH path as publisher metadata, so they respect version immutability (change a tag → new version). The server interprets three keys — tags, importance, and title; everything else stays opaque and is reachable only through filter:

  • tags: comma-joined string, e.g. tags: go,auth,middleware. The subject match target. Multi-valued via comma split + membership. Interpreted (split + matched).
  • importance: float in [0,1], the ranking weight. Default 0.5 when absent or invalid. Interpreted (validated, ranking weight).
  • title (optional): one-line human title shown in results and part of the subject match. Fallback chain: declared title → first # H1 of the body (extracted cheaply during the startup walk, which already reads the body) → path basename. Interpreted for display + query matching, with the fallback; unlike tags/importance it needs no validation or splitting.

A /guidelines.md note should ask the authoring agent to set tags + importance on PUBLISH for soul docs; catalog quality depends on it, and since the agent authors the docs it is enforceable.

Ranking

No TF-IDF. Deterministic sort by:

  1. number of distinct query terms found in tags or title (desc) — tag matches may weight slightly above title matches (refinement, optional in v1),
  2. importance (desc),
  3. modified time (desc),
  4. path (asc), stable tiebreaker.

Importance is a prior, not an override: a doc only ranks if it matched the subject at all, so importance orders genuine matches and never floats an unrelated doc onto the list. At soul scale (hundreds of docs) this is a linear scan over the in-memory catalog — sub-millisecond, no inverted index.

Catalog design

A new server/internal/catalog package, owned by the server, updated by the handler after successful store writes. The store stays pure storage; the catalog is a server feature.

  • In memory: map[path]entry, entry = {tags []string, importance float, title string, modified time}. One small entry per doc — no token map, no postings.
  • Built at startup by walking the store, piggybacking the same walk the hash index already does (one pass reads each current body once, feeds the hash index and the catalog, and grabs the first H1 for the title fallback).
  • Maintained inline on writes — the handler updates the catalog immediately after a successful PUBLISH (add/replace), APPEND (re-read tags/title), ARCHIVE (remove), unarchive (re-add). All four are the consistency surface; missing one drifts the catalog.
  • Content directory stays the sole source of truth; the catalog is derived and fully rebuildable from a walk. No persistence — restart re-walks (mirrors the hash index; gob only if startup cost ever proves painful on an Orange Pi).
  • Guarded by sync.RWMutex (single writer, concurrent readers).

Out-of-band writes (known edge)

The inline model assumes writes flow through the handler. The hardened read-only / demarkus-publish deployment writes straight to the on-disk store, bypassing the handler, so the running server's catalog (and the existing hash index) won't see those writes until a rebuild. Pre-existing limitation of the in-memory-index pattern. v1 accepts staleness; clean fix is to hang a rebuild off the existing configwatch/SIGHUP reload path. Noted, not solved in v1.

Security: read-auth filtering

The catalog holds all docs including read-auth-protected ones, but results are filtered per entry against the caller's token using the same check as FETCH (authorizeRead). Unauthorized matches are dropped silently — no path, no title, no existence signal. Single most important correctness requirement; gets dedicated leakage tests.

Module touch points

  • protocol/: VerbLookup = "LOOKUP" (done, Slice 1) + add to IsValidVerb (with the handler, Slice 3). matches reserved response key (done, Slice 1). query/filter/limit are read directly by the handler, not globally reserved.
  • protocol/store/: expose a current-version walk hook so the catalog builds from the same pass; cheap first-H1 extraction for title; validate importance as float in [0,1].
  • server/internal/catalog/: build-from-walk, tag/importance/title extraction, match + rank, filter eval, RWMutex. Small (no tokenizer-as-inverted-index, no postings, no snippets-from-body).
  • server/internal/handler/: handleLookup + dispatch (and IsValidVerb activation); parse query/filter/limit; read-auth filter results; build markdown response; query validation. Update catalog after PUBLISH/APPEND/ARCHIVE/unarchive.
  • client/fetch/: Client.Lookup(host, scope, query, token, opts).
  • client/cmd/demarkus/: lookup subcommand.
  • client/cmd/demarkus-mcp/: mark_lookup tool + add to markClient interface.
  • docs/SPEC.md: §6.7 LOOKUP (catalog semantics), metadata tables, matches key, verb count 6→7. Done 2026-05-30.
  • Soul /architecture.md + /roadmap.md: verb set 6→7; retract the "SEARCH removed" note, framed as a catalog/card-catalog supplement to the index (full-text stays in the sidecar); /guidelines.md note to tag + rate importance on PUBLISH.

Phasing (small, incremental, each tested + pre-commit)

  1. Protocol verb — DONE (PR #166): VerbLookup constant + matches reserved key + test. IsValidVerb deferred to Slice 3 so the verb is never recognised before it works.
  2. catalog package: build-from-walk, extract tags/importance/title, match (tags+title) + rank, filter. Table tests.
  3. Handler + wiring: handleLookup, IsValidVerb activation, dispatch, query validation, markdown response, read-auth result filtering (+ leakage tests), catalog updates on PUBLISH/APPEND/ARCHIVE/unarchive.
  4. Client surfaces: fetch.Client.Lookup, demarkus lookup CLI, mark_lookup MCP tool + interface. Keep CLI/MCP consistent.
  5. Spec + soul: SPEC §6.7 (done), architecture/roadmap/guidelines updates.

Out of core (sidecar / client layer, settled)

  • Full-text / semantic matching — never in core. Opt-in sidecar binary that reads demarkus over LIST/FETCH, builds its own (Bleve/vector) index, exposes its own surface. Core never depends on it.
  • Federated lookup across worlds/hubs — client/agent layer, mirrors federation over FETCH.
  • Match-centered snippets, positional indexing, query DSLs, phrase matching.

Risks

  • Catalog/store drift if any write path (esp. APPEND/ARCHIVE/unarchive) misses an update. Mitigated by build-from-walk being authoritative and rebuildable.
  • Out-of-band writes (demarkus-publish, direct-to-store sync) skip inline updates; stale until restart/rebuild. Known edge, deferred.
  • Read-auth leakage via results. Mitigated by per-result auth filtering + leakage tests.
  • Recall: a subject neither tagged nor in any title returns nothing. This is the honest catalog contract, not a defect — mitigated by the authoring agent tagging reliably (guideline) and the index/links remaining the fallback. The sidecar is the escape hatch if misses prove common.
  • Tagging discipline is the quality lever — if tags/importance go unset, the catalog degrades to title-only lookup. Acceptable (no worse than the index), but worth measuring.

Implementation status (2026-05-30, PR #166)

Slices 1-4 + broker parity shipped on branch feat/lookup-verb:

  • Slice 1 — VerbLookup constant, matches reserved key. (IsValidVerb was deliberately deferred to Slice 3 so the verb was never recognised before it worked.)
  • Slice 2 — server/internal/catalog package (build-from-walk extraction, tag+title match, rank, filter), fully unit-tested.
  • Slice 3 — IsValidVerb activation, handleLookup + dispatch, read-auth result filtering, markdown table, catalog updates on PUBLISH/APPEND/ARCHIVE/unarchive; store.WalkCurrent (shares traversal with BuildHashIndex); startup build in main.go. Spec §6.7 + metadata tables shipped here too.
  • Slice 4 — fetch.Client.Lookup, demarkus lookup CLI, MCP mark_lookup.
  • Broker parity — mark_lookup added to the gateway as its 14th tool (worldDispatcher.Lookup, handleMarkLookup, registration, MCP-API.md + helm README updated).

End-to-end smoke test passed: real server + CLI over QUIC — publish two docs → demarkus lookup returned the ranked table (title-derived), scoped lookups and matches: 0 worked, and ARCHIVE removed a doc from the catalog.

Decision deviations from the original plan, all flagged in review and accepted:

  • Only matches is a reserved key; query/filter/limit are read directly by the handler (not added to the global PUBLISH/APPEND control-key set — they are generic words a publisher might legitimately use as metadata).
  • Response Path column is server-relative (the server can't know its public host); the client composes the full mark://host/path.
  • title is the third server-interpreted key (spec/plan both corrected from "two keys" after CodeRabbit caught it).

Slice 6 — Publish-side metadata (the tags/importance gap)

The gap (found in the smoke test): no shipping client sets tags/importance on PUBLISH. The CLI's requestMain/editMain pass nil metadata; MCP mark_publish passes only agentMeta(ctx) = {"agent": <name>}. The server accepts publisher metadata via PUBLISH request frontmatter (proven by TestHandleLookupCatalogUpdates, whose mock request sets tags: middleware,go), but nothing exposes it. So in real use the catalog only ever gets title (from the body H1) and the default importance 0.5 — the tag matching and importance ranking that justify the verb are unreachable.

This means LOOKUP is currently a title-only lookup in practice. Useful, but not the designed feature. Closing this is the difference between "find docs whose title contains a word" and "find docs an author filed under this subject, ranked by stated importance."

Scope — expose publisher metadata at every publish surface, reusing the server's existing support (no core change):

  1. CLI (client/cmd/demarkus): add a repeatable -meta key=value flag to PUBLISH (and edit?), or dedicated -tags/-importance flags. Repeatable -meta is the general primitive (tags, importance, and any future publisher key flow through one mechanism); -tags/-importance could be convenience aliases. Validate key against protocol.IsValidMetaKey; pass the assembled map into client.Publish(..., meta) instead of nil.
  2. MCP mark_publish (client/cmd/demarkus-mcp): add an optional metadata object parameter (or explicit tags/importance params) merged into the meta map alongside the existing agent key. This is the load-bearing one — agents publish via MCP.
  3. Broker mark_publish (tools/demarkus-broker): mirror the MCP mark_publish parameter for byte-for-byte parity, threading meta through worldDispatcher.Publish (which already takes a meta map).
  4. APPEND carries publisher metadata too (it already accepts a meta map); decide whether -meta applies to demarkus -X APPEND and mark_append for consistency. Note the existing APPEND quirk: an APPEND with no meta drops the prior version's publisher metadata — worth documenting, possibly worth carry-forward, but out of scope unless it bites.
  5. Guidelines (/guidelines.md): once settable, add the authoring note — set tags (subject labels) and importance (0-1) on PUBLISH for soul docs so LOOKUP ranks them well. (Deferred until the capability exists — an unfulfillable guideline is worse than none.)
  6. Obsidian plugin — separate repo / cadence; note it should expose the same on publish, but not blocking.

Tests: CLI publishes -meta tags=go,auth → FETCH/LOOKUP shows the tags; MCP mark_publish with metadata → meta reaches client.Publish; broker parity test. Reuse the existing reserved/invalid-key rejection paths.

Open question for Fritz: repeatable generic -meta key=value vs. dedicated -tags/-importance flags (and the MCP equivalent — a generic metadata object vs. named tags/importance params). Generic is more future-proof and matches the server's open publisher-metadata model; named is more discoverable for the two keys LOOKUP actually interprets. Leaning generic -meta / metadata with the two interpreted keys called out in the description.

MERGED to main (2026-05-30, PR #166)

All slices (1-6) + broker parity merged. Soul updated: architecture v14, roadmap v37 (LOOKUP = DONE), patterns v11 (LOOKUP authoring note: tag + rate on publish). Plan complete; this doc stays as the design + decision record.

Deferred tail: mark_append metadata; plugin knowledge-join.md tool-count + version-pin bump (next plugin update).

Status — SHIPPED (verified 2026-05-31)

Merged to main as PR #166: protocol verb + spec §6.7, server/internal/catalog, fetch.Client.Lookup, demarkus lookup CLI, mark_lookup MCP, broker parity (14th gateway tool). Publisher tags/importance metadata shipped the same effort (CLI -meta, MCP/broker metadata). Plugin surfacing (mark_lookup wiring + knowledge-join tool-count fix) shipped in plugin v0.3.0 (#168). Remaining tail: mark_append metadata — deferred by design (you tag on publish, not append). The frontmatter status: draft is stale; this plan is complete and now lives under Completed Plans in /index.md.

trail
  1. soul.demarkus.io:6309 v6