soul.demarkus.io:6309/plans/lookup-verb.md/v1 draft reader meta

Plan: LOOKUP verb (topic/importance catalog)

Status: draft, pending Fritz review (opened 2026-05-30). Tracks issue #113 (filed as "SEARCH"; the verb is named LOOKUP — see Naming).

(Supersedes the earlier /plans/search-verb.md, which is archived. That doc's history holds the full-text TF-IDF design we descoped.)

What it is

LOOKUP is the card catalog for a demarkus world. Given a subject, it returns which documents are about it and how important they are, so the agent can go FETCH the right one without fetching and parsing the whole index hub and guessing. It is a token-efficient supplement to the index, not a search engine.

This fits the project's core metaphor exactly. The architecture's first principle is "the agent as the librarian"; LOOKUP is the catalog the librarian consults — look up a subject, get pointed at the shelf, FETCH the book. It is deliberately dumb: match a subject against author-declared catalog fields, order by importance. No relevance modeling, no body reading.

Naming

Named LOOKUP, not SEARCH, on purpose. "Search" sets a full-text, relevance-engine expectation this does not meet, and its worst failure (an obvious body match that wasn't tagged returns nothing) reads as broken. "Look up" carries the catalog contract honestly: a catalog only knows what was filed in it, so a miss on an uncatalogued subject is expected behavior, not a bug. LOOKUP also keeps the verb cadence of FETCH / LIST / VERSIONS / PUBLISH / ARCHIVE / APPEND.

Settled consequence: full-text / semantic matching never enters core. It lives permanently in an opt-in sidecar (separate binary that reads demarkus over LIST/FETCH and builds its own index). A LOOKUP that did full-text would read wrong, and adding a separate SEARCH verb later would break "the verb set is final." Choosing LOOKUP closes that door on purpose. This finalizes the core verb set at 7.

The verb

LOOKUP is a scoped, importance-ranked lookup over a server-maintained catalog of declared document metadata. It searches current versions only, excludes archived docs, skips versions/. Path scoping works like LIST: LOOKUP /docs/ scopes to a subtree, LOOKUP / covers everything.

Two axes — keep them separate:

  • query (the subject) matches tags + title by term overlap. This is the "what is it about" axis: topical labels and the human title.
  • filter matches any frontmatter key=value, exact. This is the "constrain by attribute" axis: project, type, author, dates, status, etc.

Conflating them was rejected: matching the subject query against all frontmatter values turns bookkeeping attributes (type: journal, author: fritz) into accidental subject hits and makes results impossible to reason about. Attributes belong on the filter axis. tags can serve double duty — fuzzy subject match via query, exact membership via filter: tag=go.

Request (subject + options in metadata, scope in the request line):

LOOKUP /docs/
---
query: auth middleware                          # matched against tags + title
filter: project=broker,modified-after=2025-01-01  # optional, exact key=value over any frontmatter
limit: 10
auth: <token>                                   # optional, gates read-auth-protected paths
---
  • query (REQUIRED): subject text, lowercased and split into terms. A doc matches if its tags or title share at least one term. Min 2 chars, max ~256 bytes.
  • filter (OPTIONAL): comma-separated key=value over declared frontmatter, plus built-ins modified-after / modified-before. Applied before ranking.
  • limit (OPTIONAL): default 10, hard cap 1000 (matches LIST).
  • auth (OPTIONAL): token for read-auth-gated paths.

query, filter, limit are control keys (never stored). matches is a reserved response key.

Response — markdown table (consistent with LIST/VERSIONS, parseable by the index table parser). No body snippets — bodies cost tokens and defeat the purpose; the title is the hint, the agent FETCHes the chosen doc:

---
status: ok
matches: 2
---
# Lookup matches for "auth middleware" in /docs/

| URI | Importance | Title | Tags |
|-----|------------|-------|------|
| mark://soul.demarkus.io/docs/auth-middleware | 0.90 | Auth middleware design | go, auth, middleware |
| mark://soul.demarkus.io/docs/gateway         | 0.50 | Gateway overview       | go |

Declared metadata (set on PUBLISH)

Catalog fields ride the existing PUBLISH path as publisher metadata, so they respect version immutability (change a tag → new version). The server interprets exactly two keys; everything else stays opaque and is reachable only through filter:

  • tags: comma-joined string, e.g. tags: go,auth,middleware. The subject match target. Multi-valued via comma split + membership. Interpreted.
  • importance: float in [0,1], the ranking weight. Default 0.5 when absent or invalid. The first interpreted key. Interpreted.
  • title (optional): one-line human title shown in results, also part of the subject match. Fallback chain: declared title → first # H1 of the body (extracted cheaply during the startup walk, which already reads the body) → path basename. Not "interpreted" beyond display/match; just read.

A /guidelines.md note should ask the authoring agent to set tags + importance on PUBLISH for soul docs; catalog quality depends on it, and since the agent authors the docs it is enforceable.

Ranking

No TF-IDF. Deterministic sort by:

  1. number of distinct query terms found in tags or title (desc) — tag matches may weight slightly above title matches (refinement, optional in v1),
  2. importance (desc),
  3. modified time (desc),
  4. path (asc), stable tiebreaker.

Importance is a prior, not an override: a doc only ranks if it matched the subject at all, so importance orders genuine matches and never floats an unrelated doc onto the list. At soul scale (hundreds of docs) this is a linear scan over the in-memory catalog — sub-millisecond, no inverted index.

Catalog design

A new server/internal/catalog package, owned by the server, updated by the handler after successful store writes. The store stays pure storage; the catalog is a server feature.

  • In memory: map[path]entry, entry = {tags []string, importance float, title string, modified time}. One small entry per doc — no token map, no postings.
  • Built at startup by walking the store, piggybacking the same walk the hash index already does (one pass reads each current body once, feeds the hash index and the catalog, and grabs the first H1 for the title fallback).
  • Maintained inline on writes — the handler updates the catalog immediately after a successful PUBLISH (add/replace), APPEND (re-read tags/title), ARCHIVE (remove), unarchive (re-add). All four are the consistency surface; missing one drifts the catalog.
  • Content directory stays the sole source of truth; the catalog is derived and fully rebuildable from a walk. No persistence — restart re-walks (mirrors the hash index; gob only if startup cost ever proves painful on an Orange Pi).
  • Guarded by sync.RWMutex (single writer, concurrent readers).

Out-of-band writes (known edge)

The inline model assumes writes flow through the handler. The hardened read-only / demarkus-publish deployment writes straight to the on-disk store, bypassing the handler, so the running server's catalog (and the existing hash index) won't see those writes until a rebuild. Pre-existing limitation of the in-memory-index pattern. v1 accepts staleness; clean fix is to hang a rebuild off the existing configwatch/SIGHUP reload path. Noted, not solved in v1.

Security: read-auth filtering

The catalog holds all docs including read-auth-protected ones, but results are filtered per entry against the caller's token using the same check as FETCH (authorizeRead). Unauthorized matches are dropped silently — no path, no title, no existence signal. Single most important correctness requirement; gets dedicated leakage tests.

Module touch points

  • protocol/: VerbLookup = "LOOKUP" + IsValidVerb; control keys (query, filter, limit), reserved response key matches; reserve interpreted importance and tags.
  • protocol/store/: expose a current-version walk hook so the catalog builds from the same pass; cheap first-H1 extraction for title; validate importance as float in [0,1].
  • server/internal/catalog/: build-from-walk, tag/importance/title extraction, match + rank, filter eval, RWMutex. Small (no tokenizer-as-inverted-index, no postings, no snippets-from-body).
  • server/internal/handler/: handleLookup + dispatch; parse query/filter/limit; read-auth filter results; build markdown response; query validation. Update catalog after PUBLISH/APPEND/ARCHIVE/unarchive.
  • client/fetch/: Client.Lookup(host, scope, query, token, opts).
  • client/cmd/demarkus/: lookup subcommand.
  • client/cmd/demarkus-mcp/: mark_lookup tool + add to markClient interface.
  • docs/SPEC.md: §6.7 LOOKUP (catalog semantics), metadata tables, matches key, verb count 6→7.
  • Soul /architecture.md + /roadmap.md: verb set 6→7; retract the "SEARCH removed" note, framed as a catalog/card-catalog supplement to the index (full-text stays in the sidecar); /guidelines.md note to tag + rate importance on PUBLISH.

Phasing (small, incremental, each tested + pre-commit)

  1. Protocol verb: VerbLookup + IsValidVerb + key reservations. Tiny.
  2. catalog package: build-from-walk, extract tags/importance/title, match (tags+title) + rank, filter. Table tests.
  3. Handler + wiring: handleLookup, dispatch, query validation, markdown response, read-auth result filtering (+ leakage tests), catalog updates on PUBLISH/APPEND/ARCHIVE/unarchive.
  4. Client surfaces: fetch.Client.Lookup, demarkus lookup CLI, mark_lookup MCP tool + interface. Keep CLI/MCP consistent.
  5. Spec + soul: SPEC §6.7, architecture/roadmap/guidelines updates.

Out of core (sidecar / client layer, settled)

  • Full-text / semantic matching — never in core. Opt-in sidecar binary that reads demarkus over LIST/FETCH, builds its own (Bleve/vector) index, exposes its own surface. Core never depends on it.
  • Federated lookup across worlds/hubs — client/agent layer, mirrors federation over FETCH.
  • Match-centered snippets, positional indexing, query DSLs, phrase matching.

Risks

  • Catalog/store drift if any write path (esp. APPEND/ARCHIVE/unarchive) misses an update. Mitigated by build-from-walk being authoritative and rebuildable.
  • Out-of-band writes (demarkus-publish, direct-to-store sync) skip inline updates; stale until restart/rebuild. Known edge, deferred.
  • Read-auth leakage via results. Mitigated by per-result auth filtering + leakage tests.
  • Recall: a subject neither tagged nor in any title returns nothing. This is the honest catalog contract, not a defect — mitigated by the authoring agent tagging reliably (guideline) and the index/links remaining the fallback. The sidecar is the escape hatch if misses prove common.
  • Tagging discipline is the quality lever — if tags/importance go unset, the catalog degrades to title-only lookup. Acceptable (no worse than the index), but worth measuring.
trail
  1. soul.demarkus.io:6309 v1