Budgeted evidence selection

For demarkus maintainers: the design for the roadmap slice that makes one budgeted lookup answer a multi-document question. Status: done (2026-10-10; PR 562 merged as ffe6093abd0ab1806b821306a7c5571210916c01; harness on bench/context-selection). Roadmap entry: budgeted evidence selection. Design rationale: graph focus.

What the before arms say

The OpenCode 2 before arms (r2, 2026-10-10) spend about 11% of model tokens on tool results and about 85% on the prompt replayed per turn. Cost tracks model turns, not evidence size: a three-turn attempt costs about 5,800 tokens, a seven-turn one about 17,000. Every attempt is a lookup followed by one to four fetches, with mark_explore in between for lineage, supersession and dependency questions. Budgeted expansion fired on 7 of 43 lookups because the reader policy says to omit it by default, and even then the reader fetched afterwards.

So the lever is turns. A budgeted lookup that already holds the direct passage and the one hop the question needs removes the explore call and one or two fetches.

Decision

Keep mark_lookup and its budget argument. Change what the shared expansion (client/lookupexpand, used by the local MCP and the gateway) puts inside the budget, in five parts.

1. The budget covers the whole result. The table and envelope already printed count against it; expansion gets the remainder, and relation lines are budgeted like blocks. When the table alone exceeds the budget the result says so and names the expansion path (raise the budget or fetch a row), instead of silently adding nothing.
2. Overlapping and duplicate evidence coalesces. Rows on one document expand once per byte range: a later row whose section lies inside, or encloses, an emitted section is reported as covered by that block rather than printed again. A section whose text is byte-identical to one already printed, from another document or anchor, prints as a one-line block naming the first location, so distinct provenance and its version pin survive without a second copy; the stub covers its range too. Rank order is kept; the strongest direct match always prints first.
3. Every block is citable as a section. A whole document opened by its H1 is pinned to that anchor (>>> /genres/heavy-metal.md/v1#heavy-metal); a document with a preamble or a second H1 stays a whole-document block. The answer contract asks for an immutable version and an observed section, and the first paid arm showed readers citing path/vN from unpinned blocks and failing on that alone.
4. One-hop additions come from the strongest match's own rel-* metadata. Every expanded document prints one grouped line of its declared relations under its block (>>> related: rel-by /a.md, /b.md; rel-genre /g.md; the subject is the block above). After every direct row has had its turn, the relations of the first document printed, and only that one, expand under the remaining budget, up to three targets, evidence predicates first (supersedes, superseded-by, depends-on, implements, derived-from) and then the rest in declaration order, each whole when small, else the sections holding the query terms, else the outline, under a plain header; the relation line above is what names the hop, and no hop follows a line that did not fit. A related document prints its own relation line but is not followed further. Targets already in the table are skipped; targets on another authority or outside the caller's read scope are never fetched, so scope and token never widen. No graph store is consulted, so the hop is fresh by construction and identical on both surfaces. Incoming relations known only to the store are a measured follow-on.
5. Omissions are visible. Rows that did not fit are listed with their size, capped, so the reader can fetch the one it wants. The closing note counts rows and related documents expanded.

Fetch count stays bounded by MaxFetches across direct and related documents. Body links are navigation, not evidence, and stay out of expansion. Tool descriptions are unchanged, so the reader schema is byte-identical to main's and arms compare. The section match rule lives once as mdoutline.Find; headings are parsed once per fetched document.

Why the strongest match only, and why every predicate: the first replay expanded typed relations of every printed row and added rock, hardcore punk and thrash metal to a question about doom metal, 30% more result tokens for no new evidence. Restricting the hop to the top document kept the one hop that mattered (Nightwish to Helloween through rel-influenced-by, which the roadmap's four families would have excluded) and dropped the rest. Which predicates carry evidence is a property of the world's vocabulary (ADR 0004 leaves curation to authors), so the slice prefers the roadmap's families when it must choose and otherwise trusts the author's declaration, inside the budget.

Not in this slice

• Any new tool or default behavior. A lookup without budget is byte-identical to today.
• Relation edges from the graph store (incoming relations, seeded rows). Added only if measured misses justify a store dependency inside lookup.
• Semantic deduplication. Only byte-identical text coalesces.
• Shortest paths, centrality and clustering remain evidence-gated follow-ons.

Evaluation

The benchmarks branch was rebuilt the same day as main at a8d037c0 plus the bench overlay (bench-only directories, the read-fence patch to client/fetch and demarkus-mcp, the answer-bench Make target and the artifact ignores), because its client had been 85 commits behind main and the r2 before arms had measured a demarkus-mcp without PR 561. The GCS corpus pack, the catalog ranking probe and the fan-out bench were dropped in the refresh; everything else was ported to the current client and store APIs.

Offline replay first (TestContextSelectionReplay, env-gated): the r2 readers' recorded first lookups with budget 1500 through both binaries against the frozen corpora. Final binaries: result tokens 14,254 to 14,134 on latebit (no hop fires: the supersession pair is already in the table) and 11,328 to 13,914 on music with six hops; required passages under a citable header rose from 4 to 6 and from 0 to 8 of 32 fields. Mechanics only, not a model score.

Paid arms, 16 attempts each, OpenCode 2.0.23 private-server reader, prepared cache scope-crawl-v1, scoring v3, report with hashes at tools/benchmarks/answers/context-selection-2026-10-10/README.md on the benchmarks branch:

Strict compare B against C r2 (same RunSpec, MCP hash only): music -7.53% tokens per correct, 15 to 16 correct, six related expansions, improved point estimate; latebit -25.72%, 9 to 13 correct, no regressions, feature gate inconclusive because no hop fires on latebit and the gain is citation validity from the H1 pins. compare-policies A′ against B: latebit +21.75% with four tasks regressed, music -29.20% with one. The remaining latebit misses under the budget policy are the reader's own (one field cited from the wrong page; ADR 0007 where the rubric wants 0007), identical in B and C.

A first after arm (C r1) is retained as superseded: it changed the tool descriptions (a reader-schema change compare rejects), its hop headers carried a suffix the scorer could not read as an observation, and its whole-document blocks were unpinned. About 1.1M reader tokens were spent in all. The merged binary differs from C r2 only in the two review fixes (stub span, budgeted relation line) and the simplification pass; neither changes output on the measured traces.

Reading: the software improves the budgeted path on both worlds with no task regression. Whether a budget-first reader beats a section-first one depends on the world: it does on music, it does not on latebit, where the current-client section-first baseline stays 16/16. Shipped guidance keeps section-first as the default; budget-first stays opt-in until a mixed policy (budget on multi-document questions, sections on direct facts) is measured on held-out tasks.

Mixed policy follow-on (2026-10-11)

graph-mixed-v1: section-first with the two budget sentences replaced by a rule keyed on the question (direct facts and historical revisions omit budget; lineage, supersession, dependency, influence and project roles pass budget 1500 first). v8 datasets are v7 re-pinned. Measured against a fresh section-first arm on the merged client, identical binaries (MCP 6ac61b23), report tools/benchmarks/answers/mixed-policy-2026-10-11/README.md:

Direct categories stayed 12/12 and 10/10 at fewer tokens; multi-document categories went 5/6 to 6/6 on music and 4/4 to 3/4 on latebit. The latebit miss is a reader attribution slip (client page cited for the service field after a stray Go query), not an expansion defect. Mixed readers still chose budget on some direct categories and never called mark_explore. Decision open: ship the mixed rule as default guidance or keep section-first.