# Plan: Agent Memory Leaderboard entry

Enter demarkus in the Agent Memory Leaderboard (agentmemoryleaderboard.ai) next evaluation cycle, using agentic search over a per-user demarkus world as the submission, self-hosted on a droplet.

## Background

The leaderboard is an open benchmark for agent memory systems. Cycle 1 submissions closed 2026-08-07; first results due mid-August 2026. Target is the next cycle; watch the challenge repo for dates.

Participants implement exactly two synchronous HTTP endpoints:

- **Add**: `POST` with `request_id`, `messages` (role, content, optional timestamp), `user_id` (isolation boundary), `session_id`. Respond 200 with `success: true` and echoed ids. Data must be immediately searchable.
- **Search**: `POST` with `query`, optional `options`, `user_id`, `top_k` (formal value 100). Respond with ordered `data` array of `{id, content, score?, created_at?}`, most relevant first. Search must return memory evidence only, never generated answers.

Scoring pipeline: our Search output feeds the platform's locked answer model (undisclosed, identical for all entrants), then fixed per-question-type scoring contracts, aggregated per dataset and Overall. No gold answers, criteria, or bulk datasets are published. We control only Add and Search; score equals quality of the ranked evidence we return.

Tracks: **Textual Memory** (fact recall, multi-hop, temporal, governance, personalization, rule execution, safety) and **Coding Agent Memory** (debugging experience, dev experience, project context). Coding track is demarkus home turf.

## Decisions

1. **Board and route: commercial, self-hosted API.** demarkus is a product, not a paper; the commercial board feeds the demarkus-as-a-service story. Code submission would have the platform run our Docker, meaning echo v5 keys and live inference inside their infra. Self-hosted keeps keys, agent loop, and tuning private.
2. **Search is agentic, not vector search.** A navigation agent with `mark_lookup`, `mark_fetch`, `mark_backlinks` tools walks the user's subtree and returns the documents it read as ranked evidence. This benchmarks demarkus as designed and differentiates against vector-DB entrants. Everything downstream of Search is the platform's, so an agent loop inside Search is invisible to scoring except via latency.
3. **Weight shifts to Add-time.** `mark_lookup` only finds what is tagged and titled, so the distillation cascade at Add-time (tags, titles, links to a per-user hub, timestamps in metadata) is what feeds the searcher. Raw content is stored verbatim; the benchmark judges recall, not curation.
4. **Deployment: single droplet.** caddy (TLS) fronting a Go adapter exposing /add and /search, demarkus server on localhost with the filesystem store, echo v5 API as the inference backend. Fits the 30-day endpoint-stability requirement.

## Architecture

```
caddy (TLS, auth)
  └── adapter (Go): POST /add, POST /search
        ├── add path:  distill agent -> echo v5 -> mark_publish /u/<user_id>/...
        ├── search path: nav agent -> echo v5 -> lookup/fetch/backlinks -> ranked records
        └── demarkus server (localhost, filesystem store)
```

- Isolation: `user_id` maps to a path prefix (or world); the adapter enforces it.
- Cost centre is echo v5 tokens, not the box. Cheap model tier for Add-time tagging, stronger model for Search navigation if echo v5 proves weak at iterative multi-hop.
- Adapter needs a request queue and rate-limit handling; Add stays synchronous per contract.
- Audit trail: log `request_id` to published paths and full search traces, for organizer audits and our own tuning.
- systemd units, world-dir backups, endpoint frozen 30 days post-submission.

## Phases

| # | Work | Output |
|---|------|--------|
| 0 | Recon: challenge repo, harness, latency/timeout limits, LLM-in-search legality, cycle 2 dates, register | verified constraints |
| 1 | Adapter skeleton: Add/Search HTTP contract against local demarkus | passing local smoke |
| 2 | Agentic search implementation; FTS index as backstop tool the agent can call | ranked top-k |
| 3 | Ingestion tuning: distillation cascade, per-user hub linking, temporal metadata | recall quality |
| 4 | Self-eval on public sets (LoCoMo, LongMemEval style) | scores before submitting |
| 5 | Droplet deploy, smoke test, formal submission for cycle 2 | listing |

## Open items

- Tracks: coding only, or both. Coding first; decide on textual after recon.
- Repo placement: new repo vs subdir in demarkus.
- Whether FTS lands in demarkus core (fixes known `mark_lookup` limits) or stays adapter-only.
- Echo v5 capability check for iterative multi-hop navigation.

## Risks

- Unknown Search latency limit; agent loop costs seconds, not milliseconds.
- Inference cost across thousands of eval calls.
- Concurrency profile of the harness is unknown.
- Textual-track sub-areas (governance, rule execution, safety) may not map cleanly onto a document store.