soul.demarkus.io:6309/plans/agent-memory-leaderboard.md/v3 draft reader meta

Plan: Agent Memory Leaderboard entry

Enter demarkus in the Agent Memory Leaderboard (agentmemoryleaderboard.ai) next evaluation cycle, using agentic search over a per-user demarkus world as the submission, self-hosted on a droplet.

Background

The leaderboard is an open benchmark for agent memory systems. Cycle 1 submissions closed 2026-08-07; first results due mid-August 2026. Target is the next cycle; watch the challenge repo for dates.

Participants implement exactly two synchronous HTTP endpoints:

  • Add: POST with request_id, messages (role, content, optional timestamp), user_id (isolation boundary), session_id. Respond 200 with success: true and echoed ids. Data must be immediately searchable.
  • Search: POST with query, optional options, user_id, top_k (formal value 100). Respond with ordered data array of {id, content, score?, created_at?}, most relevant first. Search must return memory evidence only, never generated answers.

Scoring pipeline: our Search output feeds the platform's locked answer model (undisclosed, identical for all entrants), then fixed per-question-type scoring contracts, aggregated per dataset and Overall. No gold answers, criteria, or bulk datasets are published. We control only Add and Search; score equals quality of the ranked evidence we return.

Tracks: Textual Memory (fact recall, multi-hop, temporal, governance, personalization, rule execution, safety) and Coding Agent Memory (debugging experience, dev experience, project context). Coding track is demarkus home turf.

Decisions

  1. Board and route: commercial, self-hosted API. demarkus is a product, not a paper; the commercial board feeds the demarkus-as-a-service story. Code submission would have the platform run our Docker, meaning echo v5 keys and live inference inside their infra. Self-hosted keeps keys, agent loop, and tuning private.
  2. Search is agentic, not vector search. A navigation agent with mark_lookup, mark_fetch, mark_backlinks tools walks the user's subtree and returns the documents it read as ranked evidence. This benchmarks demarkus as designed and differentiates against vector-DB entrants. Everything downstream of Search is the platform's, so an agent loop inside Search is invisible to scoring except via latency.
  3. Weight shifts to Add-time. mark_lookup only finds what is tagged and titled, so the distillation cascade at Add-time (tags, titles, links to a per-user hub, timestamps in metadata) is what feeds the searcher. Raw content is stored verbatim; the benchmark judges recall, not curation.
  4. Deployment: single droplet. caddy (TLS) fronting a Go adapter exposing /add and /search, demarkus server on localhost with the filesystem store, echo v5 API as the inference backend. Fits the 30-day endpoint-stability requirement.

Architecture

caddy (TLS, auth)
  └── adapter (Go): POST /add, POST /search
        ├── add path:  distill agent -> echo v5 -> mark_publish /u/<user_id>/...
        ├── search path: nav agent -> echo v5 -> lookup/fetch/backlinks -> ranked records
        └── demarkus server (localhost, filesystem store)
  • Isolation: user_id maps to a path prefix (or world); the adapter enforces it.
  • Cost centre is echo v5 tokens, not the box. Cheap model tier for Add-time tagging, stronger model for Search navigation if echo v5 proves weak at iterative multi-hop.
  • Adapter needs a request queue and rate-limit handling; Add stays synchronous per contract.
  • Audit trail: log request_id to published paths and full search traces, for organizer audits and our own tuning.
  • systemd units, world-dir backups, endpoint frozen 30 days post-submission.

Phases

# Work Output
0 Recon: challenge repo, harness, latency/timeout limits, LLM-in-search legality, cycle 2 dates, register verified constraints
1 Adapter skeleton: Add/Search HTTP contract against local demarkus passing local smoke
2 Agentic search implementation; FTS index as backstop tool the agent can call ranked top-k
3 Ingestion tuning: distillation cascade, per-user hub linking, temporal metadata recall quality
4 Self-eval on public sets (LoCoMo, LongMemEval style) scores before submitting
5 Droplet deploy, smoke test, formal submission for cycle 2 listing

Open items

  • Tracks: coding only, or both. Coding first; decide on textual after recon.
  • Repo placement: new repo vs subdir in demarkus.
  • Whether FTS lands in demarkus core (fixes known mark_lookup limits) or stays adapter-only.
  • Echo v5 capability check for iterative multi-hop navigation.

Risks

  • Unknown Search latency limit; agent loop costs seconds, not milliseconds.
  • Inference cost across thousands of eval calls.
  • Concurrency profile of the harness is unknown.
  • Textual-track sub-areas (governance, rule execution, safety) may not map cleanly onto a document store.

Related documents

trail
  1. soul.demarkus.io:6309 v3