Plan: Agent Memory Leaderboard entry

Enter demarkus in the Agent Memory Leaderboard next evaluation cycle as an Academic Methods submission, using agentic search over a per-user demarkus world and the hosted Add/Search API route.

Background

The leaderboard is an open benchmark for agent memory systems. Cycle 1 submissions closed 2026-08-07. Target is a later cycle; watch the challenge repository for dates.

Participants implement exactly two synchronous HTTP endpoints:

• Add: POST with request_id, messages (role, content, optional timestamp), user_id (isolation boundary), and session_id. Respond 200 with success: true and echoed IDs. Data must be immediately searchable.
• Search: POST with query, optional options, user_id, and top_k. Respond with an ordered data array of {id, content, score?, created_at?}, most relevant first. Search returns memory evidence only, never generated answers.

The platform's locked answer and scoring flow consumes Search evidence. We control Add and Search, so score depends on retrieval quality and evidence ordering.

The submission targets Coding Agent Memory first. Decide whether to add Textual Memory before requesting evaluation access.

Decisions

1. Board and route: Academic Methods, hosted API. Demarkus is an open-source framework and method, not a production product. The official Academic route accepts a public repository plus hosted Add/Search endpoints.
2. Repository disclosure: public Apache-2.0 adapter. The adapter repository is public and Apache-2.0. Demarkus remains AGPL-3.0, nib remains Apache-2.0, and all versions and modifications are disclosed.
3. Search is agentic, not vector search. A scoped navigation agent uses mark_lookup, mark_list, and mark_fetch, then returns fetched documents as ranked evidence.
4. Add-time distillation feeds retrieval. Raw content stays verbatim while the model produces title, summary, tags, and importance for catalog discovery.
5. Academic model: OpenAI gpt-4o-mini. Compose pins https://api.openai.com/v1 and gpt-4o-mini for both Add and Search, matching the Academic full-run checklist.
6. Deployment is a single DigitalOcean host. Caddy terminates TLS, the Go adapter exposes Add/Search, and Demarkus remains private. Docker data currently uses Droplet root ext4 after the user declined an encrypted block volume; this departs from the original explicit encrypted-storage requirement.

Echo v5 in the repository is the Go web framework. It is not an LLM provider.

Architecture

Caddy (public TLS at benchmark.demarkus.io)
  -> adapter (Go): POST /add, POST /search
       -> add: gpt-4o-mini distillation -> mark_publish /u/<user_id>/...
       -> search: gpt-4o-mini navigation -> lookup/list/fetch -> ranked records
       -> Demarkus server (private Docker network)

• Isolation: user_id maps to an encoded path prefix enforced by the adapter.
• Add remains synchronous. Search has a 90-second navigation budget and 120-second response timeout.
• Daily maintenance creates a local backup and executes 30-day data/backup retention with brief downtime.
• Deployment details: production deployment.

Phases

Validation Results

• Endpoint: https://benchmark.demarkus.io, release 874c86f.
• Live smoke: relevant evidence returned, no cross-user data, unauthenticated Search returns 401.
• Provider-backed quality before deployment: recall_any, recall_all, evidence recall, nDCG, and MRR all 1.0; one extra rejected Redis proposal.
• Deployed conformance: 18 pass, 0 fail, 3 known permissive-parser warnings.
• Deployed 16-worker capacity: 32 Adds and 64 Searches across 32 users, all 200; zero schema, isolation, or visibility failures; Search p95 4.63 seconds.
• Local backup checksum and isolated restored-volume Demarkus startup pass.

Active Phase 5

1. Configure encrypted off-host backup storage with a lifecycle of 30 days or less.
2. Resolve or formally accept the unencrypted guest Docker-storage exception before non-evaluation customer data.
3. Request Academic evaluation access with Coding Memory and hosted API route.
4. Run platform Smoke and Full against https://benchmark.demarkus.io.
5. Keep the submitted endpoint stable and retain evaluation data only for the allowed period.

Open Items

• Confirm Coding-only or Coding plus Textual before applying.
• Obtain the next-cycle application and evaluation schedule.
• Decide whether repeated-memory exhaustive Top-K recall needs further work before submission.
• Record final evaluation pricing and platform results.

Risks

• Docker data is not guest/block-volume encrypted; user explicitly chose Droplet root storage.
• Off-host backups are not configured yet.
• Daily filesystem-consistent maintenance creates a brief 503 window.
• Agentic retrieval can return fewer than Top-K when many records are equally relevant.
• Provider rolling daily buckets may produce transient 429s; short advertised delays are retried twice.

Related Documents