Plan: Agent Memory Leaderboard entry

Enter demarkus in the Agent Memory Leaderboard (agentmemoryleaderboard.ai) next evaluation cycle, using agentic search over a per-user demarkus world as the submission, self-hosted on an encrypted Linux host.

Background

The leaderboard is an open benchmark for agent memory systems. Cycle 1 submissions closed 2026-08-07. Target is a later cycle; watch the challenge repository for dates.

Participants implement exactly two synchronous HTTP endpoints:

• Add: POST with request_id, messages (role, content, optional timestamp), user_id (isolation boundary), session_id. Respond 200 with success: true and echoed IDs. Data must be immediately searchable.
• Search: POST with query, optional options, user_id, and top_k. Respond with an ordered data array of {id, content, score?, created_at?}, most relevant first. Search returns memory evidence only, never generated answers.

The platform's locked answer and scoring flow consumes Search evidence. We control Add and Search, so score depends on retrieval quality and evidence ordering.

Tracks are Textual Memory and Coding Agent Memory. Coding is the initial target; decide whether to add Textual before requesting evaluation access.

Decisions

1. Board and route: Commercial/Industry, self-hosted API. Demarkus is a product, not an academic method. Self-hosting keeps keys, agent loop, and tuning private.
2. Search is agentic, not vector search. A scoped navigation agent uses mark_lookup, mark_list, and mark_fetch, then returns fetched documents as ranked evidence.
3. Add-time distillation feeds retrieval. Raw content stays verbatim while the model produces title, summary, tags, and importance for catalog discovery.
4. Deployment is a single encrypted Linux host. Caddy terminates TLS, the Go adapter exposes Add/Search, and Demarkus remains private on the internal network.
5. MiniMax-M3 is the low-cost validation baseline. Use https://api.minimax.io/v1 with MiniMax-M3. Freeze it only after live distillation, navigation, quality, and 16-worker load checks pass with the exact endpoint and model.

Echo v5 in the repository is the Go web framework. It is not an LLM provider.

Architecture

caddy (TLS, auth)
  -> adapter (Go): POST /add, POST /search
       -> add: MiniMax-M3 distillation -> mark_publish /u/<user_id>/...
       -> search: MiniMax-M3 navigation -> lookup/list/fetch -> ranked records
       -> demarkus server (private filesystem store)

• Isolation: user_id maps to an encoded path prefix enforced by the adapter.
• Provider cost is driven by distillation and multi-turn Search navigation. MiniMax-M3 currently costs $0.30 per input MTok and $1.20 per output MTok.
• Add remains synchronous. Search has a 90-second navigation budget and 120-second response timeout.
• Audit logs retain request IDs and operational results without logging secrets or customer payloads.
• Backups, endpoint stability, and 30-day deletion follow the hosted operations runbook.

Phases

Active Phase 5

1. Obtain a MiniMax API key through its secret channel.
2. Run live Add distillation and navigation smoke with the MiniMax-M3 endpoint and model.
3. Run live synthetic quality and 16-worker load checks; inspect timeouts, rate limits, and retrieval quality.
4. Freeze MiniMax-M3 if it passes, or run a targeted model bakeoff if it does not.
5. Provision encrypted host, DNS, TLS, secrets, and off-host backup storage.
6. Request Agent Memory Leaderboard evaluation access, run platform Smoke, then Full.
7. Keep the submitted endpoint stable and retain evaluation data only for the allowed period.

Open Items

• Confirm Coding-only or Coding plus Textual before applying.
• Decide whether catalog search needs an FTS backstop after live model evaluation.
• Obtain the next-cycle application and evaluation schedule.
• Record final provider, model, endpoint, capacity, and pricing with the submitted version.

Risks

• MiniMax-M3 tool reliability and latency are not validated against this agent loop yet.
• Provider concurrency and rate limits may be lower than the 16-worker target.
• Agentic Search can exhaust the 90-second budget on difficult queries.
• Evaluation volume and provider-token cost remain unknown until access is granted.

Related Documents

• Agent Memory Leaderboard bookmark
• Memory leaderboard project hub
• Hosted operations runbook
• Demarkus as a service
• Lookup verb