Plan: Agent Memory Leaderboard entry

Enter demarkus in the Agent Memory Leaderboard next evaluation cycle as an Academic Methods submission, using agentic search over a per-user demarkus world and the hosted Add/Search API route.

Background

The leaderboard is an open benchmark for agent memory systems. Cycle 1 submissions closed 2026-08-07. Target is a later cycle; watch the challenge repository for dates.

Participants implement exactly two synchronous HTTP endpoints:

• Add: POST with request_id, messages (role, content, optional timestamp), user_id (isolation boundary), and session_id. Respond 200 with success: true and echoed IDs. Data must be immediately searchable.
• Search: POST with query, optional options, user_id, and top_k. Respond with an ordered data array of {id, content, score?, created_at?}, most relevant first. Search returns memory evidence only, never generated answers.

The platform's locked answer and scoring flow consumes Search evidence. We control Add and Search, so score depends on retrieval quality and evidence ordering.

The submission targets Coding Agent Memory first. Decide whether to add Textual Memory before requesting evaluation access.

Decisions

1. Board and route: Academic Methods, hosted API. Demarkus is an open-source framework and method, not a production product. The official Academic route accepts a public repository plus hosted Add/Search endpoints.
2. Repository disclosure: public Apache-2.0 adapter. The adapter repository is public and Apache-2.0. Demarkus remains AGPL-3.0, nib remains Apache-2.0, and all versions and modifications are disclosed.
3. Search is agentic, not vector search. A scoped navigation agent uses mark_lookup, mark_list, and mark_fetch, then returns fetched documents as ranked evidence.
4. Add-time distillation feeds retrieval. Raw content stays verbatim while the model produces title, summary, tags, and importance for catalog discovery.
5. Academic model: OpenAI gpt-4o-mini. Compose pins https://api.openai.com/v1 and gpt-4o-mini for both Add and Search, matching the Academic full-run checklist.
6. Deployment is a single encrypted Linux host. Caddy terminates TLS, the Go adapter exposes Add/Search, and Demarkus remains private on the internal network. The Demarkus GHCR image is public and pinned by digest.

Echo v5 in the repository is the Go web framework. It is not an LLM provider.

Architecture

caddy (TLS, auth)
  -> adapter (Go): POST /add, POST /search
       -> add: gpt-4o-mini distillation -> mark_publish /u/<user_id>/...
       -> search: gpt-4o-mini navigation -> lookup/list/fetch -> ranked records
       -> demarkus server (private filesystem store)

• Isolation: user_id maps to an encoded path prefix enforced by the adapter.
• Add remains synchronous. Search has a 90-second navigation budget and 120-second response timeout.
• Audit logs retain request IDs and operational results without logging secrets or customer payloads.
• Backups, endpoint stability, and 30-day deletion follow the hosted operations runbook.

Phases

Active Phase 5

Completed: PR #5 merged Academic public-readiness as 9f4ff7e; the repository is public with Apache-2.0 detection and anonymous access verified.

1. Obtain an OpenAI API key through its secret channel.
2. Run live Add distillation and navigation smoke with gpt-4o-mini.
3. Run live synthetic quality and 16-worker load checks; inspect timeouts, rate limits, and retrieval quality.
4. Provision encrypted host, DNS, TLS, secrets, and off-host backup storage.
5. Request Academic evaluation access with Coding Memory and the hosted API route, then run platform Smoke and Full.
6. Keep the submitted endpoint stable and retain evaluation data only for the allowed period.

Open Items

• Confirm Coding-only or Coding plus Textual before applying.
• Obtain the next-cycle application and evaluation schedule.
• Record final endpoint, capacity, pricing, public commit, and deployment digest with the submitted version.

Risks

• gpt-4o-mini tool reliability and latency are not validated against the live agent loop yet.
• Provider concurrency and rate limits may be lower than the 16-worker target.
• Agentic Search can exhaust the 90-second budget on difficult queries.
• Evaluation volume and provider-token cost remain unknown until access is granted.

Related Documents

• Agent Memory Leaderboard bookmark
• Memory leaderboard project hub
• Hosted operations runbook
• Demarkus as a service
• Lookup verb