# Plan: Agent Memory Leaderboard entry

Enter demarkus in the Agent Memory Leaderboard next evaluation cycle as an Academic Methods submission, using agentic search over a per-user demarkus world and the hosted Add/Search API route.

## Background

The leaderboard is an open benchmark for agent memory systems. Cycle 1 submissions closed 2026-08-07. Target is a later cycle; watch the challenge repository for dates.

Participants implement exactly two synchronous HTTP endpoints:

- **Add**: `POST` with `request_id`, `messages` (role, content, optional timestamp), `user_id` (isolation boundary), and `session_id`. Respond 200 with `success: true` and echoed IDs. Data must be immediately searchable.
- **Search**: `POST` with `query`, optional `options`, `user_id`, and `top_k`. Respond with an ordered `data` array of `{id, content, score?, created_at?}`, most relevant first. Search returns memory evidence only, never generated answers.

The platform's locked answer and scoring flow consumes Search evidence. We control Add and Search, so score depends on retrieval quality and evidence ordering.

The submission targets Coding Agent Memory first. Decide whether to add Textual Memory before requesting evaluation access.

## Decisions

1. **Board and route: Academic Methods, hosted API.** Demarkus is an open-source framework and method, not a production product. The official Academic route accepts a public repository plus hosted Add/Search endpoints.
2. **Repository disclosure: public Apache-2.0 adapter.** The adapter repository is public and Apache-2.0. Demarkus remains AGPL-3.0, nib remains Apache-2.0, and all versions and modifications are disclosed.
3. **Search is agentic, not vector search.** A scoped navigation agent uses `mark_lookup`, `mark_list`, and `mark_fetch`, then returns fetched documents as ranked evidence.
4. **Add-time distillation feeds retrieval.** Raw content stays verbatim while the model produces title, summary, tags, and importance for catalog discovery.
5. **Academic model: OpenAI `gpt-4o-mini`.** Compose pins `https://api.openai.com/v1` and `gpt-4o-mini` for both Add and Search, matching the Academic full-run checklist.
6. **Deployment is a single DigitalOcean host.** Caddy terminates TLS, the Go adapter exposes Add/Search, and Demarkus remains private. Docker data currently uses Droplet root ext4 after the user declined an encrypted block volume; this departs from the original explicit encrypted-storage requirement.

Echo v5 in the repository is the Go web framework. It is not an LLM provider.

## Architecture

```text
Caddy (public TLS at benchmark.demarkus.io)
  -> adapter (Go): POST /add, POST /search
       -> add: gpt-4o-mini distillation -> mark_publish /u/<user_id>/...
       -> search: gpt-4o-mini navigation -> lookup/list/fetch -> ranked records
       -> Demarkus server (private Docker network)
```

- Isolation: `user_id` maps to an encoded path prefix enforced by the adapter.
- Add remains synchronous. Search has a 90-second navigation budget and 120-second response timeout.
- Daily maintenance creates a local backup and executes 30-day data/backup retention with brief downtime.
- Deployment details: [production deployment](/memoryleaderboard/deployment.md).

## Phases

| # | Work | Status |
|---|------|--------|
| 0 | Recon: challenge contract, route, cycle dates | complete |
| 1 | Add/Search adapter against Demarkus | merged |
| 2 | Agentic search with catalog fallback | merged in PR #2 |
| 3 | Distillation, temporal metadata, idempotency | merged in PR #3 |
| 4 | Local and live-provider conformance, quality, and load evaluation | complete with repeated-memory Top-K gap recorded |
| 5 | Academic public release, host deploy, platform Smoke and Full | active; host live and validated |

## Validation Results

- Endpoint: `https://benchmark.demarkus.io`, release `874c86f`.
- Live smoke: relevant evidence returned, no cross-user data, unauthenticated Search returns 401.
- Provider-backed quality before deployment: recall_any, recall_all, evidence recall, nDCG, and MRR all `1.0`; one extra rejected Redis proposal.
- Deployed conformance: 18 pass, 0 fail, 3 known permissive-parser warnings.
- Deployed 16-worker capacity: 32 Adds and 64 Searches across 32 users, all 200; zero schema, isolation, or visibility failures; Search p95 4.63 seconds.
- Local backup checksum and isolated restored-volume Demarkus startup pass.

## Active Phase 5

1. Configure encrypted off-host backup storage with a lifecycle of 30 days or less.
2. Resolve or formally accept the unencrypted guest Docker-storage exception before non-evaluation customer data.
3. Request Academic evaluation access with Coding Memory and hosted API route.
4. Run platform Smoke and Full against `https://benchmark.demarkus.io`.
5. Keep the submitted endpoint stable and retain evaluation data only for the allowed period.

## Open Items

- Confirm Coding-only or Coding plus Textual before applying.
- Obtain the next-cycle application and evaluation schedule.
- Decide whether repeated-memory exhaustive Top-K recall needs further work before submission.
- Record final evaluation pricing and platform results.

## Risks

- Docker data is not guest/block-volume encrypted; user explicitly chose Droplet root storage.
- Off-host backups are not configured yet.
- Daily filesystem-consistent maintenance creates a brief 503 window.
- Agentic retrieval can return fewer than Top-K when many records are equally relevant.
- Provider rolling daily buckets may produce transient 429s; short advertised delays are retried twice.

## Related Documents

- [Agent Memory Leaderboard bookmark](/memoryleaderboard/agent-memory-leaderboard.md)
- [Memory leaderboard project hub](/memoryleaderboard/index.md)
- [Production deployment](/memoryleaderboard/deployment.md)
- [Hosted operations runbook](https://github.com/latebit-io/memoryleaderboard/blob/main/docs/operations.md)
- [Demarkus as a service](/plans/demarkus-as-a-service.md)
- [Lookup verb](/plans/lookup-verb.md)

## Status Update: Access Request Submitted

Submitted the Academic Methods hosted-API access request on 2026-08-18, targeting Coding Memory and fixed source commit `874c86f19306bf1161bcb0c7898a1435f23a1d6b`. Await the AML Leaderboard/Eval Key by email. On approval: store the key privately, run official Smoke once, inspect all errors/results, then confirm the eight-item Full gate before using the once-per-three-month Full quota.