# Agent Memory Leaderboard application readiness Audit against the live official participation guide, Evaluation Access form, API guide, and Full gate on 2026-08-18. ## Route - Division: Academic Methods - Evaluation route: participant-hosted Add/Search API - Intended evaluation type: Coding Memory - System: `demarkus` - Public repository: `https://github.com/latebit-io/memoryleaderboard` - Hosted endpoint origin: `https://benchmark.demarkus.io` ## Requirement Audit | Requirement | Status | Evidence | |---|---|---| | Public verifiable GitHub repository | Pass | Anonymous raw README and repository access work | | Fixed source version | Pass, update pending | Deployed `874c86f`; local README application-URL correction needs a new commit before final submission | | Apache-2.0 adapter license | Pass | GitHub detects Apache-2.0 | | Original method/source/change disclosure | Pass | README identifies original Latebit demarkus/nib work, licenses, adapter changes, and no external paper/method claim | | Docker startup and API run instructions | Pass | README, Compose, Dockerfile, `.env.example`, bootstrap script, and operations runbook | | Separate Add/Search URLs | Pass after README correction | `/add` and `/search`; current application form requires full URLs | | Public unauthenticated health | Pass | `/health` and `/healthz` return 200 with nav/distill enabled | | Dedicated Memory System Key | Pass | Independent 64-hex production adapter key; X-Api-Key, Bearer, and Token verified | | Synchronous Add and exact ID echo | Pass | Deployed conformance | | Search `data` schema, array options, Top-K 100 | Pass | Deployed conformance | | Stable ranked IDs and no-result empty array | Pass | Deployed conformance | | Strict `user_id` isolation | Pass | Deployed conformance and 16-worker load | | Search returns evidence, not answers | Pass | Scoped lookup/list/fetch navigator returns stored documents | | Add/Search model is `gpt-4o-mini` | Pass | Compose and running container environment both fixed to OpenAI `gpt-4o-mini` | | At least 16 concurrent Add/Search requests | Pass | 32 Adds and 64 Searches at concurrency 16, all 200; no schema/isolation/visibility failures; Search p95 4.63s | | Capacity, timeout, rate details available | Pass | Operations runbook; recommend application run settings Add 16, Search 16, Top K 100, timeout 130s | | Stable for 30 days | Human confirmation required | Host is live; application checkbox is a future commitment | | Delete evaluation data within 30 days | Pass | Daily scheduled local backup and 30-day data/backup purge | | Secret/history hygiene | Pass | Git history leak scan clean; ignored local OpenAI key is expected and not tracked | | Official platform Smoke | Pending Eval Key | Can only run after application approval | ## Current Blockers 1. Commit and merge the README correction replacing stale base-URL submission guidance with exact Add/Search/Health URLs; update the host checkout and declare the new fixed commit. 2. Confirm contact identity, work email, organization/team, final system/version display names, and public display information. 3. Confirm the hosted API will remain stable for at least 30 days. 4. Submit the access request. The inaugural-cycle deadline was 2026-08-07, but the live site still says Academic submissions are open and the request API is active; acceptance into the first release is not guaranteed. ## Application Fields - Leaderboard: Academic - Academic method: Provide Add/Search APIs - Work email: `fritz@latebit.io` if approved by user - Name: user confirmation required - Organization/Team: `Latebit` if approved by user - System name: `demarkus` - Version name: `agentic-nav-` - Add API URL: `https://benchmark.demarkus.io/add` - Search API URL: `https://benchmark.demarkus.io/search` - Authentication: `X-Api-Key` - Memory System Key: production adapter API key, entered only in the encrypted field - Add status URL: leave blank - Health URL: `https://benchmark.demarkus.io/health` - Public GitHub repository: `https://github.com/latebit-io/memoryleaderboard` - Invitation code: optional ## Submission Notes Draft Academic Methods hosted-API submission targeting Coding Memory. `demarkus` is original Latebit work; `nib` is original Latebit work. There is no external paper or third-party memory method/technical report claimed by this submission. The Apache-2.0 adapter integrates the AGPL-3.0 Demarkus server and Apache-2.0 nib agent with the official synchronous Add/Search contract, adding injective per-user storage scopes, immutable raw-message retention, Add-time metadata distillation, scoped agentic lookup/list/fetch navigation, authenticated HTTP endpoints, local evaluation tooling, and hosted operations. Fixed source: https://github.com/latebit-io/memoryleaderboard at commit ``. Reproduction: copy `.env.example` to `.env`, set DOMAIN and ACME_EMAIL, run `./scripts/bootstrap-secrets.sh`, `docker compose build adapter`, then `docker compose up -d`. Compose fixes OpenAI `gpt-4o-mini` for both Add distillation and Search navigation. API wrapper: `internal/api/server.go`; production deployment: `compose.yaml`; operations: `docs/operations.md`. Hosted API: Add `https://benchmark.demarkus.io/add`; Search `https://benchmark.demarkus.io/search`; Health `https://benchmark.demarkus.io/health`; authentication `X-Api-Key` with the Memory System Key supplied separately in the encrypted form field. Add is synchronous. Search returns ranked stored evidence only and enforces `user_id` isolation. Capacity: use Add concurrency 16, Search concurrency 16, Top K 100, and client timeout at least 130 seconds. No edge request-rate limit is configured; model capacity is the practical bound. The endpoint will remain stable for at least 30 days. Evaluation data is locally backed up and purged, with backups, within 30 days. ## Non-Application Risks - Encrypted off-host backup remains deferred. - Docker data remains on Droplet root ext4 by explicit user decision. - Daily filesystem-consistent maintenance creates a brief retryable 503 window; disable the timer during official Smoke/Full and re-enable afterward. - Repeated-memory exhaustive Top-K retrieval can return fewer than all equally relevant records, affecting score but not contract eligibility.