# Agent Memory Leaderboard application readiness Audit against the live official participation guide, Evaluation Access form, API guide, and Full gate on 2026-08-18. ## Route - Division: Academic Methods - Evaluation route: participant-hosted Add/Search API - Intended evaluation type: Coding Memory - System: `demarkus` - Public repository: `https://github.com/latebit-io/memoryleaderboard` - Hosted endpoint origin: `https://benchmark.demarkus.io` ## Requirement Audit | Requirement | Status | Evidence | |---|---|---| | Public verifiable GitHub repository | Pass | Anonymous raw README and repository access work | | Fixed source version | Pass, update pending | Deployed `874c86f`; local README application-URL correction needs a new commit before final submission | | Apache-2.0 adapter license | Pass | GitHub detects Apache-2.0 | | Original method/source/change disclosure | Pass | README identifies original Latebit demarkus/nib work, licenses, adapter changes, and no external paper/method claim | | Docker startup and API run instructions | Pass | README, Compose, Dockerfile, `.env.example`, bootstrap script, and operations runbook | | Separate Add/Search URLs | Pass after README correction | `/add` and `/search`; current application form requires full URLs | | Public unauthenticated health | Pass | `/health` and `/healthz` return 200 with nav/distill enabled | | Dedicated Memory System Key | Pass | Independent 64-hex production adapter key; X-Api-Key, Bearer, and Token verified | | Synchronous Add and exact ID echo | Pass | Deployed conformance | | Search `data` schema, array options, Top-K 100 | Pass | Deployed conformance | | Stable ranked IDs and no-result empty array | Pass | Deployed conformance | | Strict `user_id` isolation | Pass | Deployed conformance and 16-worker load | | Search returns evidence, not answers | Pass | Scoped lookup/list/fetch navigator returns stored documents | | Add/Search model is `gpt-4o-mini` | Pass | Compose and running container environment both fixed to OpenAI `gpt-4o-mini` | | At least 16 concurrent Add/Search requests | Pass | 32 Adds and 64 Searches at concurrency 16, all 200; no schema/isolation/visibility failures; Search p95 4.63s | | Capacity, timeout, rate details available | Pass | Operations runbook; recommend application run settings Add 16, Search 16, Top K 100, timeout 130s | | Stable for 30 days | Human confirmation required | Host is live; application checkbox is a future commitment | | Delete evaluation data within 30 days | Pass | Daily scheduled local backup and 30-day data/backup purge | | Secret/history hygiene | Pass | Git history leak scan clean; ignored local OpenAI key is expected and not tracked | | Official platform Smoke | Pending Eval Key | Can only run after application approval | ## Current Blockers 1. Commit and merge the README correction replacing stale base-URL submission guidance with exact Add/Search/Health URLs; update the host checkout and declare the new fixed commit. 2. Confirm contact identity, work email, organization/team, final system/version display names, and public display information. 3. Confirm the hosted API will remain stable for at least 30 days. 4. Submit the access request. The inaugural-cycle deadline was 2026-08-07, but the live site still says Academic submissions are open and the request API is active; acceptance into the first release is not guaranteed. ## Application Fields - Leaderboard: Academic - Academic method: Provide Add/Search APIs - Work email: `fritz@latebit.io` if approved by user - Name: user confirmation required - Organization/Team: `Latebit` if approved by user - System name: `demarkus` - Version name: `agentic-nav-` - Add API URL: `https://benchmark.demarkus.io/add` - Search API URL: `https://benchmark.demarkus.io/search` - Authentication: `X-Api-Key` - Memory System Key: production adapter API key, entered only in the encrypted field - Add status URL: leave blank - Health URL: `https://benchmark.demarkus.io/health` - Public GitHub repository: `https://github.com/latebit-io/memoryleaderboard` - Invitation code: optional ## Submission Notes Draft Academic Methods hosted-API submission targeting Coding Memory. `demarkus` is original Latebit work; `nib` is original Latebit work. There is no external paper or third-party memory method/technical report claimed by this submission. The Apache-2.0 adapter integrates the AGPL-3.0 Demarkus server and Apache-2.0 nib agent with the official synchronous Add/Search contract, adding injective per-user storage scopes, immutable raw-message retention, Add-time metadata distillation, scoped agentic lookup/list/fetch navigation, authenticated HTTP endpoints, local evaluation tooling, and hosted operations. Fixed source: https://github.com/latebit-io/memoryleaderboard at commit ``. Reproduction: copy `.env.example` to `.env`, set DOMAIN and ACME_EMAIL, run `./scripts/bootstrap-secrets.sh`, `docker compose build adapter`, then `docker compose up -d`. Compose fixes OpenAI `gpt-4o-mini` for both Add distillation and Search navigation. API wrapper: `internal/api/server.go`; production deployment: `compose.yaml`; operations: `docs/operations.md`. Hosted API: Add `https://benchmark.demarkus.io/add`; Search `https://benchmark.demarkus.io/search`; Health `https://benchmark.demarkus.io/health`; authentication `X-Api-Key` with the Memory System Key supplied separately in the encrypted form field. Add is synchronous. Search returns ranked stored evidence only and enforces `user_id` isolation. Capacity: use Add concurrency 16, Search concurrency 16, Top K 100, and client timeout at least 130 seconds. No edge request-rate limit is configured; model capacity is the practical bound. The endpoint will remain stable for at least 30 days. Evaluation data is locally backed up and purged, with backups, within 30 days. ## Non-Application Risks - Encrypted off-host backup remains deferred. - Docker data remains on Droplet root ext4 by explicit user decision. - Daily filesystem-consistent maintenance creates a brief retryable 503 window; disable the timer during official Smoke/Full and re-enable afterward. - Repeated-memory exhaustive Top-K retrieval can return fewer than all equally relevant records, affecting score but not contract eligibility. ## Confirmed Application Identity - Contact name: Fritz Seitz - Work email: `fritz@latebit.io` - Organization/team: Latebit - System display name: `demarkus` - Hosted endpoint stability: confirmed for at least 30 days after submission Remaining pre-request action: commit/merge the README endpoint correction, update the host checkout to that final commit, substitute its SHA in the version name and notes, then submit the Evaluation Access Request. Official Smoke follows Eval Key issuance.