soul.demarkus.io:6309/memoryleaderboard/application-readiness.md/v3 active reader meta

Agent Memory Leaderboard application readiness

Audit against the live official participation guide, Evaluation Access form, API guide, and Full gate on 2026-08-18.

Route

  • Division: Academic Methods
  • Evaluation route: participant-hosted Add/Search API
  • Intended evaluation type: Coding Memory
  • System: demarkus
  • Public repository: https://github.com/latebit-io/memoryleaderboard
  • Hosted endpoint origin: https://benchmark.demarkus.io

Requirement Audit

Requirement Status Evidence
Public verifiable GitHub repository Pass Anonymous raw README and repository access work
Fixed source version Pass, update pending Deployed 874c86f; local README application-URL correction needs a new commit before final submission
Apache-2.0 adapter license Pass GitHub detects Apache-2.0
Original method/source/change disclosure Pass README identifies original Latebit demarkus/nib work, licenses, adapter changes, and no external paper/method claim
Docker startup and API run instructions Pass README, Compose, Dockerfile, .env.example, bootstrap script, and operations runbook
Separate Add/Search URLs Pass after README correction /add and /search; current application form requires full URLs
Public unauthenticated health Pass /health and /healthz return 200 with nav/distill enabled
Dedicated Memory System Key Pass Independent 64-hex production adapter key; X-Api-Key, Bearer, and Token verified
Synchronous Add and exact ID echo Pass Deployed conformance
Search data schema, array options, Top-K 100 Pass Deployed conformance
Stable ranked IDs and no-result empty array Pass Deployed conformance
Strict user_id isolation Pass Deployed conformance and 16-worker load
Search returns evidence, not answers Pass Scoped lookup/list/fetch navigator returns stored documents
Add/Search model is gpt-4o-mini Pass Compose and running container environment both fixed to OpenAI gpt-4o-mini
At least 16 concurrent Add/Search requests Pass 32 Adds and 64 Searches at concurrency 16, all 200; no schema/isolation/visibility failures; Search p95 4.63s
Capacity, timeout, rate details available Pass Operations runbook; recommend application run settings Add 16, Search 16, Top K 100, timeout 130s
Stable for 30 days Human confirmation required Host is live; application checkbox is a future commitment
Delete evaluation data within 30 days Pass Daily scheduled local backup and 30-day data/backup purge
Secret/history hygiene Pass Git history leak scan clean; ignored local OpenAI key is expected and not tracked
Official platform Smoke Pending Eval Key Can only run after application approval

Current Blockers

  1. Commit and merge the README correction replacing stale base-URL submission guidance with exact Add/Search/Health URLs; update the host checkout and declare the new fixed commit.
  2. Confirm contact identity, work email, organization/team, final system/version display names, and public display information.
  3. Confirm the hosted API will remain stable for at least 30 days.
  4. Submit the access request. The inaugural-cycle deadline was 2026-08-07, but the live site still says Academic submissions are open and the request API is active; acceptance into the first release is not guaranteed.

Application Fields

  • Leaderboard: Academic
  • Academic method: Provide Add/Search APIs
  • Work email: fritz@latebit.io if approved by user
  • Name: user confirmation required
  • Organization/Team: Latebit if approved by user
  • System name: demarkus
  • Version name: agentic-nav-<final commit short SHA>
  • Add API URL: https://benchmark.demarkus.io/add
  • Search API URL: https://benchmark.demarkus.io/search
  • Authentication: X-Api-Key
  • Memory System Key: production adapter API key, entered only in the encrypted field
  • Add status URL: leave blank
  • Health URL: https://benchmark.demarkus.io/health
  • Public GitHub repository: https://github.com/latebit-io/memoryleaderboard
  • Invitation code: optional

Submission Notes Draft

Academic Methods hosted-API submission targeting Coding Memory. demarkus is original Latebit work; nib is original Latebit work. There is no external paper or third-party memory method/technical report claimed by this submission. The Apache-2.0 adapter integrates the AGPL-3.0 Demarkus server and Apache-2.0 nib agent with the official synchronous Add/Search contract, adding injective per-user storage scopes, immutable raw-message retention, Add-time metadata distillation, scoped agentic lookup/list/fetch navigation, authenticated HTTP endpoints, local evaluation tooling, and hosted operations.

Fixed source: https://github.com/latebit-io/memoryleaderboard at commit <FINAL_SHA>. Reproduction: copy .env.example to .env, set DOMAIN and ACME_EMAIL, run ./scripts/bootstrap-secrets.sh, docker compose build adapter, then docker compose up -d. Compose fixes OpenAI gpt-4o-mini for both Add distillation and Search navigation. API wrapper: internal/api/server.go; production deployment: compose.yaml; operations: docs/operations.md.

Hosted API: Add https://benchmark.demarkus.io/add; Search https://benchmark.demarkus.io/search; Health https://benchmark.demarkus.io/health; authentication X-Api-Key with the Memory System Key supplied separately in the encrypted form field. Add is synchronous. Search returns ranked stored evidence only and enforces user_id isolation.

Capacity: use Add concurrency 16, Search concurrency 16, Top K 100, and client timeout at least 130 seconds. No edge request-rate limit is configured; model capacity is the practical bound. The endpoint will remain stable for at least 30 days. Evaluation data is locally backed up and purged, with backups, within 30 days.

Non-Application Risks

  • Encrypted off-host backup remains deferred.
  • Docker data remains on Droplet root ext4 by explicit user decision.
  • Daily filesystem-consistent maintenance creates a brief retryable 503 window; disable the timer during official Smoke/Full and re-enable afterward.
  • Repeated-memory exhaustive Top-K retrieval can return fewer than all equally relevant records, affecting score but not contract eligibility.

Confirmed Application Identity

  • Contact name: Fritz Seitz
  • Work email: fritz@latebit.io
  • Organization/team: Latebit
  • System display name: demarkus
  • Hosted endpoint stability: confirmed for at least 30 days after submission

Remaining pre-request action: commit/merge the README endpoint correction, update the host checkout to that final commit, substitute its SHA in the version name and notes, then submit the Evaluation Access Request. Official Smoke follows Eval Key issuance.

Submitted

Evaluation Access Request submitted on 2026-08-18 for Academic Methods, participant-hosted API, targeting Coding Memory. Declared source version: 874c86f19306bf1161bcb0c7898a1435f23a1d6b. Hosted Add/Search/Health endpoints and the production Memory System Key were supplied through the official form.

Await AML review and the Leaderboard/Eval Key at fritz@latebit.io. Do not start Full before official Smoke passes. Preserve the declared endpoint behavior and commit throughout review/evaluation.

trail
  1. soul.demarkus.io:6309 v3