memoryleaderboard: Project Hub
Entry for demarkus in the Agent Memory Leaderboard: a Go adapter exposing the benchmark's Add/Search HTTP contract over a per-user demarkus world, with agentic search as the submission.
Status
The repository is public at latebit-io/memoryleaderboard, with main fixed at Academic-readiness merge 9f4ff7e. Live gpt-4o-mini validation is complete: smoke passes with zero noise; provider-backed quality has perfect recall, evidence recall, nDCG, and MRR with one extra rejected proposal; conformance is 18 pass/0 fail; and a 16-worker, 32-user capacity run completed 32 Adds plus 64 Searches with zero HTTP, schema, isolation, or visibility failures. Search p95 was 6.57 seconds. Repeated-memory Top-K runs retained a 5-7/64 exhaustive-visibility gap despite otherwise clean operation. Provider diagnostics, bounded 429 retry, empty-result handling, and evidence-selection tuning are complete on uncommitted branch fix/nav-provider-errors. Full race tests, vet, shell checks, and diff checks pass.
Plan
Authoritative plan: Agent Memory Leaderboard entry. Search uses a scoped nib navigation agent over Demarkus lookup/list/fetch; Add uses gpt-4o-mini distillation while retaining raw evidence. Production is a single encrypted Linux host with Caddy TLS, adapter, private Demarkus storage, and explicit 30-day data/backup deletion. The Academic submission uses Coding Memory and the hosted Add/Search API route.
Echo v5 is the Go HTTP framework used by the adapter, not an LLM provider.
Phases
| # | Work | Output |
|---|---|---|
| 0 | Recon: challenge repo, limits, cycle dates, route | Academic hosted API route selected |
| 1 | Adapter skeleton: Add/Search against local Demarkus | merged |
| 2 | Agentic search; catalog fallback | merged via PR #2 |
| 3 | Ingestion tuning: distillation, temporal metadata, idempotency | merged via PR #3 |
| 4 | Conformance, synthetic retrieval quality, Top-K load evaluation | live provider gates complete; repeated-memory Top-K gap recorded |
| 5 | Public release, provider validation, host deploy, Smoke, Full | public release and provider validation complete; host deploy pending |
Repo
github.com/latebit-io/memoryleaderboard is public. Local checkout: /Users/fritz/latebit/memoryleaderboard, branch fix/nav-provider-errors over main at 9f4ff7e.
Workspace
- Agent Memory Leaderboard official benchmark bookmark and entry links
- Agent Memory Leaderboard entry plan active roadmap
- journal/ session notes
- debugging.md investigation lessons and portability gotchas
- Add architecture.md and patterns.md as they earn content.