# memoryleaderboard production deployment

Verified hosted Academic API deployment and operational state.

## Endpoint

- Public endpoint: `https://benchmark.demarkus.io`
- Health: public `/healthz`; `/add` and `/search` require the production adapter API key
- DNS: Cloudflare DNS-only `A` record to `165.22.236.152`; no `AAAA`
- TLS: Caddy ACME/Let's Encrypt with HSTS

## Host

- DigitalOcean Droplet `memoryleaderboard-prod`, ID `593339479`, Toronto `tor1`
- Premium AMD Basic `s-2vcpu-4gb-amd`: 2 vCPU, 4 GiB RAM, 80 GB NVMe, $28/month
- Ubuntu 24.04 LTS; Docker 29.1.3; Compose 2.40.3
- Non-root `deploy` SSH only; password/root SSH disabled; unattended upgrades enabled
- DigitalOcean firewall `847aabb3-460a-45b7-88d6-9cd04a8ad20a` plus UFW expose public TCP 80/443; SSH is source-restricted; adapter 8080 and Demarkus UDP 6309 are not public

## Release

- Git commit: `874c86f19306bf1161bcb0c7898a1435f23a1d6b`
- Adapter image: `sha256:c421063c5aa388c32205bc03aa587418109f10697d37c70d127ed9beb05ec2b1`, built on host
- Demarkus: `0.22.7@sha256:bb42d2b2f322d08dc30248b3ec659882dea2c2f3ec2068470a28705ff0f5a854`
- Caddy: `2.10.2-alpine@sha256:4c6e91c6ed0e2fa03efd5b44747b625fec79bc9cd06ac5235a779726618e530d`
- Independent production adapter/Demarkus credentials were generated on host; only the protected OpenAI key was transferred

## Validation

- Public health: `status=ok`, `nav=true`, `distill=true`
- Conformance: 18 pass, 0 fail, 3 documented permissive-parser warnings
- Authenticated Add/Search production smoke: pass; unauthenticated Search: 401
- 16-worker capacity: 32/32 Adds and 64/64 Searches returned 200; zero schema, isolation, or visibility failures; Search p95 4.63 seconds, max 8.19 seconds
- Resource snapshot after load: 623 MiB host RAM used, 5% disk used, each container under 20 MiB
- Public TCP 80/443 reachable; adapter TCP 8080 blocked

## Backups And Retention

- Local filesystem-consistent backup and checksum verified; a restored copy booted pinned Demarkus successfully in an isolated temporary volume
- `memoryleaderboard-maintenance.timer` runs daily around 03:00 UTC with up to 15 minutes jitter
- Maintenance backs up, stops data services, executes 30-day data and local-backup retention, and guarantees service restart; brief public 503 downtime is expected
- Encrypted off-host storage with a 30-day lifecycle remains pending

## Accepted Storage Gap

Docker data currently lives on the Droplet root `ext4` filesystem. The user declined an additional encrypted DigitalOcean block volume, so the runbook's explicit encrypted-Docker-storage control is not met at the guest/storage-product layer. Resolve or formally accept this before handling non-evaluation customer data.

## Operational Notes

- Local repository path on host: `/opt/memoryleaderboard`
- Production `.env` and secrets exist only on host and are ignored by Git
- SSH cloud-firewall access is restricted to the provisioning source IP; update the firewall if that address changes
- See the public [operations runbook](https://github.com/latebit-io/memoryleaderboard/blob/main/docs/operations.md) for deploy, rotation, retention, restore, and rollback commands