Version Retention (Keep Last N)

Context

Hot documents accumulate versions without bound. The knowledge system's graph document sits at 545 versions; the soul's own index.md is at v56 and the roadmap at v43. Generated documents are the worst case: mark_graph_publish rewrites the whole graph document on every run, and the store's duplicate check (prepareExistingDoc) only no-ops on byte-identical content, so nearly every publish mints a version. The store has no delete path today. Versions are immutable files kept forever.

Permanence is the right default for authored knowledge (the soul's pitch is "all versioned, all permanent"). It is the wrong default for generated artifacts whose history has near-zero value. Retention should therefore be targeted, not global.

Scope

Server-side store change plus one recognized publish-metadata key. No new verb, no protocol message changes. Clients gain a documented metadata key that already travels through the existing PUBLISH surface, plus a confirmation guard at the tool layer (see Accidental-set guard below).

Why pruning is structurally safe (verified against protocol/store)

• Versions live as immutable files versions/<doc>/v{N} with a current symlink. CurrentVersion() is the max version present on disk (store.go:670) and the next version is current + 1 (store.go:975), so removing old files never disturbs numbering.
• VerifyChain (store.go:1197) verifies consecutive pairs among the versions actually present and never checks the oldest listed version's previous-hash. Removing a contiguous oldest prefix keeps the remaining chain verifiable. A mid-chain gap breaks it — this dictates the deletion order rule below.
• Get on a pruned version returns not-found and VERSIONS lists what remains. Both degrade cleanly.

Options Considered

A: Global server knob (-max-versions)

Simple, blunt. Prunes ADR and decision history with the same scythe as generated artifacts, and a store-wide destructive default contradicts the permanence ethos of a knowledge base. Rejected as the primary mechanism; could return later as an optional operator ceiling if abuse shows up.

B: Per-path retention config on the server

Deploy config maps path patterns to N. Keeps policy out of the data, but the policy lives far from the publisher who actually knows a document is generated, every deployment rediscovers the same rule for /graph.md, and it is awkward through broker-fronted multi-world systems. Rejected.

C: Per-document retention declared at publish — recommended

The publisher sets retention: N in publish metadata. It is stored in the version frontmatter as a recognized field and the store enforces it on every subsequent write. Generated-document publishers (mark_graph_publish) declare disposable history at the source; authored documents stay permanent by default because absent retention means keep everything.

Trust consideration: any writer with publish capability can lower retention and destroy history. Acceptable — write capability already implies stewardship (the same capability can archive the document). Called out in the spec.

Plan: Option C, prune-on-write

Mechanics

1. retention becomes a recognized publish metadata key: integer, minimum 1, validated in validateMeta (reject non-integer or < 1). It joins the recognized field set and serializes bare in frontmatter like tags and importance. The latest version's value is authoritative.
2. After a successful Write (version file created, symlink flipped), if the just-written version carries retention R and more than R versions exist: delete version files oldest-first up to the cutoff (next − R). Abort on the first deletion error and log it — aborting preserves contiguity, so VerifyChain can never observe a gap. The current version is never deleted; R ≥ 1 guarantees at least one version remains.
3. Backfill falls out for free: the first retained publish after upgrade prunes the whole backlog (545 → R) with no separate migration tool.
4. Absent or removed retention means no pruning. Default behavior is unchanged everywhere.
5. Legacy flat layout needs no special handling: Write already migrates a document to the per-doc layout before writing (migrateToPerDocDir), and pruning runs after the write, so prune only ever sees the per-doc layout.

Crash safety: pruning runs after the write is durable. A crash mid-prune leaves extra old versions, which the next write retries. Deletion is idempotent.

Concurrency: a reader fetching an old version concurrently with a prune gets not-found. Acceptable and documented. The store is already lock-free with an O_EXCL write guard; prune touches only versions strictly older than the cutoff, never the current symlink target.

Accidental-set guard (tool layer)

The store prunes immediately on the publish that carries the key — an accidental retention: 1 on a valued document destroys its history with no in-store undo. Server semantics stay simple (a two-write engage rule was considered and deferred; see Open questions). The guard lives where humans and agents act:

• CLI — demarkus publish with retention in -meta warns that pruning is destructive (showing how many versions the next write will delete, when cheaply known) and asks for interactive confirmation. A -yes flag skips the prompt for automation; a non-TTY invocation without -yes fails rather than silently confirming.
• MCP tool descriptions (local demarkus-mcp + broker gateway) — the mark_publish description states that retention permanently deletes older versions on this and every subsequent write, and instructs the agent to confirm with the user before publishing with it.
• Plugin gate — a PreToolUse hook on mark_publish/mark_append fires when metadata contains retention, at ask severity (same awk/bash machinery as the existing tag-gate and destination gate; no runtime deps). Covers the Claude Code path even when the agent ignores the description text.

mark_graph_publish is exempt from all three: it sets retention by design on a generated document.

Surfaces

• store — validateMeta + recognized-field serialization + a pruneVersions helper called from Write (and via it WriteVersion; Append lands through WriteVersion too).
• handler — no change expected; metadata already passes through.
• spec — document retention semantics and the chain-verification interaction in the store/versions section of docs/SPEC.md.
• MCP (local demarkus-mcp + broker gateway) — metadata objects already pass string values through; add retention to the mark_publish tool description on both surfaces (with the destructive-operation warning above). mark_graph_publish sets retention on the graph document — this is the concrete fix for the 545.
• CLI — -meta retention=20 works once the key is recognized; publish command gains the confirmation prompt + -yes flag.
• Plugins — demarkus-memory (and the knowledge plugin's KS-scoped gate) gain the retention ask-gate; separate plugin release with the usual pin bump.

Implementation steps

1. Store: recognize and validate retention; serialize in frontmatter; parse on read.
2. Store: pruneVersions — list versions, sort ascending, delete oldest-first below cutoff, stop on first error, log every failure explicitly (no silent swallow, per guidelines).
3. Wire into the write path after success.
4. mark_graph_publish sets a default retention; surface retention with the destructive-operation warning in mark_publish descriptions on both MCP surfaces.
5. CLI confirmation prompt + -yes flag in the publish command.
6. Plugin ask-gate on retention in publish/append metadata (separate PR, its own version bump).
7. Spec + docs.
8. Tests, bash pre-commit.sh.

Files to modify

• protocol/store/store.go + store_test.go
• docs/SPEC.md
• client/cmd/demarkus-mcp/main.go (graph publish + tool description)
• client/cmd/demarkus publish command (confirmation prompt + -yes)
• tools/demarkus-broker/internal/broker/mcp_tools_write.go, mcp_tools_graph.go (description parity)
• plugins/claude-code/ hooks (retention ask-gate; separate PR)

Verification

1. Keep-N on write: count and lowest remaining version correct.
2. VerifyChain passes after prune (contiguous suffix).
3. Never deletes the current version; R = 1 keeps exactly the current version.
4. Retention raised, lowered, or removed between writes behaves correctly.
5. Injected deletion failure aborts without creating a gap.
6. Backlog case: many existing versions, first retained write prunes them all.
7. Flat-layout document migrates then prunes.
8. Get of a pruned version returns not-found; VERSIONS lists the remainder.
9. CLI: prompt shown when retention present; -yes skips; non-TTY without -yes fails.
10. Plugin: gate fires on retention in metadata, stays silent otherwise (shell tests alongside the existing gate tests).
11. Manual smoke: repeated mark_graph_publish with retention set, watch the versions directory stay bounded.

PR slicing

• PR 1 — store + spec + MCP descriptions + graph-publish default + CLI confirmation. Nothing ships half-done (conventions: never ship anything broken).
• PR 2 — plugin retention ask-gate + pin bump (plugin changes ride their own release train).

Knowledge-system rollout is the normal release train: server release, broker repin, deploy repo bump; the 545 clears on the next graph publish after upgrade.

Open questions

• Default retention for mark_graph_publish — propose 20. Enough to debug a bad crawl, small enough to stay bounded. Alternatively make it a tool parameter with a default.
• Should journals or other append-heavy soul docs adopt retention? Probably not — they are authored history. Decide per document, never globally.
• Two-write engage rule (retention only takes effect when the previous version also carried it) — considered as a server-side accident guard, deferred to keep store semantics simple; the tool-layer confirmation covers the accident path. Revisit if an accidental prune actually happens despite the guards.
• Operator ceiling (Option A as a supplement) only if a hostile or buggy writer becomes a real problem.
• Recovery story: pruned versions are gone from the store, but the GKE deploy keeps CSI volume snapshots, so catastrophic mistakes have a coarse undo.

Status: PR 1 implemented (2026-07-06, branch feat/version-retention)

Everything in PR 1 built and verified the same day the plan was written: store (retention validation, pruneVersions, PruneResult on Document), handler prune audit logging with token_label, MCP descriptions + graph-publish default (retention tool param, default 20, 0 disables) on both surfaces, CLI confirmation (-yes, non-TTY fail-closed), SPEC §9.9. Verified live over QUIC: backlog prune, audit line, not-found on pruned reads, chain intact.

Discovered during implementation: migrateFlatFile checked for v1 specifically to detect flat files, so pruning v1 made the next write resurrect a bogus v1 from the current symlink and break the hash chain. Fixed in the same change (detect any per-doc version history); see journal 2026-07-06.

Remaining: PR 2 (plugin retention ask-gate + pin bump), knowledge-system rollout via the release train.

MERGED: PR #236 (db9beae, 2026-07-06) — same day as the plan. Review rounds added: CLI confirmation gated to PUBLISH/APPEND only, confirmation skipped for server-rejectable values (0/negative/non-numeric — they cannot prune), store.ParseRetention as the single shared predicate (protocol/store, not client/internal: the broker can't import client internals), APPEND handler retention tests, SPEC §9.6 verification steps rewritten around the oldest retained version as verification root, and the os.Root delete-path hardening. Remaining: PR 2 (plugin retention ask-gate + pin bump), then the release train (server release → broker repin → deploy bump); the graph doc's 545 versions clear on the first mark_graph_publish after the world servers upgrade.

COMPLETE — verified in production (2026-07-07). Full chain shipped in two days: #236 store core → #237 gate binary → #239 plugin guidance + repins → #240 agent retention → deploy repo rollout (worlds server 0.20.0, broker 0.9.0, agent 0.19.0). Production verification on knowledge.demarkus.io: /graph.md pruned 556 → 20 versions and the four hub hash indexes pruned their backlogs in the same crawl (~1,714 version files deleted, all audit-logged with token label, hash chains valid). PR 2's plugin ask-gate ships to plugin users via the tools/v0.8.0+ pins. Open questions resolved in practice: graph default 20 (tool param + agent config); journals stay unlimited; no operator ceiling needed so far.