# Version Retention (Keep Last N)

## Context

Hot documents accumulate versions without bound. The knowledge system's graph
document sits at 545 versions; the soul's own `index.md` is at v56 and the
roadmap at v43. Generated documents are the worst case: `mark_graph_publish`
rewrites the whole graph document on every run, and the store's duplicate check
(`prepareExistingDoc`) only no-ops on byte-identical content, so nearly every
publish mints a version. The store has no delete path today. Versions are
immutable files kept forever.

Permanence is the right default for authored knowledge (the soul's pitch is
"all versioned, all permanent"). It is the wrong default for generated
artifacts whose history has near-zero value. Retention should therefore be
targeted, not global.

## Scope

Server-side store change plus one recognized publish-metadata key. No new verb,
no protocol message changes. Clients gain a documented metadata key that
already travels through the existing PUBLISH surface, plus a confirmation
guard at the tool layer (see Accidental-set guard below).

## Why pruning is structurally safe (verified against `protocol/store`)

- Versions live as immutable files `versions/<doc>/v{N}` with a current
  symlink. `CurrentVersion()` is the max version present on disk
  (store.go:670) and the next version is current + 1 (store.go:975), so
  removing old files never disturbs numbering.
- `VerifyChain` (store.go:1197) verifies consecutive pairs among the versions
  actually present and never checks the oldest listed version's
  `previous-hash`. Removing a contiguous oldest prefix keeps the remaining
  chain verifiable. A mid-chain gap breaks it — this dictates the deletion
  order rule below.
- `Get` on a pruned version returns not-found and VERSIONS lists what remains.
  Both degrade cleanly.

## Options Considered

### A: Global server knob (`-max-versions`)

Simple, blunt. Prunes ADR and decision history with the same scythe as
generated artifacts, and a store-wide destructive default contradicts the
permanence ethos of a knowledge base. Rejected as the primary mechanism; could
return later as an optional operator ceiling if abuse shows up.

### B: Per-path retention config on the server

Deploy config maps path patterns to N. Keeps policy out of the data, but the
policy lives far from the publisher who actually knows a document is generated,
every deployment rediscovers the same rule for `/graph.md`, and it is awkward
through broker-fronted multi-world systems. Rejected.

### C: Per-document retention declared at publish — recommended

The publisher sets `retention: N` in publish metadata. It is stored in the
version frontmatter as a recognized field and the store enforces it on every
subsequent write. Generated-document publishers (`mark_graph_publish`) declare
disposable history at the source; authored documents stay permanent by default
because absent retention means keep everything.

Trust consideration: any writer with publish capability can lower retention and
destroy history. Acceptable — write capability already implies stewardship (the
same capability can archive the document). Called out in the spec.

## Plan: Option C, prune-on-write

### Mechanics

1. `retention` becomes a recognized publish metadata key: integer, minimum 1,
   validated in `validateMeta` (reject non-integer or < 1). It joins the
   recognized field set and serializes bare in frontmatter like `tags` and
   `importance`. The latest version's value is authoritative.
2. After a successful `Write` (version file created, symlink flipped), if the
   just-written version carries retention R and more than R versions exist:
   delete version files oldest-first up to the cutoff (`next − R`). Abort on
   the first deletion error and log it — aborting preserves contiguity, so
   `VerifyChain` can never observe a gap. The current version is never
   deleted; R ≥ 1 guarantees at least one version remains.
3. Backfill falls out for free: the first retained publish after upgrade prunes
   the whole backlog (545 → R) with no separate migration tool.
4. Absent or removed retention means no pruning. Default behavior is unchanged
   everywhere.
5. Legacy flat layout needs no special handling: `Write` already migrates a
   document to the per-doc layout before writing (`migrateToPerDocDir`), and
   pruning runs after the write, so prune only ever sees the per-doc layout.

Crash safety: pruning runs after the write is durable. A crash mid-prune leaves
extra old versions, which the next write retries. Deletion is idempotent.

Concurrency: a reader fetching an old version concurrently with a prune gets
not-found. Acceptable and documented. The store is already lock-free with an
`O_EXCL` write guard; prune touches only versions strictly older than the
cutoff, never the current symlink target.

### Accidental-set guard (tool layer)

The store prunes immediately on the publish that carries the key — an
accidental `retention: 1` on a valued document destroys its history with no
in-store undo. Server semantics stay simple (a two-write engage rule was
considered and deferred; see Open questions). The guard lives where humans and
agents act:

- **CLI** — `demarkus publish` with `retention` in `-meta` warns that pruning
  is destructive (showing how many versions the next write will delete, when
  cheaply known) and asks for interactive confirmation. A `-yes` flag skips
  the prompt for automation; a non-TTY invocation without `-yes` fails rather
  than silently confirming.
- **MCP tool descriptions** (local `demarkus-mcp` + broker gateway) — the
  `mark_publish` description states that `retention` permanently deletes older
  versions on this and every subsequent write, and instructs the agent to
  confirm with the user before publishing with it.
- **Plugin gate** — a PreToolUse hook on `mark_publish`/`mark_append` fires
  when metadata contains `retention`, at ask severity (same awk/bash machinery
  as the existing tag-gate and destination gate; no runtime deps). Covers the
  Claude Code path even when the agent ignores the description text.

`mark_graph_publish` is exempt from all three: it sets retention by design on
a generated document.

### Surfaces

- **store** — `validateMeta` + recognized-field serialization + a
  `pruneVersions` helper called from `Write` (and via it `WriteVersion`;
  `Append` lands through `WriteVersion` too).
- **handler** — no change expected; metadata already passes through.
- **spec** — document `retention` semantics and the chain-verification
  interaction in the store/versions section of `docs/SPEC.md`.
- **MCP** (local `demarkus-mcp` + broker gateway) — metadata objects already
  pass string values through; add `retention` to the `mark_publish` tool
  description on both surfaces (with the destructive-operation warning above).
  `mark_graph_publish` sets retention on the graph document — this is the
  concrete fix for the 545.
- **CLI** — `-meta retention=20` works once the key is recognized; publish
  command gains the confirmation prompt + `-yes` flag.
- **Plugins** — demarkus-memory (and the knowledge plugin's KS-scoped gate)
  gain the retention ask-gate; separate plugin release with the usual pin
  bump.

### Implementation steps

1. Store: recognize and validate `retention`; serialize in frontmatter; parse
   on read.
2. Store: `pruneVersions` — list versions, sort ascending, delete oldest-first
   below cutoff, stop on first error, log every failure explicitly (no silent
   swallow, per guidelines).
3. Wire into the write path after success.
4. `mark_graph_publish` sets a default retention; surface `retention` with the
   destructive-operation warning in `mark_publish` descriptions on both MCP
   surfaces.
5. CLI confirmation prompt + `-yes` flag in the publish command.
6. Plugin ask-gate on retention in publish/append metadata (separate PR, its
   own version bump).
7. Spec + docs.
8. Tests, `bash pre-commit.sh`.

### Files to modify

- `protocol/store/store.go` + `store_test.go`
- `docs/SPEC.md`
- `client/cmd/demarkus-mcp/main.go` (graph publish + tool description)
- `client/cmd/demarkus` publish command (confirmation prompt + `-yes`)
- `tools/demarkus-broker/internal/broker/mcp_tools_write.go`,
  `mcp_tools_graph.go` (description parity)
- `plugins/claude-code/` hooks (retention ask-gate; separate PR)

### Verification

1. Keep-N on write: count and lowest remaining version correct.
2. `VerifyChain` passes after prune (contiguous suffix).
3. Never deletes the current version; R = 1 keeps exactly the current version.
4. Retention raised, lowered, or removed between writes behaves correctly.
5. Injected deletion failure aborts without creating a gap.
6. Backlog case: many existing versions, first retained write prunes them all.
7. Flat-layout document migrates then prunes.
8. `Get` of a pruned version returns not-found; VERSIONS lists the remainder.
9. CLI: prompt shown when retention present; `-yes` skips; non-TTY without
   `-yes` fails.
10. Plugin: gate fires on retention in metadata, stays silent otherwise
    (shell tests alongside the existing gate tests).
11. Manual smoke: repeated `mark_graph_publish` with retention set, watch the
    versions directory stay bounded.

### PR slicing

- **PR 1** — store + spec + MCP descriptions + graph-publish default + CLI
  confirmation. Nothing ships half-done (conventions: never ship anything
  broken).
- **PR 2** — plugin retention ask-gate + pin bump (plugin changes ride their
  own release train).

Knowledge-system rollout is the normal release train: server release, broker
repin, deploy repo bump; the 545 clears on the next graph publish after
upgrade.

## Open questions

- Default retention for `mark_graph_publish` — propose 20. Enough to debug a
  bad crawl, small enough to stay bounded. Alternatively make it a tool
  parameter with a default.
- Should journals or other append-heavy soul docs adopt retention? Probably
  not — they are authored history. Decide per document, never globally.
- Two-write engage rule (retention only takes effect when the previous version
  also carried it) — considered as a server-side accident guard, deferred to
  keep store semantics simple; the tool-layer confirmation covers the accident
  path. Revisit if an accidental prune actually happens despite the guards.
- Operator ceiling (Option A as a supplement) only if a hostile or buggy
  writer becomes a real problem.
- Recovery story: pruned versions are gone from the store, but the GKE deploy
  keeps CSI volume snapshots, so catastrophic mistakes have a coarse undo.

---

## Status: PR 1 implemented (2026-07-06, branch `feat/version-retention`)

Everything in PR 1 built and verified the same day the plan was written:
store (`retention` validation, `pruneVersions`, `PruneResult` on `Document`),
handler prune audit logging with token_label, MCP descriptions + graph-publish
default (retention tool param, default 20, 0 disables) on both surfaces, CLI
confirmation (`-yes`, non-TTY fail-closed), SPEC §9.9. Verified live over QUIC:
backlog prune, audit line, not-found on pruned reads, chain intact.

Discovered during implementation: `migrateFlatFile` checked for v1
specifically to detect flat files, so pruning v1 made the next write resurrect
a bogus v1 from the current symlink and break the hash chain. Fixed in the
same change (detect any per-doc version history); see journal 2026-07-06.

Remaining: PR 2 (plugin retention ask-gate + pin bump), knowledge-system
rollout via the release train.

**MERGED: PR #236 (`db9beae`, 2026-07-06)** — same day as the plan. Review
rounds added: CLI confirmation gated to PUBLISH/APPEND only, confirmation
skipped for server-rejectable values (0/negative/non-numeric — they cannot
prune), `store.ParseRetention` as the single shared predicate (protocol/store,
not client/internal: the broker can't import client internals), APPEND handler
retention tests, SPEC §9.6 verification steps rewritten around the oldest
retained version as verification root, and the os.Root delete-path hardening.
Remaining: PR 2 (plugin retention ask-gate + pin bump), then the release train
(server release → broker repin → deploy bump); the graph doc's 545 versions
clear on the first `mark_graph_publish` after the world servers upgrade.