Versions Directory Sharding

Context

All version files for documents in a given directory share a single flat versions/ subdirectory. A frequently-edited document or a directory with many documents accumulates thousands of files in one directory. findVersions() calls os.ReadDir() on the entire directory and filters by prefix, making it O(all entries) instead of O(document versions). This is the hot path — called on every Get, CurrentVersion, Write, and Versions request.

Problem Dimensions

Two axes of growth compound in the flat layout:

1. Many documents in one directory — 1000 docs x 50 versions = 50K entries
2. Many versions of one document — 1 doc with 10K versions = 10K entries

Options Considered

A: Per-Document Subdirectory

versions/
  doc.md/
    v1
    v2
    v3
  other.md/
    v1
    v2

Symlink: versions/doc.md/v3

Pros: Eliminates cross-product, findVersions scoped to one doc, simple naming. Cons: Extra directory per document, migration needed.

B: Name-Prefix Sharding

versions/
  do/
    doc.md.v1
    doc.md.v2
  ot/
    other.md.v1

Pros: Spreads entries across shard buckets. Cons: Uneven distribution, does NOT help many-versions-of-one-doc, findVersions still scans shard dir.

C: Version-Range Bucketing

versions/
  doc.md/
    v0001-1000/
      v1 ... v1000
    v1001-2000/
      v1001 ...

Pros: Caps leaf directories at fixed size. Cons: Overkill for typical use (< 100 versions), findVersions must merge across buckets, cross-bucket complexity in chain verification.

Recommended: Option A — Per-Document Subdirectory with Lazy Migration

New Layout

root/
  doc.md              → versions/doc.md/v3
  versions/
    doc.md/
      v1
      v2
      v3

Why This Option

1. Solves both scaling axes. Each document's versions are isolated. The versions/ top-level contains only subdirectories (one per document).
2. findVersions becomes O(versions) instead of O(all entries in directory). This is the biggest win since it is the hot path.
3. Minimal code surface. All 12 path construction sites in store.go follow the same pattern. Two helper functions replace all of them.
4. Zero-downtime migration. The lazy migration pattern already exists (migrateFlatFile at line 1087). Extend it for flat-to-subdir migration.
5. Symlinks stay simple. versions/doc.md.v3 becomes versions/doc.md/v3 — still relative, one level deeper.

Implementation

Step 1: Extract path helpers (pure refactor)

Add two functions in store.go:

// versionFilePath returns the path to a version file on disk.
func versionFilePath(versionsDir, base string, version int) string {
    return filepath.Join(versionsDir, base, fmt.Sprintf("v%d", version))
}

// versionSymlinkTarget returns the relative symlink target for a version.
func versionSymlinkTarget(base string, version int) string {
    return filepath.Join("versions", base, fmt.Sprintf("v%d", version))
}

Update all 12 call sites to use these helpers. Initially they produce the OLD layout paths — this is a no-behavior-change refactor that can be tested independently.

Step 2: Add layout detection

// isPerDocLayout returns true if the document uses per-document subdirectory layout.
func isPerDocLayout(versionsDir, base string) bool {
    info, err := os.Stat(filepath.Join(versionsDir, base))
    return err == nil && info.IsDir()
}

Step 3: Update helpers to use new layout

• Writes always create the per-document subdirectory and use new layout
• Reads detect which layout exists and handle both

Step 4: Update findVersions

New layout: os.ReadDir(filepath.Join(versionsDir, base)) — reads only that document's versions, parses v{N} names. Old layout: falls back to current scan-and-filter logic.

Step 5: Add migrateToPerDocDir

Modeled on existing migrateFlatFile (line 1087). Called from Write before creating new version when old layout is detected:

1. Create versions/{base}/ directory
2. Move all {base}.v{N} files into versions/{base}/v{N}
3. Update symlink atomically

Step 6: Update BuildHashIndex

Currently skips versions/ via filepath.SkipDir. With nested subdirectories inside versions/, this already works — SkipDir skips the entire subtree.

Files to Modify

• server/internal/store/store.go — all changes (path helpers, layout detection, migration, findVersions)
• server/internal/store/store_test.go — tests for both layouts, migration, findVersions performance

Verification

1. Run existing tests — they must all pass with no layout change (Step 1 refactor)
2. Add tests for new layout: write, read, findVersions, getVersion, verifyChain, archive
3. Add test for migration: create documents in old layout, trigger write, verify files moved to new layout
4. Add test for mixed layout: old and new layout documents coexist in same directory
5. Run bash pre-commit.sh
6. Manual smoke test: start server with existing content directory, publish a document, verify versions are in per-doc subdirectory