# Journal — 2026-03-01 ## 2026-03-01 — Hardening APPEND Through Code Review The APPEND implementation from earlier today went through a thorough Copilot-driven code review — roughly 20 suggestions, each one a real edge case or a missing guard. The session evolved from fixing individual issues to a fundamental simplification of the design. ### The Spiral The original APPEND had two paths: with `expected-version` (optimistic concurrency) and without (last-write-wins with internal retry). Copilot kept finding issues with the no-version path — TOCTOU races in `AppendVersion`, retry loops that could duplicate content, post-write conflict detection that was unreliable. Each fix introduced a new edge case. The complexity was spiraling. Fritz asked the right question: "do we need to have a no conflict option?" The answer was no. APPEND is non-idempotent — appending the same content twice produces a different result. Without a version check, retries after network timeouts could silently duplicate content. Making `expected-version` mandatory eliminates the entire class of problems. ### What Changed - **Removed `store.Append` (no-version path)** and the old `AppendVersion`. Replaced with a single `Append(reqPath, expectedVersion, content)` that requires `expectedVersion >= 1`. - **Handler rejects missing `expected-version`** with `bad-request` status. No more falling through to last-write-wins. - **Client validates early** — `Client.Append` rejects `expectedVersion < 1` and empty body before making a network call. MCP tool does the same. - **Store validates at entry** — guards against `expectedVersion < 1` and empty content before any file I/O. - **Extracted `resolveAuthToken` and `resolveBody`** from `requestMain()` to fix cyclomatic complexity (was 26, limit 25). - **SPEC and DESIGN updated** — examples now show `expected-version`, DESIGN.md explains why it's mandatory for APPEND specifically. ### Other Fixes from Review - `ErrSizeLimit` sentinel error (was falling through to generic "internal error") - `joinContent` helper with size-before-allocate and trailing newline handling - Path traversal check moved to `HandleStream` level (before routing to any handler) - `containsDotDot` guard added to `Store.Append` - SPEC section 6.4 now documents `expected-version` semantics for PUBLISH too ### What I Learned The best simplification came from stepping back and asking whether a feature was needed at all. The no-version path for APPEND seemed like a nice convenience, but it was fundamentally at odds with the operation's semantics. Non-idempotent + retry = duplication risk. No amount of clever retry logic fixes that — you need the caller to participate in conflict resolution. Defense in depth works: the same validation (`expectedVersion >= 1`, non-empty body) now exists at four layers — MCP tool, CLI, client library, store. Each layer catches misuse before it propagates further. Redundant? Yes. But each layer has different callers, and the cost of a guard clause is near zero. ### Note to Future Sessions APPEND is now self-consistent: mandatory `expected-version`, validated at every layer, documented in spec and design. The flow is always fetch → append → handle conflict if needed. No shortcuts, no surprise duplications. --- ## 2026-03-01 — APPEND Verb, End to End Fritz found the use case that was missing for APPEND: journal entries and thoughts. When appending to demarkus-soul pages, the current workflow is fetch → concatenate → republish the entire document. Wasteful for adding a few lines to a growing journal. APPEND sends only the new content; the server handles concatenation. What changed: - **`protocol/protocol.go`**: Added `VerbAppend = "APPEND"`. - **`protocol/request.go`**: Added `VerbAppend` to `isValidVerb`. - **`server/internal/store/store.go`**: New `Append` and `AppendVersion` methods. `Append` reads the existing document, strips store-managed frontmatter via `extractBody`, concatenates with `\n`, checks combined size against `protocol.MaxBodyLength`, and writes a new version. `AppendVersion` wraps it with optimistic concurrency (same pattern as `WriteVersion`). - **`server/internal/handler/handler.go`**: New `handleAppend` method, routed from the verb switch. Mirrors `handlePublish` — store check, body size check, empty body rejection, auth check (`publish` capability), expected-version parsing, error handling for conflict/archived/not-found. - **`client/internal/fetch/fetch.go`**: New `Append` method, same pattern as `Publish`. - **`client/cmd/demarkus/main.go`**: CLI support — APPEND verb reads body from stdin. - **`client/cmd/demarkus-mcp/main.go`**: `markAppendTool` with url, body, expected_version params. Registered with `s.AddTool`. - **Tests**: `TestHandleAppend` (6 subtests), `TestAppend`, `TestAppend_NotFound`, `TestAppend_ExceedsMaxBody`, `TestAppendVersion`. - **Docs**: SPEC.md section 6.6, DESIGN.md updated (APPEND no longer deferred). ### Design Decisions - **Reuses `publish` auth capability** — no new capability needed. If you can publish to a path, you can append to it. - **Document must exist** — unlike PUBLISH which can create. APPEND to a non-existent path returns `not-found`. - **Archived documents rejected** — returns `archived`, same as PUBLISH. - **Empty body rejected** — returns `server-error`. No-op appends are meaningless. - **Combined size checked** — existing body + `\n` + new content must not exceed `MaxBodyLength`. Prevents documents from growing unbounded. - **Creates a new immutable version** — same versioning, hash chain, and audit trail as PUBLISH. ### What I Learned APPEND is a natural fit for the protocol. The verb set now covers the full document lifecycle: create (PUBLISH), read (FETCH), update (PUBLISH), append (APPEND), soft-delete (ARCHIVE), list (LIST), history (VERSIONS). The key insight is that append-only patterns (journals, logs, running notes) are common enough in both human and agent workflows to justify a dedicated verb rather than forcing fetch-concat-republish. The implementation was clean because all the infrastructure was already there — auth, versioning, hash chains, optimistic concurrency. APPEND is just a different way to produce the content that gets versioned. ### Note to Future Sessions APPEND is now available in the MCP server as `mark_append`. Use it for journal entries and thoughts instead of fetch-concat-republish. The `expected_version` parameter works the same as PUBLISH for conflict detection. --- ## 2026-03-01 — Tightening the Size Limits Fritz wanted to reduce the maximum document size from 10MB to 1MB. The largest markdown file in the entire project is 39KB, so 10MB was absurdly generous. 1MB still gives 25x headroom over the largest real document — plenty for any legitimate markdown use case. What changed: - **`protocol/request.go`**: Added `MaxBodyLength = 1 * 1024 * 1024` as the single protocol-level constant. Sits alongside `MaxRequestLineLength` and `MaxRequestFrontmatterLength` — all three size limits now live in one place. - **`ParseRequest`**: Previously read the entire request body with `io.ReadAll` — unbounded allocation, DoS risk. Now uses `io.LimitReader` capped at `MaxRequestFrontmatterLength + MaxBodyLength + 64` (delimiters). Body size is also checked independently on both code paths (with and without frontmatter). - **`server/internal/store/store.go`**: Removed the old `MaxFileSize` constant. On-disk size checks now account for store-managed frontmatter overhead (`maxStoreFrontmatter = 256`), so a document exactly at the body limit doesn't get rejected because of the `---\nversion: N\n...---\n` prefix. - **`server/internal/handler/handler.go`**: Uses `protocol.MaxBodyLength` directly — no more referencing the store constant. - **Tests**: Added `TestParseRequestPayloadTooLarge` covering both body paths. Updated `store_test.go` to reference the protocol constant. - **Docs**: SPEC.md, DESIGN.md, and site/server/index.md all updated to 1 MB. ### What I Learned Size limits need to be enforced at the parsing layer, not just downstream. The original code read the entire stream into memory before any check ran — an attacker could trigger unbounded allocation with an arbitrarily large request. The `LimitReader` fix is defense in depth: cap the read, then validate the parts. The store frontmatter overhead was a subtle bug. `MaxBodyLength` is a *body* limit, but the store prepends its own frontmatter (version, archived flag, hash chain). Checking the on-disk file size against the body limit would reject valid documents near the cap. Separating the concerns — body limit for protocol, stored size limit for disk — keeps the invariants clean. ### Note to Future Sessions Three layers of size enforcement now: 1. `ParseRequest` — caps total read with `LimitReader`, checks body size on both paths 2. `handler` — checks body before passing to store 3. `store` — checks raw content on write, checks on-disk size (with frontmatter allowance) on read All reference `protocol.MaxBodyLength`. Change the constant, change the limit everywhere.