2026-05-27

Tokens-file hot-reload via parent-directory watch

Investigated why world-a wasn't picking up broker-minted token rotations: the broker writes tokens to a k8s Secret, kubelet atomically retargets the ..data symlink under the mount path, and the demarkus-server only refreshes its in-memory TokenStore on SIGHUP — no signal, no reload. The architecture doc's "Hot-reloadable via SIGHUP" claim was accurate but insufficient for the k8s-mount case where there is no operator to send SIGHUP.

The existing wiring turned out to be already perfectly shaped for hot-reload:

currentTokenStore is a package-level *auth.TokenStore behind tokenMu sync.RWMutex in server/cmd/demarkus-server/main.go.
loadTokenStore(path) re-reads the TOML and atomically swaps the pointer under the write lock.
Handler.GetTokenStore is a callback (func() *auth.TokenStore), not a captured pointer. All handler sites read the store through it — read-side hot-reload was already free.
reload_unix.go already calls loadTokenStore on SIGHUP.

The only missing piece was something to call loadTokenStore on file change. Added a generic file watcher at server/internal/configwatch/:

Watches filepath.Dir(target) rather than the target itself — the canonical pattern for tolerating atomic rename swaps and symlink retargets (the inode under a stable path changes, so a watch on the leaf would silently never fire after the first swap).
Debounces events through a single time.Timer (default 150ms) so a burst of write/rename/create events from one logical update coalesces into a single reload.
Retries once on transient os.ErrNotExist from the reload callback (75ms delay) to cover the brief window when a directory swap can leave the path momentarily unresolvable before the new contents are linked in.
Generic by design — no k8s-specific naming, no ..data strings, no Secret-mount assumptions. The watcher just reacts to "something in the directory changed" and lets the reload callback re-resolve the path through whatever symlink chain happens to exist. The same pattern handles vim atomic writes, cp + mv rotation, helm rewrites, kubelet symlink swaps, and CSI direct-write injectors with identical code.
Tests cover in-place writes, atomic rename, symlink retarget (constructed exactly like the k8s ..data indirection: a current → vN symlink that's atomically swapped via Rename of a staged symlink), debounce coalescing, ctx-cancel shutdown, and the ENOENT retry path.

Wired into main.go via startTokenFileWatcher(cfg.TokensFile, logger) right after the initial loadTokenStore call. Reload callback is just func() error { return loadTokenStore(path) }, so SIGHUP and file-change route through the exact same atomic-swap function. Watcher is unconditional and platform-agnostic (fsnotify supports Linux/macOS/Windows), so this also gives Windows servers a hot-reload path they didn't have before.

Net effect: the broker's firstMintMaxAttempts: 6 retry budget (~16s) now matches reality. The world picks up freshly minted tokens within one debounce window of the kubelet swap.

Out of scope for this change (deliberate): TLS cert file-watch. Same staleness problem applies (cert rotation through a Secret mount), and the generic watcher would handle it identically, but that's a separate change and not what's currently broken. Leaving the watcher generic means a one-line wire-up will cover it later.

Architecture doc updated to v12 to reflect that token reload happens via SIGHUP or file change, both routing through loadTokenStore.

Broker stable-mint — Slices 1 & 2 implemented

Plan published at /plans/broker-stable-mint.md v2 (the v1 design assumed we could "return the same raw token on next join"; v2 simplified to "don't return tokens at join time at all" once the issuer's structural constraint surfaced — raw tokens are never recorded server-side by design).

Slice 1 — Lazy provisioning at join

/me/install (install.go) and /auth/callback bare-code branch (server.go:386-434) no longer call Issuer.Mint*. They return identity + the list of worlds the caller is authorized for, no per-world credential material. Response shape:

{
  "email": "alice@example.com",
  "worlds": [{"name": "team-a", "publicURL": "mark://..."}]
}

installWorld.Label, AccessToken, ExpiresAt and installResponse.PartialFailure all removed. Confirmed with Fritz that nothing consumes these on the knowledge-system flow — direct-QUIC soul flow uses a separate ~/.mark/tokens.toml mechanism that is untouched.

Minting moves to the place it had to happen anyway — the MCP gateway's sessionCache miss in dispatchWithAuth. Kubelet-lag is now hit at most once per (broker-pod, user, world) triple, on the first dispatch after a cold cache, rather than on every join.

Tests deleted (no longer have semantics under lazy provisioning): TestMeInstallPartialFailureReturns200WithFlag, TestMeInstallHardFailureReturns500. Tests updated: TestMeInstallHappyPathSingleWorld, TestMeInstallFiltersWorldsWithoutPublicURL, TestMeInstallEmptyWorldsReturns200EmptySlice, TestAuthCallbackSuccess, TestAuthCallbackUnchangedWithoutDeviceCookie. New assertions verify no Secret writes happen during /me/install.

Slice 2 — Cache-stable retries

The field bug was: dispatchWithAuth invalidated the sessionCache on every 401 (including fresh-mint 401s), so each backoff tick triggered a new Issuer.MintFiltered round-trip, which wrote a new hash to the world's tokens.toml Secret. Under realistic GKE Secret-sync (~30-40s), the broker burned its full retry budget on a cascade of doomed tokens — Fritz observed ~20 tokens written in one 78-second session, with the world reloading only twice in that window.

Fix structure in mcp_tools_read.go:

New local flag mintedInDispatch flips true the moment GetOrMint reports isFresh=true (or coalesced behind a singleflight peer that did).
!mintedInDispatch && !isFresh (initial cache hit on a stale token) is the only branch that calls sessionCache.Invalidate and re-mints — that path is for genuine rotation races (operator hand-revoke / sweeper retire / Secret restore).
Once mintedInDispatch == true, the cache is preserved across the entire propagation-race retry loop. The same raw token is presented to the world on every backoff tick. The world catches up exactly once when kubelet projects the Secret — no junk hashes left behind.

FirstMintMaxAttempts is now a propagation-budget knob (how long to wait for kubelet) rather than a doomed-token cascade limit. Slice 3 (shrink the default) is deferred until we observe the new behavior in the field.

New regression test: TestDispatchReadPropagationRetriesReuseSameToken — asserts all dispatcher calls in a 2x-401-then-OK scenario receive the SAME token string. Existing four tests (TestDispatchReadRetriesOnFreshMint401, TestDispatchReadExhaustsRetriesOnPersistent401, TestDispatchReadCacheHit401InvalidatesAndReMints, TestDispatchReadCacheHit401DoesNotConsumeFreshMintBudget) all still pass under the new logic — the cache-hit vs fresh-mint separation is preserved.

Verification

All broker tests pass.
All four modules (protocol, server, client, tools) pass go test ./... and pre-commit.sh (fmt + vet + lint clean).

Composed effect with PR #158

Once PR #158 (fsnotify watcher on the server side) and these two slices all land:

Server-side reload floor goes from "never without SIGHUP" → "one kubelet sync cycle" (PR #158).
Broker stops the per-join token churn (Slice 1).
Broker stops the per-retry token cascade during the kubelet wait (Slice 2).

Net effect: a healthy first-dispatch for a brand-new (user, world) pair takes roughly one kubelet projection cycle (~30-40s on GKE default) and writes exactly one hash to the world's Secret. Returning users hit the cache on subsequent dispatches and never touch kubelet. The user-visible "propagation lag exceeded broker deadline" error becomes a signal of an actual outage, not a routine event.