2026-05-27

Tokens-file hot-reload via parent-directory watch

Investigated why world-a wasn't picking up broker-minted token rotations: the broker writes tokens to a k8s Secret, kubelet atomically retargets the ..data symlink under the mount path, and the demarkus-server only refreshes its in-memory TokenStore on SIGHUP — no signal, no reload. The architecture doc's "Hot-reloadable via SIGHUP" claim was accurate but insufficient for the k8s-mount case where there is no operator to send SIGHUP.

The existing wiring turned out to be already perfectly shaped for hot-reload:

• currentTokenStore is a package-level *auth.TokenStore behind tokenMu sync.RWMutex in server/cmd/demarkus-server/main.go.
• loadTokenStore(path) re-reads the TOML and atomically swaps the pointer under the write lock.
• Handler.GetTokenStore is a callback (func() *auth.TokenStore), not a captured pointer. All handler sites read the store through it — read-side hot-reload was already free.
• reload_unix.go already calls loadTokenStore on SIGHUP.

The only missing piece was something to call loadTokenStore on file change. Added a generic file watcher at server/internal/configwatch/:

• Watches filepath.Dir(target) rather than the target itself — the canonical pattern for tolerating atomic rename swaps and symlink retargets (the inode under a stable path changes, so a watch on the leaf would silently never fire after the first swap).
• Debounces events through a single time.Timer (default 150ms) so a burst of write/rename/create events from one logical update coalesces into a single reload.
• Retries once on transient os.ErrNotExist from the reload callback (75ms delay) to cover the brief window when a directory swap can leave the path momentarily unresolvable before the new contents are linked in.
• Generic by design — no k8s-specific naming, no ..data strings, no Secret-mount assumptions. The watcher just reacts to "something in the directory changed" and lets the reload callback re-resolve the path through whatever symlink chain happens to exist. The same pattern handles vim atomic writes, cp + mv rotation, helm rewrites, kubelet symlink swaps, and CSI direct-write injectors with identical code.
• Tests cover in-place writes, atomic rename, symlink retarget (constructed exactly like the k8s ..data indirection: a current → vN symlink that's atomically swapped via Rename of a staged symlink), debounce coalescing, ctx-cancel shutdown, and the ENOENT retry path.

Wired into main.go via startTokenFileWatcher(cfg.TokensFile, logger) right after the initial loadTokenStore call. Reload callback is just func() error { return loadTokenStore(path) }, so SIGHUP and file-change route through the exact same atomic-swap function. Watcher is unconditional and platform-agnostic (fsnotify supports Linux/macOS/Windows), so this also gives Windows servers a hot-reload path they didn't have before.

Net effect: the broker's firstMintMaxAttempts: 6 retry budget (~16s) now matches reality. The world picks up freshly minted tokens within one debounce window of the kubelet swap.

Out of scope for this change (deliberate): TLS cert file-watch. Same staleness problem applies (cert rotation through a Secret mount), and the generic watcher would handle it identically, but that's a separate change and not what's currently broken. Leaving the watcher generic means a one-line wire-up will cover it later.

Architecture doc updated to v12 to reflect that token reload happens via SIGHUP or file change, both routing through loadTokenStore.