# 2026-05-27

## Tokens-file hot-reload via parent-directory watch

Investigated why world-a wasn't picking up broker-minted token rotations: the broker writes tokens to a k8s Secret, kubelet atomically retargets the `..data` symlink under the mount path, and the demarkus-server only refreshes its in-memory `TokenStore` on SIGHUP — no signal, no reload. The architecture doc's "Hot-reloadable via SIGHUP" claim was accurate but insufficient for the k8s-mount case where there is no operator to send SIGHUP.

The existing wiring turned out to be already perfectly shaped for hot-reload:

- `currentTokenStore` is a package-level `*auth.TokenStore` behind `tokenMu sync.RWMutex` in `server/cmd/demarkus-server/main.go`.
- `loadTokenStore(path)` re-reads the TOML and atomically swaps the pointer under the write lock.
- `Handler.GetTokenStore` is a callback (`func() *auth.TokenStore`), not a captured pointer. All handler sites read the store through it — read-side hot-reload was already free.
- `reload_unix.go` already calls `loadTokenStore` on SIGHUP.

The only missing piece was something to call `loadTokenStore` on file change. Added a generic file watcher at `server/internal/configwatch/`:

- Watches `filepath.Dir(target)` rather than the target itself — the canonical pattern for tolerating atomic rename swaps and symlink retargets (the inode under a stable path changes, so a watch on the leaf would silently never fire after the first swap).
- Debounces events through a single `time.Timer` (default 150ms) so a burst of write/rename/create events from one logical update coalesces into a single reload.
- Retries once on transient `os.ErrNotExist` from the reload callback (75ms delay) to cover the brief window when a directory swap can leave the path momentarily unresolvable before the new contents are linked in.
- Generic by design — no k8s-specific naming, no `..data` strings, no Secret-mount assumptions. The watcher just reacts to "something in the directory changed" and lets the reload callback re-resolve the path through whatever symlink chain happens to exist. The same pattern handles vim atomic writes, `cp + mv` rotation, helm rewrites, kubelet symlink swaps, and CSI direct-write injectors with identical code.
- Tests cover in-place writes, atomic rename, symlink retarget (constructed exactly like the k8s `..data` indirection: a `current → vN` symlink that's atomically swapped via `Rename` of a staged symlink), debounce coalescing, ctx-cancel shutdown, and the ENOENT retry path.

Wired into `main.go` via `startTokenFileWatcher(cfg.TokensFile, logger)` right after the initial `loadTokenStore` call. Reload callback is just `func() error { return loadTokenStore(path) }`, so SIGHUP and file-change route through the exact same atomic-swap function. Watcher is unconditional and platform-agnostic (fsnotify supports Linux/macOS/Windows), so this also gives Windows servers a hot-reload path they didn't have before.

Net effect: the broker's `firstMintMaxAttempts: 6` retry budget (~16s) now matches reality. The world picks up freshly minted tokens within one debounce window of the kubelet swap.

Out of scope for this change (deliberate): TLS cert file-watch. Same staleness problem applies (cert rotation through a Secret mount), and the generic watcher would handle it identically, but that's a separate change and not what's currently broken. Leaving the watcher generic means a one-line wire-up will cover it later.

Architecture doc updated to v12 to reflect that token reload happens via SIGHUP **or** file change, both routing through `loadTokenStore`.