# 2026-05-27

## Tokens-file hot-reload via parent-directory watch

Investigated why world-a wasn't picking up broker-minted token rotations: the broker writes tokens to a k8s Secret, kubelet atomically retargets the `..data` symlink under the mount path, and the demarkus-server only refreshes its in-memory `TokenStore` on SIGHUP — no signal, no reload. The architecture doc's "Hot-reloadable via SIGHUP" claim was accurate but insufficient for the k8s-mount case where there is no operator to send SIGHUP.

The existing wiring turned out to be already perfectly shaped for hot-reload:

- `currentTokenStore` is a package-level `*auth.TokenStore` behind `tokenMu sync.RWMutex` in `server/cmd/demarkus-server/main.go`.
- `loadTokenStore(path)` re-reads the TOML and atomically swaps the pointer under the write lock.
- `Handler.GetTokenStore` is a callback (`func() *auth.TokenStore`), not a captured pointer. All handler sites read the store through it — read-side hot-reload was already free.
- `reload_unix.go` already calls `loadTokenStore` on SIGHUP.

The only missing piece was something to call `loadTokenStore` on file change. Added a generic file watcher at `server/internal/configwatch/`:

- Watches `filepath.Dir(target)` rather than the target itself — the canonical pattern for tolerating atomic rename swaps and symlink retargets (the inode under a stable path changes, so a watch on the leaf would silently never fire after the first swap).
- Debounces events through a single `time.Timer` (default 150ms) so a burst of write/rename/create events from one logical update coalesces into a single reload.
- Retries once on transient `os.ErrNotExist` from the reload callback (75ms delay) to cover the brief window when a directory swap can leave the path momentarily unresolvable before the new contents are linked in.
- Generic by design — no k8s-specific naming, no `..data` strings, no Secret-mount assumptions. The watcher just reacts to "something in the directory changed" and lets the reload callback re-resolve the path through whatever symlink chain happens to exist. The same pattern handles vim atomic writes, `cp + mv` rotation, helm rewrites, kubelet symlink swaps, and CSI direct-write injectors with identical code.
- Tests cover in-place writes, atomic rename, symlink retarget (constructed exactly like the k8s `..data` indirection: a `current → vN` symlink that's atomically swapped via `Rename` of a staged symlink), debounce coalescing, ctx-cancel shutdown, and the ENOENT retry path.

Wired into `main.go` via `startTokenFileWatcher(cfg.TokensFile, logger)` right after the initial `loadTokenStore` call. Reload callback is just `func() error { return loadTokenStore(path) }`, so SIGHUP and file-change route through the exact same atomic-swap function. Watcher is unconditional and platform-agnostic (fsnotify supports Linux/macOS/Windows), so this also gives Windows servers a hot-reload path they didn't have before.

Net effect: the broker's `firstMintMaxAttempts: 6` retry budget (~16s) now matches reality. The world picks up freshly minted tokens within one debounce window of the kubelet swap.

Out of scope for this change (deliberate): TLS cert file-watch. Same staleness problem applies (cert rotation through a Secret mount), and the generic watcher would handle it identically, but that's a separate change and not what's currently broken. Leaving the watcher generic means a one-line wire-up will cover it later.

Architecture doc updated to v12 to reflect that token reload happens via SIGHUP **or** file change, both routing through `loadTokenStore`.


## Broker stable-mint — Slices 1 & 2 implemented

Plan published at `/plans/broker-stable-mint.md` v2 (the v1 design assumed we could "return the same raw token on next join"; v2 simplified to "don't return tokens at join time at all" once the issuer's structural constraint surfaced — raw tokens are never recorded server-side by design).

### Slice 1 — Lazy provisioning at join

`/me/install` (`install.go`) and `/auth/callback` bare-code branch (`server.go:386-434`) no longer call `Issuer.Mint*`. They return identity + the list of worlds the caller is authorized for, no per-world credential material. Response shape:

```json
{
  "email": "alice@example.com",
  "worlds": [{"name": "team-a", "publicURL": "mark://..."}]
}
```

`installWorld.Label`, `AccessToken`, `ExpiresAt` and `installResponse.PartialFailure` all removed. Confirmed with Fritz that nothing consumes these on the knowledge-system flow — direct-QUIC soul flow uses a separate `~/.mark/tokens.toml` mechanism that is untouched.

Minting moves to the place it had to happen anyway — the MCP gateway's `sessionCache` miss in `dispatchWithAuth`. Kubelet-lag is now hit at most once per `(broker-pod, user, world)` triple, on the first dispatch after a cold cache, rather than on every join.

Tests deleted (no longer have semantics under lazy provisioning): `TestMeInstallPartialFailureReturns200WithFlag`, `TestMeInstallHardFailureReturns500`. Tests updated: `TestMeInstallHappyPathSingleWorld`, `TestMeInstallFiltersWorldsWithoutPublicURL`, `TestMeInstallEmptyWorldsReturns200EmptySlice`, `TestAuthCallbackSuccess`, `TestAuthCallbackUnchangedWithoutDeviceCookie`. New assertions verify no Secret writes happen during `/me/install`.

### Slice 2 — Cache-stable retries

The field bug was: `dispatchWithAuth` invalidated the sessionCache on every 401 (including fresh-mint 401s), so each backoff tick triggered a new `Issuer.MintFiltered` round-trip, which wrote a new hash to the world's `tokens.toml` Secret. Under realistic GKE Secret-sync (~30-40s), the broker burned its full retry budget on a cascade of doomed tokens — Fritz observed ~20 tokens written in one 78-second session, with the world reloading only twice in that window.

Fix structure in `mcp_tools_read.go`:

- New local flag `mintedInDispatch` flips true the moment `GetOrMint` reports `isFresh=true` (or coalesced behind a singleflight peer that did).
- `!mintedInDispatch && !isFresh` (initial cache hit on a stale token) is the only branch that calls `sessionCache.Invalidate` and re-mints — that path is for genuine rotation races (operator hand-revoke / sweeper retire / Secret restore).
- Once `mintedInDispatch == true`, the cache is preserved across the entire propagation-race retry loop. The same raw token is presented to the world on every backoff tick. The world catches up exactly once when kubelet projects the Secret — no junk hashes left behind.

`FirstMintMaxAttempts` is now a propagation-budget knob (how long to wait for kubelet) rather than a doomed-token cascade limit. Slice 3 (shrink the default) is deferred until we observe the new behavior in the field.

New regression test: `TestDispatchReadPropagationRetriesReuseSameToken` — asserts all dispatcher calls in a 2x-401-then-OK scenario receive the SAME token string. Existing four tests (`TestDispatchReadRetriesOnFreshMint401`, `TestDispatchReadExhaustsRetriesOnPersistent401`, `TestDispatchReadCacheHit401InvalidatesAndReMints`, `TestDispatchReadCacheHit401DoesNotConsumeFreshMintBudget`) all still pass under the new logic — the cache-hit vs fresh-mint separation is preserved.

### Verification

- All broker tests pass.
- All four modules (protocol, server, client, tools) pass `go test ./...` and `pre-commit.sh` (fmt + vet + lint clean).

### Composed effect with PR #158

Once PR #158 (fsnotify watcher on the server side) and these two slices all land:

1. Server-side reload floor goes from "never without SIGHUP" → "one kubelet sync cycle" (PR #158).
2. Broker stops the per-join token churn (Slice 1).
3. Broker stops the per-retry token cascade during the kubelet wait (Slice 2).

Net effect: a healthy first-dispatch for a brand-new (user, world) pair takes roughly one kubelet projection cycle (~30-40s on GKE default) and writes exactly one hash to the world's Secret. Returning users hit the cache on subsequent dispatches and never touch kubelet. The user-visible "propagation lag exceeded broker deadline" error becomes a signal of an actual outage, not a routine event.