# 2026-05-27

## Tokens-file hot-reload via parent-directory watch

Investigated why world-a wasn't picking up broker-minted token rotations: the broker writes tokens to a k8s Secret, kubelet atomically retargets the `..data` symlink under the mount path, and the demarkus-server only refreshes its in-memory `TokenStore` on SIGHUP — no signal, no reload. The architecture doc's "Hot-reloadable via SIGHUP" claim was accurate but insufficient for the k8s-mount case where there is no operator to send SIGHUP.

The existing wiring turned out to be already perfectly shaped for hot-reload:

- `currentTokenStore` is a package-level `*auth.TokenStore` behind `tokenMu sync.RWMutex` in `server/cmd/demarkus-server/main.go`.
- `loadTokenStore(path)` re-reads the TOML and atomically swaps the pointer under the write lock.
- `Handler.GetTokenStore` is a callback (`func() *auth.TokenStore`), not a captured pointer. All handler sites read the store through it — read-side hot-reload was already free.
- `reload_unix.go` already calls `loadTokenStore` on SIGHUP.

The only missing piece was something to call `loadTokenStore` on file change. Added a generic file watcher at `server/internal/configwatch/`:

- Watches `filepath.Dir(target)` rather than the target itself — the canonical pattern for tolerating atomic rename swaps and symlink retargets (the inode under a stable path changes, so a watch on the leaf would silently never fire after the first swap).
- Debounces events through a single `time.Timer` (default 150ms) so a burst of write/rename/create events from one logical update coalesces into a single reload.
- Retries once on transient `os.ErrNotExist` from the reload callback (75ms delay) to cover the brief window when a directory swap can leave the path momentarily unresolvable before the new contents are linked in.
- Generic by design — no k8s-specific naming, no `..data` strings, no Secret-mount assumptions. The watcher just reacts to "something in the directory changed" and lets the reload callback re-resolve the path through whatever symlink chain happens to exist. The same pattern handles vim atomic writes, `cp + mv` rotation, helm rewrites, kubelet symlink swaps, and CSI direct-write injectors with identical code.
- Tests cover in-place writes, atomic rename, symlink retarget (constructed exactly like the k8s `..data` indirection: a `current → vN` symlink that's atomically swapped via `Rename` of a staged symlink), debounce coalescing, ctx-cancel shutdown, and the ENOENT retry path.

Wired into `main.go` via `startTokenFileWatcher(cfg.TokensFile, logger)` right after the initial `loadTokenStore` call. Reload callback is just `func() error { return loadTokenStore(path) }`, so SIGHUP and file-change route through the exact same atomic-swap function. Watcher is unconditional and platform-agnostic (fsnotify supports Linux/macOS/Windows), so this also gives Windows servers a hot-reload path they didn't have before.

Net effect: the broker's `firstMintMaxAttempts: 6` retry budget (~16s) now matches reality. The world picks up freshly minted tokens within one debounce window of the kubelet swap.

Out of scope for this change (deliberate): TLS cert file-watch. Same staleness problem applies (cert rotation through a Secret mount), and the generic watcher would handle it identically, but that's a separate change and not what's currently broken. Leaving the watcher generic means a one-line wire-up will cover it later.

Architecture doc updated to v12 to reflect that token reload happens via SIGHUP **or** file change, both routing through `loadTokenStore`.


## Broker stable-mint — Slices 1 & 2 implemented

Plan published at `/plans/broker-stable-mint.md` v2 (the v1 design assumed we could "return the same raw token on next join"; v2 simplified to "don't return tokens at join time at all" once the issuer's structural constraint surfaced — raw tokens are never recorded server-side by design).

### Slice 1 — Lazy provisioning at join

`/me/install` (`install.go`) and `/auth/callback` bare-code branch (`server.go:386-434`) no longer call `Issuer.Mint*`. They return identity + the list of worlds the caller is authorized for, no per-world credential material. Response shape:

```json
{
  "email": "alice@example.com",
  "worlds": [{"name": "team-a", "publicURL": "mark://..."}]
}
```

`installWorld.Label`, `AccessToken`, `ExpiresAt` and `installResponse.PartialFailure` all removed. Confirmed with Fritz that nothing consumes these on the knowledge-system flow — direct-QUIC soul flow uses a separate `~/.mark/tokens.toml` mechanism that is untouched.

Minting moves to the place it had to happen anyway — the MCP gateway's `sessionCache` miss in `dispatchWithAuth`. Kubelet-lag is now hit at most once per `(broker-pod, user, world)` triple, on the first dispatch after a cold cache, rather than on every join.

Tests deleted (no longer have semantics under lazy provisioning): `TestMeInstallPartialFailureReturns200WithFlag`, `TestMeInstallHardFailureReturns500`. Tests updated: `TestMeInstallHappyPathSingleWorld`, `TestMeInstallFiltersWorldsWithoutPublicURL`, `TestMeInstallEmptyWorldsReturns200EmptySlice`, `TestAuthCallbackSuccess`, `TestAuthCallbackUnchangedWithoutDeviceCookie`. New assertions verify no Secret writes happen during `/me/install`.

### Slice 2 — Cache-stable retries

The field bug was: `dispatchWithAuth` invalidated the sessionCache on every 401 (including fresh-mint 401s), so each backoff tick triggered a new `Issuer.MintFiltered` round-trip, which wrote a new hash to the world's `tokens.toml` Secret. Under realistic GKE Secret-sync (~30-40s), the broker burned its full retry budget on a cascade of doomed tokens — Fritz observed ~20 tokens written in one 78-second session, with the world reloading only twice in that window.

Fix structure in `mcp_tools_read.go`:

- New local flag `mintedInDispatch` flips true the moment `GetOrMint` reports `isFresh=true` (or coalesced behind a singleflight peer that did).
- `!mintedInDispatch && !isFresh` (initial cache hit on a stale token) is the only branch that calls `sessionCache.Invalidate` and re-mints — that path is for genuine rotation races (operator hand-revoke / sweeper retire / Secret restore).
- Once `mintedInDispatch == true`, the cache is preserved across the entire propagation-race retry loop. The same raw token is presented to the world on every backoff tick. The world catches up exactly once when kubelet projects the Secret — no junk hashes left behind.

`FirstMintMaxAttempts` is now a propagation-budget knob (how long to wait for kubelet) rather than a doomed-token cascade limit. Slice 3 (shrink the default) is deferred until we observe the new behavior in the field.

New regression test: `TestDispatchReadPropagationRetriesReuseSameToken` — asserts all dispatcher calls in a 2x-401-then-OK scenario receive the SAME token string. Existing four tests (`TestDispatchReadRetriesOnFreshMint401`, `TestDispatchReadExhaustsRetriesOnPersistent401`, `TestDispatchReadCacheHit401InvalidatesAndReMints`, `TestDispatchReadCacheHit401DoesNotConsumeFreshMintBudget`) all still pass under the new logic — the cache-hit vs fresh-mint separation is preserved.

### Verification

- All broker tests pass.
- All four modules (protocol, server, client, tools) pass `go test ./...` and `pre-commit.sh` (fmt + vet + lint clean).

### Composed effect with PR #158

Once PR #158 (fsnotify watcher on the server side) and these two slices all land:

1. Server-side reload floor goes from "never without SIGHUP" → "one kubelet sync cycle" (PR #158).
2. Broker stops the per-join token churn (Slice 1).
3. Broker stops the per-retry token cascade during the kubelet wait (Slice 2).

Net effect: a healthy first-dispatch for a brand-new (user, world) pair takes roughly one kubelet projection cycle (~30-40s on GKE default) and writes exactly one hash to the world's Secret. Returning users hit the cache on subsequent dispatches and never touch kubelet. The user-visible "propagation lag exceeded broker deadline" error becomes a signal of an actual outage, not a routine event.


## Open reads — smallest viable slice

Fritz redirected the design mid-stream toward the original demarkus instinct: "the knowledge system should be open to all — that's the whole point, not to hide things. SSO keeps it closed to the org." After a sequence of progressively simpler proposals, settled on the smallest unit that delivers it:

**`mark_fetch`, `mark_list`, `mark_versions` dispatch with an empty token. No mint, no sessionCache, no retry loop.**

Three handlers in `mcp_tools_read.go` each lose ~6 lines of auth machinery (`claimsFromCtx` + `dispatchWithAuth` closure) and gain a one-line direct dispatcher call:

```go
result, err := g.dispatcher.Fetch(worldName, path, "")
```

The world's `tokens.toml` is write-only by design (no `read` operation granted to any token), so the empty bearer flows through and the document is returned. No world-side change required. `requireAuth` upstream still validates the OIDC bearer, so SSO remains the org gate.

### Deletions

Six tests in `mcp_tools_read_test.go` were checking machinery the read path no longer reaches:

- `TestHandleMarkFetchCacheHitReusesToken` — no sessionCache for reads anymore.
- `TestDispatchReadRetriesOnFreshMint401`, `TestDispatchReadPropagationRetriesReuseSameToken`, `TestDispatchReadExhaustsRetriesOnPersistent401`, `TestDispatchReadCacheHit401InvalidatesAndReMints`, `TestDispatchReadCacheHit401DoesNotConsumeFreshMintBudget` — no retry loop for reads.
- `TestHandleMarkFetchUnauthorizedIdentitySurfacesAsToolError` — under the new model, any SSO-authed identity can read any world; the per-world `Allow` predicate is a writer allowlist, not a reader gate.

`dispatchWithAuth` itself is preserved — still used by writes (`mark_publish`/`append`/`archive`), federation (`mark_discover`/`resolve`), and graph (`mark_graph`/`index`/`graph_export`/`graph_publish`). Coverage of the retry loop is now via `TestWriteHandlersInheritPropagationRaceRetry` (in `mcp_tools_write_test.go`).

### Deliberately deferred

- **Federation + graph reads.** `mark_discover`/`resolve`/`graph`/`index` etc. still call `dispatchWithAuth` internally. Auditing each and simplifying is the natural follow-on, but not part of the smallest unit Fritz asked for. The three core reads (`fetch`/`list`/`versions`) are what most agents hit.
- **`/me/install` "all worlds visible to all readers"** behavior. Right now the endpoint still filters by `Allow` (writer predicate), so read-only users see no worlds in their bundle. They can still issue `mark_fetch mark://team-x/...` if they know the world name, but discovery is incomplete until `/me/install` is updated to enumerate every configured world (with a `canWrite` annotation).
- **Renaming `WorldConfig.Allow` → `WorldConfig.Writers`** for clarity. Purely cosmetic; the predicate's meaning has changed but the code reads identically.

### Net effect of all three branches landed together

Once PR #158 (fsnotify watcher), the previous broker slices (lazy provisioning + cache-stable retries), and this open-reads slice all land:

1. **Reads** (the majority of MCP traffic): zero issuer round-trips, zero Secret writes, zero kubelet-lag exposure. Just OIDC at the broker → QUIC to the world → back.
2. **First write for a `(broker-pod, writer, world)` triple**: one mint, one Secret write, one kubelet propagation cycle (~30-40s on GKE default), one same-token retry loop. No more 20-token cascades.
3. **Subsequent writes for the same triple**: cache hit, instant dispatch.

The "propagation lag exceeded broker deadline" failure mode is reduced from "every user, every join, every read" to "the very first write of every writer per broker pod restart." That residual case is the right place to spend a one-time short wait.