2026-05-27

Tokens-file hot-reload via parent-directory watch

Investigated why world-a wasn't picking up broker-minted token rotations: the broker writes tokens to a k8s Secret, kubelet atomically retargets the ..data symlink under the mount path, and the demarkus-server only refreshes its in-memory TokenStore on SIGHUP — no signal, no reload. The architecture doc's "Hot-reloadable via SIGHUP" claim was accurate but insufficient for the k8s-mount case where there is no operator to send SIGHUP.

The existing wiring turned out to be already perfectly shaped for hot-reload:

• currentTokenStore is a package-level *auth.TokenStore behind tokenMu sync.RWMutex in server/cmd/demarkus-server/main.go.
• loadTokenStore(path) re-reads the TOML and atomically swaps the pointer under the write lock.
• Handler.GetTokenStore is a callback (func() *auth.TokenStore), not a captured pointer. All handler sites read the store through it — read-side hot-reload was already free.
• reload_unix.go already calls loadTokenStore on SIGHUP.

The only missing piece was something to call loadTokenStore on file change. Added a generic file watcher at server/internal/configwatch/:

• Watches filepath.Dir(target) rather than the target itself — the canonical pattern for tolerating atomic rename swaps and symlink retargets (the inode under a stable path changes, so a watch on the leaf would silently never fire after the first swap).
• Debounces events through a single time.Timer (default 150ms) so a burst of write/rename/create events from one logical update coalesces into a single reload.
• Retries once on transient os.ErrNotExist from the reload callback (75ms delay) to cover the brief window when a directory swap can leave the path momentarily unresolvable before the new contents are linked in.
• Generic by design — no k8s-specific naming, no ..data strings, no Secret-mount assumptions. The watcher just reacts to "something in the directory changed" and lets the reload callback re-resolve the path through whatever symlink chain happens to exist. The same pattern handles vim atomic writes, cp + mv rotation, helm rewrites, kubelet symlink swaps, and CSI direct-write injectors with identical code.
• Tests cover in-place writes, atomic rename, symlink retarget (constructed exactly like the k8s ..data indirection: a current → vN symlink that's atomically swapped via Rename of a staged symlink), debounce coalescing, ctx-cancel shutdown, and the ENOENT retry path.

Wired into main.go via startTokenFileWatcher(cfg.TokensFile, logger) right after the initial loadTokenStore call. Reload callback is just func() error { return loadTokenStore(path) }, so SIGHUP and file-change route through the exact same atomic-swap function. Watcher is unconditional and platform-agnostic (fsnotify supports Linux/macOS/Windows), so this also gives Windows servers a hot-reload path they didn't have before.

Net effect: the broker's firstMintMaxAttempts: 6 retry budget (~16s) now matches reality. The world picks up freshly minted tokens within one debounce window of the kubelet swap.

Out of scope for this change (deliberate): TLS cert file-watch. Same staleness problem applies (cert rotation through a Secret mount), and the generic watcher would handle it identically, but that's a separate change and not what's currently broken. Leaving the watcher generic means a one-line wire-up will cover it later.

Architecture doc updated to v12 to reflect that token reload happens via SIGHUP or file change, both routing through loadTokenStore.

Broker stable-mint — Slices 1 & 2 implemented

Plan published at /plans/broker-stable-mint.md v2 (the v1 design assumed we could "return the same raw token on next join"; v2 simplified to "don't return tokens at join time at all" once the issuer's structural constraint surfaced — raw tokens are never recorded server-side by design).

Slice 1 — Lazy provisioning at join

/me/install (install.go) and /auth/callback bare-code branch (server.go:386-434) no longer call Issuer.Mint*. They return identity + the list of worlds the caller is authorized for, no per-world credential material. Response shape:

{
  "email": "alice@example.com",
  "worlds": [{"name": "team-a", "publicURL": "mark://..."}]
}

installWorld.Label, AccessToken, ExpiresAt and installResponse.PartialFailure all removed. Confirmed with Fritz that nothing consumes these on the knowledge-system flow — direct-QUIC soul flow uses a separate ~/.mark/tokens.toml mechanism that is untouched.

Minting moves to the place it had to happen anyway — the MCP gateway's sessionCache miss in dispatchWithAuth. Kubelet-lag is now hit at most once per (broker-pod, user, world) triple, on the first dispatch after a cold cache, rather than on every join.

Tests deleted (no longer have semantics under lazy provisioning): TestMeInstallPartialFailureReturns200WithFlag, TestMeInstallHardFailureReturns500. Tests updated: TestMeInstallHappyPathSingleWorld, TestMeInstallFiltersWorldsWithoutPublicURL, TestMeInstallEmptyWorldsReturns200EmptySlice, TestAuthCallbackSuccess, TestAuthCallbackUnchangedWithoutDeviceCookie. New assertions verify no Secret writes happen during /me/install.

Slice 2 — Cache-stable retries

The field bug was: dispatchWithAuth invalidated the sessionCache on every 401 (including fresh-mint 401s), so each backoff tick triggered a new Issuer.MintFiltered round-trip, which wrote a new hash to the world's tokens.toml Secret. Under realistic GKE Secret-sync (~30-40s), the broker burned its full retry budget on a cascade of doomed tokens — Fritz observed ~20 tokens written in one 78-second session, with the world reloading only twice in that window.

Fix structure in mcp_tools_read.go:

• New local flag mintedInDispatch flips true the moment GetOrMint reports isFresh=true (or coalesced behind a singleflight peer that did).
• !mintedInDispatch && !isFresh (initial cache hit on a stale token) is the only branch that calls sessionCache.Invalidate and re-mints — that path is for genuine rotation races (operator hand-revoke / sweeper retire / Secret restore).
• Once mintedInDispatch == true, the cache is preserved across the entire propagation-race retry loop. The same raw token is presented to the world on every backoff tick. The world catches up exactly once when kubelet projects the Secret — no junk hashes left behind.

FirstMintMaxAttempts is now a propagation-budget knob (how long to wait for kubelet) rather than a doomed-token cascade limit. Slice 3 (shrink the default) is deferred until we observe the new behavior in the field.

New regression test: TestDispatchReadPropagationRetriesReuseSameToken — asserts all dispatcher calls in a 2x-401-then-OK scenario receive the SAME token string. Existing four tests (TestDispatchReadRetriesOnFreshMint401, TestDispatchReadExhaustsRetriesOnPersistent401, TestDispatchReadCacheHit401InvalidatesAndReMints, TestDispatchReadCacheHit401DoesNotConsumeFreshMintBudget) all still pass under the new logic — the cache-hit vs fresh-mint separation is preserved.

Verification

• All broker tests pass.
• All four modules (protocol, server, client, tools) pass go test ./... and pre-commit.sh (fmt + vet + lint clean).

Composed effect with PR #158

Once PR #158 (fsnotify watcher on the server side) and these two slices all land:

1. Server-side reload floor goes from "never without SIGHUP" → "one kubelet sync cycle" (PR #158).
2. Broker stops the per-join token churn (Slice 1).
3. Broker stops the per-retry token cascade during the kubelet wait (Slice 2).

Net effect: a healthy first-dispatch for a brand-new (user, world) pair takes roughly one kubelet projection cycle (~30-40s on GKE default) and writes exactly one hash to the world's Secret. Returning users hit the cache on subsequent dispatches and never touch kubelet. The user-visible "propagation lag exceeded broker deadline" error becomes a signal of an actual outage, not a routine event.

Open reads — smallest viable slice

Fritz redirected the design mid-stream toward the original demarkus instinct: "the knowledge system should be open to all — that's the whole point, not to hide things. SSO keeps it closed to the org." After a sequence of progressively simpler proposals, settled on the smallest unit that delivers it:

mark_fetch, mark_list, mark_versions dispatch with an empty token. No mint, no sessionCache, no retry loop.

Three handlers in mcp_tools_read.go each lose ~6 lines of auth machinery (claimsFromCtx + dispatchWithAuth closure) and gain a one-line direct dispatcher call:

result, err := g.dispatcher.Fetch(worldName, path, "")

The world's tokens.toml is write-only by design (no read operation granted to any token), so the empty bearer flows through and the document is returned. No world-side change required. requireAuth upstream still validates the OIDC bearer, so SSO remains the org gate.

Deletions

Six tests in mcp_tools_read_test.go were checking machinery the read path no longer reaches:

• TestHandleMarkFetchCacheHitReusesToken — no sessionCache for reads anymore.
• TestDispatchReadRetriesOnFreshMint401, TestDispatchReadPropagationRetriesReuseSameToken, TestDispatchReadExhaustsRetriesOnPersistent401, TestDispatchReadCacheHit401InvalidatesAndReMints, TestDispatchReadCacheHit401DoesNotConsumeFreshMintBudget — no retry loop for reads.
• TestHandleMarkFetchUnauthorizedIdentitySurfacesAsToolError — under the new model, any SSO-authed identity can read any world; the per-world Allow predicate is a writer allowlist, not a reader gate.

dispatchWithAuth itself is preserved — still used by writes (mark_publish/append/archive), federation (mark_discover/resolve), and graph (mark_graph/index/graph_export/graph_publish). Coverage of the retry loop is now via TestWriteHandlersInheritPropagationRaceRetry (in mcp_tools_write_test.go).

Deliberately deferred

• Federation + graph reads. mark_discover/resolve/graph/index etc. still call dispatchWithAuth internally. Auditing each and simplifying is the natural follow-on, but not part of the smallest unit Fritz asked for. The three core reads (fetch/list/versions) are what most agents hit.
• /me/install "all worlds visible to all readers" behavior. Right now the endpoint still filters by Allow (writer predicate), so read-only users see no worlds in their bundle. They can still issue mark_fetch mark://team-x/... if they know the world name, but discovery is incomplete until /me/install is updated to enumerate every configured world (with a canWrite annotation).
• Renaming WorldConfig.Allow → WorldConfig.Writers for clarity. Purely cosmetic; the predicate's meaning has changed but the code reads identically.

Net effect of all three branches landed together

Once PR #158 (fsnotify watcher), the previous broker slices (lazy provisioning + cache-stable retries), and this open-reads slice all land:

1. Reads (the majority of MCP traffic): zero issuer round-trips, zero Secret writes, zero kubelet-lag exposure. Just OIDC at the broker → QUIC to the world → back.
2. First write for a (broker-pod, writer, world) triple: one mint, one Secret write, one kubelet propagation cycle (~30-40s on GKE default), one same-token retry loop. No more 20-token cascades.
3. Subsequent writes for the same triple: cache hit, instant dispatch.

The "propagation lag exceeded broker deadline" failure mode is reduced from "every user, every join, every read" to "the very first write of every writer per broker pod restart." That residual case is the right place to spend a one-time short wait.

Per-world write tokens — one token, one Secret, shared by all writers

Fritz pushed the model further: one persistent write token per world, raw bytes in a broker-owned Secret, used by every authorized writer through the broker. No per-user mint, ever. SSO is the org gate, WorldConfig.Allow is the writer allowlist, the broker is the only party that holds the raw token. Confirmed earlier in the session that the world is not exposed outside the cluster, so leaving raw tokens in a broker Secret doesn't change the data-plane access guarantee.

New: worldWriteTokenStore

tools/demarkus-broker/internal/broker/world_write_tokens.go:

• Per-world Secret in the broker namespace, named demarkus-broker-write-token-<worldName>. JSON blob with {label, rawToken, entry}. Stable label broker-write-<worldName> so the world's tokens.toml entry is idempotent across re-provision.
• Provision(ctx, worldName) is lazy and idempotent. mutateSecret-backed read-modify-write on the broker Secret handles concurrent broker pods — first commit wins, losers converge on the winner's token without churn. After the broker Secret has a record, syncWorldHash ensures the world's tokens.toml has the matching hash under the stable label; ErrLabelExists is treated as success because the existing hash matches our raw token (both derive from the same Secret record).
• Get(worldName) is the hot-path in-memory check for the dispatch loop.
• Unknown world surfaces as *errWorldNotFound so federation/graph callers map it to the existing "not a knowledge-system world" message — no callsite changes needed in those handlers.
• Expires left empty on the world's token entry — the server treats that as "no expiry," matching "long persistent."

Refactored: dispatchWithAuth

Three changes inside the function, no signature change:

1. mintFuncFor + sessionCache.GetOrMint replaced with a single worldWriteTokens.Provision(ctx, worldName) up-front.
2. The mintedInDispatch flag and the cache-hit vs fresh-mint branching are gone — there is no cache hit semantically distinct from a fresh mint anymore. One world, one token.
3. Retry loop bounded by FirstMintMaxAttempts (knob preserved). On unauthorized, sleep with the existing exponential backoff and retry the same token. This absorbs the one-time kubelet projection lag for the very first write to a fresh world; after that the token is permanent.

claims parameter stays on the signature for callsite stability; it's intentionally unused inside (writer authz happens at the handler boundary).

Writer-allow gate at the write handlers

New helper gateWrite(claims, worldName) in mcp_tools_write.go:

• Email-verified check (defense in depth).
• Canonicalize email.
• World must be configured.
• worldAllows(&worldCfg.Allow, claims) must pass.

Called at the top of handleMarkPublish, handleMarkAppend, and handleMarkArchive. A denied caller gets a "write access denied for world ..." tool error and the dispatcher is never invoked — no broker Secret write, no kubelet round-trip.

Removed: mcpGateway.mintFuncFor

Dead after the dispatch refactor. Removed.

Still dormant (follow-up cleanup)

• Issuer.Mint, Issuer.MintFiltered, Issuer.mintForWorld, the per-user Issuance records and the issuances Secret.
• Issuer.Revoke, Issuer.RotateLabel, the /tokens and /tokens/{label}/rotate HTTP routes.
• sessionCache and mcp_session.go — referenced by mcpGateway initialization but no longer consulted by the dispatch path.
• Sweeper's per-issuance reconciliation.

All of this is now unreachable from the live code paths but kept to avoid a sprawling diff. A follow-up slice can strip it without touching dispatch semantics.

New tests

• world_write_tokens_test.go — TestWorldWriteTokenStoreProvisionIsIdempotent (two Provision calls return the same token; broker Secret has one record; world tokens.toml carries the stable-label entry once), TestWorldWriteTokenStoreProvisionUnknownWorld (returns *errWorldNotFound, no Secret created).
• mcp_tools_write_test.go — TestHandleMarkPublishDeniesNonWriter (non-writer SSO identity gets "write access denied", dispatcher.Publish is never invoked).

Verification

All four modules (protocol, server, client, tools) green on go test ./.... pre-commit.sh clean (fmt + vet + lint). The full write surface, all federation + graph tools, and the existing propagation-race tests pass without modification because the retry semantics from Slice 2 carry over to the new per-world dispatch path verbatim.

Net architectural state after everything lands

The "propagation lag exceeded broker deadline" error mode now requires:

1. A fresh world being provisioned for the first time AND
2. Kubelet failing to project the new Secret within the FirstMintMaxAttempts × FirstMintMaxBackoff budget (~few seconds default).

Both conditions together describe an actual cluster outage, not the routine onboarding flow that produced the original field bug.

Related documents

• Broker stable mint: the plan whose slices 1 and 2 landed here
• Read auth: open-reads model this session moved toward
• Architecture: updated to v12 for token hot-reload
• Broker HTTPS gateway: gateway dispatch path being refactored