Broker — Lazy Token Provisioning + Cache-Stable Retries

Context

Today the broker rotates per-world demarkus tokens on every /auth/callback and /me/install call AND on every 401 retry inside dispatchWithAuth. Combined with kubelet's Secret-projection lag (~30-40s on GKE), the result is ~20 tokens written to the world's tokens.toml in 78 seconds during one user session, almost all of which fail before kubelet propagates any of them, producing the user-visible "broker: world rejected freshly-minted token after N attempts" failure.

The fsnotify watcher (PR #158) reduced the world-side reload floor from "never without SIGHUP" to "kubelet sync interval," but propagation lag is still the bottleneck.

Network shape (confirmed 2026-05-27): in knowledge.demarkus.io, world-a is not exposed outside the cluster. The broker is the only public ingress. Combined with requireAuth re-validating the OIDC bearer on every MCP request, this means SSO removal at the IdP = no path to the world's data plane. The broker's per-world demarkus tokens can therefore be a pure internal implementation detail; the client never needs to hold them.

The broker structurally cannot return the same raw token twice (issuer.go:40-43: "Raw token material is never recorded here"). So "tokens don't change on next join" is satisfied not by stable-token-recall but by not minting at join time at all.

Design

Two changes, both small, both targeted at the bug they fix:

1. Lazy provisioning

/me/install and /auth/callback stop calling Issuer.Mint*. They become identity-confirm endpoints: verify the bearer, list the worlds the caller passes worldAllows against, return {email, worlds: [...]}. No tokens in the response.

Minting moves to the place it has to happen anyway: the dispatch path's sessionCache miss in dispatchWithAuth. The first mark_* call to a given world from a given broker pod mints, writes the world Secret, populates the cache. Subsequent calls cache-hit.

Consequence: kubelet-lag is hit at most once per (broker-pod, user, world) triple — only on the first dispatch after a cold cache. Pod restarts re-warm lazily on first call. For HPA-heavy deployments where this becomes painful, a startup pre-warm is a follow-on; not in this slice.

2. Cache-stable retries

dispatchWithAuth currently calls sessionCache.Invalidate(...) + GetOrMint(...) inside its 401 retry loop, which re-mints on every iteration. That's the cascade producing ~20 junk tokens per first-call burst.

The retry loop was meant to wait for the freshly-minted token to become recognized by the world after kubelet projects the Secret update — not to re-mint each tick. The fix is:

• On 401 after fresh-mint: sleep the backoff, retry the same cached token. No Invalidate, no re-mint.
• On 401 after cache-hit (token was cached but world rejected): a genuine rotation race — Invalidate once, mint once, retry the new token. Bound this to a single rotation per request.
• Only after the budget exhausts entirely does the request error out.

Why this is small

• No new protocol surface on the world.
• No probe-during-mint, no sidecar, no encrypted-at-rest token storage.
• No client-side token-shape changes (~/.mark/tokens.toml and DEMARKUS_AUTH keep working for the direct-QUIC personal-soul flow; only the knowledge-system endpoints change).
• No data migration: existing issuances on the live cluster are still consulted by Revoke/List/RotateLabel; they just stop accumulating fresh entries per-join.

Slices

Slice 1 — Lazy provisioning (smallest)

• /me/install (install.go): replace MintFiltered call with a pure authorizedWorlds + keep filter; drop accessToken from response struct; return {email, worlds: [...]}.
• /auth/callback device path (device.go) and auth-code path (oauth_authorize.go / wherever the new RFC 6749 handler shipped in #156 lives): same shape change.
• Tests: handlers return identity + worlds without invoking the issuer; verify no Secret writes happen during these flows on the test k8s client.

Slice 2 — Cache-stable retries

• dispatchWithAuth (mcp_tools_read.go:147-211) — restructure so the retry loop does NOT re-mint on every 401 from a freshly-minted token; reserve Invalidate+re-mint for one cache-hit-rejection per request.
• The FirstMintMaxAttempts knob now governs propagation waits, not mint cascades.
• Tests: simulate world returning N consecutive 401s on a fresh-mint token, then 200; confirm exactly one mint, N+1 dispatch attempts, success.

Slice 3 (deferred — only if needed)

• Shrink FirstMintMaxAttempts and FirstMintMaxBackoff once Slices 1+2 land and we observe the new behavior in the field. Not blocking.

Out of scope (deliberately)

• Access-change propagation via tokens.toml metadata rewrites. requireAuth-claims-check is the source of truth for revocation; the world's tokens.toml just needs to recognize the token, the broker decides whether it should be honored.
• Background sweeper changes. Sweeper still has work to do for genuine rotations and orphans.
• Pre-warming sessionCache at broker startup. Future optimization if HPA churn shows a problem.

Status

• 2026-05-27 — Plan v2 published, simpler model than v1. Starting Slice 1.
• PR #158 (fsnotify watcher) open, prerequisite for any field improvement but not a code-level dependency for these slices.

Follow-up cleanup (next) — dead DefaultToken knobs

After the per-world write-token model + full issuance-subsystem retirement landed (PR #164, branch refactor/broker-retire-issuance-subsystem), two WorldConfig.DefaultToken fields are now no-op knobs:

• operations — worldWriteTokenStore.Provision hardcodes ["publish"] regardless of config (deliberate, so open-reads can't regress via a stray "read"; see #163 and the comment in world_write_tokens.go). Already documented as "kept but ignored."
• expiresAfter — was only consumed by the removed mintForWorld. Provision sets Expires: "" (long-lived) and reads use no token, so nothing reads it now.

Both currently sit as validated-but-ignored config (chart comments updated in #164 to say so). They're a footgun for the same reason the /tokens API was: operator-facing knobs that silently do nothing.

Task: remove expiresAfter (and decide on operations) from WorldConfig.DefaultToken in config.go + its validation/defaults, the worlds[].defaultToken.* example in the broker chart values.yaml, and any config_test cases. Small standalone change — no behavior change since they're already ignored. Keep paths (still consumed by Provision).

Open question: whether to drop operations too or keep it as an explicit "this is always publish" marker. Lean toward removing both for honesty; if kept, the config struct should reject any value other than ["publish"] rather than silently ignoring it.

Status — COMPLETE (verified 2026-05-31)

All three slices plus the named follow-up shipped and verified in tools/demarkus-broker (broker tests green, run fresh).

• Lazy per-world provisioning (#159, #164): worldWriteTokenStore.Provision mints once per world (Secret + world tokens.toml + in-memory cache); subsequent calls hit Get. /me/install reduced to a pure identity-confirm endpoint — no minting, no accessToken in the response.
• Cache-stable retries (#159): dispatchWithAuth provisions once before the loop, then retries the same token on 401 with backoff (FirstMint* knobs). No re-mint inside the loop — the ~20-token mint cascade is structurally impossible. Regression test TestWriteHandlersInheritPropagationRaceRetry confirms exactly 2 dispatch attempts on one 401.
• Dead-knob follow-up (#163, #165): TokenScope now carries only Paths; operations/expiresAfter removed; operations hardcoded to ["publish"], Expires empty (long-lived). rg expiresAfter → zero hits.

Design simplified beyond the plan to one long-lived write token per world (reads dispatch open to any SSO identity); deliberate, not a gap. PRs: #158, #159, #163, #164, #165.

Related documents

• Broker dead-code cleanup: issuance subsystem retirement that followed this work
• Broker MCP gateway: dispatchWithAuth and session cache this plan reshapes
• Broker authorization code grant: auth-code callback path covered by slice 1