# Broker — Lazy Token Provisioning + Cache-Stable Retries

## Context

Today the broker rotates per-world demarkus tokens on every `/auth/callback` and `/me/install` call AND on every 401 retry inside `dispatchWithAuth`. Combined with kubelet's Secret-projection lag (~30-40s on GKE), the result is ~20 tokens written to the world's `tokens.toml` in 78 seconds during one user session, almost all of which fail before kubelet propagates any of them, producing the user-visible "broker: world rejected freshly-minted token after N attempts" failure.

The fsnotify watcher (PR #158) reduced the world-side reload floor from "never without SIGHUP" to "kubelet sync interval," but propagation lag is still the bottleneck.

Network shape (confirmed 2026-05-27): in `knowledge.demarkus.io`, world-a is **not** exposed outside the cluster. The broker is the only public ingress. Combined with `requireAuth` re-validating the OIDC bearer on every MCP request, this means SSO removal at the IdP = no path to the world's data plane. The broker's per-world demarkus tokens can therefore be a pure internal implementation detail; the client never needs to hold them.

The broker structurally cannot return the same raw token twice (issuer.go:40-43: "Raw token material is never recorded here"). So "tokens don't change on next join" is satisfied not by stable-token-recall but by **not minting at join time at all**.

## Design

**Two changes, both small, both targeted at the bug they fix:**

### 1. Lazy provisioning

`/me/install` and `/auth/callback` stop calling `Issuer.Mint*`. They become identity-confirm endpoints: verify the bearer, list the worlds the caller passes `worldAllows` against, return `{email, worlds: [...]}`. No tokens in the response.

Minting moves to the place it has to happen anyway: the dispatch path's `sessionCache` miss in `dispatchWithAuth`. The first `mark_*` call to a given world from a given broker pod mints, writes the world Secret, populates the cache. Subsequent calls cache-hit.

Consequence: kubelet-lag is hit at most once per `(broker-pod, user, world)` triple — only on the first dispatch after a cold cache. Pod restarts re-warm lazily on first call. For HPA-heavy deployments where this becomes painful, a startup pre-warm is a follow-on; not in this slice.

### 2. Cache-stable retries

`dispatchWithAuth` currently calls `sessionCache.Invalidate(...)` + `GetOrMint(...)` inside its 401 retry loop, which re-mints on every iteration. That's the cascade producing ~20 junk tokens per first-call burst.

The retry loop was meant to wait for *the freshly-minted token to become recognized* by the world after kubelet projects the Secret update — not to re-mint each tick. The fix is:

- On 401 after fresh-mint: sleep the backoff, retry the *same* cached token. No `Invalidate`, no re-mint.
- On 401 after cache-hit (token was cached but world rejected): a genuine rotation race — `Invalidate` once, mint once, retry the new token. Bound this to a single rotation per request.
- Only after the budget exhausts entirely does the request error out.

## Why this is small

- No new protocol surface on the world.
- No probe-during-mint, no sidecar, no encrypted-at-rest token storage.
- No client-side token-shape changes (`~/.mark/tokens.toml` and `DEMARKUS_AUTH` keep working for the direct-QUIC personal-soul flow; only the knowledge-system endpoints change).
- No data migration: existing issuances on the live cluster are still consulted by `Revoke`/`List`/`RotateLabel`; they just stop accumulating fresh entries per-join.

## Slices

### Slice 1 — Lazy provisioning (smallest)

- `/me/install` (install.go): replace `MintFiltered` call with a pure `authorizedWorlds + keep` filter; drop `accessToken` from response struct; return `{email, worlds: [...]}`.
- `/auth/callback` device path (device.go) and auth-code path (oauth_authorize.go / wherever the new RFC 6749 handler shipped in #156 lives): same shape change.
- Tests: handlers return identity + worlds without invoking the issuer; verify no Secret writes happen during these flows on the test k8s client.

### Slice 2 — Cache-stable retries

- `dispatchWithAuth` (mcp_tools_read.go:147-211) — restructure so the retry loop does NOT re-mint on every 401 from a freshly-minted token; reserve `Invalidate`+re-mint for one cache-hit-rejection per request.
- The `FirstMintMaxAttempts` knob now governs propagation waits, not mint cascades.
- Tests: simulate world returning N consecutive 401s on a fresh-mint token, then 200; confirm exactly one mint, N+1 dispatch attempts, success.

### Slice 3 (deferred — only if needed)

- Shrink `FirstMintMaxAttempts` and `FirstMintMaxBackoff` once Slices 1+2 land and we observe the new behavior in the field. Not blocking.

## Out of scope (deliberately)

- Access-change propagation via tokens.toml metadata rewrites. `requireAuth`-claims-check is the source of truth for revocation; the world's tokens.toml just needs to *recognize* the token, the broker decides whether it *should* be honored.
- Background sweeper changes. Sweeper still has work to do for genuine rotations and orphans.
- Pre-warming sessionCache at broker startup. Future optimization if HPA churn shows a problem.

## Status

- 2026-05-27 — Plan v2 published, simpler model than v1. Starting Slice 1.
- PR #158 (fsnotify watcher) open, prerequisite for any field improvement but not a code-level dependency for these slices.


## Follow-up cleanup (next) — dead `DefaultToken` knobs

After the per-world write-token model + full issuance-subsystem retirement landed (PR #164, branch `refactor/broker-retire-issuance-subsystem`), two `WorldConfig.DefaultToken` fields are now no-op knobs:

- **`operations`** — `worldWriteTokenStore.Provision` hardcodes `["publish"]` regardless of config (deliberate, so open-reads can't regress via a stray `"read"`; see #163 and the comment in `world_write_tokens.go`). Already documented as "kept but ignored."
- **`expiresAfter`** — was only consumed by the removed `mintForWorld`. Provision sets `Expires: ""` (long-lived) and reads use no token, so nothing reads it now.

Both currently sit as validated-but-ignored config (chart comments updated in #164 to say so). They're a footgun for the same reason the `/tokens` API was: operator-facing knobs that silently do nothing.

**Task:** remove `expiresAfter` (and decide on `operations`) from `WorldConfig.DefaultToken` in `config.go` + its validation/defaults, the `worlds[].defaultToken.*` example in the broker chart `values.yaml`, and any config_test cases. Small standalone change — no behavior change since they're already ignored. Keep `paths` (still consumed by `Provision`).

Open question: whether to drop `operations` too or keep it as an explicit "this is always publish" marker. Lean toward removing both for honesty; if kept, the config struct should reject any value other than `["publish"]` rather than silently ignoring it.


## Status — COMPLETE (verified 2026-05-31)

All three slices plus the named follow-up shipped and verified in `tools/demarkus-broker` (broker tests green, run fresh).

- **Lazy per-world provisioning (#159, #164):** `worldWriteTokenStore.Provision` mints once per world (Secret + world `tokens.toml` + in-memory cache); subsequent calls hit `Get`. `/me/install` reduced to a pure identity-confirm endpoint — no minting, no `accessToken` in the response.
- **Cache-stable retries (#159):** `dispatchWithAuth` provisions once before the loop, then retries the *same* token on 401 with backoff (`FirstMint*` knobs). No re-mint inside the loop — the ~20-token mint cascade is structurally impossible. Regression test `TestWriteHandlersInheritPropagationRaceRetry` confirms exactly 2 dispatch attempts on one 401.
- **Dead-knob follow-up (#163, #165):** `TokenScope` now carries only `Paths`; `operations`/`expiresAfter` removed; `operations` hardcoded to `["publish"]`, `Expires` empty (long-lived). `rg expiresAfter` → zero hits.

Design simplified beyond the plan to **one long-lived write token per world** (reads dispatch open to any SSO identity); deliberate, not a gap. PRs: #158, #159, #163, #164, #165.