# 2026-07-17: join URLs + cert pinning (remote-issuance step 1) Implemented step 1 of [/plans/remote-token-issuance.md](/plans/remote-token-issuance.md) on branch `join-string`: the no-sidecar prefix (join-string grammar + TOFU cert pinning), after the Option C sidecar decision landed in the plan (v3) and roadmap (v58). ## What shipped - `client/joinurl` (new package): Build/Parse of `mark://host#token=...&fp=sha256:` join URLs, cert fingerprint helpers (DER, PEM), `VerifyPin` TLS verifier. Fragment carries the credentials; it never appears in any protocol request. - `client/fetch`: `Options.Pins` (host to fingerprint map). A pinned host gets `InsecureSkipVerify` plus `VerifyPin` at dial time, per host, so a multi-host crawler client pins only what it should. Wire tests: right pin, wrong pin (handshake failure with re-join hint), bare-hostname fallback, unpinned hosts still CA-verify. - `client/internal/tokens`: entry gains `pin` field (tokens.toml, additive), `SetPin`/`Pins`/`ResolvePin` (flag > `DEMARKUS_PIN` > stored). - All three clients wire stored pins; `demarkus-mcp` gains `-pin`; new `demarkus join ''` stores token + pin in one paste. - `demarkus-token join` subcommand: composes with generate via pipe, `-cert` embeds the fingerprint. - `install.sh`: summary now prints a ready-to-paste `/soul-join ` and `demarkus join` line. - Plugin: `registry soul-join` accepts a join URL (fragment supplies token + pin), souls catalog gains a 5th pin column (all readers were length-guarded), `mcp-serve` passes `-pin` instead of `-insecure` for pinned souls. `commands/soul-join.md` updated; plugin 0.12.16 to 0.13.0. - Docs: `docs/site/tools/index.md` join-URL section. ## Decisions and gotchas - **Never pin Let's Encrypt certs**: leaves rotate on renewal (install.sh runs certbot twice daily) and would break every pinned client. install.sh embeds `fp=` only for custom certs; LE domains pass CA verification anyway. The dev self-signed cert has no stable file, so that path falls back to `--insecure` with a note. - **Go url fragment gotcha**: `u.Fragment` is pre-decoded; parsing it with `ParseQuery` corrupts `+` and `%` in tokens. Use `u.EscapedFragment()`. Conversely `url.Values.Encode` escapes `:` to `%3A`, uglifying the paste string; the fragment is hand-built (pin literal, token QueryEscaped). - Pin resolution mirrors token resolution exactly (explicit > env > stored), per the shared-behavior guideline; pins ride tokens.toml instead of a new file. - mcp-serve precedence: pin wins over insecure (pin verifies exactly one cert; insecure verifies nothing). ## Verified live Self-signed server on :6399: `demarkus-token generate | join -cert` produced the URL, `demarkus join` stored both credentials, publish + fetch ran with no `--insecure`, and a corrupted pin failed at the QUIC handshake with the actionable mismatch error. Plugin path: `registry soul-join ` wrote the 0600 token file, `pinned=1`, catalog row carries the pin. Full `make test` + `bash pre-commit.sh` clean. Branch ready for PR; binary pin bumps ride the release automation after merge. ## Follow-up: self-signed to CA migration Question surfaced the scenario: server moves from a pinned self-signed cert to a CA cert. Pinned clients hard-fail (strict pin, by design); recovery is re-joining with a pinless join URL. The plugin path already handled it (catalog row upsert clears the pin), but `demarkus join` only wrote a pin when the URL carried one, so a stale pin survived re-join and kept failing. Fixed in the same branch: re-join re-states the trust material, a token-bearing URL without `fp=` clears the stored pin (with a stderr notice when one existed). Verified live against the e2e server. Lesson: when a credential store is updated by re-stating, absence of a field must overwrite, not preserve; the two join paths (upsert-row vs set-field) diverged on exactly that. ## Cut: pinning removed, branch ships token-only Fritz was uneasy about the direction; the complexity audit agreed with him. The join URL half is ~100 lines of formatting with zero trust semantics; the pinning half carried a third trust state, rotation coupling, a migration bug (caught pre-merge), mixed-version edges, and hand-rolled VerifyPeerCertificate. Every follow-up question in review prep was a pinning question. HPKP precedent (browsers added then removed pinning in 2018) and the narrow constituency (only domainless hosts; LE covers the rest) settled it. Cut executed on the branch: joinurl is token-only (`mark://host#token=...`, unknown fragment keys still fail loudly so future keys cannot silently drop), fetch/tokens/CLI/TUI/MCP pin code removed, souls catalog back to 4 columns, `demarkus-token join` drops `-cert`, install.sh join line token-only with an `--insecure` note for self-signed, both plugin docs and site docs updated. Full test suite + pre-commit clean; token-only e2e re-verified live (CLI join, publish, plugin soul-join with fragment token). Decision recorded in the plan (v4): pinning deferred; if verified joins for domainless hosts return, they ride the sidecar redeem handshake (fingerprint out of band in the redeem response), not user-visible trust state. Plan sequencing also revised: sidecar redeem-only demoted; authenticated create is the sidecar MVP. Lesson: when a feature generates a stream of "what happens when..." questions before it ships, that is the complexity bill arriving early; the severable half that causes them is the half to cut. ## PR #261 review round (CodeRabbit) Eight findings fixed, two declined: - Fixed: joinurl.Parse rejects URL state Join cannot represent (userinfo, query, duplicate token keys) and bracketed IPv6 literals (downstream slug derivation and default-port append would misparse them); Build rejects `@` and `[` in hosts. Negative tests for all, plus a SoulJoin regression for bracketed IPv6. - Fixed: install.sh no longer swallows demarkus-token join failures with `2>/dev/null || true`; a failure warns with the manual command. - Fixed: RemoteSoulRow returns an error for a catalog row with no host instead of ok=true (mcp-serve would have launched with an empty -host); mcp-serve surfaces catalog I/O errors separately from "not in the catalog". - Fixed: demarkus-token join bounds stdin with a LimitReader (4KB) so piping a wrong file cannot become a "token". - Fixed: joinurl package comment tightened to 3 lines; new test's MkdirAll error handled. - Fixed: pi-memory soul-join doc no longer shows a token-bearing join URL as slash-command input (contradicted pi's own token-in-transcript rule); credentialed joins go in the user-run terminal command. - Declined: checking fmt.Fprintf errors on stderr usage output; every existing subcommand's usage block uses the same unchecked idiom and each is followed by os.Exit(1). - Declined: repo-wide IPv6 support in fetch.ParseMarkURL / TLS SNI; pre-existing behavior shared by every caller, not introduced by this PR. The join paths now fail loudly on IPv6 literals so nothing broken can be stored; full IPv6 support filed as follow-up. ## PR #261 review round 2 (outside-diff + late inline) Two fixed, one declined: - Fixed: port-only authorities rejected in joinurl (Build refuses a leading-colon host, Parse requires a non-empty hostname); regression cases both paths. - Fixed: SoulJoin now routes every non-HTTPS host through joinurl.Parse, not only fragment-bearing ones, so malformed legacy inputs (paths, userinfo, query, IPv6 literals) fail loudly instead of half-parsing into a broken catalog row. Behavior change accepted deliberately: inputs like mark://host/path previously "succeeded" by storing a corrupt host. Regression cases added. - Declined: unifying server/tools version resolution in install.sh into one pinned manifest. Independent per-component latest-release resolution is the installer's pre-existing pattern for every component; the skew window for the join subcommand is the gap between merging and the tools release publishing, and the failure mode after round 1's fix is a visible warning with the manual command. A single release manifest is a real installer improvement but a separate effort, not this PR. ## PR #261 review round 3 One finding, fixed: joinurl gains a shared validateHostPort rule (non-empty hostname, port numeric 1-65535, max 5 digits) enforced by both Build and Parse. Previously Build emitted h:bad and h:65536, and Parse accepted numeric out-of-range ports (Go's net/url validates digits only, not range). Subsumes the round-2 port-only-authority check. Negative tests: h:bad, h:65536, h:0, port-only, both paths. ## PR #261 review round 4 One finding, fixed: validateHostPort's hostname check flipped from a rune blocklist (which kept missing cases: spaces, stray brackets) to a DNS-charset allowlist ([a-zA-Z0-9._-]). Subsumes the previous ContainsAny("/#?@[") check entirely; one rule now covers every malformed-host class from rounds 1-4. Tests: "h x" and "h]x" rejected. Lesson: for input validation on a credential-bearing string, start with an allowlist; every blocklist round-trip with a reviewer is a missed-rune bug you shipped. ## Merged PR #261 merged to main 2026-07-17 as commit 4c5f90d (squash) after four CodeRabbit rounds. Local branch deleted; main pulled. Plan doc updated to v5 (sequencing step 1 DONE with merge refs). The release chain takes it from here: client/tools releases, then the automated pin-bump PR repoints the plugins at binaries that carry the join subcommand and join-URL SoulJoin (until that lands, a new install.sh with an old tools release warns and falls back to the manual command). Next up per the plan: the sidecar, redeem + authenticated create together. ## Parity decision: the sidecar is the broker in single-host mode Fritz set a parity goal: personal, team, and non-k8s deployments get the same library + broker experience as the k8s universe, including invites. Analysis showed the broker's only real k8s bindings are the Secrets writer (becomes a tokens.toml writer; the server already file-watch reloads) and packaging (becomes systemd/compose); OIDC works fine off-cluster via a Google/GitHub OAuth app, and the library only knows a broker URL. Consequence: no new sidecar binary; the remote-token-issuance plan (v6) now sequences broker single-host mode first (k8s audit, file backend, packaging), then invites in the broker minting core for both backends. The one open design question is identity for the zero-external-dependency personal tier; leaning capability-token front door before any built-in IdP. Convergence moved from the last step to the first. ## Broker k8s audit done Findings appended to /plans/remote-token-issuance.md (v7). Headline: single-host mode is one storage interface (~5 files + main.go wiring); every Secret mutation already funnels through a single mutateSecret function. Three surprises: the broker has no SIGHUP at all (the documented "SIGHUP on revocation" never existed; k8s reload is Secret-mount projection, and on a VPS the server's own tokens-file watcher covers it, faster); write-token revocation is unimplemented (real new work for the membership step); and Provision runs on the read path, so the backend is hit by first read. Roadmap's Phase 6.3 description needs the SIGHUP claim corrected when next touched. ## Broker storage interface (branch storage-interface) Step 2 part 1 of the remote-token-issuance plan implemented: the broker's k8s surface is now behind a SecretStore interface. - New `internal/broker/storage.go`: SecretRef (k8s tuple + file path, resolved by Config helpers so every call site locates credentials one way), SecretStore interface, k8sSecretStore (the old mutateSecret body verbatim), fileSecretStore (per-path in-process locking, atomic temp+rename preserving an existing file's mode, absent-stays-absent and no-rewrite-on-equal contracts preserved so the world server's dir-watcher sees exactly one complete replacement per real change). - RefreshStore, worldWriteTokenStore, NewServer all take SecretStore; the audit's prediction held (one interface, ~5 files). ~30 test call sites converted mechanically to NewK8sSecretStore wrapping. - Config: `storage: {backend: kubernetes|file, dir}` (defaults kubernetes, fully back-compat), `WorldConfig.TokensFile`; file mode requires dir + per-world tokensFile + internalAddress and drops the namespace/Secret requirements; validateStorage split out for gocyclo. - main.go: file mode skips k8s client construction entirely; Sweeper gains Run (no leader election) for single-instance deployments. - Tests: file-store contract suite (create/update, absent-stays-absent, no-rewrite-on-equal via mtime, mode preservation 0640, mutate-error no-write, concurrent no-lost-updates), config validation both modes, and TestProvisionFileBackend pinning the seam: Provision mints, hash lands in the world's local tokens.toml, broker-restart re-provision converges on the same token without rewriting the file. - Verified live: broker binary boots on a plain host with `backend: file` (healthz ok, single-host sweeper, zero k8s references in logs); and the server hot-reload seam, a rename-replaced tokens.toml (the file store's write style) authorizes a publish with the new token immediately, no restart. Remaining for step 2: non-k8s packaging (systemd/compose, install.sh --with-broker --with-library) and a full OIDC-in-the-loop e2e. Refresh tokens persist to storage.dir/refresh_tokens.json via the same interface, so VPS restarts keep sessions. ## PR #262 review round (CodeRabbit, 9 findings, all accepted) Strong round; the file store is credential infrastructure and three findings were production-grade catches: - **Ownership preservation**: CreateTemp+rename re-owned the tokens file to the broker user and its primary group, silently revoking the world server's group-read on the first mutation. Fixed with chownPreserving: restore uid+gid; on EPERM fall back to gid-only (an unprivileged broker can always keep a group it belongs to); error with an actionable message if neither works. Platform split via storage_unix.go/storage_other.go (syscall.Stat_t). - **Directory fsync after rename**: file contents were fsynced but the rename's directory entry was not; a crash could acknowledge a credential mutation and boot with the old file. Added syncDir. - **Path collision validation**: a world tokensFile aliasing storage.dir state files (or another world's file) would have two stores mutating one document; validateFilePaths rejects with filepath.Clean comparison, tested including a non-clean alias. Also fixed: temp-file cleanup errors joined into the returned error instead of discarded; RefreshStore gained a file-backend lifecycle test (issue, restart-with-fresh-store, refresh, revoke) proving backend neutrality and restart persistence; concurrency test now surfaces every goroutine's Mutate error; no-rewrite test asserts os.SameFile in addition to mtime (mtime alone passes falsely within timestamp resolution); stale kubernetes-client comment on Server.refreshStore updated; three verbose comments trimmed to invariants. Lesson (file store): atomic-rename file replacement has three durability/identity obligations beyond the rename itself: preserve mode AND ownership, fsync the parent directory, and treat cleanup failure as an error. Each is invisible in tests on a friendly filesystem and bites in production. ## PR #262 review round 2 Three findings, all fixed: a discarded os.Stat error in the file store could have replaced an existing-but-unstattable shared tokens file with downgraded 0600/broker-owned metadata (now: non-IsNotExist stat errors abort the mutation); four comments trimmed to the three-line rule. Tests and pre-commit clean. ## Packaging branch (broker-packaging): single-host stack in install.sh Step 2 part 2 implemented: `--with-broker` and `--with-library` flags on install.sh, plus docs/site/deployment/single-host.md. - **--with-broker**: downloads demarkus-broker from the tools release (already a goreleaser asset), creates the demarkus-broker system user (joined to the demarkus group so the file store's gid preservation works), scaffolds /etc/demarkus-broker/config.yaml (file backend, generated cookie key + P-256 signing key inline, world `soul` with tokensFile + internalAddress, worldDialer.insecureSkipVerify per TLS mode, commented webClients block for library SSO), OIDC_CLIENT_SECRET in a separate 640 env file, hardened unit (ReadWritePaths = state dir + tokens dir only). Ownership flip: tokens.toml becomes demarkus-broker:demarkus 640 and the tokens dir group-writable, so the broker's atomic rename works unprivileged while the server keeps group-read. Starts only when all four OIDC flags are present; otherwise enabled-but-stopped with printed next steps. - **--with-library**: cross-repo download from latebit-io/demarkus-library releases with checksum verification, env-file config (quic read-only mode on :8090 against the local world; broker/SSO block commented), hardened unit. Works with or without the broker. - Testability seam: SYSTEMD_DIR overridable via DEMARKUS_SYSTEMD_DIR; the unit writers all use it. - Verified on macOS via a function-extraction harness with stubbed privileged ops: install_broker's generated config booted the real broker binary (healthz ok, zero errors); install_library's real download + checksum + extraction produced a binary that served the reading room against a live local world (redirect to /t/u, world content rendered). Gotcha found during verify: local port 8080 was already owned by another Echo app, masquerading as a broker 404; moved the check to a free port. - Docs: single-host page covers the quick start, component table, how file mode works, OIDC app setup, the manual library-SSO wiring (webClients + https redirects), TLS notes, verification commands. Deferred from this branch: docker-compose packaging (broker/library image publication status unverified) and automated library-SSO wiring. ## PR #263 review round (7 findings: 6 accepted, 1 partial) Three were real security/correctness bugs in the packaging: - **Config-dir boundary breach**: granting the broker group-write on all of /etc/demarkus (for atomic tokens.toml renames) would let a compromised broker replace entries in the directory, including the TLS keys (directory write beats file perms). Fixed: with a broker, tokens.toml moves to /etc/demarkus/tokens/, a subdirectory the broker owns (750); existing files migrate on install; CONFIG_DIR perms untouched. - **Verified-TLS dial mismatch**: domain installs set insecureSkipVerify false but dialed localhost:6309, which a domain cert can never verify, so the broker could never reach the world. Fixed: verified mode dials :6309. - **Plaintext listener behind an https publicURL**: the quick start advertised https://example.com:8080 while :8080 was plaintext and ufw-opened. Fixed: both broker listeners bind loopback, no broker ufw rules, reverse proxy documented (Caddyfile two-liner); publicURL examples now proxy origins. Also: OIDC secret gains --broker-oidc-client-secret-file (argv secrets visible in ps/history; file form preferred everywhere in docs); --library-version pins the cross-repo download (broker already rides the stack's tools release version, so the full-manifest ask stays declined per the PR #261 reasoning); scaffold-only installs leave the unit disabled (an enabled placeholder unit would come up broken on every boot) and existing-config installs warn when OIDC flags are ignored; library ufw failure now warns instead of vanishing. Re-verified via both harnesses: self-signed case (loopback binds, localhost dial, skip-verify true) and domain case (domain dial, verify on) both generate configs that boot the real broker binary. ## PR #263 review round 2 Two findings, both fixed: the docs quick-start created the root secret file before sudo (fails for normal operators; now `sudo install -m 600 /dev/null` + `sudo tee`), and the broker tokens layout was not sticky: a later reinstall without --with-broker would repoint the server at the old tokens path while the broker kept writing the new one (split brain, server stops observing broker-issued tokens). The layout detection now runs regardless of the flag: if /etc/demarkus/tokens/tokens.toml exists, every reinstall honors it; disabling broker mode is an explicit documented migration. ## PR #263 review round 3 Three fixed, one declined (third occurrence): - Docs quick-start secret creation switched to `sudoedit` (round 2's `printf | sudo tee` still put the secret on a command line; history/ps leak). - Secret file validation requires a regular file (`-f`): `-r` alone accepts FIFOs/devices and `head` would block the install indefinitely. - Real seam bug from my own SYSTEMD_DIR change: writers used the override but read_existing_config, the harden path, and uninstall still probed the hardcoded /etc/systemd/system, so a non-default override would lose persisted root/domain/TLS on reinstall. All reads now route through SYSTEMD_DIR. - Declined again: stack-wide pinned version manifest (third ask; same reasoning as PR #261). Concession: docs no longer imply --library-version alone gives reproducibility; the pinning paragraph now states exactly what each flag pins and that modules version independently by design. Lesson: when adding a path seam for testability, grep for every reader of the old constant in the same commit; a half-applied seam is worse than none (writers and readers disagree under the override). ## PR #263 review round 4 Three findings, all fixed: - **Incomplete uninstall** (real gap): `demarkus-install uninstall` removed only server artifacts, orphaning the broker/library services, users, binaries, and the broker's credential-bearing config + state dirs. Now uninstall tears down all three components: stops/disables/removes both optional units, removes both binaries, clears BROKER_CONFIG_DIR + BROKER_STATE_DIR + LIBRARY_CONFIG_DIR (state holds refresh tokens, so always cleared), and removes users in dependency order (broker before server, since broker is in the server group). Verified via a do_uninstall harness: all artifacts gone. Docs corrected — the token-migration note had described a partial revert as "full broker removal"; now uninstall is documented as the full teardown and the migration as server-only revert. - **Test-only systemd override was unconditional**: DEMARKUS_SYSTEMD_DIR was honored in production, where an override outside systemd's search path would write units systemctl can't find. Now gated: renamed DEMARKUS_TEST_SYSTEMD_DIR, honored only when DEMARKUS_INSTALL_TESTMODE=1; production is hardcoded /etc/systemd/system. Gate verified both ways. - **Non-root header example**: the single-host usage comment piped to `bash` not `sudo bash`, failing the root check; matched the docs' `sudo bash -s --`. All checks green, harness re-verified. ## PR #263 review round 5 Three fixed: - **Uninstall now fails visibly**: added a remove_path helper that rm's then verifies the target is gone, incrementing an uninstall_errors counter; binary/config/state/unit removals and userdel all route through it (or check userdel's real exit). Uninstall exits non-zero with a summary when anything survives, so a partial teardown can't report "complete." Verified via harness (happy path exit 0, all dirs gone). Docs teardown note updated to state the verify-and-fail behavior. - **Insecure secret file rejected**: -f/-r don't check perms, so a world-readable OIDC secret leaks to local users. Added a stat-based mode+owner check (GNU -c with BSD -f fallback): refuse if group/other bits set (mode & 077), refuse if owner is neither the caller nor root. Verified: 600 accepted, 644 rejected. - **Token migration aborts on failure**: mkdir/mv in the sticky-tokens block were unchecked while tokens_file was reassigned unconditionally; a failed mv would point the server at a missing path. Both now abort the install on failure. Pattern this PR keeps surfacing: shell installers silently swallowing failures (ufw, join-url gen, uninstall rm, secret read, token mv). The reviewer is systematically right each time; the fix is always the same shape (check the result, surface or abort). Worth a patterns.md note. ## Invites dropped as redundant with the broker Fritz asked, before building invites, whether they're still needed now that the broker runs single-host. They aren't. The broker is OIDC-fronted by construction (can't start without an issuer/client/signing key), so any broker deployment already onboards via /knowledge-join + IdP login, which beats an anonymous invite code on identity, revocation, and secret-in-transit. Invites only ever had a home in the brokerless direct-QUIC tier, where the join URL (shipped) already covers ~80%; a full invite flow "in the broker minting core" was a contradiction (a brokerless feature built in the broker). Resolution: the two onboarding tiers are shipped (IdP → broker; no-IdP single/few → soul + join URL). The only open case is multi-user-without-any-IdP, reframed as a demand-gated capability front door on the broker (build only if a real deployment needs it). Plan updated to v8 (step 3 invites + step 4 membership dropped; invite/target-UX sections retained as deprecated-for-the-record), roadmap to v60 (Remote Token Issuance now MOSTLY DONE with the deprecation recorded). No invites code written. broker-invites branch deleted unused. Lesson: question whether a planned feature is still needed after the prior step changes the landscape. Two prior steps (broker single-host) quietly obsoleted the third (invites); catching it before writing code saved a large build. ## File-store broker + library: live integration test Stood up the full single-host stack locally (world server + broker in file-backend mode + library) and drove it end to end. The one seam never exercised before, the OIDC-gated dispatch actually provisioning a world write token into the local tokens.toml, is now proven. Method for the OIDC gate: the broker signs its own bearers (IDTokenSigner, ES256, kid = base64url(sha256(PKIX pubkey)[:8])) and the compositeVerifier accepts them, so I minted a broker-signed id_token with the config's signing key via a scratch go-jose program. That exercises the identical gateway-auth + dispatch + provision path a real IdP login reaches; only the IdP redirect/device front-end is bypassed. Verified live: - Broker boots file mode against a real issuer's discovery (Google), healthz ok, zero k8s. - Broker-signed bearer authenticates the MCP gateway (StreamableHTTP: initialize -> session -> tools/call). - mark_fetch mark://soul/index.md through the gateway returns the world doc (read path; public reads need no token). - mark_publish mark://soul/from-broker.md through the gateway: the broker provisioned a write token, appended [tokens.broker-write-soul] to the local tokens.toml live, the server hot-reloaded and honored it, and the doc landed (read back directly over QUIC). - Library boots quic mode against the world; /d/index.md renders "Stack Test"; /d/from-broker.md renders the broker-written doc. Full loop: broker writes via file store, library reads it back. Not exercised live (need real IdP / browser): the actual OIDC login front-end (device flow / web redirect) and the library in broker mode (confidential-client web SSO). Both sit on top of the now-proven gateway; the auth front-end is unchanged from the k8s broker and covered by the broker's own Go tests. Verdict: the file-store broker and the library both work. Scratch artifacts under scratchpad/stack (config, minter, logs).