soul.demarkus.io:6309/journal/2026-07-17.md/v18 draft reader meta

2026-07-17: join URLs + cert pinning (remote-issuance step 1)

Implemented step 1 of /plans/remote-token-issuance.md on branch join-string: the no-sidecar prefix (join-string grammar + TOFU cert pinning), after the Option C sidecar decision landed in the plan (v3) and roadmap (v58).

What shipped

  • client/joinurl (new package): Build/Parse of mark://host#token=...&fp=sha256:<hex> join URLs, cert fingerprint helpers (DER, PEM), VerifyPin TLS verifier. Fragment carries the credentials; it never appears in any protocol request.
  • client/fetch: Options.Pins (host to fingerprint map). A pinned host gets InsecureSkipVerify plus VerifyPin at dial time, per host, so a multi-host crawler client pins only what it should. Wire tests: right pin, wrong pin (handshake failure with re-join hint), bare-hostname fallback, unpinned hosts still CA-verify.
  • client/internal/tokens: entry gains pin field (tokens.toml, additive), SetPin/Pins/ResolvePin (flag > DEMARKUS_PIN > stored).
  • All three clients wire stored pins; demarkus-mcp gains -pin; new demarkus join '<join-url>' stores token + pin in one paste.
  • demarkus-token join subcommand: composes with generate via pipe, -cert embeds the fingerprint.
  • install.sh: summary now prints a ready-to-paste /soul-join <join-url> and demarkus join line.
  • Plugin: registry soul-join accepts a join URL (fragment supplies token + pin), souls catalog gains a 5th pin column (all readers were length-guarded), mcp-serve passes -pin instead of -insecure for pinned souls. commands/soul-join.md updated; plugin 0.12.16 to 0.13.0.
  • Docs: docs/site/tools/index.md join-URL section.

Decisions and gotchas

  • Never pin Let's Encrypt certs: leaves rotate on renewal (install.sh runs certbot twice daily) and would break every pinned client. install.sh embeds fp= only for custom certs; LE domains pass CA verification anyway. The dev self-signed cert has no stable file, so that path falls back to --insecure with a note.
  • Go url fragment gotcha: u.Fragment is pre-decoded; parsing it with ParseQuery corrupts + and % in tokens. Use u.EscapedFragment(). Conversely url.Values.Encode escapes : to %3A, uglifying the paste string; the fragment is hand-built (pin literal, token QueryEscaped).
  • Pin resolution mirrors token resolution exactly (explicit > env > stored), per the shared-behavior guideline; pins ride tokens.toml instead of a new file.
  • mcp-serve precedence: pin wins over insecure (pin verifies exactly one cert; insecure verifies nothing).

Verified live

Self-signed server on :6399: demarkus-token generate | join -cert produced the URL, demarkus join stored both credentials, publish + fetch ran with no --insecure, and a corrupted pin failed at the QUIC handshake with the actionable mismatch error. Plugin path: registry soul-join <join-url> wrote the 0600 token file, pinned=1, catalog row carries the pin.

Full make test + bash pre-commit.sh clean. Branch ready for PR; binary pin bumps ride the release automation after merge.

Follow-up: self-signed to CA migration

Question surfaced the scenario: server moves from a pinned self-signed cert to a CA cert. Pinned clients hard-fail (strict pin, by design); recovery is re-joining with a pinless join URL. The plugin path already handled it (catalog row upsert clears the pin), but demarkus join only wrote a pin when the URL carried one, so a stale pin survived re-join and kept failing. Fixed in the same branch: re-join re-states the trust material, a token-bearing URL without fp= clears the stored pin (with a stderr notice when one existed). Verified live against the e2e server. Lesson: when a credential store is updated by re-stating, absence of a field must overwrite, not preserve; the two join paths (upsert-row vs set-field) diverged on exactly that.

Cut: pinning removed, branch ships token-only

Fritz was uneasy about the direction; the complexity audit agreed with him. The join URL half is ~100 lines of formatting with zero trust semantics; the pinning half carried a third trust state, rotation coupling, a migration bug (caught pre-merge), mixed-version edges, and hand-rolled VerifyPeerCertificate. Every follow-up question in review prep was a pinning question. HPKP precedent (browsers added then removed pinning in 2018) and the narrow constituency (only domainless hosts; LE covers the rest) settled it.

Cut executed on the branch: joinurl is token-only (mark://host#token=..., unknown fragment keys still fail loudly so future keys cannot silently drop), fetch/tokens/CLI/TUI/MCP pin code removed, souls catalog back to 4 columns, demarkus-token join drops -cert, install.sh join line token-only with an --insecure note for self-signed, both plugin docs and site docs updated. Full test suite + pre-commit clean; token-only e2e re-verified live (CLI join, publish, plugin soul-join with fragment token).

Decision recorded in the plan (v4): pinning deferred; if verified joins for domainless hosts return, they ride the sidecar redeem handshake (fingerprint out of band in the redeem response), not user-visible trust state. Plan sequencing also revised: sidecar redeem-only demoted; authenticated create is the sidecar MVP.

Lesson: when a feature generates a stream of "what happens when..." questions before it ships, that is the complexity bill arriving early; the severable half that causes them is the half to cut.

PR #261 review round (CodeRabbit)

Eight findings fixed, two declined:

  • Fixed: joinurl.Parse rejects URL state Join cannot represent (userinfo, query, duplicate token keys) and bracketed IPv6 literals (downstream slug derivation and default-port append would misparse them); Build rejects @ and [ in hosts. Negative tests for all, plus a SoulJoin regression for bracketed IPv6.
  • Fixed: install.sh no longer swallows demarkus-token join failures with 2>/dev/null || true; a failure warns with the manual command.
  • Fixed: RemoteSoulRow returns an error for a catalog row with no host instead of ok=true (mcp-serve would have launched with an empty -host); mcp-serve surfaces catalog I/O errors separately from "not in the catalog".
  • Fixed: demarkus-token join bounds stdin with a LimitReader (4KB) so piping a wrong file cannot become a "token".
  • Fixed: joinurl package comment tightened to 3 lines; new test's MkdirAll error handled.
  • Fixed: pi-memory soul-join doc no longer shows a token-bearing join URL as slash-command input (contradicted pi's own token-in-transcript rule); credentialed joins go in the user-run terminal command.
  • Declined: checking fmt.Fprintf errors on stderr usage output; every existing subcommand's usage block uses the same unchecked idiom and each is followed by os.Exit(1).
  • Declined: repo-wide IPv6 support in fetch.ParseMarkURL / TLS SNI; pre-existing behavior shared by every caller, not introduced by this PR. The join paths now fail loudly on IPv6 literals so nothing broken can be stored; full IPv6 support filed as follow-up.

PR #261 review round 2 (outside-diff + late inline)

Two fixed, one declined:

  • Fixed: port-only authorities rejected in joinurl (Build refuses a leading-colon host, Parse requires a non-empty hostname); regression cases both paths.
  • Fixed: SoulJoin now routes every non-HTTPS host through joinurl.Parse, not only fragment-bearing ones, so malformed legacy inputs (paths, userinfo, query, IPv6 literals) fail loudly instead of half-parsing into a broken catalog row. Behavior change accepted deliberately: inputs like mark://host/path previously "succeeded" by storing a corrupt host. Regression cases added.
  • Declined: unifying server/tools version resolution in install.sh into one pinned manifest. Independent per-component latest-release resolution is the installer's pre-existing pattern for every component; the skew window for the join subcommand is the gap between merging and the tools release publishing, and the failure mode after round 1's fix is a visible warning with the manual command. A single release manifest is a real installer improvement but a separate effort, not this PR.

PR #261 review round 3

One finding, fixed: joinurl gains a shared validateHostPort rule (non-empty hostname, port numeric 1-65535, max 5 digits) enforced by both Build and Parse. Previously Build emitted h:bad and h:65536, and Parse accepted numeric out-of-range ports (Go's net/url validates digits only, not range). Subsumes the round-2 port-only-authority check. Negative tests: h:bad, h:65536, h:0, port-only, both paths.

PR #261 review round 4

One finding, fixed: validateHostPort's hostname check flipped from a rune blocklist (which kept missing cases: spaces, stray brackets) to a DNS-charset allowlist ([a-zA-Z0-9._-]). Subsumes the previous ContainsAny("/#?@[") check entirely; one rule now covers every malformed-host class from rounds 1-4. Tests: "h x" and "h]x" rejected. Lesson: for input validation on a credential-bearing string, start with an allowlist; every blocklist round-trip with a reviewer is a missed-rune bug you shipped.

Merged

PR #261 merged to main 2026-07-17 as commit 4c5f90d (squash) after four CodeRabbit rounds. Local branch deleted; main pulled. Plan doc updated to v5 (sequencing step 1 DONE with merge refs). The release chain takes it from here: client/tools releases, then the automated pin-bump PR repoints the plugins at binaries that carry the join subcommand and join-URL SoulJoin (until that lands, a new install.sh with an old tools release warns and falls back to the manual command). Next up per the plan: the sidecar, redeem + authenticated create together.

Parity decision: the sidecar is the broker in single-host mode

Fritz set a parity goal: personal, team, and non-k8s deployments get the same library + broker experience as the k8s universe, including invites. Analysis showed the broker's only real k8s bindings are the Secrets writer (becomes a tokens.toml writer; the server already file-watch reloads) and packaging (becomes systemd/compose); OIDC works fine off-cluster via a Google/GitHub OAuth app, and the library only knows a broker URL. Consequence: no new sidecar binary; the remote-token-issuance plan (v6) now sequences broker single-host mode first (k8s audit, file backend, packaging), then invites in the broker minting core for both backends. The one open design question is identity for the zero-external-dependency personal tier; leaning capability-token front door before any built-in IdP. Convergence moved from the last step to the first.

Broker k8s audit done

Findings appended to /plans/remote-token-issuance.md (v7). Headline: single-host mode is one storage interface (~5 files + main.go wiring); every Secret mutation already funnels through a single mutateSecret function. Three surprises: the broker has no SIGHUP at all (the documented "SIGHUP on revocation" never existed; k8s reload is Secret-mount projection, and on a VPS the server's own tokens-file watcher covers it, faster); write-token revocation is unimplemented (real new work for the membership step); and Provision runs on the read path, so the backend is hit by first read. Roadmap's Phase 6.3 description needs the SIGHUP claim corrected when next touched.

Broker storage interface (branch storage-interface)

Step 2 part 1 of the remote-token-issuance plan implemented: the broker's k8s surface is now behind a SecretStore interface.

  • New internal/broker/storage.go: SecretRef (k8s tuple + file path, resolved by Config helpers so every call site locates credentials one way), SecretStore interface, k8sSecretStore (the old mutateSecret body verbatim), fileSecretStore (per-path in-process locking, atomic temp+rename preserving an existing file's mode, absent-stays-absent and no-rewrite-on-equal contracts preserved so the world server's dir-watcher sees exactly one complete replacement per real change).
  • RefreshStore, worldWriteTokenStore, NewServer all take SecretStore; the audit's prediction held (one interface, ~5 files). ~30 test call sites converted mechanically to NewK8sSecretStore wrapping.
  • Config: storage: {backend: kubernetes|file, dir} (defaults kubernetes, fully back-compat), WorldConfig.TokensFile; file mode requires dir + per-world tokensFile + internalAddress and drops the namespace/Secret requirements; validateStorage split out for gocyclo.
  • main.go: file mode skips k8s client construction entirely; Sweeper gains Run (no leader election) for single-instance deployments.
  • Tests: file-store contract suite (create/update, absent-stays-absent, no-rewrite-on-equal via mtime, mode preservation 0640, mutate-error no-write, concurrent no-lost-updates), config validation both modes, and TestProvisionFileBackend pinning the seam: Provision mints, hash lands in the world's local tokens.toml, broker-restart re-provision converges on the same token without rewriting the file.
  • Verified live: broker binary boots on a plain host with backend: file (healthz ok, single-host sweeper, zero k8s references in logs); and the server hot-reload seam, a rename-replaced tokens.toml (the file store's write style) authorizes a publish with the new token immediately, no restart.

Remaining for step 2: non-k8s packaging (systemd/compose, install.sh --with-broker --with-library) and a full OIDC-in-the-loop e2e. Refresh tokens persist to storage.dir/refresh_tokens.json via the same interface, so VPS restarts keep sessions.

PR #262 review round (CodeRabbit, 9 findings, all accepted)

Strong round; the file store is credential infrastructure and three findings were production-grade catches:

  • Ownership preservation: CreateTemp+rename re-owned the tokens file to the broker user and its primary group, silently revoking the world server's group-read on the first mutation. Fixed with chownPreserving: restore uid+gid; on EPERM fall back to gid-only (an unprivileged broker can always keep a group it belongs to); error with an actionable message if neither works. Platform split via storage_unix.go/storage_other.go (syscall.Stat_t).
  • Directory fsync after rename: file contents were fsynced but the rename's directory entry was not; a crash could acknowledge a credential mutation and boot with the old file. Added syncDir.
  • Path collision validation: a world tokensFile aliasing storage.dir state files (or another world's file) would have two stores mutating one document; validateFilePaths rejects with filepath.Clean comparison, tested including a non-clean alias.

Also fixed: temp-file cleanup errors joined into the returned error instead of discarded; RefreshStore gained a file-backend lifecycle test (issue, restart-with-fresh-store, refresh, revoke) proving backend neutrality and restart persistence; concurrency test now surfaces every goroutine's Mutate error; no-rewrite test asserts os.SameFile in addition to mtime (mtime alone passes falsely within timestamp resolution); stale kubernetes-client comment on Server.refreshStore updated; three verbose comments trimmed to invariants.

Lesson (file store): atomic-rename file replacement has three durability/identity obligations beyond the rename itself: preserve mode AND ownership, fsync the parent directory, and treat cleanup failure as an error. Each is invisible in tests on a friendly filesystem and bites in production.

PR #262 review round 2

Three findings, all fixed: a discarded os.Stat error in the file store could have replaced an existing-but-unstattable shared tokens file with downgraded 0600/broker-owned metadata (now: non-IsNotExist stat errors abort the mutation); four comments trimmed to the three-line rule. Tests and pre-commit clean.

Packaging branch (broker-packaging): single-host stack in install.sh

Step 2 part 2 implemented: --with-broker and --with-library flags on install.sh, plus docs/site/deployment/single-host.md.

  • --with-broker: downloads demarkus-broker from the tools release (already a goreleaser asset), creates the demarkus-broker system user (joined to the demarkus group so the file store's gid preservation works), scaffolds /etc/demarkus-broker/config.yaml (file backend, generated cookie key + P-256 signing key inline, world soul with tokensFile + internalAddress, worldDialer.insecureSkipVerify per TLS mode, commented webClients block for library SSO), OIDC_CLIENT_SECRET in a separate 640 env file, hardened unit (ReadWritePaths = state dir + tokens dir only). Ownership flip: tokens.toml becomes demarkus-broker:demarkus 640 and the tokens dir group-writable, so the broker's atomic rename works unprivileged while the server keeps group-read. Starts only when all four OIDC flags are present; otherwise enabled-but-stopped with printed next steps.
  • --with-library: cross-repo download from latebit-io/demarkus-library releases with checksum verification, env-file config (quic read-only mode on :8090 against the local world; broker/SSO block commented), hardened unit. Works with or without the broker.
  • Testability seam: SYSTEMD_DIR overridable via DEMARKUS_SYSTEMD_DIR; the unit writers all use it.
  • Verified on macOS via a function-extraction harness with stubbed privileged ops: install_broker's generated config booted the real broker binary (healthz ok, zero errors); install_library's real download + checksum + extraction produced a binary that served the reading room against a live local world (redirect to /t/u, world content rendered). Gotcha found during verify: local port 8080 was already owned by another Echo app, masquerading as a broker 404; moved the check to a free port.
  • Docs: single-host page covers the quick start, component table, how file mode works, OIDC app setup, the manual library-SSO wiring (webClients + https redirects), TLS notes, verification commands.

Deferred from this branch: docker-compose packaging (broker/library image publication status unverified) and automated library-SSO wiring.

PR #263 review round (7 findings: 6 accepted, 1 partial)

Three were real security/correctness bugs in the packaging:

  • Config-dir boundary breach: granting the broker group-write on all of /etc/demarkus (for atomic tokens.toml renames) would let a compromised broker replace entries in the directory, including the TLS keys (directory write beats file perms). Fixed: with a broker, tokens.toml moves to /etc/demarkus/tokens/, a subdirectory the broker owns (750); existing files migrate on install; CONFIG_DIR perms untouched.
  • Verified-TLS dial mismatch: domain installs set insecureSkipVerify false but dialed localhost:6309, which a domain cert can never verify, so the broker could never reach the world. Fixed: verified mode dials :6309.
  • Plaintext listener behind an https publicURL: the quick start advertised https://example.com:8080 while :8080 was plaintext and ufw-opened. Fixed: both broker listeners bind loopback, no broker ufw rules, reverse proxy documented (Caddyfile two-liner); publicURL examples now proxy origins.

Also: OIDC secret gains --broker-oidc-client-secret-file (argv secrets visible in ps/history; file form preferred everywhere in docs); --library-version pins the cross-repo download (broker already rides the stack's tools release version, so the full-manifest ask stays declined per the PR #261 reasoning); scaffold-only installs leave the unit disabled (an enabled placeholder unit would come up broken on every boot) and existing-config installs warn when OIDC flags are ignored; library ufw failure now warns instead of vanishing.

Re-verified via both harnesses: self-signed case (loopback binds, localhost dial, skip-verify true) and domain case (domain dial, verify on) both generate configs that boot the real broker binary.

PR #263 review round 2

Two findings, both fixed: the docs quick-start created the root secret file before sudo (fails for normal operators; now sudo install -m 600 /dev/null + sudo tee), and the broker tokens layout was not sticky: a later reinstall without --with-broker would repoint the server at the old tokens path while the broker kept writing the new one (split brain, server stops observing broker-issued tokens). The layout detection now runs regardless of the flag: if /etc/demarkus/tokens/tokens.toml exists, every reinstall honors it; disabling broker mode is an explicit documented migration.

PR #263 review round 3

Three fixed, one declined (third occurrence):

  • Docs quick-start secret creation switched to sudoedit (round 2's printf | sudo tee still put the secret on a command line; history/ps leak).
  • Secret file validation requires a regular file (-f): -r alone accepts FIFOs/devices and head would block the install indefinitely.
  • Real seam bug from my own SYSTEMD_DIR change: writers used the override but read_existing_config, the harden path, and uninstall still probed the hardcoded /etc/systemd/system, so a non-default override would lose persisted root/domain/TLS on reinstall. All reads now route through SYSTEMD_DIR.
  • Declined again: stack-wide pinned version manifest (third ask; same reasoning as PR #261). Concession: docs no longer imply --library-version alone gives reproducibility; the pinning paragraph now states exactly what each flag pins and that modules version independently by design.

Lesson: when adding a path seam for testability, grep for every reader of the old constant in the same commit; a half-applied seam is worse than none (writers and readers disagree under the override).

PR #263 review round 4

Three findings, all fixed:

  • Incomplete uninstall (real gap): demarkus-install uninstall removed only server artifacts, orphaning the broker/library services, users, binaries, and the broker's credential-bearing config + state dirs. Now uninstall tears down all three components: stops/disables/removes both optional units, removes both binaries, clears BROKER_CONFIG_DIR + BROKER_STATE_DIR + LIBRARY_CONFIG_DIR (state holds refresh tokens, so always cleared), and removes users in dependency order (broker before server, since broker is in the server group). Verified via a do_uninstall harness: all artifacts gone. Docs corrected — the token-migration note had described a partial revert as "full broker removal"; now uninstall is documented as the full teardown and the migration as server-only revert.
  • Test-only systemd override was unconditional: DEMARKUS_SYSTEMD_DIR was honored in production, where an override outside systemd's search path would write units systemctl can't find. Now gated: renamed DEMARKUS_TEST_SYSTEMD_DIR, honored only when DEMARKUS_INSTALL_TESTMODE=1; production is hardcoded /etc/systemd/system. Gate verified both ways.
  • Non-root header example: the single-host usage comment piped to bash not sudo bash, failing the root check; matched the docs' sudo bash -s --.

All checks green, harness re-verified.

trail
  1. soul.demarkus.io:6309 v18