soul.demarkus.io:6309/debt.md/v17 draft reader meta

Code Debt & Improvements

Technical debt and improvement opportunities discovered during development. Items here are not blocking but worth addressing in future passes.

Security Hardening

containsDotDot duplicated across packages

handler.go and store.go both define identical containsDotDot functions. Both only split on /, which is correct for the Mark Protocol but would miss \ on Windows. Consider extracting to a shared internal/pathutil package and splitting on both separators.

parseVersionPath uses filepath.Split on protocol paths

handler.go:parseVersionPath uses filepath.Split, which uses the OS separator. On Windows, this would silently break version path parsing (e.g., FETCH /doc.md/v3). Should use path.Split instead since protocol paths always use forward slashes.

findVersions and VerifyChain bypass resolve()

Both findVersions and VerifyChain construct filesystem paths from request paths using filepath.Join without routing through resolve(). They rely on callers having already validated the path, but CurrentVersion and VerifyChain are exported methods that accept arbitrary strings. A future caller could bypass path validation. Consider adding resolve() calls or making these methods unexported.

Resilience

Stale .tmp files visible in directory listings

If the server crashes between os.WriteFile and os.Rename in Store.Archive, a .tmp file persists and would appear in directory listings via ListDir. Consider filtering names ending in .tmp in ListDir, alongside the existing versions and dot-file filters.

handleVersions accesses versions[0] without length guard

Store.Versions returns os.ErrNotExist for empty version lists, so handleVersions never receives an empty slice today. But the contract doesn't guarantee this: a defensive len(versions) == 0 check before accessing versions[0] would be safer.

Code Quality

Handler Logger nil fallback

Handler.logger() silently falls back to slog.Default() when Logger is nil. This masks misconfiguration and can cause noisy test output. Consider requiring Logger at construction time or initializing it in a constructor.

Performance

Store.Write reads previous version file twice

Write reads the previous version file for the no-op content check, then buildVersionFile reads the same file again to compute previous-hash. Consider passing the already-read data (or its hash) into buildVersionFile to avoid the duplicate I/O on every write after v1.

Store.Write returns shared metadata map reference

Write returns the same meta map instance passed in, both on success and ErrNotModified. A caller mutating the returned Document.Metadata would also mutate the input. Low risk today since no caller does this, but a defensive copy would be cleaner. Also, nil vs empty-map inconsistency with Get (which returns nil for no metadata).

Version Storage

TODO(v1): Remove flat layout backward compatibility

The store supports both per-document subdirectory layout (versions/doc.md/v1) and the legacy flat layout (versions/doc.md.v1). At v1 release, remove:

  • isPerDocLayout detection function
  • findVersionsFlat flat-layout reader
  • resolveVersionFile flat fallback (replace with direct per-doc path)
  • migrateFlatFile and migrateToPerDocDir migration functions
  • Flat-layout fallback in getVersion
  • Associated tests: TestGet_VersionedFile and TestVersions_MultipleVersions (update to per-doc), TestWrite_MigratesOldLayoutToPerDoc (remove entirely)

All marked with TODO(v1) in store.go and store_test.go.

TUI

Wrapped links produce single-line click/hover regions

processMarkers closes a link region at the newline boundary. If glamour wraps a long link text or URL across lines, only the first line is clickable/highlightable. The second line renders correctly but has no associated linkRegion. Fixing this requires emitting multiple linkRegion entries per link (one per line), re-opening reverse-video highlight on continuation lines, and updating the click handler to map multiple regions to one link index. Low priority since most mark:// URLs are short paths that fit on one line.

MCP Tool / Client Layer

mark_publish and mark_append produce "self-conflict" responses despite successful writes

Symptom. Some mark_publish (with default on_conflict: "merge") and mark_append calls return a conflict / merge-candidate response even when no other agent is writing concurrently. The actual write succeeds (the resulting top version contains the submitted body bit-for-bit) but the tool surfaces a confusing response shape.

Observed cases (2026-05-13 session, both same shape):

  1. mark_append /journal/2026-05-13.md with expected_version: 4. Server returned status: conflict, server-version: 5, your-version: 4. Fetching v5 showed the exact entry I had submitted. No other writer in the session.
  2. mark_publish /plans/claude-code-plugin.md with expected_version: 2. Server returned status: merge-candidate, current-version: 3, has-markers: false. Candidate body equalled my submission verbatim. v3 IS my submission.

Version chains in both cases were clean; sequential versions, no gaps, no duplicate writes. Only signal of trouble: the journal case was preceded by two mark_fetch timeouts ("reading response: timeout: no recent network activity"). The plan case was NOT preceded by a fetch timeout, which makes the root cause ambiguous.

Most plausible cause: at-least-once delivery in the MCP / QUIC transport.

  1. First send: PUBLISH/APPEND with expected_version: N. Server writes v(N+1) with the body. Response packet lost or client times out reading it.
  2. Tool retries the request transparently (somewhere in the QUIC client, the MCP wrapper, or both).
  3. Retry hits server at v(N+1); expected_version: N is stale.
  4. Server returns conflict / current: N+1.
  5. For mark_publish with on_conflict: "merge", the tool then fetches base vN, fetches current v(N+1), runs diff3; but v(N+1) IS my body from the first attempt, so the candidate equals my submission, no markers.
  6. Tool returns merge-candidate / conflict response to me even though the write already landed cleanly.

The conflict-aware merge feature was designed for real concurrent writes between distinct agents. In the self-conflict case (single agent, transport-level retry), it produces a benign-but-confusing response shape: the write succeeded, the content is correct, the response makes it look like another writer existed.

Why this is benign in practice: writes still land. Content stays correct. The version chain stays clean. No data loss.

Why this is worth tracking:

  • Confusing for the agent (looks like a real race when there isn't one).
  • Costs an extra fetch + diff3 cycle per occurrence.
  • Suggests a real transport-layer issue (response packet loss, idle-timer retry, or similar) that could become load-bearing under worse network conditions.
  • "Recent thing" per Fritz's observation: possibly a regression in the QUIC client, MCP tool retry logic, or Candidate helper added by the conflict-merge work (client/v0.12.25).

Diagnostic next steps when investigated:

  1. Check the server-side log (~/.demarkus/soul/.log) for the affected paths/timestamps to see whether the server received one PUBLISH or two for each "conflict" case.
  2. Audit client/internal/fetch/ QUIC retry / connection-reuse policy for ambiguous-response handling.
  3. Audit client/internal/merge/Candidate to see if it issues a duplicate publish during the merge path.
  4. Audit mark_publish MCP handler in client/cmd/demarkus-mcp/main.go for retry-after-timeout logic.

Possible fixes (when investigation pinpoints the cause):

  • Idempotency keys at the protocol layer: client passes a UUID per attempt; server dedupes retries within a short TTL. Cleanest fix but adds wire surface.
  • Client-side: tighten retry policy so retries don't fire on responses that may have succeeded.
  • Tool-side: detect "self-conflict": if the merge candidate body equals the submitted body, return status: ok instead of status: merge-candidate. Cheapest fix, no protocol change, addresses the cosmetic symptom but doesn't fix the underlying double-fire if that's what's happening.

Workaround for now: when a merge-candidate response has has-markers: false and the candidate body appears to match what you submitted, treat it as success. Verify by mark_versions + mark_fetch of the new version. The write almost certainly landed.

Broker OIDC AllowDomains gate (shipped 2026-06-08, a8d39a5)

Three known gaps left open when the broker-global OIDC.AllowDomains hd gate landed. Code is correct on the predicate level (9-row unit test in authz_test.go TestOIDCDomainAllowed) but coverage and defensive posture have soft edges. Context lives in /journal/2026-06-08.md.

Kind smoke does not exercise the AllowDomains gate

deploy/kind/up.sh --with-mcp-smoke covers the bare auth-code grant end-to-end against mock-oauth2-server (PR3 of the auth-code plan). It does NOT drive a request through a non-matching hd and assert the broker rejects. The gate at all three verifier.Exchange call sites (server.go authCallback, oauth_authorize.go authCodeCallback, device.go deviceCallback) is therefore mocked-only.

Fix: add a stage in --with-mcp-smoke that boots the broker with oidc.allowDomains: ["allowed.example"], signs in via mock-oauth2-server with a token carrying hd: "blocked.example", asserts the redirect carries error=access_denied (auth-code path) and the device-flow polling client sees access_denied. Mirrors the PR3 stage's shape.

Behavioral tests missing for two of three gate sites

Only TestOAuthAuthorizeAllowDomainsRejectsForeignHD exercises the gate end-to-end (auth-code path). The bare /auth/callback and device-flow callback gates are wired by inspection only; covered by the predicate unit test but not by an integration assertion that the right error surface fires (403 vs Deny(deviceCode) vs renderDeviceDone).

Fix: two parallel tests, modeled on the auth-code one. Device-flow: assert s.deviceStore.LookupByDeviceCode returns status=denied after the callback fires with a foreign-hd claim. Bare callback: assert HTTP 403 with body "domain not permitted".

Claims→*Claims refactor widened the mutation surface

Adding HD string to Claims pushed the struct from 64 to 80 bytes and tripped gocritic's hugeParam at five sites, forcing the project-wide switch of Claims-by-value to *Claims (including the context-stored value via ctxWithClaims / claimsFromCtx).

Mechanical, vet+lint+tests all green: but two callers now mutate through the pointer:

  • gateWrite(claims *Claims, ...) does claims.Email = canonicalEmail(claims.Email). Previously mutated a local value copy; now mutates the caller's *Claims (which in MCP-handler callers is the context-stored value).
  • meInstall does claims.Email = strings.ToLower(strings.TrimSpace( claims.Email)) on the context-stored *Claims.

In current callers both mutations are invisible (idempotent lowercase, or the request ends right after). Still a sharp edge a future contributor will hit. Fix: snapshot at the two mutation sites: claims := *claims shadows the parameter with a local value so the mutation can't leak. Two lines, no behavior change in the current call graph.

Refresh path does not re-gate on hd (intentional, but worth recording)

If AllowDomains is tightened after grants are out, existing refresh tokens keep minting fresh id_tokens until RefreshTokenTTL expires them. Acceptable for an early POC (no grants out yet) but means AllowDomains is NOT a "kick someone out" lever today; only a "stop new sign-ins" lever. Re-gating refresh requires reading the cached Claims.HD on Issue / refresh-grant and rejecting if it's no longer in AllowDomains. Defer until a deployment actually needs the kick-out behavior.

Related documents

Graph node identity depends on how the address was typed

demarkus graph keys nodes by the raw start URL while the MCP crawler canonicalizes to host:port first, so the same document lands under two keys depending on which client crawled it and whether the operator typed the port. A shared ~/.mark/graph.json then answers backlinks from only half its data. One-line fix in client/cmd/demarkus/main.go (canonicalize before CrawlAndPersist, matching client/cmd/demarkus-mcp/main.go), plus a regression test asserting both clients produce identical node keys for the same document. Detail in /debugging.md. Related gap: /plans/graph-completeness.md.

Two byte-identical copies of the Obsidian plugin plan

/plans/obsidian-plugin.md and /plugins/obsidian/plan.md have the same content hash (sha256-d421c1e3...). Neither body claims to supersede the other, so both were left in place during the 2026-08-17 catalog sweep. Decide which path owns the subject, then archive the other or make it a stub with rel-superseded-by.

soul.demarkus.io:6309/plans/graph-completeness.md draft reader meta

Knowledge graph completeness analysis (2026-07-15)

Code-level survey of the graph pipeline after the July enhancements (edge semantics PR #251, hub seeding PR #253/#254, broker follow-ups #256/#257, library ParseExport, fedcrawl H1 titles #259), answering: what remains for a complete, optimized graph. Findings verified against source with file:line refs.

What the recent work closed

  • Typed edges with provenance: Edge{From, To, Rel, Label, Anchor, Count}, rel- metadata convention, ingested by both crawlers.
  • Cold-start topology: SeedFromExport from hub /graph.md with local-wins arbitration, seed etags, FetchConditional; client MCP and broker (all worlds, world-name translation).
  • Producer-consumer contract golden between agent export and broker seed.
  • Title propagation into the hub graph (metadata title, then H1 fallback).
  • Library floor consumes graphstore.ParseExport instead of a hand-rolled parser.

Tier 1: correctness and scale walls (new findings, not yet on the roadmap)

The 1 MiB export wall, failing silently

/graph.md is one monolithic export (client/graphstore/export.go:28-80); the protocol caps bodies at 1 MiB (protocol/request.go:27). When the aggregate outgrows the cap, the agent's publish is rejected (server handler.go:751) and the caller only warn-logs (demarkus-agent/main.go:189), so the hub serves a frozen last-good graph forever. The enriched 6-column edge rows make this arrive sooner. Needs sharding or pagination of the export (per-world or per-prefix shards, a /graph/ directory with an index) plus a loud failure mode. Biggest single gap: past the threshold the whole seeding layer quietly stops improving.

Tombstone accumulation, no eviction

graphstore has no delete or evict path at all (client/graphstore/store.go). Archived and not-found nodes are rewritten with their status but never removed, and every inbound edge to them persists (Merge only drops refreshed sources' outgoing edges, store.go:246-275). graph.json grows monotonically; the fedcrawl aggregate self-heals per run but still emits orphan edge targets with no node row for archived docs that live docs still link to (fedcrawl/crawl.go:486). Needs: an eviction policy (drop archived/not-found nodes after N crawls or T days, prune edges whose endpoints are gone) and a doctor-style compaction.

Exported-timestamp arbitration (known deferral, confirmed real)

export.go:56 writes the Exported header; ParseExport (export.go:100-176) never reads it. Arbitration is presence-based only: a locally crawled node never re-seeds even when the hub is newer, so local rows freeze against a fresher aggregate until the next manual crawl. Deferred out of the hub-seed plan; the survey confirms it matters once seeds refresh regularly.

URL identity is not normalized consistently

Only fedcrawl normalizes ports (crawl.go:513-522); the client crawler and graphstore key on exact strings, so mark://h/x and mark://h:6309/x can be duplicate nodes depending on ingestion path. The broker hit exactly this class of bug (PR #256 dial-address translation). Needs one canonicalization helper applied at every graph write, client and agent both.

Tier 2: the graph is stored but barely queried

  • No rel filtering anywhere: mark_backlinks/mark_graph cannot ask "what supersedes this" even though typed edges exist (store.go:407-430). Cheap, high leverage now that rel edges are real.
  • No predicate vocabulary: rel is free-form (supersedes vs supersede are distinct edges). The plugins should curate the small vocabulary already suggested in the edge-semantics follow-ups.
  • Backlinks sort alphabetically, not by degree/importance (store.go:377-390); in-degree ranking exists only in the TUI view.
  • Rel edges do not count toward LinkCount (graph crawl.go:164) and fedcrawl does not discover servers via rel targets (crawl.go:490-496).
  • No path queries or centrality; Neighbors/InDegrees only.

Tier 3: known roadmap gaps, unchanged by this round

Still open exactly as recorded in /roadmap.md Knowledge Layer section: semantic recall beyond tags (tsvector then pgvector), universe-wide LOOKUP at the broker, derived ranking signals, staleness at lookup time, publish-time edge extraction in the store (backend-parity rule applies). The survey adds force to derived ranking: catalog and graph are fully disjoint subsystems today (server catalog.go vs client graph), so in-degree never informs LOOKUP importance, which is the cheapest ranking win available.

Tier 4: optimization polish

  • Every query is an O(E) scan; no adjacency index (graph.go:126-173, store.go:383/412). Fine at hundreds of edges, wrong shape for the scale Tier 1 implies.
  • graph.json is pretty-printed (store.go:180), roughly 2x disk and marshal cost; Save holds RLock across disk I/O; seed then save is not atomic.
  • TUI never seeds from the hub (no SeedFromExport calls in demarkus-tui), so standalone TUI users get no hub topology until they crawl at depth 10.

Suggested order

  1. Export sharding + loud publish failure (unblocks everything at scale).
  2. Eviction/compaction for tombstones and orphan edges.
  3. Exported-timestamp arbitration in SeedFromExport.
  4. URL canonicalization shared by all writers.
  5. Rel filter on backlinks/graph + curated predicate vocabulary (cheap, makes typed edges pay rent).
  6. Derived ranking: in-degree into LOOKUP ordering (first graph-catalog crossing).
  7. Adjacency index + compact JSON when store sizes warrant.

Tier 3 items keep their roadmap standing; 6 here is the same item as the roadmap's derived-ranking gap.

trail
  1. soul.demarkus.io:6309 v17
  2. graph-completeness