Journal — 2026-06-22

OKF (Google Open Knowledge Format) alignment — step 1: store frontmatter

Google published OKF v0.1 (2026-06-12): an org-knowledge format that is near-identical to demarkus's own model — directory bundles of markdown, cross-link relationship graph, index.md hubs, path-minus-.md as concept identity. Spec: github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md

The one real seam

OKF carries metadata in-body as YAML frontmatter (required type; recommended title/description/resource/tags (a YAML list)/timestamp). demarkus carries metadata out-of-band and treats a body that opens with --- as literal content. So syntactically identical (--- frontmatter), but different data model: in demarkus the on-disk --- block is a store-owned envelope (version/previous-hash/archived + publisher data), it is stripped before serving, and the catalog reads only the publisher keys. Naively publishing a raw OKF file double-wraps the frontmatter and the OKF type/tags never reach the catalog.

Decision — split namespace in store frontmatter (protocol/store/store.go)

Pre-1.0, so changed the storage format. Promote the recognized OKF field names to bare frontmatter; keep everything else namespaced:

• Recognized OKF keys (type,title,description,resource,tags,timestamp) → written bare. tags serialized as a YAML flow list [a, b] to match spec.
• Non-spec publisher keys (importance, custom) → keep the meta. prefix.
• Store-operational (version,previous-hash,archived) → bare, unchanged.

The non-obvious why

The meta. prefix was not decoration — it was the integrity boundary that stopped a publisher from supplying archived: true / version: 999 and forging store state (extractMetadata read only meta.*). Dropping it for the OKF fields means that boundary now has to be enforced by name: added a reservedMetaKeys denylist (validateMeta rejects them) and extractMetadata excludes them. Keeping arbitrary publisher keys under meta. preserves the boundary for the open-ended set — only a closed, known set of OKF names goes bare.

Key property that kept the change contained: the in-memory metadata map stays bare-keyed, so catalog/handler/filter were untouched. Only buildVersionFile (write) and extractMetadata (read) changed. tags round-trips list↔csv, so the map contract ("a,b") is unchanged.

Back-compat: extractMetadata still reads old meta.tags/meta.type writes — live soul.demarkus.io data and existing immutable versions keep working.

What this does NOT do

Vocabulary alignment only. The frontmatter is still stripped before serving and the store's versions/ layout is not an OKF bundle tree — so demarkus does not yet serve or ingest OKF bundles. That is the next layer: an import/export codec (and possibly server-native bundle serving). Because the stored field names are now the literal OKF names, that codec is 1:1 rather than a translation table.

Open scoping question parked: import-only vs round-trip codec vs server-native content-negotiated bundle serving; and where it lives (tools/demarkus-okf CLI vs client/okf package).

OKF — spec + ADR follow-up

Documented the alignment rather than claiming compatibility:

• docs/SPEC.md §9.4 rewritten to match the code (was already stale — never documented archived or the old meta. keys). Now specifies reserved operational fields, bare OKF field names, tags as a YAML flow list, the meta. prefix for non-spec keys, and that the store block is stripped before serving. §8.1 notes arbitrary publisher metadata + reserved-key rejection. §13 Future Extensions adds OKF interop as a planned codec, not a conformance claim.
• docs/adr/0002-okf-metadata-alignment.md records the split-namespace decision and the integrity-boundary reasoning.

Framing decision (Fritz agreed): do not assert "OKF compatible / superset" in normative spec text yet — it would overclaim. Accurate framing is two-layer: a single demarkus document's content model is OKF-compatible (the doc itself), but at the system level demarkus is a superset — it builds versioning + hash chain + QUIC + capability auth + LOOKUP around an OKF-compatible document. "The doc is OKF; we built more around it." Earn the compatibility badge with the codec + a conformance check against OKF sample bundles.

Publisher metadata caps raised (10/512 → 50/1024)

Bumped while sizing for OKF producer-defined fields. Key insight: MaxMetaBytes (sum of key+value lengths) is the binding limit, not key count — the 6 OKF fields + importance already use ~7 keys / ~350 bytes, so you hit the byte wall before the key wall. Doubling keys alone would have bought little.

Final: MaxMetaKeys 10→50, MaxMetaBytes 512→1024, maxStoreFrontmatter 1024→2048 (must cover 1024 bytes meta + operational fields + per-line serialization overhead; worst case ~1.5 KB on disk). 50 keys is generous headroom but harmless — the 1024-byte total still bounds real on-disk size. Pre-1.0 is the cheap moment to right-size a wire limit. Constants in protocol.go + store.go; both store and handler validation read the shared constants. SPEC §9.4 updated. Boundary test now ties to MaxMetaKeys+1 with short keys so it isolates the key-count limit and won't rot on future bumps.

Gotcha: tags counted as csv but stored as a longer YAML list

MaxMetaBytes (1024) is validated on the metadata map, where tags is a csv string ("a,b"), but on disk tags serializes as a YAML flow list ([a, b]) — N+1 bytes longer for N tags. So the byte cap undercounted the real on-disk size, and a tag-heavy doc could pass validation then bloat the frontmatter. Adversarial worst case (~460 single-char tags + 49 tiny keys) reached ~2005 of the 2048 maxStoreFrontmatter budget — safe but only ~40 B margin, far thinner than the 1024 number suggests.

Fix: extracted store.SerializedMetaSize(key, value) — counts the value at its on-disk length (tags → formatTagsList) — and used it in BOTH the store validateMeta and the handler size check (they duplicated the accounting; per guidelines, the tricky piece now has one source of truth). The 1024 cap is now an honest on-disk bound, so maxStoreFrontmatter's margin is provable (worst case ~1542 < 2048) and a doc that passes meta validation always fits the frontmatter. Regression test: 400 one-char tags (csv 803 B passes, serialized ~1204 B rejected). protocol.go MaxMetaBytes comment updated to say values are counted as serialized.

OKF codec — first slice: demarkus okf validate landed

New package client/internal/okf (parse primitives import/export will reuse) + demarkus okf validate command (wired into the main.go subcommand switch).

Primitives: SplitFrontmatter (CRLF + EOF-delimiter tolerant), ParseFrontmatter (flat string map; flow [a,b] and block - a lists → csv; tolerates comments; errors on malformed lines), ConceptID, IsReserved. Reuses client/links (goldmark) for link extraction — no new deps.

ValidateBundle checks OKF v0.1 conformance: every non-reserved .md has parseable frontmatter with non-empty type (Error); non-root index.md with frontmatter (Error); broken/escaping internal .md links (Warn, per spec's tolerate-broken-links rule); log.md non-ISO date headings + non-okf_version root-index keys (Warn). Findings sorted (path, message) for stable output. Command: --strict (warns→failure), --quiet; exit 1 on errors.

Deliberately hand-rolled the frontmatter parser (no YAML dep) to match the project's existing manual-frontmatter approach in the store. Limitation: handles scalars/flow/block lists, not nested maps or multi-line scalars — fine for OKF v0.1's flat field set; revisit if a sample bundle needs more. Next slices: import (bundle → world, enforcing the metadata caps with sanitize-on-overflow) then export. Tested against synthetic bundles; should run it against Google's GA4/StackOverflow/Bitcoin reference bundles once import exists.

OKF codec — second slice: demarkus okf import landed

client/internal/okf/import.go + demarkus okf import [--dry-run] [--auth] [--insecure] <bundle-dir> <mark://host/prefix>.

Pure transform BuildImport(root, prefix) []PublishItem (no network, fully tested): per file — SplitFrontmatter, ParseFrontmatter, map OKF fields → demarkus metadata (recognized names are identity), strip frontmatter from body, rewrite bundle-absolute links under the prefix. The command publishes each item via fetch.Client.Publish with expectedVersion -1 (upsert; content-hash dedup means re-import of an unchanged bundle creates no new versions). The target URL's path IS the prefix (via ParseMarkURL) — no separate flag.

Decisions / non-obvious bits:

• Metadata sanitation, never silent. Invalid keys (e.g. data_steward) sanitized to data-steward with a warning; un-sanitizable or reserved keys (version/archived/previous-hash) dropped with a warning. Added store.IsReservedMetaKey so import reuses the store's reserved set rather than duplicating it.
• Cap enforcement reuses store.SerializedMetaSize (honest tags-as-list accounting). Overflow drops lowest-priority keys first via a priority ladder (title>tags>type>importance>description>resource>timestamp>producer), each drop warned. Kept caps at 50/1024 per earlier decision; sanitize-on-overflow.
• Reserved files (index.md/log.md) published verbatim; root index.md's okf_version frontmatter is stripped from the body (else it double-wraps) and kept as okf-version metadata.
• Link rewriting: only bundle-absolute (/x.md) links get the prefix — relative links resolve unchanged since the tree is preserved. Limitation: empty-text links [](/x.md) aren't rewritten (ExtractWithPositions reports no bracket span); rare, noted.

Deferred: live server round-trip. Server-side acceptance of bare OKF metadata + tags-list round-trip is already covered by handler/store unit tests, and the wire path by other CLI tests, so the marginal risk is low and the setup (TLS + TOML tokens + 2 procs) is high. Becomes natural once export lands: import → export → diff. Next slice: export.

OKF codec — third slice: demarkus okf export + verified live round-trip

client/internal/okf/export.go + demarkus okf export [--auth] [--insecure] <mark://host/prefix> <out-dir>. Completes the codec.

Pure BuildExport(docs, prefix) []BundleFile (no network): per doc — strip the world prefix from path and bundle-absolute links, reattach OKF frontmatter from metadata (canonical field order: type, title, description, resource, tags, timestamp, then producer keys sorted). Synthesizes a type (default "Document") when absent and timestamp from the doc's modified time. tags re-serialized via the shared store.FormatTagsList (one source of truth with the on-disk form). Reserved files written body-only; root index.md re-emits okf_version from the okf-version metadata the importer stashed.

Refactors / reuse:

• Extracted mapAbsLinks(body, fn) from import's link rewriter; import prepends the prefix, export strips it (boundary-safe: only /pfx or /pfx/...).
• yamlScalar quotes only when a bare scalar would reparse differently (: , trailing :, #, surrounding space, empty, leading YAML indicator). URLs (no : ) stay bare.
• Command: enumerateDocs recurses LIST (the LIST verb always lists, never serves index.md — confirmed in handler; versions/ already filtered by ListDir). publisherMeta strips server-owned response keys (status/version/ modified/etag/content-hash/current-version/entries).

Bug found + fixed via the live test: import treated PUBLISH status created as failure — a new-doc publish returns created, only an update returns ok. Now accepts both.

Live round-trip verified (self-signed server on :16309, minted read+publish token): import bundle → on-disk frontmatter exactly as designed (bare tags: [sales, revenue], title, type; meta.data-steward for the sanitized producer key; absolute links rewritten under /vendor, relative preserved) → re-import upserts with content-hash dedup (no new versions) → export → validate reports 0 errors / 0 warnings → exported body byte-identical to original, links restored to bundle-relative, root okf_version round-tripped.

Codec complete: validate + import + export, all three demarkus okf subcommands, shared logic in client/internal/okf, no protocol/server changes beyond the two small exported store helpers (IsReservedMetaKey, FormatTagsList). Next: point validate/round-trip at Google's real GA4/StackOverflow/Bitcoin sample bundles as fixtures; consider log.md generation from VERSIONS on export.

OKF codec — timestamp format handling closed

Closed the gap where incoming timestamp values were passed through unchecked.

• validate: validateConcept now warns when a present timestamp doesn't parse as RFC 3339 (isRFC3339). Stays a Warn — OKF lists timestamp as recommended, not a hard conformance rule. Refactored validateConcept to accumulate findings (type error + timestamp warn) instead of returning early.
• export: normalizeTimestamp(declared, modified) re-emits a present value in canonical RFC 3339 (accepts RFC 3339 or date-only YYYY-MM-DD, converts offsets to UTC); empty or unparseable → falls back to the doc's modified (already RFC 3339). So exported timestamps are always canonical.

Required type was already covered: validate errors on missing/empty; export synthesizes type: Document; import warns but stays permissive. Tests added for both timestamp paths. All client tests + pre-commit green.

OKF made a core write-time value: default type on publish

Fritz wanted OKF conformance to be an opinionated, core demarkus value enforced on write. Pushed back on the obvious hard-reject gate: it would break every existing writer (soul.demarkus.io, knowledge broker worlds, journals, ADRs, docs site — all publish typeless markdown via mark_publish), and OKF's own first principle is "minimally opinionated." There was also no existing policy/gate machinery in the codebase to hook into.

Chose opinionated-by-construction: on PUBLISH, when no type is declared, the server assigns Document (protocol.OKFDefaultType) — applyOKFTypeDefault in handlePublish, after extractPublisherMeta. Reserved files (index.md, log.md) exempt; explicit types preserved. Same constant backs export's synthesis (one source of truth). ADR 0003 records it; SPEC §6.4 + §14 updated.

Now every served demarkus document carries an OKF type by construction — maximal per-doc conformance at write time. Still not a served bundle (frontmatter stripped, versions/ layout); full bundle conformance stays an export concern.

Gotcha caught by the existing dedup test: the default is real metadata, so a legacy doc (written without type) republished identically now differs by the added type → one new version, then dedups. Fixed the test to seed v1 with type:Document; documented the one-time bump in SPEC + ADR. Deferred: a strict reject mode as opt-in config/per-world policy (never global default).

OKF type default folded into APPEND too

Extended applyOKFTypeDefault to handleAppend (after extractPublisherMeta), so every write path — PUBLISH and APPEND — guarantees a typed OKF concept.

Noteworthy finding while doing it: store.Append does not merge metadata — it passes the request's metadata straight to WriteVersion, so an appended version's publisher metadata comes from the APPEND request alone, replacing the prior version's. An append with no metadata therefore drops the doc's tags/title/type. With the default applied, a typeless append at least lands type: Document instead of nothing. The broader "append replaces rather than carries forward metadata" behavior is pre-existing and orthogonal — flagged as a possible PR-comment item (should append inherit the prior version's metadata when the request omits it?). SPEC §6.4/§6.6 + ADR 0003 updated to cover APPEND; test added. All suites + pre-commit green; restaged on feat/okf-compatibility.

Shipped — OKF compatibility merged (#195)

Merged to main as squash commit 69c1473 feat(okf): Open Knowledge Format compatibility (#195). Whole feature in one PR: store metadata alignment (bare OKF field names, tags-as-list, 50/1024 caps, reserved-key boundary), the demarkus okf validate/import/export codec, server default type on publish + append, SPEC updates, ADR 0002/0003. CodeRabbit review addressed (7 fixed: export traversal guard, import symlink rejection, LIST decode/cycle hardening, reserved-key read guard, cap re-validation→bad-request, link query strip, test assertions; 1 skipped: SPEC markdownlint, not CI-gated, deferred to roadmap Docs Hygiene backlog). Local feature branch deleted.