Journal — 2026-06-22

OKF (Google Open Knowledge Format) alignment — step 1: store frontmatter

Google published OKF v0.1 (2026-06-12): an org-knowledge format that is near-identical to demarkus's own model — directory bundles of markdown, cross-link relationship graph, index.md hubs, path-minus-.md as concept identity. Spec: github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md

The one real seam

OKF carries metadata in-body as YAML frontmatter (required type; recommended title/description/resource/tags (a YAML list)/timestamp). demarkus carries metadata out-of-band and treats a body that opens with --- as literal content. So syntactically identical (--- frontmatter), but different data model: in demarkus the on-disk --- block is a store-owned envelope (version/previous-hash/archived + publisher data), it is stripped before serving, and the catalog reads only the publisher keys. Naively publishing a raw OKF file double-wraps the frontmatter and the OKF type/tags never reach the catalog.

Decision — split namespace in store frontmatter (protocol/store/store.go)

Pre-1.0, so changed the storage format. Promote the recognized OKF field names to bare frontmatter; keep everything else namespaced:

• Recognized OKF keys (type,title,description,resource,tags,timestamp) → written bare. tags serialized as a YAML flow list [a, b] to match spec.
• Non-spec publisher keys (importance, custom) → keep the meta. prefix.
• Store-operational (version,previous-hash,archived) → bare, unchanged.

The non-obvious why

The meta. prefix was not decoration — it was the integrity boundary that stopped a publisher from supplying archived: true / version: 999 and forging store state (extractMetadata read only meta.*). Dropping it for the OKF fields means that boundary now has to be enforced by name: added a reservedMetaKeys denylist (validateMeta rejects them) and extractMetadata excludes them. Keeping arbitrary publisher keys under meta. preserves the boundary for the open-ended set — only a closed, known set of OKF names goes bare.

Key property that kept the change contained: the in-memory metadata map stays bare-keyed, so catalog/handler/filter were untouched. Only buildVersionFile (write) and extractMetadata (read) changed. tags round-trips list↔csv, so the map contract ("a,b") is unchanged.

Back-compat: extractMetadata still reads old meta.tags/meta.type writes — live soul.demarkus.io data and existing immutable versions keep working.

What this does NOT do

Vocabulary alignment only. The frontmatter is still stripped before serving and the store's versions/ layout is not an OKF bundle tree — so demarkus does not yet serve or ingest OKF bundles. That is the next layer: an import/export codec (and possibly server-native bundle serving). Because the stored field names are now the literal OKF names, that codec is 1:1 rather than a translation table.

Open scoping question parked: import-only vs round-trip codec vs server-native content-negotiated bundle serving; and where it lives (tools/demarkus-okf CLI vs client/okf package).

OKF — spec + ADR follow-up

Documented the alignment rather than claiming compatibility:

• docs/SPEC.md §9.4 rewritten to match the code (was already stale — never documented archived or the old meta. keys). Now specifies reserved operational fields, bare OKF field names, tags as a YAML flow list, the meta. prefix for non-spec keys, and that the store block is stripped before serving. §8.1 notes arbitrary publisher metadata + reserved-key rejection. §13 Future Extensions adds OKF interop as a planned codec, not a conformance claim.
• docs/adr/0002-okf-metadata-alignment.md records the split-namespace decision and the integrity-boundary reasoning.

Framing decision (Fritz agreed): do not assert "OKF compatible / superset" in normative spec text yet — it would overclaim. Accurate framing is two-layer: a single demarkus document's content model is OKF-compatible (the doc itself), but at the system level demarkus is a superset — it builds versioning + hash chain + QUIC + capability auth + LOOKUP around an OKF-compatible document. "The doc is OKF; we built more around it." Earn the compatibility badge with the codec + a conformance check against OKF sample bundles.

Publisher metadata caps raised (10/512 → 50/1024)

Bumped while sizing for OKF producer-defined fields. Key insight: MaxMetaBytes (sum of key+value lengths) is the binding limit, not key count — the 6 OKF fields + importance already use ~7 keys / ~350 bytes, so you hit the byte wall before the key wall. Doubling keys alone would have bought little.

Final: MaxMetaKeys 10→50, MaxMetaBytes 512→1024, maxStoreFrontmatter 1024→2048 (must cover 1024 bytes meta + operational fields + per-line serialization overhead; worst case ~1.5 KB on disk). 50 keys is generous headroom but harmless — the 1024-byte total still bounds real on-disk size. Pre-1.0 is the cheap moment to right-size a wire limit. Constants in protocol.go + store.go; both store and handler validation read the shared constants. SPEC §9.4 updated. Boundary test now ties to MaxMetaKeys+1 with short keys so it isolates the key-count limit and won't rot on future bumps.

Gotcha: tags counted as csv but stored as a longer YAML list

MaxMetaBytes (1024) is validated on the metadata map, where tags is a csv string ("a,b"), but on disk tags serializes as a YAML flow list ([a, b]) — N+1 bytes longer for N tags. So the byte cap undercounted the real on-disk size, and a tag-heavy doc could pass validation then bloat the frontmatter. Adversarial worst case (~460 single-char tags + 49 tiny keys) reached ~2005 of the 2048 maxStoreFrontmatter budget — safe but only ~40 B margin, far thinner than the 1024 number suggests.

Fix: extracted store.SerializedMetaSize(key, value) — counts the value at its on-disk length (tags → formatTagsList) — and used it in BOTH the store validateMeta and the handler size check (they duplicated the accounting; per guidelines, the tricky piece now has one source of truth). The 1024 cap is now an honest on-disk bound, so maxStoreFrontmatter's margin is provable (worst case ~1542 < 2048) and a doc that passes meta validation always fits the frontmatter. Regression test: 400 one-char tags (csv 803 B passes, serialized ~1204 B rejected). protocol.go MaxMetaBytes comment updated to say values are counted as serialized.

OKF codec — first slice: demarkus okf validate landed

New package client/internal/okf (parse primitives import/export will reuse) + demarkus okf validate command (wired into the main.go subcommand switch).

Primitives: SplitFrontmatter (CRLF + EOF-delimiter tolerant), ParseFrontmatter (flat string map; flow [a,b] and block - a lists → csv; tolerates comments; errors on malformed lines), ConceptID, IsReserved. Reuses client/links (goldmark) for link extraction — no new deps.

ValidateBundle checks OKF v0.1 conformance: every non-reserved .md has parseable frontmatter with non-empty type (Error); non-root index.md with frontmatter (Error); broken/escaping internal .md links (Warn, per spec's tolerate-broken-links rule); log.md non-ISO date headings + non-okf_version root-index keys (Warn). Findings sorted (path, message) for stable output. Command: --strict (warns→failure), --quiet; exit 1 on errors.

Deliberately hand-rolled the frontmatter parser (no YAML dep) to match the project's existing manual-frontmatter approach in the store. Limitation: handles scalars/flow/block lists, not nested maps or multi-line scalars — fine for OKF v0.1's flat field set; revisit if a sample bundle needs more. Next slices: import (bundle → world, enforcing the metadata caps with sanitize-on-overflow) then export. Tested against synthetic bundles; should run it against Google's GA4/StackOverflow/Bitcoin reference bundles once import exists.

OKF codec — second slice: demarkus okf import landed

client/internal/okf/import.go + demarkus okf import [--dry-run] [--auth] [--insecure] <bundle-dir> <mark://host/prefix>.

Pure transform BuildImport(root, prefix) []PublishItem (no network, fully tested): per file — SplitFrontmatter, ParseFrontmatter, map OKF fields → demarkus metadata (recognized names are identity), strip frontmatter from body, rewrite bundle-absolute links under the prefix. The command publishes each item via fetch.Client.Publish with expectedVersion -1 (upsert; content-hash dedup means re-import of an unchanged bundle creates no new versions). The target URL's path IS the prefix (via ParseMarkURL) — no separate flag.

Decisions / non-obvious bits:

• Metadata sanitation, never silent. Invalid keys (e.g. data_steward) sanitized to data-steward with a warning; un-sanitizable or reserved keys (version/archived/previous-hash) dropped with a warning. Added store.IsReservedMetaKey so import reuses the store's reserved set rather than duplicating it.
• Cap enforcement reuses store.SerializedMetaSize (honest tags-as-list accounting). Overflow drops lowest-priority keys first via a priority ladder (title>tags>type>importance>description>resource>timestamp>producer), each drop warned. Kept caps at 50/1024 per earlier decision; sanitize-on-overflow.
• Reserved files (index.md/log.md) published verbatim; root index.md's okf_version frontmatter is stripped from the body (else it double-wraps) and kept as okf-version metadata.
• Link rewriting: only bundle-absolute (/x.md) links get the prefix — relative links resolve unchanged since the tree is preserved. Limitation: empty-text links [](/x.md) aren't rewritten (ExtractWithPositions reports no bracket span); rare, noted.

Deferred: live server round-trip. Server-side acceptance of bare OKF metadata + tags-list round-trip is already covered by handler/store unit tests, and the wire path by other CLI tests, so the marginal risk is low and the setup (TLS + TOML tokens + 2 procs) is high. Becomes natural once export lands: import → export → diff. Next slice: export.

OKF codec — third slice: demarkus okf export + verified live round-trip

client/internal/okf/export.go + demarkus okf export [--auth] [--insecure] <mark://host/prefix> <out-dir>. Completes the codec.

Pure BuildExport(docs, prefix) []BundleFile (no network): per doc — strip the world prefix from path and bundle-absolute links, reattach OKF frontmatter from metadata (canonical field order: type, title, description, resource, tags, timestamp, then producer keys sorted). Synthesizes a type (default "Document") when absent and timestamp from the doc's modified time. tags re-serialized via the shared store.FormatTagsList (one source of truth with the on-disk form). Reserved files written body-only; root index.md re-emits okf_version from the okf-version metadata the importer stashed.

Refactors / reuse:

• Extracted mapAbsLinks(body, fn) from import's link rewriter; import prepends the prefix, export strips it (boundary-safe: only /pfx or /pfx/...).
• yamlScalar quotes only when a bare scalar would reparse differently (: , trailing :, #, surrounding space, empty, leading YAML indicator). URLs (no : ) stay bare.
• Command: enumerateDocs recurses LIST (the LIST verb always lists, never serves index.md — confirmed in handler; versions/ already filtered by ListDir). publisherMeta strips server-owned response keys (status/version/ modified/etag/content-hash/current-version/entries).

Bug found + fixed via the live test: import treated PUBLISH status created as failure — a new-doc publish returns created, only an update returns ok. Now accepts both.

Live round-trip verified (self-signed server on :16309, minted read+publish token): import bundle → on-disk frontmatter exactly as designed (bare tags: [sales, revenue], title, type; meta.data-steward for the sanitized producer key; absolute links rewritten under /vendor, relative preserved) → re-import upserts with content-hash dedup (no new versions) → export → validate reports 0 errors / 0 warnings → exported body byte-identical to original, links restored to bundle-relative, root okf_version round-tripped.

Codec complete: validate + import + export, all three demarkus okf subcommands, shared logic in client/internal/okf, no protocol/server changes beyond the two small exported store helpers (IsReservedMetaKey, FormatTagsList). Next: point validate/round-trip at Google's real GA4/StackOverflow/Bitcoin sample bundles as fixtures; consider log.md generation from VERSIONS on export.

OKF codec — timestamp format handling closed

Closed the gap where incoming timestamp values were passed through unchecked.

• validate: validateConcept now warns when a present timestamp doesn't parse as RFC 3339 (isRFC3339). Stays a Warn — OKF lists timestamp as recommended, not a hard conformance rule. Refactored validateConcept to accumulate findings (type error + timestamp warn) instead of returning early.
• export: normalizeTimestamp(declared, modified) re-emits a present value in canonical RFC 3339 (accepts RFC 3339 or date-only YYYY-MM-DD, converts offsets to UTC); empty or unparseable → falls back to the doc's modified (already RFC 3339). So exported timestamps are always canonical.

Required type was already covered: validate errors on missing/empty; export synthesizes type: Document; import warns but stays permissive. Tests added for both timestamp paths. All client tests + pre-commit green.