# Journal — 2026-06-22

## OKF (Google Open Knowledge Format) alignment — step 1: store frontmatter

Google published OKF v0.1 (2026-06-12): an org-knowledge format that is
near-identical to demarkus's own model — directory bundles of markdown,
cross-link relationship graph, `index.md` hubs, path-minus-`.md` as concept
identity. Spec: github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md

### The one real seam

OKF carries metadata **in-body** as YAML frontmatter (required `type`;
recommended `title`/`description`/`resource`/`tags` (a YAML list)/`timestamp`).
demarkus carries metadata **out-of-band** and treats a body that opens with
`---` as literal content. So syntactically identical (`---` frontmatter), but
different data model: in demarkus the on-disk `---` block is a store-owned
envelope (`version`/`previous-hash`/`archived` + publisher data), it is
**stripped before serving**, and the catalog reads only the publisher keys.
Naively publishing a raw OKF file double-wraps the frontmatter and the OKF
`type`/`tags` never reach the catalog.

### Decision — split namespace in store frontmatter (`protocol/store/store.go`)

Pre-1.0, so changed the storage format. Promote the recognized OKF field names
to **bare** frontmatter; keep everything else namespaced:

- Recognized OKF keys (`type`,`title`,`description`,`resource`,`tags`,`timestamp`)
  → written bare. `tags` serialized as a YAML flow list `[a, b]` to match spec.
- Non-spec publisher keys (`importance`, custom) → keep the `meta.` prefix.
- Store-operational (`version`,`previous-hash`,`archived`) → bare, unchanged.

### The non-obvious why

The `meta.` prefix was not decoration — it was the **integrity boundary** that
stopped a publisher from supplying `archived: true` / `version: 999` and forging
store state (`extractMetadata` read only `meta.*`). Dropping it for the OKF
fields means that boundary now has to be enforced by name: added a
`reservedMetaKeys` denylist (`validateMeta` rejects them) and `extractMetadata`
excludes them. Keeping arbitrary publisher keys under `meta.` preserves the
boundary for the open-ended set — only a closed, known set of OKF names goes
bare.

Key property that kept the change contained: the **in-memory metadata map stays
bare-keyed**, so catalog/handler/filter were untouched. Only `buildVersionFile`
(write) and `extractMetadata` (read) changed. `tags` round-trips list↔csv, so
the map contract (`"a,b"`) is unchanged.

Back-compat: `extractMetadata` still reads old `meta.tags`/`meta.type` writes —
live `soul.demarkus.io` data and existing immutable versions keep working.

### What this does NOT do

Vocabulary alignment only. The frontmatter is still **stripped before serving**
and the store's `versions/` layout is not an OKF bundle tree — so demarkus does
not yet serve or ingest OKF bundles. That is the next layer: an import/export
codec (and possibly server-native bundle serving). Because the stored field
names are now the literal OKF names, that codec is 1:1 rather than a translation
table.

Open scoping question parked: import-only vs round-trip codec vs server-native
content-negotiated bundle serving; and where it lives (`tools/demarkus-okf` CLI
vs `client/okf` package).


## OKF — spec + ADR follow-up

Documented the alignment rather than claiming compatibility:

- `docs/SPEC.md` §9.4 rewritten to match the code (was already stale — never
  documented `archived` or the old `meta.` keys). Now specifies reserved
  operational fields, bare OKF field names, `tags` as a YAML flow list, the
  `meta.` prefix for non-spec keys, and that the store block is stripped before
  serving. §8.1 notes arbitrary publisher metadata + reserved-key rejection.
  §13 Future Extensions adds OKF interop as a planned codec, not a conformance
  claim.
- `docs/adr/0002-okf-metadata-alignment.md` records the split-namespace
  decision and the integrity-boundary reasoning.

Framing decision (Fritz agreed): do **not** assert "OKF compatible / superset"
in normative spec text yet — it would overclaim. Accurate framing is two-layer:
a single demarkus document's **content model** is OKF-compatible (the doc
itself), but at the **system level** demarkus is a superset — it builds
versioning + hash chain + QUIC + capability auth + LOOKUP around an
OKF-compatible document. "The doc is OKF; we built more around it." Earn the
compatibility badge with the codec + a conformance check against OKF sample
bundles.


## Publisher metadata caps raised (10/512 → 50/1024)

Bumped while sizing for OKF producer-defined fields. Key insight: `MaxMetaBytes`
(sum of key+value lengths) is the **binding** limit, not key count — the 6 OKF
fields + importance already use ~7 keys / ~350 bytes, so you hit the byte wall
before the key wall. Doubling keys alone would have bought little.

Final: `MaxMetaKeys` 10→50, `MaxMetaBytes` 512→1024, `maxStoreFrontmatter`
1024→2048 (must cover 1024 bytes meta + operational fields + per-line
serialization overhead; worst case ~1.5 KB on disk). 50 keys is generous
headroom but harmless — the 1024-byte total still bounds real on-disk size.
Pre-1.0 is the cheap moment to right-size a wire limit. Constants in
`protocol.go` + `store.go`; both store and handler validation read the shared
constants. SPEC §9.4 updated. Boundary test now ties to `MaxMetaKeys+1` with
short keys so it isolates the key-count limit and won't rot on future bumps.


## Gotcha: tags counted as csv but stored as a longer YAML list

`MaxMetaBytes` (1024) is validated on the metadata map, where `tags` is a csv
string (`"a,b"`), but on disk `tags` serializes as a YAML flow list
(`[a, b]`) — `N+1` bytes longer for `N` tags. So the byte cap *undercounted*
the real on-disk size, and a tag-heavy doc could pass validation then bloat the
frontmatter. Adversarial worst case (~460 single-char tags + 49 tiny keys)
reached ~2005 of the 2048 `maxStoreFrontmatter` budget — safe but only ~40 B
margin, far thinner than the 1024 number suggests.

Fix: extracted `store.SerializedMetaSize(key, value)` — counts the value at its
on-disk length (tags → `formatTagsList`) — and used it in BOTH the store
`validateMeta` and the handler size check (they duplicated the accounting; per
guidelines, the tricky piece now has one source of truth). The 1024 cap is now
an honest on-disk bound, so `maxStoreFrontmatter`'s margin is provable (worst
case ~1542 < 2048) and a doc that passes meta validation always fits the
frontmatter. Regression test: 400 one-char tags (csv 803 B passes, serialized
~1204 B rejected). `protocol.go` MaxMetaBytes comment updated to say values are
counted as serialized.


## OKF codec — first slice: `demarkus okf validate` landed

New package `client/internal/okf` (parse primitives import/export will reuse) +
`demarkus okf validate` command (wired into the main.go subcommand switch).

Primitives: `SplitFrontmatter` (CRLF + EOF-delimiter tolerant), `ParseFrontmatter`
(flat string map; flow `[a,b]` and block `- a` lists → csv; tolerates comments;
errors on malformed lines), `ConceptID`, `IsReserved`. Reuses `client/links`
(goldmark) for link extraction — no new deps.

`ValidateBundle` checks OKF v0.1 conformance: every non-reserved `.md` has
parseable frontmatter with non-empty `type` (Error); non-root `index.md` with
frontmatter (Error); broken/escaping internal `.md` links (Warn, per spec's
tolerate-broken-links rule); `log.md` non-ISO date headings + non-okf_version
root-index keys (Warn). Findings sorted (path, message) for stable output.
Command: `--strict` (warns→failure), `--quiet`; exit 1 on errors.

Deliberately hand-rolled the frontmatter parser (no YAML dep) to match the
project's existing manual-frontmatter approach in the store. Limitation: handles
scalars/flow/block lists, not nested maps or multi-line scalars — fine for OKF
v0.1's flat field set; revisit if a sample bundle needs more. Next slices:
`import` (bundle → world, enforcing the metadata caps with sanitize-on-overflow)
then `export`. Tested against synthetic bundles; should run it against Google's
GA4/StackOverflow/Bitcoin reference bundles once import exists.


## OKF codec — second slice: `demarkus okf import` landed

`client/internal/okf/import.go` + `demarkus okf import [--dry-run] [--auth] [--insecure] <bundle-dir> <mark://host/prefix>`.

Pure transform `BuildImport(root, prefix) []PublishItem` (no network, fully
tested): per file — SplitFrontmatter, ParseFrontmatter, map OKF fields → demarkus
metadata (recognized names are identity), strip frontmatter from body, rewrite
bundle-absolute links under the prefix. The command publishes each item via
`fetch.Client.Publish` with expectedVersion -1 (upsert; content-hash dedup means
re-import of an unchanged bundle creates no new versions). The target URL's path
IS the prefix (via ParseMarkURL) — no separate flag.

Decisions / non-obvious bits:
- **Metadata sanitation, never silent.** Invalid keys (e.g. `data_steward`)
  sanitized to `data-steward` with a warning; un-sanitizable or reserved keys
  (`version`/`archived`/`previous-hash`) dropped with a warning. Added
  `store.IsReservedMetaKey` so import reuses the store's reserved set rather than
  duplicating it.
- **Cap enforcement reuses `store.SerializedMetaSize`** (honest tags-as-list
  accounting). Overflow drops lowest-priority keys first via a priority ladder
  (title>tags>type>importance>description>resource>timestamp>producer), each drop
  warned. Kept caps at 50/1024 per earlier decision; sanitize-on-overflow.
- **Reserved files** (index.md/log.md) published verbatim; root index.md's
  okf_version frontmatter is stripped from the body (else it double-wraps) and
  kept as `okf-version` metadata.
- **Link rewriting**: only bundle-absolute (`/x.md`) links get the prefix —
  relative links resolve unchanged since the tree is preserved. Limitation:
  empty-text links `[](/x.md)` aren't rewritten (ExtractWithPositions reports no
  bracket span); rare, noted.

Deferred: live server round-trip. Server-side acceptance of bare OKF metadata +
tags-list round-trip is already covered by handler/store unit tests, and the
wire path by other CLI tests, so the marginal risk is low and the setup (TLS +
TOML tokens + 2 procs) is high. Becomes natural once export lands: import →
export → diff. Next slice: `export`.