OKF + RAG: Why the Future of AI Agent Knowledge Isn’t One or the Other

Google Cloud’s new markdown standard doesn’t kill RAG — it exposes exactly where RAG was never meant to work alone

Ask a support agent a simple question about your own business, and watch what happens.

“What’s our refund window?”

“30 days.”

Instant. Confident. Wrong. The agent never opened your actual policy document — it pattern-matched from whatever it had seen during training, or worse, from a fuzzy retrieval pass that grabbed three loosely related paragraphs and stitched them into something that only sounded right.

For the last few years, the fix for this has been one thing: RAG (Retrieval-Augmented Generation). It’s genuinely powerful. It’s also, on its own, not enough — and in June 2026, Google Cloud shipped something that makes the gap obvious: the Open Knowledge Format (OKF).

This article breaks down what RAG actually does under the hood, what OKF is and isn’t, why teams shipping serious agents in production are converging on both at once, and how to decide what belongs where.

The problem: RAG shreds your knowledge to search it

RAG’s core idea is elegant. Take every document you own, cut it into chunks of a few hundred tokens, run each chunk through an embedding model to turn it into a vector, and store all those vectors in a database. When a question comes in, it also becomes a vector, and the database returns whichever chunks sit closest to it in that vector space.

That’s the real trick: retrieval by meaning, not keyword matching. Different phrasing, same underlying idea — RAG still finds it. It’s why an entire industry grew up around this pattern: Pinecone, Weaviate, Qdrant, Chroma, Milvus, Postgres with pgvector — systems built to store billions of vectors and search them in milliseconds.

But chunking has a cost that’s easy to ignore until it bites you.

• Structure dies at the cut. Slice a table, a step-by-step procedure, or a contract clause down the middle, and the two halves stop making sense on their own. The model gets disconnected fragments and quietly fills the gaps — which is exactly how a confident hallucination is born.
• Retrieval is probabilistic, not certain. It hands back what’s likely relevant, not guaranteed to be the correct answer. That’s fine for a fuzzy, exploratory question. It’s dangerous when the exact refund window, the exact price, or the exact dosage is what’s actually being asked.
• It’s a black box you can’t audit. Change one policy, and you have to rerun the entire ingestion pipeline. You can’t diff a vector index in a pull request. You can’t easily prove what your agent actually “knows” at any given moment, and outdated chunks can sit in the index long after the source of truth has moved on.

RAG isn’t broken. It’s just being asked to do a job — exact, authoritative answers — that it was never designed for.

The other half: what OKF actually is

Google Cloud’s answer, published as OKF v0.1 in June 2026, takes the opposite bet: no pipeline, no embeddings, no vector store. Just a directory of markdown files with a small block of YAML frontmatter on top. The spec is deliberately blunt about how simple this is meant to be — if you can cat a file, you can read OKF; if you can git clone a repo, you can ship it.

Here’s the shape of it:

• Every concept — a table, a metric, a runbook, an API, a policy — is a single markdown file.
• The frontmatter has exactly one required field: type. Everything else — title, description, resource (a link to the real underlying asset), tags, timestamp — is optional.
• The body underneath is ordinary structured markdown: headings, tables, lists, the things a model reads cleanly.
• Concepts link to each other with normal markdown links. Follow those links and the whole bundle becomes a graph — a table pointing to its dataset, a playbook pointing to the table it repairs — instead of a bag of disconnected chunks, because a human (or an agent acting on a human’s behalf) authored the relationships on purpose.

A minimal OKF concept file looks like this:

---
type: Policy
title: Refund Window
description: Standard refund eligibility period for consumer purchases.
resource: https://internal.example.com/policies/refunds
tags: [support, billing]
timestamp: 2026-06-01
---

# Refund Window

Customers are eligible for a full refund within **14 days** of purchase,
provided the item is unused and in its original packaging.

Because retrieval against this is deterministic — you decide exactly what the model sees, no cosine-distance lottery — and because the whole bundle lives in git, you get things RAG structurally can’t offer: you can diff a change, review it in a pull request, assign an owner to a concept, and roll back a bad edit like any other code change.

If this feels familiar, it should. It’s the pattern behind CLAUDE.md-style files and the "LLM wiki" idea that's been floating around informally for a while — deliberately hand-engineered context. OKF's real contribution isn't the idea itself; it's writing that pattern down as one shared, vendor-neutral spec that different producers and different agents can all target without translating between formats.

But OKF has an honest ceiling: it does not scale itself. Every concept is somebody’s curation work. It’s excellent for the knowledge you genuinely cannot afford to get wrong. It’s hopeless as a dumping ground for a decade of unsorted PDFs, tickets, and transcripts.

The architecture that actually wins: a router, not a referendum

The wrong question is “RAG or OKF?” The right one is: which layer should answer this specific query?

Picture an 80/20 split:

• OKF is the spine — the curated, structured, high-trust core of what your agent knows. The answers you simply can’t get wrong.
• RAG is the reach — the huge, messy, unbounded long tail you could never realistically sit down and hand-write.

A thin router sits in front of both. A high-stakes, canonical question — “What’s our refund window?” — routes to OKF and comes back exact, with a link straight to the source policy. An open-ended, exploratory question — “Has anyone hit this weird billing bug before?” — routes to RAG, which searches the full archive of forty thousand old support tickets.

Same agent. Two completely different knowledge paths. And the two layers actively make each other better:

• OKF gives RAG ground truth. When a curated concept exists for a question, the agent trusts it over a fuzzy retrieved chunk — a single rule that measurably cuts down on hallucinations.
• OKF gives the agent a map. Its index files list what knowledge exists before any search even runs, so the agent can practice progressive disclosure: drill into RAG only where the curated answer runs out, instead of blindly searching from zero every time.
• RAG gives OKF reach. Point semantic search at the OKF bundle and everything sprawling beyond it. Curation covers the critical core; retrieval covers the endless rest.

And critically, the agent itself doesn’t need to know which layer answered. Put the OKF bundle inside the same vector index, sit both behind one retrieval interface — even a single MCP server — and the model just asks for knowledge. All the routing stays hidden underneath.

The scorecard

OKF RAG Strength Precision Scale Trust High (human-curated) Probabilistic Structure Preserved (tables, hierarchy) Fragmented into chunks Change tracking Git diff, PR review Re-embed the pipeline Discovery Index files, explicit links Cosine similarity search Best for Policies, metrics, runbooks, contracts you must get exact Tickets, transcripts, heterogeneous unstructured piles Running cost Text + git — write once, edit a line Embedding, re-embedding, hosting a vector DB that never sleeps

When to reach for each, practically

OKF on its own, when you have:

• A small, curated knowledge base
• High-stakes answers that must be exact
• Highly structured tables or procedures
• Anything you want versioned in git and owned by a real, accountable human

RAG on its own, when you have:

• A huge, unstructured pile of text
• Open-ended questions you genuinely can’t predict in advance
• Contracts, support tickets, transcripts, PDFs, heterogeneous sources where “good enough” semantic matching is actually good enough

Both, with a router in the middle — which is where almost every serious agent headed for production ends up. The high-stakes core lives in OKF. The long tail lives in RAG. A thin routing layer decides, query by query, who answers.

Two myths worth killing

“OKF makes RAG obsolete.” No. The moment your corpus is too big or too fuzzy to curate by hand, vector search wins, and it isn’t close. RAG isn’t going anywhere.

“Just dump everything into one giant context window and skip retrieval entirely.” Also no. Irrelevant knowledge actively degrades answer quality and quietly burns tokens — selective, routed retrieval still beats brute-force stuffing, even at very large context windows.

What this looks like as a team sport

In practice, this splits cleanly across roles: data owners curate the OKF bundles for the knowledge that has to be exact. Engineers index the messy archive into a vector store. Orchestration frameworks wire the two retrieval paths together behind a shared interface. And the agent — whichever model is powering it — quietly consumes both without caring which layer answered.

OKF is only weeks old as a published spec, and it’s already moving fast, with agents starting to write OKF bundles themselves rather than waiting for humans to author every concept by hand. Curated structure and raw retrieval are converging, not competing.

How to actually implement this

Don’t start by ripping out your RAG pipeline. Start by carving out the narrow slice of knowledge that deserves to be exact, and build the router around it. In practice, it breaks into five steps:

1. Audit for canonical answers first. Go through your support logs, your agent’s failure cases, or just ask your team: what are the 20–50 questions where there is exactly one correct answer, and getting it wrong is expensive? Refund windows, pricing tiers, SLA terms, security policies, API rate limits, escalation paths. That list is your first OKF bundle. Resist the urge to curate everything — OKF’s value comes from precision on a small, deliberately scoped set of concepts, not from becoming a second copy of your entire knowledge base.

2. Structure the OKF bundle. Create one markdown file per concept, with type as the only mandatory frontmatter field, and link related concepts to each other so the bundle forms a real graph instead of a flat pile:

/knowledge
  /policies
    index.md
    refund-window.md
    escalation-path.md
  /metrics
    index.md
    monthly-active-users.md
  /apis
    index.md
    payments-api.md

Each index.md acts as a map — the thing your agent reads first, before deciding whether it even needs to go further.

3. Keep your existing RAG pipeline for everything else. Don’t touch your chunking, embedding, or vector store setup for the unstructured long tail — tickets, transcripts, contracts, PDFs. That pipeline already does its job. The only change is that it now sits alongside OKF instead of being the only source of truth.

4. Build the router. The router’s only job is classification: is this query canonical/high-stakes, or open-ended/exploratory? Three common ways to build it, roughly in order of how most teams actually start:

• Rules-based: keyword or intent matching against your OKF bundle’s tags and type fields. Cheapest, fastest, and surprisingly effective once your bundle is tagged well.
• A small classifier call: one cheap LLM call that returns okf or rag (or both) before the real answer is generated. Slightly more latency, much better recall on phrasing you didn't anticipate.
• Retrieve from both, merge, and let the model decide which to trust. Simplest to build, since you skip the router entirely — feed both OKF hits and RAG chunks into context and instruct the model to prefer the OKF concept when one exists. This is the fastest way to prototype the hybrid stack before investing in a dedicated router.

5. Expose both through one interface. Whether that’s a single MCP server, a retrieval function, or a REST endpoint, the model should only ever see “here’s your knowledge” — not “here’s OKF, here’s RAG, you figure out which to call.” Keep the plumbing invisible.

Choosing the stack

There’s no single required toolchain — that’s the point of OKF being just markdown and git — but here’s what most teams landed on for each layer:

Layer Solid default When to reach further OKF storage A git repo, plain and simple A wiki tool (Obsidian, Notion export) syncing into the repo, once you have many contributors Embeddings OpenAI text-embedding-3-large, Voyage AI, or Cohere Embed Self-hosted models (e.g., BGE, E5) if data can't leave your infra Vector store pgvector on Postgres — if you already run Postgres, this is one extension away and keeps your data in one place Pinecone, Qdrant, or Weaviate once you're past tens of millions of vectors or need managed scaling Orchestration LangChain or LlamaIndex for the retrieval + routing glue A hand-rolled service if your routing logic is simple — often less overhead than a full framework Agent-facing interface A single MCP server exposing one retrieve_knowledge tool Multiple specialized tools, once OKF and RAG genuinely need different parameters Router A cheap classifier call (Haiku-class model) or rules on tags A fine-tuned small classifier, once you have enough labeled query traffic to train one

The honest recommendation for most teams: pgvector over a dedicated vector database, unless you’re already at a scale where Postgres is the bottleneck. It means one less system to operate, one less place for data to go stale, and it plays nicely with OKF’s own git-based philosophy — everything stays close to your existing infrastructure instead of adding a new managed service you now have to keep in sync.

Implementing it in Rails

Rails is a genuinely good fit here, mostly because Postgres is already the default database for most Rails apps, which means pgvector is a single migration away rather than a new piece of infrastructure.

1. Add pgvector.

# Gemfile
gem "pgvector"
gem "neighbor"   # ActiveRecord integration for pgvector

# db/migrate/xxxx_enable_pgvector.rb
class EnablePgvector < ActiveRecord::Migration[7.1]
  def change
    enable_extension "vector"
  end
end

2. Model your knowledge chunks (the RAG side).

# db/migrate/xxxx_create_knowledge_chunks.rb
class CreateKnowledgeChunks < ActiveRecord::Migration[7.1]
  def change
    create_table :knowledge_chunks do |t|
      t.text :content
      t.string :source
      t.vector :embedding, limit: 1536
      t.timestamps
    end
  end
end

class KnowledgeChunk < ApplicationRecord
  has_neighbors :embedding
  def self.search(query_embedding, k: 5)
      nearest_neighbors(:embedding, query_embedding, distance: "cosine").first(k)
    end
 end

3. Model your OKF concepts (the curated side).

Rather than embedding OKF concepts into an opaque vector, load the bundle directly off disk (or a git-backed volume) and index it by tags and type — deterministic lookup, not similarity search:

class OkfConcept
  BUNDLE_PATH = Rails.root.join("knowledge")
  def self.all
    Dir.glob("#{BUNDLE_PATH}/**/*.md").map { |path| load(path) }
  end
  def self.load(path)
    raw = File.read(path)
    frontmatter, body = raw.split(/^---$/, 3)[1..2]
    metadata = YAML.safe_load(frontmatter)
    { path: path, type: metadata["type"], tags: metadata["tags"] || [], body: body.to_s.strip }
  end
  def self.find_by_tag(tag)
    all.select { |c| c[:tags].include?(tag) }
  end
end

Since OKF is just files, this can be as simple as a plain Ruby class with no database table at all — the git repo is the source of truth.

4. Build the router as a service object.

class KnowledgeRouter
  CANONICAL_TAGS = %w[policy pricing sla security].freeze
  def self.call(query)
    if canonical?(query)
      { source: :okf, results: OkfConcept.find_by_tag(classify_tag(query)) }
    else
      embedding = EmbeddingClient.embed(query)
      { source: :rag, results: KnowledgeChunk.search(embedding) }
    end
  end
  def self.canonical?(query)
    # start rules-based, upgrade to a classifier call once you have traffic to learn from
    CANONICAL_TAGS.any? { |tag| query.downcase.include?(tag) }
  end
end

5. Wire it into your agent call.

class SupportAgent
  def answer(query)
    knowledge = KnowledgeRouter.call(query)
    Anthropic::Client.new.messages.create(
      model: "claude-sonnet-4-6",
      max_tokens: 1000,
      messages: [{
        role: "user",
        content: "Context:\n#{knowledge[:results]}\n\nQuestion: #{query}"
      }]
    )
  end
end

A few Rails-specific notes:

• Keep OKF bundles in the same repo as your app (a /knowledge directory works fine) so changes ship through the same PR review and deploy process as your code — this is where OKF's git-native design pays off most directly in a Rails workflow.
• Use an ActiveJob to re-embed KnowledgeChunk records asynchronously whenever source documents change, rather than blocking a request cycle on embedding calls.
• If you’re deploying on AWS ECS Fargate, the OKF bundle can ship as part of the same container image or be mounted from an EFS volume synced from git — no separate service needed, since there’s no runtime to deploy for the OKF side at all.

The takeaway

It was never OKF versus RAG. It’s OKF plus RAG — a router up top, a curated spine underneath for the answers you can’t afford to get wrong, and a retrieval layer stretching out to cover everything else. Neither one has to pretend it can do the other’s job, and for the first time, you stop trading precision away just to get scale, or scale away just to get precision.

If you’re building an agent against your own company’s knowledge right now, the practical first step isn’t ripping out your RAG pipeline. It’s picking the five to ten answers your agent gets wrong most often — the ones with a single correct answer — and writing those down as OKF concepts first. Let RAG keep doing what it’s good at everywhere else.

About the author: Qasim Ali Zahid is a Senior Full-Stack Engineer and Tech Lead at Infinikorn, with 7+ years of experience across Ruby on Rails, the MERN stack, and AWS-based DevOps. He writes about AI systems architecture, agentic tooling, and production engineering. Find him on GitHub @iamqasimali, LinkedIn (qasim-ali-zahid), and X (@QasimA6).

Tags: #AIAgents #RAG #GoogleCloud #OpenKnowledgeFormat #LLM #AIEngineering #MachineLearning #KnowledgeManagement #AgenticAI #SoftwareArchitecture