reading · 8 min

Grounding & RAG

Objective: explain RAG as “retrieve relevant context, then condition the answer on it” and why it cuts hallucination and enables citations.

Parametric vs retrieved knowledge

A language model has two kinds of knowledge. Parametric knowledge is baked into the weights during training — frozen at the cutoff, unverifiable at inference time. Retrieved knowledge is fetched at query time, passed in as text, and can be current and source-attributed without retraining the model.

Most failures that look like hallucinations are parametric-knowledge failures: the model extrapolates from stale or incomplete training signal. Retrieval-Augmented Generation (RAG) is the engineering response: instead of asking the model to recall a fact, give it the fact, then ask it to reason over what you provided.

The RAG shape

A RAG pipeline has a fixed shape: query → retrieve → assemble context → answer with citation.

  1. A user query arrives.
  2. The retriever scores candidate chunks against the query and returns the top-k.
  3. The top chunks are assembled into the context window alongside the query.
  4. The model answers conditioned on that context and cites which chunk supports each claim.

The model’s job shifts from “recall this fact” to “read these passages and synthesise an answer” — a verifiable task: if the answer is not supported by a retrieved chunk, the gap is visible.

Chunking

Documents are rarely fed whole. Chunking splits them into smaller units — paragraphs, fixed token windows, or semantic sections — that can be scored individually. Chunk size is a tuning variable: too small and a chunk loses surrounding context; too large and retrieval precision falls. A starting point of 256–512 tokens with small overlap is common.

Embeddings as similarity scoring

Keyword overlap is the simplest retrieval signal. Dense retrieval uses embeddings: both query and chunk are encoded as vectors, and similarity is measured by cosine similarity in that vector space. Semantically related text lands near each other even without shared keywords. Real embedding models are external API calls (real integration: subsystem D); keyword overlap captures the same structural pattern for learning.

Retrieval as a tool

From the agent’s perspective, retrieval is just another tool in the registry (see the Tool Use module). The agent calls retrieve(query), receives chunks with source ids, and uses only those chunks to produce its answer — the agent can call the retriever multiple times and cite exactly which source supports each claim.

Best practice: ground every factual claim in a retrieved chunk and carry its source id through to the answer — a claim with no attributable source is a hallucination risk waiting to surface.

Next: A Tiny Retriever