Retrieval-augmented generation (RAG) is the architecture in which a large language model (LLM) answers questions by first retrieving relevant passages from a document corpus and then generating a response grounded in what it retrieved. Since the original RAG formulation by Lewis and colleagues in 2020, the pattern has become the default way enterprises connect language models to their own knowledge. The reason is structural: the model can only be as good as what retrieval hands it. A generation step grounded in the wrong passage produces a fluent, confident, wrong answer.
What is less widely internalized is that RAG quality is dominated by data engineering decisions that happen before any model runs. Three decisions matter most: how documents are divided into chunks, whether relevance judgments exist to measure and tune retrieval, and whether the corpus actually covers the questions users ask.
Teams debug the model, swap the embedding, and tune the prompt. Meanwhile, the failure sits upstream: in a chunk that severed a definition from its term, in a retrieval metric that was never measured against human judgment, or in a coverage gap that guarantees hallucination for a whole category of questions.
This blog treats RAG as a data problem and covers its three pillars: chunk quality, relevance data, and coverage. The comprehensive survey of RAG methods by Gao and colleagues documents how much architectural variety now exists; the data requirements below apply across nearly all of it.
Key Takeaways
- Retrieval quality bounds generation quality. A RAG system’s ceiling is set at ingestion time by chunking and corpus decisions, and no amount of prompt engineering recovers information that retrieval never surfaced.
- Chunks are semantic units, not character counts. Fixed-size splitting severs definitions from terms, steps from procedures, and cells from table headers. Structure-aware chunking with the right metadata is the highest-leverage single improvement in most underperforming RAG systems.
- Relevance data is what makes retrieval measurable. Without human relevance judgments on real queries, teams tune embeddings and rerankers against intuition. Graded relevance labels with hard negatives convert retrieval tuning from guesswork into engineering.
- Coverage determines the hallucination floor. Questions the corpus cannot answer will be answered anyway unless unanswerable queries are identified, labeled, and handled. Coverage mapping against the real query distribution is how those gaps become visible before users find them.
- Evaluation sets are corpus infrastructure. A maintained golden set of query, passage, and answer triples, refreshed as the corpus and the query distribution drift, is what separates RAG programs that improve from those that oscillate.
Why RAG Is a Data Problem Before It Is a Model Problem
Every RAG answer is the product of a chain. The corpus was chunked, the chunks were embedded, a query retrieved some of them, and the model generated from what arrived.
The generation step gets the attention because it produces the visible output, but each upstream link imposes a hard limit. If the relevant content was split across two chunks, neither of which is individually similar enough to the query, retrieval returns something else. If the corpus never contained the answer, retrieval returns the nearest irrelevant neighbor, and the model, given plausible-looking context, generates a plausible-looking answer. These are not model failures. They are data failures wearing a model failure’s symptoms, which is why they survive so many rounds of prompt and model iteration.
Pillar One: Chunk Quality
Why Chunk Boundaries Carry So Much Weight
A chunk is the unit of retrieval: it is what gets embedded (converted into a numeric vector that captures its meaning for similarity search), what gets matched against queries, and what the model reads.
When a fixed-size splitter cuts every 500 tokens regardless of content, the damage is systematic. Definitions are severed from the terms they define. A procedure’s steps land in different chunks, so no single retrieved unit contains the whole method. A table is split from its header row, leaving cells with no column meaning. A contract clause is separated from the section heading that establishes its scope. Each of these produces chunks that are individually retrievable and individually useless.
Structure-Aware Chunking and Chunk Metadata
The alternative is chunking that follows document architecture: sections, headings, list and table boundaries, and semantic breaks, with size limits applied within structural units rather than across them. Two practices carry most of the benefit. First, contextual anchoring: each chunk carries its ancestry, document title, section path, and, for tables, the header row, so that a retrieved fragment arrives with the context that makes it interpretable. Second, chunk-level metadata: document type, date, jurisdiction or product version where applicable, and source authority, which enables filtered retrieval and lets freshness and authority participate in ranking.
Reviewing a random sample of chunks by hand is the fastest diagnostic for an underperforming RAG system. Asking of each one whether a person could act on it in isolation routinely explains failures that had been attributed to the embedding model.
Chunk QA as a Labeling Task
At corpus scale, chunk quality becomes an annotation task: human reviewers sample chunks and label them as self-contained, context-dependent, or fragmentary, with fragment labels traced back to the chunking rules that produced them. This converts chunking from a one-time engineering guess into a measured process with an error rate, which is what allows the chunking configuration to be tuned against evidence.
Pillar Two: Relevance Data
What Relevance Judgments Are and Why Binary Is Not Enough
A relevance judgment is a human label on a query and passage pair, recording how well the passage answers the query. Binary labels (relevant or not) are cheap but blunt: they cannot distinguish a passage that fully answers a question from one that merely mentions its keywords.
Graded judgment practice follows the standard established by retrieval benchmarks such as BEIR: typically a three or four-level scale distinguishing passages that fully answer, partially answer, are topically related, or are irrelevant. The distinctions matter because retrieval tuning optimizes whatever the labels can express. A system tuned on binary labels learns keyword adjacency; a system tuned on graded labels learns to rank complete answers above mentions.
Hard Negatives and Where Judgment Effort Goes
The most valuable relevance labels are the difficult ones: hard negatives, passages that look relevant, share vocabulary with the query, and score high on similarity, yet do not answer the question. The near-miss policy document from an adjacent product, the outdated version of the right procedure, the section that discusses the topic without containing the answer. These are exactly the passages retrieval confuses, and they only become training and evaluation signals when human judgment marks them.
Annotator calibration for relevance work follows the same discipline as other subjective labeling: written guidelines with worked examples per grade, calibration rounds measured by inter-annotator agreement, and adjudication for disagreements. Domain-expert annotators handle corpora where relevance is a professional judgment, as it is in legal, medical, and financial content.
The Golden Evaluation Set
Relevance data culminates in a golden set: a maintained collection of real queries, each with its graded passage judgments and, for end-to-end evaluation, a verified reference answer.
Against this set, retrieval is measured with three standard metrics. Recall at k asks whether a relevant passage appears in the top k results. Mean reciprocal rank (MRR) asks how high the first relevant passage ranks. Normalized discounted cumulative gain (nDCG) asks how well the full ranking orders passages by their graded relevance.
The golden set is what turns every subsequent change, a new embedding model, a chunking revision, a reranker (a second-pass model that reorders retrieved passages for relevance), into a measured comparison rather than a vibe check.
Pillar Three: Coverage
Mapping the Corpus Against the Query Distribution
Coverage asks a question that neither chunking nor relevance tuning can answer: does the corpus contain what users ask about? The map is built from real query logs, clustered into intents, with each cluster assessed against the corpus: fully answerable, partially answerable, or unanswerable. The output is a prioritized content gap list, and it routinely surprises teams because query distributions reflect what users actually need rather than what the documentation team assumed they would need.
Unanswerable Queries and the Hallucination Floor
The unanswerable cluster deserves specific handling because it sets the hallucination floor. A RAG system, when asked a question its corpus cannot answer, will retrieve the nearest content anyway, and the model will generate from it. Labeling a representative set of unanswerable queries and evaluating whether the system declines or deflects appropriately on them is the only way to measure this failure mode. The label set also feeds the fix: either the content gap is filled, or the system is trained and prompted to recognize the boundary and say so.
Freshness as Ongoing Coverage
Coverage decays. Products change, policies are revised, and the corpus quietly falls behind the world it describes, at which point retrieval serves confident answers from superseded documents. Freshness discipline is metadata plus process: effective dates and version fields on chunks, retrieval that prefers current versions, and a refresh cycle that re-runs the coverage map as the query distribution and the document base drift.
How Digital Divide Data Can Help
Whether a team builds this data layer internally or with a partner, the same three artifacts decide RAG quality: a chunk corpus that survives sampling, a relevance-judged golden set, and a coverage map against real queries. Producing them at production scale is the work we do.
Relevance data with calibrated judgment: text annotation teams produce graded relevance labels with hard-negative mining, domain-expert annotators for professional content, and the inter-annotator agreement discipline that makes the labels trustworthy enough to tune against.
Golden sets that stay golden: model evaluation services build and maintain the query, judgment, and answer sets, refreshed on a cadence, so recall, MRR, and nDCG remain measurements of the present system rather than of last quarter’s corpus.
Corpus and chunk quality at scale: AI data preparation runs chunk sampling and labeling programs, coverage mapping against query logs, and the freshness metadata work, with data engineering for AI building the ingestion pipelines that keep all of it current.
If your team can state its retrieval recall on a human-judged set and its coverage rate against last month’s queries, this layer exists. If it cannot, that is the gap. Talk to an expert.
Conclusion
RAG moved grounding from the model’s parameters into the data pipeline, and it moved the quality problem with it. The systems that answer reliably are built on three data assets that never appear in an architecture diagram: chunks that preserve meaning, relevance judgments that make retrieval measurable, and a coverage map that knows what the corpus cannot answer. Each one is produced by disciplined human labeling and maintained by process, not discovered by model iteration.
The diagnostic for any RAG program fits into three questions. Could a person act on a randomly sampled chunk in isolation? Is retrieval measured against human relevance judgments or against intuition? And when a user asks something the corpus cannot answer, does anyone know before the user does?
References
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems (NeurIPS). https://arxiv.org/abs/2005.11401
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023). Retrieval-augmented generation for large language models: A survey. arXiv. https://arxiv.org/abs/2312.10997
Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., & Gurevych, I. (2021). BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In NeurIPS Datasets and Benchmarks Track. https://arxiv.org/abs/2104.08663
Frequently Asked Questions
Q1. Our embeddings are state of the art. Why does retrieval still miss obvious answers?
Because embeddings can only represent what chunking preserved. If the answer was split across two chunks, neither fragment embeds close enough to the query; if the chunk lost its section context, the embedding represents the fragment rather than its meaning. Before changing models, run the sampling diagnostic: pull the queries that failed, inspect which chunks the answer actually lives in, and check whether those chunks are self-contained. In a large share of cases, the state-of-the-art embedding is faithfully representing a broken unit of text, and the fix is upstream in chunking and metadata, not in the model.
Q2. How large does a relevance-judged golden set need to be?
Large enough to cover the query distribution’s major intents, not a fixed universal number. The construction sequence matters more than the count: cluster real query logs into intents, sample queries proportionally across clusters including tail intents and known unanswerables, then judge retrieved and mined candidate passages per query with a graded scale.
A few hundred well-distributed, carefully judged queries typically produce more reliable tuning signal than thousands of hastily labeled ones. The set’s value also depends on maintenance: judgments must be refreshed as the corpus changes, or the golden set silently becomes a measurement of a system that no longer exists.
Q3. Can we generate relevance labels and QA pairs synthetically with an LLM instead of using human annotators?
Synthetic generation has a legitimate role and a specific danger. It is effective for scaling coverage of easy cases, drafting candidate QA pairs for human verification, and generating query variations. The danger is circularity: labels produced by a model correlate with model beliefs, and hard negatives, the near-miss passages retrieval actually confuses, are precisely where model judgment is least trustworthy and where human judgment carries the value. The workable pattern is hybrid: synthetic drafting with human verification for the general population, and fully human judgment for hard negatives, professional-domain content, and the golden evaluation set that everything else is measured against.
Q4. How do we handle documents that update frequently without rebuilding everything?
Design ingestion for versioned incremental updates from the start. Each document carries version and effective-date metadata that its chunks inherit; an update re-chunks and re-embeds only the affected document, marks superseded chunks rather than deleting them where audit requirements apply, and retrieval filters or down-ranks stale versions. The corresponding evaluation discipline is a freshness slice in the golden set: queries whose correct answer changed with a known update, verified to confirm the system now serves the current answer. Programs that skip the versioning metadata discover the cost later as confident answers from documents that were superseded months earlier.
Q5. Which retrieval metric should we optimize: recall at k, MRR, or nDCG?
Match the metric to how generation consumes retrieval. If the model reads the full top-k context window, recall at k is primary: what matters is that a fully answering passage is present anywhere in what the model sees. If the system feeds few passages or users see ranked citations, rank position matters, and MRR or nDCG better reflect experienced quality, with nDCG preferred when graded judgments exist because it credits ranking complete answers above partial ones. In practice, report recall at k and nDCG together and watch their divergence: rising recall with flat nDCG means the right passages are being found but buried, which points the tuning effort at reranking rather than at retrieval.
Udit Khanna leads the delivery of scalable AI and data solutions at Digital Divide Data, with a deep specialization in Physical AI. With a background in presales, solutioning, and customer success, he brings a mix of technical depth and business fluency, helping global enterprises move their AI projects from prototype to real-world deployment without losing momentum.
