How to Test a RAG System: Metrics, Datasets, and Tests That Catch Real Failures

A RAG system can look convincing long before it is reliable. Ask it a question, get a reasonable answer—perhaps even with a citation—and it is easy to assume it is ready. The trouble starts when a query changes, a document is updated, a reranker pushes down an important result, or the model fills a gap with a plausible-sounding claim.

You do not validate a RAG system because it “sounds good.” You validate it when you can show that it retrieves the right evidence, answers from that evidence, and does not regress when the pipeline changes.

Why RAG testing is different

An incorrect answer can have several causes:

• The required document was never indexed.
• The document was indexed, but chunking separated the fact from its context.
• The retriever failed to return it near the top.
• The reranker moved it too far down.
• The LLM received the right evidence but ignored it or added unsupported information.
• The system should have abstained, but answered with confidence.

Evaluating only the final answer mixes all of these failures together. When an answer fails, you need to know whether to fix embeddings, metadata, chunking, top-k, prompting, or generation.

The three layers to measure

The most useful way to evaluate a RAG system is to separate it into three levels.

Retrieval. Did the relevant documents or chunks appear in the results? This can be evaluated without calling a generative model.

Generation. Given a particular context, is the answer correct, relevant, and faithful to the evidence? This tests whether the LLM uses the information it receives well.

End-to-end behavior. Does the combination of retrieval, reranking, prompting, and generation perform well under production-like conditions, with acceptable latency and cost?

This separation saves a great deal of debugging time. If recall drops, changing the prompt is unlikely to help. If the context is correct but the model invents a fact, rebuilding the index is unnecessary.

What to measure in retrieval

Traditional information-retrieval metrics remain essential.

• Recall@k: whether a relevant document appears in the first k results.
• Precision@k: how much irrelevant context is passed to the LLM.
• MRR: rewards placing the first relevant result near the top.
• nDCG: useful when there are multiple relevant documents with different relevance levels.
• Coverage: the percentage of questions for which supporting evidence exists in the corpus.

In practice, RAG systems often prioritize recall before precision. A reranker can sort the candidates you already retrieved, but it cannot recover a document that never made it into the candidate set.

For example, if the cancellation policy is ranked twelfth and your pipeline passes only five chunks to the model, the quality of the LLM does not matter. The evidence never arrived.

What to measure in the answer

An answer can be fluent and relevant while still being wrong. Measure separate dimensions:

• Faithfulness or groundedness: each claim in the answer is supported by retrieved context.
• Answer relevance: the answer actually addresses the question.
• Factual correctness: it matches a reference answer when one exists.
• Citation quality: it points to the right, sufficiently specific source.
• Abstention: when there is no evidence, the system says so instead of guessing.

Abstention deserves special attention. “I could not find enough information in the available documentation” is often much better than a confident, precise-looking answer that is false.

Start with a small, realistic dataset

You do not need thousands of examples to uncover meaningful issues. Fifty to one hundred real or representative queries are an excellent starting point.

Each case should include:

Question
Reference answer, if applicable
Relevant document or chunks
Query type
Expected behavior: answer or abstain

Include straightforward questions, ambiguous queries, multi-document questions, exact terms such as error codes or product names, outdated information, and questions with no answer.

Synthetic data is useful for getting started or increasing coverage, but it should not be your only source. Real user queries contain ambiguity, typos, and vocabulary that question generators rarely reproduce fully.

A reproducible test with DeepEval

DeepEval lets you treat evaluations as tests: define a case, choose metrics, and set minimum thresholds. It is a convenient option for Python projects.

from deepeval import assert_test
from deepeval.metrics import (
    AnswerRelevancyMetric,
    ContextualRecallMetric,
    FaithfulnessMetric,
)
from deepeval.test_case import LLMTestCase

case = LLMTestCase(
    input="What is the cancellation deadline?",
    actual_output=(
        "You can cancel up to 30 days before the start date."
    ),
    expected_output=(
        "Cancellation is allowed up to 30 days before the start date."
    ),
    retrieval_context=[
        "Reservations can be cancelled without charge up to "
        "30 days before the start date."
    ],
)

assert_test(
    case,
    metrics=[
        ContextualRecallMetric(threshold=0.8),
        FaithfulnessMetric(threshold=0.9),
        AnswerRelevancyMetric(threshold=0.8),
    ],
)

This test checks three distinct things. ContextualRecallMetric detects whether the retrieved context contains the required evidence; FaithfulnessMetric detects claims unsupported by that context; and AnswerRelevancyMetric checks whether the output answers the query.

There are no universal thresholds. A low-risk internal assistant can tolerate a different level than a system used for legal, financial, or medical support. What matters is choosing thresholds explicitly and versioning them alongside the dataset.

The same case with Ragas

Ragas is focused specifically on evaluating RAG applications. Its current API can score each interaction with an evaluator model.

import asyncio

from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import ContextRecall, Faithfulness


async def evaluate_answer():
    evaluator_llm = llm_factory(
        "gpt-4o-mini",
        client=AsyncOpenAI(),
    )

    question = "What is the cancellation deadline?"
    context = [
        "Reservations can be cancelled without charge up to "
        "30 days before the start date."
    ]
    reference_answer = (
        "Cancellation is allowed up to 30 days before the start date."
    )
    rag_answer = (
        "You can cancel up to 30 days before the start date."
    )

    context_recall = ContextRecall(llm=evaluator_llm)
    faithfulness = Faithfulness(llm=evaluator_llm)

    recall = await context_recall.ascore(
        user_input=question,
        retrieved_contexts=context,
        reference=reference_answer,
    )
    fidelity = await faithfulness.ascore(
        user_input=question,
        response=rag_answer,
        retrieved_contexts=context,
    )

    print(f"Context recall: {recall.value:.2f}")
    print(f"Faithfulness: {fidelity.value:.2f}")


asyncio.run(evaluate_answer())

ContextRecall asks whether the retrieved context contains the evidence required by the reference answer. Faithfulness takes the generated answer and checks whether each claim can be inferred from the retrieved context. Both scores range from zero to one: a low score points to a different failure and therefore a different fix.

DeepEval and Ragas solve similar problems. Pick one to start, build a representative set of cases, and make every change run through the same tests. Adding more tools before you have a solid dataset rarely improves quality.

Observability: catching degradation before users do

Tests tell you whether a new version is better or worse on a known dataset. Observability answers a different question: what is happening in production right now?

OpenTelemetry can record traces and metrics for each stage of the pipeline: extraction, cleaning, chunking, embeddings, vector-store upserts, retrieval, and generation. The goal is not to save everything. It is to explain a degradation without guessing.

The following example instruments indexing. It assumes that chunk_document, embed, and upsert_vectors are application functions, and that the OpenTelemetry SDK is already configured to export telemetry with OTLP.

import time

from opentelemetry import metrics, trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("rag.indexer")
meter = metrics.get_meter("rag.indexer")

chunks_indexed = meter.create_counter(
    "rag.indexing.chunks",
    description="Number of chunks sent to the vector store",
)
embedding_latency = meter.create_histogram(
    "rag.embedding.duration",
    unit="s",
    description="Embedding latency per batch",
)
indexing_failures = meter.create_counter(
    "rag.indexing.failures",
    description="Failures during indexing",
)


def index_document(document):
    with tracer.start_as_current_span("rag.index_document") as root_span:
        root_span.set_attribute("rag.document.id", document.id)
        root_span.set_attribute("rag.document.source", document.source)

        chunks = chunk_document(document.text)
        root_span.set_attribute("rag.chunk.count", len(chunks))

        try:
            with tracer.start_as_current_span("rag.create_embeddings") as span:
                span.set_attribute("gen_ai.operation.name", "embeddings")
                span.set_attribute(
                    "gen_ai.request.model",
                    "text-embedding-3-large",
                )
                span.set_attribute("rag.embedding.batch_size", len(chunks))

                started_at = time.perf_counter()
                vectors = embed(chunks)
                elapsed = time.perf_counter() - started_at

                span.set_attribute(
                    "gen_ai.embeddings.dimension.count",
                    len(vectors[0]),
                )
                embedding_latency.record(
                    elapsed,
                    attributes={"model": "text-embedding-3-large"},
                )

            with tracer.start_as_current_span("rag.upsert_vectors") as span:
                span.set_attribute("rag.vector_store", "my-vector-store")
                span.set_attribute("rag.vector.count", len(vectors))
                upsert_vectors(document.id, chunks, vectors)

            chunks_indexed.add(
                len(chunks),
                attributes={"source": document.source},
            )

        except Exception as error:
            root_span.record_exception(error)
            root_span.set_status(Status(StatusCode.ERROR, str(error)))
            indexing_failures.add(
                1,
                attributes={"stage": "index_document"},
            )
            raise

The parent span, rag.index_document, shows total time per document. The child spans separate embedding creation from writing to the vector store. If indexing becomes slow, you can identify whether the problem is the embedding provider, an oversized batch, or the vector database.

The latency histogram supports degradation alerts; counters show throughput and failures; and attributes make it possible to filter by model, source, or stage. OpenTelemetry defines conventions for embedding operations and attributes such as model and dimensions, so it is worth using them when available. See the instrumentation documentation and GenAI semantic conventions.

Do not record full documents, embeddings, unfiltered prompts, or personal data as trace attributes. Use IDs, hashes, and low-cardinality metadata instead. In production, export telemetry to an OpenTelemetry Collector and route it from there to the backend you use; this is the recommended pattern for traces and metrics. OpenTelemetry explains how to configure exporters.

Turn evaluation into a safety net

Every change to embeddings, chunking, model choice, index configuration, top-k, reranking, or prompting should run against the same evaluation dataset. Compare results with a baseline, define which regressions block deployment, and manually review cases that get worse.

An average score is not enough. A change can improve the global average while breaking the questions that matter most: return policies, commercial terms, technical documentation, or compliance requirements.

RAG quality is not a fixed property of a model. It is a continuous practice: measure before deployment, observe after deployment, and turn every real failure into a new evaluation case.

The question is not whether your RAG answered well today. It is whether you can show that the next change will not break the answers that matter.