How Do We Evaluate a RAG System? An Overview of Current Approaches

Author – Eva Paraschou

If you have been following the rapid rise of large language models (LLMs), you have likely also run into retrieval-augmented generation (RAG) systems. When we need an LLM to answer domain-specific questions without hallucinating, we hook it up to a library of documents so it can search for the right context before answering. It is a highly effective way of “teaching an LLM to use a library”, as previously explained in another blogpost.

But once you build a RAG system, a critical question emerges: How do you actually know if it is working well?

Evaluating these systems is tricky. If your RAG system outputs a poor response, is it because the search component retrieved the wrong documents or because the LLM failed to synthesise the retrieved information properly?

RAG in a Nutshell: From Naive to Modular

At its core, any RAG system consists of two major components working in tandem: the retrieval, which searches your document library to extract relevant context based on a user’s query; and the generation, which takes that retrieved context alongside the query and prompts an LLM to generate a clear, accurate response.

Depending on how these two components are designed, RAG architectures generally fall into three categories (Amugongo et al., 2025):

Naive RAG: A simple, linear process where the system indexes documents, retrieves the top results for a query and passes them straight to the LLM.
Advanced RAG: Introduces sophisticated pre-retrieval steps (like optimising how text chunks are indexed) and post-retrieval steps (like re-ranking the search results so the most important context is fed to the LLM first).
Modular RAG: A highly flexible architecture that integrates multiple advanced techniques, sometimes employing agentic workflows.

The Dimensions of RAG Evaluation

To evaluate a RAG system, we cannot treat it as a single “black box”. Instead, we assess both the retrieval and generation components independently, alongside several system-wide dimensions (Yu et al., 2024).

In the retrieval component, the main evaluation subject is the retrieved context, meaning the parts of the documents that have been retrieved as the most relevant to contain the information requested in the user’s query. Indicative evaluation dimensions of this component are:

Relevance: How well the retrieved context matches the user’s information needs.
Accuracy: How accurate the retrieved context is compared to a known set of “correct” contexts (ground-truth).
Completeness: The proportion of retrieved context that is actually relevant to the query.

In the generation component, the primary evaluation subject is the generated response to the user’s query. Indicative evaluation dimensions of this component are:

Relevance: Whether the response directly aligns with the intent and content of the user’s initial query.
Faithfulness: Whether the response is strictly grounded in the retrieved context (i.e. avoiding “hallucinations”).
Correctness: The overall accuracy of the generated response compared to a ground-truth sample response.

Additional evaluation dimensions consist of latency, which measures the speed at which the system retrieves data and delivers its response; diversity, which checks whether the system can pull from varied source documents and generate diverse responses; noise robustness, which evaluates how well the system ignores irrelevant background details that slip through the search filters; negative rejection, which checks if the system declines answering when the retrieved context lacks sufficient information; counterfactual robustness, which detects and rejects outright falsehoods hidden within the source documents; and human preferences, which assess subjective but critical qualitative elements like readability, toxicity and overall user safety.

Evaluation When You Have “Ground-Truth” Data

In ideal settings, researchers have a gold-standard dataset: a list of queries paired with the exact parts in the documents that should be retrieved and the exact answers that should be generated. When this ground-truth is available, we can use statistical and mathematical metrics to evaluate a RAG system (Yu et al., 2024).

To evaluate the retrieval component two types of metrics can be used:

Non-rank based metrics, which treat retrieval as binary (either the retrieved context is relevant or not). Examples include:
- Hit@k: whether at least one correct retrieved context is in the top-k results
- Precision: the proportion of retrieved contexts that are actually relevant
- Recall: the proportion of all existing relevant contexts that the system successfully retrieved

Rank-based metrics, which care about where the relevant retrieved context appears in the search results. Examples include:
- Mean Reciprocal Rank (MRR): how high up the first correct context appears
- Mean Average Precision (MAP): how high up all the correct documents are positioned

When it comes to evaluating the generation component, we want to measure if the generated response is semantically correct given a ground-truth sample response. An indicative metric is the BERTScore (Zhang et al., 2020), which uses contextual embeddings (dense mathematical representations of language) to assess semantic similarity between the generated response and the ground-truth response. It can also be adapted to measure faithfulness and relevance.

Evaluation When You Do NOT Have “Ground-Truth” Data

In the real world though, you rarely start with a perfect list of correct responses and contexts. Creating enough ground-truth data manually is incredibly time-consuming. When ground-truth is missing, we can turn to “LLM-as-a-Judge” frameworks to evaluate the system automatically.

One of the most prominent of these is RAGAS (retrieval augmented generation assessment) (Es et al., 2024). RAGAS bypasses the need for ground-truth by using a powerful LLM to evaluate the RAG system across three core dimensions:

Faithfulness: It uses an LLM to break the generated response down into individual statements. It then prompts the LLM to verify if each statement is explicitly supported by the retrieved context. The score is the percentage of supported statements.
Answer Relevance: It prompts an LLM to generate several potential questions based only on the generated response. It then measures how semantically similar those generated questions are to the user’s original query.
Context Relevance: It asks an LLM to extract only the sentences from the retrieved context that are absolutely necessary to answer the query. Unnecessary filler sentences penalise the score.

In their validation study, RAGAS scores aligned with human evaluators well: 95% agreement on faithfulness, 78% on answer relevance and 70% on context relevance (Es et al., 2024). Other popular frameworks operating on similar principles include TruLens, DeepEval and Umbrela (Upadhyay et al., 2024).

Where Can “Ground-Truth” Data Be Found?

If you are developing a domain-agnostic RAG architecture, you can use established, open-source benchmarks like the Knowledge Intensive Language Tasks (KILT) benchmark (Petroni et al., 2021), Natural Questions (Kwiatkowski et al., 2019) or SuperGLUE (Wang et al., 2019), among many others.

However, if your RAG is designed for a highly specific downstream task (such as navigating a country’s healthcare system through its national guidelines), generic benchmarks fall short. You will probably need a custom dataset.

For example, when the creators of RAGAS needed to test their framework (task-specific), they built a human-annotated dataset by selecting 50 Wikipedia pages, asking an LLM to generate questions based on the text, having the LLM answer them, and then utilising human annotators to verify and grade the quality of the questions, contexts and answers (Es et al., 2024).

Other specialised datasets leverage collaborative human-AI creation to achieve both scale and quality. For instance, BiasDPO was designed to test and mitigate LLM biases by starting with a small human-curated seed, expanding it using GPT-4 and subsequently verifying the samples with human reviewers (Allam, 2024). Similarly, Evals explores language model behaviours and value alignment by blending LLM-generated evaluation samples with varying levels of human oversight (Perez et al., 2023). Finally, ToxiGen provides a large-scale framework for detecting implicit toxicity, which was generated by prompting LLMs and then selectively reviewed by humans to ensure rigorous data quality (Hartvigsen et al., 2022).

Where is RAG Evaluation Heading?

As the field matures, evaluation methodologies are transitioning from basic scoring to more holistic, automated pipelines. Researchers are aiming to establish standardised matrices mapping complex and diverse evaluation dimensions directly to concrete computational metrics (Brehme et al., 2025). Moreover, the evaluation focus is shifting increasingly toward value-alignment aspects such as safety, fairness and non-toxicity, making it interesting to explore the effectiveness of either filtering biased context during retrieval or auditing the LLM during generation. Finally, the field is continuously moving toward hybrid workflows to balance human expertise with automated scale. This can include hybrid evaluation, which runs precise metrics on small, human-curated datasets before scaling up to ground-truth-free LLM judges, as well as hybrid dataset creation, which seeds evaluation pools with real-world, expert-created queries before using LLMs to expand them into thousands of test cases. By blending these approaches, researchers aim to ensure RAG systems remain robust, accurate and safe for real-world applications.

References

Allam, A. (2024). Biasdpo: Mitigating bias in language models through direct preference optimization. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), 42–50.

Amugongo, L. M., Mascheroni, P., Brooks, S., Doering, S., & Seidel, J. (2025). Retrieval augmented generation for large language models in healthcare: A systematic review. PLOS Digital Health, 4(6), e0000877.

Brehme, L., Ströhle, T., & Breu, R. (2025, June). Can LLMs Be trusted for evaluating RAG systems? A survey of methods and datasets. In 2025 IEEE Swiss Conference on Data Science (SDS) (pp. 16-23). IEEE.

Es, S., James, J., Anke, L. E., & Schockaert, S. (2024). Ragas: Automated evaluation of retrieval augmented generation. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, 150–158.

Hartvigsen, T., Gabriel, S., Palangi, H., Sap, M., Ray, D., & Kamar, E. (2022). Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 3309–3326.

Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., … & Devlin, J. (2019). Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7, 453–466.

Perez, E., Ringer, S., Lukosiute, K., Nguyen, K., Chen, E., Heiner, S., … & Plaza, J. (2023). Discovering language model behaviors with model-written evaluations. Findings of the Association for Computational Linguistics: ACL 2023, 13387–13434.

Petroni, F., Piktus, A., Fan, A., Lewis, P., Yazdani, M., De Cao, N., … & Riedel, S. (2021). Kilt: a benchmark for knowledge intensive language tasks. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2523–2544.

Upadhyay, S., Pradeep, R., Thakur, N., Craswell, N., & Lin, J. (2024). Umbrela: Umbrela is the (open-source reproduction of the) bing relevance assessor. arXiv preprint arXiv:2406.06519.

Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., … & Bowman, S. (2019). Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in Neural Information Processing Systems, 32.

Yu, H., Gan, A., Zhang, K., Tong, S., Liu, Q., & Zhaofeng, L. (2024). Evaluation of retrieval-augmented generation: A survey. CCF Conference on Big Data, 102–120. Springer.

Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2020). Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.

Further Reading/Watching/Listening:

Gan, A., Yu, H., Zhang, K., Liu, Q., Yan, W., Huang, Z., … & Hu, G. (2025). Retrieval augmented generation evaluation in the era of large language models: A comprehensive survey. arXiv preprint arXiv:2504.14891.

Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., … & Wang, H. (2023). Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.

RAG: Teaching Large Language Models to Use a Library https://alignai.eu/2025/09/23/rag-teaching-large-language-models-to-use-a-library/.

Image Attribution

Generated by: Nano Banana Pro

Date: 17 July 2026

Prompt: “Create an image with the animation of a little girl sitting on an office chair in front of a big complex computer system. Make the girl look like a detective trying to evaluate how this big computer system performs. The figure of the girl needs to look like an animation, not an actual person. The image must not have text on it.”