Evaluating Retrieval Quality in Retrieval-Augmented Generation

Evaluating Retrieval Quality in Retrieval-Augmented Generation

• Evaluating retrieval-augmented generation (RAG) presents challenges, especially for retrieval models within these systems.
• Traditional end-to-end evaluation methods are computationally expensive and have limitations, such as lacking transparency and being resource-intensive.

Limitations of End-to-End Evaluation

• End-to-end evaluation lacks transparency regarding which retrieved document contributed to the generated output.
• It is resource-intensive, consuming significant time and computational power.
• Many ranking systems rely on interleaving for evaluation and optimization, which further complicates the evaluation.

Novel Evaluation Approach: eRAG

• eRAG proposes using the LLM in RAG system to generate labels for evaluating the retrieval model.
• Each document in the retrieval list is individually utilized by the LLM to generate an output, which is then evaluated based on the downstream task ground truth labels.

Advantages of eRAG

• eRAG achieves a higher correlation with downstream RAG performance compared to baseline methods.
• It offers significant computational advantages, improving runtime and consuming up to 50 times less GPU memory than end-to-end evaluation.

Retrieval Evaluation Metrics

• Evaluation metrics for retrieval include Precision (P), Recall (R), Mean Average Precision (MAP), Mean Reciprocal Rank (MRR), Normalized Discounted Cumulative Gain (NDCG), and Hit Rate.

Experiments and Results

• The proposed approach is evaluated on question answering, fact-checking, and dialogue generation from the knowledge-intensive language tasks (KILT) benchmark.
• Results demonstrate that eRAG achieves the highest correlation with the downstream performance of the RAG system in comparison with the baselines.

eRAG Implementation

• eRAG's implementation is publicly available at https://github.com/alirezasalemi7/eRAG.### Retrieval-Augmented Generation (RAG)
• RAG is a pipeline that consists of a retriever and a language model (LM)
• The retriever fetches relevant documents, and the LM uses them to generate a response

Evaluating Retrieval Models in RAG

• The goal is to evaluate the retrieval models in the RAG pipeline
• The authors introduce eRAG, a novel approach for evaluating retrieval models

eRAG Approach

• eRAG leverages the per-document performance of the LM on the downstream task to generate relevance labels
• It provides a more accurate assessment of the retrieval model's performance

Correlation with Downstream Performance

• The authors compare the correlation between different evaluation methods and the downstream performance of the LM
• eRAG consistently exhibits a higher correlation with the downstream performance compared to other evaluation methods

Impact of Retrieval Model and LLM Size

• The authors vary the number of retrieved documents and compute the correlation between the metric with the highest correlation and the downstream performance of the LM
• The results show that the correlation decreases as the number of retrieved documents increases
• The authors also experiment with different LLM sizes and find that the correlation remains consistent

Retrieval-Augmentation Approaches

• The authors compare the correlation between eRAG and the downstream performance of FiD and IPA LLMs
• FiD processes each document individually, while IPA concatenates all documents and feeds them to the LM
• The results show that eRAG exhibits a higher correlation with the FiD LM

Efficiency Comparison

• The authors compare the efficiency of eRAG with end-to-end evaluation
• eRAG is 2.468 times faster than end-to-end evaluation on average
• eRAG also demonstrates greater memory efficiency, requiring 7-15 times less memory in the query-level configuration and 30-48 times less memory in the document-level configuration

Conclusion

• eRAG is a novel approach for evaluating retrieval models in the RAG pipeline

• It provides a more accurate assessment of the retrieval model's performance and is more efficient than end-to-end evaluation### Research Papers and Conferences

• Proceedings of the Second ACM International Conference on Web Search and Data Mining (WSDM '09) published in 2009.

• Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2021) published in 2021.

• Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL 2017) published in 2017.

• Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020) published in 2020.

• Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '23) published in 2023.

• Proceedings of the 47th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '24) published in 2024.

Models and Frameworks

• Mistral 7B: a model introduced by Albert Q. Jiang et al. in 2023.
• Dense Passage Retrieval (DPR) for Open-Domain Question Answering: a model introduced by Vladimir Karpukhin et al. in 2020.
• Unsupervised Dense Information Retrieval with Contrastive Learning: a framework introduced by Gautier Izacard et al. in 2022.
• Retrieval-Augmented Generation (RAG) pipelines: a framework for evaluating retrieval-augmented generation pipelines introduced by Jithin James and Shahul Es in 2023.
• LaMP: a framework for personalizing large language models through retrieval augmentation introduced by Alireza Salemi et al. in 2023.

Evaluation Methods and Datasets

• ROUGE: a package for automatic evaluation of summaries introduced by Chin-Yew Lin in 2004.
• IR evaluation methods for retrieving highly relevant documents: methods introduced by Kalervo Järvelin and Jaana Kekäläinen in 2000.
• TriviaQA: a large-scale distant supervised challenge dataset for reading comprehension introduced by Mandar Joshi et al. in 2017.
• FEVER: a large-scale dataset for fact extraction and verification introduced by James Thorne et al. in 2018.
• HotpotQA: a dataset for diverse, explainable multi-hop question answering introduced by Zhilin Yang et al. in 2018.

Authors and Contributions

• Gautier Izacard and Edouard Grave contributed to unsupervised dense information retrieval with contrastive learning and distilling knowledge from reader to retriever for question answering.
• Alireza Salemi et al. contributed to optimization methods for personalizing large language models through retrieval augmentation and symmetric dual encoding dense retrieval framework for knowledge-intensive visual question answering.
• Hamed Zamani et al. contributed to stochastic RAG: end-to-end retrieval-augmented generation through expected utility maximization and retrieval-enhanced machine learning.