Evaluating Retrieval-Augmented Generation (RAG) Models

Challenges in Evaluating Retrieval-Augmented Generation (RAG)

• Evaluating RAG presents challenges, particularly for retrieval models within these systems.
• Traditional end-to-end evaluation methods are computationally expensive.

Limitations of Traditional Evaluation Methods

• Evaluating the retrieval model's performance based on query-document relevance labels shows a small correlation with the RAG system's downstream performance.

eRAG: A Novel Evaluation Approach

• eRAG is a novel evaluation approach where each document in the retrieval list is individually utilized by the large language model within the RAG system.
• The output generated for each document is then evaluated based on the downstream task ground truth labels.

Document-Level Annotations and Aggregation

• Various downstream task metrics are employed to obtain document-level annotations.
• These annotations are aggregated using set-based or ranking metrics.

Experimental Results

• Extensive experiments on a wide range of datasets demonstrate that eRAG achieves a higher correlation with downstream RAG performance compared to baseline methods.
• Improvements in Kendall's 𝜏 correlation range from 0.168 to 0.494.

Computational Advantages of eRAG

• eRAG offers significant computational advantages, improving runtime and consuming up to 50 times less GPU memory than end-to-end evaluation.