RAG and retrieval evals, compared
Retrieval evals measure whether the right documents are found; RAG evals measure whether the final answer is grounded in them. Most teams need one of each.
| Eval | Made by | What it tests | License | Last updated | Source |
|---|---|---|---|---|---|
| Ragas | Exploding Gradients | Faithfulness, relevance and retrieval quality of your own RAG system. | — | — | docs.ragas.io |
| BEIR | UKP Lab & collaborators | Zero-shot retrieval across diverse domains. | Apache-2.0 | Repo last commit 2025-10-16 | github.com |
| MTEB Leaderboard | MTEB community (Hugging Face Space) | Embedding model quality for search and RAG. | — | — | huggingface.co |
| CRAG | Meta & collaborators | End-to-end RAG answer correctness and hallucination. | — | — | arxiv.org |
| FRAMES | Multi-hop retrieval-augmented answering. | — | — | arxiv.org |
In short
Ragas — An open-source library of metrics for RAG apps and agents, such as whether an answer is faithful to the retrieved context.
BEIR — A collection of retrieval datasets for testing how well search and embedding models work on domains they were not trained on.
MTEB Leaderboard — The standard leaderboard for comparing embedding models on retrieval, classification, clustering and more.
CRAG — A question-answering benchmark for RAG with mock web and knowledge-graph search, including questions whose answers change over time.
FRAMES — Questions that need facts from several documents combined, testing retrieval, reasoning and factuality together.
Browse the full RAG & retrieval category, or see other comparisons.