BEIR vs MTEB Leaderboard
BEIR or MTEB? BEIR is a zero-shot retrieval benchmark across 18 public datasets for comparing lexical, sparse, dense, late-interaction and re-ranking models. MTEB benchmarks embedding models across 8 task types, 58 datasets and 112 languages in its original release.
| BEIR | MTEB Leaderboard | |
|---|---|---|
| Made by | UKP Lab & collaborators | MTEB community (Hugging Face Space) |
| Focus | Retrieval only | Embeddings across many task types, including retrieval |
| Coverage | 18 datasets | 8 tasks, 58 datasets, 112 languages (expanded in MMTEB) |
| Key finding | BM25 is a strong baseline; re-ranking best but costly | No single embedding method best across all tasks |
| Doesn't measure | Whether a generated answer is correct | Fit to your own documents |
| License | Apache-2.0 | — |
Verdict: when to use which
Use BEIR if you are choosing a retrieval architecture and care about zero-shot generalisation and cost.
Use MTEB Leaderboard if you are choosing an embedding model and want results by task type or language.
Sources
Other comparisons: SWE-bench vs SWE-bench-Live · HumanEval vs BigCodeBench · HumanEval vs LiveCodeBench · MMLU vs GPQA · AlpacaEval vs MT-Bench (LLM-as-a-judge paper) · MT-Bench (LLM-as-a-judge paper) vs LMArena (Chatbot Arena) · WebArena vs OSWorld · HarmBench vs StrongREJECT · GAIA vs BrowseComp