# Evals Review

> A curated, source-linked directory of AI evals, benchmarks and rubric frameworks. Every fact links to the eval's own paper, repository or official page. Last updated 2026-10-15.

Full plain-text summaries of every eval: https://evalsreview.com/llms-full.txt

## Coding

- [SWE-bench](https://evalsreview.com/eval/swe-bench): Fixing real bugs in real repositories, judged by unit tests.
- [HumanEval](https://evalsreview.com/eval/humaneval): Writing a short correct function from a docstring.
- [LiveCodeBench](https://evalsreview.com/eval/livecodebench): Code generation, self-repair and test prediction on fresh problems.
- [Aider leaderboards (Polyglot)](https://evalsreview.com/eval/aider-polyglot): Practical multi-language code editing.
- [BigCodeBench](https://evalsreview.com/eval/bigcodebench): Library-heavy, practical Python tasks.
- [SWE-bench-Live](https://evalsreview.com/eval/swe-bench-live): Fixing live software issues across languages and platforms, judged by tests in Docker.

## Agents & tool use

- [GAIA](https://evalsreview.com/eval/gaia): General assistant skills: browsing, file handling, multi-step reasoning.
- [WebArena](https://evalsreview.com/eval/webarena): Autonomous web navigation and task completion.
- [OSWorld](https://evalsreview.com/eval/osworld): Computer-use agents operating real operating systems and apps.
- [Terminal-Bench](https://evalsreview.com/eval/terminal-bench): Agents working in the shell end to end.
- [BrowseComp](https://evalsreview.com/eval/browsecomp): Deep, creative web research by browsing agents.
- [MLE-bench](https://evalsreview.com/eval/mle-bench): End-to-end ML engineering: data prep, training, experimentation and submission quality.
- [Berkeley Function Calling Leaderboard (BFCL)](https://evalsreview.com/eval/bfcl): Correct function/tool calling across simple, parallel, multi-turn and agentic tasks.

## Customer support

- [τ-bench (tau-bench)](https://evalsreview.com/eval/tau-bench): Policy-following support agents with tools, plus consistency across repeated runs.

## Writing & instruction following

- [EQ-Bench Creative Writing](https://evalsreview.com/eval/eq-bench-creative): Creative writing quality judged against a rubric.
- [AlpacaEval](https://evalsreview.com/eval/alpacaeval): General helpfulness of chat responses.
- [MT-Bench (LLM-as-a-judge paper)](https://evalsreview.com/eval/mt-bench): Multi-turn conversation quality; how reliable LLM judges are.
- [IFEval](https://evalsreview.com/eval/ifeval): Following explicit, verifiable instructions.
- [LMArena (Chatbot Arena)](https://evalsreview.com/eval/lmarena): Which models people prefer on their own real prompts, by pairwise human votes.

## Safety & factuality

- [HarmBench](https://evalsreview.com/eval/harmbench): Robustness to jailbreaks and harmful requests.
- [TruthfulQA](https://evalsreview.com/eval/truthfulqa): Truthfulness versus imitating human misconceptions.
- [StrongREJECT](https://evalsreview.com/eval/strongreject): Real effectiveness of jailbreak attacks.
- [SimpleQA](https://evalsreview.com/eval/simpleqa): Short-form factuality and calibration.
- [XSTest](https://evalsreview.com/eval/xstest): Over-refusal (exaggerated safety).

## Healthcare

- [HealthBench](https://evalsreview.com/eval/healthbench): Emergency referrals, handling uncertainty, global health, communication quality and more.
- [MedQA](https://evalsreview.com/eval/medqa): Medical exam knowledge.
- [MedHELM](https://evalsreview.com/eval/medhelm): Broad clinical tasks: documentation, decision support, patient communication and more.
- [KARMA-OpenMedEvalKit](https://evalsreview.com/eval/karma-openmedevalkit): Medical AI in Indian contexts, including speech recognition and record parsing.
- [SAKHI](https://evalsreview.com/eval/sakhi): Sensitive maternal-health answers in Indian languages.

## Multilingual & Indian languages

- [IndicGenBench](https://evalsreview.com/eval/indicgenbench): Text generation quality in Indic languages.
- [MILU](https://evalsreview.com/eval/milu): Multi-subject understanding in Indian languages and culture.
- [Global-MMLU](https://evalsreview.com/eval/global-mmlu): Knowledge across languages and cultural sensitivity.
- [Samiksha](https://evalsreview.com/eval/samiksha): Culturally grounded health chatbot answers in Indian languages.

## RAG & retrieval

- [Ragas](https://evalsreview.com/eval/ragas): Faithfulness, relevance and retrieval quality of your own RAG system.
- [BEIR](https://evalsreview.com/eval/beir): Zero-shot retrieval across diverse domains.
- [MTEB Leaderboard](https://evalsreview.com/eval/mteb): Embedding model quality for search and RAG.
- [CRAG](https://evalsreview.com/eval/crag): End-to-end RAG answer correctness and hallucination.
- [FRAMES](https://evalsreview.com/eval/frames): Multi-hop retrieval-augmented answering.

## Reasoning & knowledge

- [GPQA](https://evalsreview.com/eval/gpqa): Expert-level science reasoning.
- [Humanity's Last Exam](https://evalsreview.com/eval/humanitys-last-exam): Frontier academic knowledge and reasoning.
- [LiveBench](https://evalsreview.com/eval/livebench): Hard, contamination-limited tasks with verifiable ground-truth answers (no LLM judge).
- [MMLU](https://evalsreview.com/eval/mmlu): World knowledge and problem solving across many school and professional subjects.

## Rubric libraries & frameworks

- [Inspect Evals](https://evalsreview.com/eval/inspect-evals): Many benchmarks under one runner: coding, agents, safety, knowledge.
- [OpenAI Evals](https://evalsreview.com/eval/openai-evals): Building and running your own evals; community registry.
- [LLMJury rubric library](https://evalsreview.com/eval/llmjury-rubrics): LLM-as-judge rubrics for support, RAG, summaries, code and tone.
- [Prompt Assay judge templates](https://evalsreview.com/eval/promptassay-judge-templates): Starter LLM-as-judge prompts.
- [Jev (TypeSafe AI)](https://evalsreview.com/eval/typesafe-jev): Not a benchmark: a tool for AI-judge / LLM-as-judge style scoring in your own evals.
- [HELM](https://evalsreview.com/eval/helm): Standardised, multi-metric comparison of models across shared scenarios.

## Image & vision

- [MMMU](https://evalsreview.com/eval/mmmu): Expert-level multimodal understanding and reasoning.
- [MathVista](https://evalsreview.com/eval/mathvista): Mathematical reasoning over figures, plots and images.
- [MMBench](https://evalsreview.com/eval/mmbench): Broad perception and reasoning skills of vision-language models.
- [ChartQA](https://evalsreview.com/eval/chartqa): Question answering over charts.

## Comparisons

- [Coding benchmarks for LLMs, compared](https://evalsreview.com/best/coding-benchmarks)
- [AI agent and tool-use evals, compared](https://evalsreview.com/best/agent-evals)
- [LLM-as-judge tools, rubrics and eval frameworks](https://evalsreview.com/best/llm-as-judge-tools)
- [Image and vision benchmarks, compared](https://evalsreview.com/best/vision-benchmarks)
- [RAG and retrieval evals, compared](https://evalsreview.com/best/rag-evals)
- [General knowledge, reasoning and preference benchmarks, compared](https://evalsreview.com/best/general-llm-benchmarks)
- [SWE-bench vs SWE-bench-Live](https://evalsreview.com/compare/swe-bench-vs-swe-bench-live): SWE-bench or SWE-bench-Live?
- [HumanEval vs BigCodeBench](https://evalsreview.com/compare/humaneval-vs-bigcodebench): HumanEval or BigCodeBench?
- [HumanEval vs LiveCodeBench](https://evalsreview.com/compare/humaneval-vs-livecodebench): HumanEval or LiveCodeBench?
- [MMLU vs GPQA](https://evalsreview.com/compare/mmlu-vs-gpqa): MMLU or GPQA?
- [AlpacaEval vs MT-Bench (LLM-as-a-judge paper)](https://evalsreview.com/compare/alpacaeval-vs-mt-bench): AlpacaEval or MT-Bench?
- [MT-Bench (LLM-as-a-judge paper) vs LMArena (Chatbot Arena)](https://evalsreview.com/compare/mt-bench-vs-lmarena): MT-Bench or LMArena (Chatbot Arena)?
- [BEIR vs MTEB Leaderboard](https://evalsreview.com/compare/beir-vs-mteb): BEIR or MTEB?
- [WebArena vs OSWorld](https://evalsreview.com/compare/webarena-vs-osworld): WebArena or OSWorld?
- [HarmBench vs StrongREJECT](https://evalsreview.com/compare/harmbench-vs-strongreject): HarmBench or StrongREJECT?
- [GAIA vs BrowseComp](https://evalsreview.com/compare/gaia-vs-browsecomp): GAIA or BrowseComp?

## Optional

- [About and methodology](https://evalsreview.com/about)
- [How to choose an eval](https://evalsreview.com/guide)