{"note":"Evals Review dataset, 2026-10-16. Facts restate each eval's own sources (see the sources column). Licensed CC BY 4.0: credit Evals Review (https://evalsreview.com/data) when you reuse it.","updated":"2026-10-16","license":"CC-BY-4.0","license_url":"https://creativecommons.org/licenses/by/4.0/","attribution":"Evals Review (https://evalsreview.com/data)","evals":[{"slug":"swe-bench","name":"SWE-bench","creator":"Princeton NLP & collaborators","category":"Coding","source_url":"https://www.swebench.com/","page_url":"https://evalsreview.com/eval/swe-bench","tests":"Fixing real bugs in real repositories, judged by unit tests.","summary":"Gives a model a real GitHub issue from an open-source Python project and checks whether its code change makes the project's tests pass.","license":"","last_updated":"","added":"2026-10-15","sources":"https://arxiv.org/abs/2310.06770 https://github.com/SWE-bench/SWE-bench https://www.swebench.com/ https://openai.com/index/introducing-swe-bench-verified/ https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/ https://openai.com/index/separating-signal-from-noise-coding-evaluations/"},{"slug":"humaneval","name":"HumanEval","creator":"OpenAI","category":"Coding","source_url":"https://github.com/openai/human-eval","page_url":"https://evalsreview.com/eval/humaneval","tests":"Writing a short correct function from a docstring.","summary":"A classic set of hand-written Python function problems, each checked by running hidden tests. Useful as a baseline, though top models have largely mastered it.","license":"MIT","last_updated":"Repo last commit 2025-01-17","added":"2026-10-15","sources":"https://arxiv.org/abs/2107.03374 https://github.com/openai/human-eval"},{"slug":"livecodebench","name":"LiveCodeBench","creator":"LiveCodeBench team (academic)","category":"Coding","source_url":"https://livecodebench.github.io/","page_url":"https://evalsreview.com/eval/livecodebench","tests":"Code generation, self-repair and test prediction on fresh problems.","summary":"Keeps collecting new contest problems over time, so you can test a model only on problems released after its training cutoff.","license":"","last_updated":"","added":"2026-10-15","sources":"https://arxiv.org/abs/2403.07974 https://github.com/LiveCodeBench/LiveCodeBench https://livecodebench.github.io/"},{"slug":"aider-polyglot","name":"Aider leaderboards (Polyglot)","creator":"Aider","category":"Coding","source_url":"https://aider.chat/docs/leaderboards/","page_url":"https://evalsreview.com/eval/aider-polyglot","tests":"Practical multi-language code editing.","summary":"Measures how well a model edits code in many programming languages inside the Aider coding assistant, including whether it follows the edit format.","license":"","last_updated":"","added":"2026-10-15","sources":"https://aider.chat/docs/leaderboards/ https://github.com/Aider-AI/aider https://github.com/Aider-AI/aider/blob/main/benchmark/README.md"},{"slug":"bigcodebench","name":"BigCodeBench","creator":"BigCode project","category":"Coding","source_url":"https://github.com/bigcode-project/bigcodebench","page_url":"https://evalsreview.com/eval/bigcodebench","tests":"Library-heavy, practical Python tasks.","summary":"Programming tasks that require calling many real libraries correctly, closer to everyday developer work than puzzle-style problems.","license":"Apache-2.0","last_updated":"Repo last commit 2026-01-03","added":"2026-10-15","sources":"https://arxiv.org/abs/2406.15877 https://github.com/bigcode-project/bigcodebench https://huggingface.co/spaces/bigcode/bigcodebench-leaderboard"},{"slug":"gaia","name":"GAIA","creator":"Meta, Hugging Face & collaborators","category":"Agents & tool use","source_url":"https://arxiv.org/abs/2311.12983","page_url":"https://evalsreview.com/eval/gaia","tests":"General assistant skills: browsing, file handling, multi-step reasoning.","summary":"Questions that are easy for a person with a browser and tools but hard for AI assistants, each with one short, checkable answer.","license":"","last_updated":"","added":"2026-10-15","sources":"https://arxiv.org/abs/2311.12983 https://huggingface.co/datasets/gaia-benchmark/GAIA https://huggingface.co/spaces/gaia-benchmark/leaderboard"},{"slug":"tau-bench","name":"τ-bench (tau-bench)","creator":"Sierra Research","category":"Customer support","source_url":"https://github.com/sierra-research/tau-bench","page_url":"https://evalsreview.com/eval/tau-bench","tests":"Policy-following support agents with tools, plus consistency across repeated runs.","summary":"Simulates a customer talking to an AI agent in domains like retail and airlines. The agent must follow company policy and use tools; the final database state is checked.","license":"MIT","last_updated":"Repo last commit 2026-03-18","added":"2026-10-15","sources":"https://arxiv.org/abs/2406.12045 https://github.com/sierra-research/tau-bench https://github.com/sierra-research/tau2-bench"},{"slug":"webarena","name":"WebArena","creator":"Carnegie Mellon University","category":"Agents & tool use","source_url":"https://webarena.dev/","page_url":"https://evalsreview.com/eval/webarena","tests":"Autonomous web navigation and task completion.","summary":"Self-hosted copies of realistic websites (shop, forum, code host and more) where an agent must complete tasks by clicking and typing.","license":"Apache-2.0","last_updated":"Repo last commit 2025-11-26","added":"2026-10-15","sources":"https://arxiv.org/abs/2307.13854 https://github.com/web-arena-x/webarena https://webarena.dev/"},{"slug":"osworld","name":"OSWorld","creator":"XLANG Lab & collaborators","category":"Agents & tool use","source_url":"https://os-world.github.io/","page_url":"https://evalsreview.com/eval/osworld","tests":"Computer-use agents operating real operating systems and apps.","summary":"Puts an agent in a real computer environment and asks it to do tasks across desktop apps, checked by scripts.","license":"","last_updated":"","added":"2026-10-15","sources":"https://arxiv.org/abs/2404.07972 https://github.com/xlang-ai/OSWorld https://os-world.github.io/"},{"slug":"terminal-bench","name":"Terminal-Bench","creator":"Terminal-Bench team","category":"Agents & tool use","source_url":"https://www.tbench.ai/","page_url":"https://evalsreview.com/eval/terminal-bench","tests":"Agents working in the shell end to end.","summary":"Tasks an agent must finish inside a sandboxed command-line terminal, such as compiling code, training models or setting up servers, checked by tests. It is now versioned continuously: 2.0 (89 tasks), 2.1 (the same 89 with 26–28 fixed), then 3.0 and 4.0 (4.0 recalibrated resources, removed 8 tasks and fixed 19).","license":"","last_updated":"4.0 is current (checked 16 Oct 2026)","added":"2026-10-15","sources":"https://arxiv.org/abs/2601.11868 https://github.com/laude-institute/terminal-bench https://github.com/harbor-framework/harbor https://www.tbench.ai/ https://www.tbench.ai/news/terminal-bench-2-1 https://www.tbench.ai/news/terminal-bench-4-0 https://github.com/harbor-framework/terminal-bench-2-1 https://www.tbench.ai/leaderboard https://www.vals.ai/benchmarks/terminal-bench-4 https://snorkel.ai/leaderboard/terminal-bench/ https://artificialanalysis.ai/evaluations/terminalbench-4-0"},{"slug":"browsecomp","name":"BrowseComp","creator":"OpenAI","category":"Agents & tool use","source_url":"https://openai.com/index/browsecomp/","page_url":"https://evalsreview.com/eval/browsecomp","tests":"Deep, creative web research by browsing agents.","summary":"1,266 very hard-to-find facts with short answers that are easy to verify, built to test persistent web-browsing agents.","license":"","last_updated":"Published 2025-04-10","added":"2026-10-15","sources":"https://arxiv.org/abs/2504.12516 https://github.com/openai/simple-evals https://openai.com/index/browsecomp/"},{"slug":"eq-bench-creative","name":"EQ-Bench Creative Writing","creator":"EQ-Bench (Sam Paech)","category":"Writing & instruction following","source_url":"https://eqbench.com/creative_writing.html","page_url":"https://evalsreview.com/eval/eq-bench-creative","tests":"Creative writing quality judged against a rubric.","summary":"Scores creative writing from models with an LLM judge using a detailed rubric, aiming to capture quality rather than just correctness.","license":"","last_updated":"","added":"2026-10-15","sources":"https://eqbench.com/creative_writing.html https://github.com/EQ-bench/creative-writing-bench https://eqbench.com/about.html#creative-writing-v3"},{"slug":"alpacaeval","name":"AlpacaEval","creator":"Stanford Tatsu Lab","category":"Writing & instruction following","source_url":"https://github.com/tatsu-lab/alpaca_eval","page_url":"https://evalsreview.com/eval/alpacaeval","tests":"General helpfulness of chat responses.","summary":"Compares a model's answers against a reference model's answers using an automatic LLM judge, with a correction for judges that prefer longer answers.","license":"Apache-2.0","last_updated":"Repo last commit 2025-08-09","added":"2026-10-15","sources":"https://arxiv.org/abs/2404.04475 https://github.com/tatsu-lab/alpaca_eval"},{"slug":"mt-bench","name":"MT-Bench (LLM-as-a-judge paper)","creator":"LMSYS","category":"Writing & instruction following","source_url":"https://arxiv.org/abs/2306.05685","page_url":"https://evalsreview.com/eval/mt-bench","tests":"Multi-turn conversation quality; how reliable LLM judges are.","summary":"The paper that popularised using a strong model as a judge, with multi-turn questions and an analysis of where judges agree with humans and where they are biased.","license":"","last_updated":"","added":"2026-10-15","sources":"https://arxiv.org/abs/2306.05685 https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge"},{"slug":"ifeval","name":"IFEval","creator":"Google","category":"Writing & instruction following","source_url":"https://arxiv.org/abs/2311.07911","page_url":"https://evalsreview.com/eval/ifeval","tests":"Following explicit, verifiable instructions.","summary":"Prompts with instructions a program can check automatically (word counts, formats, keywords), so no judge model is needed.","license":"","last_updated":"","added":"2026-10-15","sources":"https://arxiv.org/abs/2311.07911 https://github.com/google-research/google-research/tree/master/instruction_following_eval"},{"slug":"harmbench","name":"HarmBench","creator":"Center for AI Safety & collaborators","category":"Safety & factuality","source_url":"https://www.harmbench.org/","page_url":"https://evalsreview.com/eval/harmbench","tests":"Robustness to jailbreaks and harmful requests.","summary":"A standard framework for red-teaming: it tests how easily attacks can make a model produce harmful content and how well defences hold.","license":"","last_updated":"","added":"2026-10-15","sources":"https://arxiv.org/abs/2402.04249 https://github.com/centerforaisafety/HarmBench https://www.harmbench.org/"},{"slug":"truthfulqa","name":"TruthfulQA","creator":"Oxford & OpenAI researchers","category":"Safety & factuality","source_url":"https://github.com/sylinrl/TruthfulQA","page_url":"https://evalsreview.com/eval/truthfulqa","tests":"Truthfulness versus imitating human misconceptions.","summary":"Questions designed so that common misconceptions are tempting; checks whether the model repeats popular falsehoods.","license":"Apache-2.0","last_updated":"Repo last commit 2025-01-16","added":"2026-10-15","sources":"https://arxiv.org/abs/2109.07958 https://github.com/sylinrl/TruthfulQA"},{"slug":"strongreject","name":"StrongREJECT","creator":"UC Berkeley researchers","category":"Safety & factuality","source_url":"https://arxiv.org/abs/2402.10260","page_url":"https://evalsreview.com/eval/strongreject","tests":"Real effectiveness of jailbreak attacks.","summary":"A better way to score jailbreaks: it checks whether the harmful answer is actually useful, not just whether the model refused.","license":"","last_updated":"","added":"2026-10-15","sources":"https://arxiv.org/abs/2402.10260 https://github.com/dsbowen/strong_reject https://strong-reject.readthedocs.io/en/latest/"},{"slug":"simpleqa","name":"SimpleQA","creator":"OpenAI","category":"Safety & factuality","source_url":"https://openai.com/index/introducing-simpleqa/","page_url":"https://evalsreview.com/eval/simpleqa","tests":"Short-form factuality and calibration.","summary":"4,326 short fact questions with one indisputable answer each; rewards models that answer correctly or admit they don't know instead of hallucinating.","license":"","last_updated":"Published 2024-10-30","added":"2026-10-15","sources":"https://arxiv.org/abs/2411.04368 https://github.com/openai/simple-evals https://openai.com/index/introducing-simpleqa/"},{"slug":"xstest","name":"XSTest","creator":"Bocconi, Oxford & collaborators","category":"Safety & factuality","source_url":"https://arxiv.org/abs/2308.01263","page_url":"https://evalsreview.com/eval/xstest","tests":"Over-refusal (exaggerated safety).","summary":"Safe prompts that merely sound dangerous, to catch models that refuse too much.","license":"","last_updated":"","added":"2026-10-15","sources":"https://arxiv.org/abs/2308.01263 https://github.com/paul-rottger/xstest https://aclanthology.org/2024.naacl-long.301/"},{"slug":"healthbench","name":"HealthBench","creator":"OpenAI","category":"Healthcare","source_url":"https://openai.com/index/healthbench/","page_url":"https://evalsreview.com/eval/healthbench","tests":"Emergency referrals, handling uncertainty, global health, communication quality and more.","summary":"5,000 realistic, multi-turn health conversations, each graded against rubrics written by 262 physicians. Includes harder and consensus-validated subsets.","license":"","last_updated":"Published 2025-05-12","added":"2026-10-15","sources":"https://arxiv.org/abs/2505.08775 https://github.com/openai/simple-evals https://openai.com/index/healthbench/"},{"slug":"medqa","name":"MedQA","creator":"Jin et al.","category":"Healthcare","source_url":"https://github.com/jind11/MedQA","page_url":"https://evalsreview.com/eval/medqa","tests":"Medical exam knowledge.","summary":"Multiple-choice questions from medical licensing exams. Easy to run, but exam-style and widely used, so it reveals little about real clinical conversations.","license":"MIT","last_updated":"Repo last commit 2022-12-01","added":"2026-10-15","sources":"https://arxiv.org/abs/2009.13081 https://github.com/jind11/MedQA"},{"slug":"medhelm","name":"MedHELM","creator":"Stanford CRFM","category":"Healthcare","source_url":"https://crfm.stanford.edu/helm/medhelm/latest/","page_url":"https://evalsreview.com/eval/medhelm","tests":"Broad clinical tasks: documentation, decision support, patient communication and more.","summary":"A holistic medical leaderboard covering many real clinical task types, not only exam questions.","license":"","last_updated":"","added":"2026-10-15","sources":"https://arxiv.org/abs/2505.23802 https://crfm.stanford.edu/helm/medhelm/latest/ https://github.com/stanford-crfm/helm https://crfm.stanford.edu/helm/"},{"slug":"karma-openmedevalkit","name":"KARMA-OpenMedEvalKit","creator":"Eka Care","category":"Healthcare","source_url":"https://www.eka.care/services/introducing-karma-openmedevalkit-an-open-source-framework-for-medical-ai-evaluation","page_url":"https://evalsreview.com/eval/karma-openmedevalkit","tests":"Medical AI in Indian contexts, including speech recognition and record parsing.","summary":"An open toolkit for evaluating medical AI with an emphasis on Indian healthcare, covering text, image and audio models and adding new India-focused datasets.","license":"","last_updated":"","added":"2026-10-15","sources":"https://www.eka.care/services/introducing-karma-openmedevalkit-an-open-source-framework-for-medical-ai-evaluation https://karma.eka.care https://github.com/eka-care/KARMA-OpenMedEvalKit"},{"slug":"sakhi","name":"SAKHI","creator":"SimPPL & collaborators","category":"Healthcare","source_url":"https://www.simppl.org/papers/multilingual-health-bench.pdf","page_url":"https://evalsreview.com/eval/sakhi","tests":"Sensitive maternal-health answers in Indian languages.","summary":"A maternal-health benchmark built around a WhatsApp chatbot for Hindi- and Marathi-speaking mothers, with rubric verdicts from Indian doctors to calibrate the LLM judge.","license":"","last_updated":"","added":"2026-10-15","sources":"https://www.simppl.org/papers/multilingual-health-bench.pdf https://huggingface.co/datasets/SimPPL/sakhi https://ai4health.simppl.org/projects/sakhi"},{"slug":"indicgenbench","name":"IndicGenBench","creator":"Google Research","category":"Multilingual & Indian languages","source_url":"https://github.com/google-research-datasets/indic-gen-bench","page_url":"https://evalsreview.com/eval/indicgenbench","tests":"Text generation quality in Indic languages.","summary":"Generation tasks such as summarisation, translation and question answering across many Indian languages, including low-resource ones.","license":"","last_updated":"Repo last commit 2024-09-01","added":"2026-10-15","sources":"https://arxiv.org/abs/2404.16816 https://github.com/google-research-datasets/indic-gen-bench https://huggingface.co/collections/google/indicgenbench-663185166ea914038209b723"},{"slug":"milu","name":"MILU","creator":"AI4Bharat","category":"Multilingual & Indian languages","source_url":"https://arxiv.org/abs/2411.02538","page_url":"https://evalsreview.com/eval/milu","tests":"Multi-subject understanding in Indian languages and culture.","summary":"A knowledge benchmark in Indian languages that includes India-specific subjects, not just translated Western exams.","license":"","last_updated":"","added":"2026-10-15","sources":"https://arxiv.org/abs/2411.02538 https://github.com/AI4Bharat/MILU https://huggingface.co/datasets/ai4bharat/MILU"},{"slug":"global-mmlu","name":"Global-MMLU","creator":"Cohere For AI & collaborators","category":"Multilingual & Indian languages","source_url":"https://huggingface.co/datasets/CohereForAI/Global-MMLU","page_url":"https://evalsreview.com/eval/global-mmlu","tests":"Knowledge across languages and cultural sensitivity.","summary":"MMLU improved and translated into many languages, with labels marking which questions are culturally specific.","license":"","last_updated":"","added":"2026-10-15","sources":"https://arxiv.org/abs/2412.03304 https://huggingface.co/datasets/CohereForAI/Global-MMLU"},{"slug":"samiksha","name":"Samiksha","creator":"Microsoft Research India & collaborators","category":"Multilingual & Indian languages","source_url":"https://arxiv.org/abs/2509.24506","page_url":"https://evalsreview.com/eval/samiksha","tests":"Culturally grounded health chatbot answers in Indian languages.","summary":"A community-built pipeline for evaluating health chatbots in Hindi, Kannada and Malayalam, combining human raters with LLM judges.","license":"","last_updated":"","added":"2026-10-15","sources":"https://arxiv.org/abs/2509.24506 https://doi.org/10.1145/3772318.3791172"},{"slug":"ragas","name":"Ragas","creator":"Exploding Gradients","category":"RAG & retrieval","source_url":"https://docs.ragas.io/","page_url":"https://evalsreview.com/eval/ragas","tests":"Faithfulness, relevance and retrieval quality of your own RAG system.","summary":"An open-source library of metrics for RAG apps and agents, such as whether an answer is faithful to the retrieved context.","license":"","last_updated":"","added":"2026-10-15","sources":"https://arxiv.org/abs/2309.15217 https://github.com/vibrantlabsai/ragas https://docs.ragas.io/ https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/"},{"slug":"beir","name":"BEIR","creator":"UKP Lab & collaborators","category":"RAG & retrieval","source_url":"https://github.com/beir-cellar/beir","page_url":"https://evalsreview.com/eval/beir","tests":"Zero-shot retrieval across diverse domains.","summary":"A collection of retrieval datasets for testing how well search and embedding models work on domains they were not trained on.","license":"Apache-2.0","last_updated":"Repo last commit 2025-10-16","added":"2026-10-15","sources":"https://arxiv.org/abs/2104.08663 https://github.com/beir-cellar/beir https://dl.acm.org/doi/10.1145/3626772.3657862"},{"slug":"mteb","name":"MTEB Leaderboard","creator":"MTEB community (Hugging Face Space)","category":"RAG & retrieval","source_url":"https://huggingface.co/spaces/mteb/leaderboard","page_url":"https://evalsreview.com/eval/mteb","tests":"Embedding model quality for search and RAG.","summary":"The public leaderboard of the Massive Text Embedding Benchmark, comparing embedding models on retrieval, classification, clustering, reranking and other tasks.","license":"","last_updated":"","added":"2026-10-15","sources":"https://arxiv.org/abs/2210.07316 https://arxiv.org/abs/2502.13595 https://github.com/embeddings-benchmark/mteb https://huggingface.co/spaces/mteb/leaderboard"},{"slug":"crag","name":"CRAG","creator":"Meta & collaborators","category":"RAG & retrieval","source_url":"https://arxiv.org/abs/2406.04744","page_url":"https://evalsreview.com/eval/crag","tests":"End-to-end RAG answer correctness and hallucination.","summary":"A question-answering benchmark for RAG with mock web and knowledge-graph search, including questions whose answers change over time.","license":"","last_updated":"","added":"2026-10-15","sources":"https://arxiv.org/abs/2406.04744 https://github.com/facebookresearch/CRAG"},{"slug":"frames","name":"FRAMES","creator":"Google","category":"RAG & retrieval","source_url":"https://arxiv.org/abs/2409.12941","page_url":"https://evalsreview.com/eval/frames","tests":"Multi-hop retrieval-augmented answering.","summary":"Questions that need facts from several documents combined, testing retrieval, reasoning and factuality together.","license":"","last_updated":"","added":"2026-10-15","sources":"https://arxiv.org/abs/2409.12941 https://huggingface.co/datasets/google/frames-benchmark"},{"slug":"gpqa","name":"GPQA","creator":"NYU & collaborators","category":"Reasoning & knowledge","source_url":"https://github.com/idavidrein/gpqa","page_url":"https://evalsreview.com/eval/gpqa","tests":"Expert-level science reasoning.","summary":"Graduate-level science questions written by experts so that web search doesn't easily help.","license":"MIT","last_updated":"Repo last commit 2024-09-30","added":"2026-10-15","sources":"https://arxiv.org/abs/2311.12022 https://github.com/idavidrein/gpqa https://epoch.ai/benchmarks/gpqa-diamond"},{"slug":"humanitys-last-exam","name":"Humanity's Last Exam","creator":"Center for AI Safety & Scale AI","category":"Reasoning & knowledge","source_url":"https://lastexam.ai/","page_url":"https://evalsreview.com/eval/humanitys-last-exam","tests":"Frontier academic knowledge and reasoning.","summary":"Very hard questions contributed by experts across many fields, made to stay difficult after older benchmarks saturated.","license":"MIT","last_updated":"","added":"2026-10-15","sources":"https://arxiv.org/abs/2501.14249 https://github.com/centerforaisafety/hle https://lastexam.ai/ https://www.nature.com/articles/s41586-025-09962-4"},{"slug":"inspect-evals","name":"Inspect Evals","creator":"UK AI Security Institute & community","category":"Rubric libraries & frameworks","source_url":"https://github.com/UKGovernmentBEIS/inspect_evals","page_url":"https://evalsreview.com/eval/inspect-evals","tests":"Many benchmarks under one runner: coding, agents, safety, knowledge.","summary":"A large, actively maintained collection of ready-to-run evals implemented in the Inspect framework.","license":"MIT","last_updated":"Repo last commit 2026-10-15","added":"2026-10-15","sources":"https://github.com/UKGovernmentBEIS/inspect_evals"},{"slug":"openai-evals","name":"OpenAI Evals","creator":"OpenAI","category":"Rubric libraries & frameworks","source_url":"https://github.com/openai/evals","page_url":"https://evalsreview.com/eval/openai-evals","tests":"Building and running your own evals; community registry.","summary":"A framework and open registry of evals, including templates for model-graded evaluation.","license":"","last_updated":"Repo last commit 2026-04-14","added":"2026-10-15","sources":"https://github.com/openai/evals"},{"slug":"llmjury-rubrics","name":"LLMJury rubric library","creator":"LLMJury","category":"Rubric libraries & frameworks","source_url":"https://llmjury.com/rubrics","page_url":"https://evalsreview.com/eval/llmjury-rubrics","tests":"LLM-as-judge rubrics for support, RAG, summaries, code and tone.","summary":"Paste-ready judge rubrics for questions like 'did the support reply resolve the request?', each with score meanings, examples and known failure modes. Made by a vendor.","license":"","last_updated":"","added":"2026-10-15","sources":"https://llmjury.com/rubrics"},{"slug":"promptassay-judge-templates","name":"Prompt Assay judge templates","creator":"Prompt Assay","category":"Rubric libraries & frameworks","source_url":"https://promptassay.ai/blog/llm-as-a-judge-prompt-templates","page_url":"https://evalsreview.com/eval/promptassay-judge-templates","tests":"Starter LLM-as-judge prompts.","summary":"Copy-paste YAML judge templates covering basics like relevance and tone plus RAG, code, summarisation and agent traces. Made by a vendor.","license":"","last_updated":"","added":"2026-10-15","sources":"https://promptassay.ai/blog/llm-as-a-judge-prompt-templates"},{"slug":"swe-bench-live","name":"SWE-bench-Live","creator":"Microsoft Research & collaborators","category":"Coding","source_url":"https://swe-bench-live.github.io/","page_url":"https://evalsreview.com/eval/swe-bench-live","tests":"Fixing live software issues across languages and platforms, judged by tests in Docker.","summary":"A continuously updated version of SWE-bench: real GitHub issues with Docker sandboxes, now covering multiple languages and operating systems, with new verified tasks added over time.","license":"MIT","last_updated":"Repo last commit 2026-09-25","added":"2026-10-15","sources":"https://arxiv.org/abs/2505.23419 https://github.com/microsoft/SWE-bench-Live https://swe-bench-live.github.io/"},{"slug":"mle-bench","name":"MLE-bench","creator":"OpenAI","category":"Agents & tool use","source_url":"https://github.com/openai/mle-bench","page_url":"https://evalsreview.com/eval/mle-bench","tests":"End-to-end ML engineering: data prep, training, experimentation and submission quality.","summary":"75 real Kaggle machine-learning competitions where an agent must prepare data, train models and submit results, scored against human leaderboard medal thresholds.","license":"","last_updated":"Repo last commit 2026-04-24","added":"2026-10-15","sources":"https://arxiv.org/abs/2410.07095 https://github.com/openai/mle-bench"},{"slug":"bfcl","name":"Berkeley Function Calling Leaderboard (BFCL)","creator":"UC Berkeley (Gorilla)","category":"Agents & tool use","source_url":"https://gorilla.cs.berkeley.edu/leaderboard.html","page_url":"https://evalsreview.com/eval/bfcl","tests":"Correct function/tool calling across simple, parallel, multi-turn and agentic tasks.","summary":"Measures how accurately a model calls tools and functions, including parallel calls, multi-turn tool use and agentic scenarios, with an actively updated public leaderboard.","license":"Apache-2.0","last_updated":"Repo last commit 2026-03-23","added":"2026-10-15","sources":"https://gorilla.cs.berkeley.edu/leaderboard.html https://github.com/ShishirPatil/gorilla/tree/main/berkeley-function-call-leaderboard https://gorilla.cs.berkeley.edu/blogs/13_bfcl_v3_multi_turn.html"},{"slug":"livebench","name":"LiveBench","creator":"LiveBench team (academic)","category":"Reasoning & knowledge","source_url":"https://livebench.github.io/","page_url":"https://evalsreview.com/eval/livebench","tests":"Hard, contamination-limited tasks with verifiable ground-truth answers (no LLM judge).","summary":"A regularly refreshed LLM benchmark with objective, auto-checkable answers across math, coding, reasoning and more, designed to limit training-data contamination.","license":"","last_updated":"Repo last commit 2026-10-14","added":"2026-10-15","sources":"https://arxiv.org/abs/2406.19314 https://github.com/LiveBench/LiveBench https://livebench.ai/"},{"slug":"typesafe-jev","name":"Jev (TypeSafe AI)","creator":"TypeSafe AI","category":"Rubric libraries & frameworks","source_url":"https://docs.typesafe.ai/","page_url":"https://evalsreview.com/eval/typesafe-jev","tests":"Not a benchmark: a tool for AI-judge / LLM-as-judge style scoring in your own evals.","summary":"A commercial model API built for structured judgments rather than text: you send a state plus typed questions (pick a choice, score against a rubric, true/false likelihood) and get back typed answers with probabilities and confidence. Version 1.14.0 (announced 14 Oct 2026) adds experimental image inputs, a bounding-box answer type and longer context behind opt-in flags. Made by a vendor; paid API.","license":"","last_updated":"Jev 1.14.0 announced 2026-10-14","added":"2026-10-15","sources":"https://docs.typesafe.ai/ https://docs.typesafe.ai/introduction/quickstart https://docs.typesafe.ai/api"},{"slug":"mmmu","name":"MMMU","creator":"MMMU team (OSU, Waterloo & collaborators)","category":"Image & vision","source_url":"https://mmmu-benchmark.github.io/","page_url":"https://evalsreview.com/eval/mmmu","tests":"Expert-level multimodal understanding and reasoning.","summary":"About 11.5K college-level questions across six disciplines and 30 subjects, each pairing text with images such as charts, diagrams, maps and chemical structures.","license":"Apache-2.0","last_updated":"Repo last commit 2026-07-28","added":"2026-10-15","sources":"https://arxiv.org/abs/2311.16502 https://arxiv.org/abs/2409.02813 https://github.com/MMMU-Benchmark/MMMU https://mmmu-benchmark.github.io/"},{"slug":"mathvista","name":"MathVista","creator":"UCLA, University of Washington & Microsoft Research","category":"Image & vision","source_url":"https://mathvista.github.io/","page_url":"https://evalsreview.com/eval/mathvista","tests":"Mathematical reasoning over figures, plots and images.","summary":"6,141 math problems set in visual contexts, drawn from 28 existing multimodal datasets plus three new ones (IQTest, FunctionQA, PaperQA).","license":"CC-BY-SA-4.0","last_updated":"Repo last commit 2026-07-27","added":"2026-10-15","sources":"https://arxiv.org/abs/2310.02255 https://github.com/lupantech/MathVista https://mathvista.github.io/"},{"slug":"mmbench","name":"MMBench","creator":"OpenCompass (Shanghai AI Lab & collaborators)","category":"Image & vision","source_url":"https://github.com/open-compass/MMBench","page_url":"https://evalsreview.com/eval/mmbench","tests":"Broad perception and reasoning skills of vision-language models.","summary":"A bilingual (English and Chinese) multiple-choice benchmark that breaks vision-language ability into fine-grained skills, and re-asks each question with the answer options shuffled, counting it correct only if the model is right every time.","license":"Apache-2.0","last_updated":"Repo last commit 2025-05-22","added":"2026-10-15","sources":"https://arxiv.org/abs/2307.06281 https://github.com/open-compass/MMBench https://github.com/open-compass/VLMEvalKit"},{"slug":"chartqa","name":"ChartQA","creator":"Masry et al. (academic)","category":"Image & vision","source_url":"https://github.com/vis-nlp/ChartQA","page_url":"https://evalsreview.com/eval/chartqa","tests":"Question answering over charts.","summary":"9.6K human-written and 23.1K generated questions about real charts that need visual reading plus logical and arithmetic reasoning.","license":"GPL-3.0","last_updated":"Repo last commit 2025-04-18","added":"2026-10-15","sources":"https://arxiv.org/abs/2203.10244 https://github.com/vis-nlp/ChartQA"},{"slug":"mmlu","name":"MMLU","creator":"Hendrycks et al. (UC Berkeley & collaborators)","category":"Reasoning & knowledge","source_url":"https://github.com/hendrycks/test","page_url":"https://evalsreview.com/eval/mmlu","tests":"World knowledge and problem solving across many school and professional subjects.","summary":"Multiple-choice questions across 57 subjects, from elementary mathematics and US history to computer science and law, testing broad academic and professional knowledge.","license":"MIT","last_updated":"Repo last commit 2023-04-24","added":"2026-10-15","sources":"https://arxiv.org/abs/2009.03300 https://github.com/hendrycks/test https://huggingface.co/datasets/cais/mmlu https://arxiv.org/abs/2406.01574 https://github.com/TIGER-AI-Lab/MMLU-Pro"},{"slug":"lmarena","name":"LMArena (Chatbot Arena)","creator":"LMArena (started by LMSYS)","category":"Writing & instruction following","source_url":"https://lmarena.ai/","page_url":"https://evalsreview.com/eval/lmarena","tests":"Which models people prefer on their own real prompts, by pairwise human votes.","summary":"A public site where people chat with two anonymous models side by side and vote for the better answer; the votes feed public leaderboards for text, coding, vision, image and video models.","license":"","last_updated":"","added":"2026-10-15","sources":"https://arxiv.org/abs/2403.04132 https://lmarena.ai/how-it-works https://github.com/lm-sys/FastChat"},{"slug":"helm","name":"HELM","creator":"Stanford CRFM","category":"Rubric libraries & frameworks","source_url":"https://crfm.stanford.edu/helm/","page_url":"https://evalsreview.com/eval/helm","tests":"Standardised, multi-metric comparison of models across shared scenarios.","summary":"Stanford's open framework and set of leaderboards for evaluating language models on many scenarios with several metrics at once, not just accuracy. Domain leaderboards include medicine (MedHELM), safety and vision-language models.","license":"Apache-2.0","last_updated":"Repo last commit 2026-06-05","added":"2026-10-15","sources":"https://arxiv.org/abs/2211.09110 https://github.com/stanford-crfm/helm https://crfm.stanford.edu/helm/"},{"slug":"arc-agi-2","name":"ARC-AGI-2","creator":"ARC Prize","category":"Reasoning & knowledge","source_url":"https://github.com/arcprize/ARC-AGI-2","page_url":"https://evalsreview.com/eval/arc-agi-2","tests":"Abstract reasoning and adapting to new problems from very few examples.","summary":"Grid puzzles where the solver sees a few input-output examples, infers the hidden rule and must draw the exact output grid, with two tries per test input. The repo has 1,000 public training tasks and 120 public evaluation tasks; people in the creators' test sample averaged 66% on the evaluation set, and two private sets are kept back to limit leakage.","license":"Apache-2.0","last_updated":"Repo last commit 2025-05-15","added":"2026-10-16","sources":""},{"slug":"mmlu-pro","name":"MMLU-Pro","creator":"TIGER-AI-Lab","category":"Reasoning & knowledge","source_url":"https://github.com/TIGER-AI-Lab/MMLU-Pro","page_url":"https://evalsreview.com/eval/mmlu-pro","tests":"Expert-level knowledge plus multi-step reasoning across subjects such as law, physics, health and economics.","summary":"A harder follow-up to MMLU with over 12,000 exam and textbook questions in 14 subject areas and ten answer options instead of four. The authors report scores 16% to 33% lower than on MMLU and less sensitivity to prompt wording.","license":"Apache-2.0","last_updated":"Repo last commit 2026-03-18","added":"2026-10-16","sources":""},{"slug":"ruler","name":"RULER","creator":"NVIDIA","category":"RAG & retrieval","source_url":"https://github.com/NVIDIA/RULER","page_url":"https://evalsreview.com/eval/ruler","tests":"How much of a long context window a model can really use.","summary":"Generates synthetic long-context tests at whatever length you choose, with 13 tasks in 4 categories that go beyond finding a single needle in a haystack. Its results compare each model's advertised context length with the length at which it still performs well.","license":"Apache-2.0","last_updated":"Repo last commit 2026-07-22","added":"2026-10-16","sources":""},{"slug":"arena-hard-auto","name":"Arena-Hard-Auto","creator":"LMArena","category":"Writing & instruction following","source_url":"https://github.com/lmarena/arena-hard-auto","page_url":"https://evalsreview.com/eval/arena-hard-auto","tests":"Open-ended answers to hard user prompts, as a preview of how a model might fare on LMArena.","summary":"An automatic benchmark built from hard prompts taken from Chatbot Arena, scored by LLM judges as a cheaper stand-in for human votes. Version 2.0 has 500 challenging real-world queries plus 250 creative writing prompts.","license":"Apache-2.0","last_updated":"Repo last commit 2025-06-21","added":"2026-10-16","sources":""},{"slug":"agentdojo","name":"AgentDojo","creator":"ETH Zurich (SPY Lab) & Invariant Labs","category":"Safety & factuality","source_url":"https://github.com/ethz-spylab/agentdojo","page_url":"https://evalsreview.com/eval/agentdojo","tests":"Agent task success and robustness to prompt injection attacks.","summary":"An extensible environment for testing whether tool-using agents can be hijacked by prompt injections hidden in the data their tools return. The paper's release has 97 realistic tasks, such as handling email, e-banking and travel bookings, and 629 security test cases, with pluggable attacks and defences.","license":"MIT","last_updated":"Repo last commit 2026-06-02","added":"2026-10-16","sources":""},{"slug":"swe-bench-pro","name":"SWE-Bench Pro","creator":"Scale AI","category":"Coding","source_url":"https://github.com/scaleapi/SWE-bench_Pro-os","page_url":"https://evalsreview.com/eval/swe-bench-pro","tests":"Coding agents implementing larger, well-specified changes in real codebases.","summary":"Long, multi-file software tasks from 41 maintained repositories (Python, JavaScript, TypeScript, Go). An agent must produce a patch that passes human-reviewed tests. Public set of 731 tasks (v1) or 642 re-validated tasks (V2); commercial and held-out sets stay private.","license":"MIT","last_updated":"Repo last commit 2026-09-22","added":"2026-10-16","sources":"https://arxiv.org/abs/2509.16941 https://github.com/scaleapi/SWE-bench_Pro-os https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro https://scale.com/leaderboard/swe_bench_pro_public https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/ https://openai.com/index/separating-signal-from-noise-coding-evaluations/ https://arxiv.org/abs/2609.08149 https://labs.scale.com/blog/swe-bench-pro-v2"},{"slug":"deepeval","name":"DeepEval","creator":"Confident AI","category":"Rubric libraries & frameworks","source_url":"https://github.com/confident-ai/deepeval","page_url":"https://evalsreview.com/eval/deepeval","tests":"Your own LLM app, RAG pipeline or agent, as pass/fail tests in CI.","summary":"An open-source Python framework for unit-testing LLM apps like pytest, with LLM-judged metrics for RAG (faithfulness, answer relevancy, contextual precision and recall), agents (task completion, tool correctness) and custom criteria (G-Eval).","license":"Apache-2.0","last_updated":"Repo last commit 2026-10-15","added":"2026-10-16","sources":"https://github.com/confident-ai/deepeval https://deepeval.com/docs/getting-started https://deepeval.com/docs/metrics-faithfulness https://deepeval.com/docs/metrics-ragas"},{"slug":"arc-agi-3","name":"ARC-AGI-3","creator":"ARC Prize Foundation","category":"Reasoning & knowledge","source_url":"https://arcprize.org/arc-agi/3","page_url":"https://evalsreview.com/eval/arc-agi-3","tests":"Exploration, goal inference, world-modelling and planning in new environments.","summary":"Interactive, turn-based game environments with no instructions: an agent must explore, work out the goal and the rules, and finish each level in as few actions as a first-time human. 25 public demo environments, 55 semi-private and 55 fully private; scored by action efficiency against human baselines (RHAE).","license":"","last_updated":"Launched Mar 2026","added":"2026-10-16","sources":"https://arxiv.org/abs/2603.24621 https://arcprize.org/arc-agi/3 https://docs.arcprize.org/methodology https://github.com/arcprize/ARC-AGI"},{"slug":"promptfoo","name":"Promptfoo","creator":"Promptfoo (now part of OpenAI)","category":"Rubric libraries & frameworks","source_url":"https://github.com/promptfoo/promptfoo","page_url":"https://evalsreview.com/eval/promptfoo","tests":"Your own prompts, models and LLM apps, with deterministic and LLM-graded assertions.","summary":"An open-source (MIT) command-line tool for testing prompts, agents and RAG apps from declarative YAML configs, comparing models side by side, running checks in CI, and red-teaming LLM apps for vulnerabilities.","license":"MIT","last_updated":"Repo last commit 2026-10-16","added":"2026-10-16","sources":"https://github.com/promptfoo/promptfoo https://www.promptfoo.dev/docs/configuration/expected-outputs/ https://www.promptfoo.dev/docs/red-team/"},{"slug":"agent-as-a-judge","name":"Agent-as-a-Judge (DevAI)","creator":"Meta AI & KAUST","category":"Agents & tool use","source_url":"https://arxiv.org/abs/2410.10934","page_url":"https://evalsreview.com/eval/agent-as-a-judge","tests":"Coding agents on full AI-development tasks, judged step by step instead of pass/fail.","summary":"A method that uses an agent, which can read files, locate code and inspect trajectories, to judge other agents requirement by requirement, plus DevAI: 55 AI-development tasks with 365 hierarchical requirements. In the paper it agreed with human consensus about 90% of the time versus about 70% for LLM-as-a-judge.","license":"MIT","last_updated":"Repo last commit 2026-03-28","added":"2026-10-16","sources":"https://arxiv.org/abs/2410.10934 https://github.com/metauto-ai/agent-as-a-judge https://huggingface.co/devai-benchmark"}]}