Arena-Hard-Auto
by LMArena
What is Arena-Hard-Auto? Arena-Hard-Auto is a writing & instruction following eval made by LMArena. An automatic benchmark built from hard prompts taken from Chatbot Arena, scored by LLM judges as a cheaper stand-in for human votes. Version 2.0 has 500 challenging real-world queries plus 250 creative writing prompts.
Editor note
An automatic benchmark built from hard prompts taken from Chatbot Arena, scored by LLM judges as a cheaper stand-in for human votes. Version 2.0 has 500 challenging real-world queries plus 250 creative writing prompts.
- What it tests
- Open-ended answers to hard user prompts, as a preview of how a model might fare on LMArena.
- Original source
- github.com/lmarena/arena-hard-auto
- Last updated
- Repo last commit 2025-06-21
- License
- Apache-2.0
Reviews
No reviews yet.
Reviews from practitioners who sign in with GitHub open soon.
Related writing & instruction following evals
- EQ-Bench Creative Writing — Creative writing quality judged against a rubric.
- AlpacaEval — General helpfulness of chat responses.
- MT-Bench (LLM-as-a-judge paper) — Multi-turn conversation quality; how reliable LLM judges are.
- IFEval — Following explicit, verifiable instructions.