LMArena (Chatbot Arena)
by LMArena (started by LMSYS)
What is LMArena (Chatbot Arena)? LMArena (Chatbot Arena) is a writing & instruction following eval made by LMArena (started by LMSYS). A public site where people chat with two anonymous models side by side and vote for the better answer; the votes feed public leaderboards for text, coding, vision, image and video models.
Editor note
A public site where people chat with two anonymous models side by side and vote for the better answer; the votes feed public leaderboards for text, coding, vision, image and video models.
- What it tests
- Which models people prefer on their own real prompts, by pairwise human votes.
- Original source
- lmarena.ai/
- Last updated
- Not checked yet
- License
- Not checked yet — see source
When to use LMArena (Chatbot Arena)
- You want to know which models people prefer on their own everyday prompts, rather than on a fixed test set.
- You are choosing between models for chat, coding (WebDev), vision, image or video work. The site keeps separate leaderboards for these.
- You want real preference data: the project has released open datasets of human votes on Hugging Face.
Limitations
- Votes reflect preference, not correctness. A confident, well-formatted answer can win even if it is wrong; the team has published style-control analysis on this.
- The prompts come from site visitors, so they may not look like your users' tasks.
- Rankings come from votes on the public site; it is not a test suite you run on your own model.
How to run it
- Browse the leaderboards on lmarena.ai, or vote yourself in battle mode, where two anonymous models answer and their names are revealed after you vote.
- For offline analysis, use the open preference datasets the project lists on Hugging Face.
Notable results
- The 2024 paper describes the pairwise crowdsourced method. At the time it had over 240K votes, and the authors found the crowd votes agreed well with expert raters. [Paper (arXiv)]
- The FastChat repository, which powered the original arena, reports over 1.5M human votes collected from side-by-side battles. [FastChat repository]
Sources
Compare LMArena (Chatbot Arena)
Reviews
No reviews yet.
Reviews from practitioners who sign in with GitHub open soon.
Related writing & instruction following evals
- EQ-Bench Creative Writing — Creative writing quality judged against a rubric.
- AlpacaEval — General helpfulness of chat responses.
- MT-Bench (LLM-as-a-judge paper) — Multi-turn conversation quality; how reliable LLM judges are.
- IFEval — Following explicit, verifiable instructions.