WildBench
by Allen Institute for AI (Ai2)
What is WildBench? WildBench is Ai2's benchmark of 1,024 hard tasks taken from real user conversations (WildChat), scored by an LLM judge that follows a 5-10 question checklist written for each task.
Editor note
1,024 challenging tasks picked from real user conversations in the WildChat logs, judged automatically by an LLM using a checklist for each task. It reports WB-Score for single responses and WB-Reward for pairwise comparisons against three baseline models, with an optional length penalty to reduce bias toward longer answers.
- What it tests
- Chat assistants on hard, real-world user requests across 12 task types.
- Original source
- github.com/allenai/WildBench
- Last updated
- Repo last push 2024-11-03
- License
- Apache-2.0 (code); data AI2 ImpACT license
When to use WildBench
- You want a chat-assistant score based on messy, real user requests rather than exam questions.
- You want both a single-model score (WB-Score) and head-to-head comparisons (WB-Reward) from the same task set.
- You worry that LLM judges favour long answers: WildBench's length penalty turns a slight win into a tie when the winner is longer by more than K characters.
Limitations
- Scoring uses GPT-4-turbo as the judge through the OpenAI Batch API, so you need an OpenAI key and pay for judge calls; scores depend on that judge.
- The repo's last push was 3 Nov 2024 and the install pins vLLM 0.5.1, so newer models may need extra setup.
- The data uses the AI2 ImpACT license, not a standard open licence; check it before commercial use.
How to run it
- Create a Python 3.10 environment and install the pinned requirements (vLLM 0.5.1).
- Run inference with scripts/_common_vllm.sh (Hugging Face model id, a short name, number of GPUs); use --engine hf or the API engines for models vLLM doesn't support.
- Submit the WB-Score judging batch to OpenAI, check its status, then print the table with leaderboard/show_eval.sh.
- To get on the public leaderboard, open an issue and the maintainers run inference and scoring for you.
bash scripts/_common_vllm.sh <hf_model_id> <model_name> <num_gpus>
bash evaluation/run_score_eval_batch.sh <model_name>
bash leaderboard/show_eval.sh score_onlyNotable results
- README: WB-Score is the GPT-4-turbo 1-10 rating averaged over 1,024 examples and rescaled as (score - 5) x 2, so 0 means borderline acceptable. [Official repository]
Sources
Related writing & instruction following evals
- EQ-Bench Creative Writing — Creative writing quality judged against a rubric.
- AlpacaEval — General helpfulness of chat responses.
- MT-Bench (LLM-as-a-judge paper) — Multi-turn conversation quality; how reliable LLM judges are.
- IFEval — Following explicit, verifiable instructions.
Discussion
No one has discussed WildBench yet. Have you run it? Share what it caught, what it missed or how you set it up.