SO-Arena
by National University of Singapore (Lichen Li, Ziping Dong, Sicheng Feng, Qi Li, Xinchao Wang)
What is SO-Arena? SO-Arena is a reasoning & knowledge eval made by National University of Singapore (Lichen Li, Ziping Dong, Sicheng Feng, Qi Li, Xinchao Wang). A benchmark for fast 'System One' decision models, which answer without long chains of generated reasoning. It has 2,919 tasks over 11 domains and five capability families: 2,300 static decisions, 282 tool-use episodes in native simulators and 337 text-based web tasks, with Easy, Medium and Hard tiers in graded domains. Labels are checked by hand against what the model can actually see. Reliability is scored separately from capability, by reordering answer options and by including items with no valid option, and the authors find the top-scoring System One model ranks only seventh on reliability across the 21 models tested. Code is on GitHub and the data on Hugging Face.
Editor note
A benchmark for fast 'System One' decision models, which answer without long chains of generated reasoning. It has 2,919 tasks over 11 domains and five capability families: 2,300 static decisions, 282 tool-use episodes in native simulators and 337 text-based web tasks, with Easy, Medium and Hard tiers in graded domains. Labels are checked by hand against what the model can actually see. Reliability is scored separately from capability, by reordering answer options and by including items with no valid option, and the authors find the top-scoring System One model ranks only seventh on reliability across the 21 models tested. Code is on GitHub and the data on Hugging Face.
- What it tests
- Breadth of capability and decision reliability (stable under option changes, recognises when no answer fits) for fast decision models.
- Original source
- lichenlillc.github.io/SO-Arena/
- Last updated
- arXiv 2611.15777 (HF papers date 2026-11-11); repo last push 2026-11-13
- License
- MIT (code repository)
Related reasoning & knowledge evals
- GPQA — Expert-level science reasoning.
- Humanity's Last Exam — Frontier academic knowledge and reasoning.
- LiveBench — Hard, contamination-limited tasks with verifiable ground-truth answers (no LLM judge) in v1; LiveBench 2.0 (14 Oct 2026) adds long-horizon real-work tasks.
- MMLU — World knowledge and problem solving across many school and professional subjects.
Discussion
No one has discussed SO-Arena yet. Have you run it? Share what it caught, what it missed or how you set it up.