Reasoning & knowledge

SO-Arena

by National University of Singapore (Lichen Li, Ziping Dong, Sicheng Feng, Qi Li, Xinchao Wang)

What is SO-Arena? SO-Arena is a reasoning & knowledge eval made by National University of Singapore (Lichen Li, Ziping Dong, Sicheng Feng, Qi Li, Xinchao Wang). A benchmark for fast 'System One' decision models, which answer without long chains of generated reasoning. It has 2,919 tasks over 11 domains and five capability families: 2,300 static decisions, 282 tool-use episodes in native simulators and 337 text-based web tasks, with Easy, Medium and Hard tiers in graded domains. Labels are checked by hand against what the model can actually see. Reliability is scored separately from capability, by reordering answer options and by including items with no valid option, and the authors find the top-scoring System One model ranks only seventh on reliability across the 21 models tested. Code is on GitHub and the data on Hugging Face.

Page last updated 2026-10-18 · Evals Review

Editor note

A benchmark for fast 'System One' decision models, which answer without long chains of generated reasoning. It has 2,919 tasks over 11 domains and five capability families: 2,300 static decisions, 282 tool-use episodes in native simulators and 337 text-based web tasks, with Easy, Medium and Hard tiers in graded domains. Labels are checked by hand against what the model can actually see. Reliability is scored separately from capability, by reordering answer options and by including items with no valid option, and the authors find the top-scoring System One model ranks only seventh on reliability across the 21 models tested. Code is on GitHub and the data on Hugging Face.

What it tests
Breadth of capability and decision reliability (stable under option changes, recognises when no answer fits) for fast decision models.
Original source
lichenlillc.github.io/SO-Arena/
Last updated
arXiv 2611.15777 (HF papers date 2026-11-11); repo last push 2026-11-13
License
MIT (code repository)

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Sign in with GitHub to review

Related reasoning & knowledge evals

  • GPQA — Expert-level science reasoning.
  • Humanity's Last Exam — Frontier academic knowledge and reasoning.
  • LiveBench — Hard, contamination-limited tasks with verifiable ground-truth answers (no LLM judge) in v1; LiveBench 2.0 (14 Oct 2026) adds long-horizon real-work tasks.
  • MMLU — World knowledge and problem solving across many school and professional subjects.

All Reasoning & knowledge evals · Comparison table

Discussion

No one has discussed SO-Arena yet. Have you run it? Share what it caught, what it missed or how you set it up.

Sign in with GitHub to comment

Be specific and kind. No promotion or affiliate links. Community rules · All discussions