Reasoning & knowledge

S1CogBench

by Dengzhe Hou and Kazunori D. Yamada

What is S1CogBench? S1CogBench is a reasoning & knowledge eval made by Dengzhe Hou and Kazunori D. Yamada. Puts fast "System One" decision models through 24 established experiments from cognitive science, run in 27 evaluation modes. Each model is checked three ways: how well it predicts the choices people actually made, how it does when it plays the task itself, and how its stated probabilities behave, with controlled manipulations of the tasks to see whether it reacts as people do. The paper evaluates seven open models plus one API model and finds that good scores on some learning tasks sit alongside failures to gather information or adjust to feedback, and that rankings partly depend on how choices are read out. The repository ships the original runners and scorers, a quick smoke test and a guide for adding a model; some modes need the gated Psych-101 test set (arXiv:2611.00911).

Page last updated 2026-10-18 · Evals Review

Editor note

Puts fast "System One" decision models through 24 established experiments from cognitive science, run in 27 evaluation modes. Each model is checked three ways: how well it predicts the choices people actually made, how it does when it plays the task itself, and how its stated probabilities behave, with controlled manipulations of the tasks to see whether it reacts as people do. The paper evaluates seven open models plus one API model and finds that good scores on some learning tasks sit alongside failures to gather information or adjust to feedback, and that rankings partly depend on how choices are read out. The repository ships the original runners and scorers, a quick smoke test and a guide for adding a model; some modes need the gated Psych-101 test set (arXiv:2611.00911).

What it tests
Cognitive-experiment behaviour of fast decision models: human-choice prediction, autonomous play and explicit probability reports.
Original source
github.com/dengzhe-hou/S1CogBench
Last updated
arXiv:2611.00911 (14 Nov 2026); repo last pushed 2026-11-20 (GitHub API, read 22 Nov 2026)
License
MIT (code, GitHub API, read 22 Nov 2026); aggregate results CC-BY-4.0 (HF tag)

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Sign in with GitHub to review

Related reasoning & knowledge evals

  • GPQA — Expert-level science reasoning.
  • Humanity's Last Exam — Frontier academic knowledge and reasoning.
  • LiveBench — Hard, contamination-limited tasks with verifiable ground-truth answers (no LLM judge) in v1; LiveBench 2.0 (14 Oct 2026) adds long-horizon real-work tasks.
  • MMLU — World knowledge and problem solving across many school and professional subjects.

All Reasoning & knowledge evals · Comparison table

Discussion

No one has discussed S1CogBench yet. Have you run it? Share what it caught, what it missed or how you set it up.

Sign in with GitHub to comment

Be specific and kind. No promotion or affiliate links. Community rules · All discussions