MuSR
by University of Texas at Austin (Sprague, Ye, Bostrom, Chaudhuri, Durrett)
What is MuSR? MuSR is a reasoning & knowledge eval made by University of Texas at Austin (Sprague, Ye, Bostrom, Chaudhuri, Durrett). Multistep soft reasoning over long natural-language stories, such as murder mysteries, that were generated with an LLM from logic trees so each answer follows from the story plus commonsense. It was built to test chain-of-thought reasoning and appeared at ICLR 2024 as a spotlight paper.
Use it when: You want to test multi-step commonsense reasoning over long stories, such as solving a murder mystery, rather than short logic puzzles.
Key fact: Its stories are built with a neurosymbolic generation method that produces instances such as murder mysteries of roughly 1,000 words that challenged GPT-4, and the method can scale as models improve.
Our note: Useful for checking whether chain-of-thought prompting actually helps your model. Since the stories are generated, check a few items by hand to see if they match the kind of reasoning you need.
Editor note
Multistep soft reasoning over long natural-language stories, such as murder mysteries, that were generated with an LLM from logic trees so each answer follows from the story plus commonsense. It was built to test chain-of-thought reasoning and appeared at ICLR 2024 as a spotlight paper.
- What it tests
- Multi-step commonsense reasoning over long narratives.
- Original source
- zayne-sprague.github.io/MuSR/
- Last updated
- Repo last push 2024-08-10
- License
- MIT
Related reasoning & knowledge evals
- GPQA — Expert-level science reasoning.
- Humanity's Last Exam — Frontier academic knowledge and reasoning.
- LiveBench — Hard, contamination-limited tasks with verifiable ground-truth answers (no LLM judge) in v1; LiveBench 2.0 (14 Oct 2026) adds long-horizon real-work tasks.
- MMLU — World knowledge and problem solving across many school and professional subjects.
Discussion
No one has discussed MuSR yet. Have you run it? Share what it caught, what it missed or how you set it up.