SciCode
by SciCode team
What is SciCode? SciCode is a benchmark of research-level scientific coding: 80 main problems from real research, broken into 338 subproblems that each need knowledge recall, reasoning and code. It is one of the evaluations in the Artificial Analysis Intelligence Index.
Editor note
80 research-level scientific coding problems split into 338 subproblems across the natural sciences and maths. Part of the Artificial Analysis Intelligence Index.
- What it tests
- Writing code for real scientific research problems.
- Original source
- github.com/scicode-bench/SciCode
- Last updated
- Repo last commit 2026-10-12
- License
- Apache-2.0
When to use SciCode
- You care about code for physics, chemistry, biology or maths research rather than software engineering.
- You want partial credit through subproblems.
Limitations
- 80 main problems is a small set, so scores move a lot between runs.
- Optional background descriptions change difficulty; report whether you used them.
How to run it
- Clone the repository and follow its evaluation instructions; it also ships an Inspect-based evaluation.
Notable results
- README: 338 subproblems decomposed from 80 challenging main problems. [Official repository]
Sources
Reviews
No reviews yet.
Reviews from practitioners who sign in with GitHub open soon.
Related coding evals
- SWE-bench — Fixing real bugs in real repositories, judged by unit tests.
- HumanEval — Writing a short correct function from a docstring.
- LiveCodeBench — Code generation, self-repair and test prediction on fresh problems.
- Aider leaderboards (Polyglot) — Practical multi-language code editing.