ALE-Bench
by Imajuku, Horie, Iwata, Aoki, Takahashi et al. (repo in the SakanaAI GitHub org)
What is ALE-Bench? ALE-Bench is a coding eval made by Imajuku, Horie, Iwata, Aoki, Takahashi et al. (repo in the SakanaAI GitHub org). A benchmark for long-horizon algorithm engineering, built from real AtCoder Heuristic Contest problems such as routing and scheduling that are computationally hard and have no known exact solution. Instead of pass/fail unit tests, each submission gets a contest score, so an agent can keep refining its solution using test runs and visualizations. The authors found frontier models strong on some problems but still behind humans on consistency across problems and on long-horizon work. A public leaderboard, a Hugging Face dataset and a Docker-based toolkit are available.
Editor note
A benchmark for long-horizon algorithm engineering, built from real AtCoder Heuristic Contest problems such as routing and scheduling that are computationally hard and have no known exact solution. Instead of pass/fail unit tests, each submission gets a contest score, so an agent can keep refining its solution using test runs and visualizations. The authors found frontier models strong on some problems but still behind humans on consistency across problems and on long-horizon work. A public leaderboard, a Hugging Face dataset and a Docker-based toolkit are available.
- What it tests
- Whether AI agents can iteratively improve heuristic solutions to hard optimization problems over long time horizons.
- Original source
- github.com/SakanaAI/ALE-Bench
- Last updated
- Paper arXiv 2506.09050 (June 2025); repo last push 2026-10-28
- License
- Apache-2.0
Related coding evals
- SWE-bench — Fixing real bugs in real repositories, judged by unit tests.
- HumanEval — Writing a short correct function from a docstring.
- LiveCodeBench — Code generation, self-repair and test prediction on fresh problems.
- Aider leaderboards (Polyglot) — Practical multi-language code editing.
Discussion
No one has discussed ALE-Bench yet. Have you run it? Share what it caught, what it missed or how you set it up.