DecompGym
by Sikai Cheng, Tinghan Ye, Arnaud Deza et al.
What is DecompGym? DecompGym is a coding eval made by Sikai Cheng, Tinghan Ye, Arnaud Deza et al.. A benchmark for coding agents that must implement two classic large-scale optimization methods, Benders decomposition and column generation, across 81 instances from nine problem families. A solver only passes if it runs within 60 seconds, gives a feasible answer within 1% of the known optimum, and survives a method audit in which an LLM judge reads the code to confirm it really uses the required method. The paper tests 12 GPT, Claude and Gemini models inside coding-agent tools and finds that checking outputs alone would have accepted 93 of the 108 attempts that failed the method audit. Retrying with a diagnosis of the failure first (reflection) succeeded far more often than simply regenerating.
Editor note
A benchmark for coding agents that must implement two classic large-scale optimization methods, Benders decomposition and column generation, across 81 instances from nine problem families. A solver only passes if it runs within 60 seconds, gives a feasible answer within 1% of the known optimum, and survives a method audit in which an LLM judge reads the code to confirm it really uses the required method. The paper tests 12 GPT, Claude and Gemini models inside coding-agent tools and finds that checking outputs alone would have accepted 93 of the 108 attempts that failed the method audit. Retrying with a diagnosis of the failure first (reflection) succeeded far more often than simply regenerating.
- What it tests
- Whether coding agents implement a prescribed optimization algorithm correctly, judged on runtime, answer quality and how the code reaches its answer.
- Original source
- github.com/SIKAI-C/DecompGym
- Last updated
- Paper arXiv 2610.38175 (28 Oct 2026); repo last push 2026-10-30
- License
- MIT
Related coding evals
- SWE-bench — Fixing real bugs in real repositories, judged by unit tests.
- HumanEval — Writing a short correct function from a docstring.
- LiveCodeBench — Code generation, self-repair and test prediction on fresh problems.
- Aider leaderboards (Polyglot) — Practical multi-language code editing.
Discussion
No one has discussed DecompGym yet. Have you run it? Share what it caught, what it missed or how you set it up.