DistDebug-Bench

Name: DistDebug-Bench
Creator: Yuxuan Jiang, Chenglin Liu, John Rose, Mason Miller, Ruthesh Thavamani, Juntao Wu, Yinwei Huang, Jiahao Xu, Peng Huang (OrderLab / University of Michigan)
License: https://spdx.org/licenses/MIT.html

by Yuxuan Jiang, Chenglin Liu, John Rose, Mason Miller, Ruthesh Thavamani, Juntao Wu, Yinwei Huang, Jiahao Xu, Peng Huang (OrderLab / University of Michigan)

What is DistDebug-Bench? DistDebug-Bench is a coding eval made by Yuxuan Jiang, Chenglin Liu, John Rose, Mason Miller, Ruthesh Thavamani, Juntao Wu, Yinwei Huang, Jiahao Xu, Peng Huang (OrderLab / University of Michigan). A distributed-systems diagnosis benchmark: 120 real bugs from 11 open-source systems across five languages, each with an issue report, attached evidence and a pre-fix repository snapshot. Agents must name the root-cause source blocks (file, function, line range); scoring is against 218 expert-annotated blocks, not whether a patch passes tests. The authors' NeurIPS 2026 paper and leaderboard report that the strongest Mini-SWE-Agent run reaches high file coverage but at most about 68% block coverage, with run-to-run flakiness on opaque reports. Code, evaluator, dataset and traces are public.

Page last updated 2026-10-18 · Evals Review

Editor note

A distributed-systems diagnosis benchmark: 120 real bugs from 11 open-source systems across five languages, each with an issue report, attached evidence and a pre-fix repository snapshot. Agents must name the root-cause source blocks (file, function, line range); scoring is against 218 expert-annotated blocks, not whether a patch passes tests. The authors' NeurIPS 2026 paper and leaderboard report that the strongest Mini-SWE-Agent run reaches high file coverage but at most about 68% block coverage, with run-to-run flakiness on opaque reports. Code, evaluator, dataset and traces are public.

What it tests: Whether coding agents can locate the code-level root cause of real distributed-system failures from issue evidence alone.
Original source: github.com/OrderLab/distdebug-bench
Last updated: Repo last push 2026-11-05
License: MIT

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Related coding evals

SWE-bench — Fixing real bugs in real repositories, judged by unit tests.
HumanEval — Writing a short correct function from a docstring.
LiveCodeBench — Code generation, self-repair and test prediction on fresh problems.
Aider leaderboards (Polyglot) — Practical multi-language code editing.

All Coding evals · Comparison table

Discussion

No one has discussed DistDebug-Bench yet. Have you run it? Share what it caught, what it missed or how you set it up.

Be specific and kind. No promotion or affiliate links. Community rules · All discussions