AscendGYM
by SLAI-AITP (AI Training Platform Team; Bingwei Zhang, Jing Li et al., 27 authors)
What is AscendGYM? AscendGYM is a coding eval made by SLAI-AITP (AI Training Platform Team; Bingwei Zhang, Jing Li et al., 27 authors). A kernel-optimisation benchmark for Huawei Ascend NPUs. Rather than writing kernels from scratch, the model or agent must speed up existing kernels, and each kernel is tested on data types and tensor shapes taken from real training and inference jobs, so a gain on the benchmark maps to a real workload. The paper (arXiv:2611.17621) describes 48 kernels across three difficulty levels; the public repository currently releases Batch 1 (17 tasks), with later batches announced. It supports two modes, scoring a bare model through a built-in harness or scoring your own agent harness, and proposes a Speedup-Token Score that weighs speedup against token cost. A public leaderboard compares models and harnesses.
Editor note
A kernel-optimisation benchmark for Huawei Ascend NPUs. Rather than writing kernels from scratch, the model or agent must speed up existing kernels, and each kernel is tested on data types and tensor shapes taken from real training and inference jobs, so a gain on the benchmark maps to a real workload. The paper (arXiv:2611.17621) describes 48 kernels across three difficulty levels; the public repository currently releases Batch 1 (17 tasks), with later batches announced. It supports two modes, scoring a bare model through a built-in harness or scoring your own agent harness, and proposes a Speedup-Token Score that weighs speedup against token cost. A public leaderboard compares models and harnesses.
- What it tests
- Whether models and agent harnesses can make real-workload Ascend NPU kernels faster without breaking correctness, and at what token cost.
- Original source
- github.com/SLAI-AITP/AscendGYM
- Last updated
- arXiv:2611.17621 (13 Nov 2026); repo last push 2026-11-17 (GitHub API, read 19 Nov 2026)
- License
- Apache-2.0 (GitHub API, read 19 Nov 2026)
Related coding evals
- SWE-bench — Fixing real bugs in real repositories, judged by unit tests.
- HumanEval — Writing a short correct function from a docstring.
- LiveCodeBench — Code generation, self-repair and test prediction on fresh problems.
- Aider leaderboards (Polyglot) — Practical multi-language code editing.
Discussion
No one has discussed AscendGYM yet. Have you run it? Share what it caught, what it missed or how you set it up.