SWE Atlas
by Scale AI
What is SWE Atlas? SWE Atlas is a coding eval made by Scale AI. A suite from Scale AI that looks at coding agents more like junior engineers, inside real open-source repositories. Its first part, Codebase QnA, has 124 questions across 11 repositories in Go, Python, C and TypeScript that need the agent to run and trace the software, with an LLM judge checking each expert rubric item; a task only counts if every item passes. Test Writing and Refactoring tracks are in the same repo with their own leaderboards.
Editor note
A suite from Scale AI that looks at coding agents more like junior engineers, inside real open-source repositories. Its first part, Codebase QnA, has 124 questions across 11 repositories in Go, Python, C and TypeScript that need the agent to run and trace the software, with an LLM judge checking each expert rubric item; a task only counts if every item passes. Test Writing and Refactoring tracks are in the same repo with their own leaderboards.
- What it tests
- Deep codebase understanding, test writing and behaviour-preserving refactoring.
- Original source
- github.com/scaleapi/SWE-Atlas
- Last updated
- Repo last push 2026-10-13
- License
- Apache-2.0 (per the GitHub repo)
Related coding evals
- SWE-bench — Fixing real bugs in real repositories, judged by unit tests.
- HumanEval — Writing a short correct function from a docstring.
- LiveCodeBench — Code generation, self-repair and test prediction on fresh problems.
- Aider leaderboards (Polyglot) — Practical multi-language code editing.
Discussion
No one has discussed SWE Atlas yet. Have you run it? Share what it caught, what it missed or how you set it up.