CRUXEval
by Meta AI (FAIR) & collaborators
What is CRUXEval? CRUXEval is a coding eval made by Meta AI (FAIR) & collaborators. 800 short Python functions, each with an input-output pair. The model either predicts the output from the input (CRUXEval-O) or finds an input that yields the output (CRUXEval-I), so it measures reading and executing code rather than writing it.
Editor note
800 short Python functions, each with an input-output pair. The model either predicts the output from the input (CRUXEval-O) or finds an input that yields the output (CRUXEval-I), so it measures reading and executing code rather than writing it.
- What it tests
- Code reasoning and mental execution.
- Original source
- crux-eval.github.io
- Last updated
- Repo last push 2024-10-11
- License
- MIT
Related coding evals
- SWE-bench — Fixing real bugs in real repositories, judged by unit tests.
- HumanEval — Writing a short correct function from a docstring.
- LiveCodeBench — Code generation, self-repair and test prediction on fresh problems.
- Aider leaderboards (Polyglot) — Practical multi-language code editing.
Discussion
No one has discussed CRUXEval yet. Have you run it? Share what it caught, what it missed or how you set it up.