ARC-AGI-2 vs ARC-AGI-3
ARC-AGI-2 or ARC-AGI-3? They test different things. ARC-AGI-2 (March 2025) is static grid puzzles: infer a rule from a few examples and draw the output. ARC-AGI-3 (March 2026) is interactive games with no instructions, scored on how few actions the agent needs compared with humans. Frontier models scored under 1% on ARC-AGI-3 at launch.
| ARC-AGI-2 | ARC-AGI-3 | |
|---|---|---|
| Made by | ARC Prize | ARC Prize Foundation |
| Format | Static input/output grid puzzles | Interactive turn-based games on a 64x64 grid |
| What the solver gets | A few solved examples | No instructions; must explore to find the goal |
| Scoring | Exact output grid, two tries per test input | RHAE: (human actions ÷ AI actions)² per level, capped at 1.15 |
| Human time | About 300 seconds per task on average | Most environments take a few minutes; 20-minute soft limit in testing |
| Data | 1,000 public training + 120 public evaluation tasks; private sets | 25 public demo, 55 semi-private, 55 fully private environments |
| Saturation | Its authors believe frontier models have ARC-AGI-2 data well represented in training | Under 1% for frontier models at launch (Mar 2026) |
| ARC Prize 2026 | Final year of the ARC-AGI-2 track | Main track going forward |
| License | Apache-2.0 | — |
Verdict: when to use which
Use ARC-AGI-2 if you want a static, cheap-to-run abstraction test with a long history of published results.
Use ARC-AGI-3 if you want to test agents that must learn by acting, and you need a benchmark far from saturation.
Dated numbers
- ARC Prize 2025 (ARC-AGI-2): the winning Kaggle team, NVARC, scored 24% with a 4B-parameter model; the 85% grand prize went unclaimed. [ARC-AGI-3 technical paper (arXiv)]
- ARC-AGI-3 launch (Mar 2026, semi-private set): Opus 4.6 (Max) 0.50%, Gemini 3.1 Pro Preview 0.40%, GPT 5.4 (High) 0.20%. [ARC-AGI-3 technical paper (arXiv)]
FAQ
Is ARC-AGI-3 harder than ARC-AGI-2?
For current AI, yes: frontier models scored under 1% at launch. Both are designed so humans can solve every task.
Can ARC-AGI-2 and ARC-AGI-3 scores be compared?
No. ARC-AGI-2 scores exact answers; ARC-AGI-3 scores action efficiency against humans.
Sources
Other comparisons: SWE-bench vs SWE-bench-Live · HumanEval vs BigCodeBench · HumanEval vs LiveCodeBench · MMLU vs GPQA · AlpacaEval vs MT-Bench (LLM-as-a-judge paper) · MT-Bench (LLM-as-a-judge paper) vs LMArena (Chatbot Arena) · BEIR vs MTEB Leaderboard · WebArena vs OSWorld · HarmBench vs StrongREJECT · GAIA vs BrowseComp · SWE-Bench Pro vs SWE-bench Verified · Terminal-Bench vs SWE-bench · MMLU vs MMLU-Pro · GPQA vs Humanity's Last Exam · Ragas vs DeepEval · Terminal-Bench 2.1 vs Terminal-Bench 4.0 · LiveCodeBench vs SWE-bench · LLM-as-a-Judge vs Agent-as-a-Judge · DeepEval vs Promptfoo