State of LLM eval tools and benchmarks, October 2026

Key numbers

• 122 evals and benchmarks in 11 categories.
• 79 have a checked licence; 73 of them (92%) are MIT or Apache-2.0.
• 95 list a dated last update (repo commit, release or publication).
• 12 eval platforms and frameworks tracked (9 platforms, 3 frameworks); 6 publish their main code on GitHub.
• 15 open-source eval repositories with a GitHub star count taken on 27 Oct 2026.
• 4 tools shut down since 2025, plus 1 scheduled closure.

Evals by category

Shares are rounded. Compare benchmarks head to head on comparison pages or see best-of lists. For reasoning, ARC-AGI-2 vs ARC-AGI-3 sets static grid puzzles against interactive games.

Licence mix of evals

Licences come from each eval's repository (GitHub API) or source page. Entries like "MIT (code); data CC BY-NC-SA 4.0" are grouped by the first licence named; the CSV keeps the full text.

Open-source eval tools by GitHub stars

From api.github.com on 27 Oct 2026. Stars measure attention, not quality. Tool pages: LLM evaluation tools compared · observability tools · Langfuse vs LangSmith · Braintrust vs Langfuse · DeepEval and Confident AI alternatives · Braintrust pricing and plans.

Tools shut down in 2025-26

Notices last read 23 Oct 2026. Looking for a replacement? See Langfuse alternatives or the pricing calculator (prices checked 29 Oct 2026). Helicone is not listed because it has not announced a shut-down, but it has been in maintenance mode since Mintlify acquired it on 3 Mar 2026 (Mintlify): see Helicone alternatives and a migration guide.

How we compiled this

• Every number is computed when the site is built, from the same data files behind our eval and tool pages. Nothing is estimated.
• Evals: each entry's creator, category, licence and last update come from its paper, repository or official page (links first checked 15 Oct 2026). "Not checked" means we haven't verified the licence.
• Tools: features and prices were read on each vendor's own pages on 27 Oct 2026. GitHub stars and releases came from the GitHub API on 27 Oct 2026.
• Shut-downs: listed only when the vendor's own site or docs say so.
• Nothing here is ranked, and no tool paid to be included. See how we review and our disclosure.

Compiled by Evals Review, 25 Oct 2026.

Cite this page

Evals Review. "State of LLM eval tools and benchmarks (October 2026)." evalsreview.com, 25 Oct 2026. https://evalsreview.com/state-of-llm-evals

Licence: CC BY 4.0. Credit Evals Review with a link. Files: evals.csv · tools.csv · all downloads. Also on GitHub: evalsreview/llm-evals-dataset (CC BY 4.0).

FAQ