LLM evaluation tools compared: 12 platforms and the open-source frameworks
Prices checked 2026-10-17 on each vendor's own pricing page. Every cell links to its source; anything we could not confirm says “not verified”.
Short answer: If you want to self-host for free, look at Langfuse, Arize Phoenix, Comet Opik or Helicone. If you want a hosted tool with published prices, the lowest entry plans we verified are Comet Opik (Pro Cloud $19/month), Langfuse (Core $29/month), Maxim AI (Professional $29/seat/month). If you only need tests in CI, an open-source framework (Promptfoo, DeepEval, Ragas) may be enough. Most comparison pages for these searches are written by the vendors themselves; this page lists all twelve alphabetically with the same fields.
Pricing: free tier and entry price
| Tool | Best for | Licence / self-host | Meter | Free tier | Entry paid price |
|---|---|---|---|---|---|
| Arize AX and Phoenix | Teams that want an OpenTelemetry-native stack: Phoenix to self-host for free, Arize AX when they want a managed service. | Phoenix: Elastic License 2.0. Arize AX: hosted service [source] Phoenix self-hosts; AX self-hosted only on Enterprise [source] | Trace spans and ingestion volume (GB) [source] | AX Free: 25k spans/month, 1 GB ingestion, 15-day retention, unlimited users [source, checked 2026-10-17] | AX Pro $50/month: 50k spans, 10 GB, 30-day retention (overage rates not stated on the pricing page) [source, checked 2026-10-17] |
| Braintrust | Product and engineering teams that run many experiments and want evals, human review and prompt playgrounds in one hosted tool. | Hosted service; its autoevals scorer library is MIT [source] Self-hosted data plane; on-prem or hosted deployment on Enterprise [source] | Processed data (GB) plus scores, plus model credits at token rates [source] | Starter $0: 1 GB processed data, 10k scores, 14-day retention, unlimited users; then $4/GB and $2.50 per 1k scores [source, checked 2026-10-17] | Pro $249/month: 5 GB and 50k scores included, then $3/GB and $1.50 per 1k scores; 30-day retention [source, checked 2026-10-17] |
| Comet Opik | Teams that want a fully open-source tracing and evaluation platform with a low-cost cloud option. | Apache-2.0 [source] Open-source version self-hosts with unlimited spans [source] | Spans per month [source] | Free Cloud: 25k spans/month, up to 10 team members, 60-day retention; open source self-hosted is free [source, checked 2026-10-17] | Pro Cloud $19/month: 100k spans, up to 50 members, then $5 per 100k spans [source, checked 2026-10-17] |
| DeepEval and Confident AI | Python teams that want pytest-style LLM tests with ready-made RAG and agent metrics, with an optional hosted platform. | DeepEval: Apache-2.0. Confident AI: hosted service [source] DeepEval runs locally; Confident AI on-prem on Enterprise [source] | Flat monthly fee per organization plus trace spans in GB-months [source] | Free: 2 seats, 1 project, 5 test runs/week, 1 GB-month of trace spans [source, checked 2026-10-17] | Starter $200/month per organization: unlimited seats, 5 projects, 5 GB-months, then $1 per GB-month [source, checked 2026-10-17] |
| Galileo (Splunk Agent Observability) | Enterprises already on Splunk Observability that want agent evaluation and guardrails in the same contract. | Commercial product (Splunk) [source] Observability Cloud and self-hosted deployments [source] | Spans (per 1.2 million spans), billed annually [source] | not verified | From $100/month per 1.2 million spans, billed annually; includes 500 million Luna tokens and 5 Luna fine-tuning runs per year [source, checked 2026-10-17] |
| Helicone | Teams that want request logging, cost tracking and a gateway first, with scores and datasets on top. | Apache-2.0 [source] Docker or Kubernetes self-hosting [source] | Requests and storage (GB) beyond the free allowance [source] | Hobby: 10,000 requests/month, 1 GB storage, 1 seat, 7-day retention [source, checked 2026-10-17] | Pro $79/month plus usage-based pricing (usage rates are shown only via a calculator, not verified) [source, checked 2026-10-17] |
| Langfuse | Teams that want open-source, self-hostable tracing, prompt management and evals with usage-based cloud pricing. | MIT for code outside the ee/ directories; ee/ code under a separate licence [source] Free to self-host (Docker Compose or Kubernetes) [source] | Billable units: traces, observations (spans, events, generations) and scores [source] | Hobby: 50k units/month, 30 days data access, 2 users [source, checked 2026-10-17] | Core $29/month: 100k units included, then $8 per 100k units (graduated down to $6 per 100k above 50M) [source, checked 2026-10-17] |
| LangSmith | Teams building on LangChain or LangGraph that want tracing, evals and agent deployment from the same vendor. | Hosted service; self-hosted and hybrid options on Enterprise [source] Self-hosted or hybrid on Enterprise only [source] | Seats plus base traces (14-day retention) and extended traces (180-day) [source] | Developer $0: 1 seat, up to 5k base traces/month, then pay as you go [source, checked 2026-10-17] | Plus $39/seat/month: up to 10k base traces/month, then pay as you go (per-trace rate not stated on the pricing page) [source, checked 2026-10-17] |
| Maxim AI | Teams testing agents with simulations and wanting experimentation, evals and observability priced per seat. | Platform is a hosted service; Bifrost gateway is Apache-2.0 [source] In-VPC deployment on Enterprise [source] | Seats plus logs per month [source] | Developer: free for up to 3 seats, 1 workspace, 10k logs/month, 3-day retention, no overages [source, checked 2026-10-17] | Professional $29/seat/month: 100k logs/month, 7-day retention, $1 per 10k extra logs [source, checked 2026-10-17] |
| Promptfoo | Developers who want config-driven prompt and model tests plus red-teaming that run locally and in CI. | MIT [source] Run locally or self-host; on-premise Enterprise tier [source] | Red-team probes (10k/month free); Enterprise priced on team size and needs [source] | Community: free, all LLM evaluation features, red teaming up to 10k probes/month [source, checked 2026-10-17] | Enterprise: custom pricing (no public price) [source, checked 2026-10-17] |
| Ragas | Teams that need RAG and agent metrics as a Python library inside their own pipeline or another platform. | Apache-2.0 [source] A library you install and run yourself [source] | No hosted plan published; metrics call a judge model you configure [source] | Free, open source [source, checked 2026-10-17] | No paid plan published [source, checked 2026-10-17] |
| W&B Weave | Teams already using W&B for model training that want LLM tracing and evals in the same account. | Weave SDK: Apache-2.0; the platform is a hosted service [source] Self-managed Weave instance [source] | Weave data ingestion (GB/MB) [source] | Free $0: 1 GB/month Weave data ingestion [source, checked 2026-10-17] | Pro from $60/month: 1.5 GB/month ingestion, then $0.10 per MB; for teams under 50 employees [source, checked 2026-10-17] |
Features
| Tool | OpenTelemetry | CI gating | LLM-as-judge | Human annotation | Prompt management | Dataset versioning | SOC 2 / HIPAA |
|---|---|---|---|---|---|---|---|
| Arize AX and Phoenix | Yes [src] | Yes [src] | Yes [src] | Yes [src] | Yes [src] | not verified | Yes [src] |
| Braintrust | Yes [src] | Yes [src] | Yes [src] | Yes [src] | Yes [src] | Yes [src] | Yes [src] |
| Comet Opik | Yes [src] | Yes [src] | Yes [src] | Yes [src] | Yes [src] | Yes [src] | Yes [src] |
| DeepEval and Confident AI | Yes [src] | Yes [src] | Yes [src] | Yes [src] | Yes [src] | Yes [src] | Yes [src] |
| Galileo (Splunk Agent Observability) | Yes [src] | not verified | Yes [src] | Yes [src] | not verified | not verified | not verified |
| Helicone | Yes [src] | Yes [src] | Yes [src] | not verified | Yes [src] | not verified | Yes [src] |
| Langfuse | Yes [src] | Yes [src] | Yes [src] | Yes [src] | Yes [src] | Yes [src] | Yes [src] |
| LangSmith | Yes [src] | Yes [src] | Yes [src] | Yes [src] | Yes [src] | Yes [src] | Yes [src] |
| Maxim AI | Yes [src] | Yes [src] | Yes [src] | Yes [src] | Yes [src] | not verified | Yes [src] |
| Promptfoo | Yes [src] | Yes [src] | Yes [src] | not verified | not verified | not verified | not verified |
| Ragas | not verified | Yes [src] | Yes [src] | not verified | not verified | not verified | not verified |
| W&B Weave | Yes [src] | not verified | Yes [src] | Yes [src] | Yes [src] | Yes [src] | Yes [src] |
Open-source evaluation frameworks
Frameworks are libraries and CLIs you run yourself. Many teams pair one with a platform above.
| Framework | Made by | What it tests | Licence | Last update |
|---|---|---|---|---|
| Promptfoo | Promptfoo (now part of OpenAI) | Your own prompts, models and LLM apps, with deterministic and LLM-graded assertions. | MIT | Repo last commit 2026-10-16 |
| DeepEval | Confident AI | Your own LLM app, RAG pipeline or agent, as pass/fail tests in CI. | Apache-2.0 | Repo last commit 2026-10-15 |
| Ragas | Exploding Gradients | Faithfulness, relevance and retrieval quality of your own RAG system. | not verified | not verified |
| Langfuse | Langfuse | Your own LLM app, on production traces and test datasets. | MIT (core) | Repo last commit 2026-10-16 |
| Arize Phoenix | Arize AI | Your own LLM app or agent, on traces and test datasets. | Elastic-2.0 | Repo last commit 2026-10-17 |
| lm-evaluation-harness | EleutherAI | Models on standard academic benchmarks with consistent prompts. | MIT | Repo last commit 2026-09-14 |
| OpenAI simple-evals | OpenAI | Chat models on standard benchmarks with simple zero-shot prompts. | MIT | Repo last commit 2026-04-22 |
| HELM | Stanford CRFM | Standardised, multi-metric comparison of models across shared scenarios. | Apache-2.0 | Repo last commit 2026-06-05 |
| Inspect Evals | UK AI Security Institute & community | Many benchmarks under one runner: coding, agents, safety, knowledge. | MIT | Repo last commit 2026-10-15 |
How to choose
- Decide where the data may live. If traces cannot leave your network, start with a self-hostable option (Langfuse, Phoenix, Opik, Helicone) or an Enterprise plan with self-hosting (Braintrust, LangSmith, Arize AX, Maxim, Splunk).
- Price on your own volume. The meters differ (units, spans, GB, seats, requests). Take one week of real traffic, count traces, spans and scores, and plug them into each vendor's own calculator.
- Check the CI path. If you want evals to block a merge, confirm the tool has a CI runner or a pytest/Vitest integration (see the Features table).
- Check the compliance plan, not just the logo. SOC 2 reports and HIPAA BAAs are often limited to higher plans; each tool page says which.
- Pick the evals, not just the tool. A platform runs your tests; you still need good test sets. See LLM-as-judge rubrics and frameworks and RAG evals.
Head-to-head and alternatives
- What is the difference between Braintrust and Langfuse?
- Langfuse or LangSmith?
- Braintrust or LangSmith?
- Arize Phoenix or Langfuse?
- Comet Opik or Langfuse?
- Promptfoo or DeepEval: pricing and platform?
- Alternatives: Langfuse · LangSmith · Braintrust · Arize AX and Phoenix
- Pricing calculator · LLM observability tools
FAQ
What is an LLM evaluation tool?
Software that scores the outputs of an LLM app or agent against tests: code checks, LLM-as-judge rubrics or human review. Platforms add tracing, datasets, prompt versioning and dashboards; frameworks are libraries you run in your own code or CI.
Which LLM evaluation tools are free?
Every platform here has a free tier or a free self-hosted version. Free to self-host from public code: Langfuse (MIT core), Arize Phoenix (Elastic License 2.0), Comet Opik (Apache-2.0), Helicone (Apache-2.0), plus the frameworks Promptfoo (MIT), DeepEval (Apache-2.0) and Ragas (Apache-2.0). Hosted free tiers have usage caps, listed in the table. Prices checked 2026-10-17.
What is the cheapest paid plan?
Of the published entry prices we verified on 2026-10-17: Comet Opik Pro Cloud $19/month; Langfuse Core $29/month; Maxim AI Professional $29/seat/month; LangSmith Plus $39/seat/month; Arize AX and Phoenix AX Pro $50/month; W&B Weave Pro from $60/month; Helicone Pro $79/month plus usage-based pricing (usage rates are shown only via a calculator, not verified); Galileo (Splunk Agent Observability) From $100/month per 1.2 million spans, billed annually; includes 500 million Luna tokens and 5 Luna fine-tuning runs per year; DeepEval and Confident AI Starter $200/month per organization; Braintrust Pro $249/month. Per-seat plans (LangSmith, Maxim) cost more as the team grows, and most plans add usage fees above the included allowance.
How do these tools charge?
By different meters, so compare on your own volume: Langfuse counts units (traces, observations and scores); Braintrust counts processed GB and scores; LangSmith charges per seat plus traces; Arize, Opik and Splunk (Galileo) count spans; Maxim charges per seat plus logs; Helicone counts requests and storage; Weave and Confident AI count ingested GB.
Which tools support OpenTelemetry?
Verified on a vendor page: Arize AX and Phoenix, Braintrust, Comet Opik, DeepEval and Confident AI, Galileo (Splunk Agent Observability), Helicone, Langfuse, LangSmith, Maxim AI, Promptfoo, W&B Weave. Not verified: Ragas.
Which tools can gate a CI build on eval results?
Verified on a vendor page: Arize AX and Phoenix, Braintrust, Comet Opik, DeepEval and Confident AI, Helicone, Langfuse, LangSmith, Maxim AI, Promptfoo, Ragas. Not verified: Galileo (Splunk Agent Observability), W&B Weave.
Which publish SOC 2 or HIPAA support?
Arize AX and Phoenix: SOC 2 Type II on paid AX plans; HIPAA on Enterprise. Braintrust: SOC 2 Type II listed; BAA for HIPAA on Enterprise. Comet Opik: SOC 2, ISO 27001, HIPAA and GDPR on Enterprise. DeepEval and Confident AI: SOC 2 on Team; HIPAA on Enterprise. Helicone: SOC 2 and HIPAA on Team ($799/month). Langfuse: SOC 2 Type II and ISO 27001 reports and HIPAA (BAA) on Pro ($199/month) and Enterprise. LangSmith: LangChain states it maintains SOC 2 Type II, GDPR and HIPAA compliance. Maxim AI: SOC 2 Type II, ISO 27001, HIPAA and GDPR on Enterprise. W&B Weave: SOC 2 Type II listed; the HIPAA option relies on bring-your-own-bucket, which the page says is not available with Weave. Not verified: Galileo (Splunk Agent Observability), Promptfoo, Ragas.
Is this a ranking?
No. Tools are listed alphabetically and we publish no star ratings until real practitioner reviews exist. Sponsored placements, if any, are labelled and never change the order or the facts.
Method
On 2026-10-17 we read each vendor's pricing page and docs and recorded only what they state, with a link per fact. We did not sign up for paid plans or run benchmarks for this page. The full data is free to reuse under CC BY 4.0: tools.json, tools.csv. Vendors can send corrections with a source link through the submit form.
Interested in a labelled featured slot or a newsletter mention? See sponsorship options.