Cursor Agent SDK Evals: What They Test (Hands-on, Oct 2026)
Agent SDK is Cursor's product (Cursor is made by Anysphere). Evals Review is independent of Cursor.
Hands-on tested 27 Oct 2026 (docs review plus a small hands-on run)
One 3-case eval run on our own Cursor account; see What we ran. Checked 2026-10-27. Anything we could not confirm says “unknown”. How we review.
Short answer: Agent SDK lets you build agents as a folder of Markdown prompts and TypeScript tools and run them on Cursor's hosting or your own server. Its evals are regression tests for those agents: fixed inputs, assertions on which tools ran, and code or LLM-judge checks on the reply, with JUnit output for CI. It is not a general eval platform; use it if your agent is built on Agent SDK.
When the built-in evals are enough: if your agent is built on Agent SDK and you need tool-call and reply checks with code or LLM judges and JUnit output in CI, the docs cover that. Look at Braintrust or Langfuse if you need a hosted experiments UI or production tracing, and at promptfoo or DeepEval if you need open-source evals that work with any LLM app, not just Agent SDK agents.
Quick pick
| If you need | Pick | Why |
|---|---|---|
| Regression tests for an agent built on Agent SDK | Agent SDK evals | Built in; asserts on tool calls and replies of the real agent |
| Open-source evals for any LLM app in CI | promptfoo | Open source, not tied to one agent framework |
| Python test-style metrics | DeepEval | Open-source pytest-style metrics |
| Tracing plus evals plus human review, self-hosted | Langfuse | Open-source platform with annotation queues |
| Hosted eval platform with experiments UI | Braintrust | Commercial platform for datasets, scorers and experiments |
What's new (checked 27 Oct 2026)
- 23 Oct 2026: version 0.3.0 marked “GA Release” in the Agent SDK changelog.
- 26 Oct 2026: Cursor announced Agent SDK on X. We could not load the post, so no fact on this page comes from it.
- Renamed: the package is now @grok/agent-sdk; the docs say it “was @cursor/bdk before” (Agent SDK overview).
Eval features compared (documented facts only)
| Cursor Agent SDK | promptfoo | DeepEval | |
|---|---|---|---|
| Assertion types | Trajectory checks (tool calls, order, events, subagents), text/regex, schema, similarity, numeric scores | equals, JSON structure, similarity, custom JavaScript/Python/Ruby, model-graded | Metrics (e.g. G-Eval) and custom metrics |
| Dataset formats | JSON, JSONL, YAML | YAML; CSV, JSONL, XLSX and Google Sheets test files | CSV and JSON |
| Graders | Code matchers and LLM judges (factuality, summarizes, closedQA, sql, custom) | Code assertions and LLM rubrics | LLM-as-a-judge metrics and custom metrics |
| CI output | JSON, JUnit, artifact folder; exit code 1 on failure | results.json, HTML, JUnit | deepeval test run (Pytest), GitHub Actions example |
| Sources (checked 27 Oct 2026) | Evals reference | assertions · test cases · CI/CD | metrics · datasets · CI/CD |
The eval feature, field by field
| Field | What the docs say | Source |
|---|---|---|
| What it measures | Run success, tool calls made or avoided, tool and event order, max tool calls, failed actions, subagent calls, loaded skills, tagged artifacts, reply content | Reference |
| Test format | TypeScript files evals/**/*.eval.ts with defineEval; one test or a list of cases; multi-turn via repeated t.send | Evals guide |
| Datasets | JSON, JSONL or YAML loaders from the project; workspaceFiles seed inputs; GitHub webhook payloads can be replayed as fixtures | Reference |
| Code graders | Matchers such as includes (text/regex), schema matching (e.g. zod), similarity, t.score 0 to 1 | Reference |
| LLM-judge graders | factuality, summarizes, closedQA, sql and a custom model prompt; .gate, .soft and .atLeast thresholds; judge model set in config | Reference |
| Human review | Unknown: not described in the eval docs we read | — |
| Repeats | iterations runs a case 1 to 100 times; maxConcurrency 1 to 200 | Reference |
| CI | --json, --junit, --artifacts; exit code 1 on failures; --strict for soft thresholds | CLI reference |
| Results storage | Local runs write summary.json, results.jsonl and one JSON per case; a running server keeps remote batches (playground history, default 20) | Reference |
| Results UI | Unknown beyond the CLI, JSON files and server-side playground history mentioned in the reference | — |
| Side effects | Eval sessions carry purpose: "eval" so tools can skip real writes | Evals guide |
Hosting, licence and price
| Finding | Source | |
|---|---|---|
| Package | @grok/agent-sdk on npm, version 0.3.0; previously @cursor/bdk | npm |
| Release | 0.3.0 marked GA on 23 Oct 2026 in the changelog | Changelog |
| Licence | Proprietary: "Cursor SDK License … Use is subject to Cursor's Terms of Service" | LICENSE.md in package |
| Self-hosting | Yes, agent-sdk serve on Node 22.13+; Cursor-managed hosting also offered | Deployment |
| Credential | Model turns and judges need a Cursor credential (CURSOR_API_KEY) | Evals guide |
| Price | Unknown: no Agent SDK or eval price found on the pages we read | — |
Not the same as “run Cursor in your evals”
Cursor also documents using its @cursor/sdk agent inside your own benchmark harness, then scoring the result with your own tests or judge. That is for benchmarking Cursor's agent; Agent SDK evals are for testing an agent you build. Run Cursor in your evals.
Who should use it
- Use it if you build a PR reviewer, triage or Slack agent on Agent SDK and want each prompt or tool change checked before merge.
- Look elsewhere if you need open-source code, framework-neutral evals, production tracing or human annotation: see LLM evaluation tools and AI agent evals.
What we ran
On 27 Oct 2026 at 14:24–14:26 IST we installed @grok/agent-sdk 0.3.0 on Node 22 and built a tiny agent: one Markdown prompt and one TypeScript tool (lookup_benchmark, which returns static data for two benchmarks). Agent and judge model: composer-2.5. Built-in Cursor tools were turned off. Each case ran once.
| Case | What it checks | Result |
|---|---|---|
| swe-tool | Tool was called; reply mentions GitHub issues | Passed (7.4 s) |
| gpqa-json | Tool was called; reply parses as JSON matching a Zod schema with task_count 198 | Passed (6.8 s) |
| unknown-judge | Tool was called; LLM judge (closedQA, composer-2.5) checks it admits no data rather than inventing details | Passed, judge score 1.000 (bar 0.8) |
Totals: 3 passed, 0 failed. The --json run took 15.8 s by the CLI's own timer (about 18 s wall clock); the --junit run took 14.6 s by the JUnit file (15.6 s wall clock). The CLI printed no token, usage or cost figures, so we can't report the cost; Cursor's docs say SDK runs use the same pricing and request pools as the IDE. Issues we saw: harmless [bdk:otel] … You can only call end() on a span once log errors and a “GitHub auth: unavailable” notice. Neither affected results. Three cases on a toy agent show the workflow runs. They say nothing about quality at scale.
Raw output (session and trace IDs redacted): summary.json · results.jsonl · junit.xml · terminal transcript · source: instructions.md, tool, agent.ts, evals.config.ts, benchmarks.eval.ts.
How we checked
On 2026-10-27 we read Cursor's Agent SDK overview, evals guide, evals reference, CLI reference, deployment page and changelog, and the npm package metadata and LICENSE.md for @grok/agent-sdk 0.3.0. We then ran the small eval described above. We did not use the launch post on X as a source for any fact here. We give no ratings or benchmark scores. The only numbers are from our own 3-case run.
FAQ
What do Agent SDK evals measure?
Each eval sends fixed input to the real agent and checks the run: whether it succeeded, which tools it called or avoided, tool order, subagent calls, events, and the reply. Replies can be checked with matchers (text, regex, schema), similarity, numeric scores from 0 to 1, or LLM judges.
Which graders are built in?
Code assertions and matchers, plus model judges: factuality, summarizes, closedQA, sql, and t.judge.model for a custom prompt. The docs we read describe no human-review or annotation queue.
Can I run them in CI?
Yes. agent-sdk eval has --json, --junit and --artifacts flags, exits 1 on failed cases, and --strict also fails scored cases below their soft threshold. The docs suggest smoke-tagged evals on pull requests and the full suite on a schedule.
Is it open source, and what does it cost?
No open-source licence: the npm package's LICENSE.md says use is subject to Cursor's Terms of Service. Model turns need a Cursor API key. We found no published price for Agent SDK or its evals on the pages we read, so cost is unknown. Our 3-case test run printed no usage or cost figures.
Can I self-host it?
The deployment docs describe Cursor-managed hosting or self-hosting with agent-sdk serve on Node 22.13+. A self-hosted server still uses a CURSOR_API_KEY.
Does it replace Braintrust, Langfuse or promptfoo?
Only for agents built on Agent SDK. Its evals test those agents' trajectories; they are not a general tool for evaluating any LLM app, tracing production traffic or managing human review.