Cybench
by Stanford (Cybench team)
What is Cybench? Cybench is a framework and benchmark for testing AI agents on cybersecurity: 40 professional Capture the Flag tasks from four CTF competitions, with subtasks for partial credit.
Editor note
40 professional Capture the Flag tasks from four CTF competitions, with subtasks for partial credit, for measuring AI agents' cybersecurity capabilities and risks.
- What it tests
- Agents on offensive cybersecurity tasks.
- Original source
- cybench.github.io/
- Last updated
- Repo last commit 2026-09-24
- License
- Apache-2.0
When to use Cybench
- You assess an agent's offensive security abilities or risks.
- You want graded subtasks instead of only solved or unsolved.
Limitations
- 40 tasks is a small set.
- Tasks come from public CTF competitions, whose write-ups may be online.
How to run it
- Clone the repository and run tasks in the provided Docker environment with your agent.
Notable results
- README: 40 tasks from four distinct Capture the Flag competitions, with subtasks for more gradated evaluation. [Official repository]
Sources
Reviews
No reviews yet.
Reviews from practitioners who sign in with GitHub open soon.
Related safety & factuality evals
- HarmBench — Robustness to jailbreaks and harmful requests.
- TruthfulQA — Truthfulness versus imitating human misconceptions.
- StrongREJECT — Real effectiveness of jailbreak attacks.
- SimpleQA — Short-form factuality and calibration.