HarmBench vs StrongREJECT
HarmBench or StrongREJECT? HarmBench is a standard setup for comparing automated red-teaming attacks and defences, with 18 methods tested against 33 target LLMs and defences in the paper. StrongREJECT measures how useful jailbroken responses really are, and finds that earlier methods overstate jailbreak success.
| HarmBench | StrongREJECT | |
|---|---|---|
| Made by | Center for AI Safety & collaborators | UC Berkeley researchers |
| Question answered | How well do attacks and defences compare? | Did the jailbreak actually yield useful harmful information? |
| Scale in paper | 18 red-teaming methods, 33 targets and defences | Automated grader checked against human judgements |
| Running cost | Compute-heavy; examples use Slurm | Lighter: grade responses with the evaluator |
| Doesn't measure | Over-refusal of safe requests | General safety or over-refusal |
| License | — | — |
Verdict: when to use which
Use HarmBench if you are benchmarking attacks or defences in a shared setup.
Use StrongREJECT if you need an honest score for whether a jailbreak works. Pair either with XSTest for over-refusal.
Other comparisons: SWE-bench vs SWE-bench-Live · HumanEval vs BigCodeBench · HumanEval vs LiveCodeBench · MMLU vs GPQA · AlpacaEval vs MT-Bench (LLM-as-a-judge paper) · MT-Bench (LLM-as-a-judge paper) vs LMArena (Chatbot Arena) · BEIR vs MTEB Leaderboard · WebArena vs OSWorld · GAIA vs BrowseComp