DualChem
by Scale AI Preparedness & Policy team (Sehwag, Dalvi, Maurya et al.), built under a research agreement with NIST CAISSI
What is DualChem? DualChem is a safety & factuality eval made by Scale AI Preparedness & Policy team (Sehwag, Dalvi, Maurya et al.), built under a research agreement with NIST CAISSI. A dual-use chemistry benchmark of 600 questions written by chemistry experts across seven lab domains, mostly multiple choice with some free-form items. Each question is realistic lab work whose answer could serve legitimate research or misuse, and each comes with an expert-written benign and harmful use case plus severity scores. The paper runs 30+ frontier models on it (mean accuracy 55.2%, best 67.7%) and a human study in which 120 chemistry graduates gained 16 percentage points with LLM help. It also checks whether models notice the dual-use risk: on questions they flagged as dual-use they still answered about 36% of the time, and their severity ratings matched experts only weakly. Scale says it is the first benchmark from its research agreement with NIST's CAISSI.
Editor note
A dual-use chemistry benchmark of 600 questions written by chemistry experts across seven lab domains, mostly multiple choice with some free-form items. Each question is realistic lab work whose answer could serve legitimate research or misuse, and each comes with an expert-written benign and harmful use case plus severity scores. The paper runs 30+ frontier models on it (mean accuracy 55.2%, best 67.7%) and a human study in which 120 chemistry graduates gained 16 percentage points with LLM help. It also checks whether models notice the dual-use risk: on questions they flagged as dual-use they still answered about 36% of the time, and their severity ratings matched experts only weakly. Scale says it is the first benchmark from its research agreement with NIST's CAISSI.
- What it tests
- Whether a model gives real uplift on ambiguous dual-use chemistry questions and whether it recognizes the risk, not just whether it answers correctly.
- Original source
- scale.com/blog/dualchem-evaluating-ai-models-dual-use-chemistry-risks
- Last updated
- Announced 2026-10-28 (Scale blog and paper)
- License
- Not checked yet — see source
Related safety & factuality evals
- HarmBench — Robustness to jailbreaks and harmful requests.
- TruthfulQA — Truthfulness versus imitating human misconceptions.
- StrongREJECT — Real effectiveness of jailbreak attacks.
- SimpleQA — Short-form factuality and calibration.
Discussion
No one has discussed DualChem yet. Have you run it? Share what it caught, what it missed or how you set it up.