ScrapeBench
by Joseph Marvin Imperial, Daniel Slate and Noam Kolt
What is ScrapeBench? ScrapeBench is a safety & factuality eval made by Joseph Marvin Imperial, Daniel Slate and Noam Kolt. A benchmark that checks whether AI agents respect a website's terms of service and robots.txt when asked to scrape it. The agent gets a fetch tool and a request to scrape a site's landing page; it may look up the ToS or robots.txt first. The data covers 1,626 popular sites whose ToS or robots.txt explicitly prohibits scraping and 269 that don't, with saved policy snapshots, and eight prompt styles that range from a bare request to legal wording to an instruction to ignore the rules. The paper reports that most agents comply less than 5% of the time on prohibited sites, even when they read the policy, while some over-refuse on permitted sites; it also includes a human baseline of 180 participants. Code is MIT, original data CC BY-SA 4.0; the paper is listed for the NeurIPS 2026 Evaluations and Datasets Track.
Editor note
A benchmark that checks whether AI agents respect a website's terms of service and robots.txt when asked to scrape it. The agent gets a fetch tool and a request to scrape a site's landing page; it may look up the ToS or robots.txt first. The data covers 1,626 popular sites whose ToS or robots.txt explicitly prohibits scraping and 269 that don't, with saved policy snapshots, and eight prompt styles that range from a bare request to legal wording to an instruction to ignore the rules. The paper reports that most agents comply less than 5% of the time on prohibited sites, even when they read the policy, while some over-refuse on permitted sites; it also includes a human baseline of 180 participants. Code is MIT, original data CC BY-SA 4.0; the paper is listed for the NeurIPS 2026 Evaluations and Datasets Track.
- What it tests
- Whether a web agent follows explicit scraping restrictions without refusing tasks that are allowed.
- Original source
- scrapebench.github.io/
- Last updated
- Code repo last push 2026-11-03; site repo 2026-11-11; arXiv 2611.11306 v1 2026-11-08
- License
- MIT (code); CC BY-SA 4.0 (original data)
Related safety & factuality evals
- HarmBench — Robustness to jailbreaks and harmful requests.
- TruthfulQA — Truthfulness versus imitating human misconceptions.
- StrongREJECT — Real effectiveness of jailbreak attacks.
- SimpleQA — Short-form factuality and calibration.
Discussion
No one has discussed ScrapeBench yet. Have you run it? Share what it caught, what it missed or how you set it up.