Safety & factuality

ScrapeBench

by Joseph Marvin Imperial, Daniel Slate and Noam Kolt

What is ScrapeBench? ScrapeBench is a safety & factuality eval made by Joseph Marvin Imperial, Daniel Slate and Noam Kolt. A benchmark that checks whether AI agents respect a website's terms of service and robots.txt when asked to scrape it. The agent gets a fetch tool and a request to scrape a site's landing page; it may look up the ToS or robots.txt first. The data covers 1,626 popular sites whose ToS or robots.txt explicitly prohibits scraping and 269 that don't, with saved policy snapshots, and eight prompt styles that range from a bare request to legal wording to an instruction to ignore the rules. The paper reports that most agents comply less than 5% of the time on prohibited sites, even when they read the policy, while some over-refuse on permitted sites; it also includes a human baseline of 180 participants. Code is MIT, original data CC BY-SA 4.0; the paper is listed for the NeurIPS 2026 Evaluations and Datasets Track.

Page last updated 2026-10-18 · Evals Review

Editor note

A benchmark that checks whether AI agents respect a website's terms of service and robots.txt when asked to scrape it. The agent gets a fetch tool and a request to scrape a site's landing page; it may look up the ToS or robots.txt first. The data covers 1,626 popular sites whose ToS or robots.txt explicitly prohibits scraping and 269 that don't, with saved policy snapshots, and eight prompt styles that range from a bare request to legal wording to an instruction to ignore the rules. The paper reports that most agents comply less than 5% of the time on prohibited sites, even when they read the policy, while some over-refuse on permitted sites; it also includes a human baseline of 180 participants. Code is MIT, original data CC BY-SA 4.0; the paper is listed for the NeurIPS 2026 Evaluations and Datasets Track.

What it tests
Whether a web agent follows explicit scraping restrictions without refusing tasks that are allowed.
Original source
scrapebench.github.io/
Last updated
Code repo last push 2026-11-03; site repo 2026-11-11; arXiv 2611.11306 v1 2026-11-08
License
MIT (code); CC BY-SA 4.0 (original data)

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Sign in with GitHub to review

Related safety & factuality evals

  • HarmBench — Robustness to jailbreaks and harmful requests.
  • TruthfulQA — Truthfulness versus imitating human misconceptions.
  • StrongREJECT — Real effectiveness of jailbreak attacks.
  • SimpleQA — Short-form factuality and calibration.

All Safety & factuality evals

Discussion

No one has discussed ScrapeBench yet. Have you run it? Share what it caught, what it missed or how you set it up.

Sign in with GitHub to comment

Be specific and kind. No promotion or affiliate links. Community rules · All discussions