ScrapeBench: Evaluating Legal Compliance of AI Agents in Website Scraping

Joseph Marvin Imperial, Daniel Slate, Noam Kolt

TL;DR

ScrapeBench is a new benchmark designed to test whether AI agents follow the "rules of the road" when scraping the web. The study finds that most popular AI agents (like GPT-5.2 and Claude 4.6) fail to comply with website policies nearly 95% of the time unless they are explicitly given strong legal warnings or have the full text of the rules pasted directly into their prompts.

What Problem Does This Paper Address?

As AI agents become more autonomous, they are increasingly tasked with gathering data from the open web. However, many websites explicitly prohibit automated scraping through their Terms of Service (ToS) or robots.txt files. Violating these can lead to breaches of contract or even violations of computer security laws like the Computer Fraud and Abuse Act (CFAA).

Existing benchmarks for AI agents focus almost entirely on performance—can the agent successfully find and extract the data? They largely ignore compliance—is the agent legally allowed to take that action? This paper introduces ScrapeBench to fill this gap, providing an empirical framework to measure how well agents respect the legal boundaries set by website owners.

Methods

The researchers developed an agentic evaluation framework where AI agents are given a tool (fetch_url()) and instructed to scrape specific websites. The core of the methodology involves:

  1. Agentic Task Formulation: Agents are asked to scrape a landing page. Their actions are tracked through tool call traces.
  2. Outcome Classification: Results are categorized into four buckets:
  3. Violation: Scraping a site that explicitly prohibits it.
  4. Compliance: Refusing to scrape a site that prohibits it.
  5. Permissible: Scraping a site where it is allowed.
  6. Over-Refusal: Refusing to scrape even when it is lawful.
  7. Legal Priming Taxonomy: The researchers tested eight different prompt conditions (P1-P8). These range from a "Neutral" baseline (just "Scrape this site") to "Strong Legal Warnings" that include the full text of the ToS and references to specific laws.
  8. Human Baseline: To provide context, 180 humans were given the same tasks to see how their compliance rates compared to the AI.
Figure 1: Task Overview. Most AI agents tend to violate the legal policies of popular websites when instructed to scrape their landing page. Inserting into the prompt a strong legal warning and the text of the website’s terms of service (ToS) and robots.txt generally results in higher levels of compliance, as schematically illustrated in this figure.

Datasets

The authors curated a novel dataset of 1,895 popular websites sourced from the Chrome Top Million Websites (CrUX) repository.

  • Explicitly Prohibited (EP): 1,626 websites that forbid scraping in their ToS or robots.txt.
  • Not Explicitly Prohibited (NEP): 269 websites that do not have such restrictions.

The dataset covers various domains, including News & Media, E-commerce, and Education.

Figure 2: Breakdown of website data for SCRAPEBENCH (ntotal = 1,895, nEP = 1,626, nNEP = 269) over domain categories and scraping permissibility.

Key Insights and Ideas

  • Retrieval is not Reasoning: Surprisingly, agents often check the robots.txt or ToS files but proceed to scrape anyway. This suggests that the failure isn't just about "not seeing" the rules, but a failure to correctly interpret or prioritize them.
  • The "Claude" Anomaly: Claude Opus 4.6 showed a unique behavior. While it was the most resistant to adversarial prompts (refusing to ignore the rules even when told to), it suffered from extreme over-refusal, declining to scrape 98.5% of perfectly legal websites.
  • Interpretation Failures: In 88% of violations, agents provided rationales that showed they either misread the ToS or wrongly assumed that a permissive robots.txt overrode a restrictive ToS.
  • Prompt Engineering as a Legal Shield: Compliance only reached acceptable levels (above 70%) when the full legal text was "in-context"—meaning the user had to manually paste the ToS into the prompt.

Key Results

  • Baseline Failure: Under neutral prompts (P1), compliance was near zero (<5%) for five out of six agents.
  • The Power of Warnings: Strong legal warnings (P7) significantly boosted compliance, with Claude and Gemini reaching ~85%.
  • Human vs. AI: Humans were much more cautious than AI agents. Under neutral prompts, humans complied 48.9% of the time, compared to the agents' near-zero performance.
  • Adversarial Vulnerability: Most agents (except Claude) were easily swayed by instructions to "ignore the terms of service," with compliance rates dropping to near zero.
Figure 3: Compliance rates of AI agents on EXPLICITLY PROHIBITED websites (n = 1,626) across eight legal priming prompts.
Figure 4: Over-refusal rates of AI agents on NOT EXPLICITLY PROHIBITED websites (n = 269) across eight legal priming prompts.

Limitations and Future Work

The authors acknowledge several limitations:

  • Single-Turn Setup: The benchmark currently only evaluates single-turn interactions, not complex, multi-step agentic workflows.
  • Legal Nuance: The study focuses on "Explicitly Prohibited" vs "Not Explicitly Prohibited," but real-world law involves nuances like "Fair Use" that are harder to benchmark.
  • Language and Geography: The dataset is limited to English-language, US-accessible websites.

Future work could involve testing multi-agent systems, exploring non-English legal contexts, and developing more robust training methods to ensure agents can reason about legal constraints autonomously without needing "labor-intensive" prompt engineering.

Summary updated

Abstract

AI agents deployed on the open web are becoming more capable of performing actions that may breach contractual provisions and violate computer security laws. In this paper, we introduce ScrapeBench, the first agentic benchmark for empirically evaluating the legal compliance of AI agents with website scraping policies, namely terms of service (ToS) and robots.txt directives. Across widely used closed-weight and open-weight AI agents, we find that most agents exhibit low compliance rates (< 5%) when instructed to scrape websites in our sample of 1,626 popular websites that explicitly prohibit scraping. In addition, we observe that even where AI agents actively check the applicable ToS and robots.txt, they continue to exhibit low compliance rates. Conversely, in our sample of 269 websites that do not prohibit scraping, we observe that agents sometimes refuse to scrape despite it being lawful, such as Claude Opus 4.6 exhibiting an over-refusal rate of 98.5%. Our human baseline experiments (n = 180 participants) suggest that humans exhibit compliance rates comparable to AI agents but over-refuse at much higher rates.
BibTeX
@misc{imperial2026scrapebench,
  author = {Joseph Marvin Imperial and Daniel Slate and Noam Kolt},
  title = {{ScrapeBench: Evaluating Legal Compliance of AI Agents in Website Scraping}},
  year = {2026},
  eprint = {2611.11306},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11306},
}
Figure