The uneven rise of AI-generated web content

Adam Visokay, Soumyakanti Pan, Molly Griston, Polina Iasakova, Mark Nepf, Denis Peskoff, Riya Sinha, Jikai Sun, Isabelle Thiara, Letao Wang, Yiwei Wu, Lauren Yan, Tyler H. McCormickarXiv 2026 · arXiv

TL;DR

By August 2026, nearly 25% of the crawlable web consisted of AI-generated or AI-edited text, a staggering rise from near-zero levels in 2022. However, this "AI flood" is highly uneven: it is concentrated in the web's periphery (low-centrality sites) and English-language pages, while institutional domains like .gov and .edu remain significantly more human-authored.

What Problem Does This Paper Address?

While we intuitively know that generative AI is reshaping the internet, quantifying this shift is notoriously difficult. Previous studies often focused on narrow niches (like Wikipedia or scientific papers) or relied on raw AI detector scores. The problem is that AI detectors are imperfect—they have varying false-positive and false-negative rates depending on the language, text length, and topic.

The authors aim to provide the first large-scale, longitudinal, and statistically rigorous estimate of AI prevalence across the entire public web. They tackle the "measurement error" problem head-on, moving beyond simple "flags" to a model that accounts for detector bias across different contexts.

Methods

The researchers analyzed a massive sample of 5.1 million webpages from 71 Common Crawl snapshots spanning 2019 to 2026. Their approach stands out due to two main innovations:

  1. Stratified Sampling by Centrality: Instead of a simple random sample, they stratified the web by PageRank (a measure of how "central" or well-connected a domain is). This allowed them to compare the "Top" tier (platforms like Wikimedia or the EU) against the "Tail" (small businesses, personal blogs).
  2. Bayesian Hierarchical Measurement Error Model: Rather than trusting detectors blindly, they built a model that treats detector outputs as noisy proxies. They used pre-2022 crawls as a "negative control" to calibrate false-positive rates and used multiple labeled benchmarks to estimate sensitivity. This allowed them to propagate uncertainty and correct for the fact that a detector might perform differently on a .com site versus a .gov site.

Datasets

  • Primary Data: Common Crawl (2019–2026).
  • Calibration Benchmarks:
    • EditLens: Labeled pairs of human, AI-edited, and AI-generated text.
  • RAID: A large-scale benchmark covering multiple LLM generators.
  • Multilingual GenAI Content Detection: For non-English performance.
  • AITDNA: Focused on human-AI collaboration (co-writing).

Key Insights and Ideas

  • The "Tail" is Wagging the Dog: The vast majority of AI growth is happening on the periphery of the web. Sites in the "Tail" (the least central 90% of domains) account for nearly 80% of the net increase in AI content.
  • English as the AI Lingua Franca: AI prevalence is nearly double in English-only pages compared to non-English pages. This suggests that LLM-based content generation is currently an English-centric phenomenon.
  • Institutional Resilience: Domains like .gov and .edu show significantly lower AI prevalence (around 6%) compared to .com (nearly 30%). This likely reflects stricter editorial standards or different incentives for content creation in institutional vs. commercial spaces.
  • The Feedback Loop Risk: Because the crawlable web is the primary training data for future LLMs, the fact that AI content is flooding the web's periphery means future models may increasingly "learn" from the conventions and styles of current AI-generated "junk" or low-quality peripheral content.

Key Results

  • Prevalence: AI-influenced text rose from near zero (pre-2022) to 24% (95% CI: 22.9–24.4%) by August 2026.
  • Acceleration: The rate of increase nearly doubled, jumping from 4.6 percentage points per year (early 2023–2025) to 8.1 percentage points per year in the most recent period.
The overall prevalence of AI text has surged since 2023, with the rate of change accelerating sharply in 2025.
  • Centrality Gap: The most central "Top" tier sites have ~11.6% AI prevalence, while the "Tail" sites reach ~27.5%.
AI prevalence is significantly higher in the "Tail" (least central) domains compared to the "Top" (most central) domains.

Limitations and Future Work

  • The "Dark" Web: The study only looks at the crawlable web. It excludes content behind logins, such as most of Facebook, X (Twitter), or private apps, where AI use patterns might be entirely different.
  • Short-form Content: Detectors are unreliable for very short text. The authors restricted their study to pages with at least 500 characters, meaning they likely missed the rise of AI in comments, reviews, and social media posts.
  • Detector Arms Race: The study assumes detector performance shifts once (at the start of the GenAI era). In reality, as LLMs become more sophisticated at mimicking human nuances, detectors may face a permanent decline in sensitivity that is difficult to model.

Summary updated

Abstract

In less than five years, generative artificial intelligence (AI) has dramatically reshaped the public-facing web. Using a sample of more than 5 million crawlable webpages, we demonstrate that the share containing AI-generated text has increased precipitously, but unevenly. Our analyses reveal substantial differences in AI composition across language and domain categories (e.g., .edu, .com, or .gov). A higher fraction of English-language webpages contain AI-generated text compared to pages in other languages, and prevalence is substantially lower on .gov and .edu domains compared to .com. These distinctions persist even after accounting for compositional effects. We also find heterogeneity based on a page's position in the connectivity structure of the web. AI-generated text is most common and has increased most rapidly in pages that are the least central in the graph, which drives much of the overall growth in AI-generated text. A slower increase has taken place for more central pages, with the most central tier emerging as the least likely to contain AI-generated text. Methodologically, we use multiple AI detection tools, and generate estimates using a Bayesian hierarchical model that incorporates corrections for the detectors' statistical properties (both false positives and false negatives) and allows for different detector performance across groups, while propagating uncertainty in detector error rates into prevalence estimates. Our work suggests that AI is a prominent, but varied, contributor to public information and internet-based communication.
BibTeX
@misc{visokay2026uneven,
  author = {Adam Visokay and Soumyakanti Pan and Molly Griston and Polina Iasakova and Mark Nepf and Denis Peskoff and Riya Sinha and Jikai Sun and Isabelle Thiara and Letao Wang and Yiwei Wu and Lauren Yan and Tyler H. McCormick},
  title = {{The uneven rise of AI-generated web content}},
  year = {2026},
  eprint = {2610.36631},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36631},
}
Figure