TL;DR
By August 2026, nearly 25% of the crawlable web consisted of AI-generated or AI-edited text, a staggering rise from near-zero levels in 2022. However, this "AI flood" is highly uneven: it is concentrated in the web's periphery (low-centrality sites) and English-language pages, while institutional domains like .gov and .edu remain significantly more human-authored.
What Problem Does This Paper Address?
While we intuitively know that generative AI is reshaping the internet, quantifying this shift is notoriously difficult. Previous studies often focused on narrow niches (like Wikipedia or scientific papers) or relied on raw AI detector scores. The problem is that AI detectors are imperfect—they have varying false-positive and false-negative rates depending on the language, text length, and topic.
The authors aim to provide the first large-scale, longitudinal, and statistically rigorous estimate of AI prevalence across the entire public web. They tackle the "measurement error" problem head-on, moving beyond simple "flags" to a model that accounts for detector bias across different contexts.
Methods
The researchers analyzed a massive sample of 5.1 million webpages from 71 Common Crawl snapshots spanning 2019 to 2026. Their approach stands out due to two main innovations:
- Stratified Sampling by Centrality: Instead of a simple random sample, they stratified the web by PageRank (a measure of how "central" or well-connected a domain is). This allowed them to compare the "Top" tier (platforms like Wikimedia or the EU) against the "Tail" (small businesses, personal blogs).
- Bayesian Hierarchical Measurement Error Model: Rather than trusting detectors blindly, they built a model that treats detector outputs as noisy proxies. They used pre-2022 crawls as a "negative control" to calibrate false-positive rates and used multiple labeled benchmarks to estimate sensitivity. This allowed them to propagate uncertainty and correct for the fact that a detector might perform differently on a
.comsite versus a.govsite.
Datasets
- Primary Data: Common Crawl (2019–2026).
- Calibration Benchmarks:
- EditLens: Labeled pairs of human, AI-edited, and AI-generated text.
- RAID: A large-scale benchmark covering multiple LLM generators.
- Multilingual GenAI Content Detection: For non-English performance.
- AITDNA: Focused on human-AI collaboration (co-writing).
Key Insights and Ideas
- The "Tail" is Wagging the Dog: The vast majority of AI growth is happening on the periphery of the web. Sites in the "Tail" (the least central 90% of domains) account for nearly 80% of the net increase in AI content.
- English as the AI Lingua Franca: AI prevalence is nearly double in English-only pages compared to non-English pages. This suggests that LLM-based content generation is currently an English-centric phenomenon.
- Institutional Resilience: Domains like
.govand.edushow significantly lower AI prevalence (around 6%) compared to.com(nearly 30%). This likely reflects stricter editorial standards or different incentives for content creation in institutional vs. commercial spaces. - The Feedback Loop Risk: Because the crawlable web is the primary training data for future LLMs, the fact that AI content is flooding the web's periphery means future models may increasingly "learn" from the conventions and styles of current AI-generated "junk" or low-quality peripheral content.
Key Results
- Prevalence: AI-influenced text rose from near zero (pre-2022) to 24% (95% CI: 22.9–24.4%) by August 2026.
- Acceleration: The rate of increase nearly doubled, jumping from 4.6 percentage points per year (early 2023–2025) to 8.1 percentage points per year in the most recent period.
- Centrality Gap: The most central "Top" tier sites have ~11.6% AI prevalence, while the "Tail" sites reach ~27.5%.
Limitations and Future Work
- The "Dark" Web: The study only looks at the crawlable web. It excludes content behind logins, such as most of Facebook, X (Twitter), or private apps, where AI use patterns might be entirely different.
- Short-form Content: Detectors are unreliable for very short text. The authors restricted their study to pages with at least 500 characters, meaning they likely missed the rise of AI in comments, reviews, and social media posts.
- Detector Arms Race: The study assumes detector performance shifts once (at the start of the GenAI era). In reality, as LLMs become more sophisticated at mimicking human nuances, detectors may face a permanent decline in sensitivity that is difficult to model.
Summary updated