TL;DR
An HTTP 200 "OK" status code is often a lie. This paper demonstrates that approximately 21% of web pages fetched by AI agents return unusable content—such as bot challenges, empty app shells, or "soft 404" errors—despite claiming success. This "silent failure" significantly distorts measurements of AI reliability, inflating apparent citation drift by nearly fourfold.
What Problem Does This Paper Address?
As Large Language Model (LLM) agents and search-augmented assistants (like Perplexity or GPT-4o with search) increasingly browse the live web, they rely on a fundamental assumption: if a server returns an HTTP 200 status code, the fetch was successful.
However, the modern web is hostile to automated clients. Servers often use "soft 404s" (returning 200 for missing pages), serve JavaScript-heavy "app shells" that appear empty to simple scrapers, or present bot-management challenges (like Cloudflare Turnstile). If an agent treats these non-content pages as valid evidence, it leads to hallucinations, incorrect citations, and flawed evaluations of agent performance. The authors set out to quantify this gap and provide better ways to detect these "silent failures."
Methods
The researchers conducted a large-scale measurement study using a dual-client approach:
- Static Client (K1): A simple GET request (similar to
curl), which is the cheapest and most common way agents fetch data. - Rendered Client (K2): A full headless Chromium browser (via Playwright) that executes JavaScript, serving as the "ground truth" for what a human would see.
They developed a fetch-outcome taxonomy to categorize 200 responses into seven types: Content, Block, Challenge, Shell, Error/Substitute, Parked, and Consent Wall. You can see the breakdown and examples in Table 1 below:
To detect these failures without the high cost of full rendering, they compared simple heuristics (like text length and "soft 404" probes) against more complex content-reading detectors, including zero-shot LLMs and fine-tuned small language models (SLMs).
Datasets
The study analyzed 10,307 URLs across six distinct populations relevant to AI agents:
- S1: Links from
llms.txtfiles (a new standard for AI-friendly site maps). - S2-S5: Various rank bands from the Tranco list (popular vs. niche sites).
- P: 4,795 URLs cited in ExpertQA, a benchmark for expert-verified AI claims.
The researchers also introduced a set of random-path probes (fetching non-existent URLs on the same domains) to identify sites that habitually return "soft 404s." The population details are summarized in Table 2:
Key Insights and Ideas
- The Popularity Paradox: The most popular sites (Tranco top 10k) are actually the most difficult for agents to read. They have the highest rates of bot challenges and blocks (see Figure 1).
- Heuristics are Surprisingly Effective: You don't need a massive LLM to tell if a fetch failed. A "tuned combination of cheap heuristics"—specifically checking text length and comparing the page to a random-path probe—recovers 86% of unusable responses with a low false-alarm rate.
- Silent Drift is a Measurement Mirage: When researchers check if an AI's citations are still "live," they often see high "drift" (content changing or disappearing). This paper shows that 79% of apparent citation drift in the ExpertQA dataset was actually just the agent failing to fetch the page correctly (e.g., hitting a block screen), not the content itself changing.
- Rendering is Not a Panacea: While rendering helps with "app shells," it is 12x slower than static fetching and can still be defeated by consent walls or headless-browser detection.
Key Results
- 20.9% of static 200 responses are unusable. This varies by population, as shown in Figure 1, with cited sources and popular sites being the most problematic.
- Soft 404s are rampant: 13.1% of sites return a 200 status for a random, non-existent path.
- Distortion: Treating all 200s as content inflates the apparent silent citation-drift rate from a true 5.8% to a misleading 21.9%.
- Detector Performance: The best heuristic-based detector (H-best) achieved a recall of 0.86. While a text classifier (M2) had higher recall (0.91), it also had a much higher false-alarm rate (13% vs 6%).
Limitations and Future Work
- Vantage Point Bias: The study was conducted from a single vantage point. Block rates and challenges are highly sensitive to the crawler's IP address, location, and time of day.
- The Cat-and-Mouse Game: Bot management services constantly evolve. A heuristic that works today might be bypassed tomorrow.
- Future Work: The authors suggest that agents should adopt "outcome-aware fetching," where a cheap static fetch is performed first, and the system only "escalates" to expensive rendering or LLM-based verification if a failure flag is triggered.
Summary updated