When 200 OK Is Not an Answer: Silent Fetch Failures in Agentic Web Content Acquisition

Shuyang ZhangarXiv 2026 · arXiv

TL;DR

An HTTP 200 "OK" status code is often a lie. This paper demonstrates that approximately 21% of web pages fetched by AI agents return unusable content—such as bot challenges, empty app shells, or "soft 404" errors—despite claiming success. This "silent failure" significantly distorts measurements of AI reliability, inflating apparent citation drift by nearly fourfold.

What Problem Does This Paper Address?

As Large Language Model (LLM) agents and search-augmented assistants (like Perplexity or GPT-4o with search) increasingly browse the live web, they rely on a fundamental assumption: if a server returns an HTTP 200 status code, the fetch was successful.

However, the modern web is hostile to automated clients. Servers often use "soft 404s" (returning 200 for missing pages), serve JavaScript-heavy "app shells" that appear empty to simple scrapers, or present bot-management challenges (like Cloudflare Turnstile). If an agent treats these non-content pages as valid evidence, it leads to hallucinations, incorrect citations, and flawed evaluations of agent performance. The authors set out to quantify this gap and provide better ways to detect these "silent failures."

Methods

The researchers conducted a large-scale measurement study using a dual-client approach:

  1. Static Client (K1): A simple GET request (similar to curl), which is the cheapest and most common way agents fetch data.
  2. Rendered Client (K2): A full headless Chromium browser (via Playwright) that executes JavaScript, serving as the "ground truth" for what a human would see.

They developed a fetch-outcome taxonomy to categorize 200 responses into seven types: Content, Block, Challenge, Shell, Error/Substitute, Parked, and Consent Wall. You can see the breakdown and examples in Table 1 below:

Table 1: Outcome types for 200 responses, with examples from the reviewed sample. Auto: automatic type of all 4,560 static 200 responses (pages and probes). Rev.: reviewed type of the 300 sampled items.

To detect these failures without the high cost of full rendering, they compared simple heuristics (like text length and "soft 404" probes) against more complex content-reading detectors, including zero-shot LLMs and fine-tuned small language models (SLMs).

Datasets

The study analyzed 10,307 URLs across six distinct populations relevant to AI agents:

  • S1: Links from llms.txt files (a new standard for AI-friendly site maps).
  • S2-S5: Various rank bands from the Tranco list (popular vs. niche sites).
  • P: 4,795 URLs cited in ExpertQA, a benchmark for expert-verified AI claims.

The researchers also introduced a set of random-path probes (fetching non-existent URLs on the same domains) to identify sites that habitually return "soft 404s." The population details are summarized in Table 2:

Table 2: Populations. Pages are fetched with both clients; every site in S1–S5 also receives two random-path probes.

Key Insights and Ideas

  • The Popularity Paradox: The most popular sites (Tranco top 10k) are actually the most difficult for agents to read. They have the highest rates of bot challenges and blocks (see Figure 1).
  • Heuristics are Surprisingly Effective: You don't need a massive LLM to tell if a fetch failed. A "tuned combination of cheap heuristics"—specifically checking text length and comparing the page to a random-path probe—recovers 86% of unusable responses with a low false-alarm rate.
  • Silent Drift is a Measurement Mirage: When researchers check if an AI's citations are still "live," they often see high "drift" (content changing or disappearing). This paper shows that 79% of apparent citation drift in the ExpertQA dataset was actually just the agent failing to fetch the page correctly (e.g., hitting a block screen), not the content itself changing.
  • Rendering is Not a Panacea: While rendering helps with "app shells," it is 12x slower than static fetching and can still be defeated by consent walls or headless-browser detection.

Key Results

  • 20.9% of static 200 responses are unusable. This varies by population, as shown in Figure 1, with cited sources and popular sites being the most problematic.
Figure 1: Outcome of static fetches by population
  • Soft 404s are rampant: 13.1% of sites return a 200 status for a random, non-existent path.
  • Distortion: Treating all 200s as content inflates the apparent silent citation-drift rate from a true 5.8% to a misleading 21.9%.
  • Detector Performance: The best heuristic-based detector (H-best) achieved a recall of 0.86. While a text classifier (M2) had higher recall (0.91), it also had a much higher false-alarm rate (13% vs 6%).
Figure 3: Recall by reviewed type (types with at least 12 un- usable items) and false-alarm rate on usable items, reviewed test sample. Thresholds fixed on validation.

Limitations and Future Work

  • Vantage Point Bias: The study was conducted from a single vantage point. Block rates and challenges are highly sensitive to the crawler's IP address, location, and time of day.
  • The Cat-and-Mouse Game: Bot management services constantly evolve. A heuristic that works today might be bypassed tomorrow.
  • Future Work: The authors suggest that agents should adopt "outcome-aware fetching," where a cheap static fetch is performed first, and the system only "escalates" to expensive rendering or LLM-based verification if a failure flag is triggered.

Summary updated

Abstract

Web agents and answer engines fetch pages at question time, and liveness checks of cited sources treat an HTTP 200 response as success. We measure how often that assumption fails and what it does to measurements built on fetched pages. We fetched 10,307 URLs in three days with a polite static client and a rendered browser client: pages from 1,507 sites in five strata, random non-existent paths on every site, and the 4,795 URLs cited by expert-verified claims in ExpertQA. We define a taxonomy of what a 200 response contains, label the static 200 responses automatically, and check the labels against 300 items reviewed by an LLM-based assistant; a human check of 60 of them retained every reviewed usability label after discussion. By the automatic labels, 20.9% of the pages that a static client receives with status 200 do not contain the page's main content: app shells, bot challenges, block pages, error and placeholder pages, and partial content. On the held-out test split, the reviewed sample lowers this estimate from 25.5% to 20.0%. 13.1% of the sites whose random-path probe was answered returned 200. The error propagates: for the cited sources, treating 200 as content overstates the share of claim-source pairs that can be checked against live content by 12.2 points and inflates the apparent silent citation-drift rate from 5.8% to 21.9%. Cheap heuristics catch most of it. Text length, probe comparison and redirect-to-root, combined and tuned on held-out sites, recover 86% of unusable reviewed responses at a false-alarm rate near 5%. Content-reading detectors perform comparably: the best, the text classifier, gains five to seven points of recall at more than twice the false-alarm rate. We argue that agents and their evaluations should report fetch outcomes explicitly instead of inferring them from the status line and will release the labelled data and code.
BibTeX
@misc{zhang2026when,
  author = {Shuyang Zhang},
  title = {{When 200 OK Is Not an Answer: Silent Fetch Failures in Agentic Web Content Acquisition}},
  year = {2026},
  eprint = {2611.00884},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.00884},
}
Figure