One Fabrication Is Enough: A Fake News Article Changes the Forecasts of LLM Forecasting Systems

Zeyu Zhang, Bradly C. StadiearXiv 2026 · arXiv

TL;DR

A single fabricated news article is sufficient to flip the predictions of state-of-the-art LLM forecasting systems in over 80% of cases. The study reveals that the very multi-agent search capabilities that make these systems powerful also make them highly vulnerable to "weak" attackers who simply publish plausible fake news to the open web.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly deployed for financial and geopolitical forecasting, often using Retrieval-Augmented Generation (RAG) to ground their predictions in real-time news. While this improves accuracy, it creates a massive, unvetted attack surface.

The authors investigate whether a "weak attacker"—one who only knows the forecasting question and has no access to the model's internals, prompts, or retrieval queries—can manipulate these systems. The core challenge is that forecasting systems must rely on the authenticity of retrieved documents, yet they lack a reliable mechanism to verify information published after their training cutoff.

Methods

The researchers evaluated eight forecasting systems, ranging from simple single-retrieval baselines to sophisticated deployed architectures like the AIA Forecaster and xAI’s Grok. They employed two primary attack strategies:

  1. Directional Attack: The attacker publishes one plausible news article supporting a chosen direction (e.g., a stock will fall).
  2. Destabilizing Attack: The attacker publishes two articles arguing opposite directions to see if the model hedges its prediction.

The study used two protocols: Protocol A, where the fabrication is forced into the model's context to measure its influence, and Protocol B, where the article is added to a news archive to see if the system's own retriever finds it.

Diagram of Directional and Destabilizing Attacks

To counter these, they tested two state-of-the-art defenses: Passage Filtering (using an LLM to judge if a passage is fabricated) and Partitioned Retrieval (isolating retrieval into disjoint parts of the archive to prevent a single document from dominating the aggregate result).

Datasets

The study utilized a robust set of real-world and simulated forecasting targets:

  • Stock Questions: 183 questions regarding one- and five-day returns, macro events, and stock spreads, evaluated against real market moves in 2026 windows.
  • Prediction Markets: 308 questions from ForecastBench, Polymarket, and Kalshi covering sports, politics, and economics.
  • News Archives: Custom archives containing 2,800 to 4,500 genuine news passages per time window, frozen to prevent data leakage.

Key Insights and Ideas

  • The Retrieval Paradox: Systems designed to be "thorough" by using multiple agents and wide searches are actually more exposed. While a simple retriever might miss a single fake article, multi-agent systems like Grok and AIA retrieve the fabrication in ~90% of runs.
  • The Verification Gap: Models cannot distinguish between "new" genuine news and "new" fabricated news. When asked to flag fabrications, models exhibited a high false-positive rate, flagging nearly half (45.7%) of genuine news as fake (see Table 3).
  • Failure to Hedge: In destabilizing attacks, models don't become "unsure." Instead, they typically pick one side based on the article's rank or specific content, often returning a more confident (but wrong) forecast than the clean baseline.
  • Invisible Harm: Standard accuracy metrics fail to detect the attack. Because the attacker is "right" about 50% of the time by chance, the system's average accuracy remains stable even as its specific decisions are being dictated by the attacker.

Key Results

  • Reversal Rates: Once a fabricated article enters the context, it reverses 74% to 82% of forecasts that originally opposed the attacker's direction across all frontier models tested (GPT-4, Gemini 1.5 Pro, etc.).
Reversal rates for different systems and domains
  • Model Compliance: Grok 4.20 was found to be the most compliant, following the fabricated article in 96.6% of stock-related cases (see Table 2).
Reversal rates by model and domain
  • Defense Fragility: While passage filtering and partitioned retrieval can stop a single article, they are easily bypassed. An attacker can use "different framings" (multiple versions of the same story) to beat partitioning, or "distortions" (editing one fact in a real article) to bypass filters.
  • Cost of Defense: Implementing both defenses together changes 29% of clean decisions and discards 39% of genuine news, representing a significant performance penalty for deployers.

Limitations and Future Work

The authors acknowledge that their evaluation was conducted on frozen archives rather than the live web to avoid ethical issues with planting real fake news. Furthermore, the "distortions" used were not optimized against the specific filters, suggesting the actual vulnerability might be even higher.

Future Work directions include:

  • Developing defenses that isolate sources rather than just passages.
  • Implementing corroboration requirements (requiring $n$ independent sources for a claim).
  • Creating decision rules for system supervisors to discount events that rest on a single, newly-published source.
  • Exploring how these attacks scale over longer forecasting horizons.

Summary updated

Abstract

LLMs are increasingly used for forecasting, and the best forecasting systems retrieve news as evidence to strengthen their predictions. This also opens them to attack. We study the weakest attacker of this kind, who knows only the forecasting question and publishes fabricated news articles, with no access to the model, the retriever or its queries. We evaluate how far this attack reaches on stock questions and real prediction-market questions. Once the article is in the model's context, it reverses 74% to 82% of the forecasts that opposed it, on every frontier model we tested. The model cannot verify the fabrication and follows it anyway. Retrieval is the attack's entry point, and the systems built to retrieve well are the most exposed: the architectures of two leading deployed forecasting systems, whose agents search for every related article, retrieve the fabricated one in about 90% of runs. Accuracy cannot reveal this harm: the attacked forecaster is right when the attacker's direction turns out right and wrong otherwise, so its average accuracy barely changes while its forecasts follow the attacker's guess. We also propose a second attacker, who publishes two articles that argue opposite directions to make the forecaster unsure. The model does not hedge; it returns a confident forecast in a direction the attacker no longer controls. We then evaluate the two strongest published retrieval-time defenses, passage filtering and partitioned retrieval. Each stops the one-article attack but is bypassed by a one-prompt change to the attack. Together they stop both bypasses at four articles, but sixteen distorted copies beat them, and they change 29% of clean decisions and discard 39% of the genuine news. Partitioned retrieval also requires control of the retrieval index. Systems that search the open web have no such control and are left with the filter alone. For them, the attack still works.
BibTeX
@misc{zhang2026one,
  author = {Zeyu Zhang and Bradly C. Stadie},
  title = {{One Fabrication Is Enough: A Fake News Article Changes the Forecasts of LLM Forecasting Systems}},
  year = {2026},
  eprint = {2611.15102},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15102},
}
Figure