ForecastGym: A Live, Controlled, and Replayable Benchmark for Forecasting Agents

Yicheng Tao, Yuante Li, Zipei FanarXiv 2026 · arXiv

TL;DR

ForecastGym is a novel benchmark for LLM-based forecasting agents that solves the "information leakage" problem by prospectively recording web search results while questions are still open. The study reveals a surprising truth: the choice of search API can impact a model's accuracy more than the choice of the language model itself, and current LLMs still significantly trail human prediction markets.

What Problem Does This Paper Address?

Evaluating AI agents that forecast real-world events is notoriously difficult due to four main challenges:

  1. Leaky Reconstruction: Retrospective evaluations (searching for info after an event has happened) often leak the outcome. Even with date filters, search engines might return pages updated after the fact or rank historical pages based on today’s knowledge.
  2. Entangled Retrieval and Reasoning: In live benchmarks, it’s hard to tell if a bad forecast is due to a poor model (reasoning) or a poor search tool (retrieval).
  3. Uncontrolled Evidence Dynamics: Information evolves as an event approaches. To compare how agents revise their beliefs, they must experience the exact same "trajectory" of incoming evidence.
  4. Search-Source Variation: Different search providers (e.g., Google vs. Tavily) return different results, but their specific impact on forecasting hasn't been isolated from model performance.

Methods

The authors introduce a two-phase lifecycle for the benchmark:

  • Phase 1: Live Capture. While a prediction market question (from platforms like Polymarket or Kalshi) is still open, ForecastGym queries four commercial search APIs (Tavily, Exa, Serper, and Parallel) every two days. This "freezes" the evidence exactly as it appeared to an agent at that time.
  • Phase 2: Controlled Replay. Once the event is resolved, researchers can replay these frozen evidence states to any LLM agent. This allows for "single-factor interventions"—changing only the model, only the search source, or only the memory strategy—while keeping everything else identical.
Figure 1: Design principles of ForecastGym.

The benchmark also implements "Replay Safeguards," such as hiding items dated later than the state and removing market prices from search snippets to prevent the model from simply "copying" the crowd's homework.

Datasets

The first release of ForecastGym includes:

  • 150 questions from real-world prediction markets.
  • 614 evidence states (snapshots in time).
  • 131,000 search results.
  • Coverage across six domains: sports, politics, economics, world, business, and science.

It differs from existing benchmarks like ForecastQA or Autocast by being "Live" (collected before resolution) and "Source-Aware" (comparing multiple search providers).

Key Insights and Ideas

  • Search APIs are the real MVP (or Villain): Switching the search API changed a model’s Brier score (a measure of forecast error) by up to 0.043. This is a larger delta than the spread between the top-performing LLMs themselves.
  • Date Filters are Broken: The authors found that rerunning queries after an event is settled—even with strict date filters—leaks the outcome for up to 92% of questions. This effectively invalidates many retrospective forecasting benchmarks that rely on post-hoc searching.
  • Retrieval is the Bottleneck: Diagnostic analysis showed that "missed public information" (the search didn't find the fact) contributed three times more to the performance gap than "misreading" (the model saw the fact but ignored it).
  • Memory Steadies the Ship: Using persistent notes (where the model summarizes key evidence over time) helps stabilize forecasts and prevents wild swings in probability, all while using roughly half the tokens of "rereading" all previous evidence.
The ForecastGym lifecycle: from live capture of evidence to controlled offline replay for agent evaluation.

Key Results

  • LLMs vs. The Market: All tested models (GPT-4o/5.6 Terra, Claude 3.5 Sonnet, Gemini 1.5/3.8 Flash, DeepSeek-V2.5/V4.1) trailed the prediction market. The market achieved a Brier score of 0.176, while the best model (GPT-5.6 Terra) scored 0.217.
  • Provider Disagreement: For the same query at the same time, any two search providers' URL sets overlapped by very little (Jaccard overlap of 0.09 to 0.18). Four out of five URLs returned were unique to a single provider.
  • Calibration vs. Resolution: Models are actually well-calibrated (if they say 70%, it happens ~70% of the time), but they lack "resolution"—the ability to distinguish between events that resolve differently as well as the market does.

Limitations and Future Work

  • Batch Size: The initial release is relatively small (150 questions).
  • Short Horizons: The questions were limited to a 14-day window; longer-term forecasting might reveal different dynamics in evidence evolution.
  • Binary Focus: The current benchmark only handles binary (Yes/No) questions.
  • Future Directions: The authors suggest that future work should focus on improving "evidence acquisition" (better query planning) and "belief revision" (how models update their internal state as new info arrives).

Summary updated

Abstract

Agents built on large language models (LLMs) increasingly forecast real-world events by searching the Web and reasoning over what they find. How well such an agent forecasts depends on both the retrieved evidence and the model, making control over that evidence essential for evaluation. Live evaluations can mix retrieval with reasoning when each system searches on its own, while retrospective search can leak later information through date filters or today's ranking of historical pages. Comparing models, search sources, and memory strategies requires both fixed evidence and a record of how it evolved before resolution. We introduce ForecastGym, which records what four commercial web search APIs return for open prediction-market questions every two days and replays this frozen evidence offline once outcomes are known, comparing models, memory, and captured search sources under a fixed evidence-acquisition policy. Its first release covers 150 questions, 614 evidence states, one per question and collection time, and 131k search results. Rerunning the same queries after settlement returns results that all but state the outcome for up to 92% of the questions, despite the tested date filters. On the frozen evidence, all tested LLMs trail the market in forecasting accuracy. Our diagnostic analysis suggests that missed public information contributes more to the gap in high-gap states than misreading does. Switching the search API changes a model's Brier score by up to 0.043, more than the spread across models, while persistent notes score similarly to rereading all evidence at half the input tokens. The evidence a forecaster sees can matter as much as the model that reads it.
BibTeX
@misc{tao2026forecastgym,
  author = {Yicheng Tao and Yuante Li and Zipei Fan},
  title = {{ForecastGym: A Live, Controlled, and Replayable Benchmark for Forecasting Agents}},
  year = {2026},
  eprint = {2611.23045},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23045},
}
Figure