TL;DR
ForecastGym is a novel benchmark for LLM-based forecasting agents that solves the "information leakage" problem by prospectively recording web search results while questions are still open. The study reveals a surprising truth: the choice of search API can impact a model's accuracy more than the choice of the language model itself, and current LLMs still significantly trail human prediction markets.
What Problem Does This Paper Address?
Evaluating AI agents that forecast real-world events is notoriously difficult due to four main challenges:
- Leaky Reconstruction: Retrospective evaluations (searching for info after an event has happened) often leak the outcome. Even with date filters, search engines might return pages updated after the fact or rank historical pages based on today’s knowledge.
- Entangled Retrieval and Reasoning: In live benchmarks, it’s hard to tell if a bad forecast is due to a poor model (reasoning) or a poor search tool (retrieval).
- Uncontrolled Evidence Dynamics: Information evolves as an event approaches. To compare how agents revise their beliefs, they must experience the exact same "trajectory" of incoming evidence.
- Search-Source Variation: Different search providers (e.g., Google vs. Tavily) return different results, but their specific impact on forecasting hasn't been isolated from model performance.
Methods
The authors introduce a two-phase lifecycle for the benchmark:
- Phase 1: Live Capture. While a prediction market question (from platforms like Polymarket or Kalshi) is still open, ForecastGym queries four commercial search APIs (Tavily, Exa, Serper, and Parallel) every two days. This "freezes" the evidence exactly as it appeared to an agent at that time.
- Phase 2: Controlled Replay. Once the event is resolved, researchers can replay these frozen evidence states to any LLM agent. This allows for "single-factor interventions"—changing only the model, only the search source, or only the memory strategy—while keeping everything else identical.
The benchmark also implements "Replay Safeguards," such as hiding items dated later than the state and removing market prices from search snippets to prevent the model from simply "copying" the crowd's homework.
Datasets
The first release of ForecastGym includes:
- 150 questions from real-world prediction markets.
- 614 evidence states (snapshots in time).
- 131,000 search results.
- Coverage across six domains: sports, politics, economics, world, business, and science.
It differs from existing benchmarks like ForecastQA or Autocast by being "Live" (collected before resolution) and "Source-Aware" (comparing multiple search providers).
Key Insights and Ideas
- Search APIs are the real MVP (or Villain): Switching the search API changed a model’s Brier score (a measure of forecast error) by up to 0.043. This is a larger delta than the spread between the top-performing LLMs themselves.
- Date Filters are Broken: The authors found that rerunning queries after an event is settled—even with strict date filters—leaks the outcome for up to 92% of questions. This effectively invalidates many retrospective forecasting benchmarks that rely on post-hoc searching.
- Retrieval is the Bottleneck: Diagnostic analysis showed that "missed public information" (the search didn't find the fact) contributed three times more to the performance gap than "misreading" (the model saw the fact but ignored it).
- Memory Steadies the Ship: Using persistent notes (where the model summarizes key evidence over time) helps stabilize forecasts and prevents wild swings in probability, all while using roughly half the tokens of "rereading" all previous evidence.
Key Results
- LLMs vs. The Market: All tested models (GPT-4o/5.6 Terra, Claude 3.5 Sonnet, Gemini 1.5/3.8 Flash, DeepSeek-V2.5/V4.1) trailed the prediction market. The market achieved a Brier score of 0.176, while the best model (GPT-5.6 Terra) scored 0.217.
- Provider Disagreement: For the same query at the same time, any two search providers' URL sets overlapped by very little (Jaccard overlap of 0.09 to 0.18). Four out of five URLs returned were unique to a single provider.
- Calibration vs. Resolution: Models are actually well-calibrated (if they say 70%, it happens ~70% of the time), but they lack "resolution"—the ability to distinguish between events that resolve differently as well as the market does.
Limitations and Future Work
- Batch Size: The initial release is relatively small (150 questions).
- Short Horizons: The questions were limited to a 14-day window; longer-term forecasting might reveal different dynamics in evidence evolution.
- Binary Focus: The current benchmark only handles binary (Yes/No) questions.
- Future Directions: The authors suggest that future work should focus on improving "evidence acquisition" (better query planning) and "belief revision" (how models update their internal state as new info arrives).
Summary updated