TL;DR
The paper introduces EVOLVECAST, a framework designed to evaluate how LLMs revise their forecasts when presented with new, post-training information. The researchers found that while modern LLMs (like DeepSeek-R1) can often identify the correct direction of a forecast update, they are consistently under-responsive in magnitude and remain poorly calibrated compared to human crowd-forecasters.
What Problem Does This Paper Address?
Traditional LLM evaluation often treats forecasting as a static, one-shot task: "Will X happen?" However, real-world forecasting is a dynamic process of belief revision. As new evidence emerges, a rational agent should update their "prior" belief to a "posterior" probability.
The authors identify a critical gap: current benchmarks focus on final accuracy rather than the reasoning process of updating. They argue that simply retrieving news (RAG) isn't enough; models must treat that news as evidence that shifts a probability distribution. The paper investigates whether LLMs act like "Bayesian agents" or if they are anchored too heavily to their pre-training data.
Methods
The authors formalize the task as a dynamic update: given an initial belief ($p_0$) and new information ($x_t$), the model must produce a revised belief ($p_t$).
Key methodological components include:
- Two Confidence Lenses:
- Black-box (Verbalized): The model explicitly states a probability on a 1–10 scale.
- White-box (Logit-based): Confidence is derived from the average token probabilities of the model's output.
- Dynamic Anchoring: To prevent "hindsight bias," the researchers restricted the benchmark to questions posted after the models' training cutoffs (October 2023) and explicitly instructed models to ignore any information released after a specific "today" timestamp.
- Evaluation Metrics: They used Mean Directional Accuracy (MDA) to check if the model moved in the same direction as humans, and $\Delta$Brier scores to measure if the model's calibration improved or worsened after seeing new info.
Datasets
- Metaculus: The primary source for "ground truth." Metaculus is a crowd-forecasting platform where aggregate human predictions serve as a proxy for the rational latent probability of an event.
- EVOLVECAST Benchmark: A newly curated dataset of 1,613 instances derived from 203 unique binary (Yes/No) questions.
- News Retrieval: For every human forecast shift on Metaculus, the authors used the Google Search API to retrieve the most relevant news articles from that specific timeframe to present to the LLMs.
Key Insights and Ideas
- The "Anchoring" Problem: LLMs suffer from a massive "pre-training anchor." They treat new information as mere text to be summarized in context rather than a signal to fundamentally shift their internal probability distribution.
- Qualitative vs. Quantitative Gap: Models are much better at Directional QA (simply saying if a probability should go "Up" or "Down") than they are at actually adjusting the numbers. For example, the DeepSeek-R1 LLaMA-8B model jumped from 0.34 MDA in probabilistic mode to 0.49 MDA when asked for a simple directional label.
- The Accumulation Paradox: Counter-intuitively, providing a model with a full history of news updates (Accumulated) often degraded performance compared to just providing the latest update (Single). The models seem to treat all updates with equal salience, leading to "diluted" adjustments and accumulated noise.
- Reasoning Models Help, But Aren't a Cure: While "reasoning-tuned" models (like the R1 series) outperformed base models, the performance gap between verbalized and logit-based confidence remained marginal and inconsistent.
Key Results
- Under-responsiveness: In a typical scenario (see Figure 1), if humans shifted a forecast by +15% based on news, the LLM might only shift by +5%.
- Poor Calibration: Most models showed a positive $\Delta$Brier score, indicating that their alignment with the human reference standard actually worsened or failed to improve after processing new information.
- Model Performance: The DeepSeek-R1-Distill-Qwen-7B was a top performer in directional agreement (0.3534 MDA for verbalized confidence), but even it struggled with magnitude alignment (0.0232 MSE).
- Human Reference as Context: Even when the model was explicitly told what the current human "market consensus" was, it failed to use that anchor to produce better-calibrated updates.
Limitations and Future Work
- Binary Limitation: The study focused exclusively on binary (Yes/No) questions. Future work is needed to see if these findings hold for continuous variables (e.g., "What will the inflation rate be?").
- Retrieval Dependency: The framework relies on the Google Search API; since search results change over time, exact replication of the retrieval stage is difficult (though the authors released their specific retrieved snippets).
- The "Why" of Anchoring: The paper identifies that models are anchored to pre-training data but doesn't provide a training-level solution to break that anchor. The authors suggest that future research should focus on architectural mechanisms that allow external knowledge to more "forcefully" update the model's posterior beliefs.
Summary updated