TL;DR
Measuring how LLM forecasts improve with new evidence (the "update slope") is a popular but deeply flawed metric. This paper demonstrates that raw update slopes are inflated by up to 4x when evidence is constructed by someone who knows the outcome and are mathematically constrained by how accurate the model was to begin with. By "blinding" the evidence construction and adjusting for starting calibration, researchers can recover a much more robust measure of a model's true updating skill.
What Problem Does This Paper Address?
As Large Language Models (LLMs) are increasingly used for forecasting, researchers want to measure more than just their final accuracy. They want to know if models are "good Bayesians"—that is, do they update their beliefs appropriately as new evidence arrives?
The standard tool for this is the evidence ladder: a sequence of prompts for a single question that provides increasing levels of context (from no context to a full briefing). The "update slope" measures the rate at which the model's forecast moves toward the truth across these levels.
However, the authors identify two major "confounds" that make raw update slopes untrustworthy:
- Construction Leakage: If the person (or AI) building the evidence ladder knows the eventual outcome, they may inadvertently select or emphasize facts that point toward the correct answer, even without explicitly stating it.
- The Structural Ceiling: Update slope is a "change score." If a model is already very accurate at the start (high prior calibration), it has very little "headroom" to improve. Conversely, a model that starts with a wild guess has plenty of room for a steep slope. Comparing these raw slopes is like comparing the improvement of a straight-A student to a failing student; the latter has much more room for a "steep" improvement curve regardless of actual learning ability.
Methods
The authors introduce several methodological corrections to fix these issues:
- Blinded Evidence Ladders: They compare "outcome-aware" ladders (where the builder knows the result) against "blinded" ladders (where the builder is denied the result).
- Proposition 1 (The Prior Bound): They mathematically derive a tight structural bound showing that the maximum possible slope is strictly limited by the initial forecast.
- Headroom Utilization ($u_m$): They propose a new metric that decomposes the slope into "available headroom" (how much room there is to improve) and "utilization" (how much of that room the model actually used).
- Prior-Conditioned Analysis: They use hierarchical mixed-effects models to "control" for the starting Brier score (a measure of forecast error), allowing them to isolate the effect of updating skill from starting accuracy.
Datasets
The study evaluates a panel of 7 frontier LLMs (including models like GPT-5, o3, and Claude 3.5 variants) across three distinct question pools:
- Manifold Markets: 40 resolved binary questions from a prediction market (primary pool).
- ForecastBench: A dynamic benchmark of AI forecasting capabilities.
- ACLED: Templated event counts related to political violence and protest.
Key Insights and Ideas
- The Invisible Leak: Even when an audit finds no explicit "hindsight markers" (like dates or spoilers), outcome-aware evidence construction still inflates slopes by roughly 4x. This suggests that "leakage" is a subtle property of how evidence is curated, not just a matter of accidentally revealing the answer.
- The Ceiling is Real: You cannot have a steep update slope if you are already right. This "mathematical coupling" means that raw rankings of models by update slope are often just rankings of how poorly they started.
- Question Identity Dominates: In their variance decomposition, the authors found that 85.2% of the variance in performance was due to the specific question being asked, while only 0.5% was due to the model identity. This suggests that current forecasting benchmarks are highly sensitive to "luck of the draw" in question selection.
Key Results
The paper's quantitative findings provide a stark warning for LLM evaluators:
-
Slope Inflation: As seen in Figure 1, roughly 72% of the apparent "evidence effect" disappears when the ladder builder is blinded to the outcome. Fig.
-
Aggregate dose response under the two ladder constructions. The outcome-aware ladder produces roughly 3.5x the movement of the blinded ladder.
- Tripled Association: Once you condition on prior calibration, the statistical association between intermediate updating and final forecasting skill triples. This means that "updating skill" is a real, measurable trait, but it's hidden behind the noise of the starting point unless you control for it.
- Metric Performance: Table II shows that raw slopes ($\bar{\beta}_m$) have almost no correlation with final skill (Kendall $\tau = 0.143$). However, their proposed "resolution residual" ($DRR^\rho_m$)—which adjusts for the prior—reaches a much more meaningful $\tau = 0.714$. Table II showing Kendall tau between candidate metrics and final skill.

Limitations and Future Work
The authors acknowledge several limitations:
- Sample Size: The study uses a relatively small panel of 7 models and 40 questions per pool, making the cross-model correlations "descriptive" rather than definitive.
- Single Builder Family: The evidence ladders were built using a single model family, which might introduce specific biases in how evidence is presented.
- Contamination: While they used questions resolving after training cutoffs, they cannot entirely rule out "residual contamination" from public discussions of these topics.
Future Work: The authors suggest that future forecasting benchmarks must report question-stratified scores and headroom-normalized measures ($u_m$) to provide a fair "leaderboard" for model updating capabilities.
Summary updated