TL;DR
When using Large Language Models (LLMs) for forecasting, standard systems dump all gathered evidence into a single prompt, often resulting in uncalibrated, "confidently wrong" predictions. LEAP solves this by forcing the LLM to evaluate each piece of evidence independently to extract local likelihood parameters, which a deterministic Bayesian model then aggregates into a well-calibrated posterior distribution. This modular approach drastically reduces overconfidence and makes the final forecast fully auditable down to individual sources.
What Problem Does This Paper Address?
State-of-the-art LLM forecasting agents typically operate in two stages: first, they search the web to gather a bundle of evidence, and second, they feed all that text into an LLM to generate a final prediction. The authors call this second stage Monolithic Prediction.
Monolithic prediction suffers from several critical flaws:
- Opacity: It acts as a black box. It is impossible to isolate exactly how much a specific news article or data point influenced the final forecast.
- Information Loss & Hallucination: Shoving massive amounts of text into a long-context prompt makes the LLM susceptible to "lost-in-the-middle" forgetting and hallucination.
- Uncertainty Collapse: Monolithic models tend to prematurely commit to a single outcome, outputting highly confident but incorrect point estimates instead of nuanced probabilistic distributions.
Methods
LEAP (Likelihood Elicitation and Aggregation for Probabilistic forecasting) replaces the monolithic prediction step with a structured, three-step pipeline:
- Parameter Elicitation: Instead of seeing all evidence at once, the LLM acts as a local interpreter. It reviews the forecasting question alongside exactly one piece of evidence at a time. It outputs structured qualitative support/opposition labels (for discrete tasks) or target-scale observations (for continuous tasks), along with a source dependency key. Separately, a prior distribution (base rate) is established using historical data or a blind LLM call.
- Engineering Safeguards: Before math takes over, LEAP applies three algorithmic safeguards:
- Dependency Clustering: Uses the dependency keys to group common-source articles so the same rumor or report isn't double-counted.
- Reliability Sampling: Repeats the local LLM query multiple times to shrink the weight of noisy or inconsistent interpretations.
- Outlier Rejection: Discards continuous evidence that deviates wildly (more than 4 standard deviations) from the data-derived prior.
- Deterministic Aggregation: A closed-form Bayesian update combines the prior and the elicited likelihoods into a final posterior distribution. Crucially, no LLM is used during aggregation.
Because the aggregation is purely mathematical, LEAP can compute exact leave-one-out contributions ($\Delta_j$) for each piece of evidence, showing users precisely how much each source shifted the final forecast.
Datasets
To evaluate LEAP across diverse tasks, the authors constructed a 347-task evaluation manifest by adapting three existing benchmarks:
- FutureX: A live benchmark for real-world future prediction.
- GAIA: A general AI assistant benchmark used here for complex information-seeking tasks.
- BrowseComp: A web-browsing benchmark.
The final dataset spans three target types: single_choice (266 tasks), multi_choice (29 tasks), and continuous numeric targets (52 tasks). A separate 60-task stratified subset was reserved for detailed diagnostic and ablation analyses.
Key Insights and Ideas
- Divide and Conquer for Calibration: Delegating text interpretation to the LLM but leaving the probability math to a deterministic Bayesian framework prevents the LLM from collapsing uncertainty.
- Graceful Degradation over Time: As the forecast horizon lengthens (e.g., predicting an event 60 days out vs. 7 days out), monolithic models remain stubbornly overconfident. LEAP, however, naturally recognizes the weaker, more indirect support in the evidence and broadens its posterior distribution, degrading gracefully by becoming conservative rather than confidently wrong.
- Priors Matter Most: Ablation studies revealed that the prior is the single most critical component of the system. Without a stable statistical starting point, structured aggregation struggles when local evidence is sparse or noisy.
Key Results
The authors evaluated LEAP across five prominent base models (including DeepSeek-V3.2, Gemini-3.1-Flash-Lite, Claude-Haiku-4.5, GPT-5.4-mini, and Grok-4.20-Fast) and four external agent frameworks (DeerFlow, Hermes, OpenClaw, and MiroFlow).
- Consistent Performance Boosts: LEAP improved the headline FutureX composite score and accuracy across almost all evaluated models and frameworks. For instance, macro-averaged across external frameworks, LEAP boosted the FutureX score from 0.4491 to 0.5471.
- Halving Calibration Error: On the diagnostic subset, LEAP cut Expected Calibration Error (ECE) by more than half (dropping from 0.1840 to 0.0876) and slashed overconfidence metrics from 0.3167 to 0.1500.
Limitations and Future Work
While LEAP significantly improves the prediction step, it is entirely dependent on the quality of the upstream collection phase; it cannot correct for sparse, outdated, or fundamentally biased evidence retrieval. Additionally, because LEAP processes each piece of evidence individually and utilizes reliability sampling, it requires substantially more LLM inference calls and exhibits higher $p95$ latency than monolithic baselines.
Future work includes extending LEAP to non-English forecasting tasks, specialized scientific domains, and integrating it directly into iterative, closed-loop agent search behaviors.
Summary updated