LEAP: Likelihood Elicitation and Aggregation for LLM-based Probabilistic Forecasting

Yufei Chen, Yiran Zhao, Xiaogang Xu, Qipeng Xie, Jiafei Wu, Zhe LiuarXiv 2026 · arXiv

TL;DR

When using Large Language Models (LLMs) for forecasting, standard systems dump all gathered evidence into a single prompt, often resulting in uncalibrated, "confidently wrong" predictions. LEAP solves this by forcing the LLM to evaluate each piece of evidence independently to extract local likelihood parameters, which a deterministic Bayesian model then aggregates into a well-calibrated posterior distribution. This modular approach drastically reduces overconfidence and makes the final forecast fully auditable down to individual sources.

What Problem Does This Paper Address?

State-of-the-art LLM forecasting agents typically operate in two stages: first, they search the web to gather a bundle of evidence, and second, they feed all that text into an LLM to generate a final prediction. The authors call this second stage Monolithic Prediction.

Monolithic prediction suffers from several critical flaws:

  1. Opacity: It acts as a black box. It is impossible to isolate exactly how much a specific news article or data point influenced the final forecast.
  2. Information Loss & Hallucination: Shoving massive amounts of text into a long-context prompt makes the LLM susceptible to "lost-in-the-middle" forgetting and hallucination.
  3. Uncertainty Collapse: Monolithic models tend to prematurely commit to a single outcome, outputting highly confident but incorrect point estimates instead of nuanced probabilistic distributions.
The Monolithic baseline can be confidently wrong and hard to audit, whereas LEAP provides an auditable, calibrated probability distribution.

Methods

LEAP (Likelihood Elicitation and Aggregation for Probabilistic forecasting) replaces the monolithic prediction step with a structured, three-step pipeline:

  1. Parameter Elicitation: Instead of seeing all evidence at once, the LLM acts as a local interpreter. It reviews the forecasting question alongside exactly one piece of evidence at a time. It outputs structured qualitative support/opposition labels (for discrete tasks) or target-scale observations (for continuous tasks), along with a source dependency key. Separately, a prior distribution (base rate) is established using historical data or a blind LLM call.
  2. Engineering Safeguards: Before math takes over, LEAP applies three algorithmic safeguards:
  3. Dependency Clustering: Uses the dependency keys to group common-source articles so the same rumor or report isn't double-counted.
  4. Reliability Sampling: Repeats the local LLM query multiple times to shrink the weight of noisy or inconsistent interpretations.
  5. Outlier Rejection: Discards continuous evidence that deviates wildly (more than 4 standard deviations) from the data-derived prior.
  6. Deterministic Aggregation: A closed-form Bayesian update combines the prior and the elicited likelihoods into a final posterior distribution. Crucially, no LLM is used during aggregation.

Because the aggregation is purely mathematical, LEAP can compute exact leave-one-out contributions ($\Delta_j$) for each piece of evidence, showing users precisely how much each source shifted the final forecast.

Figure 2: Overview of LEAP. After evidence collection, isolated LLM calls estimate a prior and one likelihood per evidence item. LEAP applies safeguards, performs a closed-form Bayesian update, and reports the posterior forecast with leave-one-out contributions ∆j.

Datasets

To evaluate LEAP across diverse tasks, the authors constructed a 347-task evaluation manifest by adapting three existing benchmarks:

  • FutureX: A live benchmark for real-world future prediction.
  • GAIA: A general AI assistant benchmark used here for complex information-seeking tasks.
  • BrowseComp: A web-browsing benchmark.

The final dataset spans three target types: single_choice (266 tasks), multi_choice (29 tasks), and continuous numeric targets (52 tasks). A separate 60-task stratified subset was reserved for detailed diagnostic and ablation analyses.

Key Insights and Ideas

  • Divide and Conquer for Calibration: Delegating text interpretation to the LLM but leaving the probability math to a deterministic Bayesian framework prevents the LLM from collapsing uncertainty.
  • Graceful Degradation over Time: As the forecast horizon lengthens (e.g., predicting an event 60 days out vs. 7 days out), monolithic models remain stubbornly overconfident. LEAP, however, naturally recognizes the weaker, more indirect support in the evidence and broadens its posterior distribution, degrading gracefully by becoming conservative rather than confidently wrong.
Figure 3: Absolute improvement of LEAP over Mono- lithic on the diagnostic subset as a function of the fore- cast horizon (days between evidence collection and ground truth resolution). All four reported metrics move further in LEAP’s favour as the horizon lengthens.
  • Priors Matter Most: Ablation studies revealed that the prior is the single most critical component of the system. Without a stable statistical starting point, structured aggregation struggles when local evidence is sparse or noisy.

Key Results

The authors evaluated LEAP across five prominent base models (including DeepSeek-V3.2, Gemini-3.1-Flash-Lite, Claude-Haiku-4.5, GPT-5.4-mini, and Grok-4.20-Fast) and four external agent frameworks (DeerFlow, Hermes, OpenClaw, and MiroFlow).

  • Consistent Performance Boosts: LEAP improved the headline FutureX composite score and accuracy across almost all evaluated models and frameworks. For instance, macro-averaged across external frameworks, LEAP boosted the FutureX score from 0.4491 to 0.5471.
  • Halving Calibration Error: On the diagnostic subset, LEAP cut Expected Calibration Error (ECE) by more than half (dropping from 0.1840 to 0.0876) and slashed overconfidence metrics from 0.3167 to 0.1500.

Limitations and Future Work

While LEAP significantly improves the prediction step, it is entirely dependent on the quality of the upstream collection phase; it cannot correct for sparse, outdated, or fundamentally biased evidence retrieval. Additionally, because LEAP processes each piece of evidence individually and utilizes reliability sampling, it requires substantially more LLM inference calls and exhibits higher $p95$ latency than monolithic baselines.

Future work includes extending LEAP to non-English forecasting tasks, specialized scientific domains, and integrating it directly into iterative, closed-loop agent search behaviors.

Summary updated

Abstract

LLM-based forecasting systems have improved on real-world tasks such as financial markets and sports outcomes, largely through stronger search and tool use. Many systems still ask an LLM to read all collected evidence together and produce the final forecast. We call this design Monolithic Prediction. It can obscure how individual evidence items affect the result and collapse uncertainty across competing outcomes. We propose LEAP (Likelihood Elicitation and Aggregation for Probabilistic forecasting), which reorganizes how collected evidence is used in the prediction stage. LEAP examines each evidence item separately and elicits likelihood parameters that describe its implications for the target. An explicit prior and a deterministic probabilistic model then combine these likelihoods into a posterior distribution. This procedure supports continuous, single-choice, and multi-choice forecasts while preserving reproducible evidence contributions. We build a benchmark covering forecasting, information-seeking, and browsing tasks, and evaluate LEAP on our own agent loop and several agent CLI frameworks. Given the same evidence, LEAP improves most prediction and calibration metrics across models and remains stronger under controlled comparisons of prior access, inference budget, and aggregation.
BibTeX
@misc{chen2026leap,
  author = {Yufei Chen and Yiran Zhao and Xiaogang Xu and Qipeng Xie and Jiafei Wu and Zhe Liu},
  title = {{LEAP: Likelihood Elicitation and Aggregation for LLM-based Probabilistic Forecasting}},
  year = {2026},
  eprint = {2609.01337},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.01337},
}
Figure