Competence-Gated Pooling of Language Models and Priors for Event Forecasting

Aditi Tiwari, Aashrith Bandaru, Heng JiarXiv 2026 · arXiv

TL;DR

When deploying language models (LLMs) for event forecasting, the critical metric isn't standalone accuracy, but relative competence—the marginal value an LLM adds over an existing baseline like a prediction market or statistical prior. This paper introduces a "competence-gated pooling" method that learns when to trust LLMs versus external sources across different domains, demonstrating that historical domain performance, rather than an LLM's self-reported verbal confidence, is the key to safer, more accurate hybrid forecasts.

What Problem Does This Paper Address?

Large language models have shown surprising capabilities in predicting future events. However, in real-world forecasting systems, users rarely rely on an LLM in isolation; they already have access to high-quality external signals like prediction markets (e.g., Polymarket), crowd aggregates (e.g., Metaculus), or traditional time-series models.

The core challenge is deciding when to listen to the LLM and when to defer to the baseline. Current systems often use global pooling strategies (giving fixed weights to sources regardless of the topic) or rely on the LLM's own stated confidence. This paper tackles this gap by formalizing relative competence under Brier loss and providing a principled way to route and combine forecasts based on domain-specific historical performance.

Methods

The authors propose Competence-Gated Pooling, a framework that combines an external forecast with independent predictions from multiple LLMs.

Figure 1: Competence-gated forecasting. The system combines an external forecast with independent forecasts from several language models. Domain-specific source weights are estimated from resolved training outcomes and shrunk toward a global estimate. The weighted forecasts are then pooled and recalibrated. The gate is estimated externally and does not use the models’ self-reported confidence.

The method consists of three main steps:

  1. Domain-Specific Weight Estimation: For each domain (e.g., economics vs. geopolitics), the system estimates a source-weight vector using a closed-form affine constraint ($1^\top w = 1$) minimizing Brier loss on resolved historical outcomes.
  2. Shrinkage to Global Estimates: To prevent overfitting in small domains with few questions, domain-level weights are mathematically "shrunk" toward a robust global weight vector based on the sample size of that domain.
  3. Isotonic Recalibration: The pooled forecasts are clipped to $[0,1]$ and passed through an isotonic calibrator to ensure the final probabilities are well-calibrated.

Crucially, the LLMs generate their forecasts completely independent of the external baseline's probability, preserving their unique information.

Datasets

The authors evaluated their approach on a primary pool of 2,357 resolved binary questions sourced from ForecastBench. The dataset spans eight distinct domains/sources:

  • Prediction Markets: Polymarket, Manifold
  • Forecasting Crowds: Metaculus, INFER
  • Structured/Statistical Data: FRED (Federal Reserve Economic Data), ACLED (conflict data), Wikipedia, DBnomics

To ensure robustness, they also evaluated on a leakage-controlled subset (questions resolving after June 2025 to avoid training data contamination) and an official live-market subset.

Key Insights and Ideas

  • Disagreement is only useful if it corrects errors: Mathematically, an LLM adds value to an external forecast only when its disagreement correlates with the baseline's errors (cross-source error correction). If the baseline is already strong and well-calibrated, the LLM's marginal value drops to zero.
  • LLM verbal confidence is an unreliable gatekeeper: One of the most striking findings is that an LLM's self-reported confidence ("I am 80% sure this will happen") does not reliably predict whether it will outperform an external baseline.
  • Smart deferral beats blind blending: Instead of trying to force the LLM to improve a highly efficient prediction market, a competent system learns to simply defer to the market while aggressively leveraging the LLM in data-sparse domains where the baseline (like a simple statistical prior) is weak.

Key Results

  • Main Pool Improvement: On the primary dataset, the competence gate significantly improved the main external baseline's Brier score from 0.0771 to 0.0732 ($p = 0.001$), outperforming global mixture baselines.
  • Regime Deferral: On the highly efficient live market subset, the competence gate recognized the strength of the baseline and largely deferred to it, yielding no significant change over the market forecast (retaining a Brier score around 0.076-0.077).
Figure 2: Model value depends on the strength of the external forecast. (a) On the main pool, the competence gate improves the main external baseline from 0.0771 to 0.0732 Brier, with p = 0.001. (b) On the live-market subset, neither the global mixture (p = 0.19) nor the competence gate (p = 0.42) significantly improves over the market forecast. This result is consistent with deferral when the available external forecast is already strong.
  • Superior Abstention: When using the competence gate to decide when to abstain from forecasting, it achieved a drastically lower risk-coverage area under the curve (0.019) compared to relying on the LLM's verbal confidence (0.054), enabling much safer selective prediction.

Limitations and Future Work

  • Coarse Domain Labels: The current framework uses the source (e.g., FRED or Polymarket) as a proxy for the domain. Future work could explore finer, question-level routing features like forecast horizon, market liquidity, or explicit text embeddings.
  • Cold-Start Problem: Because the competence gate relies on historically resolved outcomes to estimate weights, transferring the system to entirely unseen domains remains a challenge.
  • Operational Costs: The experiments isolated forecasting accuracy and did not factor in real-world deployment constraints such as API token costs, latency, or retrieval-augmented generation (RAG) overhead.

Summary updated

Abstract

In hybrid forecasting, a language model is often one of several available signals. A system may already have a market, crowd, or statistical forecast and must decide whether the model adds useful information or should be ignored. The relevant target is therefore not standalone model accuracy, but relative competence, defined as the model's marginal value beyond the available external forecast. Under Brier loss, we characterize when model disagreement can improve an external forecast and derive the gain from using domain-specific rather than global pooling weights. We then introduce a competence gate that estimates domain-level source weights from resolved outcomes, shrinks uncertain estimates toward a global weight, and recalibrates the pooled forecast. Across 2,357 resolved binary questions and five language models, the gate improves the main external baseline from 0.0771 to 0.0732 Brier and significantly outperforms global forecast combinations. The gain remains significant under leakage controls and against a leakage-safe time-series prior on the pooled structured set, with separate evidence on FRED. In contrast, the gate gives no significant improvement on the official ForecastBench market subset, where it largely defers to the market. Across four Qwen models, verbal confidence does not reliably identify when the model outperforms the external forecast, while outcome-estimated competence supports better abstention decisions. These results provide a practical approach for selective model use based on measured marginal value.
BibTeX
@misc{tiwari2026competence,
  author = {Aditi Tiwari and Aashrith Bandaru and Heng Ji},
  title = {{Competence-Gated Pooling of Language Models and Priors for Event Forecasting}},
  year = {2026},
  eprint = {2609.12101},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.12101},
}
Figure