TL;DR
When deploying language models (LLMs) for event forecasting, the critical metric isn't standalone accuracy, but relative competence—the marginal value an LLM adds over an existing baseline like a prediction market or statistical prior. This paper introduces a "competence-gated pooling" method that learns when to trust LLMs versus external sources across different domains, demonstrating that historical domain performance, rather than an LLM's self-reported verbal confidence, is the key to safer, more accurate hybrid forecasts.
What Problem Does This Paper Address?
Large language models have shown surprising capabilities in predicting future events. However, in real-world forecasting systems, users rarely rely on an LLM in isolation; they already have access to high-quality external signals like prediction markets (e.g., Polymarket), crowd aggregates (e.g., Metaculus), or traditional time-series models.
The core challenge is deciding when to listen to the LLM and when to defer to the baseline. Current systems often use global pooling strategies (giving fixed weights to sources regardless of the topic) or rely on the LLM's own stated confidence. This paper tackles this gap by formalizing relative competence under Brier loss and providing a principled way to route and combine forecasts based on domain-specific historical performance.
Methods
The authors propose Competence-Gated Pooling, a framework that combines an external forecast with independent predictions from multiple LLMs.
The method consists of three main steps:
- Domain-Specific Weight Estimation: For each domain (e.g., economics vs. geopolitics), the system estimates a source-weight vector using a closed-form affine constraint ($1^\top w = 1$) minimizing Brier loss on resolved historical outcomes.
- Shrinkage to Global Estimates: To prevent overfitting in small domains with few questions, domain-level weights are mathematically "shrunk" toward a robust global weight vector based on the sample size of that domain.
- Isotonic Recalibration: The pooled forecasts are clipped to $[0,1]$ and passed through an isotonic calibrator to ensure the final probabilities are well-calibrated.
Crucially, the LLMs generate their forecasts completely independent of the external baseline's probability, preserving their unique information.
Datasets
The authors evaluated their approach on a primary pool of 2,357 resolved binary questions sourced from ForecastBench. The dataset spans eight distinct domains/sources:
- Prediction Markets: Polymarket, Manifold
- Forecasting Crowds: Metaculus, INFER
- Structured/Statistical Data: FRED (Federal Reserve Economic Data), ACLED (conflict data), Wikipedia, DBnomics
To ensure robustness, they also evaluated on a leakage-controlled subset (questions resolving after June 2025 to avoid training data contamination) and an official live-market subset.
Key Insights and Ideas
- Disagreement is only useful if it corrects errors: Mathematically, an LLM adds value to an external forecast only when its disagreement correlates with the baseline's errors (cross-source error correction). If the baseline is already strong and well-calibrated, the LLM's marginal value drops to zero.
- LLM verbal confidence is an unreliable gatekeeper: One of the most striking findings is that an LLM's self-reported confidence ("I am 80% sure this will happen") does not reliably predict whether it will outperform an external baseline.
- Smart deferral beats blind blending: Instead of trying to force the LLM to improve a highly efficient prediction market, a competent system learns to simply defer to the market while aggressively leveraging the LLM in data-sparse domains where the baseline (like a simple statistical prior) is weak.
Key Results
- Main Pool Improvement: On the primary dataset, the competence gate significantly improved the main external baseline's Brier score from 0.0771 to 0.0732 ($p = 0.001$), outperforming global mixture baselines.
- Regime Deferral: On the highly efficient live market subset, the competence gate recognized the strength of the baseline and largely deferred to it, yielding no significant change over the market forecast (retaining a Brier score around 0.076-0.077).
- Superior Abstention: When using the competence gate to decide when to abstain from forecasting, it achieved a drastically lower risk-coverage area under the curve (0.019) compared to relying on the LLM's verbal confidence (0.054), enabling much safer selective prediction.
Limitations and Future Work
- Coarse Domain Labels: The current framework uses the source (e.g., FRED or Polymarket) as a proxy for the domain. Future work could explore finer, question-level routing features like forecast horizon, market liquidity, or explicit text embeddings.
- Cold-Start Problem: Because the competence gate relies on historically resolved outcomes to estimate weights, transferring the system to entirely unseen domains remains a challenge.
- Operational Costs: The experiments isolated forecasting accuracy and did not factor in real-world deployment constraints such as API token costs, latency, or retrieval-augmented generation (RAG) overhead.
Summary updated