Modeling Probability Forecasts via Information Diversity

Ville Satopää, Robin Pemantle, Lyle Ungar

TL;DR

This paper moves beyond treating expert forecasters as "noisy sensors" of a single true probability. Instead, it introduces a Partial Information Framework where forecast differences arise from experts seeing different, partially overlapping "particles" of information, providing a rigorous theoretical justification for the common empirical practice of "extremizing" (pushing) average forecasts toward 0 or 1.

What Problem Does This Paper Address?

When we want to predict the probability of a binary event (e.g., "Will a banking union be approved in the EU?"), we often ask several experts and average their answers. However, simple averages are notoriously "under-confident"—they tend to stay too close to 0.5.

To fix this, practitioners often use an ad hoc technique called extremizing: they take the average and manually push it closer to the nearest extreme (0 or 1). While this works well in practice, it lacks a solid theoretical foundation. Existing models usually rely on the "Measurement Error Framework," which assumes there is a "true" objective probability and experts just add random noise to it. The authors argue this is philosophically flawed and fails to explain why extremizing is necessary.

Methods

The authors propose the Gaussian Partial Information Model. Here’s the intuition:

  1. Information Particles: Imagine the total information available for an event is a pool of "particles" on a unit interval. Some particles favor the event happening; others favor it not happening.
  2. Expert Observations: Each expert observes a specific subset (a $\sigma$-field) of these particles. Their forecast is the expected value of the event given their specific subset of information.
  3. The Gaussian Link: By modeling these information sets using a Gaussian process (specifically, Brownian motion variation), the authors can describe the relationship between experts using a covariance matrix.
  4. The Aggregators:
    • Oracular Aggregator ($p'$): The "gold standard" forecast if you had access to the union of all experts' information.
  5. Revealed Aggregator ($p''$): The optimal forecast you can make using only the probabilities reported by the experts.

The model uses two key parameters: $\delta$ (the amount of information an individual expert has) and $\lambda$ (the proportion of information overlap between experts).

Datasets

The model was evaluated using data from the Good Judgment Project (GJP), a massive forecasting tournament sponsored by IARPA (ACE program).

  • Scope: The study focused on "super-forecasters" in the second year of the tournament.
  • Data Points: 44 super-forecasters making predictions on 123 global events (e.g., troop withdrawals, political elections).
  • Pre-processing: To avoid infinite scores in log-odds space, extreme forecasts of 0 and 1 were censored to 0.001 and 0.999.

Key Insights and Ideas

  • Extremizing is Rational: The paper proves that if experts have different information, the "Oracular" aggregate is mathematically more likely to be more extreme than the average of individual forecasts. Extremizing isn't a "hack"—it's the correct Bayesian response to information diversity.
  • The Spectrum of Aggregation: The optimal way to combine forecasts exists on a spectrum. If experts have complete overlap (they see the same things), you should simply average them. If they have zero overlap (they see entirely different things), you should use a "voting" style aggregation that results in very extreme forecasts.
  • Diversity vs. Expertise: Both the total amount of information ($\delta$) and the diversity of that information (low $\lambda$) drive the need for extremizing. If you have many diverse experts, you should push the average further toward the extremes.
  • Calibration vs. Confidence: The model shows that even if every individual expert is perfectly "calibrated" (their 70% predictions come true 70% of the time), their simple average will be uncalibrated and under-confident.

Key Results

The authors compared their Symmetric Revealed Aggregator ($p''_{cs}$) against standard methods using the Brier Score (where lower is better).

Aggregator Brier Score (BS) Reliability (REL) Resolution (RES)
Simple Average ($\bar{p}$) 0.132 0.026 0.045
Log-odds Average ($p_{log}$) 0.128 0.025 0.048
Probit Average ($p_{probit}$) 0.128 0.023 0.047
Revealed Aggregator ($p''_{cs}$) 0.123 0.020 0.049

The $p''_{cs}$ model achieved the lowest Brier Score because it was both the most resolved (confident) and the most calibrated (reliable).

Limitations and Future Work

  • Symmetry Assumption: The specific aggregator tested ($p''_{cs}$) assumes "exchangeability"—that all experts are roughly equal in expertise and overlap. In reality, some experts are much better than others.
  • Parameter Estimation: Estimating the exact information overlap ($\lambda$) between experts is difficult with small datasets. The authors suggest a Bayesian approach might help in the future.
  • High-Order Intersections: The current model focuses on pairwise overlaps. The authors note that understanding how information is shared across three or more experts (higher-order intersections) remains a complex theoretical challenge.
  • Static vs. Dynamic: The model doesn't currently account for how forecasts change over time as new information arrives.

Summary updated

Abstract

Randomness in scientific estimation is generally assumed to arise from unmeasured or uncontrolled factors. However, when combining subjective probability estimates, heterogeneity stemming from people's cognitive or information diversity is often more important than measurement noise. This paper presents a novel framework that models the heterogeneity arising from experts that use partially overlapping information sources, and applies that model to the task of aggregating the probabilities given by a group of experts who forecast whether an event will occur or not. Our model describes the distribution of information across experts in terms of easily interpretable parameters and shows how the optimal amount of extremizing of the average probability forecast (shifting it closer to its nearest extreme) varies as a function of the experts' information overlap. Our model thus gives a more principled understanding of the historically ad hoc practice of extremizing average forecasts.
BibTeX
@article{satopaa2015modeling,
  author = {Ville Satopää and Robin Pemantle and Lyle Ungar},
  title = {{Modeling Probability Forecasts via Information Diversity}},
  journal = {Journal of the American Statistical Association},
  year = {2015},
  doi = {10.1080/01621459.2015.1100621},
}

Loading related content...

Figure