Probability aggregation in time-series: Dynamic hierarchical modeling of sparse expert beliefs

Ville Satopää, Shane T. Jensen, Barbara A. Mellers, Philip E. Tetlock, Lyle UngarThe Annals of Applied Statistics 2014 · DOI · arXiv

Abstract

Most subjective probability aggregation procedures use a single probability judgment from each expert, even though it is common for experts studying real problems to update their probability estimates over time. This paper advances into unexplored areas of probability aggregation by considering a dynamic context in which experts can update their beliefs at random intervals. The updates occur very infrequently, resulting in a sparse data set that cannot be modeled by standard time-series procedures. In response to the lack of appropriate methodology, this paper presents a hierarchical model that takes into account the expert’s level of self-reported expertise and produces aggregate probabilities that are sharp and well calibrated both in- and out-of-sample. The model is demonstrated on a real-world data set that includes over 2300 experts making multiple probability forecasts over two years on different subsets of 166 international political events.

Introduction: The Challenge of Sparse Expert Beliefs

The "Wisdom of Crowds" typically relies on a single snapshot of opinions. However, in high-stakes fields like geopolitics or finance, experts continuously update their beliefs as new information emerges. Aggregating these updates is difficult because the data is inherently "sparse": experts do not check in on a regular schedule. One might provide a forecast on day 1, while another provides updates on days 40 and 60.

Standard time-series tools, such as ARIMA, fail in this context because they require regular intervals and dense data. Furthermore, experts often suffer from systematic biases—they might be consistently overconfident or, as this study surprisingly found, underconfident.

Summary statistics of the geopolitical forecasting dataset, highlighting the extreme sparsity of expert updates.

As shown in the table above, the median number of forecasts given by each expert on a specific question is 1.0, despite questions being active for a median of 72 days. This sparsity necessitates a model that can infer a continuous "true" belief from fragmented data points.

Methodology: Dynamic Hierarchical Modeling

The authors propose a Dynamic Hierarchical Model (DHM) that operates in "logit space," transforming probabilities to the real line. The core of the approach is a latent (hidden) process representing the "true" crowd belief, which evolves over time.

To estimate this process, the authors developed the Sample-And-Calibrate (SAC) procedure:

  1. Sampling Step: Using Gibbs sampling and a Kalman filter (specifically the Forward-Filtering-Backward-Sampling algorithm), the model estimates the hidden state and group-level biases. Because the model is mathematically unidentifiable, one parameter is initially fixed to a constant.
  2. Calibration Step: A one-dimensional optimization finds a scaling factor (beta) that maximizes a "proper scoring rule" (like the Brier score). This ensures the final aggregate probabilities are "calibrated"—meaning if the model predicts a 70% chance of an event, it occurs approximately 70% of the time.

The model also incorporates self-reported expertise (on a 1–5 scale) as a multiplicative bias term, allowing the system to "learn" which groups are more reliable.

Key Insights and Results

The research yielded several significant findings:

  • The Underconfidence Surprise: Contrary to many studies suggesting people are overconfident, the authors found that all expertise groups in this dataset were underconfident—their predictions were consistently too close to 0.5 (the "I don't know" mark).
  • Expertise and Calibration: There was a clear correlation between self-reported expertise and calibration. As expertise increased, underconfidence decreased. The "Extremely Expert" group was the closest to being perfectly calibrated.
  • Superior Accuracy: The SAC model achieved the lowest Brier scores (highest accuracy) across almost all categories when compared to baselines like the Simple Dynamic Linear Model (SDLM) and Exponentially Weighted Moving Averages (EWMA).
The DHM tracking two different events: one that occurred (left) and one that did not (right). The solid line represents the model's aggregate forecast amidst sparse individual data points.

The model's ability to track the evolving probability of an event is illustrated in the figure above. It successfully filters the "noise" of individual sparse forecasts to produce a coherent trend that converges toward the actual outcome.

Conclusion

This paper introduces a robust framework for aggregating sporadic expert opinions into a continuous, calibrated forecast. By balancing "sharpness" (making bold predictions) with "calibration" (being right about those predictions), the SAC method provides a significant advancement over traditional moving averages. While the current model assumes linear belief evolution and operates "offline," it lays the groundwork for real-time, non-linear probability aggregation in complex, dynamic environments.

Summary updated

Abstract

Most subjective probability aggregation procedures use a single probability judgment from each expert, even though it is common for experts studying real problems to update their probability estimates over time. This paper advances into unexplored areas of probability aggregation by considering a dynamic context in which experts can update their beliefs at random intervals. The updates occur very infrequently, resulting in a sparse data set that cannot be modeled by standard time-series procedures. In response to the lack of appropriate methodology, this paper presents a hierarchical model that takes into account the expert’s level of self-reported expertise and produces aggregate probabilities that are sharp and well calibrated both in- and out-of-sample. The model is demonstrated on a real-world data set that includes over 2300 experts making multiple probability forecasts over two years on different subsets of 166 international political events.
BibTeX
@article{satopaa2014probability,
  author = {Ville Satopää and Shane T. Jensen and Barbara A. Mellers and Philip E. Tetlock and Lyle Ungar},
  title = {{Probability aggregation in time-series: Dynamic hierarchical modeling of sparse expert beliefs}},
  journal = {The Annals of Applied Statistics},
  year = {2014},
  doi = {10.1214/14-aoas739},
}
Figure