Probability Paths and the Structure of Predictions over Time

Zhiyuan Lin, Hao Sheng, Sharad Goel

TL;DR

This paper introduces GLIM, a novel Bayesian framework for modeling dynamic probability predictions (probability paths) that evolve over time. GLIM explicitly captures the inherent structure of these paths, such as their martingale property and appropriate volatility, outperforming existing methods in quantifying future uncertainties.

What Problem Does This Paper Address?

Many real-world predictions, like weather forecasts or election outcomes, aren't static. They evolve as new information becomes available, forming "probability paths." While much research focuses on making accurate real-time predictions, less attention has been paid to understanding the structure of how these predictions change over time. This is crucial for decision-making: knowing if a forecast is likely to change soon or remain stable can significantly impact choices. For instance, a stable forecast might prompt immediate action, while a volatile one might suggest waiting for more information. Existing time-series models often fail to capture the specific properties of these probability paths, such as their martingale nature (where the expected future value equals the current value) and appropriate levels of volatility.

Methods

The core contribution is the Gaussian Latent Information Martingale (GLIM) framework. GLIM models a probability path $\{Y_t\}_{t=0}^T$ by introducing a sequence of latent variables $\{Z_t\}_{t=1}^T$. These latent variables represent the "amount of information" received at each time step, influencing the ultimate outcome.

Key aspects of GLIM:

  • Bayesian Framework: GLIM is a Bayesian model, allowing for uncertainty quantification.
  • Latent Information: It assumes an underlying latent process of information flow that drives the evolution of predictions.
  • Martingale Property: The model is designed to inherently satisfy the martingale property, meaning the expected future prediction is equal to the current prediction. This is a critical property of rational belief updating.
  • Covariate Incorporation: GLIM can incorporate covariates (e.g., meteorological data) to model heterogeneous forecast structures.
  • Recursive Inference: The paper derives a recursive formula for the log probability density function, enabling efficient inference.

GLIM is contrasted with three classes of baseline methods:

  1. Martingale Model of Forecast Evolution (MMFE): Focuses on estimating covariance of prediction increments.
  2. Probabilistic Time Series Models: Standard models like AR, ARMA, ARIMA, and GARCH.
  3. Bayesian Deep Sequence Models: Modern deep learning approaches like LSTMs.

Datasets

The paper evaluates GLIM on two real-world datasets:

  • Basketball Game Outcomes: Minute-by-minute win probabilities for professional basketball teams.
  • Weather Forecasts: Daily precipitation predictions for Australian cities.

A synthetic dataset was also generated to study GLIM's finite-sample behavior and parameter recovery.

Key Insights and Ideas

  • The "Why" Behind Forecast Changes Matters: The paper highlights that understanding how a forecast evolves (its volatility and stability) is as important as the forecast itself for informed decision-making.
  • Martingale Property is Key: Explicitly modeling the martingale property is crucial for correctly representing evolving beliefs and avoiding systematic biases in predictions.
  • Latent Information as a Unifying Concept: The introduction of latent variables provides a flexible yet principled way to model the information flow that drives prediction changes.
  • GLIM's Structural Advantage: By directly modeling the structure of probability paths, GLIM can capture nuances missed by general time-series models, leading to better uncertainty quantification.

Key Results

GLIM consistently outperformed three baseline methods (MMFE, LR, MQLSTM) across various metrics on both the basketball and weather datasets.

  • Mean Calibration MSE: GLIM showed significantly lower mean calibration MSE, indicating its predicted posterior means were closer to the observed ground truth. The paper notes GLIM's performance was often orders of magnitude better.
  • Volatility MSE: GLIM also achieved lower expected volatility MSE, suggesting it better captured the appropriate level of fluctuation in probability paths.
  • Credible Interval Coverage: GLIM demonstrated superior or indistinguishable credible interval coverage at different significance levels and time points compared to baselines.

Limitations and Future Work

  • Model Complexity: While GLIM's structure is simpler than some deep learning models, the derivation of its likelihood function involves some notational complexity, especially when latent variables are correlated.
  • Parameter Identification: The paper notes that the scale of latent variables needs to be constrained (e.g., by setting Var(Z₁) = 1) for full parameter identification.
  • Future Work: The authors suggest exploring more complex covariance matrix structures (e.g., using deep kernel learning) and applying GLIM to other domains with dynamic predictions. They also emphasize the potential for GLIM to aid decision-makers in various fields.

Summary updated

Abstract

In settings ranging from weather forecasts to political prognostications to financial projections, probability estimates of future binary outcomes often evolve over time. For example, the estimated likelihood of rain on a specific day changes by the hour as new information becomes available. Given a collection of such probability paths, we introduce a Bayesian framework -- which we call the Gaussian latent information martingale, or GLIM -- for modeling the structure of dynamic predictions over time. Suppose, for example, that the likelihood of rain in a week is 50 %, and consider two hypothetical scenarios. In the first, one expects the forecast to be equally likely to become either 25 % or 75 % tomorrow; in the second, one expects the forecast to stay constant for the next several days. A time-sensitive decision-maker might select a course of action immediately in the latter scenario, but may postpone their decision in the former, knowing that new information is imminent. We model these trajectories by assuming predictions update according to a latent process of information flow, which is inferred from historical data. In contrast to general methods for time series analysis, this approach preserves important properties of probability paths such as the martingale structure and appropriate amount of volatility and better quantifies future uncertainties around probability paths. We show that GLIM outperforms three popular baseline methods, producing better estimated posterior probability path distributions measured by three different metrics. By elucidating the dynamic structure of predictions over time, we hope to help individuals make more informed choices.
BibTeX
@inproceedings{lin2021probability,
  author = {Zhiyuan Lin and Hao Sheng and Sharad Goel},
  title = {{Probability Paths and the Structure of Predictions over Time}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=5CKM8jCfEmM},
}
Figure