Double spike Dirichlet priors for structured weighting

Huiming Lin, Meng LiarXiv 2020 · DOI · arXiv

Abstract

Assigning weights to a large pool of objects is a fundamental task in applications ranging from economic forecasting to ensemble machine learning. This paper introduces the concept of structured high-dimensional probability simplexes, where most components are zero (sparsity) and the remaining components are approximately equal (partial constancy). We propose a new class of double spike Dirichlet priors to shrink a probability simplex toward this structure. This approach addresses the "forecast combination puzzle" and improves the performance of random forests while enabling uncertainty quantification. We demonstrate competitive performance through simulations and real-world applications in economics and machine learning.

Introduction

In modern data science, we frequently encounter the challenge of weighting a large ensemble of experts or models. Whether combining GDP forecasts from dozens of institutions or aggregating predictions from hundreds of decision trees in a Random Forest, the goal is to find an optimal weighting scheme. However, two significant hurdles persist: high dimensionality (where the number of experts exceeds the available data) and the "forecast combination puzzle"—the empirical observation that a simple average often outperforms complex statistical optimizations.

Standard regularization techniques like the Lasso are effective at inducing sparsity but struggle with the simplex constraint (weights must be non-negative and sum to one) and do not naturally encourage the equal weighting that often yields robust results. This research introduces the Double Spike Dirichlet Prior to address these challenges simultaneously.

The Double Spike Dirichlet Prior

The core innovation is a Bayesian prior designed to induce a specific structure on the probability simplex. The "Double Spike" refers to a mixture of Dirichlet distributions:

  1. The Sparsity Spike: Controlled by a small parameter ($\rho_2$), this component pushes the majority of weights toward zero, effectively performing variable selection.
  2. The Equality Spike: Controlled by a large parameter ($\rho_1$), this component pushes the remaining non-zero weights toward a common value. For instance, if four experts are selected, the prior encourages each to have a weight near 0.25.

By performing selection and weighting simultaneously, the model avoids the pitfalls of "two-step" methods. To handle the computational complexity of exploring various combinations of experts, the authors developed the Add/Delete/Swap/Stay (ADSS) sampler, a specialized Markov chain Monte Carlo (MCMC) algorithm that efficiently navigates the high-dimensional parameter space.

Applications and Empirical Results

The method was evaluated across synthetic simulations and two real-world datasets: the European Central Bank (ECB) Survey of Professional Forecasters and the UCI Concrete Compressive Strength dataset.

Economic Forecasting

Using quarterly GDP growth forecasts for the Euro area from 23 forecasters, the Double Spike Dirichlet Prior achieved a Root-Mean-Squared Error (RMSE) of 1.441. This significantly outperformed the simple average (1.502) and the two-step Lasso approach (1.542). The model successfully identified a small, stable subset of forecasters whose combined insights provided superior predictive power.

Improving Random Forests

In the context of ensemble learning, the method was applied to a Random Forest consisting of 500 decision trees. The results were striking: the model reduced prediction error by 7.3% compared to standard Random Forest weighting. More importantly, it achieved this by identifying that only 10 to 18 trees were truly contributing to the model's accuracy. By pruning the forest and weighting the remaining trees nearly equally, the method produced a more parsimonious and accurate model.

Key Insights

The success of this approach stems from the synergy between sparsity and equality. Forcing non-zero weights to be similar acts as a powerful regularizer, preventing the model from over-fitting to specific experts who may have been "lucky" in past observations. Furthermore, the method demonstrates a robust ability to handle highly correlated experts, selecting a diverse group that balances individual errors rather than picking one at random.

Conclusion and Future Directions

The Double Spike Dirichlet Prior provides a principled Bayesian framework for structured weighting in high dimensions. While currently focused on the probability simplex (positive weights), future research may explore relaxing sign constraints or incorporating group sparsity structures. Despite the computational demands of MCMC, the ADSS sampler makes this approach viable for complex, modern datasets where both precision and parsimony are paramount.

Summary updated

Abstract

Assigning weights to a large pool of objects is a fundamental task in a wide variety of applications. In this article, we introduce the concept of structured high-dimensional probability simplexes, in which most components are zero or near zero and the remaining ones are close to each other. Such structure is well motivated by (i) high-dimensional weights that are common in modern applications, and (ii) ubiquitous examples in which equal weights -- despite their simplicity -- often achieve favorable or even state-of-the-art predictive performance. This particular structure, however, presents unique challenges partly because, unlike high-dimensional linear regression, the parameter space is a simplex and pattern switching between partial constancy and sparsity is unknown. To address these challenges, we propose a new class of double spike Dirichlet priors to shrink a probability simplex to one with the desired structure. When applied to ensemble learning, such priors lead to a Bayesian method for structured high-dimensional ensembles that is useful for forecast combination and improving random forests, while enabling uncertainty quantification. We design efficient Markov chain Monte Carlo algorithms for implementation. Posterior contraction rates are established to study large sample behaviors of the posterior distribution. We demonstrate the wide applicability and competitive performance of the proposed methods through simulations and two real data applications using the European Central Bank Survey of Professional Forecasters data set and a data set from the UC Irvine Machine Learning Repository (UCI).
BibTeX
@misc{lin2020double,
  author = {Huiming Lin and Meng Li},
  title = {{Double spike Dirichlet priors for structured weighting}},
  year = {2020},
  eprint = {2007.04387},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.04387},
}
Figure