Approximating Message Lengths of Hierarchical Bayesian Models Using Posterior Sampling

Daniel F. Schmidt, Enes Makalic, John L. HopperAustralasian Conference on Artificial Intelligence 2016 · DOI

TL;DR

This paper introduces MML-h, a novel and computationally efficient method for selecting between complex hierarchical Bayesian models. By leveraging posterior samples from MCMC, MML-h provides a robust way to compare models without the computational burden of traditional methods, making advanced Bayesian modeling more accessible.

What Problem Does This Paper Address?

Modern statistical modeling often involves complex hierarchical Bayesian models, which are powerful for capturing intricate relationships and prior beliefs. However, selecting the "best" model from a set of competing hierarchical structures has been a significant challenge. Traditional model selection methods, like those based on Minimum Message Length (MML), struggle with the complexity of hierarchical models, often requiring computationally intensive calculations or being limited to specific model structures. This gap leaves researchers without an easy-to-use, general-purpose tool for choosing among these sophisticated models.

Methods

The core contribution is the MML-h approximation. This method extends the established MML87 approximation to hierarchical Bayesian models by using posterior samples obtained from Markov Chain Monte Carlo (MCMC) methods. Instead of relying on the Fisher information matrix (which can be difficult to compute for hierarchical models), MML-h uses the empirical posterior covariance matrix of the sampled parameters and hyperparameters. This allows for the direct computation of message lengths from a single MCMC run. The method also incorporates the posterior expectation of the negative log-likelihood, effectively balancing model fit with complexity.

Datasets

Not discussed in the paper. The paper focuses on methodological development and demonstrates its application on simulated data and established regression models.

Key Insights and Ideas

  • Posterior Samples as a Model Selection Tool: The paper demonstrates that posterior samples, readily available from MCMC, can be directly used to approximate model selection criteria like MML. This bypasses the need for complex analytical derivations of Fisher information matrices.
  • Bridging MML and Hierarchical Models: MML-h provides a principled way to apply the information-theoretic MML principle to hierarchical structures, which were previously difficult to handle.
  • Robustness to Prior Choice: The MML-h criterion is shown to be robust in selecting between different hierarchical structures (like ridge vs. horseshoe regression), adapting to the data's properties and effectively choosing the more appropriate prior specification.
  • Computational Efficiency: By avoiding analytical calculations of complex quantities and relying on MCMC samples, MML-h offers a significant computational advantage over existing methods for hierarchical model selection.

Key Results

The paper demonstrates the effectiveness of MML-h through simulations and an example comparing ridge and horseshoe regression models.

  • Adaptive Model Selection: In simulations, MML-h successfully discriminated between ridge and horseshoe hierarchies, selecting the model that better reflected the underlying data-generating process. For sparse data, the horseshoe was favored, while for denser data, ridge regression was preferred.
  • Robustness: MML-h-based methods showed excellent robustness, rarely performing substantially worse than the best-performing method across various sparsity levels and correlation structures.
  • Poisson-Exponential Hierarchy: The method was applied to a Poisson-exponential hierarchy, showing close agreement with the exact MML87 message length for finite sample sizes and converging for large samples.
  • Ridge vs. Horseshoe: The MML-h approximation was used to compute message lengths for both ridge and horseshoe regression models, enabling data-driven selection between these two important shrinkage priors.

Limitations and Future Work

  • Invariance: The MML-h approximation is generally not invariant under reparameterizations, unlike the standard MML87. However, the authors suggest this is a minor issue for moderate to large sample sizes, where differences in message lengths between parameterizations become small.
  • Convergence Proof: While the paper establishes convergence for simple cases, a formal proof for the general setting of hierarchical models is noted as a topic for future research.
  • Block-wise Covariance: The authors suggest that for high-dimensional models, the empirical covariance matrix can be decomposed into block-wise independent matrices to improve numerical stability and computational efficiency.
  • Further Applications: The paper highlights the potential for MML-h to significantly improve model selection for penalized regression and suggests future research into mitigating weaknesses of individual prior distributions.

Summary updated

BibTeX
@inproceedings{schmidt2016approximating,
  author = {Daniel F. Schmidt and Enes Makalic and John L. Hopper},
  title = {{Approximating Message Lengths of Hierarchical Bayesian Models Using Posterior Sampling}},
  booktitle = {Australasian Conference on Artificial Intelligence},
  year = {2016},
  doi = {10.1007/978-3-319-50127-7\_41},
}
Figure