Global-Local Mixtures

Anindya Bhadra, Jyotishka Datta, Nicholas G. Polson, Brandon T. WillardarXiv 2016 · DOI · arXiv

Abstract: Global-local mixtures are derived from the Cauchy-Schlomilch and Liouville integral transformation identities. We characterize well-known normal-scale mixture distributions including the Laplace or lasso, logit and quantile as well as new global-local mixtures. We also apply our methodology to convolutions that commonly arise in Bayesian inference. Finally, we conclude with a conjecture concerning bridge and uniform correlation mixtures.

Introduction

In modern high-dimensional statistics, the challenge of identifying "sparse" signals—where only a few variables among thousands are truly significant—is a central problem. To address this, researchers frequently employ global-local mixtures. These models operate through a two-tier hierarchy: a local scale parameter ($\lambda_i$) that accounts for the specific behavior of individual variables, and a global scale parameter ($\tau$) that governs the overall sparsity of the system.

While models such as the Horseshoe or Lasso are staples of the Bayesian toolkit, deriving their exact mathematical forms and the specific penalties they impose can be algebraically exhausting. In "Global-Local Mixtures," Bhadra et al. provide a mathematical "skeleton key" that unifies these derivations using two classical integral identities, simplifying the theoretical landscape of Bayesian regularization.

The Mathematical Engine: Cauchy-Schlömilch and Liouville

Rather than relying on traditional "brute-force" integration, the authors leverage two powerful, often overlooked, integral transformation identities to characterize probability densities:

  1. The Cauchy-Schlömilch Transformation: This identity allows for the transformation of an integral involving a function of $(ax - b/x)^2$ into a significantly simpler form. The authors demonstrate that this transformation is the underlying "engine" behind the Laplace (Lasso) distribution.
  2. The Liouville Transformation: A related identity used for functions involving $(ax + b/x)$, providing a similar simplification for a different class of distributions.

The Mother-Daughter Framework

Central to the paper is the concept of a "mother-daughter" relationship between distributions. The authors show that if a simple density function $f(x)$ (the mother) exists, a more complex "daughter" density $g(x)$ can be automatically generated using the Cauchy-Schlömilch identity. This framework provides a systematic recipe for creating new types of shrinkage priors, moving away from ad-hoc derivations toward a unified generative theory.

Unifying Disparate Models

The methodology reveals that several seemingly unrelated statistical tools share the same underlying mathematical DNA:

  • The Lasso: Used for linear regression, the Lasso penalty is shown to be a normal scale mixture derived cleanly through these transformations (Proposition 1).
  • Logistic Regression (Pólya-Gamma): The paper derives marginal distributions for Pólya-Gamma mixtures, which are essential for efficient Markov Chain Monte Carlo (MCMC) algorithms in logistic modeling.
  • Quantile Regression: The check-loss function used in quantile regression is similarly integrated into this framework.
  • Support Vector Machines (SVMs): The authors identify a "pseudo-likelihood" form that represents SVMs as a global-local mixture. This insight potentially paves the way for more robust Bayesian interpretations of SVMs.

Simplifying Convolutions in Bayesian Inference

Beyond mixtures, the paper applies these identities to convolutions—the mathematical operations used to determine the distribution of the sum of random variables. This is a common requirement in Bayesian inference. Using their unified approach, the authors provide streamlined proofs for complex identities, such as why the sum of two Cauchy-distributed variables remains Cauchy-distributed, and similar results for Inverse-Gaussian distributions.

Conclusion and Future Directions

While the paper is primarily theoretical and does not provide empirical benchmarks or computational software, it establishes a rigorous foundation for future methodological development. The authors conclude with a conjecture that these identities can be extended to "bridge" and "uniform correlation" mixtures, which are vital for complex multivariate modeling.

Future research may focus on utilizing the Liouville formula to calculate higher-order moments (such as variance and kurtosis) for these distributions and exploring the practical algorithmic efficiency of the new mixtures derived through this framework.

Summary updated

Abstract

Global-local mixtures are derived from the Cauchy-Schlomilch and Liouville integral transformation identities. We characterize well-known normal-scale mixture distributions including the Laplace or lasso, logit and quantile as well as new global-local mixtures. We also apply our methodology to convolutions that commonly arise in Bayesian inference. Finally, we conclude with a conjecture concerning bridge and uniform correlation mixtures.
BibTeX
@misc{bhadra2016global,
  author = {Anindya Bhadra and Jyotishka Datta and Nicholas G. Polson and Brandon T. Willard},
  title = {{Global-Local Mixtures}},
  year = {2016},
  eprint = {1604.07487},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1604.07487},
}
Figure