Conditionally Gaussian Representations for Non-Gaussian Inference

26 concepts23 papers1 proposal

In probabilistic modeling, the Gaussian distribution is a mathematical sanctuary. Linear transformations of Gaussian variables remain Gaussian, and when paired with Gaussian likelihoods, the resulting posteriors are conjugate and available in closed form. This tractability underpins classical linear regression, Kalman filters, and standard Gaussian Processes.

However, the real world is rarely Gaussian. Real-world data exhibits heavy tails (outliers in finance or audio engineering), sharp spikes at zero (sparsity in image features or gene selection), and profound asymmetries (skewed noise from shadows or saturation). Facing these phenomena, researchers often feel forced to abandon Gaussians in favor of more complex distributions like the Student-$t$, Laplace, or $\alpha$-stable families. Unfortunately, doing so typically destroys mathematical conjugacy, turning clean analytical updates into intractable integration problems.

The core insight of scale mixture representations is that we do not have to choose between the realism of non-Gaussian distributions and the computational elegance of Gaussians. Instead, we can build non-Gaussian distributions out of Gaussians. By representing a complex random variable as a conditional Gaussian whose variance (or scale) is modulated by a hidden, secondary random variable, we decouple the non-Gaussian complexity from the core linear algebra.

Mathematically, a target distribution $p(x)$ is decomposed into a hierarchical mixture: $$x \mid \sigma^2 \sim \mathcal{N}(0, \sigma^2), \quad \sigma^2 \sim p(\sigma^2)$$ By choosing different mixing distributions $p(\sigma^2)$ for the latent variance, we can synthesize an astonishing variety of non-Gaussian behaviors while keeping the underlying model conditionally Gaussian.

Taming Heavy Tails and Outliers

When modeling phenomena with frequent extreme events, standard Gaussian models fail because their thin tails assign near-zero probability to outliers. If a model encounters a data point many standard deviations away from the mean, a Gaussian likelihood will aggressively warp the parameter estimates to accommodate it.

By treating the variance as a latent random variable rather than a fixed constant, we can absorb these outliers. If a data point is an extreme outlier, the inference engine simply infers a large latent variance for that specific observation, effectively downweighting its influence on the global parameters.

Standard Gaussian:      [Thin Tails]      --> Outliers distort the model
Scale Mixture:          [Gaussian 1] \
                        [Gaussian 2]  --> [Heavy-Tailed Student-t] --> Outliers absorbed by variance
                        [Gaussian 3] /

This foundational concept was rigorously analyzed by Carmen Fernández et al. in Bayesian Regression Analysis with scale mixtures of normals, establishing that treating scaling factors as latent variables decomposes heavy-tailed posteriors into tractable updates. For instance, placing a Gamma prior on the latent precision (inverse variance) and marginalizing it out yields a heavy-tailed Student-$t$ distribution. This exact mechanism is leveraged by Ali Golmakani et al. in A Weighted-Variance Variational Autoencoder Model for Speech Enhancement, where assigning a learnable weight to each spectrogram time-frame allows a Variational Autoencoder to remain robust against sudden acoustic bursts and noise.

This strategy scales even to highly intractable distributions like the symmetric $\alpha$-stable family, which lack analytical density functions but are vital for modeling extreme, infinite-variance phenomena. Abhimanyu Dubey et al. utilize this in Thompson Sampling on Symmetric Alpha-Stable Bandits to achieve near-optimal regret bounds in environments plagued by extreme outliers.

Similarly, in the domain of deep learning, Jorge Loria et al. demonstrate in Deep Kernel Posterior Learning under Infinite Variance Prior Weights and Posterior Inference on Shallow Infinitely Wide Bayesian Neural Networks under Weights with Unbounded Variance that infinitely wide Bayesian Neural Networks with heavy-tailed priors converge to $\alpha$-stable processes. By conditioning the network on latent variables drawn from an $\alpha/2$-stable distribution, they can deploy standard Gaussian Process machinery to model functions characterized by sharp, non-continuous jumps.

This structural decomposition is a universal property of symmetric distributions with specific tail behaviors. As shown by Alex R. Dytso et al. in Analytical properties of generalized Gaussian distributions, any Generalized Gaussian distribution with a shape parameter less than or equal to two can be perfectly represented as a product of a standard normal distribution and an independent positive random scaling factor. This insight allows researchers like Jordan Awan et al. in Elliptical Perturbations for Differential Privacy to map complex elliptical privacy-preserving noise back to standard Gaussian measures, simplifying privacy proofs in high-dimensional spaces.

Inducing Sparsity and Shrinkage

In high-dimensional regression, we often assume that most features are irrelevant. To enforce this, we use "shrinkage priors" that aggressively drive small coefficients toward zero while leaving large, true signals unpenalized. Popular examples include the Laplacian prior ($L_1$ regularization or Lasso), the Horseshoe prior, and the non-convex Bridge penalty.

The mathematical challenge is that these priors have sharp peaks or non-convex shapes that prevent direct conjugate Bayesian updating. Scale mixtures solve this by representing these sparse priors as a hierarchical cascade of variances:

  1. Laplacian Priors: A Laplace distribution can be written as a Gaussian distribution whose variance is governed by an exponential distribution. In Sparse Probabilistic Principal Component Analysis, Yue Guan et al. use this decomposition to automate feature selection in PCA, bypassing manual regularization tuning. Similarly, Junmin Liu et al. apply it in Bayesian fusion for infrared and visible images to transform a non-differentiable Total Variation penalty into a sequence of computationally efficient weighted least-squares problems.
  2. The Horseshoe Prior: The Horseshoe prior uses a half-Cauchy distribution to handle both global and local sparsity scales, creating an infinitely tall spike at zero and ultra-heavy tails. Enes Makalic demonstrates in A Simple Sampler for the Horseshoe Estimator that breaking the half-Cauchy distribution down into a hierarchy of Inverse-Gamma distributions yields a fully conjugate Gibbs sampler, replacing complex, non-standard sampling steps with standard, highly stable updates.
  3. Bridge and Generalized Shrinkage: For non-convex penalties ($\gamma < 1$), Artin Armagan introduces Variational Bridge Regression, showing that introducing latent variance components makes even non-convex sparse regression mathematically tractable. To unify these disparate approaches, Artin Armagan et al. propose the Three-Parameter Beta distribution in Generalized Beta Mixtures of Gaussians, creating a single hierarchical framework of Gamma distributions that guarantees closed-form conjugacy with Gaussian likelihoods.

Accommodating Asymmetry and Skewness

Real-world errors are not always symmetric. For instance, in image processing, shadows create a one-sided skew in pixel intensity, and in quantile regression, we deliberately focus on asymmetric loss functions (the pinball loss) to estimate specific percentiles of a distribution.

The scale mixture framework elegantly adapts to asymmetry by making the mean of the conditional Gaussian dependent on the latent variance component. Instead of just scaling the width of the Gaussian, the latent variable shifts its center: $$x \mid \sigma^2 \sim \mathcal{N}(\theta \sigma^2, \sigma^2)$$ When the latent variance $\sigma^2$ is large, the distribution is simultaneously wider and shifted further along the axis, perfectly mirroring the behavior of an Asymmetric Laplace distribution.

Small Latent Variance:     |*| (Sharp, centered spike)
Large Latent Variance:     _____*_____ (Wide, shifted to the right)
------------------------------------------------------------------
Marginalized Result:       |\_ (Asymmetric, skewed distribution)

Fabio Ramos et al. exploit this property in Variational Inference for Nonparametric Bayesian Quantile Regression, treating the non-differentiable pinball loss as an Asymmetric Laplace likelihood and decomposing it to unlock standard Gaussian Process machinery. In Robust Asymmetric Bayesian Adaptive Matrix Factorization, Xin Guo et al. apply this decomposition to handle skewed noise patterns caused by shadows and saturation in face reconstruction, converting an otherwise intractable probabilistic framework into standard variational updates.

Furthermore, Maria-Zafeiria Spyropoulou et al. extend this to individual-level variable selection in Fast Bayesian inference in a class of sparse linear mixed effects models, representing skew-$t$ distributions as a mixture of normals with latent variables to enable rapid, optimization-based inference for highly irregular error structures.

Why It Works: The Computational Engine

The ultimate justification for scale mixture representations is computational. By introducing a latent variable $\sigma^2$, we temporarily increase the dimensionality of our parameter space, but we drastically simplify the geometry of the posterior distribution.

When we condition on the latent scales, the non-Gaussian model collapses back into a standard Gaussian framework. This unlocks three primary inference engines:

1. Gibbs Sampling

Instead of using slow, rejection-heavy Metropolis-Hastings steps to sample from a complex heavy-tailed distribution, a scale mixture allows for an alternating, two-step conjugate sampler:

  • Step 1: Sample the model parameters conditioned on the latent scales (a standard, fast Gaussian linear update).
  • Step 2: Sample the latent scales conditioned on the current parameters (typically a standard Gamma or Inverse-Gamma update).

This parallelizable, conjugate updating scheme is central to the topographic filters learned by Max Welling et al. in Learning Sparse Topographic Representations with Products of Student-t Distributions, where treating the inverse variance of each filter as a hidden variable enables efficient parallel Gibbs sampling.

2. Variational Inference (VB)

Non-conjugate priors typically break Variational Bayes because the evidence lower bound (ELBO) contains expectations that cannot be solved analytically. Decomposing the prior into a hierarchy of Gamma or Inverse-Gamma distributions restores conditional conjugacy, allowing for clean, closed-form coordinate ascent updates.

3. Expectation-Maximization (EM)

By treating the latent scales as "missing data," complex non-differentiable objectives (like $L_1$ or total variation penalties) are transformed into a sequence of weighted least-squares problems. In the "E-step," we compute the expected value of the latent precisions, which act as weights for the "M-step" optimization.

Alternative Partitioning Strategies

While scale modulation (multiplicative mixing) is the dominant paradigm for taming non-Gaussianity, some complex distributions are instead tamed by partitioning them additively.

For instance, Constantinos Daskalakis et al. demonstrate in A Size-Free CLT for Poisson Multinomials and its Applications that complex probability distributions can be split into a high-variance Gaussian component and a sparse residual component. In this regime, the high-variance Gaussian part effectively dominates or "swamps" the sparse residual part. This allows researchers to simplify the overall approximation without significant loss of accuracy, proving that whether through multiplicative scale modulation or additive variance partitioning, embedding Gaussian components within complex distributions remains a premier strategy for scalable probabilistic inference.

Key Foundational Work

Bayesian Regression Analysis with scale mixtures of normals (1999) — contributed 1 concept elements to this direction.
Bayesian Learning of Sparse Classifiers (2001) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_scale_mixture_representation_5dea5077,
  title = {scale-mixture representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scale-mixture-representation/rdr_5dea50771eb14fd9b8eb602100c20eb3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_scale_mixture_representation_5dea5077_v1,
  title = {scale-mixture representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scale-mixture-representation/rdr_5dea50771eb14fd9b8eb602100c20eb3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake