Variable Splitting for Efficient Gibbs Sampling

10 concepts8 papers1 proposal

In Bayesian inference, we often need to sample from a posterior distribution $p(x|y) \propto p(y|x)p(x)$, where $p(y|x)$ represents the data likelihood and $p(x)$ represents the prior. In practical applications like image restoration, medical imaging, or tensor decomposition, this task faces a fundamental bottleneck: the likelihood and the prior are often deeply entangled, non-smooth, or computationally intractable when evaluated together.

For instance, if the likelihood encodes a complex physical forward model and the prior is a non-smooth regularizer (such as Total Variation), standard Markov Chain Monte Carlo (MCMC) methods struggle. They either require expensive gradient computations that fail on non-smooth landscapes, or they suffer from incredibly slow mixing rates.

To solve this, a recent line of research adapts a classic "divide-and-conquer" strategy from deterministic optimization—variable splitting—and ports it into the world of probabilistic sampling. The result is the Split Gibbs Sampler (SGS), a framework that breaks complex, high-dimensional posterior distributions into simpler, modular, and highly efficient conditional sampling steps.

The Core Intuition: Borrowing from ADMM

To understand why variable splitting works in sampling, it helps to look at mathematical optimization. When optimizing a difficult objective function $f(x) + g(x)$, algorithms like the Alternating Direction Method of Multipliers (ADMM) introduce an auxiliary variable $z$ to split the objective into $f(x) + g(z)$, subject to the constraint that $x = z$.

As shown by Nicolas Dobigeon et al. in Efficient Sampling through Variable Splitting-inspired Bayesian Hierarchical Models, this exact optimization trick can be mirrored within a hierarchical Bayesian framework. Instead of forcing $x = z$ strictly, the framework introduces a soft quadratic coupling term governed by a parameter $\rho$.

Instead of sampling from the intractable target posterior $p(x|y) \propto \exp(-f(x) - g(x))$, the algorithm samples from an augmented joint distribution:

$$p_\rho(x, z | y) \propto \exp\left( -f(x) - g(z) - \frac{1}{2\rho} \|x - z\|^2 \right)$$

Here, $f(x)$ represents the negative log-likelihood, $g(z)$ represents the negative log-prior, and the final term acts as a probabilistic tether between the primary variable $x$ and the auxiliary variable $z$. As the coupling parameter $\rho$ approaches zero, the constraint tightens, and the marginal distribution of $x$ mathematically converges to the true target posterior.

Why It Works: Decoupling the Conditionals

The magic of this augmentation becomes clear when we derive the conditional distributions for a Gibbs sampler. Because $x$ and $z$ are decoupled in the main energy functions and only interact through the quadratic penalty, alternating updates becomes remarkably straightforward:

  1. The Likelihood Step: Sampling from $p(x | z, y) \propto \exp\left(-f(x) - \frac{1}{2\rho}\|x-z\|^2\right)$. This step isolates the data-fidelity term $f(x)$, regularized by a simple Gaussian centered at $z$.
  2. The Prior Step: Sampling from $p(z | x, y) \propto \exp\left(-g(z) - \frac{1}{2\rho}\|x-z\|^2\right)$. This step isolates the prior $g(z)$, regularized by a Gaussian centered at $x$.

By splitting the variables, a single intractable problem becomes two standard sub-problems. As Nicolas Dobigeon et al. demonstrate in Split-and-Augmented Gibbs Sampler - Application to Large-Scale Inference Problems, this structure allows practitioners to use specialized, highly efficient algorithms (such as proximal samplers) for each individual component rather than trying to solve the joint problem all at once.

Dimension-Free Convergence

A common failure mode for MCMC algorithms in high dimensions is the "curse of dimensionality"—as the number of parameters grows, the sampler's efficiency typically plummets. Remarkably, the Split Gibbs Sampler escapes this trap.

In a rigorous theoretical analysis, Maxime Vono et al. (Efficient MCMC Sampling with Dimension-Free Convergence Rate using ADMM-type Splitting) and later Daniel Paulin et al. (Efficient MCMC Sampling with Dimension-Free Convergence Rate using ADMM-type Splitting) proved that the convergence rate of the SGS algorithm can be completely independent of the problem's dimensionality. Because the auxiliary variable absorbs the geometric complexity of the prior, the state space can be explored far more effectively. This dimension-free property makes the method uniquely suited for large-scale distributed inference and massive imaging datasets.

The Plug-and-Play Power of Modular Priors

Beyond computational speed, the most profound advantage of variable splitting in MCMC is modularity. Because the prior step $p(z | x, y)$ is mathematically equivalent to a Gaussian denoising problem, we can swap out traditional mathematical priors for entirely different, black-box generative models.

This realization underpins the "Plug-and-Play" sampling paradigm. For example, Nicolas Dobigeon et al. introduced PnP-SGS in Plug-and-Play Split Gibbs Sampler: Embedding Deep Generative Priors in Bayesian Inference, showing that a pre-trained deep neural network can be embedded directly into the Gibbs loop as a stochastic denoiser. Similarly, Zerui Tao and Qibin Zhao developed DiffBCP (Bayesian Tensor Decomposition with Diffusion Model Prior), which uses the split sampling strategy to insert pre-trained diffusion models into low-rank tensor decomposition tasks.

In these setups, the sampler alternates between a standard closed-form update for the physical data constraints and a deep learning-driven denoising step. This grants practitioners the best of both worlds: the raw expressive power of deep generative AI alongside the rigorous uncertainty quantification of Bayesian inference.

Real-World Impact: Imaging and Beyond

The practical utility of this framework spans multiple domains where fast, uncertainty-aware inference is critical:

By translating the core mechanics of variable splitting from optimization into probability spaces, these methods provide a mathematically rigorous yet flexible blueprint for modern, high-dimensional Bayesian computation.

Key Foundational Work

Efficient Sampling through Variable Splitting-inspired Bayesian Hierarchical Models (2019) — contributed 3 concept elements to this direction.
Split-and-Augmented Gibbs Sampler - Application to Large-Scale Inference Problems (2019) — contributed 3 concept elements to this direction.
Bayesian Image Restoration under Poisson Noise and Log-concave Prior (2019) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_auxiliary_variable_decoupling_split_gibb_e3fa2e17,
  title = {auxiliary variable decoupling (split Gibbs sampling)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/auxiliary-variable-decoupling-split-gibbs-sampling/rdr_e3fa2e177036495a823cdba1ab116741},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_auxiliary_variable_decoupling_split_gibb_e3fa2e17_v1,
  title = {auxiliary variable decoupling (split Gibbs sampling)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/auxiliary-variable-decoupling-split-gibbs-sampling/rdr_e3fa2e177036495a823cdba1ab116741/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake