When applying Bayesian inference to high-dimensional problems—such as fluid dynamics, geophysics, or deep neural networks—practitioners routinely encounter a fundamental barrier known as the curse of dimensionality. Standard Markov Chain Monte Carlo (MCMC) methods, like the Random Walk Metropolis (RWM) algorithm, degrade rapidly as the number of parameters grows. As Andrieu et al. demonstrate in their analysis of Metropolis chains, the step size of traditional random walks must be scaled down inversely with the dimension to maintain a reasonable acceptance rate. Consequently, the sampler is forced to take vanishingly small steps, causing it to stall.
To overcome this, a powerful class of dimension-independent MCMC algorithms has emerged. By restructuring how proposals are generated, these methods maintain a stable acceptance probability regardless of the parameter space's dimensionality, enabling efficient inference even in infinite-dimensional function spaces.
The Intuition: Why Standard MCMC Fails
To understand why dimension-independent MCMC works, we must first look at why standard random walks fail. In high dimensions, probability distributions concentrate their mass in a narrow region known as the typical set. For a high-dimensional Gaussian prior, this typical set resembles a thin, hollow spherical shell.
When an RWM algorithm proposes a move, it adds blind, isotropic noise to the current state. In high dimensions, almost every random direction points out of this thin typical set and into regions of near-zero probability. To prevent the acceptance rate from collapsing to zero, the step size must be shrunk to a fraction of the dimension ($O(d^{-1/2})$). The sampler is trapped making microscopic movements.
The Core Mechanism: Prior-Preserving Proposals
The breakthrough insight behind dimension-independent MCMC is to design proposal mechanisms that inherently respect the geometry of the prior distribution. Consider a common Bayesian setup where the prior over the parameters $u$ is a Gaussian distribution $\mathcal{N}(0, C)$.
Instead of adding blind noise, the preconditioned Crank-Nicolson (pCN) algorithm proposes a new state $u^*$ using an autoregressive update:
$$u^* = \sqrt{1 - \beta^2} u + \beta \xi, \quad \xi \sim \mathcal{N}(0, C)$$
where $\beta \in (0, 1]$ is a step-size parameter.
This formulation possesses a remarkable mathematical property: if the current state $u$ is drawn from the prior, the proposed state $u^*$ is exactly distributed according to the prior as well, completely independent of the dimension or the choice of $\beta$.
Because the proposal perfectly preserves the prior, the prior terms completely cancel out of the Metropolis-Hastings acceptance ratio. The acceptance probability simplifies to depend solely on the change in the likelihood function:
$$\alpha = \min\left(1, \frac{\text{Likelihood}(u^*)}{\text{Likelihood}(u)}\right)$$
If the likelihood function is smooth and depends primarily on a finite number of informative directions (which is typical for physical inverse problems and continuous functions), the acceptance rate remains stable even as the dimension approaches infinity.
Alternative Frameworks for Dimension-Free Convergence
This core philosophy of leveraging prior structure or problem geometry manifests across several algorithmic variants:
- Elliptical Slice Sampling (ESS): Instead of requiring manual tuning of the step size $\beta$, ESS defines an elliptical trajectory between the current state and a prior noise vector. Natarovskii et al. provided a rigorous theoretical foundation for this approach, proving that ESS achieves geometric convergence that remains stable as the state space dimensionality increases. Similarly, Bloem-Reddy et al. demonstrated that trajectory-based slice sampling exhibits dimension-free scaling when the prior accurately reflects variable dependencies.
- Non-Gaussian Priors and Splitting: While pCN natively assumes Gaussian priors, Vollmer et al. extended the mathematical framework to guarantee dimension-independent convergence rates for non-Gaussian priors. Taking a different approach to non-Gaussianity, Vono et al. and Paulin et al. introduced the Split Gibbs Sampler (SGS). By leveraging an ADMM-type splitting strategy to decouple complex targets, the authors achieved dimension-free convergence rates suitable for large-scale distributed inference.
- Exploiting Local Structure: Rather than relying solely on global prior alignment, Morzfeld et al. demonstrated that MCMC samplers can achieve dimension-independent convergence by explicitly exploiting the local coupling and structure inherent in high-dimensional posterior distributions.
- Non-Euclidean Geometries: The principle of dimension-independence has also been extended to curved spaces. For directional data constrained to manifolds, Rudolf et al. developed dimension-independent MCMC algorithms on high-dimensional spheres by lifting the problem into an ambient Hilbert space.
Applications: From Geophysics to Deep Learning
Dimension-independent sampling was originally popularized in physical inverse problems where parameters represent continuous spatial fields. For instance, Iglesias et al. used pCN as a high-fidelity "gold standard" benchmark to evaluate cheaper, ad-hoc Gaussian approximations (like the Ensemble Kalman Filter) in groundwater flow and reservoir modeling, revealing where those approximations failed to capture true non-Gaussian posteriors. In a similar vein, Friedli et al. integrated prior-preserving proposals into a correlated pseudo-marginal framework for subsurface lithological tomography, reducing sensitivity to discretization choices. For engineering design, Kumar et al. leveraged dimension-independent Bayesian Monte Carlo estimation to optimize jet engine components under manufacturing variations.
Recently, these algorithms have experienced a massive resurgence in modern machine learning, where parameter spaces routinely span millions of dimensions:
┌─────────────────────────────────────────────────────────────────────────┐
│ High-Dimensional Applications │
└─────────────────────────────────────────────────────────────────────────┘
│
┌───────────────────────────┼───────────────────────────┐
▼ ▼ ▼
┌───────────────┐ ┌───────────────┐ ┌───────────────┐
│ Bayesian NNs │ │ Infinite-Width│ │ Generative │
│ (Emulators) │ │ Limits │ │ Score Models │
└───────────────┘ └───────────────┘ └───────────────┘
Moslemi et al. Pezzetti et al. Yoon et al.
Uses pCN with Proves pCN/pCNL Uses pCNL for
surrogates for acceptance → 100% reward alignment
60x speedups. as width → ∞. in latent spaces.
In the context of Bayesian Neural Networks (BNNs), Moslemi et al. combined dimension-independent MCMC with cheap neural network emulators to accelerate posterior sampling by up to 60x. Remarkably, Pezzetti et al. proved that for wide BNNs, pCN and its Langevin variant (pCNL) actually become more efficient as the network grows wider—demonstrating that the acceptance probability approaches 100% in the infinite-width limit.
Finally, this dimension-robustness has proven vital for generative AI alignment. Yoon et al. utilized pCNL to sample reward-informed initial particles within the massive latent spaces of diffusion and score models, bypassing the vanishing acceptance rates that typically cripple standard Sequential Monte Carlo methods during inference-time alignment.
Key Foundational Work
MCMC for the evaluation of Gaussian approximations to Bayesian inverse problems in groundwater flow (2012) — contributed 1 concept elements to this direction.
Evaluation of Gaussian approximations for data assimilation in reservoir models (2012) — contributed 1 concept elements to this direction.
Efficient robust design for manufacturing process capability (2006) — contributed 1 concept elements to this direction.