Sampling from complex, high-dimensional probability distributions is a foundational challenge in machine learning and statistical physics. Traditional approaches generally fall into two camps: local Markov Chain Monte Carlo (MCMC) samplers, which explore space through small incremental steps, and global generative models, which attempt to map an entire distribution at once. Both possess complementary failure modes. Local samplers get trapped in local modes because crossing low-probability energy barriers takes exponential time. Global models, such as normalizing flows or variational approximations, can bypass these barriers but suffer from mode collapse and struggle to capture intricate local geometries.
To resolve this tension, a powerful paradigm has emerged: hybrid global-local sampling. By interleaving global transport mechanisms with local mutation kernels, these methods achieve a rigorous division of labor. The global component provides the mobility needed to bridge disconnected modes or track large-scale distribution shifts, while the local component provides the precision needed to resolve fine-grained geometric details and correct for global model inaccuracies.
The Synergy of Flow Transport and Mutation
One of the clearest realizations of this hybrid strategy is the combination of normalizing flows with Sequential Monte Carlo (SMC) or annealing frameworks. In pure variational inference, optimizing a normalizing flow often leads to mode-seeking behavior, where the flow focuses entirely on a single high-probability region and ignores others.
To counteract this, Michael Arbel et al. introduced Annealed Flow Transport Monte Carlo (AFT). Instead of relying solely on a deterministic flow to map particles from one annealed target to the next, AFT utilizes a dual-stage update: a global transport step followed by a local MCMC mutation. While the flow moves the particle population toward new target regions, the local mutation ensures exploration. If the learned transformation misses a mode, the local stochastic steps allow particles to diffuse and recover the missing mass, preventing mode collapse.
[Particle Population]
│
▼ (Global Transport Step: Normalizing Flow)
[Large-scale Mass Movement to New Target Region]
│
▼ (Local Mutation Step: MCMC Kernel)
[Fine-grained Exploration & Mode Recovery]
This division of labor makes the overall sampling process highly resilient to imperfections in the generative model. As demonstrated by Alexander G. D. G. Matthews et al. in Continual Repeated Annealed Flow Transport Monte Carlo (CRAFT), the generative model does not need to be a perfect replicator of the target distribution to provide massive efficiency gains. The flow handles the heavy lifting of transporting particles across an annealing schedule, while the MCMC kernels handle local density corrections and accommodate topological complexities that a continuous flow cannot easily map.
Concurrent Training and Latent Space Exploration
Rather than using a fixed or sequentially trained global model, some frameworks train the global proposal mechanism and the local sampler concurrently, creating a bidirectional feedback loop.
In Efficient Bayesian Sampling Using Normalizing Flows to Assist Markov Chain Monte Carlo Methods, Marylou Gabrié et al. propose a framework that alternates between local gradient-based updates and global jumps suggested by an invertible neural network. The network learns the target distribution on-the-fly from the points discovered during local exploration. Simultaneously, it provides high-quality, non-local proposals that allow the sampler to escape local optima. By using the network's inverse to calculate Metropolis-Hastings acceptance probabilities, the system executes non-local transitions between distant modes while strictly maintaining detailed balance.
┌─────────────────────────────────────────────────────────┐
│ │
▼ │
[Local Sampler] ───(Provides High-Quality Samples)───► [Generative Model]
│ ▲
└───────────────◄───(Provides Non-Local Jumps)────────────┘
Alternatively, the local exploration can be mapped entirely into a simplified latent space. Florentin Coeurdoux et al. leverage this in Normalizing flow sampling with Langevin dynamics in the latent space (NF-SAILS). Normalizing flows often generate poor, out-of-distribution samples in the low-probability gaps between disconnected data modes. NF-SAILS addresses this by running local Langevin dynamics within the latent space of a pre-trained flow, combined with global independent proposals. This hybrid strategy ensures fine-grained convergence to local density peaks while maintaining broad coverage of the entire target distribution, all without requiring retraining of the underlying flow.
Mode-Hopping via Jump Kernels
Another facet of hybrid sampling involves explicitly constructing state-independent global "jump" kernels to interleave with standard local moves. This is particularly effective for navigating highly isolated modes separated by deep energy valleys.
In Local-Global MCMC kernels: the best of both worlds, Sergey Samsonov et al. introduce Ex2MCMC. The algorithm alternates between a global importance sampling step to facilitate jumps between distant modes and a local rejuvenation step to refine the sample's position. The local sampler prevents the chain from getting trapped in low-probability regions where global proposals might occasionally land, ensuring high-fidelity exploration of the target distribution's tails.
This concept extends naturally to discrete and constrained spaces. For discrete distributions, Alkis Gotovos et al. develop the $M^3$ sampler in Discrete Sampling using Semigradient-based Product Mixtures. It pairs a local transition kernel (like a Gibbs sampler) with a state-independent global proposal constructed from discrete semigradients. The resulting hybrid chain remains locally precise while gaining the global mobility needed to rapidly mix across disjoint high-probability regions.
The mathematical intuition behind these global jumps is rooted in identifying and mapping mode geometries. Max Welling demonstrates in Generalized Darting Monte Carlo that embedding global transitions between high-probability regions directly into local MCMC moves allows samplers to hop across energy barriers that are otherwise impassable. To automate this, Sungjin Ahn et al. present Distributed and Adaptive Darting Monte Carlo through Regenerations, which fits a density estimation model to previously sampled data on-the-fly. This model identifies the location and geometry of high-probability regions, defining jump kernels that allow the sampler to transition directly between modes without ever having to traverse the low-probability valleys between them.
Why the Synergy Works: The Dimensionality Trade-off
The fundamental reason hybrid global-local sampling works so well lies in how global and local mechanisms scale with dimensionality. Louis Grenioux provides a systematic analysis of this relationship in On Sampling with Approximate Transport Maps, highlighting a core trade-off:
| Sampling Strategy | Multimodal Navigation | High-Dimensional Scaling |
|---|---|---|
| Pure Global Proposals | Excellent (can jump between distant modes) | Poor (acceptance rates collapse as dimensions grow) |
| Pure Local Reparameterization | Poor (confined to a single local mode) | Excellent (exploits local geometry robustly) |
If you rely solely on a global proposal from a learned model, your acceptance rate will decay exponentially as the dimensionality of the space increases, because even a minor mismatch between the proposal and the true target distribution becomes catastrophic in high dimensions.
Hybrid sampling resolves this conflict. By alternating between the two strategies, the global mechanism is tasked only with providing mobility—the capacity to discover and transition between disconnected modes—while the local mechanism is tasked with precision—the capacity to map complex local geometries and satisfy detailed balance. This cooperative dynamic allows modern samplers to conquer distributions that are simultaneously highly multimodal and high-dimensional.