Interleaving Global Jumps and Local Refinement for Multimodal Sampling

This direction is merged into another direction: hybrid global-local sampling. Its last version stays readable and citable here.

Version 1 of 1150 papers192 conceptsfrozen 2026-10-04

Sampling from complex probability distributions is a fundamental challenge in Bayesian inference and generative modeling. The central difficulty is multimodality: the distribution may have multiple "islands" of high probability separated by vast "oceans" of near-zero probability.

Standard gradient-based samplers, like Langevin Dynamics or Hamiltonian Monte Carlo (HMC), are excellent at exploring the local terrain of a single island. However, because they rely on local information, they are often "blind" to other islands. Conversely, global generative models like Normalizing Flows can learn to jump between islands but often struggle to capture the precise local geometry. A growing body of research focuses on hybridizing these two scales to create samplers that are both globally aware and locally precise.

The Blindness of Local Exploration

Most traditional Markov Chain Monte Carlo (MCMC) methods move by taking small steps based on the local gradient of the target density. While efficient, these methods suffer from a "trap" effect. As Jacopo Guidolin and Ondrej Kuzelka note in Jump-Diffusion Langevin Dynamics for Multimodal Posterior Sampling, gradient-only methods become trapped in the nearest high-probability mode because the probability of stepping through a low-density "energy barrier" to find another mode is exponentially small.

To overcome this, researchers have introduced non-local mechanisms. Yulong Lu and colleagues in Accelerating Langevin Sampling with Birth-death propose a birth-death process. Instead of forcing a single particle to travel across a desert, they allow particles in low-density regions to "die" while particles in high-density regions "give birth" to clones. This redistributes the particle mass across the distribution without requiring any single particle to traverse a low-probability barrier.

Global Proposals as Teleportation

If local samplers are hikers, global proposals are teleporters. By using a learned model—often a Normalizing Flow or a Neural Network—the sampler can propose a jump to a completely different part of the state space.

In Efficient Bayesian Sampling Using Normalizing Flows to Assist Markov Chain Monte Carlo Methods, Marylou Gabrié and her co-authors demonstrate a synergistic loop: a local MCMC sampler explores a mode and provides data to a Normalizing Flow, which then learns the global structure and proposes "non-local jumps" to other modes. This ensures the sampler can escape local optima while maintaining the detailed balance required for unbiased sampling.

Similarly, Evgeny Lagutin and the team behind Ex 2 MCMC: Sampling through Exploration Exploitation formalize this as an "interleaving" strategy. They alternate between:

  1. Exploration: Global jumps (often learned on-the-fly) to discover distant modes.
  2. Exploitation: Local gradient-based refinement to map the discovered mode's geometry.

Bridging the Gap with Annealing and Transport

A major risk in using global models is "mode collapse," where the model only learns a subset of the islands. To prevent this, researchers use annealing—gradually transforming a simple distribution (like a Gaussian) into the complex target distribution.

Michael Arbel and Arnaud Doucet introduced Annealed Flow Transport Monte Carlo, which combines global transport maps with local MCMC "mutation" steps. The flow moves the entire population of particles toward new high-probability regions, while the local MCMC steps act as a refinement filter, ensuring that the particles explore the local neighborhood and recover any modes the flow might have missed.

This dual-stage update is also central to Importance Corrected Neural JKO Sampling by Johannes Hertrich. By alternating between continuous flows for local refinement and importance-based rejection-resampling for global mass correction, the algorithm ensures that probability mass is correctly distributed across disconnected peaks, preventing the "winner-take-all" behavior common in standard variational methods.

Multi-Scale Fidelity and Ensembles

Another facet of this direction involves using multiple "fidelities" or "temperatures" to explore the landscape. Yating Wang and Guang Lin 0001 developed a multi-variance replica exchange method for Bayesian Physics-Informed Neural Networks. They use a "cheap" coarse solver for global exploration and a high-fidelity solver for local refinement. By swapping information between these chains, they can identify entire manifolds of valid solutions that single-chain methods would miss.

This ensemble-based intuition is echoed in Turing: Composable inference for probabilistic programming by Hong Ge and Zoubin Ghahramani. Their framework allows for composable inference, where different algorithms (like HMC for continuous local exploration and Particle Gibbs for discrete global moves) are mixed within a single model to handle disjoint state spaces.

Why It Works: The Synergy of Scales

The success of these hybrid methods stems from a simple trade-off:

  • Local samplers have high "resolution" but low "range." They are statistically rigorous but geographically limited.
  • Global models have high "range" but low "resolution." They can see the whole map but often blur the fine details.

By interleaving them, the global model provides the "blueprint" or "teleportation targets," while the local sampler provides the "ground truth" refinement. As shown in On Sampling with Approximate Transport Maps by Louis Grenioux and Eric Moulines, this combination is far more robust than either method alone, especially in high-dimensional spaces where the learned global flow is inevitably imperfect.

Papers in this version

Showing 60 of 150 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_interleaving_global_jumps_and_local_refi_ca509e2b,
  title = {Interleaving Global Jumps and Local Refinement for Multimodal Sampling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/interleaving-global-jumps-and-local-refinement-for-multimodal-sampling/rdr_ca509e2be47e46ca9e4ffdef37d00993},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_interleaving_global_jumps_and_local_refi_ca509e2b_v1,
  title = {Interleaving Global Jumps and Local Refinement for Multimodal Sampling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/interleaving-global-jumps-and-local-refinement-for-multimodal-sampling/rdr_ca509e2be47e46ca9e4ffdef37d00993/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}