Annealing Flow Generative Model Towards Sampling High-Dimensional and Multi-Modal Distributions

Dongze Wu, Yao Xie

Introduction

Sampling from complex probability distributions is a fundamental task in Bayesian inference, molecular dynamics, and physics-based machine learning. However, traditional methods face significant hurdles when dealing with high-dimensional spaces and multi-modal distributions—those with multiple isolated "peaks" of high probability. Standard Markov Chain Monte Carlo (MCMC) methods often fail to "mix" effectively, becoming trapped in a single mode. While Normalizing Flows (NFs) offer a promising alternative by mapping simple distributions to complex targets, they frequently suffer from mode collapse or training instability when the target density is highly non-convex.

The Annealing Flow (AF) Framework

To address these challenges, researchers have introduced Annealing Flow (AF). This novel generative model combines Continuous Normalizing Flows (CNF) with a structured annealing procedure and a dynamic Optimal Transport (OT) objective.

The core idea is to decompose the difficult task of mapping a simple Gaussian to a complex target into a sequence of manageable sub-tasks. AF defines a trajectory of intermediate distributions that gradually increase in complexity.

Conceptual comparison of flow-based annealing versus diffusion-based methods along an annealing trajectory.

As illustrated in the figure above, AF models the transformation between these intermediate distributions as a continuous fluid flow. Unlike diffusion-based methods that rely on stochastic, noisy paths, AF utilizes a Neural ODE to learn a smooth, deterministic velocity field. This flow is optimized using a Benamou-Brenier formulation of Optimal Transport, incorporating Wasserstein-2 regularization to ensure the paths taken are the most efficient (shortest) possible.

Annealing as a Curriculum

The annealing process serves as a "curriculum" for the neural network. By learning to sample flatter, easier versions of the distribution first, the model builds a robust representation before tackling the sharp modes of the final target.

The annealing procedure transitioning from a simple unimodal Gaussian (β0=0) to a complex six-mode target distribution (β3=1).

This block-wise training approach allows AF to bridge consecutive annealing steps effectively. A key theoretical finding of the work is that in the limit, the optimal velocity of this flow is directly related to the difference between the score functions (gradients of the log-density) of the consecutive distributions, providing a clear mathematical foundation for the model's success.

Key Results and Performance

The efficacy of AF was demonstrated across several rigorous benchmarks:

  • Extreme Multi-modality: In a 50-dimensional test involving an "Exp-Weighted Gaussian" with 1,024 modes, AF successfully captured every single mode. In contrast, competing state-of-the-art methods like CRAFT and LFIS missed hundreds of modes.
  • Efficiency and Speed: AF achieves superior results with significantly fewer annealing steps—often requiring only 12 steps compared to the 128+ steps needed by traditional annealing-based NFs. Once trained, the model is exceptionally fast, generating 10,000 samples in approximately 2.1 seconds for 50D experiments.
  • Real-World Application: On Bayesian Logistic Regression tasks using datasets from the LIBSVM repository, AF consistently matched or outperformed established baselines in terms of classification accuracy and log-posterior values.

Conclusion

Annealing Flow represents a significant step forward in sampling technology. By integrating Optimal Transport with continuous flows and annealing, it provides a stable and efficient path for exploring high-dimensional, multi-modal landscapes. While the model currently requires offline pre-training and knowledge of the analytical target density, its ability to avoid mode collapse and its independence from heavy MCMC assistance make it a powerful tool for modern probabilistic modeling. Future research aims to extend this framework to distribution-free models, potentially impacting fields ranging from adversarial robustness to differential privacy.

Summary updated

BibTeX
@inproceedings{wu2025annealing,
  author = {Dongze Wu and Yao Xie},
  title = {{Annealing Flow Generative Model Towards Sampling High-Dimensional and Multi-Modal Distributions}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=iT8qIzNSAl},
}
Figure