Abstract
Diffusion Probabilistic Models (DPMs) have achieved state-of-the-art results in image synthesis, yet their sampling process remains a bottleneck, often struggling to balance global structure with fine-grained detail. This paper introduces the Adaptive Momentum (AM) Sampler, a training-free method that treats the reverse sampling process as an optimization problem. By incorporating momentum and adaptive step sizes—techniques inspired by the Adam optimizer—the AM Sampler stabilizes sampling trajectories and reduces the "train-test gap." We demonstrate that the AM Sampler significantly improves image quality, as measured by Fréchet Inception Distance (FID), across multiple benchmarks including CelebA and LSUN, particularly in low-step-count scenarios.
Introduction
The sampling process in Diffusion Probabilistic Models (DPMs), which iteratively transforms noise into a structured image, is mathematically analogous to Stochastic Gradient Descent (SGD). However, just as vanilla SGD can suffer from slow convergence or instability in complex loss landscapes, standard diffusion samplers often fail to perfectly capture both high-level semantics and low-level textures. This discrepancy, often termed the "train-test gap," limits the efficiency and quality of pre-trained models during inference. The authors propose a "plug-and-play" solution that enhances existing models without the need for costly retraining.
Methodology: The Adaptive Momentum (AM) Sampler
The core innovation of this work is the Adaptive Momentum (AM) Sampler, which adapts the principles of the Adam optimizer to the diffusion sampling trajectory. The method introduces two key components:
- Momentum ($m_t$): By maintaining a "velocity" vector of previous update directions, the sampler stabilizes the trajectory. This ensures that the high-level structural information established in early sampling stages is preserved and refined rather than lost to noise.
- Adaptive Updating ($v_t$): Similar to RMSProp, the sampler tracks a moving average of squared increments to automatically adjust the step size for each pixel. This allows for more precise updates tailored to local image features.
The AM Sampler serves as a second-order approximation of the underlying differential equations, providing a more robust theoretical foundation than standard first-order solvers. Because it is training-free, it can be seamlessly integrated into existing frameworks like Latent Diffusion Models (LDM).
Experimental Results
The AM Sampler was rigorously tested on datasets including CIFAR-10, CelebA, and LSUN (Church and Bedroom). The results show consistent improvements in image fidelity and diversity.
Quantitative Performance
The most significant gains were observed in Fréchet Inception Distance (FID) scores. For instance, on the CelebA dataset, the AM Sampler reduced the FID of a standard DDPM from 5.78 to 2.50—an improvement of over 50%. On high-resolution LSUN Church images, the FID improved from 3.99 to 3.92 when using Latent Diffusion Models. The sampler is especially effective at low sampling steps (e.g., 25-50), making it ideal for high-speed generation scenarios.
Qualitative Analysis
Qualitative results highlight the sampler's ability to resolve fine-grained details that standard methods might blur or misinterpret. By stabilizing the update trajectory, the AM Sampler ensures that architectural features and textures are rendered with higher fidelity.
Insights and Discussion
The effectiveness of the AM Sampler suggests that the sampling trajectory is best viewed as an optimization path. Key insights include:
- Stability of $x_0$: The stability of the model's final image prediction ($x_0$) during sampling allows the momentum buffer to capture meaningful trends.
- Trajectory Smoothing: Rate-distortion analysis confirms that the AM Sampler "straightens" the sampling curve, allocating model capacity more efficiently toward meaningful image structures.
- Robustness: The method's ability to escape local traps in the probability landscape leads to more accurate and diverse final outputs.
While the introduction of momentum hyperparameters requires some initial tuning, the provided defaults offer strong performance across diverse datasets. Future research may extend these principles to continuous-time sampling or incorporate adaptive momentum into the training phase itself.
Summary updated