Denoising Diffusion Step-aware Models

Shuai Yang, Yukang Chen, Luozhou Wang, Shu Liu, Yingcong Chen

TL;DR

Denoising Diffusion Step-aware Models (DDSM) significantly accelerate image generation by using variable-sized neural networks for different denoising steps. By tailoring model capacity to the specific importance of each step via evolutionary search, DDSM achieves up to 76% computational savings without sacrificing—and sometimes even improving—image quality.

What Problem Does This Paper Address?

Denoising Diffusion Probabilistic Models (DDPMs) have become the gold standard for high-quality image synthesis. However, their iterative nature is a major bottleneck: generating a single image requires hundreds or thousands of passes through a massive neural network.

While existing methods like DDIM focus on skipping steps entirely, DDSM asks a different question: Do all diffusion steps require the same amount of "brainpower"?

Standard models use the same heavy U-Net for every single step, from the initial coarse structure formation to the final fine-grained detailing. The authors hypothesize that this leads to massive computational redundancy, as some steps are inherently "easier" or less critical to the final output than others.

Methods

The DDSM framework introduces two main components to optimize per-step computation:

  1. Slimmable Networks: Instead of training multiple separate models of different sizes, the authors use a "slimmable" U-Net. This is a single supernet trained such that it can be executed at various widths (by contiguous channel pruning) without needing re-training. During training, the largest, smallest, and a random sub-network are optimized simultaneously to ensure all possible widths perform well.
  2. Step-aware Evolutionary Search: To determine which model size to use at which step, the authors employ a multi-objective evolutionary algorithm (NSGA-II). The search balances two conflicting goals: minimizing total FLOPs (computational cost) and maximizing image quality (measured by Fréchet Inception Distance, or FID).
The overall DDSM procedure, showing slimmable network training followed by an evolutionary search to find the optimal step-aware strategy.

Datasets

The authors evaluated DDSM across five diverse image datasets to prove its generalizability:

  • CIFAR-10: Small-scale natural images.
  • CelebA-HQ (64x64 & 128x128): High-quality human faces.
  • LSUN-bedroom: Non-object-centric indoor scenes.
  • AFHQ: Animal faces.
  • ImageNet (64x64): Large-scale, highly diverse natural images.

Key Insights and Ideas

  • Step Importance is Real: A pilot study showed that using a tiny model for just 250 out of 1000 steps can either maintain quality or cause it to collapse, depending entirely on which 250 steps are pruned.
  • Dataset-Specific Strategies: The "optimal" strategy depends heavily on the dataset's attributes.
  • For ImageNet/CIFAR-10 (high diversity), the model can be small at the start because "any" coarse structure is acceptable, but it needs to be large at the end to get specific details (like bird wings or car wheels) right.
  • For CelebA-HQ (highly structured faces), the model must be large at the start to ensure the rigid face layout (eyes, nose, mouth) is correct, but it can be smaller at the end for simple color interpolation.
  • Orthogonality: DDSM is a "plug-and-play" module. It can be combined with step-skipping methods like DDIM or dimension-reduction methods like Latent Diffusion (LDM) to provide additive speedups.
Comparison of proper vs. improper model combinations. The right choice of when to use a small model saves computation without ruining the image.

Key Results

DDSM achieves dramatic efficiency gains across the board compared to the standard ADM (Ablated Diffusion Model) baseline:

  • Computational Savings: 49% for CIFAR-10, 61% for CelebA-HQ, 59% for LSUN-bedroom, 71% for AFHQ, and a staggering 76% for ImageNet.
  • Actual Speedup: On an NVIDIA RTX 3090, DDSM reduced the latency for generating a CelebA-HQ image from 2.25s to 1.13s.
  • Quality Preservation: In most cases, DDSM achieved these savings while maintaining or slightly improving the FID score compared to the full-sized baseline.

Limitations and Future Work

  • One-time Overhead: DDSM requires extra training time for the slimmable supernet (roughly 2-3x a normal training run) and a search phase (roughly equal to one training run). However, the authors argue this is a worthwhile trade-off for models intended for massive-scale inference.
  • Theoretical Grounding: While empirical results are strong, the authors note that more theoretical work is needed to fully understand the mathematical responsibility of each diffusion step.
  • Future Directions: The authors suggest exploring even more fine-grained pruning (e.g., per-block or per-layer) and applying the framework to text-to-image models like Stable Diffusion.

Summary updated

Abstract

Denoising Diffusion Probabilistic Models (DDPMs) have garnered popularity for data generation across various domains. However, a significant bottleneck is the necessity for whole-network computation during every step of the generative process, leading to high computational overheads. This paper presents a novel framework, Denoising Diffusion Step-aware Models (DDSM), to address this challenge. Unlike conventional approaches, DDSM employs a spectrum of neural networks whose sizes are adapted according to the importance of each generative step, as determined through evolutionary search. This step-wise network variation effectively circumvents redundant computational efforts, particularly in less critical steps, thereby enhancing the efficiency of the diffusion model. Furthermore, the step-aware design can be seamlessly integrated with other efficiency-geared diffusion models such as DDIMs and latent diffusion, thus broadening the scope of computational savings. Empirical evaluations demonstrate that DDSM achieves computational savings of 49% for CIFAR-10, 61% for CelebA-HQ, 59% for LSUN-bedroom, 71% for AFHQ, and 76% for ImageNet, all without compromising the generation quality.
BibTeX
@inproceedings{yang2024denoising,
  author = {Shuai Yang and Yukang Chen and Luozhou Wang and Shu Liu and Yingcong Chen},
  title = {{Denoising Diffusion Step-aware Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=c43FGk8Pcg},
}
Figure