Not All Steps are Equal: Efficient Generation with Progressive Diffusion Models

Wenhao Li, Xiu Su, Shan You, Tao Huang, Fei Wang, Chen Qian, Chang Xu

Abstract

Diffusion models have demonstrated remarkable efficacy in various generative tasks with the predictive prowess of denoising models. Currently, these models employ a uniform denoising approach across all timesteps. However, the inherent variations in noisy latents at each timestep lead to conflicts during training, constraining the potential of diffusion models. To address this challenge, we propose a novel two-stage training strategy termed Step-Adaptive Training (or TDC: Two-stage Divide-and-Conquer). In the initial stage, a base denoising model is trained to encompass all timesteps. Subsequently, we partition the timesteps into distinct groups, fine-tuning the model within each group to achieve specialized denoising capabilities. Recognizing that the difficulties of predicting noise at different timesteps vary, we introduce a diverse model size requirement. We dynamically adjust the model size for each timestep by estimating task difficulty based on its signal-to-noise ratio before fine-tuning. This adjustment is facilitated by a proxy-based structural importance assessment mechanism, enabling precise and efficient pruning of the base denoising model. Our experiments validate the effectiveness of the proposed training strategy, demonstrating an improvement in the FID score on CIFAR10 by over 0.3 while utilizing only 80% of the computational resources.

Introduction: The Inefficiency of Uniform Denoising

Standard diffusion models, such as DDPM or Stable Diffusion, typically utilize a single neural network to manage the entire denoising trajectory—from pure Gaussian noise to a clean image. However, this "one-size-fits-all" approach ignores a fundamental reality of the diffusion process: not all timesteps are equal.

Early timesteps, characterized by high noise levels, are inherently difficult as the model must establish global structure. Conversely, late timesteps with low noise are significantly easier, requiring the model only to refine fine-grained details. Forcing a single model to learn these disparate distributions leads to "training conflicts" and significant computational redundancy. Current alternatives often involve training multiple separate models, which is prohibitively expensive, or relying on a single model that is simultaneously over-engineered for easy steps and under-powered for difficult ones.

The TDC Training Framework

To bridge this efficiency gap, we introduce TDC Training (Two-stage Divide-and-Conquer). This strategy replaces the monolithic denoising model with a suite of specialized, pruned models through a two-stage pipeline:

  1. Stage 1 (The Generalist): A single "base model" is trained across all timesteps to capture the general denoising logic and establish a foundational understanding of the data distribution.
  2. Stage 2 (The Specialists): Timesteps are partitioned into $N$ groups based on their Signal-to-Noise Ratio (SNR). Timesteps with similar difficulty levels are grouped together. The base model is then pruned to a specific size optimized for each group's difficulty and fine-tuned exclusively on those timesteps.
The TDC training pipeline, illustrating the two-stage process and the GPT-4-guided pruning loop.

Adaptive Pruning via LLM Expertise

A key innovation in our approach is the use of a Large Language Model (LLM), specifically GPT-4, as an "expert pruner." Determining the optimal structural pruning for various layers is a complex heuristic task. By providing GPT-4 with the model architecture, FLOPs constraints, and a "Memory Bank" of previous pruning attempts and their resulting performance, the LLM acts as an iterative decision-maker.

This proxy-based pruning mechanism allows the system to refine its strategy over multiple rounds. The difficulty of each timestep group is mathematically estimated using SNR; lower SNR indicates higher difficulty, necessitating a larger model capacity. By dynamically adjusting model size to match task difficulty, we eliminate the computational waste inherent in using full-sized models for late-stage refinement.

Experimental Results and Efficiency

We evaluated the TDC framework across three standard benchmarks: CIFAR10, ImageNet64, and FFHQ. The results consistently demonstrate that specialized "specialist" models outperform unified "generalist" models while requiring fewer resources.

  • CIFAR10: Achieved an FID improvement of over 0.3 while utilizing only 80% of the computational resources compared to the original IDDPM.
  • ImageNet64: Improved the FID score from 20.7 to 19.2, representing a significant jump in image quality alongside reduced parameter counts.
  • FFHQ (Latent Diffusion): Improved FID from 5.0 to 4.73 while reducing FLOPs from 96.1G to 77.2G.
FID score vs. GFLOPs on CIFAR10, demonstrating that TDC with pruning achieves superior image quality with significantly fewer computational resources.

Beyond raw metrics, the two-stage process accelerates convergence. By starting with a pre-trained base model, the specialized models retain global knowledge while avoiding the gradient conflicts that arise when a single model attempts to learn every noise level simultaneously.

Conclusion

By recognizing the varying difficulty of denoising timesteps, the TDC framework introduces a more nuanced and efficient approach to diffusion model training. The integration of SNR-based difficulty estimation and LLM-guided pruning allows for the creation of specialized models that are both more accurate and less computationally demanding. This approach not only enhances model performance but also significantly reduces the environmental and financial costs of training state-of-the-art generative models.

Summary updated

Abstract

Diffusion models have demonstrated remarkable efficacy in various generative tasks with the predictive prowess of denoising model. Currently, these models employ a uniform denoising approach across all timesteps. However, the inherent variations in noisy latents at each timestep lead to conflicts during training, constraining the potential of diffusion models. To address this challenge, we propose a novel two-stage training strategy termed Step-Adaptive Training. In the initial stage, a base denoising model is trained to encompass all timesteps. Subsequently, we partition the timesteps into distinct groups, fine-tuning the model within each group to achieve specialized denoising capabilities. Recognizing that the difficulties of predicting noise at different timesteps vary, we introduce a diverse model size requirement. We dynamically adjust the model size for each timestep by estimating task difficulty based on its signal-to-noise ratio before fine-tuning. This adjustment is facilitated by a proxy-based structural importance assessment mechanism, enabling precise and efficient pruning of the base denoising model. Our experiments validate the effectiveness of the proposed training strategy, demonstrating an improvement in the FID score on CIFAR10 by over 0.3 while utilizing only 80\% of the computational resources. This innovative approach not only enhances model performance but also significantly reduces computational costs, opening new avenues for the development and application of diffusion models.
BibTeX
@inproceedings{li2023not,
  author = {Wenhao Li and Xiu Su and Shan You and Tao Huang and Fei Wang and Chen Qian and Chang Xu},
  title = {{Not All Steps are Equal: Efficient Generation with Progressive Diffusion Models}},
  booktitle = {ICME},
  year = {2023},
}
Figure