Abstract
Diffusion models have demonstrated remarkable efficacy in various generative tasks with the predictive prowess of denoising models. Currently, diffusion models employ a uniform denoising model across all timesteps. However, the inherent variations in data distributions at different timesteps lead to conflicts during training, constraining the potential of diffusion models. To address this challenge, we propose a novel two-stage divide-and-conquer training strategy termed TDC Training. It groups timesteps based on task similarity and difficulty, assigning highly customized denoising models to each group, thereby enhancing the performance of diffusion models. While two-stage training avoids the need to train each model separately, the total training cost is even lower than training a single unified denoising model. Additionally, we introduce Proxy-based Pruning to further customize the denoising models. This method transforms the pruning problem of diffusion models into a multi-round decision-making problem, enabling precise pruning of diffusion models. Our experiments validate the effectiveness of TDC Training, demonstrating improvements in FID of 1.5 on ImageNet64 compared to original IDDPM, while saving about 20% of computational resources.
Introduction: The "One-Size-Fits-All" Bottleneck
Standard diffusion models, such as Stable Diffusion or Improved Denoising Diffusion Probabilistic Models (IDDPM), typically utilize a single, static neural network to manage the entire denoising trajectory. This process involves a transition from high-entropy Gaussian noise (low Signal-to-Noise Ratio, high difficulty) to structured, nearly clean images (high SNR, low difficulty).
The uniform approach faces two primary limitations:
- Training Conflicts: Attempting to learn vastly different data distributions across thousands of timesteps within a single set of weights creates a "capacity burden." Optimization for one noise level may inadvertently degrade performance at another.
- Architectural Redundancy: High-capacity models are often overkill for the final, "easier" denoising steps, leading to unnecessary computational expenditure (GFLOPs).
TDC Training: A Divide-and-Conquer Strategy
To overcome these inefficiencies, we introduce Two-stage Divide-and-Conquer (TDC) Training. This framework replaces the monolithic model with a suite of specialized, progressively scaled denoising models.
The process begins by calculating the Signal-to-Noise Ratio (SNR) for every timestep, which serves as a proxy for task difficulty. Timesteps with similar SNR values are clustered into $N$ distinct groups.
- Stage 1 (Base Training): A standard "Base Model" is trained across the full spectrum of timesteps to establish a strong foundational representation.
- Stage 2 (Specialized Fine-tuning): For each identified group, the base model is pruned to an optimal size and fine-tuned exclusively on that group's specific noise levels. This specialization allows the model to resolve training conflicts and focus its capacity where it is most needed.
Proxy-based Pruning via LLM Reasoning
Determining the optimal pruning architecture for each timestep group is a complex search problem. We propose Proxy-based Pruning, which leverages the reasoning capabilities of Large Language Models (LLMs) like GPT-4.
Instead of traditional, resource-intensive Neural Architecture Search (NAS), we treat pruning as an iterative decision-making task. GPT-4 is provided with the model's architecture and historical performance data. It then proposes a pruning scheme (e.g., specific channels to remove). A Memory Bank tracks the results of these proposals, allowing the LLM to refine its strategy over multiple rounds. This approach bypasses the need for exhaustive manual tuning while achieving precise structural optimization.
Experimental Results
The TDC approach was evaluated on CIFAR10, ImageNet64, and FFHQ. Across all benchmarks, the method demonstrated superior efficiency and image quality.
Key Findings:
- Efficiency Gains: On ImageNet64, TDC achieved a 20% reduction in computational resources compared to the original IDDPM.
- Quality Improvements: Despite the reduced parameter count, image quality improved significantly. On ImageNet64, the Frechet Inception Distance (FID) dropped by 1.5 points (from 20.7 to 19.2).
- Latent Diffusion Scaling: When applied to Latent Diffusion Models (LDM) on the FFHQ dataset, TDC reduced FLOPs from 96.1G to 77.2G while simultaneously improving the FID from 5.0 to 4.73.
- Training Dynamics: Total training time remained competitive with baseline models. The specialized models in the second stage converge rapidly, often resulting in a total training budget lower than that of a single unified model.
Discussion and Conclusion
TDC Training demonstrates that model specialization is a powerful tool for scaling diffusion models. By aligning model capacity with task difficulty, we can achieve better generative performance with fewer resources. While the current implementation introduces some inference-time complexity due to weight switching, the substantial gains in FID and GFLOPs suggest that "Progressive Diffusion Models" are a promising direction for future research, particularly in high-resolution and video synthesis domains.
Summary updated