Diffusion models have revolutionized generative AI, but their iterative nature makes them computationally expensive. To generate an image or video, these models must simulate a reverse denoising process across dozens or hundreds of sequential timesteps, passing data through massive neural networks—such as U-Nets or Diffusion Transformers (DiTs)—at every single step. This creates a dual bottleneck: training requires massive computational resources over days or weeks, and real-time inference is hindered by high latency.
To address this, a prominent research direction focuses on accelerating convergence in diffusion models. Rather than relying purely on brute-force scaling, this paradigm re-engineers how information flows through the model's architecture and stabilizes the mathematical objectives driving optimization.
Structural Shortcuts: Skipping Layers and Tokens
One of the most direct ways to accelerate a model is to avoid redundant computation. In deep architectures like Diffusion Transformers, adjacent layers and sequential timesteps often compute highly correlated features. By introducing structural shortcuts, models can bypass unnecessary layers entirely.
For instance, Guanjie Chen et al. introduced Skip-DiT in Towards Stabilized and Efficient Diffusion Transformers through Long-Skip-Connections with Spectral Constraints, which leverages Long-Skip-Connections (LSCs) combined with spectral constraints to stabilize and accelerate both training (up to 4.4x) and inference. Similarly, Bin Xia demonstrated in DreamOmni: Unified Image Generation and Editing that integrating residual connections into DiT architectures significantly speeds up training convergence while maintaining model scalability.
Beyond static structural shortcuts, computation can be skipped dynamically at the token level. The training strategy proposed in TREAD: Token Routing for Efficient Architecture-agnostic Diffusion Training routes a subset of tokens to skip intermediate layers entirely during training, yielding up to a 37x speedup without altering the underlying architecture.
During inference, this representation redundancy can be exploited via training-free feature caching. Because the latent states change gradually across consecutive denoising steps, we can reuse features from previous steps rather than recomputing them. Zichao Yu et al. developed AB-Cache in AB-Cache: Training-Free Acceleration of Diffusion Models via Adams-Bashforth Cached Feature Reuse, reusing cached features based on the mathematical properties of the denoising process to achieve a 3x speedup. In a similar vein, Pengtao Chen et al. proposed $\Delta$-DiT: Accelerating Diffusion Transformers without training via Denoising Property Alignment, which caches intermediate feature differences ($\Delta$) to align with the inherent denoising trajectory, accelerating 20-step generation tasks by 1.6x.
Target Stabilization and Clean-State Reparameterization
While structural shortcuts optimize the computational path, another critical bottleneck lies in the training objective itself. Standard diffusion training forces the network to predict the noise injected at a specific timestep. Because this noise is highly stochastic, the training targets exhibit high variance, leading to slow and erratic convergence.
To stabilize this process, Yilun Xu et al. proposed the Stable Target Field (STF) in Stable Target Field for Reduced Variance Score Estimation in Diffusion Models. By utilizing a reference batch and importance sampling, STF drastically reduces the variance of training targets, accelerating wall-clock convergence by over 3x.
An alternative approach to variance reduction is clean-state reparameterization. Instead of predicting the highly variable noise added at each step, the model is trained to directly predict the final, fully denoised state at every step. Shuyang Gu et al. utilized this in Vector Quantized Diffusion Model for Text-to-Image Synthesis, showing that predicting the final clean image allows the system to skip multiple steps during inference, resulting in a 15-fold increase in generation speed.
This concept of cross-timestep consistency is pushed further in distillation frameworks. Simian Luo et al. introduced Latent Consistency Models (LCMs) in Latent Consistency Models: Synthesizing High-Resolution Images with Few-step Inference, utilizing a "skipping-step" technique. By enforcing consistency between timesteps that are far apart rather than just adjacent steps, this technique dramatically accelerates convergence during distillation, enabling high-quality synthesis in just 2 to 4 inference steps.
Algorithmic and Difficulty-Weighted Acceleration
The drive toward faster convergence in diffusion models mirrors broader advancements in deep learning optimization, where training efficiency is enhanced by focusing computation where it matters most.
For example, in spatial reconstruction tasks, Wei Lin et al. demonstrated in Pixel-Level Self-Paced Learning for Super-Resolution that dynamically weighting individual pixels based on their reconstruction difficulty cuts training time nearly in half. This philosophy of prioritizing high-difficulty regions aligns with the block-wise causal processing introduced by Mahdi Karami et al. in Auto-Regressive Masked Diffusion Models, which unifies autoregressive and diffusion frameworks to significantly reduce training iterations.
Finally, optimization-level adjustments provide an orthogonal boost to convergence. Techniques like Latest Weight Averaging (LAWA), proposed by Jean Kaddour in Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averaging, average the weights of recent checkpoints to accelerate training without sacrificing performance. Additionally, Manuel Graca et al. showed in Accelerated Gradient Descent for Faster Convergence with Minimal Overhead that incorporating local curvature information can reduce the number of required training epochs by an average of 33%.
Key Foundational Work
Optimization of CNN through Novel Training Strategy for Visual Classification Problems (2018) — contributed 1 concept elements to this direction.
Diagonal rnns in symbolic music modeling (2017) — contributed 1 concept elements to this direction.