Adaptive Guidance: Training-free Acceleration of Conditional Diffusion Models

Angela Castillo, Jonas Kohler, Juan C. Pérez, Albert Pumarola, Bernard Ghanem, Pablo Arbeláez, Ali K. Thabet

TL;DR

Diffusion models are powerful but slow. This paper introduces "Adaptive Guidance" (AG), a training-free method that intelligently skips computationally expensive steps in the diffusion process, achieving up to 25% speedup without sacrificing image quality. They further propose "LINEARAG," which replaces some steps entirely with fast linear approximations, offering up to 75% speedup.

What Problem Does This Paper Address?

Generating high-quality images with diffusion models, especially those conditioned on text (like Stable Diffusion or DALL-E), is computationally intensive. A key component, Classifier-Free Guidance (CFG), significantly improves image quality but doubles the number of neural network evaluations (NFEs) required per denoising step. This makes sampling slow, especially on expensive hardware. While methods like "Guidance Distillation" exist to speed things up, they often require retraining and can't handle dynamic features like negative prompts. The authors aim to find a more efficient, training-free, and flexible way to accelerate CFG without compromising image fidelity.

Methods

The core idea is to treat the choice of guidance strategy at each denoising step as a search problem. The authors leverage differentiable Neural Architecture Search (NAS), specifically inspired by the DARTS framework. They unroll the diffusion process into a graph where each step is a node, and the "operations" at each node are different guidance strategies (unconditional score, conditional score, or CFG with varying strengths).

  1. Adaptive Guidance (AG): By analyzing the diffusion process, they observe that the conditional and unconditional score estimates become highly correlated in the later stages of denoising. This suggests that CFG's extra computation is redundant when the model is already close to convergence. AG adaptively switches from CFG to cheaper conditional-only steps when this correlation (measured by cosine similarity) exceeds a certain threshold. This is a "plug-and-play" solution that doesn't require retraining.
  2. LINEARAG: Further exploring redundancies, they find that in the earlier stages of denoising, where guidance is more critical, unconditional score evaluations can be approximated by simple affine transformations of past score estimates. They train linear regression models on pre-computed diffusion trajectories to learn these transformations. LINEARAG alternates between CFG steps and these fast, approximated unconditional steps, significantly reducing computation.

Datasets

  • CC3M: Used to generate training data for the linear regression models in LINEARAG.
  • OUI (Open User Input) dataset: Used for evaluating the learned policies.
  • LDM-512 and EMU-768: These are pre-trained diffusion models used for experiments. LDM-512 (Stable Diffusion) was used for searching the guidance policies, and EMU-768 (a larger model) was used to validate the generalizability of the findings.

Key Insights and Ideas

  • CFG Redundancy: The most surprising insight is that CFG's expensive computations become increasingly redundant as the diffusion process progresses. The later steps are less sensitive to the extra guidance.
  • Correlation as a Signal: The high cosine similarity between conditional and unconditional scores is a strong indicator of convergence and a reliable signal to switch to cheaper guidance.
  • Early vs. Late Stage Guidance: The paper clearly distinguishes the roles of guidance in the early (semantic structure) and late (fine details) stages of diffusion, leading to different optimization strategies for each.
  • Linearity in Diffusion: The observation that unconditional score estimates can be well-approximated by linear combinations of past estimates is a significant finding, opening doors for further efficiency gains.

Key Results

  • Adaptive Guidance (AG): Achieves up to 25% reduction in NFEs while preserving image quality comparable to standard CFG. Human evaluations showed no significant difference in visual appeal.
  • LINEARAG: Achieves up to 75% reduction in NFEs by replacing early-stage CFG computations with linear approximations. While it deviates more from the baseline, it still produces sharper images with better contrast and more vivid colors compared to naive step reduction.
  • Generalizability: The learned policies from LDM-512 generalized well to the larger EMU-768 model, demonstrating robustness.
  • Image Editing: AG can also accelerate image editing tasks (like those using InstructPix2Pix) by up to 33.3%, as the same convergence principles apply.

Limitations and Future Work

  • LINEARAG Deviation: LINEARAG deviates from the baseline CFG, meaning it doesn't perfectly replicate the output. While it offers significant speedups, its suitability depends on the acceptable level of deviation.
  • Hyperparameter Tuning: AG introduces a new hyperparameter ($\gamma$) for the correlation threshold, which needs to be tuned.
  • Future Work: The authors suggest that the linearity observed in diffusion paths is a promising area for future research, potentially leading to even more efficient inference methods. They also highlight the potential for applying these insights to other types of diffusion models and tasks.

Summary updated

Abstract

This paper presents a comprehensive study on the role of Classifier-Free Guidance (CFG) in text-conditioned diffusion models from the perspective of inference efficiency. In particular, we relax the default choice of applying CFG in all diffusion steps and instead search for efficient guidance policies. We formulate the discovery of such policies in the differentiable Neural Architecture Search framework. Our findings suggest that the denoising steps proposed by CFG become increasingly aligned with simple conditional steps, which renders the extra neural network evaluation of CFG redundant, especially in the second half of the denoising process. Building upon this insight, we propose "Adaptive Guidance" (AG), an efficient variant of CFG, that adaptively omits network evaluations when the denoising process displays convergence. Our experiments demonstrate that AG preserves CFG's image quality while reducing computation by 25%. Thus, AG constitutes a plug-and-play alternative to Guidance Distillation, achieving 50% of the speed-ups of the latter while being training-free and retaining the capacity to handle negative prompts. Finally, we uncover further redundancies of CFG in the first half of the diffusion process, showing that entire neural function evaluations can be replaced by simple affine transformations of past score estimates. This method, termed LinearAG, offers even cheaper inference at the cost of deviating from the baseline model. Our findings provide insights into the efficiency of the conditional denoising process that contribute to more practical and swift deployment of text-conditioned diffusion models.
BibTeX
@inproceedings{castillo2025adaptive,
  author = {Angela Castillo and Jonas Kohler and Juan C. Pérez and Juan Pablo Pérez and Albert Pumarola and Bernard Ghanem and Pablo Arbeláez and Ali K. Thabet},
  title = {{Adaptive Guidance: Training-free Acceleration of Conditional Diffusion Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i2.32192},
}
Figure