TL;DR
Diffusion models are powerful but slow. This paper introduces "Adaptive Guidance" (AG), a training-free method that intelligently skips computationally expensive steps in the diffusion process, achieving up to 25% speedup without sacrificing image quality. They further propose "LINEARAG," which replaces some steps entirely with fast linear approximations, offering up to 75% speedup.
What Problem Does This Paper Address?
Generating high-quality images with diffusion models, especially those conditioned on text (like Stable Diffusion or DALL-E), is computationally intensive. A key component, Classifier-Free Guidance (CFG), significantly improves image quality but doubles the number of neural network evaluations (NFEs) required per denoising step. This makes sampling slow, especially on expensive hardware. While methods like "Guidance Distillation" exist to speed things up, they often require retraining and can't handle dynamic features like negative prompts. The authors aim to find a more efficient, training-free, and flexible way to accelerate CFG without compromising image fidelity.
Methods
The core idea is to treat the choice of guidance strategy at each denoising step as a search problem. The authors leverage differentiable Neural Architecture Search (NAS), specifically inspired by the DARTS framework. They unroll the diffusion process into a graph where each step is a node, and the "operations" at each node are different guidance strategies (unconditional score, conditional score, or CFG with varying strengths).
- Adaptive Guidance (AG): By analyzing the diffusion process, they observe that the conditional and unconditional score estimates become highly correlated in the later stages of denoising. This suggests that CFG's extra computation is redundant when the model is already close to convergence. AG adaptively switches from CFG to cheaper conditional-only steps when this correlation (measured by cosine similarity) exceeds a certain threshold. This is a "plug-and-play" solution that doesn't require retraining.
- LINEARAG: Further exploring redundancies, they find that in the earlier stages of denoising, where guidance is more critical, unconditional score evaluations can be approximated by simple affine transformations of past score estimates. They train linear regression models on pre-computed diffusion trajectories to learn these transformations. LINEARAG alternates between CFG steps and these fast, approximated unconditional steps, significantly reducing computation.
Datasets
- CC3M: Used to generate training data for the linear regression models in LINEARAG.
- OUI (Open User Input) dataset: Used for evaluating the learned policies.
- LDM-512 and EMU-768: These are pre-trained diffusion models used for experiments. LDM-512 (Stable Diffusion) was used for searching the guidance policies, and EMU-768 (a larger model) was used to validate the generalizability of the findings.
Key Insights and Ideas
- CFG Redundancy: The most surprising insight is that CFG's expensive computations become increasingly redundant as the diffusion process progresses. The later steps are less sensitive to the extra guidance.
- Correlation as a Signal: The high cosine similarity between conditional and unconditional scores is a strong indicator of convergence and a reliable signal to switch to cheaper guidance.
- Early vs. Late Stage Guidance: The paper clearly distinguishes the roles of guidance in the early (semantic structure) and late (fine details) stages of diffusion, leading to different optimization strategies for each.
- Linearity in Diffusion: The observation that unconditional score estimates can be well-approximated by linear combinations of past estimates is a significant finding, opening doors for further efficiency gains.
Key Results
- Adaptive Guidance (AG): Achieves up to 25% reduction in NFEs while preserving image quality comparable to standard CFG. Human evaluations showed no significant difference in visual appeal.
- LINEARAG: Achieves up to 75% reduction in NFEs by replacing early-stage CFG computations with linear approximations. While it deviates more from the baseline, it still produces sharper images with better contrast and more vivid colors compared to naive step reduction.
- Generalizability: The learned policies from LDM-512 generalized well to the larger EMU-768 model, demonstrating robustness.
- Image Editing: AG can also accelerate image editing tasks (like those using InstructPix2Pix) by up to 33.3%, as the same convergence principles apply.
Limitations and Future Work
- LINEARAG Deviation: LINEARAG deviates from the baseline CFG, meaning it doesn't perfectly replicate the output. While it offers significant speedups, its suitability depends on the acceptable level of deviation.
- Hyperparameter Tuning: AG introduces a new hyperparameter ($\gamma$) for the correlation threshold, which needs to be tuned.
- Future Work: The authors suggest that the linearity observed in diffusion paths is a promising area for future research, potentially leading to even more efficient inference methods. They also highlight the potential for applying these insights to other types of diffusion models and tasks.
Summary updated