Navigating with Annealing Guidance Scale in Diffusion Space

Shai Yehezkel, Omer Dahary, Andrey Voynov, Daniel Cohen-Or

Abstract

Modern text-to-image diffusion models rely heavily on Classifier-Free Guidance (CFG) to ensure that generated images align with user prompts. However, the standard approach uses a fixed guidance scale, which often forces a trade-off between image fidelity and prompt adherence, frequently resulting in over-saturation or structural artifacts. This paper introduces an annealing guidance scheduler, a lightweight, learned policy that dynamically adjusts the guidance scale throughout the denoising process. By treating guidance as a navigation problem and utilizing a "discrepancy signal" as a compass, the proposed method achieves superior visual quality and prompt alignment across benchmarks like MSCOCO and PartiPrompts, all while adding negligible computational overhead.

Introduction

In the current landscape of generative AI, text-to-image models have reached unprecedented levels of realism. Central to this success is Classifier-Free Guidance (CFG), a technique that pushes the model's predictions away from an unconditional (generic) output and toward a conditional (prompted) output using a fixed multiplier, the guidance scale ($w$).

Despite its ubiquity, CFG is notoriously difficult to tune. A low scale often results in images that ignore the prompt, while a high scale leads to "over-baked" images characterized by unnatural colors and distorted geometry. Finding the optimal scale is currently a manual process that varies significantly depending on the prompt, the random seed, and the specific model architecture. This paper argues that a fixed scale is fundamentally insufficient because the high-dimensional latent space the model navigates is non-uniform; different stages of the denoising process require different levels of guidance.

The Annealing Guidance Scheduler

To address the limitations of fixed scaling, the authors propose a learning-based scheduler that treats guidance as a dynamic navigation task. This approach builds upon CFG++, a variant that frames image sampling as an optimization problem aimed at minimizing the Score Distillation Sampling (SDS) loss.

Architecture and Inputs

The core of the method is a lightweight Multi-Layer Perceptron (MLP) with only 52,000 parameters. This MLP is trained to predict the optimal guidance scale at every step of the denoising trajectory based on three primary inputs:

  1. Timestep ($t$): The current stage in the generation process, from initial noise to final image.
  2. Discrepancy Signal ($||\delta_t||$): The norm of the difference between conditional and unconditional noise predictions. This acts as a "GPS signal," indicating how well the current latent state aligns with the prompt.
  3. User Preference ($\lambda$): A high-level control knob (ranging from 0 to 1) that allows users to prioritize either image fidelity or strict prompt adherence.

Training Objectives

The scheduler is trained to balance two competing objectives: an $\epsilon$-loss, which ensures the image remains on the natural image manifold to avoid artifacts, and a $\delta$-loss, which drives the generation toward regions where the prompt is satisfied. By learning from a curated subset of 20,000 high-resolution images from the LAION-POP dataset, the scheduler learns to navigate the latent space more effectively than any fixed-rule heuristic.

Key Insights

The Discrepancy Signal as a Compass

A pivotal insight of this research is that the discrepancy signal ($||\delta_t||$) serves as a reliable proxy for the gradient of the SDS loss. When this value is small, it suggests the image is already well-aligned with the prompt, allowing the scheduler to reduce guidance and focus on image quality. Conversely, a large discrepancy signals a need for stronger guidance to pull the generation back toward the prompt's intent.

Non-Monotonic Scaling

Unlike traditional manual schedulers that often use simple linear decay, the learned annealing scheduler frequently exhibits non-monotonic behavior. It may increase or decrease the guidance scale dynamically as the model encounters "difficult" regions of the latent space, providing a level of nuance that fixed scales cannot match.

Lifting Constraints

While previous theoretical frameworks suggested keeping the guidance scale between 0 and 1, the authors found that "lifting" this constraint allows the scheduler to explore higher values. This exploration leads to significantly better prompt adherence without the typical quality degradation associated with high-scale CFG.

Experimental Results

The annealing scheduler was benchmarked against standard CFG, CFG++, and Adaptive Pseudo-Guidance (APG) using MSCOCO 2017 and PartiPrompts.

  • Quantitative Performance: On MSCOCO, the method achieved the lowest Fréchet Inception Distance (FID), indicating higher image quality, and the highest CLIP similarity scores, indicating better prompt alignment. For instance, in high-alignment configurations, it improved FID from 27.15 (standard CFG) to 26.40.
  • Qualitative Robustness: The scheduler excels at traditionally difficult tasks for diffusion models, such as maintaining correct object counts (e.g., "two dogs, one cat") and rendering complex spatial relationships. It also significantly reduces anatomical errors, such as the "spaghetti fingers" often seen in high-guidance generations.
  • Efficiency: Because the MLP is extremely small and only runs once per denoising step, it provides these improvements at virtually zero additional cost to inference time or memory usage.

Conclusion and Future Work

The annealing guidance scheduler represents a significant step toward more autonomous and robust diffusion models. By replacing manual tuning with a learned, adaptive policy, the authors have demonstrated that models can navigate the complexities of latent space more effectively. While the current scheduler is somewhat model-specific—performing best when trained on the target model's specific latent space—future research may focus on universal schedulers or integrating this adaptive logic directly into the initial training of diffusion models and video generation frameworks.

Summary updated

BibTeX
@inproceedings{yehezkel2025navigating,
  author = {Shai Yehezkel and Omer Dahary and Andrey Voynov and Daniel Cohen-Or},
  title = {{Navigating with Annealing Guidance Scale in Diffusion Space}},
  booktitle = {SIGGRAPH Asia},
  year = {2025},
  doi = {10.1145/3757377.3763830},
}
Figure