Introduction
Classifier-Free Guidance (CFG) has become a cornerstone of modern generative AI, serving as the primary mechanism to ensure diffusion models strictly adhere to user prompts. However, this control comes at a significant "diversity tax": as guidance increases, generated images often become repetitive and "over-baked."
Existing theoretical understandings of CFG typically rely on unimodal assumptions—treating data as a single, simple distribution. In reality, a prompt like "a dog" encompasses a vast multimodal landscape, from a Golden Retriever in a park to a Bulldog on a couch. This research addresses the lack of a theoretical framework for how CFG interacts with these multimodal distributions, identifying exactly how and when diversity is lost during the generation process.
Theoretical Framework and Methods
By modeling conditional distributions as Gaussian Mixtures and analyzing the Probability Flow ODE, the authors decompose the sampling process into three distinct stages:
- Direction Shift (Early stage): At high noise levels, CFG pulls all potential image paths toward a single "weighted mean" of all possible outcomes, creating a significant initialization bias.
- Mode Separation (Intermediate stage): As noise decreases, paths should naturally diverge into different categories (modes). While CFG does not alter the boundaries of these categories, the bias from Stage 1 has already funneled most paths toward a single dominant mode.
- Concentration (Late stage): At low noise levels, CFG acts as a vacuum, pulling images tightly toward the center of a mode, which eliminates fine-grained diversity such as varying textures or poses.
Based on these dynamics, the researchers propose TV-CFG (Time-Varying CFG). This scheduling method applies low guidance initially to preserve global diversity, ramps up in the middle to ensure mode selection, and tapers off at the end to maintain fine-grained details.
Key Insights
- Initialization Bias: The loss of diversity is primarily driven by the "Direction Shift" in the earliest steps of sampling, rather than a failure of the model to recognize different modes later on.
- Local Neutrality: During the intermediate "Mode Separation" stage, CFG is surprisingly neutral regarding mode boundaries; it simply accelerates the movement toward the already-biased destinations.
- The Bell Curve Schedule: A "low-high-low" guidance schedule provides a superior balance between prompt alignment and image variety compared to the standard constant guidance approach.
Results and Validation
The researchers validated TV-CFG using Stable Diffusion v3.5 and the COCO 2017 dataset. Their findings indicate that time-varying schedules consistently outperform vanilla CFG in several metrics:
- Quality-Diversity Balance: TV-CFG achieved lower FID scores (higher quality) and higher LPIPS scores (greater diversity) simultaneously.
- Semantic Alignment: Using ImageReward (IR) and CLIP scores, the study showed that the approach maintains high prompt-following accuracy without the typical "collapse" where all outputs look identical.
Conclusion
By moving beyond unimodal assumptions, this work provides a clearer mathematical picture of why CFG behaves the way it does. The identified three-stage dynamics suggest that the "diversity tax" is not an inherent property of guidance itself, but rather a consequence of how guidance is applied across time. Future work may further refine these schedules for specific architectures or extend the theory to temporal consistency in video generation.
Summary updated