TL;DR
Diffusion models are powerful for generating images but notoriously slow. This paper introduces SCott, a method that uses stochastic differential equations (SDEs) within a consistency distillation framework to dramatically speed up image generation, achieving high-quality results in just 2-4 steps. SCott also incorporates adversarial training to further enhance sample quality and diversity.
What Problem Does This Paper Address?
Diffusion models (DMs) have revolutionized generative AI, enabling impressive text-to-image synthesis. However, their iterative sampling process during inference is computationally expensive, requiring many steps and leading to significant latency. While various methods exist to accelerate this process, they often struggle to balance speed with image quality and diversity, especially at very few sampling steps (e.g., 1-2). Existing consistency distillation (CD) methods, which distill a pre-trained DM into a faster student model, typically rely on ordinary differential equation (ODE) solvers. The authors argue that this approach doesn't fully leverage the potential of the teacher DM, as stochastic differential equation (SDE) solvers can offer better performance with sufficient sampling steps. SCott aims to bridge this gap by integrating SDE solvers into CD for faster, higher-quality, and more diverse image generation.
Methods
SCott builds upon the concept of Consistency Distillation (CD) but introduces several key innovations:
- Stochastic Consistency Distillation (SCott): Instead of using ODE solvers as in traditional CD, SCott employs SDE solvers to distill the teacher DM into a student model. This leverages the inherent stochasticity of SDEs to potentially improve sample quality and diversity.
- Controlled SDE Noise: The authors found that directly applying SDE solvers can lead to training instability. They introduce strategies to control the noise level in the SDE solver, finding a moderate level is crucial for stabilizing training while still benefiting from stochasticity.
- Multi-step SDE Sampling: To further mitigate discretization errors inherent in SDE solvers, SCott uses a multi-step sampling approach within the distillation process. This allows the student model to learn from a more refined teacher trajectory.
- Adversarial Loss: To address limitations of L2 loss in capturing complex data distributions and to improve sample quality, especially at rare sampling steps, SCott incorporates an adversarial loss. This loss is implemented using a novel low-rank adaptation (LoRA) discriminator with time and text projection, making it efficient and effective for high-resolution image generation in the latent space.
Datasets
- MSCOCO-2017 5K: Used for training and evaluation with a Stable Diffusion-V1.5 teacher.
- MJHQ-5K: Used for evaluation with a Realistic-Vision-v51 teacher.
- MSCOCO-2014 30K: Used for further comparisons.
- LAION-Aesthetics-6+: Used for training the SCott model.
Key Insights and Ideas
- SDEs unlock teacher potential: The core insight is that SDE solvers, when properly controlled, can serve as more powerful teachers for consistency distillation than ODE solvers, especially when aiming for few-step generation.
- Noise control is critical: Simply using SDEs isn't enough; carefully managing the noise injection level is essential for stable training and optimal performance. Too little noise reverts to ODE-like behavior, while too much leads to instability.
- Multi-step distillation improves accuracy: Extending the distillation process to multiple steps, even with SDEs, helps correct accumulated errors and leads to a more accurate student model.
- Adversarial training for realism: Incorporating a GAN-based discriminator, specifically a LoRA-based one, significantly boosts the realism and quality of generated images, particularly at the critical few-sampling steps.
- Flexibility in sampling steps: SCott demonstrates that it's not just about achieving 1-2 steps, but also about having a model that can gracefully improve quality by taking a few more steps (up to 4), offering a valuable trade-off.
Key Results
- State-of-the-art FID: On MSCOCO-2017 5K with a SD1.5 teacher, SCott achieved an FID of 21.9 in just 2 steps, outperforming 1-step InstaFlow (23.4) and matching 4-step UFOGen (22.1).
- Improved Diversity: SCott showed significant improvements in sample diversity (Coverage metric) compared to other consistency models like LCM.
- Scalable Quality: The model consistently improved sample quality with increasing sampling steps (e.g., from 2 to 4 steps).
- High-resolution performance: SCott demonstrated strong performance on high-resolution image generation tasks.
- Qualitative improvements: Visual comparisons showed SCott generating images with better details, text-to-image alignment, and overall quality compared to baselines like InstaFlow and LCM.
Limitations and Future Work
- Training Stability: While SCott addresses training instability with noise control, the paper notes that directly applying SDE solvers can still be challenging.
- Computational Cost: Although SCott significantly accelerates inference, training the model, especially with the adversarial component, still requires substantial computational resources.
- Future Work: The authors suggest exploring further improvements by integrating more advanced SDE solvers and investigating the impact of different teacher models and datasets. They also hint at the potential for further optimization of the adversarial training setup.
Summary updated