SCott: Accelerating Diffusion Models with Stochastic Consistency Distillation

Hongjian Liu, Qingsong Xie, Zhijie Deng, Chen Chen, Shixiang Tang, Fueyang Fu, Zhengjun Zha, Haonan Lu

TL;DR

Diffusion models are powerful for generating images but notoriously slow. This paper introduces SCott, a method that uses stochastic differential equations (SDEs) within a consistency distillation framework to dramatically speed up image generation, achieving high-quality results in just 2-4 steps. SCott also incorporates adversarial training to further enhance sample quality and diversity.

What Problem Does This Paper Address?

Diffusion models (DMs) have revolutionized generative AI, enabling impressive text-to-image synthesis. However, their iterative sampling process during inference is computationally expensive, requiring many steps and leading to significant latency. While various methods exist to accelerate this process, they often struggle to balance speed with image quality and diversity, especially at very few sampling steps (e.g., 1-2). Existing consistency distillation (CD) methods, which distill a pre-trained DM into a faster student model, typically rely on ordinary differential equation (ODE) solvers. The authors argue that this approach doesn't fully leverage the potential of the teacher DM, as stochastic differential equation (SDE) solvers can offer better performance with sufficient sampling steps. SCott aims to bridge this gap by integrating SDE solvers into CD for faster, higher-quality, and more diverse image generation.

Methods

SCott builds upon the concept of Consistency Distillation (CD) but introduces several key innovations:

  • Stochastic Consistency Distillation (SCott): Instead of using ODE solvers as in traditional CD, SCott employs SDE solvers to distill the teacher DM into a student model. This leverages the inherent stochasticity of SDEs to potentially improve sample quality and diversity.
  • Controlled SDE Noise: The authors found that directly applying SDE solvers can lead to training instability. They introduce strategies to control the noise level in the SDE solver, finding a moderate level is crucial for stabilizing training while still benefiting from stochasticity.
  • Multi-step SDE Sampling: To further mitigate discretization errors inherent in SDE solvers, SCott uses a multi-step sampling approach within the distillation process. This allows the student model to learn from a more refined teacher trajectory.
  • Adversarial Loss: To address limitations of L2 loss in capturing complex data distributions and to improve sample quality, especially at rare sampling steps, SCott incorporates an adversarial loss. This loss is implemented using a novel low-rank adaptation (LoRA) discriminator with time and text projection, making it efficient and effective for high-resolution image generation in the latent space.

Datasets

  • MSCOCO-2017 5K: Used for training and evaluation with a Stable Diffusion-V1.5 teacher.
  • MJHQ-5K: Used for evaluation with a Realistic-Vision-v51 teacher.
  • MSCOCO-2014 30K: Used for further comparisons.
  • LAION-Aesthetics-6+: Used for training the SCott model.

Key Insights and Ideas

  • SDEs unlock teacher potential: The core insight is that SDE solvers, when properly controlled, can serve as more powerful teachers for consistency distillation than ODE solvers, especially when aiming for few-step generation.
  • Noise control is critical: Simply using SDEs isn't enough; carefully managing the noise injection level is essential for stable training and optimal performance. Too little noise reverts to ODE-like behavior, while too much leads to instability.
  • Multi-step distillation improves accuracy: Extending the distillation process to multiple steps, even with SDEs, helps correct accumulated errors and leads to a more accurate student model.
  • Adversarial training for realism: Incorporating a GAN-based discriminator, specifically a LoRA-based one, significantly boosts the realism and quality of generated images, particularly at the critical few-sampling steps.
  • Flexibility in sampling steps: SCott demonstrates that it's not just about achieving 1-2 steps, but also about having a model that can gracefully improve quality by taking a few more steps (up to 4), offering a valuable trade-off.

Key Results

  • State-of-the-art FID: On MSCOCO-2017 5K with a SD1.5 teacher, SCott achieved an FID of 21.9 in just 2 steps, outperforming 1-step InstaFlow (23.4) and matching 4-step UFOGen (22.1).
  • Improved Diversity: SCott showed significant improvements in sample diversity (Coverage metric) compared to other consistency models like LCM.
  • Scalable Quality: The model consistently improved sample quality with increasing sampling steps (e.g., from 2 to 4 steps).
  • High-resolution performance: SCott demonstrated strong performance on high-resolution image generation tasks.
  • Qualitative improvements: Visual comparisons showed SCott generating images with better details, text-to-image alignment, and overall quality compared to baselines like InstaFlow and LCM.

Limitations and Future Work

  • Training Stability: While SCott addresses training instability with noise control, the paper notes that directly applying SDE solvers can still be challenging.
  • Computational Cost: Although SCott significantly accelerates inference, training the model, especially with the adversarial component, still requires substantial computational resources.
  • Future Work: The authors suggest exploring further improvements by integrating more advanced SDE solvers and investigating the impact of different teacher models and datasets. They also hint at the potential for further optimization of the adversarial training setup.

Summary updated

Abstract

The iterative sampling procedure employed by diffusion models (DMs) often leads to significant inference latency. To address this, we propose Stochastic Consistency Distillation (SCott) to enable accelerated text-to-image generation, where high-quality generations can be achieved with just 1-2 sampling steps, and further improvements can be obtained by adding additional steps. In contrast to vanilla consistency distillation (CD) which distills the ordinary differential equation solvers-based sampling process of a pretrained teacher model into a student, SCott explores the possibility and validates the efficacy of integrating stochastic differential equation (SDE) solvers into CD to fully unleash the potential of the teacher. SCott is augmented with elaborate strategies to control the noise strength and sampling process of the SDE solver. An adversarial loss is further incorporated to strengthen the sample quality with rare sampling steps. Empirically, on the MSCOCO-2017 5K dataset with a Stable Diffusion-V1.5 teacher, SCott achieves an FID (Frechet Inceptio Distance) of 22.1, surpassing that (23.4) of the 1-step InstaFlow (Liu et al., 2023) and matching that of 4-step UFOGen (Xue et al., 2023b). Moreover, SCott can yield more diverse samples than other consistency models for high-resolution image generation (Luo et al., 2023a), with up to 16% improvement in a qualified metric. The code and checkpoints are coming soon.
BibTeX
@inproceedings{liu2025scott,
  author = {Hongjian Liu and Qingsong Xie and Zhijie Deng and Chen Chen and Shixiang Tang and Fueyang Fu and Zhengjun Zha and Haonan Lu},
  title = {{SCott: Accelerating Diffusion Models with Stochastic Consistency Distillation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i5.32580},
}
Figure