SCott: Accelerating Diffusion Models with Stochastic Consistency Distillation

Hongjian Liu, Qingsong Xie, TianXiang Ye, Zhijie Deng, Chen Chen, Shixiang Tang, Xueyang Fu, Haonan Lu, Zheng-jun Zha

TL;DR

SCott (Stochastic Consistency Distillation) is a new framework that enables high-quality text-to-image generation in just 2–4 sampling steps. By integrating stochastic differential equation (SDE) solvers and an adversarial LoRA-based discriminator into the consistency distillation process, it achieves state-of-the-art image quality and significantly higher sample diversity than previous few-step models.

What Problem Does This Paper Address?

Standard Diffusion Models (DMs) are notoriously slow because they require dozens or hundreds of iterative steps to generate an image. Consistency Distillation (CD) was introduced to solve this by training a "student" model to map any point on a sampling trajectory directly to the final image.

However, existing CD methods have two main drawbacks:

  1. ODE Limitations: They typically distill Ordinary Differential Equation (ODE) solvers. While deterministic, ODE solvers often underperform Stochastic Differential Equation (SDE) solvers, which use injected noise to "correct" approximation errors during sampling.
  2. The One-Step Ceiling: Many recent "one-step" models (like InstaFlow or UFOGen) produce impressive results in a single step but fail to improve significantly if you give them more compute (sampling steps).

SCott aims to bridge this gap, creating a model that is excellent at 2 steps but continues to improve up to 4 steps, all while maintaining the diversity that purely deterministic models often lose.

Methods

The authors introduce several key innovations to make stochastic distillation stable and effective:

  • Stochastic Consistency Distillation: Instead of following a deterministic ODE path, SCott distills the trajectory of an SDE solver. This allows the student to benefit from the error-correcting properties of noise injection.
  • Multi-step SDE Training: SDE solvers have higher discretization errors than ODE solvers. To fix this, SCott uses a multi-step sampling strategy during training. Instead of one giant leap, the teacher model takes several smaller SDE steps to provide a more accurate target for the student.
  • Noise Strength Control ($\eta$): The authors found that raw SDE solvers can be unstable during training. They introduced a scaling factor to tune the intensity of injected noise, balancing training stability with the benefits of stochasticity.
  • LoRA-based Adversarial Loss: To sharpen images at very low step counts (1–2 steps), SCott incorporates a GAN-style adversarial loss. To keep training efficient, they use a pre-trained U-Net as a discriminator, freezing the encoder and only training a few Low-Rank Adaptation (LoRA) layers in the decoder.
Figure 2: Overview of SCott. SCott distills a pre-trained teacher DM into a student one for accelerated sampling. Com- pared to the vanilla consistency distillation approach, we in- troduce a multi-step SDE solver to establish a stronger and more versatile teacher. We train the student model with CD loss using SDE solvers. Additionally, we include an adver- sarial learning loss to correct student output, boosting the sample quality with rare sampling steps. Note that we omit the EMA operation for the teacher for brevity.

Datasets

The models were trained and evaluated using:

  • Training: LAION-Aesthetics-6+ (a high-quality subset of the massive LAION-5B dataset).
  • Evaluation:
    • MSCOCO-2017 5K & MSCOCO-2014 30K: Standard benchmarks for text-to-image fidelity (FID) and text alignment (CLIP Score).
  • MJHQ-5K: A high-resolution dataset used to test the model against high-quality teachers like Realistic-Vision-v51.

Key Insights and Ideas

  • Stochasticity as Data Augmentation: The randomness inherent in SDE solvers acts as a form of implicit data augmentation during distillation. This helps the student model better cover the data distribution, leading to much higher sample diversity (measured by the Coverage metric) compared to deterministic models like LCM.
  • SDEs are Better Teachers: The paper demonstrates that for a well-trained DM, SDE solvers provide a "stronger" teacher signal than ODE solvers because they are less prone to accumulating errors over the sampling path.
  • Efficiency through LoRA: By using a LoRA-based discriminator, the authors achieved the benefits of adversarial training (sharper details, less blur) with only 3.6% of the U-Net parameters being updated, making the process computationally feasible.

Key Results

SCott sets new benchmarks for few-step generation:

  • State-of-the-Art FID: On MSCOCO-2017 5K, 2-step SCott achieved an FID of 21.9, outperforming 1-step InstaFlow (23.4) and even 4-step UFOGen (22.1).
  • Superior Diversity: SCott showed a 16% improvement in the Coverage (CR) metric over Latent Consistency Models (LCM), proving it generates a wider variety of images for the same prompt.
  • Scalable Quality: Unlike many competitors, SCott's performance scales. Moving from 2 to 4 steps consistently improves both FID and CLIP scores.
Figure 1: 512×512 resolution images generated by SCott using 2 sampling steps. SCott is trained based on Realistic-Vision-v51.

Limitations and Future Work

  • Step Count: While SCott is excellent at 2–4 steps, its 1-step performance (FID 26.8) still lags behind specialized one-step models like DMD or SwiftBrush. The authors note that over-optimizing for one step often hurts the ability to improve with more steps.
  • Hyperparameter Sensitivity: The noise scaling factor $\eta$ and the number of training SDE steps $h$ require careful tuning to avoid training instability.
  • Future Directions: The authors suggest exploring higher-order SDE solvers and applying the SCott framework to other domains like video generation or 3D synthesis.

Summary updated

Abstract

The iterative sampling procedure employed by diffusion models (DMs) often leads to significant inference latency. To address this, we propose Stochastic Consistency Distillation (SCott) to enable accelerated text-to-image generation, where high-quality and diverse generations can be achieved within just 2-4 sampling steps. In contrast to vanilla consistency distillation (CD) which distills the ordinary differential equation solvers-based sampling process of a pre-trained teacher model into a student, SCott explores the possibility and validates the efficacy of integrating stochastic differential equation (SDE) solvers into CD to fully unleash the potential of the teacher. SCott is augmented with elaborate strategies to control the noise strength and sampling process of the SDE solver. An adversarial loss is further incorporated to strengthen the consistency constraints in rare sampling steps. Empirically, on the MSCOCO-2017 5K dataset with a Stable Diffusion-V1.5 teacher, SCott achieves an FID of 21.9 with 2 sampling steps, surpassing that of the 1-step InstaFlow (23.4) and the 4-step UFOGen (22.1). Moreover, SCott can yield more diverse samples than other consistency models for high-resolution image generation, with up to 16% improvement in a qualified metric.
BibTeX
@inproceedings{liu2025scott,
  author = {Hongjian Liu and Qingsong Xie and TianXiang Ye and Zhijie Deng and Chen Chen and Shixiang Tang and Xueyang Fu and Haonan Lu and Zheng-jun Zha},
  title = {{SCott: Accelerating Diffusion Models with Stochastic Consistency Distillation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i5.32580},
}
Figure