TL;DR
SCott (Stochastic Consistency Distillation) is a new framework that enables high-quality text-to-image generation in just 2–4 sampling steps. By integrating stochastic differential equation (SDE) solvers and an adversarial LoRA-based discriminator into the consistency distillation process, it achieves state-of-the-art image quality and significantly higher sample diversity than previous few-step models.
What Problem Does This Paper Address?
Standard Diffusion Models (DMs) are notoriously slow because they require dozens or hundreds of iterative steps to generate an image. Consistency Distillation (CD) was introduced to solve this by training a "student" model to map any point on a sampling trajectory directly to the final image.
However, existing CD methods have two main drawbacks:
- ODE Limitations: They typically distill Ordinary Differential Equation (ODE) solvers. While deterministic, ODE solvers often underperform Stochastic Differential Equation (SDE) solvers, which use injected noise to "correct" approximation errors during sampling.
- The One-Step Ceiling: Many recent "one-step" models (like InstaFlow or UFOGen) produce impressive results in a single step but fail to improve significantly if you give them more compute (sampling steps).
SCott aims to bridge this gap, creating a model that is excellent at 2 steps but continues to improve up to 4 steps, all while maintaining the diversity that purely deterministic models often lose.
Methods
The authors introduce several key innovations to make stochastic distillation stable and effective:
- Stochastic Consistency Distillation: Instead of following a deterministic ODE path, SCott distills the trajectory of an SDE solver. This allows the student to benefit from the error-correcting properties of noise injection.
- Multi-step SDE Training: SDE solvers have higher discretization errors than ODE solvers. To fix this, SCott uses a multi-step sampling strategy during training. Instead of one giant leap, the teacher model takes several smaller SDE steps to provide a more accurate target for the student.
- Noise Strength Control ($\eta$): The authors found that raw SDE solvers can be unstable during training. They introduced a scaling factor to tune the intensity of injected noise, balancing training stability with the benefits of stochasticity.
- LoRA-based Adversarial Loss: To sharpen images at very low step counts (1–2 steps), SCott incorporates a GAN-style adversarial loss. To keep training efficient, they use a pre-trained U-Net as a discriminator, freezing the encoder and only training a few Low-Rank Adaptation (LoRA) layers in the decoder.
Datasets
The models were trained and evaluated using:
- Training: LAION-Aesthetics-6+ (a high-quality subset of the massive LAION-5B dataset).
- Evaluation:
- MSCOCO-2017 5K & MSCOCO-2014 30K: Standard benchmarks for text-to-image fidelity (FID) and text alignment (CLIP Score).
- MJHQ-5K: A high-resolution dataset used to test the model against high-quality teachers like Realistic-Vision-v51.
Key Insights and Ideas
- Stochasticity as Data Augmentation: The randomness inherent in SDE solvers acts as a form of implicit data augmentation during distillation. This helps the student model better cover the data distribution, leading to much higher sample diversity (measured by the Coverage metric) compared to deterministic models like LCM.
- SDEs are Better Teachers: The paper demonstrates that for a well-trained DM, SDE solvers provide a "stronger" teacher signal than ODE solvers because they are less prone to accumulating errors over the sampling path.
- Efficiency through LoRA: By using a LoRA-based discriminator, the authors achieved the benefits of adversarial training (sharper details, less blur) with only 3.6% of the U-Net parameters being updated, making the process computationally feasible.
Key Results
SCott sets new benchmarks for few-step generation:
- State-of-the-Art FID: On MSCOCO-2017 5K, 2-step SCott achieved an FID of 21.9, outperforming 1-step InstaFlow (23.4) and even 4-step UFOGen (22.1).
- Superior Diversity: SCott showed a 16% improvement in the Coverage (CR) metric over Latent Consistency Models (LCM), proving it generates a wider variety of images for the same prompt.
- Scalable Quality: Unlike many competitors, SCott's performance scales. Moving from 2 to 4 steps consistently improves both FID and CLIP scores.
Limitations and Future Work
- Step Count: While SCott is excellent at 2–4 steps, its 1-step performance (FID 26.8) still lags behind specialized one-step models like DMD or SwiftBrush. The authors note that over-optimizing for one step often hurts the ability to improve with more steps.
- Hyperparameter Sensitivity: The noise scaling factor $\eta$ and the number of training SDE steps $h$ require careful tuning to avoid training instability.
- Future Directions: The authors suggest exploring higher-order SDE solvers and applying the SCott framework to other domains like video generation or 3D synthesis.
Summary updated