Abstract
Generating high-quality images with diffusion models typically requires numerous iterations, leading to significant computational overhead. We present SDXL-Lightning, a state-of-the-art distillation framework that enables high-resolution (1024px) text-to-image generation in as few as one to eight steps. By leveraging progressive adversarial distillation and a novel UNet-based discriminator, SDXL-Lightning achieves superior detail and style preservation compared to existing fast-sampling methods.
Introduction
The primary bottleneck in deploying large-scale diffusion models like SDXL is inference speed. Standard ODE-solving processes often require 50 or more steps to produce high-fidelity results. While numerical solvers and distillation techniques have attempted to mitigate this, they often face trade-offs between speed and image quality. For instance, Latent Consistency Models (LCM) frequently produce blurry outputs at low step counts, while SDXL-Turbo is limited to 512px resolution due to the memory constraints of its pixel-space discriminator.
Methodology
SDXL-Lightning introduces Progressive Adversarial Diffusion Distillation to bridge the gap between speed and quality.
Progressive Adversarial Distillation
The framework employs a progressive distillation strategy, doubling the step size at each stage (from 128 down to 1 step). This ensures the student model closely follows the teacher's probability flow. To maintain sharpness, an adversarial loss is applied at every stage, preventing the regression toward the mean (blurriness) typical of MSE-only distillation.
UNet Discriminator
A key innovation is the use of the pre-trained SDXL UNet encoder as the discriminator backbone. Operating entirely in latent space, this approach allows for efficient 1024px training without the overhead of external vision models like DINOv2. The discriminator is uniquely suited to the task as it is already optimized for the target data distribution and supports noised inputs.
Mitigation of Artifacts
To address "Janus" artifacts—where the model generates multiple heads or inconsistent structures when compressing trajectories—the authors "relax" mode coverage. By finetuning with an unconditional adversarial objective, the model prioritizes per-sample quality over exact trajectory matching. Additionally, multi-timestep training stabilizes 1-step generation by adding noise to student predictions, forcing the discriminator to focus on global structure rather than high-frequency noise.
Experimental Setup
The models were distilled using high-quality subsets of the LAION and COYO datasets, filtered for images larger than 1024px and aesthetic scores above 5.5. Sharpness was ensured via Laplacian filtering, and text prompts were refined for better training signals. Quantitative evaluation was performed using the first 10,000 prompts from the COCO validation set.
Results
SDXL-Lightning establishes a new state-of-the-art for 1, 2, 4, and 8-step generation at native 1024px resolution.
Quantitative Performance
In FID-Patch evaluations, which prioritize high-frequency detail, SDXL-Lightning significantly outperforms competitors. At 4 steps, it achieves a Patch-FID of 33.52, compared to 42.53 for LCM.
Style Preservation and Compatibility
Unlike previous methods that drift from the base model's aesthetic, SDXL-Lightning maintains the original SDXL layout and style. The release of LoRA modules further enables the community to accelerate existing finetuned models instantly.
Discussion and Limitations
While SDXL-Lightning represents a significant leap, it currently relies on discrete checkpoints for different step counts. Future work aims to unify these into a single checkpoint. Furthermore, while the model generalizes well to various aspect ratios, the primary distillation on square images can occasionally lead to artifacts at extreme ratios in 1-step mode. Architectural optimizations specifically for single-step generation remain an open area for exploration.
Summary updated