SDXL-Lightning: Progressive Adversarial Diffusion Distillation

Shanchuan Lin, Anran Wang, Xiao YangarXiv 2024 · DOI · arXiv

Abstract

Generating high-quality images with diffusion models typically requires numerous iterations, leading to significant computational overhead. We present SDXL-Lightning, a state-of-the-art distillation framework that enables high-resolution (1024px) text-to-image generation in as few as one to eight steps. By leveraging progressive adversarial distillation and a novel UNet-based discriminator, SDXL-Lightning achieves superior detail and style preservation compared to existing fast-sampling methods.

Introduction

The primary bottleneck in deploying large-scale diffusion models like SDXL is inference speed. Standard ODE-solving processes often require 50 or more steps to produce high-fidelity results. While numerical solvers and distillation techniques have attempted to mitigate this, they often face trade-offs between speed and image quality. For instance, Latent Consistency Models (LCM) frequently produce blurry outputs at low step counts, while SDXL-Turbo is limited to 512px resolution due to the memory constraints of its pixel-space discriminator.

Methodology

SDXL-Lightning introduces Progressive Adversarial Diffusion Distillation to bridge the gap between speed and quality.

Progressive Adversarial Distillation

The framework employs a progressive distillation strategy, doubling the step size at each stage (from 128 down to 1 step). This ensures the student model closely follows the teacher's probability flow. To maintain sharpness, an adversarial loss is applied at every stage, preventing the regression toward the mean (blurriness) typical of MSE-only distillation.

Figure 1: High-frequency detail and sharpness achieved through adversarial loss and Laplacian filtering.

UNet Discriminator

A key innovation is the use of the pre-trained SDXL UNet encoder as the discriminator backbone. Operating entirely in latent space, this approach allows for efficient 1024px training without the overhead of external vision models like DINOv2. The discriminator is uniquely suited to the task as it is already optimized for the target data distribution and supports noised inputs.

Mitigation of Artifacts

To address "Janus" artifacts—where the model generates multiple heads or inconsistent structures when compressing trajectories—the authors "relax" mode coverage. By finetuning with an unconditional adversarial objective, the model prioritizes per-sample quality over exact trajectory matching. Additionally, multi-timestep training stabilizes 1-step generation by adding noise to student predictions, forcing the discriminator to focus on global structure rather than high-frequency noise.

Experimental Setup

The models were distilled using high-quality subsets of the LAION and COYO datasets, filtered for images larger than 1024px and aesthetic scores above 5.5. Sharpness was ensured via Laplacian filtering, and text prompts were refined for better training signals. Quantitative evaluation was performed using the first 10,000 prompts from the COCO validation set.

Results

SDXL-Lightning establishes a new state-of-the-art for 1, 2, 4, and 8-step generation at native 1024px resolution.

Figure 2: A showcase of diverse 1024px images generated by SDXL-Lightning, demonstrating high fidelity across various subjects including animals, portraits, and landscapes.

Quantitative Performance

In FID-Patch evaluations, which prioritize high-frequency detail, SDXL-Lightning significantly outperforms competitors. At 4 steps, it achieves a Patch-FID of 33.52, compared to 42.53 for LCM.

Style Preservation and Compatibility

Unlike previous methods that drift from the base model's aesthetic, SDXL-Lightning maintains the original SDXL layout and style. The release of LoRA modules further enables the community to accelerate existing finetuned models instantly.

Discussion and Limitations

While SDXL-Lightning represents a significant leap, it currently relies on discrete checkpoints for different step counts. Future work aims to unify these into a single checkpoint. Furthermore, while the model generalizes well to various aspect ratios, the primary distillation on square images can occasionally lead to artifacts at extreme ratios in 1-step mode. Architectural optimizations specifically for single-step generation remain an open area for exploration.

Summary updated

Abstract

We propose a diffusion distillation method that achieves new state-of-the-art in one-step/few-step 1024px text-to-image generation based on SDXL. Our method combines progressive and adversarial distillation to achieve a balance between quality and mode coverage. In this paper, we discuss the theoretical analysis, discriminator design, model formulation, and training techniques. We open-source our distilled SDXL-Lightning models both as LoRA and full UNet weights.
BibTeX
@misc{lin2024sdxl,
  author = {Shanchuan Lin and Anran Wang and Xiao Yang},
  title = {{SDXL-Lightning: Progressive Adversarial Diffusion Distillation}},
  year = {2024},
  eprint = {2402.13929},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.13929},
}
Figure