Multi-student Diffusion Distillation for Better One-step Generators

Yanke Song, Jonathan Lorraine, Weili Nie, Karsten Kreis, James Lucas

TL;DR

This paper introduces Multi-Student Distillation (MSD), a novel framework that distills a large diffusion model into multiple smaller, specialized "student" models. This approach significantly improves generation quality and speed, enabling real-time applications and reducing computational costs without sacrificing performance.

What Problem Does This Paper Address?

Diffusion models are powerful tools for generating high-quality images, audio, and video. However, their inference process is notoriously slow, requiring hundreds of model evaluations for a single sample. This high computational cost makes them impractical for real-time applications like augmented reality and incurs significant daily expenses. While existing distillation methods aim to create faster one-step generators, they often face a trade-off: reducing model size to increase speed leads to a drop in generation quality. This paper tackles this speed-quality dilemma by proposing a way to increase effective model capacity without increasing inference latency.

Methods

The core innovation is Multi-Student Distillation (MSD). Instead of distilling a teacher model into a single student, MSD distills it into multiple one-step student generators. Each student is trained to handle a specific subset of the conditioning data (e.g., a range of classes or text prompts). This partitioning allows each student to specialize, leading to higher quality generation for its assigned subset.

Key aspects of the MSD framework include:

  • Partitioning the Conditioning Space: The input conditioning space (e.g., class labels, text embeddings) is divided among multiple students.
  • Specialized Training: Each student is trained on a subset of the data corresponding to its assigned partition.
  • Mixture of Experts (MoE) Principle: By having multiple specialized students, MSD effectively increases model capacity without increasing inference cost, as only one student is used per request.
  • Three-Stage Training (for smaller students): When distilling into smaller students that cannot initialize from teacher weights, MSD employs a three-stage process:
  • Teacher Score Matching (TSM): A pretraining phase to initialize the student by matching the teacher's score function.
  • Distribution Matching (DM): Standard distillation techniques to match the output distribution of the teacher.
  • Adversarial Distribution Matching (ADM): An optional stage that further enhances generation quality using adversarial loss.

The paper demonstrates MSD with two established distillation techniques: Distribution Matching (DM) and Adversarial Distribution Matching (ADM).

Datasets

The paper utilizes the following datasets:

  • ImageNet-64x64: Used for class-conditional image generation experiments.
  • MS-COCO2014: Used for zero-shot text-to-image generation experiments.
  • COYO dataset: A subset of MS-COCO used for text-to-image generation.
  • 2D toy model: A synthetic dataset with 8 classes, each a mixture of Gaussians, used for initial visualization and validation.
  • Dog-related subset of COYO: Used for distilling smaller students.

Key Insights and Ideas

  • Specialization is Key: Distilling a large model into multiple smaller, specialized students significantly outperforms a single student of equivalent or even larger capacity. This is because each student can focus on a subset of the data, leading to better mode coverage and higher fidelity.
  • Mixture of Experts for Efficiency: MSD leverages the "mixture of experts" principle to boost performance without increasing inference latency. This is achieved by routing requests to the appropriate student, effectively increasing the model's capacity without a proportional increase in computational cost per sample.
  • Smaller Students, Faster Inference: By distilling into multiple smaller students, MSD offers a flexible way to achieve faster inference times, especially for edge devices or low-latency applications, while maintaining competitive generation quality.
  • TSM Pretraining is Crucial for Smaller Students: When distilling into students with different architectures (i.e., smaller ones), a Teacher Score Matching (TSM) pretraining stage is essential for good initialization and stable convergence.

Key Results

  • ImageNet-64x64 Class-Conditional Generation:
    • MSD with 4 same-sized students (ADM stage) achieved an FID score of 1.20, outperforming single-student counterparts.
  • MSD with 4 smaller students (all three stages) achieved an FID score of 2.88, with a 42% reduction in model size and a 7% reduction in latency compared to same-sized students.
  • MS-COCO2014 Text-to-Image Generation:
    • MSD with 4 same-sized students (ADM stage) achieved an FID score of 8.20, close to state-of-the-art.
  • 2D Toy Model: Demonstrated that increasing the number of students consistently improves distillation quality, reducing the distance between generated samples and the teacher's distribution.
  • CLIP Score: MSD-ADM achieved competitive CLIP scores, beating the single-student counterpart DMD2.

Limitations and Future Work

  • Partitioning Strategy: While simple splitting strategies (sequential, random) worked well, more sophisticated routing mechanisms could potentially improve performance. The paper notes that clustering label embeddings did not offer significant advantages in their experiments.
  • Smaller Student Design: The paper used simple channel reduction for smaller students, which resulted in a smaller latency reduction than model size reduction. Exploring other architectural designs for smaller students could further improve quality and throughput.
  • Weight Sharing: The current MSD trains students separately. Investigating weight-sharing or other interaction schemes between students could enhance training efficiency.
  • Generalizability: The authors hypothesize that MSD can be applied to other state-of-the-art diffusion distillation methods and other modalities, which would be an interesting avenue for future work.
  • Optimal Resource Allocation: Finding the optimal partition of resources for student nodes in a distributed deployment setting remains an open problem.

Summary updated

Abstract

Diffusion models achieve high-quality sample generation at the cost of a lengthy multistep inference procedure. To overcome this, diffusion distillation techniques produce student generators capable of matching or surpassing the teacher in a single step. However, the student model's inference speed is limited by the size of the teacher architecture, preventing real-time generation for computationally heavy applications. In this work, we introduce Multi-Student Distillation (MSD), a framework to distill a conditional teacher diffusion model into multiple single-step generators. Each student generator is responsible for a subset of the conditioning data, thereby obtaining higher generation quality for the same capacity. MSD trains multiple distilled students, allowing smaller sizes and, therefore, faster inference. Also, MSD offers a lightweight quality boost over single-student distillation with the same architecture. We demonstrate MSD is effective by training multiple same-sized or smaller students on single-step distillation using distribution matching and adversarial distillation techniques. With smaller students, MSD gets competitive results with faster inference for single-step generation. Using 4 same-sized students, MSD significantly outperforms single-student baseline counterparts and achieves remarkable FID scores for one-step image generation: 1.20 on ImageNet-64x64 and 8.20 on zero-shot COCO2014.
BibTeX
@inproceedings{song2024multi,
  author = {Yanke Song and Jonathan Lorraine and Weili Nie and Karsten Kreis and James Lucas},
  title = {{Multi-student Diffusion Distillation for Better One-step Generators}},
  booktitle = {ES-FoMo-III @ ICML},
  year = {2024},
  url = {https://openreview.net/forum?id=EZvIFdF4HT},
}
Figure