TL;DR
This paper introduces Multi-Student Distillation (MSD), a novel framework that distills a large diffusion model into multiple smaller, specialized "student" models. This approach significantly improves generation quality and speed, enabling real-time applications and reducing computational costs without sacrificing performance.
What Problem Does This Paper Address?
Diffusion models are powerful tools for generating high-quality images, audio, and video. However, their inference process is notoriously slow, requiring hundreds of model evaluations for a single sample. This high computational cost makes them impractical for real-time applications like augmented reality and incurs significant daily expenses. While existing distillation methods aim to create faster one-step generators, they often face a trade-off: reducing model size to increase speed leads to a drop in generation quality. This paper tackles this speed-quality dilemma by proposing a way to increase effective model capacity without increasing inference latency.
Methods
The core innovation is Multi-Student Distillation (MSD). Instead of distilling a teacher model into a single student, MSD distills it into multiple one-step student generators. Each student is trained to handle a specific subset of the conditioning data (e.g., a range of classes or text prompts). This partitioning allows each student to specialize, leading to higher quality generation for its assigned subset.
Key aspects of the MSD framework include:
- Partitioning the Conditioning Space: The input conditioning space (e.g., class labels, text embeddings) is divided among multiple students.
- Specialized Training: Each student is trained on a subset of the data corresponding to its assigned partition.
- Mixture of Experts (MoE) Principle: By having multiple specialized students, MSD effectively increases model capacity without increasing inference cost, as only one student is used per request.
- Three-Stage Training (for smaller students): When distilling into smaller students that cannot initialize from teacher weights, MSD employs a three-stage process:
- Teacher Score Matching (TSM): A pretraining phase to initialize the student by matching the teacher's score function.
- Distribution Matching (DM): Standard distillation techniques to match the output distribution of the teacher.
- Adversarial Distribution Matching (ADM): An optional stage that further enhances generation quality using adversarial loss.
The paper demonstrates MSD with two established distillation techniques: Distribution Matching (DM) and Adversarial Distribution Matching (ADM).
Datasets
The paper utilizes the following datasets:
- ImageNet-64x64: Used for class-conditional image generation experiments.
- MS-COCO2014: Used for zero-shot text-to-image generation experiments.
- COYO dataset: A subset of MS-COCO used for text-to-image generation.
- 2D toy model: A synthetic dataset with 8 classes, each a mixture of Gaussians, used for initial visualization and validation.
- Dog-related subset of COYO: Used for distilling smaller students.
Key Insights and Ideas
- Specialization is Key: Distilling a large model into multiple smaller, specialized students significantly outperforms a single student of equivalent or even larger capacity. This is because each student can focus on a subset of the data, leading to better mode coverage and higher fidelity.
- Mixture of Experts for Efficiency: MSD leverages the "mixture of experts" principle to boost performance without increasing inference latency. This is achieved by routing requests to the appropriate student, effectively increasing the model's capacity without a proportional increase in computational cost per sample.
- Smaller Students, Faster Inference: By distilling into multiple smaller students, MSD offers a flexible way to achieve faster inference times, especially for edge devices or low-latency applications, while maintaining competitive generation quality.
- TSM Pretraining is Crucial for Smaller Students: When distilling into students with different architectures (i.e., smaller ones), a Teacher Score Matching (TSM) pretraining stage is essential for good initialization and stable convergence.
Key Results
- ImageNet-64x64 Class-Conditional Generation:
- MSD with 4 same-sized students (ADM stage) achieved an FID score of 1.20, outperforming single-student counterparts.
- MSD with 4 smaller students (all three stages) achieved an FID score of 2.88, with a 42% reduction in model size and a 7% reduction in latency compared to same-sized students.
- MS-COCO2014 Text-to-Image Generation:
- MSD with 4 same-sized students (ADM stage) achieved an FID score of 8.20, close to state-of-the-art.
- 2D Toy Model: Demonstrated that increasing the number of students consistently improves distillation quality, reducing the distance between generated samples and the teacher's distribution.
- CLIP Score: MSD-ADM achieved competitive CLIP scores, beating the single-student counterpart DMD2.
Limitations and Future Work
- Partitioning Strategy: While simple splitting strategies (sequential, random) worked well, more sophisticated routing mechanisms could potentially improve performance. The paper notes that clustering label embeddings did not offer significant advantages in their experiments.
- Smaller Student Design: The paper used simple channel reduction for smaller students, which resulted in a smaller latency reduction than model size reduction. Exploring other architectural designs for smaller students could further improve quality and throughput.
- Weight Sharing: The current MSD trains students separately. Investigating weight-sharing or other interaction schemes between students could enhance training efficiency.
- Generalizability: The authors hypothesize that MSD can be applied to other state-of-the-art diffusion distillation methods and other modalities, which would be an interesting avenue for future work.
- Optimal Resource Allocation: Finding the optimal partition of resources for student nodes in a distributed deployment setting remains an open problem.
Summary updated