TL;DR
This paper introduces OMGD, a novel "online multi-granularity distillation" method to significantly compress Generative Adversarial Networks (GANs) for deployment on resource-constrained devices. By training a student GAN to learn from multiple, progressively refined teacher GANs and their intermediate layers simultaneously, OMGD achieves substantial reductions in computation (up to 46.6x) and parameters (up to 82.5x) without sacrificing image quality.
What Problem Does This Paper Address?
Generative Adversarial Networks (GANs) have revolutionized image generation and translation, producing stunningly realistic results. However, their impressive performance comes at a steep cost: massive computational requirements and large memory footprints. This makes deploying state-of-the-art GANs on resource-limited devices like mobile phones or embedded systems extremely challenging, hindering their widespread adoption in real-world applications. While existing GAN compression techniques exist, they often rely on complex multi-stage processes, require pre-trained teachers, or still incur high computational costs. This paper aims to bridge this gap by proposing a more efficient and effective single-stage compression method.
Methods
The core of the proposed solution is the Online Multi-Granularity Distillation (OMGD) framework. Unlike traditional offline distillation where a pre-trained teacher guides a student, OMGD employs an online, single-stage distillation approach. Here's a breakdown of the key components:
- Discriminator-Free Student Training: The student generator ($G_s$) is trained without a discriminator, simplifying the training process and allowing for greater compression. It learns by mimicking the output of teacher generators.
- Progressively Promoted Teacher: Instead of a static pre-trained teacher, OMGD uses a teacher generator ($G_T$) that is also being trained and refined alongside the student. This "progressively promoted teacher" helps warm up the student and guides its learning more effectively.
- Multi-Granularity Distillation: This is a key innovation. OMGD leverages knowledge from multiple sources:
- Multiple Teacher Generators: Two complementary teacher generators are used: a wider one ($G_W$) with more channels and a deeper one ($G_P$) with added ResNet blocks. This provides richer, multi-dimensional cues.
- Intermediate Layer Distillation: Beyond just the output layer, OMGD also distills knowledge from intermediate layers. It uses channel-wise attention weights to identify important channels and transfers this "dark knowledge" to the student.
- Learn-Best Strategy (for CycleGAN): To combat the notorious instability of CycleGAN training, a "learn-best" strategy is employed. The generators are evaluated periodically, and the best performing one is selected as the teacher for the student's next training phase.
The overall objective function combines knowledge distillation losses from multiple teachers and intermediate layers, along with a channel distillation loss.
Datasets
The paper evaluates OMGD on four widely-used benchmark datasets for image-to-image translation:
- Edges → Shoes: For Pix2Pix.
- Cityscapes: For Pix2Pix (evaluated using mIoU).
- Horse → Zebra: For CycleGAN.
- Summer → Winter: For CycleGAN.
Key Insights and Ideas
- Online Distillation is Key: Training the teacher and student simultaneously (online) and progressively is crucial for achieving better results and avoiding issues like over-fitting that can plague offline methods.
- Complementary Teachers Boost Performance: Using teachers with different architectural strengths (wider vs. deeper) provides a more comprehensive set of supervisory signals, leading to better student performance.
- Intermediate Layer Knowledge is Valuable: Distilling knowledge from intermediate layers, not just the final output, significantly enhances the student's ability to capture fine-grained details and improve visual fidelity.
- Discriminator-Free Training Enables Aggressive Compression: Removing the discriminator from the student's training loop simplifies the process and allows for much more aggressive compression without catastrophic performance drops.
- "Learn-Best" Strategy Mitigates Instability: For notoriously unstable GAN training scenarios like CycleGAN, a dynamic teacher selection mechanism can significantly improve robustness and final performance.
Key Results
OMGD demonstrates impressive results across various GAN models and datasets:
- Significant Compression: OMGD achieves substantial reductions in computational cost (MACs) and model size (#Parameters). For example, it compresses Pix2Pix and CycleGAN by 40.3-46.6x MACs and 82.5x parameters.
- Preserved Visual Fidelity: Despite aggressive compression, OMGD maintains high visual quality, often outperforming state-of-the-art methods in terms of FID (Frechet Inception Distance) and mIoU (mean Intersection over Union).
- On cityscapes, OMGD(U) 1.0x achieves an mIoU of 48.91, outperforming the original model (44.32) and GAN Compression (40.77) with significantly fewer MACs.
- On edges→shoes, OMGD(R) 1.0x achieves an FID of 25.88, comparable to the original model's 24.18, but with a 40.3x MACs reduction.
- Real-time Potential: The compressed models show significantly reduced latency on mobile devices, with OMGD(U) 1.0x achieving a 90% latency decline on a Huawei P20. This suggests a path towards real-time image translation on resource-constrained devices.
- Ablation Studies Confirm Effectiveness: Ablation studies confirm the importance of each component: online distillation, complementary teachers, and intermediate layer distillation all contribute to the overall performance gains.
Limitations and Future Work
- CycleGAN Instability: While the "learn-best" strategy helps, the paper notes that the unstable training process of CycleGAN can still pose challenges for the deeper teacher generator.
- Channel Distillation Impact: The paper mentions that channel distillation loss did not show a significant effect for the Res-Net style generator on the cityscapes dataset, leading them to set its weight to zero in that specific case. This suggests potential dataset or architecture-specific tuning might be beneficial.
- Future Work: The authors highlight the potential for further compression, as demonstrated by their OMGD(U) 0.5x model achieving impressive results with only 0.333G MACs. This opens doors for even more aggressive compression and deployment on highly constrained devices. The success on various datasets and architectures suggests OMGD is a robust framework for GAN compression.
Summary updated