MFAGAN: A Compression Framework for Memory-Efficient On-Device Super-Resolution GAN

Wenlong Cheng, Mingbo Zhao, Zhiling Ye, Shuhang GuSSRN 2021 · DOI · OpenReview

TL;DR

This paper introduces MFAGAN, a novel compression framework for Generative Adversarial Networks (GANs) specifically designed for on-device Super-Resolution (SR). MFAGAN significantly reduces memory consumption and computational cost, enabling high-quality image SR on resource-constrained mobile devices without substantial quality degradation.

What Problem Does This Paper Address?

Single-image super-resolution (SR) aims to reconstruct high-resolution (HR) images from low-resolution (LR) inputs. While Generative Adversarial Networks (GANs) have achieved impressive photorealistic results, their high memory and computational demands make them impractical for deployment on mobile devices. These devices have limited memory bandwidth and processing power, leading to performance degradation and excessive energy consumption. Existing GAN compression techniques often struggle with the inherent instability of GAN training or fail to adequately address the memory efficiency bottleneck crucial for mobile deployment.

Methods

MFAGAN tackles the memory and computation challenge through a multi-pronged approach:

  • Memory-Efficient Generator (MFANet): The core of MFAGAN is a novel generator architecture called the Multi-Scale Feature Aggregation Network (MFANet). It employs Multi-scale Feature Aggregation Modules (MFAMs) and Concatenative Feature Aggregation Blocks (CFABs) to efficiently process features at different scales, reducing memory overhead compared to traditional dense connections.
  • Light-weight Discriminator (PatchGAN): To complement the efficient generator and stabilize training, a PatchGAN discriminator is used. This discriminator focuses on local image patches rather than the entire image, leading to fewer parameters and faster, more stable training.
  • Two-Stage Compression:
    1. Knowledge Distillation: A layer-wise knowledge distillation strategy is applied to both the generator and discriminator. This process transfers knowledge from a larger, pre-trained "teacher" model (MFAGAN_L) to a smaller "student" model, ensuring the student learns effectively and maintaining a balance between the generator and discriminator.
  • Hardware-Aware NAS Channel Pruning: After distillation, a hardware-aware Neural Architecture Search (NAS) is employed for channel pruning. This process specifically searches for an optimized "SubGenerator" tailored to the latency constraints of target mobile hardware, further reducing model size and computation. A "SuperGenerator" is trained first, containing all candidate SubGenerators, to facilitate this search.
  • Hardware-Aware Evolutionary Search: An evolutionary search algorithm is used to find the optimal SubGenerator that meets specific hardware latency requirements while maintaining high PSNR.

Datasets

  • Training: The authors used 800 HR images from the DIV2K dataset for training.
  • Evaluation: Performance was evaluated on standard SR benchmark datasets: Set5, Set14, B100, and Urban100.

Key Insights and Ideas

  • Memory Efficiency as a Primary Goal: The paper highlights that for on-device applications, memory access cost is a critical bottleneck, often more so than raw computation. MFAGAN prioritizes this from the outset.
  • Synergy of Distillation and NAS: The two-stage compression (distillation followed by NAS) is crucial. Distillation stabilizes the training of the compressed generator, while NAS efficiently prunes the network for hardware-specific optimization.
  • PatchGAN for GAN Stability: Using a PatchGAN discriminator, known for its efficiency and stability, is key to successfully training compressed GANs for SR.
  • Hardware-Awareness is Non-Negotiable: Simply optimizing for PSNR or FLOPs is insufficient for mobile deployment. The NAS component explicitly targets hardware latency, demonstrating the importance of co-design.

Key Results

  • Significant Compression: MFAGAN achieves up to 8.3x memory saving and 42.9x computation reduction compared to ESRGAN.
  • Competitive Performance: Despite aggressive compression, MFAGAN maintains competitive PSNR and LPIPS scores, with only minor visual quality degradation. For example, on the Set5 dataset, it achieves 30.16 PSNR and 0.0571 LPIPS.
  • Fast Inference on Mobile: On a Qualcomm Snapdragon 865 GPU, MFAGAN achieves an inference latency of approximately 70 milliseconds for a 128x128 input image, a 16.4x speedup over ESRGAN (1150ms).
  • Memory Reduction: The model's memory footprint is reduced to 0.55GB from ESRGAN's 4.29GB.

Limitations and Future Work

  • Capacity Gap in NAS: The authors note that a large capacity gap between the SuperGenerator and the target SubGenerator can sometimes hinder learning. Using a distilled MFAGAN-48 as the SuperGenerator helped mitigate this.
  • Future Work: The authors plan to extend their findings to video SR.

Summary updated

Abstract

Generative adversarial networks (GANs) have promoted remarkable advances in single-image super-resolution (SR) by recovering photo-realistic images. However, high memory consumption of GAN-based SR (usually generators) causes performance degradation and more energy consumption, hindering the deployment of GAN-based SR into resource-constricted mobile devices. In this paper, we propose a novel compression framework \textbf{M}ulti-scale \textbf{F}eature \textbf{A}ggregation Net based \textbf{GAN} (MFAGAN) for reducing the memory access cost of the generator. First, to overcome the memory explosion of dense connections, we utilize a memory-efficient multi-scale feature aggregation net as the generator. Second, for faster and more stable training, our method introduces the PatchGAN discriminator. Third, to balance the student discriminator and the compressed generator, we distill both the generator and the discriminator. Finally, we perform a hardware-aware neural architecture search (NAS) to find a specialized SubGenerator for the target mobile phone. Benefiting from these improvements, the proposed MFAGAN achieves up to \textbf{8.3}$\times$ memory saving and \textbf{42.9}$\times$ computation reduction, with only minor visual quality degradation, compared with ESRGAN. Empirical studies also show $\sim$\textbf{70} milliseconds latency on Qualcomm Snapdragon 865 chipset.
BibTeX
@misc{cheng2021mfagan,
  author = {Wenlong Cheng and Mingbo Zhao and Zhiling Ye and Shuhang Gu},
  title = {{MFAGAN: A Compression Framework for Memory-Efficient On-Device Super-Resolution GAN}},
  year = {2021},
  eprint = {2107.12679},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.12679},
  howpublished = {SSRN},
}
Figure