Lightweight Single Image Super-Resolution with Selective Channel Processing Network

Hongyu Zhu, Hao Tang, Yaocong Hu, Huanjie Tao, Chao XieSensors 2022 · DOI

TL;DR

This paper introduces the Selective Channel Processing Network (SCPN), a novel lightweight deep learning architecture for single image super-resolution (SR). By dynamically learning channel importance and selectively processing features, SCPN significantly reduces computational cost and model size while achieving state-of-the-art performance on both natural and remote-sensing images.

What Problem Does This Paper Address?

Single Image Super-Resolution (SR) aims to reconstruct a high-resolution (HR) image from a low-resolution (LR) input. While deep learning has revolutionized SR, leading to impressive results, many state-of-the-art methods suffer from enormous parameter counts and computational demands. This makes them impractical for deployment on resource-constrained devices or for real-time applications. The core challenge is to achieve high-quality SR without the prohibitive computational overhead.

Methods

The authors propose a novel architecture called the Selective Channel Processing Network (SCPN). The key innovations are:

  • Selective Channel Processing Module (SCPM): This module is designed to dynamically learn the importance of each feature channel.
  • Training Phase: It uses a "softened" channel selection matrix, approximated by a Gumbel-softmax distribution, to learn which channels are most relevant. This allows for gradient-based learning of channel importance.
  • Inference Phase: The selection matrix is "hardened" into a binary mask. This mask acts as a gate, deciding whether to pass a channel to the next convolutional layer or directly to an addition layer. This selective processing drastically reduces computation.
  • Differential Channel Attention (DCA) Block: This block enhances the network's ability to capture high-frequency details. Unlike traditional channel attention that relies solely on average pooling, DCA incorporates both the mean and standard deviation of channel features, providing a richer representation of data distribution.

The overall SCPN architecture consists of three main parts: shallow feature extraction, deep feature extraction (using multiple SCPMs), and up-sampling reconstruction.

Datasets

  • Training: DIV2K dataset (800 high-quality natural images).
  • Evaluation (Natural Images): Set5, Set14, B100, Urban100, and Manga109.
  • Evaluation (Remote Sensing): UC Merced Land-use dataset and NWPU-RESISC45 dataset.

Key Insights and Ideas

  • Feature Map Sparsity: The paper visually demonstrates that in deep SR networks (like EDSR and RCAN), many feature maps, especially in earlier layers, contain little useful information (i.e., are mostly zeros). This observation is a strong motivation for selective channel processing.
  • Dynamic Channel Selection: The core idea of learning channel importance dynamically during training and then using this knowledge for efficient inference is a powerful approach to create lightweight models.
  • Inference-Phase Optimization: The distinction between training and inference phases for SCPM is crucial. Softening during training allows for learning, while hardening during inference enables significant computational savings.
  • Beyond Average Pooling for Attention: The DCA block highlights that richer statistical information (like standard deviation) from feature channels can lead to better reconstruction of fine details, especially for tasks like SR.

Key Results

  • Lightweight Model: The proposed SCPN achieves state-of-the-art performance with fewer than 1 million parameters, significantly outperforming many larger models.
  • Performance Gains: On natural image benchmarks (Set5, Set14, B100, Urban100, Manga109), SCPN achieves competitive or superior PSNR/SSIM values compared to other lightweight methods.
  • Remote Sensing Performance: SCPN also demonstrates strong performance on remote sensing datasets (UCCTest, RESISCTest), outperforming IMDN-T with a higher PSNR (approx. 0.1 dB) and SSIM (approx. 0.004) on average.
  • Efficiency: The selective channel processing significantly reduces computational cost (FLOPS) while maintaining or improving performance. For example, compared to a baseline without channel selection (Module-A), SCPN shows better PSNR with comparable FLOPS but fewer parameters.
  • Generalization: The paper suggests that SCPN's fine-tuning on remote sensing data from a model pre-trained on natural images shows good generalization ability and faster convergence.

Limitations and Future Work

  • Limitations: While the paper focuses on lightweight design, the exact trade-offs between parameter count, FLOPS, and performance are always a subject of ongoing research. The paper implicitly acknowledges that some performance might be sacrificed for extreme lightness, though it aims to minimize this.
  • Future Work: The authors suggest extending their lightweight model to mobile devices. They also plan to explore other lightweight strategies, such as incorporating sparsity convolution, to further reduce model size and computational complexity. The potential for SCPN and DCA to be applied to other computer vision tasks is also mentioned.

Summary updated

Abstract

With the development of deep learning, considerable progress has been made in image restoration. Notably, many state-of-the-art single image super-resolution (SR) methods have been proposed. However, most of them contain many parameters, which leads to a significant amount of calculation consumption in the inference phase. To make current SR networks more lightweight and resource-friendly, we present a convolution neural network with the proposed selective channel processing strategy (SCPN). Specifically, the selective channel processing module (SCPM) is first designed to dynamically learn the significance of each channel in the feature map using a channel selection matrix in the training phase. Correspondingly, in the inference phase, only the essential channels indicated by the channel selection matrixes need to be further processed. By doing so, we can significantly reduce the parameters and the calculation consumption. Moreover, the differential channel attention (DCA) block is proposed, which takes into consideration the data distribution of the channels in feature maps to restore more high-frequency information. Extensive experiments are performed on the natural image super-resolution benchmarks (i.e., Set5, Set14, B100, Urban100, Manga109) and remote-sensing benchmarks (i.e., UCTest and RESISCTest), and our method achieves superior results to other state-of-the-art methods. Furthermore, our method keeps a slim size with fewer than 1 M parameters, which proves the superiority of our method. Owing to the proposed SCPM and DCA, our SCPN model achieves a better trade-off between calculation cost and performance in both general and remote-sensing SR applications, and our proposed method can be extended to other computer vision tasks for further research.
BibTeX
@inproceedings{zhu2022lightweight,
  author = {Hongyu Zhu and Hao Tang and Yaocong Hu and Huanjie Tao and Chao Xie},
  title = {{Lightweight Single Image Super-Resolution with Selective Channel Processing Network}},
  booktitle = {Sensors},
  year = {2022},
  doi = {10.3390/s22155586},
}
Figure