TL;DR
This paper introduces a novel method for making image super-resolution (SR) models more efficient by dynamically reducing both spatial and channel redundancy based on the input image. This approach allows SR models to achieve significant speedups and memory reductions without sacrificing image quality, outperforming existing methods in terms of the PSNR-FLOPs trade-off.
What Problem Does This Paper Address?
Modern deep learning-based image super-resolution (SR) systems, while achieving impressive results, are often computationally very expensive and require large amounts of memory. This makes them impractical for deployment on resource-constrained devices like mobile phones or embedded systems. The core problem is that these networks are designed to be static, meaning they use the same amount of computation for every image, even though the complexity and information content of images can vary greatly. This leads to wasted computation on simpler images and insufficient processing for more complex ones. Existing model compression techniques like channel pruning often permanently reduce a network's capacity and fail to adapt to the input-dependent nature of SR tasks.
Methods
The paper proposes a "Multi-Dimensional Dynamic Model Compression" approach that tackles redundancy in two key ways:
- Dynamic Spatial-wise Compression: Instead of always processing feature maps at their full spatial resolution, the method uses a learned "router" to dynamically downscale feature maps before applying convolutions. The degree of downscaling is determined by the input image's characteristics, specifically its spatial information content. This leverages the observation that spatial redundancy can be reduced with minimal information loss.
- Dynamic Channel-wise Compression: For each convolution, a "channel saliency predictor" dynamically determines which channels are most important for the current input image. Using a Gumbel-Softmax trick, the network can selectively activate or skip channels, further reducing computation. This is a significant improvement over static pruning, as it allows the network to adapt its channel usage based on the input.
Both the spatial router and channel saliency predictor are conditioned on feature maps and guided by classic image processing metrics like Spatial Information (SI), which better captures high-frequency details than simple pooling methods. The entire method is designed to be end-to-end trainable using standard SR losses and a sparsity criterion to enforce a computation budget. A two-stage training strategy is employed: first, pre-training the backbone SR network, and then adding and training the dynamic compression modules.
Datasets
The paper uses the following datasets:
- Training: 800 RGB images from DIV2K.
- Evaluation: Four standard benchmark datasets: Set5, Set14, BSD100, and Urban100.
Key Insights and Ideas
- Input-Dependency is Crucial: The paper strongly emphasizes that SR networks exhibit high input-dependency. The importance of channels and the degree of spatial redundancy vary significantly from one image to another. Static compression methods fail to exploit this.
- Spatial Redundancy is Underestimated: The authors demonstrate that spatial downsampling of feature maps can significantly reduce computation with little visual impact, a redundancy that is often overlooked.
- Dynamic Compression for Efficiency: By dynamically adjusting both spatial resolution and channel activation, the proposed method achieves a more efficient use of computational resources, tailoring the network's complexity to the input.
- Two-Stage Training for Better Performance: Pre-training the base SR network before introducing dynamic compression modules helps the model learn more effective compression policies without sacrificing accuracy.
Key Results
The proposed method achieves significant FLOPs reduction with negligible PSNR loss.
- General Applicability: The method is applied to both lightweight (e.g., CARN) and non-compact (e.g., RDN) SR architectures.
- Significant FLOPs Reduction: For EDSR-baseline, the method achieves 50% FLOPs reduction with comparable or slightly better PSNR than the baseline. For super-efficient SR, it achieves up to 60x FLOPs reduction (e.g., compressing EDSR-baseline to 1.9 GFLOPs) while outperforming existing methods like SESR.
- Superior PSNR-FLOPs Trade-off: The compressed models establish a new Pareto frontier, offering better PSNR for a given FLOP budget compared to state-of-the-art NAS-based and manually designed SR methods.
- Qualitative Results: Visual comparisons show that the compressed models produce image quality virtually indistinguishable from the original, with sharper edges and fewer artifacts.
Limitations and Future Work
- Limitations: The paper notes that optimizing from scratch without pre-training can lead to lower PSNR at a target computation budget, suggesting that the interaction between compression and learning is complex.
- Future Work: The authors suggest that their method can be readily applied to various SR architectures and that the learned compression policies can be further analyzed to understand the relationship between image complexity and computational demands. The effectiveness of jointly reducing spatial and channel redundancy is highlighted as a key advantage for future research in efficient deep learning models.
Summary updated