WaveMixSR: Resource-Efficient Neural Network for Image Super-Resolution

Pranav Jeevan, Akella Srinidhi, Pasunuri Prathiba, Amit SethiWACV 2024

TL;DR

This paper introduces WaveMixSR, a novel neural network for image super-resolution that leverages a 2D discrete wavelet transform for efficient spatial token-mixing. By combining the inductive biases of CNNs with the wavelet transform's ability to capture local details without the quadratic complexity of self-attention, WaveMixSR achieves state-of-the-art performance on the BSD100 dataset with significantly fewer parameters and computational resources than existing transformer-based models.

What Problem Does This Paper Address?

Image super-resolution (SR) aims to enhance the resolution of low-resolution (LR) images to produce high-resolution (HR) counterparts. This is a crucial task with applications in various fields like medical imaging, surveillance, and digital photography. While deep learning, particularly transformer-based models, has achieved remarkable progress in SR, these models often suffer from high computational complexity due to the quadratic scaling of self-attention mechanisms with respect to input size. This makes them resource-intensive and less suitable for scenarios with limited data or computational power. Convolutional Neural Networks (CNNs), on the other hand, are more efficient but struggle to capture long-range dependencies effectively. The paper addresses this gap by proposing a resource-efficient SR model that combines the strengths of both approaches.

Methods

The core innovation of WaveMixSR lies in its WaveMix block, which replaces the computationally expensive self-attention mechanism in transformers with a 2D discrete wavelet transform (2D-DWT) for spatial token-mixing. This approach offers several advantages:

  • Resource Efficiency: 2D-DWT provides a parameter-free token-mixing mechanism that reduces computational complexity and memory requirements compared to self-attention.
  • Inductive Bias: By incorporating the inductive biases of convolutions, WaveMixSR benefits from inherent image properties, leading to better generalization with less training data.
  • Lossless Downsampling: The wavelet transform's invertibility ensures that no image information is lost during the down-sampling operation, which is crucial for SR.
  • Architecture: WaveMixSR processes the Y channel of the YCbCr color space separately for SR, as the Y channel contains most of the image details. The CbCr channels are upsampled and then concatenated with the processed Y channel. The model utilizes a series of WaveMix blocks for feature extraction and enhancement.

Datasets

  • Training: DIV2K dataset was used for training WaveMixSR.
  • Evaluation: The model's performance was tested on several benchmark datasets: BSD100, Urban100, Set5, and Set14.

Key Insights and Ideas

  • Wavelet Transform as a Viable Alternative to Self-Attention: The paper demonstrates that the 2D-DWT can effectively replace self-attention for spatial token-mixing in SR, offering significant computational savings without sacrificing performance. This opens up new avenues for designing efficient deep learning models for vision tasks.
  • Combining Inductive Biases and Token-Mixing: The successful integration of CNN inductive biases with wavelet-based token-mixing highlights the power of hybrid architectures for image restoration tasks. This approach allows the model to learn effectively from limited data while still capturing complex image structures.
  • Parameter Efficiency is Key for Real-World SR: The significant reduction in parameters and computational cost (e.g., half the parameters of SwinIR and one-fourth of HAT) makes WaveMixSR a strong candidate for deployment in resource-constrained environments.
  • YCbCr Color Space Separation is Beneficial: Processing the Y channel separately for SR and then merging it with the upsampled CbCr channels proved to be an effective strategy, aligning with previous SR research.

Key Results

  • State-of-the-Art Performance: WaveMixSR achieved state-of-the-art performance on the BSD100 dataset for 2x SR, outperforming previous methods like HAT-L by 0.34 dB in PSNR. It also showed competitive results on other benchmark datasets.
  • Resource Efficiency: The model is highly parameter-efficient, requiring significantly fewer parameters and computational resources (e.g., 1.7M parameters compared to 11.8M for SwinIR and 20.8M for HAT).
  • No Pre-training Required: WaveMixSR achieved its impressive performance without the need for ImageNet pre-training or training on larger datasets like DF2K, further emphasizing its efficiency and data-friendliness.
  • Qualitative Improvements: Visual results showed WaveMixSR's ability to regenerate fine details, sharpen textures, and reconstruct facial features effectively, even in challenging cases.

Limitations and Future Work

  • Further Improvement with Pre-training: The authors suggest that further performance gains could be achieved by training with larger datasets like DF2K or employing ImageNet pre-training, similar to other state-of-the-art models.
  • Adversarial Training: Exploring adversarial training techniques is proposed as a potential avenue for further improvements.
  • Mother Wavelet Choice: While the Haar wavelet was used due to its simplicity, investigating other mother wavelets might yield different performance characteristics.
  • Loss Function Exploration: While Huber loss was found to be optimal, further exploration of loss functions and their combinations could be beneficial.

Summary updated

BibTeX
@inproceedings{jeevan2024wavemixsr,
  author = {Pranav Jeevan and Akella Srinidhi and Pasunuri Prathiba and Amit Sethi},
  title = {{WaveMixSR: Resource-Efficient Neural Network for Image Super-Resolution}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure