Going Much Wider with Deep Networks for Image Super-Resolution

Vikram Singh, Keerthan Ramnath, Subrahmanyam Arunachalam, Anurag MittalWACV 2020

TL;DR

This paper introduces a novel Wide and Deep Network (WDN) for image super-resolution that tackles the 4x upsampling problem by dividing it into 32 smaller, parallelizable subproblems. This "divide and conquer" approach, combined with a unique pixel calibration technique, leads to sharper and more visually appealing super-resolved images compared to existing state-of-the-art methods.

What Problem Does This Paper Address?

Image Super-Resolution (SR) aims to reconstruct a high-resolution (HR) image from a low-resolution (LR) input. This is crucial for applications ranging from enhancing old photographs to improving medical imaging and display technologies. While deep learning has revolutionized SR, current methods often rely on building deeper and deeper networks, which can be computationally expensive and prone to vanishing gradients. A key challenge in SR is accurately recovering high-frequency details (like edges and textures) which are often lost in the downsampling process, leading to blurry or unnatural-looking results. Existing methods have attempted to address this through various means, including adversarial training and perceptual losses, but there's still room for improvement in both accuracy and efficiency.

Methods

The core innovation of this paper is the Wide and Deep Network (WDN), which employs a "divide and conquer" strategy. For a 4x SR task, the problem is broken down into 32 independent subproblems. These are categorized into two groups:

  • 16 subproblems for all-frequency prediction: These networks predict both low and high-frequency details for different segments of the image.
  • 16 subproblems for high-frequency prediction: These networks specifically focus on reconstructing the finer details, recognizing their critical importance for image sharpness.

The input LR image is processed using a "space-to-depth" operation to create the inputs for these subproblems. The outputs from the 32 parallel networks are then aggregated and fused in an "output module" to produce the final HR image.

A significant novel technique introduced is pixel calibration. This is a self-learned, non-linear layer applied after convolutional layers, designed to scale pixel/feature intensities based on their learned relevance to the upsampling task. This is achieved by computing a relevance score using a sigmoid activation on a convolution output, and then using this score to weigh pixel values, aiming to better preserve important details.

Datasets

The paper uses four publicly available datasets for evaluation:

  • Set5
  • Set14
  • B100
  • Urban100

These datasets are standard benchmarks for image super-resolution research. The authors also mention using images from the DIV2K dataset for training.

Key Insights and Ideas

  • Divide and Conquer for SR: Breaking down a complex SR problem into many smaller, parallel subproblems is an effective strategy. This allows for specialized networks to learn specific aspects of the image reconstruction, potentially leading to better overall performance.
  • Explicit High-Frequency Prediction: Recognizing that high-frequency details are paramount for perceived sharpness, dedicating a significant portion of the network's capacity (half of the subproblems) to their explicit prediction is a key insight.
  • Pixel Calibration as a Learned Feature Enhancement: The proposed pixel calibration technique, which learns pixel relevance, offers a more sophisticated way to adjust pixel intensities than standard activation functions, leading to improved results.
  • Width vs. Depth Trade-off: The WDN architecture emphasizes width (parallel subproblems) over extreme depth, which can offer advantages in parallelization and potentially mitigate some of the training challenges associated with very deep networks.

Key Results

The WDN demonstrates superior performance across all tested datasets (Set5, Set14, B100, Urban100) compared to state-of-the-art methods, as measured by PSNR and SSIM metrics. For example, on the Urban100 dataset, WDN achieved a PSNR of 27.35 and an SSIM of 82.06, outperforming other methods. Qualitative results (shown in Figures 5 and 6) also indicate that WDN produces sharper and more visually pleasing upsampled images with better texture reconstruction. The paper also shows that the proposed pixel calibration technique outperforms alternatives like ReLU and ReLU with Batch Normalization.

Limitations and Future Work

  • Hardware Requirements: Training the full 32 subproblems simultaneously requires multiple GPUs or TPUs. For higher upsampling factors (e.g., 8x), the number of subproblems (and thus hardware requirements) increases significantly. However, the independent nature of subproblems allows for sequential training on limited hardware.
  • Future Work: The authors suggest extending the divide and conquer approach to other tasks like video super-resolution and deblurring. They also propose further improving the architecture of individual deep networks within the WDN.

Summary updated

BibTeX
@inproceedings{singh2020going,
  author = {Vikram Singh and Keerthan Ramnath and Subrahmanyam Arunachalam and Anurag Mittal},
  title = {{Going Much Wider with Deep Networks for Image Super-Resolution}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure