OverNet: Lightweight Multi-Scale Super-Resolution with Overscaling Network

Parichehr Behjati, Pau Rodríguez, Armin Mehri, Isabelle Hupont, Carles Fernández Tena, Jordi Gonzàlez

TL;DR

This paper introduces OverNet, a lightweight deep learning model for single image super-resolution (SISR) that can handle arbitrary scaling factors with a single network. It achieves this through a novel recursive feature extractor and an "Overscaling Module" (OSM) that generates intermediate overscaled feature maps, leading to improved accuracy and efficiency compared to existing methods.

What Problem Does This Paper Address?

Single Image Super-Resolution (SISR) aims to reconstruct a high-resolution (HR) image from a low-resolution (LR) input. While deep convolutional neural networks (CNNs) have significantly advanced SISR, they often suffer from two major drawbacks: computational complexity and a lack of generality. As CNNs become deeper and wider for better performance, their computational demands and memory footprint increase, making them impractical for resource-constrained devices like mobile phones. Furthermore, most state-of-the-art SISR models are trained for specific integer scale factors (e.g., x2, x3, x4), requiring separate models for each, which is inefficient and increases memory requirements. This paper addresses the need for a single, lightweight SISR model capable of handling arbitrary scaling factors without sacrificing accuracy.

Methods

OverNet is built upon two core novel components:

  1. Lightweight Recursive Feature Extractor: This module employs a recursive structure of Residual Blocks (RBs) organized into Local Dense Groups (LDGs) and a Global Dense Group (GDG). It leverages both short and long skip connections to efficiently reuse information and combat the degradation of low-level features in deep networks. The use of wide low-rank convolutions and squeeze-and-excitation operations further enhances feature representation.
  2. Overscaling Module (OSM): This is a model-agnostic reconstruction module designed to generate accurate HR images from overscaled feature maps. The key idea is that by overscaling features to a factor of (N+1) for a target maximum scale N, the module generates multiple predictions for the same pixel. This ensemble-like effect is hypothesized to reduce noise and improve accuracy. The OSM can be integrated into existing SR architectures by replacing their standard upsampling modules.

Additionally, OverNet utilizes a Multi-Scale Loss Function. This loss encourages the model to learn to generate accurate HR images across a range of target scales simultaneously, promoting generalization and eliminating the need for separate models for each scale.

Datasets

The paper uses the following datasets:

  • Training: DIV2K dataset.
  • Evaluation: Set5, Set14, B100, and Urban100 benchmarks.

No new benchmarks were introduced.

Key Insights and Ideas

  • Overscaling as an Inductive Bias: The paper proposes that overscaling feature maps before reconstruction acts as a beneficial inductive bias, leading to improved accuracy and reduced artifacts. This is counter-intuitive to the common practice of directly upsampling to the target resolution.
  • Single Model for Arbitrary Scales: The combination of the OSM and the multi-scale loss allows for a single model to perform super-resolution at any arbitrary scale factor, a significant improvement in efficiency and practicality.
  • OSM's Modularity: The OSM is designed to be model-agnostic, meaning it can be plugged into various existing SR architectures to boost their performance, demonstrating its versatility.
  • Recursive Feature Reuse: The recursive structure within the feature extractor effectively propagates information across layers, mitigating the vanishing gradient problem and enhancing feature representation with a relatively low parameter count.

Key Results

  • Performance: OverNet outperforms existing state-of-the-art SISR methods on standard benchmarks (Set5, Set14, B100, Urban100) across various scale factors, often with significantly fewer parameters. For example, on Urban100 with x4 scaling, OverNet achieves 26.31 PSNR with 0.9M parameters, outperforming models with much larger parameter counts.
  • Efficiency: OverNet is described as lightweight, with a low number of parameters and computational cost. Table 8 shows OverNet as the fastest among comparable methods on Urban100 for x4 scaling.
  • OSM Benefits: Integrating the OSM into existing architectures like RDN, EDSR, and CARN consistently improved their PSNR scores across different scales.
  • Generalization: Models trained to target larger scales (e.g., x4) with OverNet showed better performance on smaller scales (e.g., x2, x3) than models trained specifically for those smaller scales, highlighting its generalization capability.

Limitations and Future Work

  • Overscaling Factor: The paper notes that overscaling to higher factors slightly improves PSNR but at the cost of more computation. They settle for N+1 overscaling for a balance.
  • RDN Comparison: While OverNet performs very well, the paper acknowledges that RDN sometimes performs slightly better on certain BD degradation models, though RDN has a significantly higher parameter count.
  • Future Work: The authors suggest that the proposed overscaling method could be beneficial for other low-level image restoration tasks, such as denoising and dehazing. They also imply that further exploration of the OSM's integration with various architectures could yield more advancements.

Summary updated

Abstract

Super-resolution (SR) has achieved great success due to the development of deep convolutional neural networks (CNNs). However, as the depth and width of the networks increase, CNN-based SR methods have been faced with the challenge of computational complexity in practice. More-over, most SR methods train a dedicated model for each target resolution, losing generality and increasing memory requirements. To address these limitations we introduce OverNet, a deep but lightweight convolutional network to solve SISR at arbitrary scale factors with a single model. We make the following contributions: first, we introduce a lightweight feature extractor that enforces efficient reuse of information through a novel recursive structure of skip and dense connections. Second, to maximize the performance of the feature extractor, we propose a model agnostic reconstruction module that generates accurate high-resolution images from overscaled feature maps obtained from any SR architecture. Third, we introduce a multi-scale loss function to achieve generalization across scales. Experiments show that our proposal outperforms previous state-of-the-art approaches in standard benchmarks, while maintaining relatively low computation and memory requirements.
BibTeX
@inproceedings{behjati2021overnet,
  author = {Parichehr Behjati and Pau Rodríguez and Armin Mehri and Isabelle Hupont and Carles Fernández Tena and Jordi Gonzàlez},
  title = {{OverNet: Lightweight Multi-Scale Super-Resolution with Overscaling Network}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=g2icOJFYNy3},
}
Figure