Single Image Super-Resolution via a Dual Interactive Implicit Neural Network

Quan H. Nguyen, William J. BeksiWACV 2023

TL;DR

This paper introduces a novel Dual Interactive Implicit Neural Network (DIINN) for single image super-resolution (SISR) that can handle arbitrary scale factors with a single model. By decoupling content and positional features and leveraging implicit neural representations, DIINN achieves state-of-the-art results on benchmark datasets, offering improved performance and flexibility over existing methods.

What Problem Does This Paper Address?

Single Image Super-Resolution (SISR) is a fundamental computer vision task aiming to enhance the resolution of an image. This is crucial for both improving visual quality for human consumption and enabling more robust machine perception in applications like robotics and remote sensing. Traditional deep learning approaches for SISR often train separate models for specific scale factors (e.g., 2x, 3x, 4x). This is inefficient and impractical, especially for embedded systems with limited resources that might encounter images at various resolutions. Furthermore, existing methods that attempt arbitrary scaling often struggle with generalization to scales beyond their training range. This paper addresses the need for a single, flexible SISR model capable of producing high-resolution images at any desired scale factor, while maintaining high fidelity and performance.

Methods

The core innovation of this paper is the Dual Interactive Implicit Neural Network (DIINN). DIINN is built upon the concept of implicit neural representations, where an image is represented as a continuous function that maps coordinates to pixel values. This allows for sampling at any resolution.

DIINN's architecture consists of two main components:

  • Encoder: This part takes the low-resolution (LR) input image and extracts a deep feature map. Crucially, this encoder does not use any upsampling modules, preserving the spatial content of the LR image. To enrich these features, a "feature unfolding" operation is applied, which concatenates the features of a 3x3 neighborhood around each pixel. Nearest interpolation is used to sample features for query locations, avoiding artifacts from smoother interpolation methods.
  • Dual Interactive Implicit Decoder: This decoder is responsible for predicting the high-resolution (HR) pixel values. It takes two types of features as input:
  • Content Features (z): These are the features extracted by the encoder.
  • Positional Features (p): These encode the location of the pixel being predicted. The paper distinguishes between "global coordinates" (position relative to the entire image) and "local coordinates" (position relative to the nearest LR pixel).

The decoder itself is a modulated periodic activations network, inspired by recent work in implicit representations. A key novelty is the "dual interactive" aspect, where the content and positional features are processed in separate branches and then interact. Specifically, the synthesis branch of the decoder takes the output of the previous synthesis layer and the content features as input, allowing for richer interactions than simply using the output of the previous modulation layer. This design choice is argued to provide better information to the modulation network.

Datasets

The paper uses the following datasets for training and evaluation:

  • DIV2K: Used for training the model.
  • DIV2K validation set: Used for evaluation.
  • Set5, Set14, BSDS100, Urban100: Standard benchmark datasets for SISR evaluation.

No new datasets were introduced in this paper.

Key Insights and Ideas

  • Implicit Representations for SISR: The paper effectively demonstrates that implicit neural representations, which have shown success in 3D vision, are a powerful tool for SISR, enabling arbitrary scale outputs from a single model.
  • Decoupling Content and Position: Separating content features (what the image depicts) from positional features (where it is) and allowing them to interact within the decoder is a key architectural innovation that leads to improved performance.
  • Pixel-Level Representation: By operating at a pixel level rather than using coarser feature patches, the model can capture finer details crucial for SISR.
  • Arbitrary Scale Flexibility: The DIINN architecture inherently supports arbitrary scale factors without requiring retraining, a significant practical advantage over scale-specific models.
  • Efficiency: Despite its advanced capabilities, the DIINN shows competitive runtime performance, making it suitable for real-world applications.

Key Results

  • State-of-the-Art Performance: DIINN achieves state-of-the-art results on various benchmark datasets (DIV2K, Set5, Set14, Urban100) for arbitrary scale upsampling, consistently outperforming methods like Meta-SR and LIIF, especially at larger "inter" and "outer" scales. For example, on the DIV2K dataset at x15 scale, DIINN achieves a PSNR of 22.89, outperforming LIIF (22.82) and Meta-SR (21.63).
  • Improved Visual Quality: Qualitative results show that DIINN produces sharper edges and finer details compared to LIIF, while avoiding the artifacts seen in Meta-SR.
  • Competitive Runtime: Table 4 shows that DIINN's runtime is competitive, being faster than LIIF for larger output sizes (e.g., 95.51ms vs. 185.42ms for 256x256 output).

Limitations and Future Work

  • Artifacts at Higher Scales: While DIINN significantly reduces artifacts compared to other methods, some subtle artifacts can still appear, particularly at very high magnification factors. The paper notes that these artifacts are less distinguishable and do not appear with higher quality LR inputs.
  • Future Work: The authors suggest exploring ways to adapt the architecture for a larger receptive field and improved encoder-decoder interactions. They also propose adapting the method to learn a "space of predictions" to better address the inherent ill-posed nature of SISR.

Summary updated

BibTeX
@inproceedings{nguyen2023single,
  author = {Quan H. Nguyen and William J. Beksi},
  title = {{Single Image Super-Resolution via a Dual Interactive Implicit Neural Network}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure