Self-Organized Residual Blocks For Image Super-Resolution

Onur Keles, A. Murat Tekalp, Junaid Malik, Serkan KiranyazICIP 2021 · OpenReview

TL;DR

This paper introduces "Self-Organized Residual" (SOR) blocks, a novel neural network architecture for image super-resolution (SR) that uses Taylor series expansions to learn more flexible non-linear transformations. By combining these SOR blocks with traditional residual blocks in a hybrid approach, the authors achieve improved SR performance (PSNR, SSIM, and perceptual metrics) compared to state-of-the-art methods, often with fewer parameters.

What Problem Does This Paper Address?

Traditional convolutional neural networks (CNNs) for image restoration tasks like super-resolution (SR) often rely on a homogeneous structure, typically using the Rectified Linear Unit (ReLU) activation function. While the universal approximation theorem guarantees that neural networks can learn any function, it doesn't specify the optimal architecture. This homogeneity can limit the network's ability to capture complex non-linear relationships efficiently, potentially requiring a large number of neurons and layers. The paper aims to address this by introducing more flexible and "self-organizing" neuron models that can adapt their non-linear behavior, leading to potentially better performance with greater computational efficiency.

Methods

The core innovation is the Self-Organized Residual (SOR) block. These blocks are analogous to standard residual blocks but replace the conventional convolutional layers with Self-Organized Operational Neural Network (Self-ONN) layers. Self-ONN layers approximate non-linear functions using a Taylor series expansion up to a specified order ($q$). This allows the network to learn a more tailored non-linear transformation for each connection, rather than being constrained by a fixed activation like ReLU.

The authors propose two main architectural strategies:

  1. Pure Self-ONN: Replacing all residual blocks in a standard SR architecture (like EDSR) with SOR blocks.
  2. Hybrid Architecture: Combining regular residual blocks with SOR blocks, and also replacing the upsampling/output layers with Self-ONN layers. This hybrid approach aims to balance the benefits of stronger non-linearity from SOR blocks with the established performance of regular residual blocks.

Datasets

The paper uses the DIV2K dataset for training and evaluation of image super-resolution models. They specifically mention using 800 training images and 100 validation images from this dataset.

Key Insights and Ideas

  • Taylor Series for Flexibility: The use of Taylor series expansion in Self-ONN layers provides a mechanism for learning adaptive non-linear transformations, moving beyond the fixed nature of ReLU. This allows the network to better approximate complex functions required for SR.
  • Hybridization is Key: Combining regular residual blocks with SOR blocks and Self-ONN upsampling/output layers yielded the best results. This suggests that a complete shift to Self-ONN might not be optimal, and a strategic integration can leverage the strengths of both approaches.
  • Efficiency Gains: The proposed Self-ONN architectures, particularly the hybrid ones, achieve comparable or better performance than the baseline EDSR model with potentially fewer parameters and blocks, indicating improved computational efficiency.
  • "Self-Organized" Neuron Behavior: The term "self-organized" highlights the network's ability to learn the optimal non-linear kernel transformation dynamically during training, rather than relying on a pre-defined set of operators.

Key Results

  • Performance Improvement: Both pure Self-ONN and hybrid architectures consistently outperformed the EDSR baseline in terms of PSNR, SSIM, and LPIPS (a perceptual metric).
  • Hybrid Dominance: The hybrid model, which replaced the last four residual blocks with SOR blocks and the upsampling/output layers with Self-ONN layers, achieved the best results. For $\times 4$ SR, this hybrid model achieved a PSNR of 29.064, compared to EDSR's 28.970 (random initialization).
  • Parameter Efficiency: The hybrid model with 12 regular residual blocks and 4 SOR blocks achieved superior performance to a pure Self-ONN with 8 SOR blocks, despite having a similar number of parameters (around 2.3-2.7 million for $\times 2/\times 4$ SR).
  • Hyperparameter Tuning: Table 1 shows that increasing the number of SOR blocks and neurons per layer generally leads to better performance in pure Self-ONN models.

Limitations and Future Work

  • Computational Cost of SOL: While Self-ONN offers flexibility, a Self-ONN layer with order $q$ has approximately $q$ times more learnable parameters than a conventional convolutional block. The paper notes that exhaustive search for ONNs can be expensive, and Self-ONNs are presented as a more efficient alternative.
  • Generalizability of Hybrid Strategy: The paper explored replacing specific sets of blocks (first/last 4) in the hybrid architecture. Further research could investigate optimal placement and number of SOR blocks for different SR tasks or network architectures.
  • Future Work: The authors suggest that Self-ONNs offer a path towards achieving ultimate network heterogeneity and expressive power, implying further exploration of their capabilities in various domains beyond SR. The potential for faster inference due to parallelizable layer computations is also highlighted.

Summary updated

Abstract

It has become a standard practice to use the convolutional networks (ConvNet) with RELU non-linearity in image restoration and super-resolution (SR). Although the universal approximation theorem states that a multi-layer neural network can approximate any non-linear function with the desired precision, it does not reveal the best network architecture to do so. Recently, operational neural networks (ONNs) that choose the best non-linearity from a set of alternatives, and their “self-organized” variants (Self-ONN) that approximate any non-linearity via Taylor series have been proposed to address the well-known limitations and drawbacks of conventional ConvNets such as network homogeneity using only the McCulloch-Pitts neuron model. In this paper, we propose the concept of self-organized operational residual (SOR) blocks, and present hybrid network architectures combining regular residual and SOR blocks to strike a balance between the benefits of stronger non-linearity and the overall number of parameters. The experimental results demonstrate that the proposed architectures yield performance improvements in both PSNR and perceptual metrics.
BibTeX
@inproceedings{keles2021self,
  author = {Onur Keles and A. Murat Tekalp and Junaid Malik and Serkan Kiranyaz},
  title = {{Self-Organized Residual Blocks For Image Super-Resolution}},
  booktitle = {ICIP},
  year = {2021},
  url = {https://openreview.net/forum?id=DPE065b3U7r},
}
Figure