Block-Attentive Subpixel Prediction Networks for Computationally Efficient Image Restoration

Taeoh Kim, Chajin Shin, Sangjin Lee, Sangyoun LeeIEEE Access 2021 · DOI · OpenReview

Introduction

Image restoration tasks, such as denoising, deblurring, and JPEG artifact removal, require dense pixel-level predictions. Unlike classification tasks that output a single label, restoration must preserve fine spatial details. Traditional convolutional neural networks (CNNs) often avoid spatial pooling to maintain resolution, leading to high computational costs (FLOPs) that scale poorly with image size. Conversely, Encoder-Decoder architectures like U-Nets use downsampling to improve speed but often struggle to reconstruct high-frequency details during upsampling.

To address this bottleneck, researchers have proposed Subpixel Prediction Networks (SPNs). SPNs are designed to be lightweight and computationally efficient by operating primarily in a downsampled feature space and utilizing a novel attention mechanism to ensure spatial consistency.

Proposed Method

The SPN architecture departs from the standard upsampling decoder paradigm. Instead, it employs an encoder-only structure that predicts a reshaped, downsampled "block-wise" tensor.

Subpixel Prediction

The core idea is to predict a compressed tensor with reduced spatial dimensions but increased channel depth. For instance, a target $100 \times 100$ image might be predicted as a $25 \times 25 \times 16$ tensor. A final "depth-to-space" (pixel shuffling) operation rearranges these channels to produce the full-resolution output. This approach allows the network to stay in a low-resolution, computationally efficient space for the majority of its operations.

Overview of the Subpixel Prediction Network architecture, illustrating the transition from input to compressed feature blocks and the final reconstruction.

Subpixel Block Attention (SBA)

Predicting images in a block-wise fashion can lead to visible discontinuities or "seams" at block boundaries. To mitigate this, the Subpixel Block Attention (SBA) module is introduced. SBA re-calibrates features by analyzing correlations across neighboring subpixel blocks before the final reconstruction, effectively smoothing the output and improving perceptual quality.

Architectural Variants

For specific tasks like JPEG artifact removal, a spatial-to-frequency variant was developed. This version operates in the frequency domain using Discrete Cosine Transform (DCT) basis matrices for global skip connections, aligning the network's learning process with the underlying JPEG compression algorithm. The building blocks of the network are based on inverted residual blocks, utilizing Leaky ReLU to better capture the nuances of image intensity.

Experiments and Results

The SPN framework was evaluated across three restoration tasks: JPEG artifact removal (LIVE1, Classic5), color image denoising (Kodak24, CBSD68, Urban100), and image enhancement (DPED iPhone dataset).

Efficiency and Performance

SPNs were scaled into four variants (XS, S, M, and L) to demonstrate flexibility.

  • Denoising: Compared to the standard DnCNN, the SPN-M model achieved comparable PSNR and SSIM scores while requiring 11.5x fewer FLOPs.
  • JPEG Deblocking: SPN-S outperformed the well-known AR-CNN while using 7x fewer FLOPs. Qualitatively, the SBA module significantly reduced "staircase" artifacts in smooth background regions.
  • Mobile Enhancement: On the DPED dataset, SPN-M was 6.87x faster than baseline models and achieved superior NIQE (Natural Image Quality Evaluator) scores, indicating better perceived quality.

Conclusion

Subpixel Prediction Networks offer a compelling solution for high-resolution image restoration on resource-constrained devices. By prioritizing the reduction of FLOPs through subpixel prediction and ensuring spatial smoothness with the SBA module, SPNs achieve a superior trade-off between processing speed and restoration quality. Future work includes optimizing these networks for specific mobile NPUs and extending the framework to high-factor super-resolution tasks.

Summary updated

Abstract

Image restoration based on the Deep Convolutional Neural Network (CNN) based image restoration has demonstrated promising results in many sub-tasks, such as image super-resolution, compression artifacts removal, image denoising, and image enhancement. Compared to many CNN-based high-level vision tasks that predict sparse probabilities of each class, the CNN for image restoration requires dense pixel-level predictions with precise intensity-level values. Therefore, a minimum number of spatial pooling (or down-sampling) operations is required to maintain the image details. Therefore, designing a fast or lightweight model for image restoration is a difficult problem and is even more critical when the spatial resolution of an image becomes larger. In this paper, we propose a family of networks called Subpixel Prediction Networks (SPNs) that predict reshaped and spatially down-sampled block-wise tensors instead of raw images with full resolution. Under this scheme, spatial downsampling decreases the restoration performance less while making the network faster. We propose a novel Subpixel Block Attention (SBA) module that re-calibrates blockwise features to diminish blockwise discontinuity to increase the performance further. The experimental results reveal that these networks demonstrate good trade-offs between speed (number of computations) and restoration performance in the three image restoration tasks: image compression artifacts removal, color image denoising, and image enhancement.
BibTeX
@article{kim2021block,
  author = {Taeoh Kim and Chajin Shin and Sangjin Lee and Sangyoun Lee},
  title = {{Block-Attentive Subpixel Prediction Networks for Computationally Efficient Image Restoration}},
  journal = {IEEE Access},
  year = {2021},
  doi = {10.1109/access.2021.3091975},
}
Figure