Introduction
Image restoration tasks, such as denoising, deblurring, and JPEG artifact removal, require dense pixel-level predictions. Unlike classification tasks that output a single label, restoration must preserve fine spatial details. Traditional convolutional neural networks (CNNs) often avoid spatial pooling to maintain resolution, leading to high computational costs (FLOPs) that scale poorly with image size. Conversely, Encoder-Decoder architectures like U-Nets use downsampling to improve speed but often struggle to reconstruct high-frequency details during upsampling.
To address this bottleneck, researchers have proposed Subpixel Prediction Networks (SPNs). SPNs are designed to be lightweight and computationally efficient by operating primarily in a downsampled feature space and utilizing a novel attention mechanism to ensure spatial consistency.
Proposed Method
The SPN architecture departs from the standard upsampling decoder paradigm. Instead, it employs an encoder-only structure that predicts a reshaped, downsampled "block-wise" tensor.
Subpixel Prediction
The core idea is to predict a compressed tensor with reduced spatial dimensions but increased channel depth. For instance, a target $100 \times 100$ image might be predicted as a $25 \times 25 \times 16$ tensor. A final "depth-to-space" (pixel shuffling) operation rearranges these channels to produce the full-resolution output. This approach allows the network to stay in a low-resolution, computationally efficient space for the majority of its operations.
Subpixel Block Attention (SBA)
Predicting images in a block-wise fashion can lead to visible discontinuities or "seams" at block boundaries. To mitigate this, the Subpixel Block Attention (SBA) module is introduced. SBA re-calibrates features by analyzing correlations across neighboring subpixel blocks before the final reconstruction, effectively smoothing the output and improving perceptual quality.
Architectural Variants
For specific tasks like JPEG artifact removal, a spatial-to-frequency variant was developed. This version operates in the frequency domain using Discrete Cosine Transform (DCT) basis matrices for global skip connections, aligning the network's learning process with the underlying JPEG compression algorithm. The building blocks of the network are based on inverted residual blocks, utilizing Leaky ReLU to better capture the nuances of image intensity.
Experiments and Results
The SPN framework was evaluated across three restoration tasks: JPEG artifact removal (LIVE1, Classic5), color image denoising (Kodak24, CBSD68, Urban100), and image enhancement (DPED iPhone dataset).
Efficiency and Performance
SPNs were scaled into four variants (XS, S, M, and L) to demonstrate flexibility.
- Denoising: Compared to the standard DnCNN, the SPN-M model achieved comparable PSNR and SSIM scores while requiring 11.5x fewer FLOPs.
- JPEG Deblocking: SPN-S outperformed the well-known AR-CNN while using 7x fewer FLOPs. Qualitatively, the SBA module significantly reduced "staircase" artifacts in smooth background regions.
- Mobile Enhancement: On the DPED dataset, SPN-M was 6.87x faster than baseline models and achieved superior NIQE (Natural Image Quality Evaluator) scores, indicating better perceived quality.
Conclusion
Subpixel Prediction Networks offer a compelling solution for high-resolution image restoration on resource-constrained devices. By prioritizing the reduction of FLOPs through subpixel prediction and ensuring spatial smoothness with the SBA module, SPNs achieve a superior trade-off between processing speed and restoration quality. Future work includes optimizing these networks for specific mobile NPUs and extending the framework to high-factor super-resolution tasks.
Summary updated