Abstract
Existing neural networks proposed for low-level image processing tasks are usually implemented by stacking convolution layers with limited kernel size. Every convolution layer merely involves in context information from a small local neighborhood. More contextual features can be explored as more convolution layers are adopted. However it is difficult and costly to take full advantage of long-range dependencies. We propose a novel non-local module, Pyramid Non-local Block, to build up connection between every pixel and all remain pixels. The proposed module is capable of efficiently exploiting pairwise dependencies between different scales of low-level structures. The target is fulfilled through first learning a query feature map with full resolution and a pyramid of reference feature maps with downscaled resolutions. Then correlations with multi-scale reference features are exploited for enhancing pixel-level feature representation. The calculation procedure is economical considering memory consumption and computational cost. Based on the proposed module, we devise a Pyramid Non-local Enhanced Networks for edge-preserving image smoothing which achieves state-of-the-art performance in imitating three classical image smoothing algorithms. Additionally, the pyramid non-local block can be directly incorporated into convolution neural networks for other image restoration tasks. We integrate it into two existing methods for image denoising and single image super-resolution, achieving consistently improved performance.
Introduction
Most low-level vision tasks, such as image restoration and smoothing, rely on Convolutional Neural Networks (CNNs). However, standard convolutions are inherently local, capturing only a small neighborhood of pixels. To understand global context or find similar patterns far away in an image, networks typically require stacking many layers to increase the receptive field. This approach is computationally expensive and often fails to capture distant correlations effectively.
Existing non-local blocks attempt to solve this by calculating the relationship between every pair of pixels. However, this operation has a complexity of $O(N^2)$, where $N$ is the number of pixels. This leads to an explosion in memory and processing requirements as image resolution increases. Previous attempts to mitigate this involved limiting the search window or downsampling the entire feature map, both of which sacrifice the fine-grained detail necessary for high-quality restoration.
Methodology
Pyramid Non-local Block (PNB)
The authors propose the Pyramid Non-local Block (PNB), which introduces two key innovations to overcome the limitations of standard non-local operations:
- Asymmetric Resolution: The PNB maintains a query feature map at full resolution to preserve pixel-level detail, but it compares it against a pyramid of reference feature maps that are downscaled.
- Multi-Scale Correlation: Instead of a single downsampled map, it creates a pyramid (typically 3 scales) using parallel convolutions with different strides. This allows the model to find similarities between a single pixel and larger texture elements ("texels") at various scales, making the correlation estimation more robust to noise.
Pyramid Non-Local Enhanced Network (PNEN)
The Pyramid Non-Local Enhanced Network (PNEN) integrates these PNBs with Dilated Residual Blocks (DRB). Dilated convolutions further expand the receptive field without losing resolution, creating a powerful pipeline for edge-preserving smoothing. By downscaling the "keys" and "values" in the non-local operation, the memory footprint is reduced from 8.5G to 3.2G—a 62.4% reduction compared to standard non-local blocks.
Experiments and Results
Edge-Preserving Smoothing
PNEN was evaluated on its ability to reproduce classical filters like Weighted Median and $L_0$ smoothing. In $L_0$ smoothing tasks, PNEN outperformed previous state-of-the-art methods like ResNet by 1.14 dB PSNR. Qualitatively, the network produces much cleaner and flatter regions while maintaining sharp edges.
Image Denoising and Super-Resolution
The PNB is a versatile "plug-and-play" module. When integrated into existing architectures like RDN (Residual Dense Network) for image denoising ($\sigma=50$), the PNB-enhanced version achieved higher PSNR across all test sets, including a +0.25 dB gain on the Urban100 dataset. The models are particularly effective at recovering structural textures, such as grids or slats, that standard CNNs often distort.
Conclusion
PNEN introduces an efficient mechanism for capturing long-range dependencies in low-level vision tasks. By utilizing an asymmetric pyramid structure, the Pyramid Non-local Block significantly reduces memory consumption while improving performance across smoothing, denoising, and super-resolution tasks. Future work may explore recursive generation of multi-scale maps to further optimize parameter counts for mobile applications.
Summary updated