PNEN: Pyramid Non-Local Enhanced Networks

Feida Zhu, Chaowei Fang, Kai-Kuang Ma

TL;DR

PNEN introduces the Pyramid Non-local Block (PNB), a plug-and-play module that allows neural networks to capture long-range pixel dependencies across multiple scales efficiently. By using a full-resolution "query" map against a downsampled "pyramid" of reference maps, the authors achieve state-of-the-art results in edge-preserving smoothing and significant boosts in image denoising and super-resolution—all while using 62% less memory than standard non-local methods.

What Problem Does This Paper Address?

Most low-level vision tasks (like removing noise or sharpening edges) rely on Convolutional Neural Networks (CNNs). However, standard convolutions are "local"—they only look at a tiny neighborhood of pixels at a time. To understand the "big picture" or find similar patterns far away in an image, you usually have to stack dozens of layers to increase the receptive field, which is computationally expensive and often fails to capture distant correlations.

Existing "non-local" blocks attempt to solve this by calculating the relationship between every pair of pixels in an image. The catch? This operation is $O(N^2)$, meaning memory and processing requirements explode as image resolution increases. Researchers previously dodged this by either limiting the search to a small window or downsampling the entire feature map, both of which sacrifice the fine-grained detail needed for high-quality image restoration.

Methods

The authors propose the Pyramid Non-local Block (PNB). Its architecture is built on two clever design choices:

  1. Asymmetric Resolution: It maintains a query feature map at full resolution (to preserve pixel-level detail) but compares it against a pyramid of reference feature maps that are downscaled.
  2. Multi-Scale Correlation: Instead of just looking at one downsampled version, it creates a pyramid (typically 3 scales) using parallel convolutions with different strides and kernel sizes. This allows the model to find similarities between a single pixel and larger "texels" (texture elements) at different scales.

The paper also introduces the Pyramid Non-Local Enhanced Network (PNEN), which combines these PNBs with Dilated Residual Blocks (DRB). Dilated convolutions further expand the receptive field without losing resolution, creating a powerful pipeline for edge-preserving smoothing.

Datasets

  • Edge-Preserving Smoothing: A dataset of 500 clear images (400 training / 100 testing) used to reproduce classical filters like Weighted Median, $L_0$ smoothing, and SD Filter.
  • Image Denoising: Trained on the Berkeley Segmentation Dataset (BSD300); evaluated on Set12, BSD68, and Urban100.
  • Super-Resolution (SISR): Trained on DIV2K and Flickr2K; evaluated on Set5, Set14, BSD100, and Urban100.

Key Insights and Ideas

  • Efficiency through Asymmetry: You don't need the reference map to be high-resolution to find meaningful correlations. By downscaling the "keys" and "values" in the non-local operation, the memory footprint drops from 8.5G to 3.2G (a 62.4% reduction) compared to standard non-local blocks.
  • Robustness via Pyramids: Similar patterns in nature often appear at different scales (e.g., small bricks vs. large building blocks). The pyramid structure allows the network to "see" these multi-scale similarities, making the correlation estimation much more robust to noise.
  • Plug-and-Play Versatility: The PNB isn't just for one specific model. The authors proved this by inserting PNBs into existing heavyweights like RDN (Residual Dense Network) and MemNet, resulting in consistent performance gains across the board.

Key Results

  • Smoothing Superiority: In reproducing $L_0$ smoothing, PNEN outperformed the previous best method (ResNet) by 1.14 dB PSNR, producing much cleaner and flatter regions while keeping edges sharp.
  • Denoising & SR Boosts: When added to the RDN architecture for image denoising ($\sigma=50$), the PNB-enhanced version achieved higher PSNR across all test sets (e.g., +0.25 dB on Urban100) compared to the baseline.
  • Visual Quality: Qualitatively, the PNB-enhanced models are significantly better at recovering structural textures (like the slats of a fence or the grid of a skyscraper) that standard CNNs often blur or distort.

Limitations and Future Work

The authors note that while the pyramid structure is efficient, the largest scales still require relatively large convolution kernels to capture high-level feature maps. They suggest that a potential future direction is to recursively generate these multi-scale feature maps using fixed, smaller kernels to further reduce the parameter count and computational overhead. Additionally, while the PNB is efficient, it still adds some complexity compared to pure local CNNs, which might be a factor for real-time mobile applications.

Summary updated

Abstract

Existing neural networks proposed for low-level image processing tasks are usually implemented by stacking convolution layers with limited kernel size. Every convolution layer merely involves in context information from a small local neighborhood. More contextual features can be explored as more convolution layers are adopted. However it is difficult and costly to take full advantage of long-range dependencies. We propose a novel non-local module, Pyramid Non-local Block, to build up connection between every pixel and all remain pixels. The proposed module is capable of efficiently exploiting pairwise dependencies between different scales of low-level structures. The target is fulfilled through first learning a query feature map with full resolution and a pyramid of reference feature maps with downscaled resolutions. Then correlations with multi-scale reference features are exploited for enhancing pixel-level feature representation. The calculation procedure is economical considering memory consumption and computational cost. Based on the proposed module, we devise a Pyramid Non-local Enhanced Networks for edge-preserving image smoothing which achieves state-of-the-art performance in imitating three classical image smoothing algorithms. Additionally, the pyramid non-local block can be directly incorporated into convolution neural networks for other image restoration tasks. We integrate it into two existing methods for image denoising and single image super-resolution, achieving consistently improved performance.
BibTeX
@article{zhu2020pnen,
  author = {Feida Zhu and Chaowei Fang and Kai-Kuang Ma},
  title = {{PNEN: Pyramid Non-Local Enhanced Networks}},
  journal = {IEEE Trans. Image Process.},
  year = {2020},
  url = {https://openreview.net/forum?id=kQ58vGmr2b},
}

Loading related content...

Figure