Standard convolutional neural networks (CNNs) operate on a fundamental assumption: locality. They process images by sliding small filters over local neighborhoods, assuming that the most relevant information for reconstructing a pixel lies in its immediate vicinity.
However, natural images are rarely just collections of local gradients. They are filled with redundancy. A brick on the left side of a building looks nearly identical to a brick on the right; a texture of grass in the foreground repeats in the background; the left side of a human brain often mirrors the right. This property is known as Non-Local Self-Similarity (NSS).
Researchers have increasingly found that by breaking the "locality" constraint, models can borrow high-resolution details from distant but visually similar regions to reconstruct degraded areas with far greater precision than local filters ever could.
The Intuition of Global Redundancy
If you are trying to denoise a blurry patch of a checkered shirt, a local filter only sees the blur. But if the model can look across the entire image, it might find a sharp patch of the same fabric elsewhere. By "averaging" or "transferring" information from that distant, clean patch, the model can resolve the blur.
This concept was pioneered in classical algorithms like Non-Local Means and BM3D, but it has been reimagined for the deep learning era. Saeed Izadi and colleagues in Image Denoising in the Deep Learning Era describe this shift as a move toward "non-local feature aggregation," where attention layers or graph structures identify and combine related patches from distant parts of an image.
From Pixels to Graphs
One of the most effective ways to capture these relationships is to treat an image not as a grid, but as a graph. In this framing, pixels or patches are nodes, and edges are drawn between them based on feature similarity rather than spatial distance.
Diego Valsesia, Enrico Magli, and Giulia Fracastoro introduced this in Deep Graph-Convolutional Image Denoising. Their GCDN architecture uses dynamic graph convolutions to find non-local self-similarities. Similarly, Xiaobin Hu and Wenqi Ren applied this to medical imaging in Feedback Graph Attention Convolutional Network for MR Images Enhancement, leveraging the brain's natural symmetry to restore missing details in MRIs.
Patch Grouping and Sparse Coding
Before the dominance of pure neural networks, researchers used Structural Group Sparse Representation (SGSR) to exploit NSS. As explained by Jian Zhang and Debin Zhao in Structural Group Sparse Representation for Image Compressive Sensing Recovery, the idea is to stack similar patches into a "group" matrix. Because these patches are so similar, the resulting matrix is "low-rank," meaning it can be represented very sparsely.
Modern methods have "unrolled" these mathematical optimizations into neural networks. Dror Simon and Michael Elad developed CSCNet in Rethinking the CSC Model for Natural Images, which uses a lightweight network to approximate these global redundancies. By combining the interpretability of sparse coding with the speed of deep learning, they achieved state-of-the-art results with significantly fewer parameters.
Transcending the Local Window
While CNNs are limited by their kernel size, Transformers have naturally stepped into this space because their self-attention mechanism is inherently non-local. However, standard Transformers are computationally expensive for high-resolution images.
To solve this, Zhisheng Lu and Juncheng Li proposed the Efficient Super-Resolution Transformer (ESRT) in Transformer for Single Image Super-Resolution, which uses a "Feature Split" attention to capture long-range dependencies without the massive memory overhead.
Further pushing this boundary, Leheng Zhang and Yawei Li introduced an Adaptive Token Dictionary in Transcending the Limit of Local Window. Instead of looking at pixels in a fixed grid, their model groups pixels based on visual similarity, allowing it to "see" repetitive architectural details across the entire frame as a single coherent structure.
Applications Across Domains
The power of non-local search extends beyond standard photography:
- Remote Sensing: Yule Duan and Xiao Wu developed CANConv in Content-Adaptive Non-Local Convolution for Remote Sensing Pansharpening, which clusters similar textures like forests or urban grids across satellite images to apply specialized filters.
- Microscopy: In Multi-Resolution Data Fusion for Super-Resolution Electron Microscopy, Suhas Sreehari and Katherine L. Bouman used high-resolution "patches" to build a library of textures, which was then used to "fill in" details across a much larger low-resolution scan.
- Anomaly Detection: Thibaud Ehret and Jean-michel Morel took the opposite approach in How to Reduce Anomaly Detection in Images to Anomaly Detection in Noise. By identifying and removing everything that repeats (the non-local background), they could isolate the unique, non-repeating anomalies hiding in the image.
By treating an image as a global collection of related patterns rather than a local grid of pixels, these researchers have unlocked a more robust way to distinguish between random noise and the underlying structural "truth" of a scene.