Most classical image restoration models—whether for super-resolution, deblurring, or denoising—rely on a convenient lie: the assumption of spatial invariance. This assumption treats an entire image as if it were degraded by a single, uniform process, such as a global bicubic downsampling or a lone Gaussian blur kernel.
In the laboratory, this works perfectly. In the real world, it fails. A photograph of a street scene might have sharp details in the foreground, a soft "bokeh" blur in the background due to a shallow depth of field, and a directional motion blur on a passing car. As Radu Timofte and colleagues noted in the NTIRE 2018 Challenge on Single Image Super-Resolution, moving beyond these idealized conditions requires models that can handle the messy, unpredictable, and non-uniform degradations of actual photography.
The Failure of Global Assumptions
When a model assumes a uniform blur kernel but encounters an image with varying blur, it produces artifacts. If it over-sharpens to fix a blurry background, it creates "ringing" or "halos" around foreground objects. Conversely, if it optimizes for the sharp foreground, the background remains muddy.
This problem is particularly acute in "blind" restoration, where the degradation kernel is unknown. Kai Zhang and Wangmeng Zuo demonstrated with DPSR (Deep Plug-And-Play Super-Resolution) that while deep learning can adapt to arbitrary kernels, the standard framework still struggles when those kernels change across the frame. Real-world degradation is a moving target; as Xiao Jin and Lei Zhang argue, lens optics and scene depth mean that blur patterns are inherently pixel-wise variations rather than global constants.
Predicting the Map of Degradation
To solve this, researchers have shifted from predicting a single kernel to predicting a kernel map. Instead of one $k$ for the whole image, the model estimates a unique $k_{i,j}$ for every pixel $(i,j)$.
A landmark approach in this direction is the Laplacian Pyramid based Kernel Prediction Network (LP-KPN), introduced by Jianrui Cai and Lei Zhang in their work on RealSR. By using real-world image pairs captured at different focal lengths, they trained a network to recover details using these per-pixel kernels. This allows the model to apply heavy restoration where the blur is intense and light restoration where the image is already sharp.
Similarly, Guolei Sun and Jingyun Liang proposed MANet (Mutual Affine Network), which uses a localized receptive field to estimate spatially variant kernels. By constraining the network's "view" to a small neighborhood, it can more accurately identify the specific blur type affecting a local patch without being distracted by different degradations elsewhere in the image.
Adaptive Restoration and Aesthetic Intent
Restoration isn't just about mathematical sharpness; it's about preserving the photographer's intent. A major pitfall of uniform restoration is the destruction of "bokeh"—the aesthetic blur in a background. Soo Ye Kim and Munchurl Kim addressed this with KOALAnet (Kernel-Oriented Adaptive Local Adjustment).
KOALAnet works by jointly learning to estimate the spatially-variant blur and then using that estimate to guide the reconstruction. If the model identifies a region as having intentional background blur, it adapts its restoration to avoid over-sharpening, maintaining the artistic quality of the photo while still enhancing the high-frequency details of the subject.
Efficiency through Unfolding and RNNs
The computational cost of applying a different kernel to every pixel is potentially massive. To make this practical, researchers have turned to "deep unfolding" and specialized architectures.
- Spatially Variant RNNs: Jiawei Zhang and Jinshan Pan moved away from standard CNNs toward Spatially Variant Recurrent Neural Networks. By learning unique weights for every pixel and processing them through an RNN, they achieved state-of-the-art deblurring that is 10x faster and smaller than previous models.
- Linearized Optimization: Andres Almansa and Charles Laroche developed PnP-LADMM (Plug-and-Play Linearized ADMM). This framework "unfolds" a classical optimization algorithm into a neural network, allowing it to handle spatially varying kernels without the need for expensive matrix inversions. This makes it possible to apply sophisticated neural denoisers to images with complex, non-uniform motion blur.
Beyond Photography: Medical and Astronomical Imaging
The need for spatially variant modeling extends into specialized scientific fields. In medical imaging, Tzu-An Song and Joyita Dutta showed that PET scanner blur changes based on a pixel's distance from the center of the device. Similarly, in astronomy, the GREAT10 Challenge highlighted how atmospheric and instrumental distortions vary across the field of view, requiring spatially aware models to accurately map dark matter.
By treating image degradation as a local, adaptive phenomenon rather than a global constant, these researchers have bridged the gap between theoretical image processing and the complex reality of how light actually hits a sensor.