Image restoration is a tug-of-war between preserving sharp details and removing unwanted noise or artifacts. For decades, researchers have relied on "priors"—mathematical assumptions about what a clean image should look like—to guide this process. Historically, these priors fell into two camps: local and nonlocal.
Recent research has converged on a powerful realization: these two perspectives are not redundant, but complementary. By integrating local sparsity with nonlocal self-similarity, models can recover images with a level of fidelity that neither approach could achieve alone.
The Local View: Edges and Gradients
Local priors operate on the immediate neighborhood of a pixel. The most famous example is Total Variation (TV), which assumes that images are "piecewise smooth"—meaning gradients should be small except at sharp edges.
As Jian Zhang, Debin Zhao, and their colleagues explain in Improved Total Variation based Image Compressive Sensing Recovery by Nonlocal Regularization, local methods are excellent at defining edges. However, they suffer from a "staircase effect," where smooth gradients are turned into artificial blocks, and they often smooth away fine, repeating textures because they only look at a tiny window of the image at a time.
The Nonlocal View: Patterns and Redundancy
Natural images are full of repetition. A patch of pixels representing a brick on one side of a building likely looks almost identical to a patch on the other side. Nonlocal priors exploit this "self-similarity."
Instead of looking at neighbors, these methods search the entire image for similar patches and group them together. In Joint Adaptive Sparsity and Low-Rankness on the Fly, Bihan Wen and Yoram Bresler demonstrate how these groups can be treated as low-rank matrices. Because the patches are so similar, the "true" signal is low-rank, while the noise is high-rank. By enforcing a low-rank constraint, the model can effectively "average out" noise across distant parts of the image.
Why the Hybrid Approach Works
The intuition behind the hybrid direction—often called Joint Statistical Modeling (JSM) or Joint Patch-Group (JPG) modeling—is that local and nonlocal priors fail in opposite ways.
- Local priors are "short-sighted." They preserve unique, one-off details but struggle with noise and global structure.
- Nonlocal priors are "far-sighted." They are incredible at recovering repeating textures but can over-smooth unique features that don't have a "twin" elsewhere in the image.
By combining them, as seen in Joint Patch-Group Based Sparse Representation for Image Inpainting by Xin Yuan and Zhiyuan Zha, the model uses local sparsity to keep edges sharp and nonlocal similarity to ensure structural consistency.
Unified Optimization: The STROLLR Framework
The technical challenge is how to mathematically "glue" these two priors together. If you apply them sequentially, you might introduce artifacts from one that the other can't fix.
The state-of-the-art approach is to solve them as a single, joint optimization problem. In The Power of Complementary Regularizers, Bihan Wen and Yanjun Li introduce STROLLR, which uses "Transform Learning" (local) and "Low-Rank Modeling" (nonlocal) simultaneously.
This is often achieved using the Split Bregman Iteration or ADMM (Alternating Direction Method of Multipliers). These algorithms break the complex joint problem into smaller, solvable steps that pass information back and forth. The local prior says, "This edge looks too blurry," while the nonlocal prior says, "This patch doesn't match its brothers across the image." Through iteration, the model finds a solution that satisfies both constraints, leading to reconstructions that are both sharp and structurally sound.
Beyond Static Images
This dual-prior logic extends naturally to video and complex sensing. In Reconstruction for block-based compressive sensing of image with reweighted double sparse constraint, Yuanhong Zhong and Jing Zhang show that this "double sparse" constraint is particularly effective for Compressive Sensing, where the data is already highly incomplete. By leveraging both the local geometry and the global redundancy, these models can "hallucinate" missing information with much higher accuracy than traditional methods.