Some image restoration networks are small and interpretable because each layer is one step of a classical optimization algorithm. In this "unrolled" design, the network repeatedly nudges a corrupted image toward one whose pixel values change little between pixels a learned graph marks as similar, and changes sharply only across edges. Interpretable Lightweight Transformer via Unrolling of Learned Graph Smoothness Priors uses this design for interpolation and demosaicking. Its model, uGTV, learns the pixel-similarity graph from shallow CNN features and shows that learning this graph is functionally a form of self-attention. It uses about 319k parameters and beats the 10.9M-parameter MAIN transformer by roughly 0.7 dB PSNR. To keep cost linear, however, it connects each pixel only to its 5×5 neighbourhood. That brings back the limitation of convolutional networks: a pixel cannot draw directly on a distant but similar pixel, such as the matching window three facades away in an Urban100 street scene.
We propose to keep the unrolled graph-total-variation solver but run it over a sparse graph with global reach. Each pixel keeps its 5×5 local edges. It also gains a few learned, content-selected edges along its own row and column, following the criss-cross pattern of CCNet. Because two row/column hops connect any two pixels, the graph has global reach at $O(N\sqrt{N})$ selection cost and $O(N)$ solver cost. Graph total variation (an $\ell_1$ penalty on neighbour differences, which yields piecewise-flat rather than blurred solutions) then decides how much each long-range neighbour contributes. The claim is that this helps images with repeated long-range structure and costs nothing on images without it. Global attention is known to act as a blurring low-pass filter (Butterworth as Attention), and an $\ell_1$ prior should avoid that failure.
Core Claim
An unrolled solver whose graph adds a small number of similarity-selected row/column edges will beat the same solver restricted to local edges on interpolation and demosaicking. The gain should be concentrated on images with high long-range self-similarity and at high missing-pixel rates, because at high missing rates local neighbourhoods carry little information. The comparison holds parameters, unrolled iterations and edge budget fixed. The crucial qualifier is that the gain must come from content-selected distant edges, not merely from greater reach.
A local-graph solver already propagates information across the image as its iterations accumulate. Reach alone is therefore not new. What is new is that the long edges point directly at similar content.
Research Program
Each unrolled layer solves, approximately, the following problem:
$$\min_x \; \|y - Hx\|_2^2 + \mu \sum_{(i,j)\in E_{\text{loc}}\cup E_{\text{cc}}} w_{ij}\,|x_i - x_j|.$$
Here $y$ is the observed image, $H$ is the sampling or mosaicking operator, and $x$ is the restored image. $\mu$ is a learned per-layer strength. $E_{\text{loc}}$ holds the 5×5 edges. $E_{\text{cc}}$ holds, for each pixel, the top-$k$ pixels in its row and column ranked by weight.
The weights are $w_{ij}=\exp(-(f_i-f_j)^\top M (f_i-f_j))$, where $f$ are shallow CNN features and $M$ is a learned low-rank metric. These are the same Mahalanobis weights uGTV uses, so the only architectural change is the edge set.
Selecting the top $k$ neighbours makes the graph directed. We symmetrize by taking the union, and we treat a directed variant as a secondary ablation. This follows up the open question in the research file about whether asymmetric graphs help.
The ADMM iterations are unrolled exactly as in uGTV. Each iteration costs $O(N(25+2k))$, and we use $k\in\{2,4,8\}$. For contrast, Total Variation Graph Neural Networks supplies a total-variation layer only for fixed irregular graphs. Coupled Segmentation and Edge Learning via Dynamic Graph Propagation reaches far across the image, but through four fixed directional sweeps with a hard edge gate. That gate is fragile under noisy boundaries, as its authors report on COCO Panoptic. Our long edges are instead chosen by content, and a convex prior weights them softly rather than a learned gate.
Analysis accompanies the model. We will compute a per-image long-range self-similarity score: the fraction of 7×7 patches whose nearest neighbour lies outside the 5×5 window within the same row or column. We then correlate the per-image PSNR gain with this score. We will also measure how the solutions differ in frequency content. Specifically, we compare edge sharpness across boundaries against a quadratic Laplacian variant on the same long-range graph, to test whether the $\ell_1$ prior is what prevents attention-style blur.
Baselines
The naive reference is uGTV exactly as published, with 5×5 edges and trained on DIV2K. We retrain it with our code and the same iteration count, so every comparison shares data, schedule and tuning budget. We use an equal hyperparameter-search budget for all models on a DIV2K validation split.
The strongest existing method is uGTV itself, since it was shown to beat MAIN. For absolute reference, the research file reports only that uGTV "outperformed the SOTA model by approximately 0.7 dB" PSNR with 319k versus 10.9M parameters. No absolute PSNR on Urban100, Kodak or McM was found on Lacuna. We will therefore also retrain MAIN, if code is available, under the same protocol. Beating uGTV means a PSNR gain on Urban100 that holds across three seeds and exceeds the seed-to-seed spread, together with no loss on Kodak and McM.
Three ablations isolate the key ingredient. A random-long-edge control adds $2k$ row/column edges per pixel chosen at random rather than by weight. A quadratic control swaps the $\ell_1$ penalty for the Laplacian quadratic, solved by unrolled conjugate gradient as in uGLR. A matched-reach local control enlarges the window until edges-per-pixel and FLOPs match our model.
Why not just…?
Why not just run more unrolled iterations on the 5×5 graph? Information already travels further with every iteration, so this is the most serious threat. The answer is that diffusion through many local hops averages over everything along the path. A direct edge copies from the matching structure. We will sweep the local model's iteration count up to our model's FLOP budget. If it closes the gap on high-self-similarity images, the claim fails and we will report that.
Why not just enlarge the local window? An 11×11 window adds reach but no content selection, and its cost grows with window area. The matched-FLOPs local control tests this directly. We expect it to help on Kodak textures but not on repetitions farther apart than the window.
Why not just put a CCNet attention block in front of uGTV? This gives long-range context in the features, which then shape local weights. The problem is that attention aggregation behaves as a low-pass filter (Butterworth as Attention), and the pixels it would pull from never enter the solver's objective. We include this hybrid at matched parameters as a baseline. Our model should win on boundary sharpness even if PSNR ties.
Why not just apply a dense CRF afterwards? A dense CRF, a classical post-processing step that smooths predictions using all pixel pairs, can refine edges after the fact. But SmooSeg attributes its main gains to the in-training smoothness prior rather than to CRF refinement. A post-hoc step also cannot supply the missing pixels that interpolation needs. We treat it as a cheap reference rather than a competitor.
Proposed Experiments
We train on DIV2K and evaluate on Urban100, Kodak and McM, covering interpolation at several missing-pixel rates and Bayer demosaicking. We report PSNR, SSIM, parameters, FLOPs and peak memory.
Three checks are decisive. First, the gain over uGTV must be larger on Urban100 than on Kodak and McM, and it must correlate positively with the per-image self-similarity score. Second, the random-long-edge control must recover little of that gain, which shows content selection matters. Third, the model must beat the iteration-matched and FLOP-matched local controls.
A secondary test measures robustness to additive noise on the inputs. This checks whether noisy features choose wrong long edges, the restoration analogue of the edge-gate failure reported for Dynamic Graph Propagation.
If the self-similarity correlation is absent, or the iteration-matched local model matches ours, we will conclude that the 5×5 locality in uGTV was not a real limitation for these tasks.
References
- Interpretable Lightweight Transformer via Unrolling of Learned Graph Smoothness Priors
- CCNet: Criss-Cross Attention for Semantic Segmentation
- Butterworth as Attention: Anisotropic Spectral Gating for Pansharpening
- Total Variation Graph Neural Networks
- Coupled Segmentation and Edge Learning via Dynamic Graph Propagation
- SmooSeg: Smoothness Prior for Unsupervised Semantic Segmentation