Night photos taken with a handheld phone are both dark and noisy everywhere, and blurry only in some places, such as a moving pedestrian, a swinging light, or the frame edges during a hand twitch. The best current restorers for this case still spend the same computation on every pixel. The clearest example is algorithm unrolling, which turns a fixed number of iterations of a classical optimizer into network layers, each with a learned denoising sub-network. On a typical night photo, most of that computation goes to regions that needed only denoising and brightening and were finished after an iteration or two. We propose to let each image tile stop iterating independently. The stopping rule comes from the physics rather than from a learned difficulty oracle: a tile halts once its reconstruction explains the observed blurry, noisy pixels to within the noise level the sensor model predicts. Every tile gets a guaranteed cheap denoise-and-enhance pass. Only tiles whose residual stays above the noise floor, which in practice are the blurred ones, keep receiving the expensive iterations.
The idea borrows the central lesson of adaptive computation in language models: uniform budgets waste compute on easy inputs and starve hard ones (Learning How Hard to Think). It then fixes the weakness that blocks existing spatially adaptive restorers in the dark. Those restorers assume that "not blurry" means "clean." Under low light, no pixel is clean.
Core Claim
In an unrolled joint deblur–denoise solver, the right amount of computation for a region is set by how far its current estimate is from explaining the data at the noise floor. This quantity can be measured locally at every iteration, without ground truth. We claim that per-tile halting on a noise-normalized data residual matches the reconstruction quality of the full fixed-depth solver at a substantially lower average cost. We further claim it beats three simpler options at equal computation: a globally shallower solver, per-image early exit, and binary blur-mask gating. It should also avoid the visible seams that binary gating produces under low light.
The mechanism is a local version of the classical discrepancy principle, the rule that you stop an iterative inverse solver when the residual reaches the noise level. Let $B$ be the observed image, $K$ the estimated spatially varying blur, $R \circ L$ the current reflectance-times-illumination estimate at iteration $t$, and $\hat\sigma^2_p$ the per-pixel variance predicted by a signal-dependent (shot plus read) noise model evaluated on the current estimate. Tile $j$ is halted at the first iteration $t$ where
$$ \rho_j^{(t)} = \frac{1}{|j|}\sum_{p \in j} \frac{\big(B_p - (K \otimes (R^{(t)} \circ L^{(t)}))_p\big)^2}{\hat\sigma_p^2} \le \tau, $$
with $\tau$ near 1. A value of 1 means the tile explains the data exactly to the noise level. Going below it would mean fitting the noise itself, which is where the classical rule gets its justification. A learned correction on $\tau$ is permitted but is treated as an ablation, not the core method.
Relation to Prior Work
The method builds on JUDE. JUDE unrolls an augmented Lagrangian solver over a Retinex decomposition (image = reflectance × illumination) combined with blur convolution. It interleaves closed-form updates with residual-U-Net priors for kernel estimation, reflectance denoising and illumination enhancement. It runs a fixed number of iterations on every image and identifies its compute cost as a main limitation. JUDE supplies both the backbone and the explicit forward model that makes our residual computable. The halting signal is exactly the data term JUDE already optimizes. What JUDE lacks is any way to stop.
Two spatially adaptive deblurrers motivate the design by failing in the regime we target. SPAIR predicts a binary degraded/clean mask and restricts sparse convolutions to degraded pixels. LMD-ViT prunes transformer windows it judges sharp, reporting 66% fewer FLOPs than Restormer/Uformer baselines and a +0.28 dB gain on the well-lit ReLoBlur benchmark. Both methods bypass "clean" regions. In a dark scene those regions still carry heavy sensor noise, so bypassing them leaves noise untouched and creates seams. Our design differs in two ways. No tile is ever skipped entirely, since every tile gets the first iteration. And the allocation is a continuous depth rather than keep-or-prune.
U-WADN selects a nested sub-network width per image with a learned selector, reporting 32.3% fewer FLOPs than AirNet. It handles isolated degradations only, and the whole image gets one width. Its selector is a direct precedent for the learned-predictor ablation below.
The allocation logic comes from language-model work. Damani et al. train a predictor of the marginal gain from extra compute. That predictor needs verifiers on many samples, and we replace it with a residual that needs no labels at test time. Early-stopping self-consistency halts sampling once answers agree, which is the same "stop when further work changes nothing" logic, applied there to discrete outputs. Hidden-state difficulty probes (Zhu et al., Zhao et al.) suggest that encoder features may already predict how many iterations a tile needs. We test this as a competitor to the physics signal rather than assuming it.
LEDNet and VJT establish that the joint problem cannot be cascaded. LEDNet's cascade of MIMO-UNet plus Zero-DCE reaches 17.52 dB against LEDNet's 25.74 dB on LOL-Blur. This constrains our design: the cheap path for halted tiles must be the joint solver's own first iteration, not a separate denoiser.
Research Program
The first stage retrains JUDE's unrolled network with deep supervision, meaning a reconstruction loss at every iteration. This makes a halted tile's intermediate estimate a valid output. We also add a stochastic halting schedule during training so that later iterations learn to work alongside frozen neighbours.
The second stage implements tile-wise freezing. The closed-form updates are cheap and stay global. The expensive prior sub-networks run only on active tiles plus a one-tile halo, with frozen tiles supplying fixed context. Execution is tile-batched rather than per-pixel sparse, because per-pixel sparsity rarely turns into wall-clock gains. The latency-aware spatial dynamic networks in LASNet provide the engineering template.
The third stage calibrates the noise model per dataset. For LOL-Blur this uses the synthesis parameters. For real data it uses dark-frame or flat-region statistics.
The analysis stage asks whether halting depth tracks true local blur extent. On synthetic data we know the per-pixel motion, so we can measure this directly. We also ask whether the rule stays sensible on real captures where the noise model is only approximate.
A secondary extension applies the same residual gate between VJT's progressive decoder tiers on MLBN video. This is explicitly lower priority, because VJT's tiers are not iterations of a solver and the residual has a weaker justification there.
Baselines
The naive reference is JUDE at its full fixed iteration count, retrained with our deep supervision so that any quality change is attributable to halting and not to the new loss. The published JUDE numbers anchor it: 26.884 PSNR and 0.932 SSIM on LOL-Blur. No reported FLOP or latency figure for JUDE was found on Lacuna, so we will measure both ourselves on the same GPU. Beating it means staying within 0.1 dB PSNR while cutting the average prior-network cost by at least 40%.
The strongest existing competitor on quality is JUDE itself. The strongest on efficiency is LMD-ViT-style gating ported into the same backbone. A lightweight blur-confidence head predicts a binary mask, masked tiles get all iterations, and unmasked tiles get exactly one iteration. We deliberately give it the one-iteration denoise floor, which LMD-ViT itself lacks, so that it is the strongest version of the binary idea. It uses the same data, the same backbone weights, a tuning sweep over mask threshold equal in size to our sweep over $\tau$, and matched average FLOPs. Beating it means higher PSNR at matched FLOPs and a lower boundary-seam score. The seam score is the gradient discontinuity measured along routing boundaries relative to the ground truth.
A uniform-budget baseline runs the retrained solver with a smaller fixed iteration count chosen to match our average FLOPs. A per-image early-exit baseline applies the same residual rule to the whole image. We also include LEDNet (25.74 dB on LOL-Blur) as an unadaptive joint network of a different architecture.
Two ablations isolate the key ingredient. The first replaces the residual rule with a learned per-tile predictor of iterations needed, a two-layer probe on encoder features in the style of Zhao et al. and Damani et al., trained with oracle stopping depths. The second removes the noise normalization, halting on the raw residual or on the change between iterations.
Why not just…?
Why not just run fewer iterations everywhere? At the same average cost, fewer uniform iterations must shortchange the blurred tiles, and those tiles dominate the error. U-WADN's gains over static networks point the same way, although for width rather than depth. The uniform-budget baseline at matched FLOPs settles this directly. If it comes within 0.1 dB of our method, the adaptive machinery is not worth its complexity, and we would report that outcome.
Why not just halt the whole image once its residual is small? Night blur is often local, so one blurred subject would keep the whole frame iterating. The per-image early-exit baseline measures this. We expect the gap to be large on locally blurred scenes and to vanish on globally blurred ones. To make the dependence visible, we report results stratified by blurred-area fraction.
Why not just gate with a binary blur mask, but give the background a cheap denoiser? This is the most serious threat, since it captures much of our idea with a simpler mechanism. Two things separate the methods. Blur severity varies continuously, so a binary split either over-spends on mildly blurred tiles or under-spends on severe ones. And a separate cheap path invites the seams that the cascade evidence warns about. We gave the gating baseline the solver's own first iteration as its floor so that the comparison is fair. If it matches us in both PSNR and seam score, the conclusion is that continuous depth adds nothing, and the contribution reduces to "never skip, always denoise."
Why not just learn how many iterations each tile needs? A learned predictor may be sharper in-distribution. However, it needs oracle stopping labels, and like the verifier-dependent predictors in language models, it can drift under shift. The residual rule needs no labels and is tied to the sensor model. The learned-probe ablation, evaluated on both LOL-Blur and Real-LOL-Blur, decides which to prefer.
Proposed Experiments
The main benchmark is LOL-Blur, with 10,200 training and 1,800 test pairs, scored by PSNR, SSIM and LPIPS against measured FLOPs and latency, with quality–compute curves traced by sweeping $\tau$. Real-LOL-Blur (1,354 unpaired night photos) tests transfer, scored with the no-reference metrics ARNIQA and LIQE that JUDE uses. No reported numeric results for those metrics were found on Lacuna, so all methods will be scored ourselves.
The decisive diagnostic is a dark local-blur set built by applying LOL-Blur's darkening and noise synthesis to ReLoBlur, which contains moving subjects on sharp backgrounds. Here binary gating should fail visibly, and the halting map can be compared against the known moving-object masks.
Three checks decide the proposal. First, at matched FLOPs, our method must beat both uniform-depth and binary gating on dark ReLoBlur. Second, per-tile halting depth must correlate with ground-truth local blur magnitude, which confirms the rule tracks blur rather than brightness or texture. Third, on Real-LOL-Blur, the savings must hold without a no-reference quality drop relative to full-depth JUDE. If the noise-normalized rule fails the third check while the learned probe passes it, we will conclude that sensor-model mismatch, not the allocation idea, is the bottleneck.
References
- Learning How Hard to Think: Input-Adaptive Allocation of LM Computation
- Deep Joint Unrolling for Deblurring and Low-Light Image Enhancement (JUDE)
- Spatially-Adaptive Image Restoration using Distortion-Guided Networks
- Adaptive Window Pruning for Efficient Local Motion Deblurring
- Unified-Width Adaptive Dynamic Network for All-In-One Image Restoration
- Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning
- The LLM Already Knows: Estimating LLM-Perceived Question Difficulty via Hidden Representations
- Optimizing Reasoning Efficiency through Prompt Difficulty Prediction
- LEDNet: Joint Low-Light Enhancement and Deblurring in the Dark
- VJT: A Video Transformer on Joint Tasks of Deblurring, Low-light Enhancement and Denoising