Phone cameras and other small devices restore images with networks that can only afford to spend heavy computation on a few regions. One way to do this is to run a cheap network over the whole frame, then pass only the "hard" patches through a deeper network. The decision about which patches are hard is learned on synthetic training data. When the device meets real sensor noise or real rain, that decision is the first thing to break. Real noise raises high-frequency energy everywhere and varies across the frame with vignetting, brightness, and color channel. The patch selector then either sends almost everything to the deep network, which wipes out the savings, or it spends its budget chasing noise instead of structure. We propose to make the selector robust to this shift, and to concentrate domain adaptation on the patches the deep network actually sees.
The proposal has three parts. First, a local adaptive threshold, borrowed from document binarization, decides which patches are hard. Each patch is compared with its own neighborhood rather than with a fixed global top-$K$ cutoff. Second, the patch score is a single number computed from a three-channel color error estimate, so thresholding stays well defined in color. Third, an adversarial domain loss, applied only on the routed patches, adapts the deep branch from synthetic to real data. The discriminator is thrown away after training, so the deployed model is unchanged in cost. The claim we test is that this approach gets most of the quality of adapting the whole network, at a fraction of the inference compute and with far lower training cost than diffusion-based adaptation.
Core Claim
Content-adaptive restoration has two separate failures under domain shift. The routing becomes miscalibrated, and the deep branch is unadapted. Both can be fixed cheaply if the routing threshold is local and the adaptation is spatially masked by the routing decision. The prediction is specific. On synthetic-to-real denoising, measured on the Smartphone Image Denoising Dataset (SIDD, A High-Quality Denoising Dataset for Smartphone Cameras), a routed network with local thresholds and masked adaptation should get within a few tenths of a dB of a fully adapted uniform network. It should do so while refining well under half the patches. Its router should also rank true per-patch error on real data much better than a source-trained top-$K$ router.
Where This Sits
The routing design comes from Restore Globally, Refine Locally. That paper splits a super-resolution network into a light Base-Net and a deep Refine-Net, and uses a mask predictor trained on synthetic bicubic data to send the top-$K$ under-restored patches to the Refine-Net. It reports FLOP cuts of 10% to 48%. We keep the split and change two things: a fixed ratio becomes a local threshold, and a closed-domain predictor becomes an adapted one. Dynamic Domain Adaptation for Efficient Inference is the closest conceptual neighbor. It attaches domain discriminators to every early exit of a classifier, so that adaptive computation survives domain shift. It works at the level of whole images and class confidences. Our work is the dense, spatial version, where the "exit" is a per-patch decision and the quantity being adapted is a continuous restoration.
Two adaptation methods motivate and constrain the design. Denoising as Adaptation shows that plain feature-space adversarial alignment, such as the gradient-reversal approach of DANN, and pixel translation, such as CyCADA, both lose 3–4 dB to its diffusion-teacher approach on SIDD. That paper treats every region uniformly, and it needs a large pretrained diffusion model during training. This is a warning that naive global alignment destroys texture. We respond by restricting alignment to routed patches and to a branch that only refines a base output. We do not claim to beat that paper's quality. FAN-Net shows that domain "style" sits largely in low-frequency Fourier amplitude. It normalizes that amplitude globally on grayscale MRI, combined with a gradient-reversal classifier. We reuse its insight as an optional input normalization. Our point of departure is that real camera degradations vary spatially within one image, and a global spectral fix cannot capture that.
The thresholding comes from two places. Classic local binarization compares a value with local statistics in sliding windows, as in Wu and Feng. Feng's dual-window variant, also described in that line of work, adds a larger window for regional dynamic range to handle steep illumination gradients. PatchRefineNet shows that deep networks have systematic regional over- and under-confidence. It fixes this with optimal per-patch thresholds, but only on scalar logit maps. The authors note that vector-valued thresholds are combinatorial.
That limitation is where the color problem enters. Color pixels have no natural ordering, which is why Morphological Networks for Image De-raining stays single-channel despite needing only 2,700 parameters. It is also why MuLUT had to add explicit cross-channel indexing. We sidestep the ordering problem by not thresholding vectors at all. Instead, we threshold a learned scalar norm of a color error vector, so chroma noise counts without needing per-channel cutoffs.
Research Program
The backbone is NAFNet from Simple Baselines for Image Restoration, an activation-free restoration network that reaches 40.30 dB on SIDD with supervised training. We split it by depth into a shallow Base path, run on the full frame, and a deep Refine path, run on selected 64×64 patches. A small head on the Base features predicts, per pixel, a three-channel error estimate $\hat e(x) \in \mathbb{R}^3$. On source data this head is trained against the true residual. The patch score is
$$s_p = \frac{1}{|p|}\sum_{x \in p} \sqrt{\hat e(x)^\top \Sigma^{-1} \hat e(x)},$$
where $p$ is a patch, $|p|$ its pixel count, and $\Sigma$ a learned 3×3 color covariance. A diagonal $\Sigma$ with luminance-only weighting recovers the grayscale case. Patch $p$ is routed to the Refine path when
$$s_p > \mu_{N(p)} + k\,\sigma_{N(p)} \quad \text{or} \quad s_p > \tau_0,$$
where $\mu_{N(p)}$ and $\sigma_{N(p)}$ are the mean and standard deviation of scores over a larger window $N(p)$ around the patch, $k$ is a learned scalar, and $\tau_0$ is a global floor. The floor keeps uniformly bad frames from being under-refined. During training, a sigmoid relaxation of this rule produces a soft mask $m_p$.
Adaptation adds a small patch discriminator $D$ on Refine-path features $f_p$. It is trained through a gradient reversal layer with the loss $\sum_p m_p \cdot \mathrm{BCE}(D(f_p), d)$, where $d$ marks source versus target. A second, weaker reversal term aligns the Base features that feed the error head, so that the router itself sees domain-invariant inputs. On unlabeled target data, the error head gets a self-consistency signal: patches that change a lot when refined should score high. The Base path's other parameters stay frozen after source training. This keeps the update small, in the spirit of the sparse-update argument in APM.
Because routing produces a variable number of patches, which causes branch divergence on accelerators, inference gathers routed patches into fixed-size batches padded to a few bucket sizes. We report wall-clock latency, not just MACs.
Baselines
The naive reference is a source-only uniform NAFNet. It is trained on clean images with synthetic Poisson-Gaussian noise and tested on SIDD. The 26 dB versus 30 dB gap reported for synthetic versus real training of DnCNN in the SIDD paper shows how large this domain gap is. All methods share the same source data, the same unlabeled SIDD training images, the same NAFNet width, and the same number of optimizer steps and hyperparameter trials.
The routing baseline is MGA-style top-$K$ routing, trained source-only. It is evaluated both at its source-chosen $K$ and with $K$ set to match our method's average refined fraction, so the comparison isolates where compute goes rather than how much is spent. We beat it if, at equal refined fraction, we gain PSNR on SIDD and achieve higher rank correlation between router score and true per-patch error.
The strongest adaptation competitor is Denoising as Adaptation applied to the full uniform network. Its paper reports a +8.13 dB PSNR gain over a vanilla baseline on SIDD, and an improvement for Uformer-B from 29.65 dB to 33.99 dB. On real deraining it reports SPA rising from 33.04 dB to 34.39 dB. We will rerun it on our NAFNet backbone under our protocol rather than borrowing those numbers. We do not expect to exceed its PSNR. Beating it means landing within about 0.3 dB of its SIDD PSNR while running fewer than 60% of its inference MACs and using no diffusion model in training, measured in GPU-hours. A global DANN on the uniform network is included because it is the obvious cheap alternative, and that paper reports it trailing by 3–4 dB.
Supervised NAFNet trained on SIDD (40.30 dB in its paper) is an oracle ceiling, not a competitor.
Four ablations isolate the ingredients. The first replaces the local threshold with a global quantile recalibrated on target. The second removes the mask from the adversarial loss so alignment is global. The third drops the router-input alignment. The fourth swaps the Mahalanobis score for a luminance-only score and for independent per-channel thresholds.
Why not just…?
Why not just recalibrate the top-$K$ ratio or a global threshold on target data? This is the most serious threat. It is cheap, needs no adversarial training, and might fix most of the miscalibration if real noise only shifts the score distribution uniformly. The argument against it is that SIDD noise varies within a frame with brightness and sensor position, which is the illumination-inhomogeneity setting where global binarization fails and local windows help. Our first ablation settles the question directly. If global quantile recalibration matches the local threshold in PSNR and error-ranking correlation, we will report that the local threshold is unnecessary and narrow the contribution to masked adaptation.
Why not just adapt the whole network with DANN or Denoising as Adaptation and then apply MGA routing afterward? This sequential pipeline is a fair alternative, and we run it. Global DANN is reported to lose 3–4 dB to Denoising as Adaptation on SIDD, which suggests uniform feature alignment damages texture. Denoising as Adaptation itself needs a diffusion teacher during training. Adapting first and routing second also leaves the router trained on source errors. We test whether masking the adaptation to routed patches recovers DANN's lost quality, using the second ablation, and whether the post-hoc router stays miscalibrated, using error-ranking correlation.
Why not just threshold luminance, or each color channel separately? Per-channel thresholds reproduce the combinatorial vector-threshold problem PatchRefineNet flags. MuLUT found it needed cross-channel indexing for color fidelity. Luminance-only scores are cheap but blind to chroma noise, which is prominent in smartphone noise. The fourth ablation reports PSNR and a chroma error measure on SIDD. If luminance-only scoring matches the Mahalanobis score, we will use it and drop the color machinery.
Proposed Experiments
The primary experiment is synthetic-to-real denoising, with SIDD validation as the test set. We report PSNR and SSIM, inference MACs, refined-patch fraction, padded-batch latency on a mobile CPU, and training GPU-hours. A second experiment, synthetic rain to real SPA deraining, tests whether the conclusions transfer to a structured degradation. Here the 33.04 to 34.39 dB result from Denoising as Adaptation is the reference.
The decisive diagnostic is router calibration on target. Because SIDD has ground truth, we can compute true per-patch error after the Base path. We report Spearman correlation and top-decile recall of each router's choices against that error, and break the results down by SIDD brightness level, since illumination drives spatial inhomogeneity.
The proposal fails in two ways. It fails if a recalibrated global threshold matches local thresholding on these diagnostics. It also fails if masked adversarial adaptation lands more than about 0.5 dB below Denoising as Adaptation at matched refined fraction. Either outcome would be informative. The first would show that routing under shift is a simple calibration problem. The second would show that cheap adversarial alignment cannot substitute for a strong generative teacher, even when it is spatially focused.
References
- A High-Quality Denoising Dataset for Smartphone Cameras
- Restore Globally, Refine Locally: A Mask-Guided Scheme to Accelerate Super-Resolution Networks
- Dynamic Domain Adaptation for Efficient Inference
- Denoising as Adaptation: Noise-Space Domain Adaptation for Image Restoration
- Fan-Net: Fourier-Based Adaptive Normalization for Cross-Domain Stroke Lesion Segmentation
- High Recognition Ratio Image Processing Algorithm of Micro Electrical Components in Optical Microscope
- PatchRefineNet: Improving Binary Segmentation by Incorporating Signals from Optimal Patch-wise Binarization
- Morphological Networks for Image De-raining
- Toward DNN of LUTs: Learning Efficient Image Restoration With Multiple Look-Up Tables
- Simple Baselines for Image Restoration
- Gradient-Guided Parameter Mask for Multi-Scenario Image Restoration Under Adverse Weather