Borrowing Texture Statistics from Far Away to Stop Denoisers from Over-Smoothing

Image denoisers face a basic ambiguity. Real fine detail, such as fur, fabric weave or foliage, and real sensor noise occupy the same high spatial frequencies. When a network cannot tell them apart from the small neighborhood it sees, minimizing squared error pushes it to average both away, and the output looks plastic. The usual fix is a wider receptive field, the region of the image a network can draw on for each output pixel. Non-local attention, transposed attention and deformable sampling all let a pixel draw on similar content elsewhere. But these mechanisms mostly aggregate pixel values or features from distant matches. For stochastic textures, two patches of the same grass are similar in distribution, not pixel by pixel, so averaging them is itself a smoothing operation.

We propose a different use of long-range context. The network uses distant look-alike regions only to estimate how much and what kind of high-frequency energy each region of the clean image should contain, which is a local gradient histogram. It then requires the local denoiser to reproduce that budget. Averaging statistics across many noisy repeats of a texture gives a good estimate of the clean statistics, and it needs no pixel alignment. In short, wide receptive fields should supply a region-specific detail budget, not more pixels to average.

Core Claim

Over-smoothing in strong learned denoisers is mostly a miscalibrated estimate of local high-frequency energy. A wide-context module that pools gradient statistics from texture-matched regions, rather than pooling features, can correct that estimate. It should recover texture that is correlated with ground truth and not merely energetic. The claim makes two predictions that would falsify it. First, detail loss in current models should shrink when an image contains distant repeats of the same texture and our module is present, but not when only a feature-pooling module of equal reach is present. Second, the gain should concentrate at low noise levels, where deep denoisers are known to over-correct.

Relation to Prior Work

Two older results motivate the proposal. Texture Enhanced Image Denoising via Gradient Histogram Preservation showed that clean gradient histograms are stable enough to estimate from noisy images, and that enforcing them restores texture grit. Its reported result is 27.10 dB at $\sigma=30$, against 27.34 dB for CSR. It has three limits: it imposes a single histogram on the whole image, it runs as a slow variational loop, and it assumes Gaussian noise. We make the histogram regional, estimate it from learned long-range matches, and train it end to end on real noise. Non-Local Video Denoising by CNN ran an oracle experiment in which patches were searched on clean frames, and found that the main bottleneck is match accuracy under noise, not CNN capacity. That finding is why we match on noise-robust low-resolution statistic descriptors rather than raw pixel distances. It also explains why we aggregate statistics: an imperfect match corrupts a pixel average far more than it corrupts a histogram average.

Three architectures supply the competing way of widening context. Residual Non-local Attention Networks for Image Restoration uses full pairwise spatial attention with an identity shortcut that protects detail, but its quadratic cost limits it to two such blocks. Restormer gets linear cost by attending across channels, so all spatial communication is local depth-wise convolution. Spatial-Adaptive Network for Single Image Denoising bends its sampling grids along edges, but they stay within a local offset range. All three widen context by moving features. Our module moves only a small summary of texture energy, which is cheap and robust to misalignment.

Other work constrains the design. A High-Quality Denoising Dataset for Smartphone Cameras and Benchmarking Denoising Algorithms with Real Photographs show that synthetic Gaussian assumptions break on real heteroscedastic noise. We therefore cannot reuse the parametric noise deconvolution in the gradient-histogram paper, and must learn the noisy-to-clean histogram mapping on real pairs. DVDnet documents that deep denoisers over-smooth at $\sigma=10$ while beating patch methods at high noise. This sets the low-noise regime as our primary test. Dual-Path Adversarial Denoising Network Based on UNet pursues detail through adversarial training and reports instability. This is the main reason we do not lean on a discriminator.

Research Program

The work begins with a diagnostic study, done before any new architecture is built. Using SIDD and DND crops, we segment images into regions. For each region we measure the energy deficit, which is the gap between the output's local gradient histogram and the ground truth's, for Restormer and SADNet. We plot the deficit against two quantities: noise level, and the amount of similar texture elsewhere in the image, judged by clean-image histogram similarity. If the deficit does not fall with the amount of distant similar texture under an oracle statistic estimate, the premise fails and we stop.

The proposed module has three parts. First, a soft-binned, differentiable local gradient histogram is computed on early backbone features at one-eighth resolution, so that $N$ is small enough for dense or top-$k$ matching. Second, each region retrieves its $k$ most similar regions anywhere in the image in this descriptor space. A small network maps the pooled noisy statistics, together with an estimated local noise level, to a predicted clean histogram $\hat h_r$ for region $r$. This network is supervised directly by the ground-truth histogram $h_r$. Third, $\hat h_r$ conditions the backbone through per-channel scaling and shifting. The training loss adds a regional histogram term to the reconstruction term:

$$\mathcal{L} = \|\hat x - x\|_1 + \lambda \sum_r W_1\big(H_r(\hat x), \hat h_r\big) + \gamma \sum_r W_1(\hat h_r, h_r),$$

Here $\hat x$ is the denoised output and $x$ the clean image. $H_r(\cdot)$ is the soft gradient histogram of region $r$, and $W_1$ is the one-dimensional Wasserstein distance between histograms, computed from cumulative sums. $\lambda$ and $\gamma$ are weights. The first term keeps outputs faithful pixel by pixel. The second gives the model a target energy to hit instead of defaulting to zero, and the third trains the estimator.

Baselines

The naive reference points are the noisy input itself and an unmodified Restormer trained with the same recipe. Restormer is the strongest existing method in the file. Its paper reports being the first to surpass 40 dB PSNR on SIDD; the file gives no exact figure, so we will retrain it ourselves. Other reported numbers in the file are SADNet at 39.46 dB on SIDD validation and 39.59 dB on DND, and DUNetD at 39.29 dB PSNR on SIDD. We also run SADNet, because it is the strongest edge-adaptive local design, and RNAN, as the feature-pooling non-local design, on the same training data.

For fairness, every learned model uses the same Restormer backbone where applicable, the same real-noise training pairs, the same number of iterations and the same progressive patch-size curriculum. Each variant gets the same hyperparameter search budget, and the added module stays under 5% extra FLOPs.

Four controlled variants test the proposal's key ingredient. The first is a feature-pooling ablation: the identical retrieval module with the same $k$ and reach, which returns aggregated features instead of statistics. This isolates "what is moved" from "how far it reaches." The second is a global-histogram ablation: the gradient-histogram constraint with a single image-wide budget, as in the original paper but trained end to end. The third is a local-only ablation that estimates $\hat h_r$ from region $r$ alone, which tests whether distant context matters. The fourth is an oracle upper bound that conditions on ground-truth $h_r$. Beating Restormer means two things together: a statistically significant gain in regional texture fidelity on texture-rich regions, and PSNR within 0.05 dB or better. The method must also beat the feature-pooling ablation on texture fidelity at matched cost.

Why not just…?

Why not just add a perceptual or adversarial loss? These losses raise high-frequency energy but do not tie it to the true texture. The adversarial approach in Dual-Path Adversarial Denoising Network Based on UNet also reports instability under noise shifts. We include Restormer with an adversarial loss as a baseline and separate energy from correctness with a high-frequency correlation metric, described below. If that baseline matches ours on correlation, our approach is not needed.

Why not just blend some of the noisy input back into the output? This is a real threat to perceived sharpness, and it resembles the texture re-injection in Weighted median guided filtering method for single image rain removal. Blending restores energy, but what it restores is mostly noise. We tune the blend weight on validation data and report it. Our prediction is that it matches our energy deficit but loses on correlation with clean high frequencies and on PSNR. If it does not, the proposal's value collapses to a post-processing trick.

Why not just widen Restormer's receptive field with spatial attention? If reach alone fixed over-smoothing, RNAN-style attention would already solve it. The feature-pooling ablation is exactly this test at equal reach and cost. The VNLnet oracle result predicts that feature pooling suffers more from noisy matches than statistic pooling does.

Why not just use the gradient-histogram method's global prior inside a network? Its own stated limitation is heterogeneous scenes, where flat sky sits next to grass. The global-histogram ablation measures this directly, broken down by within-image texture heterogeneity.

Proposed Experiments

The evaluation sets are the SIDD validation and benchmark sets and DND in sRGB, to compare with SADNet's reported numbers. Low-noise stress tests add Gaussian noise at $\sigma \in \{5, 10, 30\}$ to SIDD ground-truth images, which probes the regime DVDnet identified. A held-out cross-camera split of SIDD tests whether the learned histogram mapping transfers across sensors.

Alongside PSNR and SSIM we report two texture metrics on texture-rich regions, selected by ground-truth gradient energy. The first is regional histogram distance, which measures energy fidelity. The second is the correlation between output and ground-truth high-pass bands, which separates real detail from hallucination.

Three results decide the question. The full model must beat the feature-pooling ablation on correlation in texture-rich regions. Its gain over Restormer must grow with the amount of distant repeated texture in an image and shrink to near zero in images with no repeats. And it must close a meaningful fraction of the gap to the oracle-histogram bound at $\sigma=10$. If the gain persists in images without repeats, the benefit comes from the histogram loss and not from long-range context, and we would report it as such.