Precision-Weighted Global Attention for Real Camera Noise

Modern image denoisers get their quality from looking at the whole image at once. Restormer and NAFNet pool statistics across every pixel to decide how to clean each one. Real camera noise is not the well-behaved bell-curve noise these networks implicitly assume. It contains hot pixels, clipped highlights, shot-noise spikes in dark regions, and correlated blotches left by the camera's processing pipeline. When a network averages over all pixels, a few extreme pixels can dominate that average. Each pixel's influence on the global summary is then set by how loud it is, not by how trustworthy it is.

This note proposes a fix. A small branch of the network predicts, for each pixel, a heavy-tailed noise scale. That is a per-pixel estimate of how far the observation is likely to sit from the truth, under a distribution that allows occasional huge deviations. The network then uses these estimates as weights inside its global pooling and attention operations, so that unreliable pixels count less when the network forms image-wide statistics. The approach joins two research threads. One models noise beyond Gaussian assumptions. The other widens receptive fields, meaning how much of the image each output pixel can see. The noise model stops being only a loss function or a source of training data and starts controlling how global context is gathered.

Core Claim

The claim is narrow and testable. Wide-receptive-field restoration networks are fragile to heavy-tailed sensor corruption because of how they aggregate, and not only because of how they are trained. Down-weighting each pixel by a learned heavy-tailed precision inside the aggregation step improves robustness more than three cheaper changes:

  • a robust output loss;
  • training on more realistic noise;
  • feeding the same noise map to the network as an extra input.

Two aggregation steps are targeted.

Restormer's attention. Restormer's transposed attention (MDTA) forms a channel-by-channel affinity matrix by summing over all $N$ pixel positions. Write $Q, K \in \mathbb{R}^{C \times N}$ for the query and key features, with $C$ channels. The standard and proposed forms are

$$A = \mathrm{softmax}\!\left(\frac{K Q^\top}{\tau}\right) \quad\longrightarrow\quad A_w = \mathrm{softmax}\!\left(\frac{K\, W\, Q^\top}{\tau \,\mathrm{tr}(W)/N}\right),$$

where $\tau$ is Restormer's learned temperature. $W = \mathrm{diag}(w_1,\dots,w_N)$ holds per-pixel weights. The trace normalization keeps the overall scale comparable to the unweighted case.

NAFNet's channel attention. NAFNet's simplified channel attention takes a plain global average over pixels. It becomes the weighted mean $\sum_i w_i x_i / \sum_i w_i$, at no extra asymptotic cost. Both changes keep linear cost in image size.

The weights come from a heteroscedastic Student-t model of the residual between noisy and clean images. A Student-t distribution is a bell curve with fat tails, controlled by a degrees-of-freedom parameter $\nu$. A small variance branch predicts a scale $s_i$ and a shared or per-image $\nu$. The network is trained with the Student-t negative log-likelihood of the clean target given the noisy input. The weights are the standard iteratively-reweighted-least-squares weights for that likelihood,

$$w_i = \frac{\nu + 1}{\nu + r_i^2 / s_i^2},$$

where $r_i$ is the current block's residual estimate at pixel $i$. This form comes from the fixed point of Student-t M-estimation. As a result, a pixel whose deviation greatly exceeds its predicted scale is down-weighted roughly in proportion to $1/r_i^2$.

Relation to Prior Work

Methods this builds on. Robust Filter Attention recasts self-attention as precision-weighted state estimation with heavy-tailed likelihoods. That work is the conceptual source for this proposal. It is confined to 1D language sequences with stationary, isotropic noise. We take the idea to 2D images, where precision must vary pixel by pixel with the signal, as camera noise does. Variational Denoising Network (VDN) supplies the second ingredient. VDN jointly estimates a clean image and a per-pixel variance map, shows that plain MSE is a degenerate case of its objective, and reaches 39.23 dB on the Smartphone Image Denoising Dataset (SIDD). VDN's likelihood is Gaussian, its two networks are purely local CNNs, and its variance map is used only in the loss. We change all three.

Work that motivates the problem. Modeling sRGB Camera Noise with Normalizing Flows shows that processed camera noise is non-Gaussian and spatially correlated: a Gaussian model has KL divergence 0.079 against their flow's 0.044. Its denoiser, however, is a local DnCNN trained on synthetic samples, and a roughly 1.7 dB gap to real-pair training remains. Restormer and NAFNet are the global-context backbones we modify. Neither has any notion of per-pixel reliability inside its pooling.

Work that constrains the design. Pureformer found that channel attention over-smooths fine texture and needed extra convolutional filter banks to recover it. Down-weighting textured high-residual regions could make this worse. We therefore check Urban100 explicitly, where Pureformer reports 29.78 dB at σ=50 against Restormer's 29.31 dB. Classic non-local filtering stays competitive in RAW space (Benchmarking Denoising Algorithms with Real Photographs reports 46.63 dB mean PSNR for BM3D on the Darmstadt Noise Dataset, DND). This is a reminder that global aggregation over similar patches already works when the aggregation is robust.

Research Program

Step one is a diagnosis, done before building anything. We take pretrained Restormer and NAFNet and inject controlled heavy-tailed corruptions into SIDD sRGB validation images: hot pixels, salt-and-pepper impulses at rates from 0.01% to 1%, and highlight clipping. We then measure how far the MDTA affinity matrices and NAFNet's pooled channel vectors move, relative to how far the corruption moves the input. If a 0.1% impulse rate shifts the attention matrix as much as a large Gaussian perturbation does, the contamination mechanism is real. If the attention is already insensitive, the proposal's premise fails and we stop, reporting a negative result.

Step two builds the model. A lightweight variance branch predicts $s_i$ and $\nu$. We give it a global receptive field by reusing the backbone's own encoder features rather than a separate local CNN. This is VDN's design with the locality limit removed. Its outputs supply weights to every MDTA block (Restormer variant) or every pooling step in the simplified channel attention (NAFNet variant). Training uses the Student-t likelihood plus a small Charbonnier term for stability. A stop-gradient on $r_i$ within each block avoids a degenerate solution in which the network inflates $s_i$ everywhere.

Step three studies what the model learned. We check calibration: whether residuals normalized by $s_i$ actually follow the predicted Student-t, measured by quantile coverage. We also check whether low weights land on clipped regions, dark shot-noise regions, and injected impulses, rather than on edges and texture.

Baselines

The naive reference. This is the unmodified backbone trained with its standard pixel loss on SIDD. For NAFNet, Chen et al. report 40.30 dB PSNR on SIDD in Simple Baselines for Image Restoration. Restormer is reported as the first model above 40 dB on SIDD. Every variant is retrained from scratch at the same model width, iteration count, crop size, and augmentation as our reproduced baseline. We compare against our reproduction, not only against the published figure.

The strongest existing method. On clean SIDD this is NAFNet at 40.30 dB. Since the file has no reported result for any method under injected heavy-tailed corruption on SIDD (no reported result found on Lacuna for that setting), robustness numbers for it come from our own runs. Beating NAFNet means two things. First, staying within 0.05 dB of it on standard SIDD and DND. Second, improving PSNR by a consistent margin across three seeds on the corrupted-SIDD stress set, with the gap growing as the corruption rate grows.

The input-conditioning ablation. This is the most important baseline. The same variance branch, trained the same way, has its map $s_i$ concatenated to the backbone's input or features, with the attention left unweighted. Comparing it to our model isolates the key ingredient: using the noise estimate as aggregation weights rather than as information.

Two further ablations. The first replaces the Student-t likelihood with VDN's Gaussian likelihood, keeping the weighting. This tests whether heavy tails matter. The second uses a VDN-style local DnCNN variance branch instead of the shared global encoder. This tests whether global context helps variance estimation. VDN itself, at its reported 39.23 dB on SIDD, is included as a historical reference.

Why not just…?

Why not just train Restormer or NAFNet with an ℓ1 or Charbonnier loss? A robust output loss bounds how much a corrupted pixel's error pulls on the gradient. It does nothing about how that pixel's activations pull on the image-wide statistics inside the network at test time. Every baseline here already uses ℓ1 or Charbonnier. If our method's gains on the stress set disappear relative to these ℓ1-trained baselines, the alternative wins, and we will say so.

Why not just feed the noise map to the network as an extra input? This is the real threat. A network given a good map of where pixels are unreliable could, in principle, learn to ignore them internally. The input-conditioning ablation answers it directly with an identical variance branch and training budget. The proposal is justified only if explicit weighting beats it by a clear margin under corruption. If the two tie, the honest conclusion is that a noise map helps but placing it inside the aggregation step does not. The paper would then become a study of noise-map conditioning for global backbones.

Why not just train on realistic synthetic noise from a normalizing flow? Better training data and robust aggregation are complementary, not competing. Kousha et al. still leave about 1.7 dB on the table versus real pairs, and their flow needs ISO and camera metadata at synthesis time. We run both backbones with and without flow-style synthetic augmentation. This shows whether precision weighting still adds anything once the training noise is realistic.

Why not just use BM3D-style non-local filtering? It is strong in RAW space but slow, and the processed sRGB images where heavy-tailed, correlated noise is worst are where the learned backbones exceed 40 dB. It remains a reference on DND RAW.

Proposed Experiments

The main benchmarks are SIDD sRGB and DND for real noise. On SIDD, both NAFNet and Restormer variants are trained at matched compute. The corrupted-SIDD stress set uses impulse rates of 0.01%, 0.1%, and 1%, hot-pixel maps, and exposure-boosted clipping. It is held out by scene. Urban100 at σ=50 serves as a texture-preservation check against Pureformer's reported 29.78 dB. Metrics are PSNR and SSIM, plus quantile-coverage calibration of the predicted scales. We also measure the attention-shift sensitivity from step one, before and after the change.

The decisive checks, in order of importance:

  1. Does the diagnosis show contamination?
  2. Does weighting beat input conditioning under corruption?
  3. Does Student-t beat Gaussian weighting?
  4. Does Urban100 PSNR hold up, showing that the weights do not suppress legitimate texture?

A clean result would be: no loss on standard SIDD, a stress-set gain over NAFNet that grows with corruption rate, a clear margin over input conditioning, and calibrated scales that concentrate low weight on clipped and impulse pixels. Any other outcome tells us which part of the story is wrong.