A camera denoiser has to decide, pixel by pixel, whether a rapid flicker in brightness is sensor noise to remove or fine texture to keep, such as fur, fabric or brickwork. Inside a small neighbourhood the two look the same, so compact networks built from small convolutions tend to fail in one of two ways. Either they erase texture as if it were noise, or they keep noise as if it were texture. Our starting observation is that the two separate cleanly once the model can see a few repetitions of the pattern. A real texture concentrates its energy in a few sharp peaks of the local frequency spectrum, while sensor noise spreads its energy over all frequencies. Seeing a few repetitions does not require a global view of the image. A tile a few texture periods wide is enough, and the frequency transform of a fixed-size tile is a fixed matrix multiply, the operation phone accelerators run best.
We propose to add a windowed spectral gate to an activation-free edge backbone. The gate is a small block that takes the discrete Fourier transform (DFT) of each tile as a fixed integer-friendly matrix product, scales each frequency bin by a learned gate conditioned on how "peaky" the tile's spectrum is, and transforms back. We train it with a loss that has one term for each of the two failure modes. It penalizes lost energy at the clean image's spectral peaks, which is over-smoothing, and it penalizes energy added away from those peaks, which is retained noise or hallucinated texture. The claim is specific and falsifiable. On real smartphone noise, most of the gap between compact convolutional denoisers and heavy transformer denoisers sits in high-periodicity regions. A gated window of roughly three or more texture periods, supervised by the two-trap loss, closes a substantial part of that gap at matched multiply-accumulate cost and under INT8 on a mobile processor.
Core Claim
The literature in the file treats the receptive-field problem as a need for global context. TCD-Net (Teacher-Guided Causal Interventions for Image Denoising) separates content from noise using a Vision Transformer backbone, an orthogonality loss between a content branch and a noise branch, and a distilled generative teacher. It reaches 40.48 dB on SIDD, but its efficiency is reported only on a desktop RTX 5090. The efficient global operators take other routes to long-range context. Criss-cross attention (CCNet) attends along rows and columns. Butterworth spectral gating (Butterworth as Attention) applies a whole-image Fourier gate with smooth, learnable cutoffs. Slice-and-scan state-space models (XYScanNet) sweep linear recurrences across the image. All three buy image-wide context, and none targets the texture-noise decision specifically. We argue that deciding between texture and noise needs a measured, finite context of a few periods, not global context.
The need for context beyond a small neighbourhood is supported by two sources. The non-local denoiser AST-NLS (Image denoising via adaptive soft-thresholding based on non-local samples) groups similar patches from across the image and thresholds coefficients in a learned basis, which is too costly for a phone. Noise Estimation from a Single Image shows that smooth priors mistake dense texture for noise. A windowed DFT with a learned gate recovers the part of that non-local benefit that comes from periodicity, at a fixed, predictable cost.
This design differs from Butterworth gating in three ways, and each has a reason. First, the gate works per tile rather than on the whole image, because texture is local and image-wide spectra mix unrelated regions. Second, the gate is a free per-bin multiplicative mask conditioned on tile statistics, not a smooth parametric roll-off. The Butterworth paper itself concedes that smooth roll-offs cannot isolate sharp peaks, which is exactly what periodic texture produces. Third, the transform is a fixed dense matrix rather than a fast Fourier transform, trading a log factor of arithmetic for regular memory access and INT8 support.
The loss is the second half of the claim. For a tile $w$, let $X_w$ and $\hat X_w$ be the DFT magnitudes of the clean and restored tiles. Let $P_w$ be the set of bins where the clean magnitude exceeds a robust floor, taken as a multiple $\tau$ of the tile's median magnitude. We train with
$$ \mathcal{L} = \|\hat x - x\|_1 + \lambda_p \sum_w \sum_{k \in P_w} \big[X_w(k) - \hat X_w(k)\big]_+ + \lambda_f \sum_w \sum_{k \notin P_w} \big[\hat X_w(k) - X_w(k)\big]_+ , $$
where $\hat x$ and $x$ are the restored and clean images and $[\cdot]_+$ keeps only positive parts. The second term charges only for peak energy the model removed, which is the over-smoothing trap. The third term charges only for energy the model added off the peaks, which is the trap of retaining noise or hallucinating texture. Neither term fires on flat tiles that were restored correctly. This addresses a concern the file raises about uniform gradient losses. In noisy flat regions, the gradient consistency loss of GradNet and the gradient-magnitude complexity signal of CADyQ are driven by the noise itself. Our peak set is computed from the clean target during training only and costs nothing at inference.
Research Program
The backbone is NAFNet (Simple Baselines for Image Restoration), which replaces nonlinear activations with elementwise multiplication of channel halves. Its channel attention is global in a limited sense: it pools the whole image into one value per channel, so it can reweight channels but cannot tell one location's spectrum from another's. We build compact NAFNet variants at two budgets, roughly 1.5M and 4M parameters. The lower budget matches GradNet's 1.6M. We insert the windowed spectral gate at the full-resolution stage, on a projected set of 8 channels, and at one mid-resolution stage.
Within a tile of side $T$, the gate proceeds in three steps. It computes separable row and column DFTs as fixed $T \times T$ matrix products. It computes a peakiness statistic, the ratio of top-bin energy to median-bin energy. It then predicts a per-bin mask from the log-magnitude spectrum and this statistic using a tiny shared linear layer, multiplies the mask in, and inverts the transform. To avoid square roots under INT8, magnitude is approximated as $|\mathrm{Re}| + |\mathrm{Im}|$. To avoid seam artefacts, tiles overlap by half and are blended with a fixed window.
The central analysis is a window-size study that tests the "few periods" hypothesis directly. We sweep $T$ over 8, 16, 32 and 64 and measure restoration quality separately for tiles grouped by their dominant texture period. The hypothesis predicts that gains appear once $T$ spans about three periods and saturate beyond that. It also predicts that textures whose period exceeds roughly $T/3$ receive no benefit. If instead gains grow smoothly with $T$ and ignore period, the mechanism is generic context rather than periodicity detection, and the core claim is wrong.
We also test what the two-trap loss does beyond generic frequency supervision. We compare it against an ordinary symmetric frequency-magnitude $L_1$ on the same tiles. We also run each of its two terms alone, to see whether each suppresses the failure mode it targets.
For deployment, we apply INT8 quantization-aware training to the full network, including the DFT matrices, which are quantized once. We measure latency and energy on a Snapdragon 888 and a Kirin 980, the phone processors used by MuLUT and MGA. This lets our numbers sit beside theirs.
Baselines
The naive reference point is compact NAFNet trained with plain $L_1$ at the same parameter and multiply-accumulate budget, with no gate and no spectral loss. We train it ourselves with the same data, schedule and tuning budget, because published NAFNet numbers use a much larger model. That large model's reported SIDD score of 40.30 dB (Simple Baselines for Image Restoration) is a ceiling for reference, not the fair comparison.
The strongest compact competitor on the specific question of preserving texture is a GradNet-style recipe. GradNet extracts Sobel gradients from a first rough denoised estimate, feeds them into a residual network, and adds a gradient consistency loss. It reports 38.34 dB on SIDD with 1.6M parameters. We run it in two ways: as published, and transplanted onto our compact NAFNet with its $L_1$-plus-gradient loss. The transplant separates GradNet's loss idea from its older backbone.
The strongest heavy method is TCD-Net at 40.48 dB on SIDD (Teacher-Guided Causal Interventions). We cite its reported number as an upper reference rather than re-running it, since its generative teacher is not available.
For long-range context at edge cost, we insert a whole-image Butterworth gate (Butterworth as Attention) and, separately, a two-pass criss-cross attention block (CCNet) at the same positions as our gate, with widths adjusted to equal multiply-accumulate counts.
The key ablations cross architecture with loss: gate with $L_1$, no gate with the two-trap loss, and gate with a symmetric frequency loss. Together these isolate which ingredient carries the gain. All models share the SIDD training split, the same number of iterations, and the same hyperparameter search budget of eight trials per method.
Beating the baselines has two parts. On SIDD overall, the gated compact model must exceed the compute-matched plain-$L_1$ NAFNet and the transplanted gradient recipe, with the margin concentrated in the high-periodicity stratum defined below. In that stratum, it must close at least half the gap between the compact baseline and the large NAFNet model. On the phone, it must stay within 15% of the compact baseline's INT8 latency.
Why not just…?
Why not just make compact NAFNet wider or deeper with the saved effort? This is the most serious threat. Extra capacity alone might learn periodicity implicitly, and NAFNet's own results show that simple designs scale well. Every comparison is therefore matched in multiply-accumulate count, and we add a baseline that spends the gate's budget on extra channels. If the wider model matches our gains in the high-periodicity stratum, the gate is not justified and we will report that. The window-size study adds a second test: capacity scaling has no reason to show a period-dependent threshold, while our mechanism predicts one.
Why not just add a gradient consistency loss, as GradNet does? A gradient loss is local and symmetric. It rewards sharp output wherever the target is sharp, but its Sobel input is computed from an estimate that is itself noisy or over-smoothed. GradNet acknowledges this by running a rough denoiser first. The transplanted-GradNet baseline measures this directly. Our off-peak energy metric shows whether the gradient loss adds spurious high-frequency energy in noisy flat regions, which the two-trap loss is designed to avoid.
Why not just widen the receptive field with dilated convolutions or a whole-image spectral gate? Dilated convolutions sample sparsely and can alias periodic textures. The Butterworth paper reports that its spectral gate ran 2.4x faster and used 2.3x less memory than the adaptive-convolution alternative, ARConv, in pansharpening. A whole-image gate, however, averages the spectra of unrelated regions and uses smooth roll-offs, both of which blur the sharp local peaks we rely on. The compute-matched Butterworth and criss-cross baselines test this directly. If the whole-image gate wins, our windowing argument fails.
Proposed Experiments
We train on the SIDD training split (A High-Quality Denoising Dataset for Smartphone Cameras) and report PSNR and SSIM on the SIDD benchmark and on DnD. For GradNet on DnD, no reported result was found on Lacuna, so all DnD numbers will be our own runs. To probe texture specifically, we add synthetic noise sampled from SIDD noise patches to Urban100 (Single image super-resolution from transformed self-exemplars) and CBSD68, since Urban100 is dense with repetitive architectural structure.
Our main diagnostic stratifies evaluation tiles into quintiles by the peakiness of their clean spectrum. Within each quintile we report PSNR, retained peak energy and added off-peak energy. The prediction is that gains lie mostly in the top quintile and are near zero on flat tiles.
A controlled synthetic test checks the period threshold. We render periodic textures with periods from 2 to 32 pixels, add real SIDD noise, and plot restoration error against period for each window size $T$. The mechanism predicts a knee near period $T/3$.
As a machine-readability check motivated by the three-way trade-off between distortion, perceived quality and classification accuracy (On The Classification-Distortion-Perception Tradeoff), we measure off-peak energy on restored images. Energy added away from the true peaks is exactly the kind of hallucinated texture that can mislead downstream recognition models.
Deployment runs the INT8 models on a Snapdragon 888 and a Kirin 980. We report latency and energy per 1080p frame alongside MuLUT's reported 240 ms and 278 μJ per HD frame on the Snapdragon 888. MuLUT's figure comes from a look-up-table method that cannot see a texture period at all, so the comparison shows what the extra context costs.
The decisive outcomes follow from these tests. If the gated compact model beats the compute-matched wider NAFNet in the top peakiness quintile and shows the predicted period threshold, the claim stands. If it does neither, texture-noise ambiguity in compact denoisers is a capacity problem rather than a context problem, and the proposal is refuted.
References
- Teacher-Guided Causal Interventions for Image Denoising: Orthogonal Content-Noise Disentanglement in Vision Transformers
- CCNet: Criss-Cross Attention for Semantic Segmentation
- Butterworth as Attention: Anisotropic Spectral Gating for Pansharpening
- XYScanNet: A State Space Model for Single Image Deblurring
- Image denoising via adaptive soft-thresholding based on non-local samples
- Noise Estimation from a Single Image
- GradNet Image Denoising
- CADyQ: Content-Aware Dynamic Quantization for Image Super-Resolution
- Simple Baselines for Image Restoration
- Toward DNN of LUTs: Learning Efficient Image Restoration With Multiple Look-Up Tables
- Restore Globally, Refine Locally: A Mask-Guided Scheme to Accelerate Super-Resolution Networks
- A High-Quality Denoising Dataset for Smartphone Cameras
- Single image super-resolution from transformed self-exemplars
- On The Classification-Distortion-Perception Tradeoff