Stochastic Frequency Masking to Improve Super-Resolution and Denoising Networks

Majed El Helou, Ruofan Zhou, Sabine Süsstrunk

TL;DR

The paper introduces Stochastic Frequency Masking (SFM), a simple yet powerful data augmentation technique that masks random frequency bands in the DCT domain during training. By forcing networks to reconstruct missing frequency components, SFM significantly reduces overfitting to specific degradation kernels and noise levels, leading to state-of-the-art performance in "blind" super-resolution and denoising.

What Problem Does This Paper Address?

Deep learning models for image restoration—like Super-Resolution (SR) and Denoising—are notoriously prone to overfitting. While they perform exceptionally well on synthetic data, they often fail when faced with real-world images where the "degradation kernel" (the specific way an image was blurred) or the exact noise level is unknown.

The authors identify that this happens because networks implicitly learn a conditional relationship: they try to predict high-frequency details based on the low-frequency information available in the input. However, they often "memorize" the specific frequency cut-off or noise profile of the training set. When a test image has a different blur or noise level, the network's assumptions fail, resulting in a "learning gap" in the frequency spectrum of the restored image.

Methods

The core contribution is Stochastic Frequency Masking (SFM). Unlike standard spatial augmentations (like flipping or rotating images), SFM operates in the frequency domain:

  1. Transform: An input training image is converted to the frequency domain using the Discrete Cosine Transform (DCT).
  2. Mask: A stochastic mask is applied to the DCT coefficients, zeroing out certain frequency bands. The authors propose two modes:
  3. Central Mode: Samples two radii to mask a random "annulus" (ring-like area) of frequencies. This is primarily used for SR to simulate various blur effects.
  4. Targeted Mode: Masks frequencies around a specific target frequency with a normal distribution decay. This is used for denoising to force the network to recover signal from high-noise regions.
  5. Invert: The masked coefficients are transformed back to the spatial domain via Inverse DCT.
  6. Train: This "corrupted" image is used as the network input, while the original clean image remains the ground-truth target.

SFM is applied to roughly 50% of the training data. Crucially, it adds zero computational cost at inference time, as it is strictly a training-side regularization technique.

Datasets

The authors evaluated their method across several standard and real-world benchmarks:

  • Super-Resolution:
    • DIV2K: Used for synthetic experiments with bicubic and Gaussian kernels.
  • RealSR & SR-RAW: Datasets containing pairs of real low-resolution and high-resolution images captured with different focal lengths.
  • Denoising:
    • BSD68: Used for Additive White Gaussian Noise (AWGN) experiments.
  • Fluorescence Microscopy Dataset: A real-world dataset featuring Poisson-Gaussian noise, where "ground truth" is obtained by averaging 50 repeated captures of the same static scene.

Key Insights and Ideas

  • Implicit Conditional Learning: The authors provide a formal frequency-domain analysis showing that SR and denoising are essentially the same task: predicting lost high-frequency components given less-affected low-frequency ones.
  • Visualizing the Overfitting: By plotting the Power Spectral Density (PSD), the authors show a literal "dip" in the frequencies of restored images when the test kernel doesn't match the training kernel. SFM effectively "fills" this dip by teaching the network to be less reliant on a specific frequency cut-off.
  • Denoising as Frequency Loss: In denoising, high-frequency signals are often completely buried by noise (low Signal-to-Noise Ratio). The authors show that high-level denoising is functionally similar to SR because the network must "hallucinate" the missing high-frequency signal from the cleaner low-frequency base.

Key Results

SFM consistently improved performance across multiple architectures (RCAN, RRDB, ESRGAN, IKC):

  • Blind SR: On unseen Gaussian kernels, SFM improved the PSNR of the IKC (Iterative Kernel Correction) method by up to 0.27 dB.
  • Real-World SR: On the RealSR and SR-RAW datasets, SFM pushed state-of-the-art results higher by up to 0.23 dB.
  • Extreme Denoising: The benefits were most pronounced at high noise levels (e.g., $\sigma=100$), where SFM helped networks maintain structural integrity that baseline models lost.
  • Real Denoising: When combined with Noise2Noise (N2N) on microscopy data, SFM achieved the highest quantitative scores across all tested noise levels.

Limitations and Future Work

The authors noted that SFM is less compatible with Noise2Self (N2S). This is because N2S relies on a spatial "J-invariant" masking (zeroing out random pixels), which creates its own artifacts in the frequency domain that interfere with the SFM masking.

Future research could explore adaptive SFM, where the frequency bands are not chosen purely at random but are instead targeted based on the specific weaknesses of the model during the training process. Additionally, extending this to other inverse problems like deblurring or JPEG artifact removal remains an open area.

Summary updated

Abstract

Super-resolution and denoising are ill-posed yet fundamental image restoration tasks. In blind settings, the degradation kernel or the noise level are unknown. This makes restoration even more challenging, notably for learning-based methods, as they tend to overfit to the degradation seen during training. We present an analysis, in the frequency domain, of degradation-kernel overfitting in super-resolution and introduce a conditional learning perspective that extends to both super-resolution and denoising. Building on our formulation, we propose a stochastic frequency masking of images used in training to regularize the networks and address the overfitting problem. Our technique improves state-of-the-art methods on blind super-resolution with different synthetic kernels, real super-resolution, blind Gaussian denoising, and real-image denoising.
BibTeX
@inproceedings{helou2020stochastic,
  author = {Majed El Helou and Ruofan Zhou and Sabine Süsstrunk},
  title = {{Stochastic Frequency Masking to Improve Super-Resolution and Denoising Networks}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=VQVVqU_vB4Z},
}

Loading related content...

Figure