CutFreq: Cut-and-Swap Frequency Components for Low-Level Vision Augmentation

Hongyang Chen, Kaisheng MaAAAI 2024 · OpenReview

Abstract

Low-level vision tasks, such as denoising and deraining, often suffer from limited dataset diversity and quality. While data augmentation is a standard solution in high-level vision, traditional spatial methods like CutMix often introduce destructive artifacts in low-level tasks. We propose CutFreq, a novel augmentation method that operates in the frequency domain using Discrete Wavelet Transforms (DWT). By "cutting and swapping" specific frequency components between input and ground-truth images, CutFreq helps models bridge the frequency gap and learn fine-grained textures. Our experiments across five low-level tasks demonstrate superior performance, particularly in low-data regimes.

Introduction

Data augmentation is essential for training robust deep learning models. However, in low-level vision, common techniques like Mixup or CutMix can be counterproductive. These methods often create unrealistic boundaries or color blocks that disrupt the local textural consistency required for precise pixel-to-pixel restoration.

A key observation in this work is the persistent "frequency domain gap" between images reconstructed by state-of-the-art models and real ground-truth images. As shown in Figure 1, even when visual results appear satisfactory, their frequency signatures often reveal a failure to capture specific high-frequency details or low-frequency global balances.

Visual comparison of generated images and ground truth alongside their frequency spectra across various tasks.

Methodology

The proposed CutFreq method shifts augmentation from the spatial domain to the frequency domain using the Discrete Wavelet Transform (DWT). Unlike the Fourier Transform, DWT preserves spatial structure by decomposing an image into four sub-bands: LL (low-frequency approximation), and LH, HL, HH (horizontal, vertical, and diagonal high-frequency details).

The Cut-and-Swap Strategy

The augmentation process involves two distinct stages designed to guide model convergence:

  1. Stage I (Fast Convergence): The model identifies frequency bands that are particularly difficult to learn for a given task. These bands are swapped from the ground-truth image into the input image, providing "directional" guidance that helps the model focus on missing features early in training.
  2. Stage II (Performance Approximation): Bands are alternately exchanged to refine the model’s ability to synthesize realistic textures.

Following the swap, an Inverse DWT brings the augmented image back into the spatial domain for standard training.

The CutFreq pipeline: Input and ground truth images undergo DWT, followed by a two-stage cut-and-swap process before inverse transformation.

Frequency Band Normalized $L_1$ Distance (FBND)

To quantify the frequency gap, we introduce the FBND metric. This metric measures how well a model converges on specific frequency bands, allowing for task-specific optimization. For instance, denoising models typically struggle with high-frequency components, while low-light enhancement is more sensitive to low-frequency information.

Experiments and Results

We evaluated CutFreq across five tasks: Smart ISP (RAW to RGB), Image Denoising (SIDD, DND), Image Deraining (Rain100H/L), Image Deblurring (GoPro, HIDE), and Low-light Enhancement (LoL).

Key Findings

  • Performance Gains: CutFreq consistently outperformed baselines and other augmentation methods like CutBlur and Mixup. For example, it achieved a +0.17 dB PSNR gain in Smart ISP and improved Restormer's denoising performance on SIDD by 0.06 dB.
  • Low-Data Efficiency: CutFreq is remarkably effective when training data is scarce. Models trained with only 5% or 10% of the dataset using CutFreq maintained a significant performance lead over baseline models trained on the same amount of data.
  • Qualitative Quality: Visual inspections confirm that CutFreq-trained models produce images with better global contrast and more vivid, realistic details.

Conclusion

CutFreq addresses the limitations of spatial data augmentation in low-level vision by operating in the frequency domain. By leveraging DWT to swap frequency components, it provides a principled way to bridge the gap between generated and real images. Future work may explore more explicit frequency-learning architectures or automated strategies for selecting optimal frequency bands for different tasks.

Summary updated

Abstract

Low-level vision plays a crucial role in a wide range of imaging quality and image recognition applications. However, the limited size, quality, and diversity of datasets often pose significant challenges for low-level tasks. Data augmentation is the most effective and practical way of sample expansion, but the commonly used augmentation methods in high-level tasks have limited improvement in the low-level due to the boundary effects or the non-realistic context information. In this paper, we propose the Cut-and-Swap Frequency Components (CutFreq) method for low-level vision, which aims to preserve high-level representations with directionality and improve image synthesis quality. Observing the significant frequency domain differences between reconstructed images and real ones, in CutFreq, we propose to transform the input and real images separately in the frequency domain, then define two stages for the model training process, and finally swap the specified frequency bands respectively and inversely transform to generate augmented samples. The experimental results show the superior performance of CutFreq on five low-level vision tasks. Moreover, we demonstrate the effectiveness of CutFreq in the low-data regime. Code is available at https://github.com/DreamerCCC/CutFreq.
BibTeX
@inproceedings{chen2024cutfreq,
  author = {Hongyang Chen and Kaisheng Ma},
  title = {{CutFreq: Cut-and-Swap Frequency Components for Low-Level Vision Augmentation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=t7BpgyxdUb},
}
Figure