DestripeCycleGAN: Stripe Simulation CycleGAN for Unsupervised Infrared Image Destriping

Shiqi Yang, Hanlin Qin, Shuai Yuan, Xiang Yan, Hossein RahmaniIEEE Trans. Instrumentation & Measurement 2024 · DOI · OpenReview

TL;DR

This paper introduces DestripeCycleGAN, a novel unsupervised deep learning framework for removing stripe noise from infrared images. By replacing the standard auxiliary generator in CycleGAN with a specialized stripe generation model and incorporating a Haar wavelet background guidance module, DestripeCycleGAN effectively preserves image details and outperforms existing methods.

What Problem Does This Paper Address?

Infrared imaging systems are crucial for applications like autonomous driving and maritime rescue. However, a common issue is the presence of irregular stripe noise, caused by variations in the infrared focal plane array's column response. This noise significantly degrades image quality and hinders downstream tasks like target detection and segmentation. While supervised methods exist, they require paired clean and noisy images, which are difficult to obtain in real-world scenarios. Existing unsupervised methods, like the vanilla CycleGAN, struggle to accurately model the directional nature of stripe noise, leading to residual noise in the output.

Methods

The core innovation is DestripeCycleGAN, which adapts the CycleGAN architecture for infrared destriping. Key components include:

  • Stripe Generation Model (SGM): Instead of a generic auxiliary generator, a custom SGM is used to explicitly generate Gaussian-distributed vertical stripe noise with varying intensities. This ensures the simulated noise closely matches the characteristics of real stripe noise.
  • Gradient Map for Cycle-Consistency: To better handle the directional nature of stripes, the cycle-consistency loss is applied to the vertical gradient map of the image, rather than the raw image itself.
  • Haar Wavelet Background Guidance Module (HBGM): This module is introduced to minimize the divergence of background details between the noisy and clean domains. It uses Haar wavelet transforms to isolate and preserve background information.
  • Multi-level Wavelet U-Net (MWUNet) Generator: A specialized U-Net architecture is employed as the denoising generator. It utilizes Haar wavelet transforms for sampling to reduce feature loss and incorporates Group Fusion Blocks (GFB) in skip connections to fuse multi-scale features and capture long-range dependencies.

Datasets

The paper uses publicly available datasets for training and evaluation:

  • Training Data:
    • Clean domain: 300 images from DLS-NUC, CVC-09.
  • Stripe domain: Simulated stripe noise added to clean images, plus 150 real stripe images from MIRE, SNRWDNN, CVC-09, and their own long-wave infrared camera.
  • Evaluation Datasets:
    • DLS_50
    • CVC09_50
    • Set12

The training data is cropped into 64x64 patches.

Key Insights and Ideas

  • Explicit Stripe Modeling is Crucial: The success of DestripeCycleGAN hinges on the SGM's ability to generate realistic stripe noise. Generic data-driven approaches are insufficient for capturing the specific directional properties of infrared stripes.
  • Gradient Domain Cycle-Consistency: Applying cycle-consistency to the gradient domain is a clever way to enforce structural consistency without being overly sensitive to pixel-level noise variations.
  • Wavelets for Background Preservation: The HBGM's use of Haar wavelets to isolate and preserve background information is key to preventing the loss of fine details during destriping.
  • MWUNet for Enhanced Feature Representation: The multi-level wavelet U-Net, with its wavelet sampling and GFB, effectively balances feature extraction and reconstruction, leading to better preservation of image structure.

Key Results

  • Quantitative Performance: DestripeCycleGAN consistently outperforms state-of-the-art methods across various datasets and noise types (Gaussian and Uniform). For instance, on DLS_50 with Uniform noise, it achieves 3.10 dB and 4.68 dB higher PSNR than SNRWDNN and TSWEU, respectively.
  • Visual Performance: Qualitative results show that DestripeCycleGAN effectively removes stripe noise while preserving vertical edges and fine details, a common failure point for other methods. It also maintains consistent brightness in real-world scenarios.
  • Efficiency: The proposed framework reduces model parameters and computational complexity compared to the vanilla CycleGAN.

Limitations and Future Work

  • Anti-noise Experiment: While robust, the paper notes that as the intensity of simulated stripe noise increases beyond the training range (0.125 for SGM, 0.1 for training), a minor residual noise may appear. This suggests a potential limitation in handling extremely intense noise.
  • Future Work: The authors plan to validate the effectiveness of their unsupervised destriping paradigm for downstream tasks in real-world scenarios.

Summary updated

Abstract

CycleGAN has been proven to be an advanced approach for unsupervised image restoration. This framework consists of two generators: a denoising one for inference and an auxiliary one for modeling noise to fulfill cycle-consistency constraints. However, when applied to the infrared destriping task, it becomes challenging for the vanilla auxiliary generator to consistently produce vertical noise under unsupervised constraints. This poses a threat to the effectiveness of the cycle-consistency loss, leading to stripe noise residual in the denoised image. To address the above issue, we present a novel framework for single-frame infrared image destriping, named DestripeCycleGAN. In this model, the conventional auxiliary generator is replaced with a priori stripe generation model (SGM) to introduce vertical stripe noise in the clean data, and the gradient map is employed to re-establish cycle-consistency. Meanwhile, a Haar wavelet background guidance module (HBGM) has been designed to minimize the divergence of background details between the different domains. To preserve vertical edges, a multi-level wavelet U-Net (MWUNet) is proposed as the denoising generator, which utilizes the Haar wavelet transform as the sampler to decline directional information loss. Moreover, it incorporates the group fusion block (GFB) into skip connections to fuse the multi-scale features and build the context of long-distance dependencies. Extensive experiments on real and synthetic data demonstrate that our DestripeCycleGAN surpasses the state-of-the-art methods in terms of visual quality and quantitative evaluation. Our code will be made public at https://github.com/0wuji/DestripeCycleGAN.
BibTeX
@article{yang2024destripecyclegan,
  author = {Shiqi Yang and Hanlin Qin and Shuai Yuan and Xiang Yan and Hossein Rahmani},
  title = {{DestripeCycleGAN: Stripe Simulation CycleGAN for Unsupervised Infrared Image Destriping}},
  journal = {IEEE Trans. Instrumentation \& Measurement},
  year = {2024},
  doi = {10.1109/tim.2024.3476560},
}
Figure