TL;DR
This paper introduces DestripeCycleGAN, a novel unsupervised deep learning framework for removing stripe noise from infrared images. By replacing the standard auxiliary generator in CycleGAN with a specialized stripe generation model and incorporating a Haar wavelet background guidance module, DestripeCycleGAN effectively preserves image details and outperforms existing methods.
What Problem Does This Paper Address?
Infrared imaging systems are crucial for applications like autonomous driving and maritime rescue. However, a common issue is the presence of irregular stripe noise, caused by variations in the infrared focal plane array's column response. This noise significantly degrades image quality and hinders downstream tasks like target detection and segmentation. While supervised methods exist, they require paired clean and noisy images, which are difficult to obtain in real-world scenarios. Existing unsupervised methods, like the vanilla CycleGAN, struggle to accurately model the directional nature of stripe noise, leading to residual noise in the output.
Methods
The core innovation is DestripeCycleGAN, which adapts the CycleGAN architecture for infrared destriping. Key components include:
- Stripe Generation Model (SGM): Instead of a generic auxiliary generator, a custom SGM is used to explicitly generate Gaussian-distributed vertical stripe noise with varying intensities. This ensures the simulated noise closely matches the characteristics of real stripe noise.
- Gradient Map for Cycle-Consistency: To better handle the directional nature of stripes, the cycle-consistency loss is applied to the vertical gradient map of the image, rather than the raw image itself.
- Haar Wavelet Background Guidance Module (HBGM): This module is introduced to minimize the divergence of background details between the noisy and clean domains. It uses Haar wavelet transforms to isolate and preserve background information.
- Multi-level Wavelet U-Net (MWUNet) Generator: A specialized U-Net architecture is employed as the denoising generator. It utilizes Haar wavelet transforms for sampling to reduce feature loss and incorporates Group Fusion Blocks (GFB) in skip connections to fuse multi-scale features and capture long-range dependencies.
Datasets
The paper uses publicly available datasets for training and evaluation:
- Training Data:
- Clean domain: 300 images from DLS-NUC, CVC-09.
- Stripe domain: Simulated stripe noise added to clean images, plus 150 real stripe images from MIRE, SNRWDNN, CVC-09, and their own long-wave infrared camera.
- Evaluation Datasets:
- DLS_50
- CVC09_50
- Set12
The training data is cropped into 64x64 patches.
Key Insights and Ideas
- Explicit Stripe Modeling is Crucial: The success of DestripeCycleGAN hinges on the SGM's ability to generate realistic stripe noise. Generic data-driven approaches are insufficient for capturing the specific directional properties of infrared stripes.
- Gradient Domain Cycle-Consistency: Applying cycle-consistency to the gradient domain is a clever way to enforce structural consistency without being overly sensitive to pixel-level noise variations.
- Wavelets for Background Preservation: The HBGM's use of Haar wavelets to isolate and preserve background information is key to preventing the loss of fine details during destriping.
- MWUNet for Enhanced Feature Representation: The multi-level wavelet U-Net, with its wavelet sampling and GFB, effectively balances feature extraction and reconstruction, leading to better preservation of image structure.
Key Results
- Quantitative Performance: DestripeCycleGAN consistently outperforms state-of-the-art methods across various datasets and noise types (Gaussian and Uniform). For instance, on DLS_50 with Uniform noise, it achieves 3.10 dB and 4.68 dB higher PSNR than SNRWDNN and TSWEU, respectively.
- Visual Performance: Qualitative results show that DestripeCycleGAN effectively removes stripe noise while preserving vertical edges and fine details, a common failure point for other methods. It also maintains consistent brightness in real-world scenarios.
- Efficiency: The proposed framework reduces model parameters and computational complexity compared to the vanilla CycleGAN.
Limitations and Future Work
- Anti-noise Experiment: While robust, the paper notes that as the intensity of simulated stripe noise increases beyond the training range (0.125 for SGM, 0.1 for training), a minor residual noise may appear. This suggests a potential limitation in handling extremely intense noise.
- Future Work: The authors plan to validate the effectiveness of their unsupervised destriping paradigm for downstream tasks in real-world scenarios.
Summary updated