TL;DR
This paper proposes using image gradients as a more robust representation for image restoration tasks, especially in the presence of illumination changes and occlusions. They introduce differentiable modules to convert between RGB and gradient domains and adapt the Video Swin-Transformer architecture to leverage this gradient information, achieving state-of-the-art results on various restoration tasks.
What Problem Does This Paper Address?
Deep learning models for computer vision typically rely on RGB images as input, assuming that the network can learn the optimal representation. However, real-world image sequences often suffer from distortions like varying illumination, shadows, and occlusions, which can significantly degrade the performance of standard RGB-based restoration models. While traditional computer vision methods have utilized image gradients for tasks like image stitching and decomposition, their full potential in end-to-end deep learning for image restoration, particularly with sequential data, has not been thoroughly explored. This paper aims to bridge this gap by demonstrating that image gradients offer a more stable and informative representation for handling these common distortions, leading to improved image restoration.
Methods
The core innovation lies in treating image gradients as a primary representation for restoration. The authors introduce two key differentiable modules:
- Tensor2Gradient: This module converts RGB images into their gradient representation using 2D convolutions with gradient filters. This process is designed to be differentiable, allowing gradients to flow back during training.
- Gradient2Tensor: This module reconstructs an RGB image from its gradient representation. It involves computing a Laplace image and then using deconvolution in the frequency domain. This module is also differentiable.
They adapt the Video Swin-Transformer architecture to process these gradient representations. The Video Swin-Transformer is known for its ability to capture spatio-temporal context in video sequences by using self-attention over shifting windows. The authors modify it to operate on sequences of gradient images.
A composite loss function is employed, combining an RGB loss (measuring error in the color domain) and a gradient loss (measuring error in the gradient domain). This hybrid loss encourages the model to focus on both accurate color reconstruction and preserving high-frequency image details, which are crucial for restoration.
They also propose four variations of their model:
- Swin-RGB: Standard RGB input, trained with RGB loss.
- Swin-RGB+: Standard RGB input, trained with composite loss.
- Swin-Mix: Concatenates RGB and gradient features.
- Swin-∇: Operates solely in the gradient domain.
Datasets
- SIDAR dataset: Used for training and evaluating the models. This dataset contains image sequences with various synthetic perturbations, including illumination changes, shadows, and occlusions, along with ground truth images.
- DIV2K dataset: Used for training when adapting the SwinIR architecture for broader image restoration tasks (denoising, deblurring, JPEG compression). Data augmentation was applied to simulate illumination changes.
- WALT dataset: Used for qualitative evaluation of zero-shot background estimation.
Key Insights and Ideas
- Gradient Robustness: Image gradients are shown to be significantly more robust to illumination changes than RGB values. This is because gradients primarily capture structural information, which is less affected by lighting variations.
- Outlier Detection: Occlusions and other outlier artifacts manifest as more distinct spikes in the gradient domain, making them easier for the model to identify and handle.
- Clustering Properties: When clustering image patches, gradient representations exhibit stronger intra-class similarity and higher inter-class variance compared to RGB representations. This suggests gradients provide a more discriminative feature space.
- Differentiable Gradient Modules: The development of differentiable modules for converting between RGB and gradient domains is crucial, enabling end-to-end training of gradient-based restoration models.
- Zero-Shot Generalization: Models trained on synthetic gradient data demonstrate impressive generalization to real-world video sequences, particularly for background estimation, highlighting the practical utility of the gradient domain.
Key Results
- Improved SSIM: Models that explicitly use gradients (Swin-∇, Swin-Mix) or incorporate gradient loss (Swin-RGB+) significantly improve the Structural Similarity Index Measure (SSIM) compared to RGB-only models, especially for sequences with varying illumination and occlusions.
- Swin-∇ Performance: The pure gradient-based model (Swin-∇) shows excellent generalization to real-world video sequences for background estimation, outperforming other methods and producing more consistent images without artifacts.
- Composite Loss Benefits: Training with a composite RGB and gradient loss generally leads to better performance across various metrics (RMSE, PSNR, SSIM) compared to using only RGB loss.
- Gradient Domain Drawback: The gradient domain inherently loses precise color range information, which can lead to less accurate color reconstruction in some cases, as noted for Swin-∇.
Limitations and Future Work
- Color Range Loss: The primary limitation of the gradient domain is the loss of precise color range, which can affect the accuracy of color reconstruction. The authors suggest that combining color and gradient features (Swin-Mix) can mitigate this.
- Blurring Distortion: Image blurring significantly distorts gradient information by attenuating high frequencies, leading to degraded performance when using gradients for deblurring tasks.
- Future Work: The authors propose exploring gradient representations across a wider range of vision tasks and investigating alternative image representations beyond gradients and color spaces. They also aim to make their models and training code publicly available.
Summary updated