Abstract
Weather degraded conditions such as rain, haze, snow, etc. may degrade the performance of most computer vision systems. Therefore, effective restoration of multi-weather degraded images is an essential prerequisite for successful functioning of such systems. The current multi-weather image restoration approaches utilize a model that is trained on a combined dataset consisting of individual images for rainy, snowy, and hazy weather degradations. These methods may face challenges when dealing with real-world situations where the images may have multiple, more intricate weather conditions. To address this issue, we propose a domain translation-based unified method for multi-weather image restoration. In this approach, the proposed network learns multiple weather degradations simultaneously, making it immune for real-world conditions. Specifically, we first propose an instance-level domain (weather) translation with multi-attentive feature learning approach to get different weather-degraded variants of the same scenario. Next, the original and translated images are used as input to the proposed novel multi-weather restoration network which utilizes a progressive multi-domain deformable alignment (PMDA) with cascaded multi-head attention (CMA). The proposed PMDA facilitates the restoration network to learn weather-invariant clues effectively. Further, PMDA and respective decoder features are merged via proposed CMA module for restoration. Extensive experimental results on synthetic and real-world hazy, rainy, and snowy image databases clearly demonstrate that our model outperforms the state-of-the-art multi-weather image restoration methods.
Introduction
Computer vision systems, particularly those used in autonomous driving and surveillance, face significant performance degradation when operating in adverse weather. Traditionally, restoration tasks were handled by specialized models—such as de-hazers or de-rainers—designed for a single type of degradation. However, real-world weather is often a complex mixture of conditions, such as rain accompanied by a thick fog, known as the "veiling effect."
Existing "all-in-one" models often struggle to generalize because they fail to effectively decouple weather-specific noise from the underlying scene content. Cascading multiple single-weather models often exacerbates errors. This paper introduces a unified framework that leverages domain translation to learn weather-invariant features, enabling robust restoration across diverse and overlapping weather conditions.
Methodology
The proposed method operates in two distinct stages: Domain Translation and Restoration.
Domain Translation (DT)
Given a degraded input image, the network first translates it into three specific weather domains: hazy ($I_{D1}$), rain with veil ($I_{D2}$), and snow with veil ($I_{D3}$). This is achieved using a U-Net style generator enhanced with a Multi-attentive Feature Extraction (MFE) module. By forcing the network to generate these variants, the encoder is trained to identify features common across all weather types—essentially uncovering the "clean" scene structure hidden beneath the degradation.
Restoration Generator ($G_r$)
The second stage takes the original image and the three translated versions as input to a novel restoration network. This stage relies on two key components:
- Progressive Multi-domain Deformable Alignment (PMDA): Recognizing that rain, snow, and haze have different spatial properties, this module uses deformable convolutions to align features from different domains progressively. It uses learned offsets to ensure scene structure is consistent across the translated variants.
- Cascaded Multi-head Attention (CMA): To merge aligned features with the decoder, the authors employ a transformer-inspired block. This captures long-range dependencies, which is critical for removing global degradations like the veiling effect.
Experiments and Results
The model was evaluated on a comprehensive suite of synthetic (SOTS, ORD, CSD) and real-world (RTTS, RID) benchmarks.
Quantitative and Qualitative Performance
The proposed method consistently outperformed state-of-the-art (SOTA) unified models like TransWeather and KD. For instance, on the SOTS haze dataset, the model achieved a PSNR of 36.26 dB, a significant improvement over the 34.64 dB achieved by the next best model. Visually, the framework is notably superior at removing the persistent "white-ish wash" or veiling effect that often remains after primary artifacts like rain streaks are removed.
Efficiency and Downstream Utility
Despite its multi-stage approach, the model is computationally efficient, utilizing approximately 11M parameters—nearly one-third the size of TransWeather (~31M parameters). Furthermore, the authors demonstrated that using this restoration as a pre-processing step significantly boosts the performance of downstream tasks like YOLOv3-based object detection and depth estimation in adverse weather.
Discussion and Conclusion
The core innovation of this work lies in "learning by creating." By utilizing domain translation as a precursor, the model implicitly learns what a clean scene looks like by understanding how different weather types obscure it. While the model shows strong generalization, future work could focus on reducing the computational overhead of multiple forward passes and extending the weather-invariant learning approach to video restoration for temporal consistency.
Code for this project is available at: https://github.com/pwp1208/Domain_Translation_Multi-weather_Restoration.
Summary updated