Learning Weather-General and Weather-Specific Features for Image Restoration Under Multiple Adverse Weather Conditions

Yurui Zhu, Tianyu Wang, Xueyang Fu, Xuanyu Yang, Xin Guo, Jifeng Dai, Yu Qiao, Xiaowei Hu

In computer vision, "adverse weather" presents a significant challenge for real-world applications like autonomous driving and surveillance. Traditionally, researchers have developed specialized models for individual conditions—such as rain, haze, or snow. However, deploying multiple heavy, task-specific models is computationally prohibitive for edge devices.

A recent CVPR paper proposes a unified, lightweight framework that restores images from multiple weather conditions using a single set of parameters. By intelligently separating "weather-general" features from "weather-specific" ones, the authors achieve state-of-the-art results with nearly 85% fewer parameters than previous all-weather models.

The Challenge of All-Weather Restoration

Most image restoration models are specialists. While they excel at their specific task (e.g., deraining), they often fail when faced with different weather types. Previous attempts at "all-weather" models typically fell into two categories:

  1. Unified Models: Using one set of parameters for all weather types, which ignores the unique physical properties of different degradations.
  2. Multi-Encoder Models: Using separate encoders for each weather type, resulting in massive, slow models unsuitable for real-time use.
Comparison of different architectural strategies for multi-weather restoration. (a) Task-specific models, (b) Shared-top models, (c) Fully shared models, and (d) the proposed adaptive expansion model.

Proposed Method: Adaptive Weather-Specific Expansion

The authors introduce a two-stage training strategy built on a U-Net backbone to find a middle ground between efficiency and flexibility.

Stage 1: Weather-General Learning

The model is first trained on all weather types simultaneously to learn shared characteristics, such as low contrast and color distortion. A key innovation is the depth consistency loss. Since weather degradation is physically tied to scene depth (based on atmospheric scattering models), forcing the model to respect depth information helps it learn more robust, general features.

Stage 2: Adaptive Expansion

Instead of adding entire encoders for each weather type, the authors use a regularization-based optimization. They introduce small weather-specific parameter blocks (such as $1 \times 1$ convolutions) and a learnable "scoring" variable. During training, the model automatically identifies the "requisite positions"—the specific layers where weather-specific logic is most beneficial—and prunes the rest.

At inference time, the model only activates the general path plus the specific blocks for the detected weather type (e.g., haze) at those learned positions. This surgical approach ensures high performance without the bloat of traditional multi-task models.

Key Insights and Datasets

  • Representation Analysis: Using Centered Kernel Alignment (CKA), the authors found that rain and snow models share similar internal representations, while haze requires distinct logic, particularly in the middle layers. This physical reality justifies the need for adaptive, rather than fully shared, parameters.
  • New Benchmarks: The study introduces several high-quality real-world datasets: SPA+ (improved deraining), RealSnow (the first large-scale real-world benchmark for desnowing), and REVIDE (real-world dehazing).

Results and Performance

The proposed method consistently outperforms previous champions like TransWeather and All-in-One, often by as much as 2 dB in PSNR.

  • Efficiency: The model uses only 5.97M parameters, compared to 44M for All-in-One.
  • Speed: It processes a $256 \times 256$ image in just 0.03 seconds on a standard NVIDIA 1080 Ti GPU.

While the current framework assumes the weather type is known at inference, it sets a new standard for efficient, physics-informed image restoration. Future work may involve integrating automatic weather classification and extending the logic to ensure temporal consistency in video applications.

Summary updated

Abstract

Image restoration under multiple adverse weather conditions aims to remove weather-related artifacts by using a single set of network parameters. In this paper, we find that image degradations under different weather conditions contain general characteristics as well as their specific characteristics. Inspired by this observation, we design an efficient unified framework with a two-stage training strategy to explore the weather-general and weather-specific features. The first training stage aims to learn the weather-general features by taking the images under various weather conditions as inputs and outputting the coarsely restored results. The second training stage aims to learn to adaptively expand the specific parameters for each weather type in the deep model, where the requisite positions for expanding weather-specific parameters are automatically learned. Hence, we can obtain an efficient and unified model for image restoration under multiple adverse weather conditions. Moreover, we build the first real-world benchmark dataset with multiple weather conditions to better deal with realworld weather scenarios. Experimental results show that our method achieves superior performance on all the synthetic and real-world benchmarks. Codes and datasets are available at this repository.
BibTeX
@inproceedings{zhu2023learning,
  author = {Yurui Zhu and Tianyu Wang and Xueyang Fu and Xuanyu Yang and Xin Guo and Jifeng Dai and Yu Qiao and Xiaowei Hu},
  title = {{Learning Weather-General and Weather-Specific Features for Image Restoration Under Multiple Adverse Weather Conditions}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.02083},
}
Figure