Always Clear Days: Degradation Type and Severity Aware All-In-One Adverse Weather Removal

Yuwei Chen, Soo‐Chang PeiarXiv 2023 · DOI · arXiv

This paper, titled "Always Clear Days: Degradation Type and Severity Aware All-In-One Adverse Weather Removal," introduces a novel deep learning framework designed to clean up images obscured by various weather conditions like rain, snow, and haze using a single, unified model.

TL;DR

The authors propose UtilityIR, an "all-in-one" restoration model that explicitly accounts for both the type and severity of weather degradation. By using a novel ranking loss and adaptive normalization, it outperforms state-of-the-art models with fewer parameters and allows users to manually "tune" the level of restoration.

What Problem Does This Paper Address?

Most computer vision algorithms (like those in self-driving cars) are trained on clear images and fail miserably in adverse weather. While researchers have built specific models to remove rain or snow individually, "all-in-one" models—which handle multiple weather types at once—are the holy grail for practical deployment.

The authors identify a major flaw in existing all-in-one models: they focus on weather types (e.g., "is this rain or snow?") but ignore weather severity (e.g., "is this a light drizzle or a torrential downpour?"). Ignoring severity leads to "intra-domain gaps," where a model might under-restore a heavy storm or over-process a light mist, creating artifacts. Furthermore, weather often varies spatially—raindrops are local spots, while fog is a global haze—requiring a model that can handle both local and global distortions simultaneously.

Methods

The core of the proposed UtilityIR (Utility Image Restoration) is the Degradation Information Encoder (DIE), which extracts two specific signals from a degraded image:

  1. Weather Type Extraction: Uses Contrastive Loss (CL) to ensure that features from the same weather type (e.g., all snow images) cluster together in the model's "mind," while different types are pushed far apart.
  2. Weather Severity Estimation: Introduces a new Marginal Quality Ranking Loss (MQRL). Unlike standard ranking losses that only care if one image is "worse" than another, MQRL forces the model to learn the interval or degree of difference in quality. This allows the model to accurately gauge exactly how much restoration is needed.

To apply this information, the authors use two clever architectural tricks:

  • LG-AdaIN (Local-Global Adaptive Instance Normalization): This layer uses the weather type to adjust pixels locally (good for raindrops) and uses the severity to adjust the image globally (good for haze).
  • DGCA (Degradation-guided Cross Attention): This uses the extracted weather information as a "query" to help the transformer-style layers focus on the most degraded parts of the image.

Datasets

The model was evaluated using two primary experimental setups:

  • Setting 1 (All-Weather): A combination of the Raindrop dataset, Outdoor-Rain (rain + fog), and Snow100K-L.
  • Setting 2 (Standard Benchmarks): Includes Rain1400 (rain streaks), RESIDE OTS (haze), and Snow100K-L.
  • Real-World Data: The authors also tested the model on various unlabelled real-world images to prove its generalization capability.

Key Insights and Ideas

  • Severity is a Spectrum: The "aha moment" here is treating weather not as a set of discrete categories, but as a continuous spectrum of intensity. By learning the variance in degradation, the model becomes much more flexible.
  • User-Controllable Restoration: Because the model represents severity as a vector in a latent space, the authors found they could "nudge" this vector. This allows a user to manually modulate the restoration level—effectively a "de-weathering" slider.
  • Zero-Shot Combination: Surprisingly, even though the model was trained on single weather types (just rain or just snow), it can handle unseen combinations (like rain + snow + fog) by iteratively processing the image.

Key Results

  • Quantitative Superiority: UtilityIR outperformed previous leaders (like Transweather and Restormer) by significant margins. For example, it achieved a +2.65 dB PSNR improvement on the challenging rain+fog dataset.
  • Efficiency: Despite the better performance, UtilityIR uses only 26.3 million parameters, making it significantly lighter than competitors like WeatherDiff (83M) or All-in-one (44M).
  • Qualitative Quality: Visually, the model avoids common pitfalls like "black stain" artifacts left behind after raindrop removal or the "over-smoothing" of textures common in haze removal.

Limitations and Future Work

The authors are transparent about a few hurdles:

  • Parameter Sensitivity: The new MQRL loss requires careful tuning of its "margin" parameter to work effectively.
  • Computational Cost: While the parameter count is low, the use of cross-attention mechanisms still demands high computational power during inference.
  • Severe Occlusion: In cases where weather completely blocks high-frequency details (like a thick snowflake covering a small sign), the model—like all current SOTA methods—struggles to "hallucinate" the missing information perfectly, sometimes resulting in slight blurring.

Future work aims to make the model more robust to completely "unseen" weather types encountered in the wild.

Summary updated

Abstract

All-in-one adverse weather removal is an emerging topic on image restoration, which aims to restore multiple weather degradations in an unified model, and the challenge are twofold. First, discover and handle the property of multi-domain in target distribution formed by multiple weather conditions. Second, design efficient and effective operations for different degradations. To resolve this problem, most prior works focus on the multi-domain caused by different weather types. Inspired by inter\&intra-domain adaptation literature, we observe that not only weather type but also weather severity introduce multi-domain within each weather type domain, which is ignored by previous methods, and further limit their performance. To this end, we propose a degradation type and severity aware model, called UtilityIR, for blind all-in-one bad weather image restoration. To extract weather information from single image, we propose a novel Marginal Quality Ranking Loss (MQRL) and utilize Contrastive Loss (CL) to guide weather severity and type extraction, and leverage a bag of novel techniques such as Multi-Head Cross Attention (MHCA) and Local-Global Adaptive Instance Normalization (LG-AdaIN) to efficiently restore spatial varying weather degradation. The proposed method can outperform the state-of-the-art methods subjectively and objectively on different weather removal tasks with a large margin, and enjoy less model parameters. Proposed method even can restore unseen combined multiple degradation images, and modulate restoration level. Implementation code and pre-trained weights will be available at \url{https://github.com/fordevoted/UtilityIR}
BibTeX
@misc{chen2023always,
  author = {Yuwei Chen and Soo‐Chang Pei},
  title = {{Always Clear Days: Degradation Type and Severity Aware All-In-One Adverse Weather Removal}},
  year = {2023},
  eprint = {2310.18293},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.18293},
}
Figure