In computer vision, "adverse weather" presents a significant challenge for real-world applications like autonomous driving and surveillance. Traditionally, researchers have developed specialized models for individual conditions—such as rain, haze, or snow. However, deploying multiple heavy, task-specific models is computationally prohibitive for edge devices.
A recent CVPR paper proposes a unified, lightweight framework that restores images from multiple weather conditions using a single set of parameters. By intelligently separating "weather-general" features from "weather-specific" ones, the authors achieve state-of-the-art results with nearly 85% fewer parameters than previous all-weather models.
The Challenge of All-Weather Restoration
Most image restoration models are specialists. While they excel at their specific task (e.g., deraining), they often fail when faced with different weather types. Previous attempts at "all-weather" models typically fell into two categories:
- Unified Models: Using one set of parameters for all weather types, which ignores the unique physical properties of different degradations.
- Multi-Encoder Models: Using separate encoders for each weather type, resulting in massive, slow models unsuitable for real-time use.
Proposed Method: Adaptive Weather-Specific Expansion
The authors introduce a two-stage training strategy built on a U-Net backbone to find a middle ground between efficiency and flexibility.
Stage 1: Weather-General Learning
The model is first trained on all weather types simultaneously to learn shared characteristics, such as low contrast and color distortion. A key innovation is the depth consistency loss. Since weather degradation is physically tied to scene depth (based on atmospheric scattering models), forcing the model to respect depth information helps it learn more robust, general features.
Stage 2: Adaptive Expansion
Instead of adding entire encoders for each weather type, the authors use a regularization-based optimization. They introduce small weather-specific parameter blocks (such as $1 \times 1$ convolutions) and a learnable "scoring" variable. During training, the model automatically identifies the "requisite positions"—the specific layers where weather-specific logic is most beneficial—and prunes the rest.
At inference time, the model only activates the general path plus the specific blocks for the detected weather type (e.g., haze) at those learned positions. This surgical approach ensures high performance without the bloat of traditional multi-task models.
Key Insights and Datasets
- Representation Analysis: Using Centered Kernel Alignment (CKA), the authors found that rain and snow models share similar internal representations, while haze requires distinct logic, particularly in the middle layers. This physical reality justifies the need for adaptive, rather than fully shared, parameters.
- New Benchmarks: The study introduces several high-quality real-world datasets: SPA+ (improved deraining), RealSnow (the first large-scale real-world benchmark for desnowing), and REVIDE (real-world dehazing).
Results and Performance
The proposed method consistently outperforms previous champions like TransWeather and All-in-One, often by as much as 2 dB in PSNR.
- Efficiency: The model uses only 5.97M parameters, compared to 44M for All-in-One.
- Speed: It processes a $256 \times 256$ image in just 0.03 seconds on a standard NVIDIA 1080 Ti GPU.
While the current framework assumes the weather type is known at inference, it sets a new standard for efficient, physics-informed image restoration. Future work may involve integrating automatic weather classification and extending the logic to ensure temporal consistency in video applications.
Summary updated