Abstract
All-Weather Image Restoration (AWIR) aims to remove environmental artifacts such as rain, snow, and haze. Traditional models often struggle with generalization across different weather types and the disentanglement of degradation from the underlying scene. We present AWRaCLe, a novel framework that leverages visual in-context learning. By utilizing a reference "context pair"—a degraded image and its clean counterpart—AWRaCLe extracts precise degradation "fingerprints" to guide the restoration of query images. Our approach achieves state-of-the-art performance across multiple benchmarks and demonstrates unique capabilities like selective restoration.
Introduction
Environmental conditions like rain streaks, snowflakes, and thick haze significantly degrade image quality, impacting downstream computer vision tasks. While deep learning has advanced All-Weather Image Restoration (AWIR), two primary challenges remain: generalization and disentanglement. "All-in-one" models often lack the specific guidance required for the unique textures of various weather types. Furthermore, isolating weather artifacts from the background scene in a single image is mathematically ill-posed.
Previous attempts to provide guidance used text prompts (e.g., "remove rain"), but text lacks the spatial resolution to describe complex weather patterns. AWRaCLe addresses this by using visual in-context learning, where a context pair (degraded and clean versions of the same scene) provides high-fidelity degradation information.
Proposed Method: AWRaCLe
The AWRaCLe architecture is designed to extract and inject degradation-specific information (DSI) into a restoration backbone.
Architecture Overview
The system consists of three primary components:
- Restoration Backbone: A modified Restormer architecture serves as the high-performance transformer-based engine for image processing.
- Degradation Context Extraction (DCE) Blocks: These blocks utilize a frozen CLIP image encoder to obtain high-level features from the context pair. Multi-Head Self-Attention (MHSA) is then applied to compare the degraded and clean context images. Since the underlying scene is identical, the attention mechanism effectively isolates the weather artifacts.
- Context Fusion (CF) Blocks: These blocks employ Multi-Head Cross-Attention (MHCA) to integrate the extracted weather "fingerprint" into the restoration network. By injecting this information at multiple levels of the decoder, the model maintains precise guidance throughout the restoration process.
Experiments and Results
AWRaCLe was evaluated on standard benchmarks including Snow100k, SRD (rain), and RESIDE (haze). It achieved state-of-the-art (SOTA) results, notably reaching a PSNR of 31.7 dB on the SOTS haze dataset and 35.7 dB on Rain100L.
Key Insights
- Visual vs. Textual Context: Visual examples provide significantly richer DSI than text prompts, especially for describing the transparency and spatial density of haze or rain.
- Selective Restoration: A unique property of AWRaCLe is its ability to follow specific context instructions. If a query image contains both haze and snow, providing a "haze-only" context pair results in the removal of haze while leaving the snow intact.
- Robustness and Generalization: The model demonstrates strong performance on out-of-distribution (OoD) datasets (e.g., Foggy Cityscapes), indicating that the in-context learning approach generalizes better than standard supervised learning.
Conclusion and Future Work
AWRaCLe introduces a powerful paradigm for image restoration by treating weather removal as an in-context learning task. While the model requires a known degradation type to select the appropriate context pair, future work could involve automating this selection. Additionally, exploring multi-context pairs for compound weather events remains an open area for research.
Summary updated