Introduction
Restoring images captured under adverse weather conditions—such as heavy rain, snow, or haze—is a challenging inverse problem. In many cases, the degradation is so severe that the original background information is mathematically unrecoverable from the pixels alone. While "all-in-one" restoration models have attempted to handle multiple weather types simultaneously, they often reach a bottleneck because they rely exclusively on the degraded input. If the input lacks critical texture and structure, the model is forced to "guess," leading to blurry or artifact-heavy results.
The AWRCP Framework
To overcome these limitations, the authors propose AWRCP (Adverse Weather Removal with Codebook Priors). This framework shifts the restoration task from simple filtering to a feature-matching and fusion problem by leveraging a pre-trained "dictionary" of high-quality visual features.
VQGAN Codebook Priors
The core of AWRCP is a codebook derived from a Vector-Quantized Generative Adversarial Network (VQGAN) pre-trained on a massive dataset of clean images. This codebook serves as a discrete collection of high-resolution "visual atoms"—textures and shapes that the model can use to reconstruct missing details.
Deformable Cross-Attention (DCA)
Because weather distortions can warp the geometry of an image, the features extracted from a degraded input rarely align perfectly with the clean entries in the codebook. AWRCP employs Deformable Cross-Attention (DCA), which uses a sparse sampling mechanism to flexibly search the codebook for the best-matching high-quality features, effectively bridging the gap between the messy input and the clean prior.
Hierarchical Texture Warping Head (HTWH)
To ensure that the restored images maintain fine-grained textures, the model utilizes a Hierarchical Texture Warping Head (HTWH). This module gradually fuses high-resolution codebook features into the output, ensuring that edges and textures (like building facades or fabric weaves) remain crisp and realistic.
Experimental Evaluation
The AWRCP model was tested on several standard benchmarks, including Snow100K, Outdoor-Rain, and RainDrop. These datasets cover a wide range of conditions, from localized raindrop distortions to global haze and heavy snow.
Quantitative and Qualitative Results
AWRCP achieved state-of-the-art (SOTA) performance across all benchmarks. For instance, on the Snow100K-S dataset, it reached a PSNR of 36.92 dB, significantly outperforming previous models like Restormer (35.03 dB). On the Outdoor-Rain dataset, it achieved 31.39 dB, a substantial improvement over TransWeather (28.83 dB).
Visually, the model excels at removing "ghosting" effects and restoring sharp edges where other models typically leave muddy patches. By using the codebook, the model effectively "re-paints" the scene using a palette of high-quality samples rather than just cleaning the noise.
Conclusion
AWRCP demonstrates that external priors are essential for handling severe information loss in adverse weather restoration. By combining a VQGAN codebook with flexible alignment mechanisms like Deformable Cross-Attention, the model achieves superior texture and structural fidelity. While the size of the codebook and potential structural distortions remain areas for future research, AWRCP represents a significant step forward in "all-in-one" image restoration.
Summary updated