Adverse Weather Removal with Codebook Priors

Tian Ye, Sixiang Chen, Jinbin Bai, Jun Shi, Chenghao Xue, Jingxia Jiang, Junjie Yin, Erkang Chen, Yun LiuICCV 2023 · DOI

Introduction

Restoring images captured under adverse weather conditions—such as heavy rain, snow, or haze—is a challenging inverse problem. In many cases, the degradation is so severe that the original background information is mathematically unrecoverable from the pixels alone. While "all-in-one" restoration models have attempted to handle multiple weather types simultaneously, they often reach a bottleneck because they rely exclusively on the degraded input. If the input lacks critical texture and structure, the model is forced to "guess," leading to blurry or artifact-heavy results.

The AWRCP Framework

To overcome these limitations, the authors propose AWRCP (Adverse Weather Removal with Codebook Priors). This framework shifts the restoration task from simple filtering to a feature-matching and fusion problem by leveraging a pre-trained "dictionary" of high-quality visual features.

VQGAN Codebook Priors

The core of AWRCP is a codebook derived from a Vector-Quantized Generative Adversarial Network (VQGAN) pre-trained on a massive dataset of clean images. This codebook serves as a discrete collection of high-resolution "visual atoms"—textures and shapes that the model can use to reconstruct missing details.

Deformable Cross-Attention (DCA)

Because weather distortions can warp the geometry of an image, the features extracted from a degraded input rarely align perfectly with the clean entries in the codebook. AWRCP employs Deformable Cross-Attention (DCA), which uses a sparse sampling mechanism to flexibly search the codebook for the best-matching high-quality features, effectively bridging the gap between the messy input and the clean prior.

The AWRCP architecture, featuring the VQGAN codebook, Deformable Cross-Attention, and the Hierarchical Texture Warping Head.

Hierarchical Texture Warping Head (HTWH)

To ensure that the restored images maintain fine-grained textures, the model utilizes a Hierarchical Texture Warping Head (HTWH). This module gradually fuses high-resolution codebook features into the output, ensuring that edges and textures (like building facades or fabric weaves) remain crisp and realistic.

Experimental Evaluation

The AWRCP model was tested on several standard benchmarks, including Snow100K, Outdoor-Rain, and RainDrop. These datasets cover a wide range of conditions, from localized raindrop distortions to global haze and heavy snow.

Quantitative and Qualitative Results

AWRCP achieved state-of-the-art (SOTA) performance across all benchmarks. For instance, on the Snow100K-S dataset, it reached a PSNR of 36.92 dB, significantly outperforming previous models like Restormer (35.03 dB). On the Outdoor-Rain dataset, it achieved 31.39 dB, a substantial improvement over TransWeather (28.83 dB).

Qualitative results showing the input and reconstruction across four weather types: Haze, Rain, Raindrop, and Snow.

Visually, the model excels at removing "ghosting" effects and restoring sharp edges where other models typically leave muddy patches. By using the codebook, the model effectively "re-paints" the scene using a palette of high-quality samples rather than just cleaning the noise.

Conclusion

AWRCP demonstrates that external priors are essential for handling severe information loss in adverse weather restoration. By combining a VQGAN codebook with flexible alignment mechanisms like Deformable Cross-Attention, the model achieves superior texture and structural fidelity. While the size of the codebook and potential structural distortions remain areas for future research, AWRCP represents a significant step forward in "all-in-one" image restoration.

Summary updated

BibTeX
@inproceedings{ye2023adverse,
  author = {Tian Ye and Sixiang Chen and Jinbin Bai and Jun Shi and Chenghao Xue and Jingxia Jiang and Junjie Yin and Erkang Chen and Yun Liu},
  title = {{Adverse Weather Removal with Codebook Priors}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01163},
}
Figure