Prior-based Domain Adaptive Object Detection for Adverse Weather Conditions

Vishwanath A. Sindagi, Poojan Oza, Rajeev Yasarla, Vishal M. Patel

TL;DR

This paper introduces a novel "prior-based domain adaptive object detection" framework to improve object detection performance in adverse weather conditions like haze and rain. By leveraging weather-specific "priors" (like transmission maps for haze or rain streaks for rain) and a new "prior-adversarial loss," the method effectively adapts detectors trained on clean images to perform well on degraded images, outperforming existing domain adaptation techniques.

What Problem Does This Paper Address?

Object detection systems, particularly those based on deep learning, often struggle when deployed in real-world scenarios with adverse weather conditions such as haze and rain. These conditions degrade image quality, causing a "domain shift" between the clean training data and the real-world, weather-affected data. This shift leads to a significant drop in detection accuracy. While some methods attempt to "de-haze" or "de-rain" images as a preprocessing step, these often require complex networks, pixel-level supervision, and can still result in a domain shift. Existing domain adaptation techniques, while promising, often treat the entire image as uniformly degraded, neglecting the spatially varying nature of weather effects. This paper aims to bridge this gap by developing an unsupervised domain adaptation method that specifically accounts for the physics of weather degradation.

Methods

The core of the proposed method is a "prior-based domain adversarial object detection" framework. Key components include:

  • Weather-Specific Priors: The authors leverage the mathematical models describing haze and rain. For haze, the "transmission map" (indicating how much light is attenuated) is used as a prior. For rain, the "rain residue" (the visual artifact of rain streaks) serves as the prior. These priors are estimated without requiring manual annotations, often using existing techniques like the "dark channel prior" for haze.
  • Prior-Adversarial Loss (PAL): This novel loss function is designed to make the features extracted by the object detector "weather-invariant." It works by training a "Prior Estimation Network" (PEN) to predict the weather-specific prior from the detector's features. Simultaneously, the detector is trained to "fool" the PEN, meaning its features should contain minimal weather-specific information. This adversarial process, guided by the weather priors, helps align the source and target domain features.
  • Residual Feature Recovery Blocks (RFRBs): To further address distortions introduced by weather in the feature space, RFRBs are incorporated into the detection pipeline. Inspired by residual learning, these blocks learn to recover or "de-distort" the features, leading to improved performance.
  • Domain Adversarial Training: Standard domain adversarial techniques are also employed, but the proposed prior-adversarial loss is integrated to provide more targeted adaptation.

The overall framework combines these elements within a Faster R-CNN architecture.

Datasets

The paper uses the following datasets for training and evaluation:

  • Source Domain: Cityscapes (clear weather images)
  • Target Domains (Hazy):
    • Foggy-Cityscapes
    • RTTS
    • UFDD-Haze (derived from WIDER-Face)
  • Target Domains (Rainy):
    • Rainy-Cityscapes (newly created synthetic dataset)
    • UFDD-Rain (derived from WIDER-Face)
  • Preliminary Study (Snow):
    • UFDD-Snow (derived from WIDER-Face)

The Rainy-Cityscapes dataset was created by synthesizing rain streaks onto Cityscapes images, providing a benchmark for rainy conditions.

Key Insights and Ideas

  • Weather Degradation is Spatially Varying: A crucial insight is that weather effects like haze and rain are not uniform across an image. This spatial variability necessitates adaptation methods that can handle localized degradations, unlike traditional domain adaptation that assumes uniform shifts.
  • Physics-Based Priors are Powerful: By modeling the physics of haze and rain, the authors extract meaningful "priors" that directly capture the essence of the weather degradation. These priors are more informative than generic domain labels for guiding the adaptation process.
  • Adversarial Learning with Priors: The proposed "prior-adversarial loss" is a clever way to enforce weather-invariance. By training a network to predict the weather prior and then training the main detector to fool it, the features become robust to weather effects.
  • Residual Feature Recovery Complements Adaptation: The RFRBs act as a "de-distorter" for features, effectively cleaning up the information corrupted by weather. This complements the domain adaptation process by directly addressing feature distortions.

Key Results

The proposed "prior-based domain adaptive object detection" framework consistently outperforms existing methods across various adverse weather conditions:

  • Hazy Conditions: Achieved state-of-the-art performance on Foggy-Cityscapes, outperforming previous domain adaptation methods by up to 3.6% mAP and surpassing the best existing approach by 2.8%. Similar improvements were observed on RTTS and UFDD-Haze.
  • Rainy Conditions: On the newly created Rainy-Cityscapes dataset, the proposed method significantly outperformed other approaches, showing improvements of up to 9% over baseline domain adaptation methods and outperforming recent deraining+detection methods.
  • Qualitative Results: Figures demonstrate that the proposed method produces higher confidence detections and misses fewer objects compared to baseline methods, especially in challenging scenarios.
  • Ablation Studies: Experiments confirmed the effectiveness of individual components: domain adaptation loss improves performance, RFRBs provide further gains, and the prior-adversarial loss (FRCNN+P5+R5) offers a significant boost over standard domain adversarial training.

Limitations and Future Work

  • Computational Overhead: While the prior computation is not required during inference, the proposed method does introduce additional network modules (PEN and RFRB), which could increase computational cost compared to simpler methods.
  • Extensibility to Other Conditions: The paper suggests that the framework is extensible to other weather conditions (e.g., low-light, water puddles, snow) by leveraging their respective mathematical models and priors. Future work will explore these conditions.
  • Dataset Creation: The creation of the Rainy-Cityscapes dataset is a step towards better evaluation, but more diverse and challenging datasets for various weather conditions would be beneficial.

Summary updated

Abstract

Adverse weather conditions such as haze and rain corrupt the quality of captured images, which cause detection networks trained on clean images to perform poorly on these images. To address this issue, we propose an unsupervised prior-based domain adversarial object detection framework for adapting the detectors to hazy and rainy conditions. In particular, we use weather-specific prior knowledge obtained using the principles of image formation to define a novel prior-adversarial loss. The prior-adversarial loss used to train the adaptation process aims to reduce the weather-specific information in the features, thereby mitigating the effects of weather on the detection performance. Additionally, we introduce a set of residual feature recovery blocks in the object detection pipeline to de-distort the feature space, resulting in further improvements. Evaluations performed on various datasets (Foggy-Cityscapes, Rainy-Cityscapes, RTTS and UFDD) for rainy and hazy conditions demonstrates the effectiveness of the proposed approach.
BibTeX
@inproceedings{sindagi2020prior,
  author = {Vishwanath A. Sindagi and Poojan Oza and Rajeev Yasarla and Vishal M. Patel},
  title = {{Prior-based Domain Adaptive Object Detection for Adverse Weather Conditions}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=rTcpvAyGH-so},
}
Figure