PEARL: Preprocessing Enhanced Adversarial Robust Learning of Image Deraining for Semantic Segmentation

Xianghao Jiao, Yaohua Liu, Jiaxin Gao, Xinyuan Chu, Xin Fan, Risheng Liu

Introduction

Deep learning models for semantic segmentation are essential for safety-critical applications like autonomous driving, where identifying every pixel—from pedestrians to road boundaries—is paramount. However, these models face a "double whammy" of degradation: natural environmental factors like rain and intentional, human-engineered adversarial attacks. While rain streaks blur visual details, adversarial perturbations (such as PGD or BIM attacks) introduce human-imperceptible noise designed to trick the model into catastrophic misclassification.

Historically, researchers have addressed these issues in isolation. A model might be trained to remove rain (deraining) or to resist adversarial noise (adversarial training), but rarely both. This paper introduces PEARL (Preprocessing Enhanced Adversarial Robust Learning), a unified framework that treats both rain and adversarial noise as a combined degradation problem, significantly enhancing the robustness of segmentation models in high-risk environments.

The Challenge: Why Traditional Methods Fail

Standard approaches often rely on a two-stage pipeline: a preprocessing module to remove rain followed by a robust segmentation model. However, this "Naive Adversarial Training" (NAT) approach has a critical flaw. Because the deraining module is not specifically trained to handle adversarial noise, it often leaves behind residual perturbations. These residuals, though small, are frequently enough to bypass the defenses of the downstream segmentation model, leading to a failure in accuracy.

The PEARL Framework

The PEARL framework shifts the defensive strategy by transferring adversarial robustness from the segmentation model directly to the preprocessing deraining module. In this architecture, the derainer is trained to treat adversarial noise as just another form of image degradation, similar to rain streaks. By cleaning both natural and artificial noise simultaneously, the derainer acts as a primary defensive shield, ensuring that the input reaching the segmentation model is significantly more reliable.

Auxiliary Mirror Attack (AMA)

To further refine this process, the authors introduce the Auxiliary Mirror Attack (AMA). While standard adversarial training uses "Negative Adversarial Attacks" (NAA) to identify and exploit model weaknesses, AMA generates a "positive" perturbation. During the training phase, this positive noise is added to the ground-truth clean image. This creates a "mirror" prior that helps the deraining model better understand the complex relationship between rain and adversarial noise, allowing for more effective decomposition and restoration of the original signal.

Experimental Evaluation

The researchers evaluated PEARL on two benchmark datasets: Cityscapes (urban street scenes) and PASCAL VOC 2012. To simulate harsh conditions, they synthesized rain streaks and applied various adversarial attacks (BIM, PGD, and CW) at varying intensities.

The results demonstrate a dramatic improvement in performance:

  • Segmentation Accuracy: On the Cityscapes dataset under a heavy BIM10 attack, a standard model's Mean Intersection over Union (mIoU) plummeted to a mere 2.41%. While the NAT baseline improved this to 31.37%, the PEARL + AMA framework achieved a significantly higher 43.75%.
  • Image Quality: Beyond segmentation accuracy, the framework maintained high Peak Signal-to-Noise Ratio (PSNR) values, indicating that the restored images were high-quality for both human viewing and machine processing.
  • Architecture Agnostic: The framework proved effective across various underlying architectures, including ResNet, PSPNet, and DeepLabv3, suggesting broad applicability.

Conclusion and Future Directions

PEARL represents a significant step forward in unifying natural and adversarial robustness. By treating the derainer as a security component and utilizing the AMA prior, the framework provides a robust defense for semantic segmentation.

Future work may focus on addressing the complexities of real-world rain—such as splashing and veiling effects—which differ from synthetic models. Additionally, while the current framework is computationally intensive due to the nature of adversarial training, further optimization could make it more viable for real-time applications. Finally, the authors suggest that this "unified degradation" approach could be extended to other environmental challenges like fog, snow, or low-light conditions, paving the way for truly all-weather autonomous systems.

Summary updated

BibTeX
@inproceedings{jiao2023pearl,
  author = {Xianghao Jiao and Yaohua Liu and Jiaxin Gao and Xinyuan Chu and Xin Fan and Risheng Liu},
  title = {{PEARL: Preprocessing Enhanced Adversarial Robust Learning of Image Deraining for Semantic Segmentation}},
  booktitle = {ACM Multimedia},
  year = {2023},
  url = {https://openreview.net/forum?id=4h2vZSUhmK},
}
Figure