Sensitivity-Informed Augmentation for Robust Segmentation

Laura Y. Zheng, Wenjie Wei, Tony Wu, Jacob Clements, Shreelekha Revankar, Andre Harrison, Yu Shen, Ming C. LinarXiv 2024 · OpenReview

Abstract

Achieving robustness in image segmentation models is challenging due to the fine-grained nature of pixel-level classification. These models, which are crucial for many real-time perception applications, particularly struggle when faced with natural corruptions in the wild for autonomous systems. While sensitivity analysis can help us understand how input variables influence model outputs, its application to natural and uncontrollable corruptions in training data is computationally expensive. In this work, we present an adaptive, sensitivity-guided augmentation method to enhance robustness against natural corruptions. Our sensitivity analysis on average runs 10x faster and requires about 200x less storage than previous sensitivity analysis, enabling practical, on-the-fly estimation during training for a model-free augmentation policy. With minimal fine-tuning, our sensitivity-guided augmentation method achieves improved robustness on both real-world and synthetic datasets compared to state-of-the-art data augmentation techniques in image segmentation.

Introduction

Image segmentation models serve as the backbone for critical real-time perception systems, including autonomous vehicles and medical imaging. Despite their sophistication, these models are notoriously brittle when encountering "natural corruptions"—unpredictable environmental variations such as heavy rain, snow, fog, or sensor noise.

Standard data augmentation techniques often employ a stochastic "spray and pray" approach, applying random transformations that may not target a model's specific vulnerabilities. While sensitivity analysis can pinpoint the most beneficial augmentations, traditional methods are computationally prohibitive, requiring massive offline datasets and extensive model training. This research introduces a framework for making sensitivity-guided augmentation efficient enough to integrate directly into standard training pipelines.

Methodology

The core of this work is an online sensitivity analysis framework that eliminates the need for pre-generated datasets and offline processing.

Overview of the proposed online sensitivity analysis pipeline for adaptive data augmentation.

The pipeline consists of three primary components:

  1. Adaptive Sensitivity Estimation: Rather than evaluating every possible perturbation intensity, the method iteratively approximates a sensitivity curve ($g$-curve). It utilizes Kernel Inception Distance (KID) to quantify image degradation and Model Accuracy (MA) to monitor performance degradation.
  2. Fast Analysis Algorithm: By employing a warmup period on clean data and Piecewise Cubic Hermite Interpolating Polynomial (PCHIP) splines, the algorithm rapidly identifies "sensitive" levels for various perturbations, such as blur, noise, and color shifts.
  3. Dynamic Sampling Policy: During training, the model samples from these identified sensitive levels using a Beta-Binomial distribution. This prioritizes "harder" levels—where the model currently underperforms—ensuring training resources are focused on the most impactful data variations.

Experimental Evaluation

The method was rigorously evaluated across a diverse set of benchmarks to demonstrate generalizability:

  • Real-World Adverse Weather: The ACDC (Adverse Conditions Dataset with Correspondences) dataset, covering fog, rain, night, and snow.
  • Domain Shift: The India Driving Dataset (IDD).
  • Synthetic Corruptions: ImageNet-C (IN-C) and the AdvSteer benchmark.
  • Standard & Specialized Domains: Cityscapes, ADE20K, VOC2012, POTSDAM (aerial), and Synapse (medical imaging).

Key Results and Insights

Robustness and Efficiency

The proposed method achieved significant gains on the ACDC dataset, notably a 6.20% relative mIoU improvement in snow and a 3.85% improvement in rain over the next best state-of-the-art technique. Crucially, this was achieved with a sensitivity analysis that runs 10x faster and requires 200x less storage than previous methods like AdvSteer.

Data Efficiency

A standout feature of the sensitivity-guided approach is its performance in low-data regimes. The method maintains a higher mIoU than competitors even as the number of training samples is reduced, demonstrating superior data efficiency.

Comparison of mIoU performance against state-of-the-art methods across different dataset sizes in adverse weather (left) and domain shift (right) conditions.

Qualitative Generalization

Qualitatively, the model demonstrated an ability to handle complex, unsimulated corruptions. For instance, it successfully learned to ignore windshield wiper occlusions in rainy scenes, despite "windshield wipers" never being an explicit part of the training augmentation set. This suggests the basis perturbations used are effective proxies for complex real-world noise.

Conclusion

This work demonstrates that adaptive, online sensitivity analysis is a practical and highly effective tool for enhancing the robustness of segmentation models. While "Night" scenarios remain a significant challenge due to unique lighting artifacts, the sensitivity-informed approach provides a clear path forward for more reliable perception in autonomous systems. Future work will explore expanding the basis perturbation set and applying this adaptive analysis to other complex tasks like video processing and 3D perception.

Summary updated

Abstract

Achieving robustness in image segmentation models is challenging due to the fine-grained nature of pixel-level classification. These models, which are crucial for many real-time perception applications, particularly struggle when faced with natural corruptions in the wild for autonomous systems. While sensitivity analysis can help us understand how input variables influence model outputs, its application to natural and uncontrollable corruptions in training data is computationally expensive. In this work, we present an adaptive, sensitivity-guided augmentation method to enhance robustness against natural corruptions. Our sensitivity analysis on average runs 10x faster and requires about 200x less storage than previous sensitivity analysis, enabling practical, on-the-fly estimation during training for a model-free augmentation policy. With minimal fine-tuning, our sensitivity-guided augmentation method achieves improved robustness on both real-world and synthetic datasets compared to state-of-the-art data augmentation techniques in image segmentation. Code implementation for this work can be found at: https://github.com/laurayuzheng/SensAug.
BibTeX
@misc{zheng2024sensitivity,
  author = {Laura Y. Zheng and Wenjie Wei and Tony Wu and Jacob Clements and Shreelekha Revankar and Andre Harrison and Yu Shen and Ming C. Lin},
  title = {{Sensitivity-Informed Augmentation for Robust Segmentation}},
  year = {2024},
  url = {https://openreview.net/forum?id=kKbiEuAk9r},
}
Figure