Quantifying the robustness of deep multispectral segmentation models against natural perturbations and data poisoning

Elise Bishoff, Charles Godfrey, Myles Mckay, Eleanor Byler2023 · arXiv

Introduction

In the field of remote sensing, the integration of multispectral data—specifically wavelengths beyond the visible spectrum such as Near-Infrared (NIR)—has become a standard approach for enhancing the accuracy of deep learning models. While the performance benefits of these additional bands for tasks like semantic segmentation are well-documented, their impact on model robustness remains under-explored. This study, conducted by researchers at the Pacific Northwest National Laboratory and the University of Washington, investigates a critical trade-off: does the inclusion of multispectral data make models more resilient to environmental challenges, or does it introduce new vulnerabilities to malicious exploitation?

Problem Statement and Objectives

Deep learning models deployed for satellite and overhead imagery analysis must operate reliably in diverse conditions. The researchers categorize robustness into two distinct domains:

  1. Natural Robustness: The ability of a model to maintain performance under adverse environmental conditions, such as fog, snow, or haze.
  2. Adversarial Robustness: The resistance of a model to "data poisoning" attacks, where a malicious actor manipulates a portion of the training data to create backdoors or specific misclassifications.

The primary objective of this research is to quantify how different multispectral fusion strategies influence a model's susceptibility to both human-engineered attacks and natural atmospheric perturbations.

Methodology

Model Architecture and Fusion Strategies

The study utilized the DeepLabv3 architecture with a ResNet50 backbone for semantic segmentation. To understand the impact of multispectral data, the researchers compared four distinct configurations:

  • RGB-only: The baseline model using standard visible light channels.
  • NIR-only: A model trained exclusively on Near-Infrared data.
  • Early Fusion: A 4-channel input approach where RGB and NIR data are stacked before being processed by the network.
  • Late Fusion: A dual-stream approach where RGB and NIR data are processed through separate backbones, with features merged at a later stage.

Data Poisoning Attacks

Three types of data poisoning were implemented to test adversarial robustness:

  • Fine-grained Attacks (Square/Line): Small, identifiable geometric shapes were inserted into 10% of the training images to trigger specific misclassifications (e.g., identifying a building as a forest).
  • Texture Attacks: A more sophisticated, "physically realizable" attack where pixels belonging to a specific class (e.g., buildings) were replaced with a foliage texture during training. This teaches the model to associate a natural texture with an incorrect label, making the attack harder to detect.

Physics-Aware Natural Perturbations

To test natural robustness, the researchers developed physically realistic versions of the ImageNet-C "Snow" and "Fog" corruptions. Unlike standard digital filters, these perturbations were adjusted for the NIR band. For instance, the model accounted for the physical reality that NIR light penetrates haze and fog more effectively than visible light, providing a more accurate simulation of real-world sensor performance.

Experimental Setup

The study utilized the Urban Semantic 3D (US3D) dataset, which includes high-resolution overhead imagery and labels for seven classes (ground, foliage, building, water, road, etc.) across three major U.S. cities: Atlanta, Jacksonville, and Omaha.

Key Insights and Results

The Performance-Robustness Paradox

The results revealed a significant paradox: multispectral models (specifically RGB+NIR) achieved the highest accuracy on clean data but were also the most vulnerable to manipulation. The RGB-NIR Early Fusion model led in performance with a mean Intersection over Union (mIOU) of 0.769. However, these same models exhibited the highest Adversarial Success Rate (ASR). The researchers conclude that the additional spectral information provides a larger "surface area" for attackers to exploit, making the models easier to poison.

Superiority of Early Fusion in Adverse Weather

While multispectral models were more susceptible to poisoning, they proved significantly more robust against natural perturbations. In heavy snow conditions, the Early Fusion model maintained an mIOU of 0.65, significantly outperforming the RGB-only baseline, which dropped to 0.46. This resilience is attributed to the model's ability to rely on the NIR signal when the visible spectrum is obscured by weather. Early fusion, in particular, outperformed late fusion in these scenarios, suggesting that early integration of features allows the network to learn more robust cross-modal dependencies.

Stealth of Texture-Based Attacks

The study found that texture-based poisoning was particularly dangerous because it was "stealthy." Unlike geometric triggers, the texture attack did not significantly degrade the model's performance on clean data. This means a compromised model could pass standard validation tests while still containing a hidden backdoor that an attacker could trigger in the field.

Conclusion and Future Directions

The research highlights a critical trade-off for practitioners in the remote sensing community. While adding NIR data via Early Fusion is highly recommended for models operating in regions with frequent adverse weather, it necessitates a higher standard of data provenance and security.

Future work is expected to expand beyond NIR to include Short-Wave Infrared (SWIR) and hyperspectral data, as well as the integration of structural data like LiDAR. Furthermore, the development of specific defense mechanisms to mitigate multispectral poisoning remains a vital area for subsequent research.

Summary updated

Abstract

In overhead image segmentation tasks, including additional spectral bands beyond the traditional RGB channels can improve model performance. However, it is still unclear how incorporating this additional data impacts model robustness to adversarial attacks and natural perturbations. For adversarial robustness, the additional information could improve the model’s ability to distinguish malicious inputs, or simply provide new attack avenues and vulnerabilities. For natural perturbations, the additional information could better inform model decisions and weaken perturbation effects or have no significant influence at all. In this work, we seek to characterize the performance and robustness of a multispectral (RGB and near infrared) image segmentation model subjected to adversarial attacks and natural perturbations. While existing adversarial and natural robustness research has focused primarily on digital perturbations, we prioritize on creating realistic perturbations designed with physical world conditions in mind. For adversarial robustness, we focus on data poisoning attacks whereas for natural robustness, we focus on extending ImageNet-C common corruptions for fog and snow that coherently and self-consistently perturbs the input data. Overall, we find both RGB and multispectral models are vulnerable to data poisoning attacks regardless of input or fusion architectures and that while physically realizable natural perturbations still degrade model performance, the impact differs based on fusion architecture and input data.
BibTeX
@misc{bishoff2023quantifying,
  author = {Elise Bishoff and Charles Godfrey and Myles Mckay and Eleanor Byler},
  title = {{Quantifying the robustness of deep multispectral segmentation models against natural perturbations and data poisoning}},
  year = {2023},
  eprint = {2305.11347},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.11347},
}
Figure