Robustness of 3D Deep Learning in an Adversarial Setting.

Matthew Wicker, Marta Kwiatkowska

TL;DR

This paper introduces a novel algorithm, Iterative Salience Occlusion (ISO), to rigorously test the robustness of 3D deep learning models against adversarial occlusion. They demonstrate that existing methods significantly overestimate model robustness, revealing that even small occlusions (as little as 6.5% of input) can reduce state-of-the-art 3D networks to zero classification accuracy.

What Problem Does This Paper Address?

Autonomous systems like self-driving cars and drones rely heavily on understanding 3D environments. Deep learning has achieved remarkable accuracy in 3D object recognition, leading to their consideration for safety-critical applications. However, the robustness of these 3D deep learning models, particularly against adversarial attacks, is poorly understood. Unlike 2D image recognition, where adversarial attacks are well-studied, 3D data representations (like point clouds and voxels) present unique challenges for existing analysis tools. This lack of understanding makes it difficult to deploy these models in real-world scenarios where data is often noisy and incomplete, and where failures can have severe consequences. The paper aims to bridge this gap by developing a method to systematically evaluate the vulnerability of 3D deep learning models to adversarial occlusion.

Methods

The authors propose a unified framework for analyzing 3D deep learning pipelines, breaking them down into a "latent translation" step (which maps raw 3D data to an intermediate representation) and a "fully connected network" (FCN) step for classification. Their core contribution is the Iterative Salience Occlusion (ISO) algorithm.

ISO works by identifying a "critical point set" – points in the input that, if removed, significantly impact the latent representation. The algorithm then iteratively removes points from this critical set, aiming to find the minimal set of occlusions that cause a misclassification. This approach is designed to be more effective than random occlusion because it targets the most influential parts of the input for the network's decision.

Key aspects of their method include:

  • Unified Framework: Generalizing analysis across both point-cloud (e.g., PointNet) and volumetric (e.g., VoxNet) representations.
  • Critical Point Set: Identifying influential input points based on their impact on the latent representation.
  • Iterative Refinement: A systematic process to find minimal occlusions that lead to misclassification.
  • Black-box Capability: While the core algorithm requires white-box access to identify the critical set, they prove it can operate in a black-box setting for PointNet architectures.
  • Verification Guarantees: The algorithm provides guarantees about finding an adversarial example if one exists and finding a minimal one.

Datasets

The authors trained and evaluated their models on the following standard 3D datasets:

  • ModelNet10: A dataset of 10 object categories.
  • ModelNet40: A dataset of 40 object categories.
  • KITTI: 3D objects extracted from the LIDAR sensor of the KITTI self-driving car dataset.

They used the standard train-test splits provided with ModelNet.

Key Insights and Ideas

  • Random Occlusion is Misleading: The most significant finding is that random occlusion vastly overestimates the robustness of 3D deep learning models. This is because random removal doesn't target the specific points that are most critical to the network's decision-making process.
  • Vulnerability to Minimal Occlusion: Even a small percentage of occluded input (as low as 6.5% in the worst case) can completely degrade a network's classification accuracy. This highlights a critical safety concern for real-world applications.
  • Critical Points are Weaknesses: Contrary to some prior assumptions, the "critical points" identified by the network's internal workings are not necessarily indicators of robustness but rather potential points of vulnerability that can be exploited.
  • Unified Analysis is Possible: The paper demonstrates that a common analytical framework can be applied to different 3D data representations (point clouds and voxels), paving the way for more standardized robustness testing.

Key Results

  • In the worst-case scenario, the proposed ISO algorithm reduced the classification accuracy of a VoxNet model trained on ModelNet40 to 0% after occluding just 6.5% of the input.
  • For PointNet models, the critical set typically comprises around 25% of the input points.
  • Across various models and datasets, the ISO algorithm consistently showed that random occlusion overestimates robustness by approximately 60% compared to their targeted approach.
  • The algorithm could achieve significant accuracy reduction (to less than 10%) by manipulating only half of the data, even with the ISO algorithm.

Limitations and Future Work

  • Volumetric Saliency: While the critical set for PointNet is well-defined, volumetric networks present a challenge as all points can contribute to the salience. The authors addressed this by using a threshold (top 25% of salience) to define the critical set for volumetric inputs.
  • Black-box for Volumetric: The paper proves black-box capability for PointNet but doesn't explicitly detail it for volumetric networks, though it suggests the framework can be extended.
  • Future Directions: The authors suggest using their saliency computation methods to derive bounds on the safety of classification for real-time, safety-critical systems. They also advocate for developing more robust pipelines for point-cloud data processing to improve generalization.

Summary updated

Abstract

Understanding the spatial arrangement and nature of real-world objects is of paramount importance to many complex engineering tasks, including autonomous navigation. Deep learning has revolutionized state-of-the-art performance for tasks in 3D environments; however, relatively little is known about the robustness of these approaches in an adversarial setting. The lack of comprehensive analysis makes it difficult to justify deployment of 3D deep learning models in real-world, safety-critical applications. In this work, we develop an algorithm for analysis of pointwise robustness of neural networks that operate on 3D data. We show that current approaches presented for understanding the resilience of state-of-the-art models vastly overestimate their robustness. We then use our algorithm to evaluate an array of state-of-the-art models in order to demonstrate their vulnerability to occlusion attacks. We show that, in the worst case, these networks can be reduced to 0% classification accuracy after the occlusion of at most 6.5% of the occupied input space.
BibTeX
@inproceedings{wicker2019robustness,
  author = {Matthew Wicker and Marta Kwiatkowska},
  title = {{Robustness of 3D Deep Learning in an Adversarial Setting.}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  doi = {10.1109/cvpr.2019.01204},
}
Figure