Abstract
Understanding the spatial arrangement and nature of real-world objects is of paramount importance to many complex engineering tasks, including autonomous navigation. Deep learning has revolutionized state-of-the-art performance for tasks in 3D environments; however, relatively little is known about the robustness of these approaches in an adversarial setting. The lack of comprehensive analysis makes it difficult to justify deployment of 3D deep learning models in real-world, safety-critical applications. In this work, we develop an algorithm for analysis of pointwise robustness of neural networks that operate on 3D data. We show that current approaches presented for understanding the resilience of state-of-the-art models vastly overestimate their robustness. We then use our algorithm to evaluate an array of state-of-the-art models in order to demonstrate their vulnerability to occlusion attacks. We show that, in the worst case, these networks can be reduced to 0% classification accuracy after the occlusion of at most 6.5% of the occupied input space.
Introduction
While 3D deep learning models achieve high accuracy on standard benchmarks, they are surprisingly fragile in the face of targeted data removal. In real-world scenarios like autonomous driving or drone navigation, objects are rarely captured in their entirety; they are frequently occluded by other objects or sensor noise.
Previous research into the robustness of 3D models often relied on random occlusion—simply deleting random points from a 3D scan. These tests suggested that models were resilient, maintaining high accuracy even with significant data loss. However, this paper argues that random testing provides a false sense of security. An adversary or a specific environmental condition will not remove points at random; they will remove the "critical points" that the model relies on most for classification. This work introduces the Iterative Saliency Occlusion (ISO) algorithm to expose these vulnerabilities.
Methodology
The authors propose a unified framework to analyze diverse 3D architectures, such as PointNet and VoxNet, by decomposing them into three stages: Latent Translation, Pooling, and a Fully Connected Network (FCN).
The core contribution is the ISO algorithm, which operates through three primary steps:
- Identify Saliency: It calculates the contribution of each point (or voxel) to the model’s final decision, identifying the "critical set."
- Iterative Removal: It greedily removes the most salient points and monitors the drop in model confidence.
- Refinement: A "cleanup" step ensures the attack is minimal by re-adding points that were not strictly necessary for misclassification.
Crucially, for PointNet-style architectures, this attack can be performed in a black-box setting, meaning the attacker does not require knowledge of the model's internal weights to identify critical points.
Vulnerability Analysis and Semantic Shifts
The study reveals that the "critical set" used by models like PointNet is a major vulnerability. While condensing thousands of points into a small set (typically ~25% of input) is efficient, it creates a high-leverage target for attackers. Targeted occlusion does not just create noise; it causes specific, dangerous semantic shifts.
As seen in the figure above, removing a mere 32 points from a 2,048-point scan of a pedestrian can cause PointNet to misclassify the object as a "plant." In safety-critical applications like autonomous driving, such a misclassification could have catastrophic consequences.
Key Results
The ISO algorithm was evaluated against standard models using the ModelNet10/40 and KITTI datasets. The results demonstrate a stark contrast between random and targeted robustness:
- Accuracy Collapse: On ModelNet40, PointNet maintains ~74% accuracy even when 75% of points are removed randomly. In contrast, the ISO attack reduces accuracy to 0% with only 6.5% of the input space occluded.
- VoxNet Fragility: VoxNet proved even more vulnerable. On the KITTI dataset, ISO reduced accuracy to 0% with 25% occlusion, while random occlusion only dropped it to 55.5%.
- Computational Efficiency: The ISO algorithm is highly practical, taking an average of only 4 seconds to find a successful adversarial occlusion for PointNet.
Conclusion
This research demonstrates that current 3D deep learning models are significantly less robust than previously believed. The ISO algorithm provides a powerful "anytime" tool for safety testing, allowing researchers to identify the minimal data removal required to fool a network. Future work must focus on developing architectures that do not rely so heavily on a tiny subset of critical points, potentially by forcing models to integrate more global features to ensure reliability in real-world, adversarial environments.
Summary updated