Seeing is Not Believing: Adversarial Natural Object Optimization for Hard-Label 3D Scene Attacks

Daizong Liu, Wei HuCVPR 2025 · OpenReview

Abstract

Deep learning models for 3D data have shown to be vulnerable to adversarial attacks, which have received increasing attention in various safety-critical applications such as autonomous driving and robotic navigation. Existing 3D attackers mainly put effort into attacking the simple 3D classification model by perturbing point cloud objects in the white/black-box setting. However, real-world 3D applications focus on tackling more complicated scene-based data while sharing no information about the model parameters and logits with users. Therefore, directly applying previous naive 3D attack methods to these applications does not work. To this end, this paper attempts to address the challenging hard-label 3D scene attack with access only to the input/output of the 3D models. To make the attack effective and stealthy, we propose to generate universal adversarial objects, which will mislead scene-aware 3D models to predict attacker-chosen labels whenever these objects are placed on any scene input. Specifically, we inject an imperceptible object trigger with further perturbations into all scenes and learn to mislead their reasoning by only querying the 3D model. We start by initializing the trigger pattern with a realistic object and searching for an appropriate location to place it naturally in the scene data. Then, we design a novel weighted gradient estimation strategy to perturb the object trigger with additive slight noise to make them adversarial in an iterative optimization procedure. Extensive experiments demonstrate that our attack can achieve superior performance on seven 3D models and three scene-based datasets, with satisfactory adversarial imperceptibility and strong resistance to defense methods.

Introduction

Adversarial attacks—inputs designed to trick AI models—are well-documented in 2D images, typically involving global noise layers invisible to humans. However, translating these attacks to 3D environments presents unique challenges. Most existing 3D adversarial research focuses on simple classification tasks in "white-box" settings, where the attacker knows the model's internal weights. These methods often perturb every point in a cloud, resulting in messy, unnatural geometries.

Real-world 3D applications, such as autonomous driving or robotic navigation, deal with complex scenes containing hundreds of objects. Furthermore, these models are usually "black boxes" to the end-user; they provide an input and return a label (hard-label), with no access to underlying gradients or confidence scores. This paper addresses the gap in launching stealthy, effective attacks on complex 3D scenes when the attacker has zero visibility into the model’s internal reasoning.

Methodology

The proposed method, UAO² (Universal Adversarial Object Optimization), moves away from global noise and instead uses realistic 3D objects to break sophisticated 3D scene-understanding models. The approach is defined by three key technical innovations:

  1. Natural Object Triggers: Instead of adding random digital noise to the whole scene, the attack injects a realistic 3D object (sourced from the ScanObjectNN dataset). By perturbing only this specific object, the rest of the scene remains geometrically intact and "natural" to human observers.
  2. Geometry-Aware Placement: To ensure the object doesn't appear to float in mid-air, the researchers utilize a 3D Hough Transform. This algorithm detects flat surfaces (like floors or tables) near the target object, allowing the adversarial trigger to be placed realistically within the environment.
  3. Weighted Gradient Estimation: In a "hard-label" setting, standard backpropagation is unavailable. UAO² employs a Monte Carlo-based estimation with a novel twist: it assigns higher weights to random noises that successfully push the model toward a miscalculation. This makes the optimization process significantly more efficient than standard "blind" querying.
A universal attacker injects a trigger object into a 3D scene to mislead the model's grounding capabilities.

Experimental Evaluation

The researchers evaluated the attack on three major 3D scene datasets: ScanRefer (36,665 language expressions), Nr3D / Sr3D (natural and synthetic language identification), and ScanObjectNN (the source for trigger objects). The attack was tested against seven state-of-the-art 3D grounding models, including SAT, 3DTrans, and EDA.

Key Results

The results demonstrated that perturbing a single object is far stealthier and more effective than global perturbations. On the ScanRefer dataset, the "Unique" accuracy at a 0.5 IoU threshold plummeted from 46.19% to 10.45% for the baseline model.

The API output showing the model's predicted bounding box (green) under the influence of the adversarial trigger.

Furthermore, the attack proved resistant to common defenses. Even when models employed anomaly detection or fine-tuning, the success rate remained high. Because the perturbations are slight and the object is placed on a detected plane, the adversarial samples remain visually indistinguishable from benign scenes to the human eye.

Conclusion

UAO² demonstrates that 3D vision models can be significantly compromised by a single, well-placed "noisy" object. While the attack currently requires a high query count, it serves as a critical benchmark for developing more robust 3D models. Future work may explore the "physical world gap" by printing these adversarial objects to test their effectiveness against real-world LiDAR sensors in physical environments.

Summary updated

Abstract

Deep learning models for 3D data have shown to be vulnerable to adversarial attacks, which have received increasing attention in various safety-critical applications such as autonomous driving and robotic navigation. Existing 3D attackers mainly put effort into attacking the simple 3D classification model by perturbing point cloud objects in the white/black-box setting. However, real-world 3D applications focus on tackling more complicated scene-based data while sharing no information about the model parameters and logits with users. Therefore, directly applying previous naive 3D attack methods to these applications does not work. To this end, this paper attempts to address the challenging hard-label 3D scene attack with access only to the input/output of the 3D models. To make the attack effective and stealthy, we propose to generate universal adversarial objects, which will mislead scene-aware 3D models to predict attacker-chosen labels whenever these objects are placed on any scene input. Specifically, we inject an imperceptible object trigger with further perturbations into all scenes and learn to mislead their reasoning by only querying the 3D model. We start by initializing the trigger pattern with a realistic object and searching for an appropriate location to place it naturally in the scene data. Then, we design a novel weighted gradient estimation strategy to perturb the object trigger with additive slight noise to make them adversarial in an iterative optimization procedure. Extensive experiments demonstrate that our attack can achieve superior performance on seven 3D models and three scene-based datasets, with satisfactory adversarial imperceptibility and strong resistance to defense methods.
BibTeX
@inproceedings{liu2025seeing,
  author = {Daizong Liu and Wei Hu},
  title = {{Seeing is Not Believing: Adversarial Natural Object Optimization for Hard-Label 3D Scene Attacks}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=ALnzXjihHp},
}
Figure