ART-Point: Improving Rotation Robustness of Point Cloud Classifiers via Adversarial Rotation

Robin Wang, Yibo Yang, Dacheng Tao

TL;DR

ART-Point introduces a novel framework that treats 3D point cloud rotation as an adversarial attack, using adversarial training to build intrinsically rotation-robust classifiers. By optimizing directly over rotation angles and leveraging a category-specific "rotation pool," the method achieves state-of-the-art robustness against arbitrary rotations without sacrificing performance on clean, aligned datasets.

What Problem Does This Paper Address?

Deep learning models for 3D point clouds (like PointNet or DGCNN) are notoriously sensitive to spatial orientations. If a robot encounters an object rotated in an unfamiliar pose, standard classifiers frequently fail.

To fix this, researchers typically rely on two strategies:

  1. Rotation-invariance methods: Transforming inputs into geometric descriptors (like distances or angles) that don't change with rotation.
  2. Rotation-equivariance methods: Designing specialized network architectures where features rotate in tandem with the input.

While effective at handling rotations, both approaches come with a steep cost: they require complex architectural modifications or time-consuming preprocessing. More importantly, they degrade the model's performance on standard, clean, aligned datasets because they restrict the network's capacity to learn highly discriminative features.

ART-Point addresses this trade-off by asking: Can we make standard point cloud classifiers robust to arbitrary rotations purely through smart training, without changing the network architecture or input space?

Methods

The authors frame rotation robustness as a classic adversarial min-max (saddle-point) optimization problem. The framework consists of three core components:

1. Axis-Wise Rotation Attack (Inner Maximization)

Instead of optimizing the full $3 \times 3$ transformation matrix (which can introduce non-rigid deformations), the authors optimize directly over the three rotation angles ($\Phi = [\phi_x, \phi_y, \phi_z]$) using gradient descent. To make the attack more potent, they introduce an axis-wise mechanism: at each step, the algorithm calculates the gradient for each axis and isolates the single most damaging axis to rotate. This finds highly aggressive rotations much faster than standard simultaneous optimization.

2. The Rotation Pool (Defense & Anti-Overfitting)

Directly training a model on its own adversarial rotations often leads to "label leaking" and severe overfitting. To counter this, the authors discovered that adversarial rotations possess strong category-level transferability—a rotation that fools the model on one "Bench" sample will likely fool it on another "Bench" sample.

Figure 2. Transferability of adversarial rotations among samples in the same categories. The adversarial rotation found on one sample in “Bench” can be applied to other samples of the same category to induce high loss and mislead the model to classify them into a wrong category “Bookshelf”.

They exploit this by saving successful adversarial attack angles into a Rotation Pool organized by class category. During training, the network samples diverse rotations from this pool, drastically increasing data variety and preventing the model from overfitting to a narrow set of self-generated gradients.

3. One-Step Optimization

Iterative min-max training is computationally expensive. Borrowing from ensemble adversarial training, the authors propose a one-step optimization method. They construct a robust, static rotation pool upfront by attacking multiple pre-trained models simultaneously. The target model is then trained on this pool in a single pass, matching the performance of slow, iterative training at a fraction of the computational cost.

Figure 1. The general pipeline of our adversarial training approach. In the upper branch, the network takes a clean batch (aligned object) as inputs and finds the most aggressive attack angles by maximizing the classification loss of the eval model. The attack angles will be stored by class in the rotation pool. In the lower branch, the network samples angles from the rotation pool to produce adversarial point clouds for re-training the classifier to obtain the rotation robust model. The red and blue dashed lines respectively indicate routes of the backward gradient in two optimization tasks and point to the final optimized parameters. In the real implementations, the one-step optimization will construct the rotation pool by attacking multiple eval models, while the iterative optimization will update the parameter of the eval model by parameters of the latest re-trained model in each min-max iterations.

Datasets

The authors evaluated their approach on two standard 3D benchmarks:

  • ModelNet40: Contains 12,311 meshed CAD models across 40 categories.
  • ShapeNet16: A larger dataset containing 16,881 shapes across 16 categories.

Models were evaluated under three protocols: Clean (unrotated), Random (arbitrarily rotated), and Attack (adversarially rotated).

Key Insights and Ideas

  • Intrinsic Robustness Over Architectural Constraints: ART-Point proves that rotation robustness can be baked directly into network weights via adversarial training. This eliminates the need for hand-crafted geometric descriptors or rigid equivariant layers.
  • Angle-Space Optimization Matters: Restricting the adversarial search space to pure rotation angles ($\Phi$) guarantees rigid-body transformations, ensuring the attack behaves like a realistic physical rotation rather than a shape-distorting perturbation.
  • Axis-Wise Stepping is More Aggressive: Breaking down the gradient descent into axis-wise selections allows the attack to climb the loss landscape more efficiently than standard multi-axis gradient steps.
Figure 4. Averaged loss values of attacked samples produced by standard attack and axis-wise attack under different attack steps.

Key Results

The authors integrated ART-Point into three baseline classifiers: PointNet, PointNet++, and DGCNN.

  • Outperforming Specialized Architectures: On ModelNet40, ART-DGCNN achieved 91.3% accuracy on clean data, 90.5% on randomly rotated data, and 91.5% under adversarial attack. This outperformed top rotation-invariant and rotation-equivariant models (like SFCNN and Vector Neurons), which typically suffer a drop in clean accuracy (often hovering between 77% and 90%).
  • Massive Gains over Standard Augmentation: Compared to simple rotation data augmentation (RA), ART-Point improved adversarial attack defense by up to 31.2%.
  • Superior Attack Success Rate: The proposed axis-wise attack achieved higher success rates (up to 99.54% on baseline PointNet) than prior state-of-the-art rotation attacks like TSI and CTRI.

Limitations and Future Work

The primary limitation of ART-Point is its reliance on a white-box setting for the inner maximization step; it requires a fully trained model with accessible parameters to generate the initial rotation pool. Additionally, the authors note a potential negative societal impact: because the axis-wise attack is highly effective, it could theoretically be exploited to compromise point-cloud-based 3D object detection systems in autonomous vehicles or robotics if left undefended.

Summary updated

Abstract

Point cloud classifiers with rotation robustness have been widely discussed in the 3D deep learning community. Most proposed methods either use rotation invariant descriptors as inputs or try to design rotation equivariant networks. However, robust models generated by these methods have limited performance under clean aligned datasets due to modifications on the original classifiers or input space. In this study, for the first time, we show that the rotation robustness of point cloud classifiers can also be acquired via adversarial training with better performance on both rotated and clean datasets. Specifically, our proposed framework named ART-Point regards the rotation of the point cloud as an attack and improves rotation robustness by training the classifier on inputs with Adversarial RoTations. We contribute an axis-wise rotation attack that uses back-propagated gradients of the pre-trained model to effectively find the adversarial rotations. To avoid model over-fitting on adversarial inputs, we construct rotation pools that leverage the transferability of adversarial rotations among samples to increase the diversity of training data. Moreover, we propose a fast one-step optimization to efficiently reach the final robust model. Experiments show that our proposed rotation attack achieves a high success rate and ART-Point can be used on most existing classifiers to improve the rotation robustness while showing better performance on clean datasets than state-of-the-art methods.
BibTeX
@inproceedings{wang2022art,
  author = {Robin Wang and Yibo Yang and Dacheng Tao},
  title = {{ART-Point: Improving Rotation Robustness of Point Cloud Classifiers via Adversarial Rotation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
}
Figure