SGMNet: Learning Rotation-Invariant Point Cloud Representations via Sorted Gram Matrix

Jianyun Xu, Xin Tang, Yushi Zhu, Jie Sun, Shiliang PuICCV 2021 · DOI

Researchers from the Hikvision Research Institute have introduced SGMNet, a novel framework designed to make 3D point cloud processing naturally resistant to rotations. By rethinking local geometry description through the lens of a "Sorted Gram Matrix" (SGM), the authors achieve state-of-the-art performance on 3D tasks without the computational burden of massive data augmentation or the instability of local reference frames.

Abstract

SGMNet addresses the inherent rotation sensitivity of 3D point clouds by introducing a Sorted Gram Matrix (SGM) descriptor. Unlike traditional methods that rely on data augmentation or unstable local reference frames, SGMNet models dense pairwise relationships within local neighborhoods to achieve mathematical rotation and permutation invariance. Evaluated on ModelNet40 and ShapeNet Part, SGMNet demonstrates superior performance and robustness under arbitrary 3D rotations, maintaining high accuracy where standard models fail.

The Problem of Rotation Sensitivity

Point clouds, represented as sets of $(x, y, z)$ coordinates, are the standard format for 3D sensors like LiDAR. However, they are fundamentally rotation-sensitive; rotating an object completely changes its raw coordinate values. Standard deep learning models like PointNet struggle with this variance. Existing solutions typically fall into three categories:

  1. Data Augmentation: Training on millions of rotated versions of an object, which is computationally expensive and lacks formal guarantees.
  2. Local Reference Frames (LRFs): Defining a "canonical" orientation for local patches, which is often sensitive to noise and point density changes.
  3. Point-Pair Features (PPFs): Using angles and distances. Most PPF methods only relate a center point to its neighbors, losing the complex geometric "web" between the neighbors themselves.

The SGM Module

The core innovation of SGMNet is the Sorted Gram Matrix (SGM). This module creates a descriptor that is mathematically guaranteed to be invariant to 3D rotations and point permutations.

For any local patch, SGMNet employs dense connectivity, connecting every point in the neighborhood to every other point. It then computes the vector inner product ($x_i^T x_j$) for every pair, forming a $K \times K$ Gram matrix (where $K$ is the number of neighbors). This matrix encodes both distances and relative angles. To ensure the descriptor remains the same regardless of the order in which points are listed (permutation invariance), the values in each row of the Gram matrix are sorted.

The SGM Module integrated into a PointNet++ Set Abstraction layer, showing the generation of the sorted Gram matrix from local geometric coordinates.

Architecture and Integration

SGMNet is designed as a "drop-in" module. In their primary implementation, the authors integrated the SGM module into the standard PointNet++ architecture. By replacing raw $xyz$ coordinates in the "Set Abstraction" layers with SGM features, the network learns from rotation-invariant geometric signatures. The authors proved mathematically that if two point patches share the same Gram matrix, they are geometrically identical up to rotation or reflection.

Experimental Results

The authors evaluated SGMNet on industry-standard benchmarks, specifically ModelNet40 (classification) and ShapeNet Part (segmentation), under three rotation scenarios: upright training/testing (z/z), upright training with rotated testing (z/SO(3)), and arbitrary rotation for both (SO(3)/SO(3)).

  • Classification: On ModelNet40, SGMNet achieved 90.0% accuracy under arbitrary rotations ($SO(3)/SO(3)$), significantly outperforming previous rotation-invariant leaders like ClusterNet (87.1%).
  • Segmentation: On ShapeNet Part, SGMNet maintained a high 79.3 mIoU (mean Intersection over Union) regardless of rotation. In contrast, standard PointNet++ performance dropped from 80.6 to 45.9 when test data was rotated.
Qualitative part segmentation results on arbitrarily rotated objects from the ShapeNet Part dataset, demonstrating consistent performance across orientations.

Conclusion

SGMNet provides a robust and efficient solution for 3D point cloud analysis. By modeling the "web" of local geometry through dense connectivity and the Gram matrix, it achieves true rotation invariance. Furthermore, the model's efficiency is notable; a "mini" version of SGMNet with only 0.23M parameters outperformed much larger, rotation-sensitive models on rotated datasets, proving that smart geometric design can replace brute-force data augmentation.

Summary updated

BibTeX
@inproceedings{xu2021sgmnet,
  author = {Jianyun Xu and Xin Tang and Yushi Zhu and Jie Sun and Shiliang Pu},
  title = {{SGMNet: Learning Rotation-Invariant Point Cloud Representations via Sorted Gram Matrix}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01030},
}
Figure