TL;DR
This paper introduces a novel viewpoint-invariant coordinate transform as input to point cloud convolution networks. This transform helps networks generalize better to varying neighborhood scales and rotations, significantly improving robustness and performance on tasks like semantic segmentation, especially under challenging conditions like test-time downsampling.
What Problem Does This Paper Address?
Convolutional Neural Networks (CNNs) have revolutionized computer vision, but applying them to 3D point clouds presents unique challenges. Unlike regular 2D image grids, point clouds are irregularly sampled. This irregularity makes it difficult for standard convolution operations to generalize, particularly when the neighborhood structure changes between training and testing. A key issue arises from the common use of k-Nearest Neighbors (kNN) to define these neighborhoods. While kNN is efficient, the spatial arrangement of neighbors can vary drastically, making it hard for a network trained on one set of neighborhoods to perform well on another. Existing solutions like using e-balls to constrain neighbor distances are computationally expensive, and data augmentation alone doesn't fully solve the generalization problem.
Methods
The core contribution is a viewpoint-invariant (VI) coordinate transform. This transform takes the surface normal of a center point and the vector to a neighboring point, along with their respective surface normals, and uses them to construct an orthonormal basis. Angles and projections onto this basis are then computed, creating an 8-dimensional descriptor. This VI descriptor is designed to be invariant to rotations and scale changes in the local neighborhood.
The authors propose feeding this VI descriptor, either alone or concatenated with the standard (x, y, z) coordinates (referred to as VI+XYZ), into the weight-generating function of point convolution networks, specifically PointConv. This allows the network to learn which aspects of the input (rotation-invariant, scale-invariant, or standard coordinates) are most important for generating effective convolution weights for different layers.
Datasets
- SemanticKITTI: Used for semantic scene segmentation.
- ScanNet: Also used for 3D semantic scene segmentation.
- ModelNet40: Used for 3D point cloud classification, specifically to evaluate rotation invariance.
Key Insights and Ideas
- kNN Neighborhoods are Inherently Unstable: The paper highlights that the irregular nature of kNN neighborhoods in point clouds is a significant bottleneck for generalization.
- Invariance is Key for Robustness: Introducing viewpoint-invariant features directly into the convolution's weight generation process significantly improves robustness to variations in neighborhood scale and rotation.
- Concatenation Offers Flexibility: Combining the VI descriptor with standard XYZ coordinates (VI+XYZ) allows the network to learn to leverage both invariant and non-invariant features, leading to the best performance. This suggests that for real-world tasks, a mix of invariances is often optimal.
- Efficiency Gains: By using kNN neighborhoods with the VI transform, the method achieves performance comparable to slower e-ball based approaches while being significantly more efficient.
- Smaller Models Benefit Greatly: The VI transform helps smaller models maintain performance even under significant downsampling, making them more practical for deployment.
Key Results
- ScanNet: The proposed VI+XYZ approach achieved state-of-the-art performance on ScanNet, outperforming standard PointConv and showing significant robustness to test-time downsampling (e.g., a 66.4% relative improvement at 20k points compared to using only XYZ coordinates).
- SemanticKITTI: The VI-PointConv method achieved state-of-the-art results among point-based approaches, outperforming KPConv and standard PointConv.
- ModelNet40: The VI transform enabled rotation-invariant convolutions, achieving competitive results without requiring extensive rotation augmentation during training.
- Efficiency: The VI transform adds minimal computational overhead, making it a practical enhancement.
Limitations and Future Work
- Computational Cost of e-balls: The paper notes that while e-ball neighborhoods offer robustness, they are significantly slower than kNN, which is a practical limitation.
- Generalization Gap: While the proposed method significantly improves robustness, the authors acknowledge that a gap still exists between point-based and sparse volumetric convolution approaches.
- Future Work: The authors suggest exploring the application of their VI transform in problems where viewpoint-invariance is extremely critical, such as relocalization in Simultaneous Localization and Mapping (SLAM). They also believe that further research into combining different types of invariances could lead to even better results.
Summary updated