Relation-Shape Convolutional Neural Network for Point Cloud Analysis

Yongcheng Liu, Bin Fan, Shiming Xiang, Chunhong Pan

Introduction

Deep learning has revolutionized 2D image analysis through Convolutional Neural Networks (CNNs). However, applying these techniques to 3D point clouds—the raw data generated by LiDAR and depth cameras—remains a significant challenge. Unlike images, which are structured as neat grids of pixels, point clouds are unordered, irregular, and highly sensitive to transformations like rotation or shifting.

Previous approaches often bypassed these issues by converting 3D data into 2D images or 3D voxel grids, which unfortunately leads to memory waste and the loss of fine geometric details. Other methods, such as PointNet, process points independently, often failing to capture the critical "local structure"—the way neighboring points relate to one another to form a coherent shape.

The Relation-Shape Convolution (RS-Conv)

To address these limitations, the authors introduce the Relation-Shape Convolution (RS-Conv). In a standard 2D CNN, a pixel is multiplied by a fixed weight within a kernel. In contrast, RS-CNN treats the "weight" as a dynamic function learned from the geometric relationship between a point and its neighbors.

The RS-Conv operator learns high-level shape-aware representations by mapping low-level geometric relations through a shared MLP.

How it Works:

  1. Learning from Relation: For any given neighborhood, the network extracts a "low-level relation" vector, such as Euclidean distance and relative coordinates.
  2. The Mapping Function ($\mathcal{M}$): This vector is passed through a shared Multi-Layer Perceptron (MLP). The MLP learns to translate raw spatial distances into high-level weights that capture the underlying geometric shape.
  3. Hierarchical Architecture: Similar to traditional CNNs, RS-CNN stacks these layers to learn increasingly abstract features, moving from simple edges and corners to complex components like airplane wings or chair legs.

Key Insights and Efficiency

RS-CNN shifts the paradigm from memorizing fixed spatial weights to learning an inductive rule for how geometry implies shape. This makes the model inherently more "shape-aware" and robust to rigid transformations and point permutations. Furthermore, because the mapping function $\mathcal{M}$ is shared across all points, the model is remarkably lightweight, utilizing approximately 60% fewer parameters than the original PointNet while delivering superior performance.

Experimental Results and Robustness

The researchers evaluated RS-CNN on major benchmarks including ModelNet40 (classification and normal estimation) and ShapeNet Part (segmentation).

  • Classification: RS-CNN achieved a state-of-the-art 93.6% accuracy on ModelNet40 using only 3D coordinates, outperforming established models like PointNet++ and DGCNN.
  • Normal Estimation: It achieved a significantly lower error (0.15) compared to PointNet++ (0.29), demonstrating a deeper understanding of local surface geometry.
  • Computational Efficiency: The model reduces FLOPs by about 33% compared to PointNet, making it highly suitable for real-time applications like autonomous driving.

One of the most striking findings is the model's resilience to "point dropout." As shown in the figure below, even when 90% of the points are removed (leaving as few as 64 points), RS-CNN maintains high accuracy, whereas the performance of competing models drops precipitously.

RS-CNN demonstrates exceptional robustness to point dropout compared to other state-of-the-art models.

Conclusion

RS-CNN provides a powerful and efficient framework for 3D point cloud analysis by effectively bringing the strengths of convolutional learning to irregular spatial data. While it may still face challenges with extremely complex or ambiguous geometries, its ability to learn from local relations sets a new standard for shape-aware representation learning in 3D vision.

Summary updated

Abstract

Point cloud analysis is very challenging, as the shape implied in irregular points is difficult to capture. In this paper, we propose RS-CNN, namely, Relation-Shape Convolutional Neural Network, which extends regular grid CNN to irregular configuration for point cloud analysis. The key to RS-CNN is learning from relation, i.e., the geometric topology constraint among points. Specifically, the convolutional weight for local point set is forced to learn a high-level relation expression from predefined geometric priors, between a sampled point from this point set and the others. In this way, an inductive local representation with explicit reasoning about the spatial layout of points can be obtained, which leads to much shape awareness and robustness. With this convolution as a basic operator, RS-CNN, a hierarchical architecture can be developed to achieve contextual shape-aware learning for point cloud analysis. Extensive experiments on challenging benchmarks across three tasks verify RS-CNN achieves the state of the arts.
BibTeX
@inproceedings{liu2019relation,
  author = {Yongcheng Liu and Bin Fan and Shiming Xiang and Chunhong Pan},
  title = {{Relation-Shape Convolutional Neural Network for Point Cloud Analysis}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  doi = {10.1109/cvpr.2019.00910},
}
Figure