Point Convolutional Neural Networks by Extension Operators

Matan Atzmon, Haggai Maron, Yaron Lipman

Researchers at the Weizmann Institute of Science have introduced Point Convolutional Neural Networks (PCNN), a mathematically rigorous framework for applying convolutional neural networks (CNNs) to irregular 3D point clouds. By treating point clouds as continuous functions rather than discrete sets, PCNN achieves translation invariance and robustness to sampling density, overcoming major hurdles in 3D deep learning.

The Challenge of 3D Point Clouds

While deep learning has transformed 2D image analysis, 3D data captured as point clouds—unorganized collections of $(x, y, z)$ coordinates—presents unique difficulties:

  1. Permutation Invariance: The output must remain the same regardless of the order in which points are listed in a file.
  2. Irregular Density: Real-world scans often have varying point densities, which can confuse standard discrete filters.
  3. Translation Invariance: A geometric feature (like a corner) should be detectable regardless of its absolute position in space.

Previous approaches often "voxelized" data into low-resolution 3D grids, which are memory-intensive and lose fine detail, or used symmetric functions like Max-Pooling, which can discard structural nuances.

The Extension-Convolution-Restriction Mechanism

PCNN addresses these issues through a three-step operator loop that bridges the gap between discrete points and continuous volumetric space:

  1. Extension ($E_X$): Discrete values at each point are "spread" into the surrounding 3D space using Radial Basis Functions (RBFs), specifically Gaussians. This transforms the point cloud into a continuous volumetric function.
  2. Convolution ($O$): A continuous volumetric convolution is applied to this function. By using Gaussians for both extension and kernels, the authors solve the convolution integral analytically in closed form, avoiding the need for a discrete 3D grid.
  3. Restriction ($R_X$): The resulting continuous function is sampled back at the original point locations.
The PCNN pipeline: Extension maps points to a continuous volume, followed by convolution and restriction back to the point set.

This mechanism allows the network to benefit from the mathematical properties of volumetric CNNs while maintaining the computational efficiency of point-based methods.

Key Insights and Robustness

A significant finding of the paper is the "Constant 1" trick: applying the extension operator to a point cloud where every point has a value of "1" results in a volumetric function that approximates the indicator function of the underlying surface. This allows the network to perceive geometry and mean curvature even before processing specific features.

Furthermore, the method is inherently robust to changes in point density. Because the extension operator effectively reconstructs a surface via Gaussian "blur," the convolution remains stable even if a large percentage of points are removed.

Robustness to sub-sampling: PCNN (Ours) maintains significantly higher accuracy than PointNet and PointNet++ as the ratio of missing data increases.

Experimental Results

The authors evaluated PCNN on several standard benchmarks, demonstrating state-of-the-art performance:

  • Classification: On ModelNet40, PCNN achieved 92.3% accuracy, outperforming PointNet (89.2%) and PointNet++ (90.7%) using only raw coordinates.
  • Segmentation: PCNN set a new benchmark for ShapeNet part segmentation with an 85.1 mIoU, surpassing both point-based and voxel-based competitors.
  • Efficiency: A "Small" version of the network with just 1.4M parameters achieved 92.2% accuracy, highlighting the parameter efficiency of the extension-restriction operators.

Limitations and Future Work

While highly effective, the current implementation requires calculating a translating tensor ($q$) to connect points to kernel translations, which can be computationally demanding for very large scenes. Future work may involve developing sparser implementations or learning optimal kernel positions through deformable convolutions to further enhance scalability.

Summary updated

Abstract

This paper presents Point Convolutional Neural Networks (PCNN): a novel framework for applying convolutional neural networks to point clouds. The framework consists of two operators: extension and restriction, mapping point cloud functions to volumetric functions and vise-versa. A point cloud convolution is defined by pull-back of the Euclidean volumetric convolution via an extension-restriction mechanism. The point cloud convolution is computationally efficient, invariant to the order of points in the point cloud, robust to different samplings and varying densities, and translation invariant, that is the same convolution kernel is used at all points. PCNN generalizes image CNNs and allows readily adapting their architectures to the point cloud setting. Evaluation of PCNN on three central point cloud learning benchmarks convincingly outperform competing point cloud learning methods, and the vast majority of methods working with more informative shape representations such as surfaces and/or normals.
BibTeX
@article{atzmon2018point,
  author = {Matan Atzmon and Haggai Maron and Yaron Lipman},
  title = {{Point Convolutional Neural Networks by Extension Operators}},
  journal = {ACM Trans. Graphics},
  year = {2018},
  url = {https://openreview.net/forum?id=3AJGuLmCW_x},
}
Figure