Point convolutional neural networks by extension operators

Matan Atzmon, Haggai Maron, Yaron Lipman

This paper, published by researchers at the Weizmann Institute of Science, introduces a mathematically elegant way to bring the power of Convolutional Neural Networks (CNNs) to the messy, irregular world of 3D point clouds.

TL;DR

The authors propose Point Convolutional Neural Networks (PCNN), a framework that applies standard volumetric convolutions to point clouds by treating them as continuous functions rather than discrete sets. By using "extension" and "restriction" operators, they achieve a translation-invariant, permutation-invariant, and sampling-robust convolution that outperforms previous state-of-the-art methods like PointNet++.

What Problem Does This Paper Address?

Deep learning has revolutionized 2D image analysis because images are regular grids of pixels. 3D data, however, is often captured as point clouds—unorganized collections of $(x, y, z)$ coordinates.

Applying convolutions to point clouds is hard for three main reasons:

  1. Lack of Order: A point cloud is a set; changing the order of points in the data file shouldn't change the output (Permutation Invariance).
  2. Irregular Density: Some areas of a scan might have thousands of points while others have ten, yet they represent the same surface.
  3. Translation Invariance: A "corner" detector should work the same way whether the corner is at the origin or ten meters away.

Previous solutions either "voxelized" the data (turning it into a low-res 3D grid, which is memory-intensive and blurry) or used symmetric functions like Max-Pooling (which can lose fine-grained structural detail).

Methods

The core of PCNN is a three-step loop called the Extension-Convolution-Restriction mechanism:

  1. Extension ($E_X$): They take the discrete values defined at each point and "spread" them into the surrounding 3D space using Radial Basis Functions (RBFs)—specifically Gaussians. This turns the point cloud into a continuous volumetric function.
  2. Convolution ($O$): They apply a continuous volumetric convolution to this "spread out" function. Because they use Gaussians for both the extension and the convolution kernels, they can use a "multiplication law for Gaussians" to solve the convolution integral analytically (in closed form). This avoids the need for a discrete 3D grid.
  3. Restriction ($R_X$): Finally, they sample the resulting continuous function back at the original point locations.

By performing the math in a continuous space but only calculating values at the point locations, they get the mathematical benefits of volumetric CNNs without the massive memory overhead of 3D grids.

Datasets

The authors evaluated PCNN on three standard 3D deep learning benchmarks:

  • ModelNet40 & ModelNet10: For 3D object classification (e.g., identifying if a point cloud is a chair or a plane).
  • ShapeNet: For part segmentation (e.g., labeling which points belong to the "wing" vs. the "body" of a plane).
  • Normal Estimation: Predicting the surface orientation at each point on ModelNet40 shapes.

Key Insights and Ideas

  • The "Constant 1" Trick: One of the most striking findings is that if you apply the extension operator to a point cloud where every point has a value of "1," the resulting volumetric function approximates the indicator function of the underlying surface. Its gradient even approximates the mean curvature. This means the network can "see" the shape's geometry even before it looks at any features.
  • Closed-Form Math: By choosing Gaussians, the authors turned a potentially impossible 3D integration problem into a simple tensor multiplication. This makes the "continuous" convolution computationally feasible.
  • Inherent Robustness: Because the extension operator effectively "reconstructs" a surface, the method is naturally robust to changes in point density. If you remove half the points, the Gaussian "blur" still covers roughly the same volume, keeping the convolution stable.

Key Results

  • Classification: On ModelNet40, PCNN achieved 92.3% accuracy, outperforming PointNet (89.2%) and PointNet++ (90.7%) using only point coordinates as input.
  • Segmentation: PCNN set a new state-of-the-art on the ShapeNet part segmentation task (85.1 mIoU), beating both point-based and voxel-based competitors.
  • Efficiency: The authors demonstrated a "Small" version of their network with only 1.4M parameters that still achieved 92.2% accuracy, proving that their operators are highly parameter-efficient compared to traditional architectures.

Limitations and Future Work

  • Computational Overhead: While more efficient than dense voxels, the method still requires calculating a "translating" tensor ($q$) that connects points to kernel translations. The authors suggest that a sparser implementation is needed to scale to very large scenes.
  • Farthest Point Sampling: The architecture relies on Farthest Point Sampling (FPS) for its pooling layers, which can be a bottleneck for extremely large point clouds.
  • Learned Translations: Currently, the convolution kernels are placed on a regular 3D grid. The authors suggest that future work could involve "learning" the best positions for these kernels (similar to deformable convolutions).

Summary updated

Abstract

This paper presents Point Convolutional Neural Networks (PCNN): a novel framework for applying convolutional neural networks to point clouds. The framework consists of two operators: extension and restriction, mapping point cloud functions to volumetric functions and vise-versa. A point cloud convolution is defined by pull-back of the Euclidean volumetric convolution via an extension-restriction mechanism. The point cloud convolution is computationally efficient, invariant to the order of points in the point cloud, robust to different samplings and varying densities, and translation invariant, that is the same convolution kernel is used at all points. PCNN generalizes image CNNs and allows readily adapting their architectures to the point cloud setting. Evaluation of PCNN on three central point cloud learning benchmarks convincingly outperform competing point cloud learning methods, and the vast majority of methods working with more informative shape representations such as surfaces and/or normals.
BibTeX
@article{atzmon2018point,
  author = {Matan Atzmon and Haggai Maron and Yaron Lipman},
  title = {{Point convolutional neural networks by extension operators}},
  journal = {ACM Trans. Graphics},
  year = {2018},
  url = {https://openreview.net/forum?id=3AJGuLmCW_x},
}

Loading related content...

Figure