Standard convolutional neural networks (CNNs) are built on the assumption of a regular grid. In an image, a pixel’s neighbors are always in the same relative positions: one up, one down, one left, and one right. This structure allows us to use fixed-weight kernels that slide across the data, sharing parameters and capturing spatial patterns efficiently.
3D point clouds break this assumption. They are unordered, sparse, and irregular. A point in a cloud doesn't have a "top-left" neighbor; it has a collection of neighbors at arbitrary distances and angles. To process this data, researchers have traditionally had to choose between two compromises: voxelizing the data into a rigid 3D grid (which wastes memory on empty space) or designing complex "point-based" operators that struggle to match the efficiency and weight-sharing power of standard CNNs.
A growing body of research proposes a third way: instead of changing the convolution to fit the points, we can learn to project the points into a structured space where standard convolutions can thrive.
Interpolating onto Discrete Kernels
The most direct way to bridge the gap between irregular points and regular grids is through interpolation. In Interpolated Convolutional Networks for 3D Point Cloud Understanding, Jiageng Mao and colleagues introduce InterpConv.
The intuition here is to treat the convolutional kernel not as a continuous function, but as a set of discrete weights located at fixed coordinates—much like a standard 3x3 image filter. When a point cloud falls near these weights, its features are distributed (interpolated) onto the nearest grid points of the kernel. By using functions like trilinear or Gaussian interpolation, the model maps the "influence" of an irregular point onto a fixed-coordinate system. This allows the network to use standard convolutional optimization techniques while remaining robust to the varying densities of raw point clouds.
Learning a Spatial Dictionary
While interpolation uses fixed geometric rules to map points to grids, another approach is to learn the coordinate system itself. In Spatial Transformer Point Convolution, Chunyan Xu and Zhen Cui propose the Spatial Transformer Point Convolution (STPC).
STPC addresses the "unorder" problem by learning a "spatial direction dictionary." Think of this dictionary as a set of learned basis vectors that define a canonical coordinate system for a local neighborhood. By calculating how well each neighbor point aligns with these learned directions, the method projects irregular neighbors into a structured, ordered space. Once the points are "sorted" into this dictionary-defined order, the model can apply weight-sharing kernels just like an image-based CNN. This makes the convolution "anisotropic" (direction-aware) without requiring the points to actually lie on a grid.
The Projection-Convolution-Resampling Cycle
If we can map local neighborhoods to grids, why not map the entire scene? Cloud Transformers, developed by Kirill Mazur and Victor S. Lempitsky, generalizes this idea into a universal building block.
The process follows a three-step cycle:
- Project: Unordered point features are mapped onto a regular grid using learnable coordinate projections.
- Process: The resulting grid is processed using standard, highly optimized 2D or 3D convolutions. This allows the model to benefit from the large receptive fields and architectural innovations of the broader computer vision field.
- Sample: The processed features are sampled back from the grid to the original point locations.
This "sandwich" architecture allows the network to maintain the high precision of raw point coordinates while leveraging the structural efficiency of grid-based operators. By using a "gradient balancing" technique, the authors ensure that the learnable projections don't collapse, allowing the network to discover the most efficient way to flatten or voxelize the 3D data for processing.
Why This Works
The success of these methods stems from a simple realization: the "grid" in a CNN is not just a data format; it is a mechanism for weight sharing. By projecting irregular points into structured spaces—whether through interpolation, learned dictionaries, or grid-sampling—these architectures allow a single set of weights to learn patterns that apply across the entire point cloud. This reduces the number of parameters needed and allows the models to generalize better than methods that treat every point as an isolated entity.