In traditional deep learning, data usually comes with a built-in map. Images are grids of pixels where "up" and "left" have fixed meanings. Text is a sequence where the order of words defines the message. But 3D point clouds—the raw data from LiDAR sensors or depth cameras—are fundamentally different. They are unordered sets.
If you have a list of 1,000 coordinates representing a chair, you can shuffle that list in any of the $1,000!$ possible permutations, and the physical chair remains identical. This property, known as permutation invariance, creates a unique challenge: how do we design neural networks that see the "shape" rather than the "list"?
The Symmetry Solution
The breakthrough in processing raw point sets came with PointNet, developed by Charles R. Qi, Leonidas Guibas, and colleagues. Their intuition was simple: to make a model's output independent of input order, use a symmetric function.
A symmetric function, like addition or finding the maximum, yields the same result regardless of the order of its arguments. PointNet processes each point independently through shared layers and then applies a global max-pooling step. This "squeezes" the unordered set into a single global feature vector that represents the skeleton of the shape, effectively bypassing the need for expensive voxelization or rendering.
Taming the Sequence
While PointNet solved basic classification, it struggled with generation. Modern generative models, like Transformers and Autoregressive (AR) models, are inherently sequential—they predict the "next" item in a series. But in a point cloud, there is no "next" point.
To bridge this gap, researchers have explored two main strategies:
- Canonical Mapping: An-Chieh Cheng and Ming-Hsuan Yang proposed "taming" the transformer by mapping point clouds onto a shared canonical sphere. By projecting the 3D data into a structured, semantically aligned sequence, they allow the transformer to learn a consistent "reading order" for shapes.
- Level-of-Detail Prediction: Instead of predicting point-by-point, Ziqiao Meng and Irwin King introduced PointNSP, which predicts "next-scale" levels of detail. This approach maintains permutation invariance by treating the generation as a refinement of global structure rather than a 1D sequence of coordinates.
Modeling Distributions Instead of Points
Another way to ignore the order of points is to stop treating them as a list and start treating them as a probability distribution.
In PointFlow, Guandao Yang and Xun Huang used Continuous Normalizing Flows (CNFs) to model 3D shapes. Instead of generating specific points, the model learns a transformation that turns a simple 3D Gaussian "cloud" into the shape of an airplane or a car. Because the model learns the underlying continuous density, it doesn't matter which point is sampled first; the permutation invariance is mathematically guaranteed.
Marin Biloš further scaled this idea by designing architectures that yield exact likelihoods in linear time, making these "distribution of distributions" models practical for large-scale datasets.
Scaling with Equivariance
As we move toward processing millions of points (like in protein folding or large-scale scene flow), the quadratic complexity of standard Transformers becomes a bottleneck. Researchers are now looking toward State Space Models (SSMs) and long-convolutional operators.
However, these models are even more sensitive to order than Transformers. To fix this, Yiming Wu and Dong Xu introduced PoinTramba, which uses a "Bi-directional Importance-aware Ordering" (BIO) to force a meaningful sequence onto the points before they enter a Mamba-based architecture.
Meanwhile, Artem Moskalev and Rui Liao have pushed the boundaries of SE(3) Equivariance. Their work on Geometric Hyena and SE(3)-Hyena replaces attention with efficient vector-valued convolutions. These models don't just ignore the order of points (invariance); they ensure that if the entire point cloud rotates or shifts, the model's internal features rotate and shift exactly in sync (equivariance), all while maintaining sub-quadratic scaling.
Why It Works
The core intuition across all these papers is that 3D geometry is defined by relationships, not indices. Whether through symmetric pooling (PointNet), learned paths (PMP-Net), or continuous flows (PointFlow), the goal is to force the neural network to look at the global context and local neighborhoods. By stripping away the arbitrary 1D ordering of the data, these models are finally able to "see" the 3D world as it actually exists.