To process 3D data, early deep learning models typically forced irregular spatial data into regular structures. They either projected 3D points into 2D multi-view images or discretized them into dense 3D voxel grids. While this allowed the use of standard 2D or 3D convolutions, it introduced severe trade-offs: 2D projections flatten depth and distort spatial relationships, while voxelization suffers from cubic memory scaling ($O(N^3)$) and quantization errors that blur fine-grained geometric details.
Direct point cloud processing avoids these intermediaries by operating directly on raw, unorganized $(x, y, z)$ coordinates. As pioneered by Qi et al. in PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation, treating the data as an unordered set allows models to preserve the native spatial resolution and inherent sparsity of the scene.
Why Point-Based Representations Preserve Geometry
When a model voxelizes a point cloud, it collapses continuous coordinates into fixed grid cells. If two distinct geometric features fall into the same voxel, their unique identities are lost. Li et al. demonstrate in NoKSR: Kernel-Free Neural Surface Reconstruction via Point Cloud Serialization that maintaining data in the point domain avoids these quantization errors entirely, leading to significantly higher surface reconstruction accuracy.
This preservation of sub-pixel geometric detail is crucial for thin or complex structures. For instance, Srivastava shows in Drought Stress Classification Using 3D Plant Models that direct point processing captures narrow, spindly biological features (like plant leaves) where voxel grids fail due to resolution limits. Bypassing fixed-grid boundaries prevents spatial blurring, allowing for high-fidelity local completion of intricate shapes.
Furthermore, analyzing raw coordinates maintains geometric fidelity on inclined or irregular surfaces. As noted by Williams et al. in 3D Segmentation of Trees Through a Flexible Multiclass Graph Cut Algorithm, processing raw 3D coordinates directly avoids the systematic errors and structural distortions that occur when spatial data is normalized or flattened against a ground model.
Overcoming Discretization and Scaling Pitfalls
How do networks process unordered sets without losing spatial context? The core mechanism relies on learning permutation-invariant functions. By applying shared multi-layer perceptrons (MLPs) to individual points and aggregating features globally via symmetric functions (such as max-pooling), networks extract both local and global geometric properties without requiring a rigid grid.
Maintaining raw coordinates also preserves the inherent sparsity of 3D data. Instead of wasting computation on empty space inside a volumetric grid, architectures like those proposed by Qi et al. in Frustum PointNets for 3D Object Detection From RGB-D Data focus parameters exclusively where data exists. This grid-free nature makes the approach highly resilient to sparse or noisy sensor data, as it does not enforce an arbitrary spatial resolution.
Even when compressing data into a latent space, keeping a point-cloud structure preserves spatial inductive biases. As shown by Zeng et al. in LION: Latent Point Diffusion Models for 3D Shape Generation, point-based latent spaces allow generative models to better capture local geometric details and spatial relationships during denoising compared to flattened vector representations.
Raw Point Cloud [(x,y,z), ...] ───► Permutation-Invariant Layer (Shared MLPs) ───► Max Pooling ───► Global Feature
│
Voxelization (Alternative) ───► Fixed 3D Grid (Quantization Error & Blurring) ─────────────────────┘
Eliminating the 2D Bottleneck in Multimodal Models
Direct point processing has fundamentally changed how 3D data interacts with other modalities like language and images. Traditional multimodal models often project 3D scenes into 2D images before aligning them with text. However, this intermediary step discards critical spatial context and depth information.
By mapping 3D point clouds directly to language embeddings, models can perform complex spatial reasoning without information loss. For example, Peng et al. in UniM-OV3D: Uni-Modality Open-Vocabulary 3D Scene Understanding with Fine-Grained Feature Representation show that direct 3D-to-text alignment preserves relative positioning and object context that are flattened or distorted in 2D projections.
Similarly, Qi et al. demonstrate in ShapeLLM: Universal 3D Object Understanding for Embodied Interaction that feeding raw point clouds directly into multimodal large language models significantly reduces hallucinations. This direct pipeline ensures that the geometric precision and depth information essential for physical interaction and complex spatial reasoning remain intact.
Key Foundational Work
3D face recognition by projection-based methods (2006) — contributed 2 concept elements to this direction.