X-3D: Explicit 3D Structure Modeling for Point Cloud Recognition

Shuofeng Sun, Yongming Rao, Jiwen Lu, Haibin Yan

Abstract

Point cloud recognition models typically map local point-to-point relations into high-dimensional spaces to implicitly learn geometry, a paradigm known as Implicit High-dimensional Structure Modeling (IHSM). However, this approach often loses the true structural shape of the data by treating neighborhoods as collections of isolated vectors. This paper introduces X-3D, a plug-and-play module that explicitly models 3D local structures in the original input space to generate shared dynamic kernels. By providing a strong geometric prior, X-3D achieves state-of-the-art performance across classification, segmentation, and detection tasks with minimal computational overhead.

Introduction

Point clouds are inherently irregular and unordered, making the application of standard grid-based convolutions challenging. Modern point-based networks, such as PointNet++ or Point Transformers, typically address this by constructing local neighborhoods and calculating "relation vectors"—such as the relative distance between a center point and its neighbors. These vectors are then embedded into high-dimensional spaces using dynamic kernels to capture local structure implicitly.

The authors argue that this IHSM paradigm creates a gap between learned embeddings and actual physical geometry. By forcing the network to learn geometry from scratch in a latent space, critical detailed information is often lost. X-3D is proposed to bridge this gap by focusing on the overall cohesive 3D shape of a neighborhood rather than individual point-to-point relations.

Methodology: Explicit 3D Structure Modeling

X-3D shifts the focus from individual vectors to the collective structure of the neighborhood through four key components:

1. Explicit Structure Construction

Instead of relying on high-dimensional embeddings, X-3D directly captures local geometry in the original 3D input space. Inspired by PointHop, the module divides a local neighborhood into eight octants and calculates the centroid of the points within each partition. This process creates a robust, symmetric descriptor of the local shape that serves as a grounded geometric prior.

2. Structure Kernel Generator

In many Transformer-based architectures, a unique dynamic kernel is generated for every single point-to-point vector, which is computationally expensive. X-3D optimizes this by using the explicit structure descriptor to generate a single, shared dynamic kernel for the entire neighborhood via a Multi-Layer Perceptron (MLP). This approach is not only more efficient but also ensures the kernel is informed by the global shape of the neighborhood.

3. Denoising via Cross-Attention

To ensure the geometric representation is not distorted by outliers, the model employs a cross-attention mechanism. By calculating the relationship between neighborhood points and the explicit local structure, the model can identify and filter out noisy points before the kernel generation phase.

4. Neighborhood Context Propagation

To mitigate the randomness inherent in kNN or ball-query sampling, X-3D fuses features from overlapping neighborhoods. To prevent the introduction of conflicting geometric information, this propagation is restricted to local neighborhoods that exhibit similar geometric properties, ensuring consistency across the manifold.

Experimental Results

The authors evaluated X-3D by embedding it into several baseline architectures, including PointNet++, PointMetaBase, DeLA, and GroupFree3D. The module consistently delivered state-of-the-art results across various benchmarks:

  • Classification: On the ScanObjectNN dataset (real-world scanned objects), X-3D achieved 90.7% Overall Accuracy (OA). This set a new SOTA, providing a +0.3% boost over the DeLA baseline while adding only 0.03 GFLOPs and 0.1M parameters.
  • Scene Segmentation: The model reached 79.2% mIoU on S3DIS (6-fold) and 76.3% mIoU on ScanNetV2. Notably, adding X-3D to the original PointNet++ architecture increased its S3DIS Area 5 mIoU by 4.3%.
  • 3D Object Detection: When integrated with GroupFree3D on ScanNetV2, X-3D boosted performance by 1.7% [email protected] (reaching 69.0%) and 2.2% [email protected] (reaching 51.1%).

Discussion and Insights

Geometric Priors and Learning Efficiency

By explicitly modeling 3D structure in the input space, X-3D acts as a powerful geometric prior. This restricts the search space for kernel parameters to physically grounded shapes, making the learning process significantly easier than forcing the network to derive geometry from high-dimensional latent vectors.

Computational Efficiency

X-3D is notably more efficient than per-vector kernel methods. For example, compared to standard Point Transformers, X-3D uses significantly fewer GFLOPs (0.24 vs 1.51 in standard configurations) because it computes one shared kernel per neighborhood rather than one per point pair.

Manifold Learning Connection

The authors draw a parallel between X-3D and Locally Linear Embedding (LLE) in manifold learning. Just as LLE preserves manifold surfaces by minimizing local structural differences in the input space, X-3D preserves the non-Euclidean manifold of point clouds more effectively than standard Euclidean relation vectors.

Conclusion

X-3D demonstrates that explicit geometric modeling in the original 3D space is a highly effective alternative to implicit high-dimensional modeling. As a plug-and-play module, it provides a significant performance boost across diverse 3D tasks with negligible computational cost. Future work may explore adaptive partitioning methods to further refine the representation of highly irregular geometries.

Summary updated

Abstract

Numerous prior studies predominantly emphasize constructing relation vectors for individual neighborhood points and generating dynamic kernels for each vector and embedding these into high-dimensional spaces to capture implicit local structures. However, we contend that such implicit high-dimensional structure modeling approch inadequately represents the local geometric structure of point clouds due to the absence of explicit structural information. Hence, we introduce X-3D, an explicit 3D structure modeling approach. X-3D functions by capturing the explicit local structural information within the input 3D space and employing it to produce dynamic kernels with shared weights for all neighborhood points within the current local region. This modeling approach introduces effective geometric prior and significantly diminishes the disparity between the local structure of the embedding space and the original input point cloud, thereby improving the extraction of local features. Experiments show that our method can be used on a variety of methods and achieves state-of-the-art performance on segmentation, classification, detection tasks with lower extra computational cost, such as \textbf{90.7\%} on ScanObjectNN for classification, \textbf{79.2\%} on S3DIS 6 fold and \textbf{74.3\%} on S3DIS Area 5 for segmentation, \textbf{76.3\%} on ScanNetV2 for segmentation and \textbf{64.5\%} mAP , \textbf{46.9\%} mAP on SUN RGB-D and \textbf{69.0\%} mAP , \textbf{51.1\%} mAP on ScanNetV2 . Our code is available at \href{https://github.com/sunshuofeng/X-3D}{https://github.com/sunshuofeng/X-3D}.
BibTeX
@inproceedings{sun2024x,
  author = {Shuofeng Sun and Yongming Rao and Jiwen Lu and Haibin Yan},
  title = {{X-3D: Explicit 3D Structure Modeling for Point Cloud Recognition}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}
Figure