Abstract
PS$^2$-Net is a novel deep learning framework designed for 3D point cloud semantic segmentation that simultaneously captures fine-grained local structures and broad global context. By stacking specialized encoders that combine EdgeConv and NetVLAD layers, the network achieves state-of-the-art performance on major benchmarks while mathematically guaranteeing permutation invariance—ensuring the model's output remains consistent regardless of the order in which points are processed.
Introduction
Semantic segmentation in 3D involves assigning a category label (such as "chair," "wall," or "floor") to every individual point in a 3D scan. Unlike 2D images, which are structured grids of pixels, 3D point clouds are unordered and irregular. A fundamental requirement for any neural network processing such data is permutation invariance: the physical scene remains identical regardless of the order in which points are listed in a data file, and the model's output must reflect this consistency.
Early architectures like PointNet achieved permutation invariance but struggled to capture local geometric relationships between neighboring points. Subsequent models improved local feature extraction but often at the expense of global context or mathematical rigor (e.g., using RNNs that break permutation invariance). PS$^2$-Net bridges this gap by being both locally and globally aware without sacrificing mathematical rigor.
Methods
The core of the architecture is the PS$^2$-Net encoder, which is stacked four times to create a deep feature-learning pipeline. Each encoder integrates two primary components to balance local and global information:
- EdgeConv (Local Awareness): Instead of processing points in isolation, EdgeConv constructs a local graph by identifying a point’s $K$-nearest neighbors. It computes the relationships (edges) between these points, enabling the network to discern geometric features such as corners, edges, and surface curvatures.
- NetVLAD (Global Awareness): Adapted from image retrieval tasks, NetVLAD aggregates features from the entire 3D scene into a single global descriptor. This global context is vital for resolving local ambiguities—for instance, distinguishing a white wall from a white whiteboard by analyzing the surrounding room context.
The authors provide a formal theoretical proof demonstrating that this specific combination of layers maintains permutation invariance throughout the entire network.
Datasets and Evaluation
The performance of PS$^2$-Net was evaluated on two large-scale indoor 3D datasets:
- S3DIS (Stanford Large-Scale 3D Indoor Spaces): Comprising 271 rooms across six areas with 13 semantic categories.
- ScanNet: A massive dataset of over 1,500 scans from 707 unique indoor scenes, featuring 21 categories.
To ensure fair comparisons, the authors utilized multiple data-preprocessing pipelines established by previous state-of-the-art (SOTA) works.
Key Results and Insights
PS$^2$-Net achieved state-of-the-art results across both major benchmarks. On the S3DIS dataset, the model reached a Mean Intersection over Union (mIoU) of 66.60%, outperforming PointCNN (65.39%) and showing a significant 9.7% improvement over DGCNN, which lacks a global context component. On ScanNet, it achieved an overall accuracy of 87.21% and an mIoU of 44.90%, setting a new high bar for point-based methods.
Ablation studies confirmed the necessity of both modules. Removing the local feature module caused the mIoU to drop from approximately 53% to 45%, proving that local geometric awareness is essential for accurate 3D understanding. Furthermore, the model demonstrated robustness to class imbalance, performing remarkably well on rare classes like bathtubs and shower curtains without specialized weighting techniques.
Conclusion
The success of PS$^2$-Net highlights the importance of hierarchical representation and scene-level context in 3D point cloud analysis. While stacking encoders improves performance, the researchers noted diminishing returns after four layers, likely due to parameter overhead relative to dataset size. Future work may focus on optimizing the computational cost of $K$-nearest neighbor (KNN) searches for real-time applications on massive point clouds.
Summary updated