Abstract
Recurrent Slice Networks (RSNet) address the fundamental challenge of processing unordered and unstructured 3D point clouds for semantic segmentation. By introducing a novel "slice pooling" layer, RSNet transforms unordered 3D points into ordered sequences, enabling the use of bidirectional Recurrent Neural Networks (RNNs) to model local geometric dependencies efficiently. Evaluated on major benchmarks including S3DIS, ScanNet, and ShapeNet, RSNet achieves state-of-the-art accuracy while maintaining a significantly lower computational and memory footprint compared to existing voxel-based or point-based architectures.
Introduction
3D point clouds, the raw output of LiDAR and depth sensors, are difficult for deep learning models to process because they lack a structured grid. Previous approaches typically relied on voxelization—which suffers from high memory consumption and quantization artifacts—or point-based global features like PointNet, which often fail to capture critical local geometric context. While some methods attempt to model local dependencies using expensive neighbor-searching algorithms (e.g., ball queries), RSNet introduces a more efficient alternative that leverages the sequential modeling power of RNNs.
Methodology
The core innovation of RSNet is the Local Dependency Module, which converts the 3D segmentation task into a sequence modeling problem through three primary stages:
- Slice Pooling Layer: This layer projects features from unordered points onto an ordered sequence of feature vectors. It "slices" the 3D space along a specific axis (x, y, or z) into discrete bins. All points within a bin are aggregated via max-pooling to create a single feature vector for that slice.
- RNN Layers: Once the slices are ordered, bidirectional Gated Recurrent Units (GRUs) process the sequence. This allows the network to propagate information across the sequence, effectively capturing dependencies between neighboring slices.
- Slice Unpooling Layer: This stage reverses the pooling process, mapping the refined features from the RNN back to the original individual points within each slice.
To capture comprehensive 3D context, the RSNet architecture typically employs three parallel branches—one for each principal axis—and aggregates their outputs to produce the final semantic labels.
Experiments and Results
RSNet was validated on three major 3D datasets: S3DIS (indoor rooms), ScanNet (real-world indoor scans), and ShapeNet (part segmentation).
- Accuracy: On the S3DIS dataset, RSNet achieved a mean Intersection over Union (mIOU) of 51.93, significantly outperforming the original PointNet (41.09) and even 3D-CNN approaches (48.92). On ScanNet, it surpassed PointNet++ by over 5% in mIOU (39.35 vs 34.26).
- Efficiency: RSNet is highly optimized for performance, proving to be 3.1x faster than the multi-scale version of PointNet++ while consuming less GPU memory than the standard PointNet. The slice pooling operation has a time complexity of $O(n)$ relative to the number of points, making it significantly cheaper than traditional neighbor-searching algorithms.
Discussion and Conclusion
RSNet demonstrates that complex 3D convolutions and expensive neighbor searches are not strictly necessary for high-quality 3D segmentation. By projecting points into 1D ordered slices, the model effectively leverages mature RNN technology to capture local context.
However, the current implementation relies on axis-aligned slicing, which may be less optimal for objects with arbitrary orientations. Additionally, the slicing resolution presents a trade-off: finer resolutions preserve detail but create longer sequences that can challenge RNN stability. Future research may explore rotation-invariant slicing or the integration of attention mechanisms to further enhance the modeling of dependencies across slices.
Summary updated