Recurrent Slice Networks for 3D Segmentation on Point Clouds

Qiangui Huang, Weiyue Wang, Ulrich Neumann

Abstract

Recurrent Slice Networks (RSNet) address the fundamental challenge of processing unordered and unstructured 3D point clouds for semantic segmentation. By introducing a novel "slice pooling" layer, RSNet transforms unordered 3D points into ordered sequences, enabling the use of bidirectional Recurrent Neural Networks (RNNs) to model local geometric dependencies efficiently. Evaluated on major benchmarks including S3DIS, ScanNet, and ShapeNet, RSNet achieves state-of-the-art accuracy while maintaining a significantly lower computational and memory footprint compared to existing voxel-based or point-based architectures.

Introduction

3D point clouds, the raw output of LiDAR and depth sensors, are difficult for deep learning models to process because they lack a structured grid. Previous approaches typically relied on voxelization—which suffers from high memory consumption and quantization artifacts—or point-based global features like PointNet, which often fail to capture critical local geometric context. While some methods attempt to model local dependencies using expensive neighbor-searching algorithms (e.g., ball queries), RSNet introduces a more efficient alternative that leverages the sequential modeling power of RNNs.

Methodology

The core innovation of RSNet is the Local Dependency Module, which converts the 3D segmentation task into a sequence modeling problem through three primary stages:

  1. Slice Pooling Layer: This layer projects features from unordered points onto an ordered sequence of feature vectors. It "slices" the 3D space along a specific axis (x, y, or z) into discrete bins. All points within a bin are aggregated via max-pooling to create a single feature vector for that slice.
  2. RNN Layers: Once the slices are ordered, bidirectional Gated Recurrent Units (GRUs) process the sequence. This allows the network to propagate information across the sequence, effectively capturing dependencies between neighboring slices.
  3. Slice Unpooling Layer: This stage reverses the pooling process, mapping the refined features from the RNN back to the original individual points within each slice.
The slicing process transforms a 3D point cloud into an ordered sequence of feature vectors, allowing for efficient sequence modeling.

To capture comprehensive 3D context, the RSNet architecture typically employs three parallel branches—one for each principal axis—and aggregates their outputs to produce the final semantic labels.

The RSNet architecture overview, illustrating the flow from input feature extraction through parallel slice-based RNN modules to the final semantic segmentation.

Experiments and Results

RSNet was validated on three major 3D datasets: S3DIS (indoor rooms), ScanNet (real-world indoor scans), and ShapeNet (part segmentation).

  • Accuracy: On the S3DIS dataset, RSNet achieved a mean Intersection over Union (mIOU) of 51.93, significantly outperforming the original PointNet (41.09) and even 3D-CNN approaches (48.92). On ScanNet, it surpassed PointNet++ by over 5% in mIOU (39.35 vs 34.26).
  • Efficiency: RSNet is highly optimized for performance, proving to be 3.1x faster than the multi-scale version of PointNet++ while consuming less GPU memory than the standard PointNet. The slice pooling operation has a time complexity of $O(n)$ relative to the number of points, making it significantly cheaper than traditional neighbor-searching algorithms.

Discussion and Conclusion

RSNet demonstrates that complex 3D convolutions and expensive neighbor searches are not strictly necessary for high-quality 3D segmentation. By projecting points into 1D ordered slices, the model effectively leverages mature RNN technology to capture local context.

However, the current implementation relies on axis-aligned slicing, which may be less optimal for objects with arbitrary orientations. Additionally, the slicing resolution presents a trade-off: finer resolutions preserve detail but create longer sequences that can challenge RNN stability. Future research may explore rotation-invariant slicing or the integration of attention mechanisms to further enhance the modeling of dependencies across slices.

Summary updated

BibTeX
@inproceedings{huang2018recurrent,
  author = {Qiangui Huang and Weiyue Wang and Ulrich Neumann},
  title = {{Recurrent Slice Networks for 3D Segmentation on Point Clouds}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2018},
  doi = {10.1109/cvpr.2018.00278},
}
Figure