Cascaded Non-local Neural Network for Point Cloud Semantic Segmentation

Mingmei Cheng, Le Hui, Jin Xie, Jian Yang, Hui Kong

Abstract

In this paper, we propose a cascaded non-local neural network for point cloud segmentation. The proposed network aims to build the long-range dependencies of point clouds for the accurate segmentation. Specifically, we develop a novel cascaded non-local module, which consists of the neighborhood-level, superpoint-level and global-level non-local blocks. First, in the neighborhood-level block, we extract the local features of the centroid points of point clouds by assigning different weights to the neighboring points. The extracted local features of the centroid points are then used to encode the superpoint-level block with the non-local operation. Finally, the global-level block aggregates the non-local features of the superpoints for semantic segmentation in an encoder-decoder framework. Benefiting from the cascaded structure, geometric structure information of different neighborhoods with the same label can be propagated. In addition, the cascaded structure can largely reduce the computational cost of the original non-local operation on point clouds. Experiments on different indoor and outdoor datasets show that our method achieves state-of-the-art performance and effectively reduces the time consumption and memory occupation.

Introduction

Semantic segmentation of 3D point clouds is a fundamental task in computer vision, yet it remains challenging due to the unordered and irregular nature of the data. While existing architectures like PointNet++ or DGCNN excel at capturing local geometric features, they often fail to model long-range dependencies—the contextual relationships between distant objects in a scene.

In 2D image processing, non-local operations (a form of self-attention) are used to capture these dependencies. However, applying them directly to 3D point clouds is computationally prohibitive, with a complexity of $O(N^2)$. For large-scale scenes, this leads to excessive GPU memory requirements and long training times. This paper introduces PointNL, a Cascaded Non-local Neural Network designed to capture global context efficiently by decomposing the non-local operation into a hierarchical structure.

Methodology: Cascaded Non-local Module

The core of PointNL is the Cascaded Non-local Module, integrated into an encoder-decoder framework. Instead of a single global attention pass, the module propagates context across three hierarchical levels:

  1. Neighborhood Level: Centroid points are selected via Farthest Point Sampling (FPS). A non-local operation is applied to the $K$-nearest neighbors of each centroid to extract refined local features.
  2. Superpoint Level: The point cloud is partitioned into superpoints—regions with similar geometric characteristics. Attention is applied to centroids within these superpoints to propagate features within geometric boundaries.
  3. Global Level: Max pooling extracts a single feature vector per superpoint. A final non-local operation is performed across these superpoint representations to capture scene-wide context.
The PointNL architecture (a) and the detailed structure of the Cascaded Non-local Module (b).

Unlike standard non-local blocks that use scalar similarity, PointNL utilizes a channel-wise vector to embed differences between point features, allowing for a more nuanced capture of 3D geometric structures.

Experimental Results

The model was evaluated on major benchmarks including S3DIS, ScanNet, vKITTI, and SemanticKITTI. PointNL achieved state-of-the-art results, such as 68.4% mIoU on S3DIS and 86.7% Overall Accuracy on ScanNet.

Beyond accuracy, the primary advantage of the cascaded approach is its efficiency. As shown in the ablation study, the full three-level cascade (PointNL$^{123}$) not only provides the highest accuracy but also maintains a significantly lower computational footprint compared to a standard non-local baseline.

Ablation study on S3DIS Area 5 comparing accuracy, training time, and GPU memory usage across different cascade configurations.

On the S3DIS dataset, the standard non-local baseline required 60 GB of GPU memory and 6 days of training. In contrast, the full PointNL model required only 8 GB of memory and 1.8 days of training, representing a massive improvement in practical usability.

Conclusion

PointNL demonstrates that hierarchical attention is a powerful and efficient way to model long-range dependencies in 3D point clouds. By using superpoints as geometric bridges and a vector-based similarity measure, the network achieves high accuracy while drastically reducing computational costs. While the current pipeline relies on a pre-processing step for superpoint partitioning, the efficiency gains make it a compelling architecture for large-scale 3D scene understanding.

Summary updated

Abstract

In this paper, we propose a cascaded non-local neural network for point cloud segmentation. The proposed network aims to build the long-range dependencies of point clouds for the accurate segmentation. Specifically, we develop a novel cascaded non-local module, which consists of the neighborhood-level, superpoint-level and global-level non-local blocks. First, in the neighborhood-level block, we extract the local features of the centroid points of point clouds by assigning different weights to the neighboring points. The extracted local features of the centroid points are then used to encode the superpoint-level block with the non-local operation. Finally, the global-level block aggregates the non-local features of the superpoints for semantic segmentation in an encoder-decoder framework. Benefiting from the cascaded structure, geometric structure information of different neighborhoods with the same label can be propagated. In addition, the cascaded structure can largely reduce the computational cost of the original non-local operation on point clouds. Experiments on different indoor and outdoor datasets show that our method achieves state-of-the-art performance and effectively reduces the time consumption and memory occupation.
BibTeX
@inproceedings{cheng2020cascaded,
  author = {Mingmei Cheng and Le Hui and Jin Xie and Jian Yang and Hui Kong},
  title = {{Cascaded Non-local Neural Network for Point Cloud Semantic Segmentation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  doi = {10.1109/iros45743.2020.9341531},
}
Figure