Cascaded Non-local Neural Network for Point Cloud Semantic Segmentation

Mingmei Cheng, Le Hui, Jin Xie, Jian Yang, Hui Kong

TL;DR

This paper introduces a Cascaded Non-local Neural Network (PointNL) for 3D point cloud semantic segmentation. By breaking down the computationally heavy non-local attention mechanism into three hierarchical levels (neighborhood, superpoint, and global), the model efficiently captures long-range dependencies in 3D space. This approach achieves state-of-the-art accuracy across multiple indoor and outdoor datasets while drastically reducing the memory footprint and training time compared to standard non-local operations.

What Problem Does This Paper Address?

Semantic segmentation of 3D point clouds is challenging due to their unordered and irregular structure. While popular architectures like PointNet++ and DGCNN are excellent at extracting local geometric features, they struggle to capture long-range dependencies—the contextual relationships between distant parts of a scene (e.g., recognizing that a chair on one side of a room implies the existence of a floor connecting to a table on the other side).

In 2D image processing, "non-local" operations (a form of self-attention) solve this by computing interactions between all pixels. However, applying this directly to 3D point clouds is computationally prohibitive. It carries an $O(N^2)$ complexity, requiring massive GPU memory and training time that makes it infeasible for large-scale point clouds. The authors aim to bring the power of non-local global context to point clouds without the crippling computational cost.

Methods

The authors build an encoder-decoder architecture based on the PointNet++ framework, replacing standard local feature extractors with a novel Cascaded Non-local Module. Instead of computing attention across all points simultaneously, the module propagates context hierarchically across three distinct scales:

  1. Neighborhood Level: The network uses Farthest Point Sampling (FPS) to select centroid points. It then applies a non-local operation to extract features by assigning learned weights to the $K$-nearest neighbors of each centroid.
  2. Superpoint Level: The point cloud is partitioned into "superpoints" (regions with isotropous/similar geometric characteristics). The network samples a small number of centroids within each superpoint and applies attention to propagate features locally within these geometric boundaries.
  3. Global Level: The network extracts a single feature vector for each superpoint using max pooling. A final non-local operation is then applied across all superpoints to capture scene-wide context.

Additionally, instead of using a simple scalar value to measure similarity between points (as is standard in image-based non-local blocks), the authors use a channel-wise vector to embed the differences between point features. This allows the network to capture 3D geometric structures much more accurately.

Datasets

The model was trained and evaluated on a mix of standard indoor and outdoor benchmarks:

  • S3DIS: An indoor 3D point cloud dataset featuring six large-scale areas annotated with 13 semantic categories.
  • ScanNet: A richly annotated indoor scene dataset containing 1,513 point clouds across 20 categories.
  • vKITTI: A synthetic outdoor dataset mimicking the real-world KITTI dataset, containing 5 sequences and 13 classes.
  • SemanticKITTI: A massive, real-world outdoor LiDAR dataset containing over 43,000 scans annotated with 19 classes.

Key Insights and Ideas

  • Hierarchical attention is highly efficient: By restricting attention to local neighborhoods, then superpoints, and finally a small set of global superpoint representations, the computational complexity drops from $O(N^2)$ to a manageable linear combination of smaller sets.
  • Superpoints act as natural geometric bridges: Using superpoints as an intermediate level allows the network to propagate context smoothly from local patches to the entire scene without losing structural integrity.
  • Vector-based similarity beats scalar similarity: In 3D space, the relationship between two points is complex. Using a channel-wise vector to compute attention weights captures directional and structural nuances better than a single scalar similarity score.

Key Results

  • Accuracy: The model (PointNL) achieved state-of-the-art or highly competitive results across the board. It hit 68.4% mIoU on S3DIS (6-fold cross-validation), 86.7% Overall Accuracy on ScanNet, 57.6% mIoU on vKITTI, and 52.2% mIoU on the massive SemanticKITTI dataset (outperforming DarkNet53Seg's 49.9%).
  • Computational Speedup: The efficiency gains are massive. On S3DIS Area 5, a standard non-local baseline took 6 days to train and required 60 GB of GPU memory. The proposed cascaded PointNL model took only 1.8 days and used just 8 GB of GPU memory.
  • Ablation Success: Adding each level of the cascade progressively improved performance. On S3DIS Area 5, the neighborhood-only model hit 60.03% mIoU, adding superpoints raised it to 62.18%, and the full 3-level cascade peaked at 63.50%.

Limitations and Future Work

Not explicitly discussed in the paper. However, it is apparent from the methodology that the network relies on a pre-processing step to partition the point clouds into superpoints (following prior work like Superpoint Graphs). This adds an external dependency to the pipeline, meaning the network is not entirely end-to-end from raw points to segmentation without this partitioning step. The authors do not explicitly outline future research directions in the provided text.

Summary updated

Abstract

In this paper, we propose a cascaded non-local neural network for point cloud segmentation. The proposed network aims to build the long-range dependencies of point clouds for the accurate segmentation. Specifically, we develop a novel cascaded non-local module, which consists of the neighborhood-level, superpoint-level and global-level non-local blocks. First, in the neighborhood-level block, we extract the local features of the centroid points of point clouds by assigning different weights to the neighboring points. The extracted local features of the centroid points are then used to encode the superpoint-level block with the non-local operation. Finally, the global-level block aggregates the non-local features of the superpoints for semantic segmentation in an encoder-decoder framework. Benefiting from the cascaded structure, geometric structure information of different neighborhoods with the same label can be propagated. In addition, the cascaded structure can largely reduce the computational cost of the original non-local operation on point clouds. Experiments on different indoor and outdoor datasets show that our method achieves state-of-the-art performance and effectively reduces the time consumption and memory occupation.
BibTeX
@inproceedings{cheng2020cascaded,
  author = {Mingmei Cheng and Le Hui and Jin Xie and Jian Yang and Hui Kong},
  title = {{Cascaded Non-local Neural Network for Point Cloud Semantic Segmentation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  doi = {10.1109/iros45743.2020.9341531},
}

Loading related content...

Figure