Global Context Reasoning for Semantic Segmentation of 3D Point Clouds

Yanni Ma, Yulan Guo, Hao Liu, Yinjie Lei, Gongjian WenWACV 2020 · OpenReview

TL;DR

This paper introduces PointGCR, a novel module for 3D point cloud semantic segmentation that captures global contextual information by modeling channel dependencies as a graph. By integrating this plug-and-play module into existing networks, the authors achieve significant performance improvements on both indoor and outdoor datasets, demonstrating its effectiveness in understanding complex scene relationships.

What Problem Does This Paper Address?

Semantic segmentation of 3D point clouds is crucial for applications like autonomous driving and robotics. However, existing methods often struggle to capture the global context of a scene. Many approaches rely on stacking feature extraction layers to increase the receptive field, which can be computationally expensive and may not fully capture long-range dependencies between different parts of the point cloud. This limitation leads to confusion between similar object categories (e.g., doors and walls, windows and boards) because the models lack a holistic understanding of how these elements relate to each other within the scene. The paper aims to address this by proposing a method to efficiently capture global contextual information, specifically focusing on the relationships between different feature channels.

Methods

The core contribution is the Point Global Context Reasoning (PointGCR) module. This module operates on the channel dimension of feature maps, unlike many prior methods that focus on spatial dimensions.

Here's a breakdown of the key techniques:

  • ChannelGraph Representation: The module constructs an undirected graph called "ChannelGraph." Each channel map in a feature representation is treated as a node in this graph. The dependencies between these channel maps are then represented as edges.
  • Node Attention (NA) Block: To embed feature channels as graph nodes, a node attention block is used. This block learns pairwise similarities between channels, effectively modeling their initial relationships. This is inspired by self-attention mechanisms.
  • Graph Convolution (GC) for Edge Learning: Graph Convolutional Networks (GCNs) are employed to learn the dependencies between these graph nodes (channels). This process captures the "edge weights" of the ChannelGraph, allowing the module to reason about the long-term contextual dependencies between different semantic responses.
  • Plug-and-Play Integration: PointGCR is designed to be a lightweight and easily integrable module. It can be appended to existing point cloud segmentation architectures without requiring significant modifications to the backbone network.
  • Residual Connection: The output of the PointGCR module is added back to the original feature map via a residual connection, allowing the network to benefit from both the original features and the enhanced contextual information.

Datasets

The paper evaluates its proposed method on three widely used 3D point cloud segmentation datasets:

  • ScanNet: A large-scale indoor dataset with 20 semantic categories.
  • S3DIS (Stanford Large-Scale 3D Indoor Spaces): An indoor dataset comprising 6 areas and 271 rooms, with 13 semantic categories.
  • Semantic3D: A large-scale outdoor dataset with 8 semantic categories.

The authors do not introduce any new benchmarks.

Key Insights and Ideas

  • Channel-wise Global Context is Crucial: The paper highlights that focusing on the relationships between feature channels (representing different semantic responses) is a powerful way to capture global context, complementing spatial context aggregation.
  • Graph Representation for Channel Dependencies: Using a graph to model channel interdependencies is an effective way to represent complex relationships that might be missed by purely spatial convolutions.
  • Attention and GCN Synergy: The combination of channel attention for node embedding and graph convolution for edge learning provides a robust mechanism for capturing long-range contextual reasoning.
  • Efficiency and Modularity: The plug-and-play nature of PointGCR is a significant advantage, allowing researchers to easily boost the performance of their existing segmentation models without extensive re-engineering.

Key Results

The PointGCR module consistently improved the performance of baseline networks across all tested datasets.

  • ScanNet: PointGCR integration led to improvements of 3.9% for PointNet++, 2.6% for PointConv, and 1.7% for PointSIFT in terms of mIoU.
  • S3DIS: The module achieved a 54.38% mIoU with PointNet++, an increase of 4.34%. Notably, it significantly improved performance on challenging categories like "bookcase" (19.81% IoU increase) and "column" (10.32% IoU increase).
  • Semantic3D: PointNet++-GCR outperformed its baseline by 4.7% in mIoU and 5.8% in overall accuracy (OA).

Qualitatively, visualizations showed that PointGCR-enhanced models produced smoother and more accurate segmentations, particularly for objects with complex shapes or those attached to walls, which were often misclassified by the baseline methods.

Limitations and Future Work

  • Computational Complexity: While designed to be lightweight, the paper acknowledges that GCNs can have relatively high computational complexity. The method addresses this by embedding original features into a smaller number of graph nodes.
  • Stacking GCN Layers: The ablation study suggests that simply stacking multiple GCN layers does not necessarily lead to further performance gains, indicating an optimal depth for graph reasoning.
  • Future Work: The paper implicitly suggests that exploring different graph structures or more advanced graph reasoning techniques could further enhance performance. The success across diverse indoor and outdoor datasets also points to the generalizability of the approach.

Summary updated

Abstract

Global contextual dependency is important for semantic segmentation of 3D point clouds. However, most existing approaches stack feature extraction layers to enlarge the receptive field to aggregate more contextual information of points along the spatial dimension. In this paper, we propose a Point Global Context Reasoning (PointGCR) module to capture global contextual information along the channel dimension. In PointGCR, an undirected graph representation (namely, ChannelGraph) is used to learn channel independencies. Specifically, channel maps are first represented as graph nodes and the independencies between nodes are then represented as graph edges. PointGCR is a plug-andplay and end-to-end trainable module. It can easily be integrated into an existing segmentation network and achieves a significant performance improvement. We conduct extensive experiments to evaluate the proposed PointGCR module on both indoor and outdoor datasets. Experimental results show that our PointGCR module efficiently captures global contextual dependencies and significantly improve the segmentation performance of several existing networks.
BibTeX
@inproceedings{ma2020global,
  author = {Yanni Ma and Yulan Guo and Hao Liu and Yinjie Lei and Gongjian Wen},
  title = {{Global Context Reasoning for Semantic Segmentation of 3D Point Clouds}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=UYQNYCZnoSce},
}
Figure