3D Graph Neural Networks for RGBD Semantic Segmentation

Xiaojuan Qi, Renjie Liao, Jiaya Jia, Sanja Fidler, Raquel UrtasunICCV 2017 · OpenReview

Abstract

RGBD semantic segmentation requires joint reasoning about 2D appearance and 3D geometric information. In this paper we propose a 3D graph neural network (3DGNN) that builds a k-nearest neighbor graph on top of 3D point cloud. Each node in the graph corresponds to a set of points and is associated with a hidden representation vector initialized with an appearance feature extracted by a unary CNN from 2D images. Relying on recurrent functions, every node dynamically updates its hidden representation based on the current status and incoming messages from its neighbors. This propagation model is unrolled for a certain number of time steps and the final per-node representation is used for predicting the semantic class of each pixel. We use back-propagation through time to train the model. Extensive experiments on NYUD2 and SUN-RGBD datasets demonstrate the effectiveness of our approach.

Introduction

Semantic segmentation—the task of labeling every pixel in an image (e.g., "chair," "floor," "table")—is significantly enhanced when depth data (RGBD) is available. However, traditional methods often treat depth as a simple fourth channel in a 2D image grid. This approach is fundamentally flawed because 2D proximity does not equate to 3D proximity. In a 2D view, a pixel on a foreground table might be adjacent to a pixel on a distant background appliance. A standard 2D Convolutional Neural Network (CNN) will attempt to relate these pixels due to their grid proximity, leading to blurred semantic boundaries.

Conversely, objects that are continuous in 3D space, such as a long sofa, may be fragmented in a 2D projection by occluding objects. While 3D voxel-based CNNs can address spatial relationships, they are computationally expensive and inefficient at handling the inherent sparsity of 3D point clouds.

Methodology: The 3DGNN Framework

The proposed 3DGNN bridges the gap between 2D appearance and 3D geometry through a multi-stage process that leverages the strengths of both domains.

The 3DGNN pipeline: from 2D image and depth to 3D graph propagation and final semantic prediction.
  1. Graph Construction: The 2D image is projected into 3D space using depth information. Pixels are treated as nodes in a graph, with each node connected to its $K$ nearest neighbors ($K=64$) in actual 3D space.
  2. Initial Features: A standard 2D CNN (such as ResNet-101) extracts appearance features from the RGB image to initialize the state of each node.
  3. Recurrent Propagation: The core of the model is a recurrent function (RNN or LSTM). In each time step, nodes collect messages from their 3D neighbors, processed via a Multi-Layer Perceptron (MLP), to update their internal representations. This allows information to flow across the true 3D structure of the scene.
  4. Prediction: After several iterations (typically 3 to 6), the final hidden state of each node is used to predict the semantic class for every pixel.

Key Insights and Results

The primary advantage of 3DGNN is its ability to use 3D context to disambiguate visually similar objects. By connecting points that are physically close in the real world, the model avoids the "bleeding" of labels between unrelated objects that happen to overlap in a 2D view.

Comparison of 2D vs. 3D prediction. The 3D graph correctly identifies the 'Table' by leveraging spatial context that the 2D model confuses with the 'Counter'.

Performance Benchmarks

The model was evaluated on two major RGBD datasets:

  • NYUD2: Achieved a 43.1% mean IoU (40-class), outperforming complex multi-scale 2D CNNs.
  • SUN-RGBD: Reached 45.9% mean IoU using a ResNet-101 backbone.

3DGNN showed significant improvements in identifying large, rigid objects like cabinets, beds, and refrigerators. These are categories where global 3D shape and spatial context are far more informative than local 2D textures. Furthermore, because the model propagates information across the graph, it effectively expands its receptive field to cover entire 3D objects that might otherwise exceed the fixed receptive field of a standard CNN.

Conclusion

While 3DGNN sets a new standard for RGBD segmentation, challenges remain regarding sensor noise in complex, thin geometries and the computational overhead of k-NN graph construction. However, the research demonstrates that injecting "physical common sense" via Graph Neural Networks is a powerful strategy for scene understanding. Future work may involve dynamic graph structures that can adapt during the propagation phase to further refine object boundaries.

Summary updated

Abstract

RGBD semantic segmentation requires joint reasoning about 2D appearance and 3D geometric information. In this paper we propose a 3D graph neural network (3DGNN) that builds a k-nearest neighbor graph on top of 3D point cloud. Each node in the graph corresponds to a set of points and is associated with a hidden representation vector initialized with an appearance feature extracted by a unary CNN from 2D images. Relying on recurrent functions, every node dynamically updates its hidden representation based on the current status and incoming messages from its neighbors. This propagation model is unrolled for a certain number of time steps and the final per-node representation is used for predicting the semantic class of each pixel. We use back-propagation through time to train the model. Extensive experiments on NYUD2 and SUN-RGBD datasets demonstrate the effectiveness of our approach.
BibTeX
@inproceedings{qi2017d,
  author = {Xiaojuan Qi and Renjie Liao and Jiaya Jia and Sanja Fidler and Raquel Urtasun},
  title = {{3D Graph Neural Networks for RGBD Semantic Segmentation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2017},
  url = {https://openreview.net/forum?id=Bk-zBWz_bB},
}
Figure