Omni-supervised Point Cloud Segmentation via Gradual Receptive Field Component Reasoning

Yuan Xie

TL;DR

This paper introduces a novel "omni-supervised" approach for 3D point cloud segmentation by introducing "Receptive Field Component Codes" (RFCCs) to supervise intermediate network layers. This method, called RFCR, decomposes segmentation into a coarse-to-fine reasoning process and significantly improves performance across multiple benchmarks, achieving state-of-the-art results.

What Problem Does This Paper Address?

3D point cloud semantic segmentation, the task of assigning a category label to each point in a 3D scan, is crucial for applications like robotics and autonomous driving. Traditional methods often rely solely on supervision at the final output layer of a neural network. This leaves intermediate layers, which learn hierarchical features, under-supervised. This lack of supervision can lead to hidden network units being inactive or having small magnitudes, hindering their ability to learn informative representations. The challenge is to effectively supervise these intermediate layers in a way that is compatible with the irregular structure of point clouds and doesn't require additional annotations.

Methods

The core innovation is the Gradual Receptive Field Component Reasoning (RFCR) framework. Instead of directly supervising intermediate features, RFCR introduces Receptive Field Component Codes (RFCCs). These are multi-hot binary vectors that encode the presence of different object categories within the receptive field of a feature at a specific layer.

Here's how it works:

  • RFCC Generation: Target RFCCs are generated in the encoder by analyzing the ground truth labels within the receptive fields of features at each layer. This process is "omni-scale" because it happens at multiple levels of the network.
  • RFCC Reasoning: In the decoder, the network is trained to predict RFCCs for each layer, starting from a global context prediction and progressively refining it to more local receptive fields. This is guided by the target RFCCs from the encoder, effectively creating a coarse-to-fine reasoning process.
  • Feature Densification: To address the issue of inactive or low-magnitude features, the authors propose Feature Densification. This technique uses a "centrifugal potential" to push features away from zero, making them more active and contributing more unambiguously to RFCC prediction. This is shown to be equivalent to entropy regularization on features.

The overall loss combines the standard cross-entropy loss for segmentation with losses for RFCC prediction and Feature Densification.

Datasets

The method was evaluated on three popular 3D point cloud segmentation datasets:

  • ScanNet v2: A dataset of indoor scenes with 20 semantic categories.
  • S3DIS: A dataset of large-scale indoor rooms across 6 areas, with 13 classes.
  • Semantic3D: A large-scale outdoor dataset with 8 categories.

The paper also mentions the Semantic3D reduced-8 task, which uses down-sampled point clouds for testing.

Key Insights and Ideas

  • Omni-Supervision is Key: The paper convincingly demonstrates that supervising intermediate network layers, not just the output, is crucial for learning better representations in 3D point cloud segmentation.
  • RFCCs as a Bridge: RFCCs provide a clever way to bridge the gap between feature representations and semantic labels at multiple scales, overcoming the challenges of irregular point cloud structures.
  • Coarse-to-Fine Reasoning: Decomposing the complex segmentation task into a series of RFCC reasoning steps makes the learning process more manageable and effective.
  • Feature Activation Matters: Activating more hidden units (through Feature Densification) significantly boosts performance, highlighting the importance of feature clarity and distinctiveness.
  • Generalizability: The method shows strong performance when integrated with various backbone architectures, indicating its broad applicability.

Key Results

The proposed RFCR method, combined with Feature Densification, achieved significant improvements across all tested backbones and datasets:

  • ScanNet v2: Achieved a state-of-the-art mIoU of 70.2%, ranking first among point-based methods.
  • S3DIS Area-5: Pushed the state-of-the-art with an mIoU of 68.73%.
  • Semantic3D (reduced-8): Achieved an mIoU of 77.8%, outperforming existing methods.
  • Ablation studies showed that RFCR alone improved performance by 2.9%, and Feature Densification further added 1.8% improvement on Semantic3D.

Qualitative results also showed noticeable improvements, particularly in distinguishing fine-grained details and object boundaries.

Limitations and Future Work

  • Backbone Dependency: While the method improves various backbones, it is noted that it is "more suitable for networks with encoder-decoder architecture."
  • Up-sampling Limitations: The paper briefly discusses and contrasts their RFCR approach with an up-sampling based multi-scale supervision strategy, showing RFCR's superiority due to its ability to trace encoding relationships better.
  • Future Work: The paper doesn't explicitly detail future work directions, but the success of RFCR and Feature Densification suggests potential for further exploration in designing more effective intermediate supervision strategies and feature enhancement techniques for 3D deep learning.

Summary updated

Abstract

Hidden features in neural network usually fail to learn informative representation for 3D segmentation as supervisions are only given on output prediction, while this can be solved by omni-scale supervision on intermediate layers. In this paper, we bring the first omni-scale supervision method to point cloud segmentation via the proposed gradual Receptive Field Component Reasoning (RFCR), where target Receptive Field Component Codes (RFCCs) are designed to record categories within receptive fields for hidden units in the encoder. Then, target RFCCs will supervise the decoder to gradually infer the RFCCs in a coarse-to-fine categories reasoning manner, and finally obtain the semantic labels. Because many hidden features are inactive with tiny magnitude and make minor contributions to RFCC prediction, we propose a Feature Densification with a centrifugal potential to obtain more unambiguous features, and it is in effect equivalent to entropy regularization over features. More active features can further unleash the potential of our omni-supervision method. We embed our method into four prevailing backbones and test on three challenging benchmarks. Our method can significantly improve the backbones in all three datasets. Specifically, our method brings new state-of-the-art performances for S3DIS as well as Semantic3D and ranks the 1st in the ScanNet benchmark among all the point-based methods. Code is publicly available at https://github.com/azuki-miho/RFCR.
BibTeX
@inproceedings{xie2021omni,
  author = {Yuan Xie},
  title = {{Omni-supervised Point Cloud Segmentation via Gradual Receptive Field Component Reasoning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=8c2s7uo99UR},
}
Figure