Segment-Fusion: Hierarchical Context Fusion for Robust 3D Semantic Segmentation

Anirud Thyagharajan, Benjamin Ummenhofer, Prashant Laddha, Om Ji Omer, Sreenivas SubramoneyCVPR 2022 · OpenReview

Abstract

3D semantic segmentation is a fundamental building block for several scene understanding applications such as autonomous driving, robotics, and AR/VR. Several state-of-the-art semantic segmentation models suffer from the part-misclassification problem, wherein parts of the same object are labeled incorrectly. Previous methods have utilized hierarchical, iterative methods to fuse semantic and instance information, but they lack learnability in context fusion and are computationally complex and heuristic-driven. This paper presents Segment-Fusion, a novel attention-based method for hierarchical fusion of semantic and instance information to address part misclassifications. The presented method includes a graph segmentation algorithm for grouping points into segments that pools point-wise features into segment-wise features, a learnable attention-based network to fuse these segments based on their semantic and instance features, followed by a simple yet effective connected component labeling algorithm to convert segment features to instance labels. Segment-Fusion can be flexibly employed with any network architecture for semantic/instance segmentation. It improves the qualitative and quantitative performance of several semantic segmentation backbones by up to 5% on the ScanNet and S3DIS datasets.

Introduction

In 3D scene understanding, semantic segmentation involves assigning a category label to every point in a 3D cloud. Despite significant progress, many current models struggle with "part misclassification." For example, a model might correctly identify the center of a door but mislabel its edges as "wall" due to local geometric similarities. This "identity crisis" stems from models processing points or voxels with limited global context. While combining semantic and instance data can mitigate this, existing solutions often rely on rigid, non-generalizable heuristics. Segment-Fusion addresses this by introducing a learnable, hierarchical framework that provides the necessary context to ensure object consistency.

Methodology

Segment-Fusion is designed as a modular, three-stage pipeline that can be integrated with various 3D backbones.

  1. Joint Graph Segmentation (J-GS): The system first groups individual points into segments (super-voxels or surfaces). This algorithm considers geometric normals and feature similarity, ensuring it functions effectively even on data lacking mesh connectivity, such as raw LiDAR scans.
  2. Segment-Fusion Network: This core component processes pooled segment features through a stack of attention-based encoder blocks. To maintain computational efficiency, the network utilizes a spatial adjacency matrix, ensuring segments only interact with their immediate neighbors. This allows the model to learn whether adjacent segments should be fused into a single object or remain separate.
  3. Connected Component Labeling (CCL): A clustering algorithm groups the fused segments. A consensus mechanism is then applied: the most frequent label within a fused object becomes the label for the entire object, effectively correcting local misclassifications.
Overview of the Segment-Fusion pipeline, illustrating the flow from point-wise features to fused segment labels.

Evaluation and Results

The framework was evaluated on the ScanNet and S3DIS datasets, representing diverse indoor environments with and without mesh information.

Quantitative Performance

Segment-Fusion consistently enhanced the performance of various backbone architectures. It improved the mean Intersection-over-Union (mIoU) for models such as SparseConvNet, PointConv, and MinkNet by 2.3% to 5.3% on ScanNet. Notably, when paired with the VMNet backbone, the system achieved the 2nd position on the ScanNet leaderboard at the time of publication.

Qualitative Improvements

The primary strength of Segment-Fusion is its ability to resolve part misclassifications. By moving from point-wise predictions to a segment-based voting mechanism, the model ensures that noisy local features do not derail the classification of large, coherent objects like walls or furniture.

Comparison between a baseline model and the Segment-Fusion enhanced model, highlighting the correction of part misclassifications.

Conclusion

Segment-Fusion demonstrates that a learnable fusion step significantly outperforms traditional heuristic-based clustering in 3D segmentation. By constraining attention spatially and utilizing a hierarchical approach, the framework achieves high-quality context fusion without excessive computational overhead. Future work may explore extending this hierarchical strategy to semi-supervised or self-supervised learning contexts to reduce the reliance on densely labeled training data.

Summary updated

Abstract

3D semantic segmentation is a fundamental building block for several scene understanding applications such as autonomous driving, robotics and AR/VR. Several state-of-the-art semantic segmentation models suffer from the part-misclassification problem, wherein parts of the same object are labelled incorrectly. Previous methods have utilized hierarchical, iterative methods to fuse semantic and instance information, but they lack learnability in context fusion, and are computationally complex and heuristic driven. This paper presents Segment-Fusion, a novel attention-based method for hierarchical fusion of semantic and instance in-formation to address the part misclassifications. The presented method includes a graph segmentation algorithmfor grouping points into segments that pools point-wise features into segment-wise features, a learnable attention-based net-work to fuse these segments based on their semantic and instance features, and followed by a simple yet effective connected component labelling algorithm to convert seg-ment features to instance labels. Segment-Fusion can be flexibly employed with any network architecture for semantic/instance segmentation. It improves the qualitative and quantitative performance of several semantic segmentation backbones by upto 5% on the ScanNet and S3DIS datasets.
BibTeX
@inproceedings{thyagharajan2022segment,
  author = {Anirud Thyagharajan and Benjamin Ummenhofer and Prashant Laddha and Om Ji Omer and Sreenivas Subramoney},
  title = {{Segment-Fusion: Hierarchical Context Fusion for Robust 3D Semantic Segmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  url = {https://openreview.net/forum?id=8hvJ-EAulx},
}
Figure