GroupContrast: Semantic-Aware Self-Supervised Representation Learning for 3D Understanding

Chengyao Wang, Li Jiang, Xiaoyang Wu, Zhuotao Tian, Bohao Peng, Hengshuang Zhao, Jiaya Jia

Abstract

Traditional self-supervised 3D learning methods often suffer from "semantic conflict" by treating different parts of the same object as negative pairs. GroupContrast addresses this by grouping points into semantically meaningful regions without human labels. By performing contrastive learning at the group level, it achieves state-of-the-art performance across 3D segmentation and object detection tasks on benchmarks like ScanNet and S3DIS.

Introduction: The Semantic Conflict in 3D Learning

Most current self-supervised 3D learning methods rely on point discrimination. When comparing two "views" of a 3D room, these methods attempt to match the exact same point in both views (positive pair) while pushing all other points away as negative pairs.

This approach creates a significant semantic conflict. For instance, if a model is forced to treat a point on the left side of a table as a "negative" to a point on the right side of the same table, it fails to learn that they belong to the same object. This leads to high false-negative rates and fragmented representations that struggle to identify objects as cohesive units.

Comparison of point-wise similarity heatmaps. GroupContrast (GC) shows much higher semantic coherence compared to traditional CSC methods.

Methodology

GroupContrast introduces a teacher-student framework designed to learn semantic groupings and contrastive features simultaneously.

The GroupContrast framework, featuring segment grouping with learnable prototypes and semantic-aware contrastive learning.

1. Segment Grouping

The model first partitions the 3D point cloud into small geometric segments using a standard graph-cut method. It then employs a set of learnable prototypes (cluster centers) to group these segments into larger, semantically meaningful regions. This process uses "informative-aware distillation," where a teacher network guides a student network to assign segments to the same prototypes across different views.

2. Semantic-aware Contrastive Learning

Once groups are formed, the model performs contrastive learning where points within the same group are treated as positive pairs, and points in different groups are negatives. To handle initial noise, a confidence weight is introduced, ensuring the model does not over-rely on uncertain groupings during early training phases.

3. Architecture

The system uses a SparseUNet backbone. The teacher network is updated via an Exponential Moving Average (EMA) of the student’s weights, providing a stable target for learning.

Experiments and Results

The researchers evaluated GroupContrast on major 3D indoor scene benchmarks, including ScanNet, ScanNet200, S3DIS, and SUN RGB-D.

Performance Benchmarks

GroupContrast established new state-of-the-art results for self-supervised 3D representation:

  • Semantic Segmentation: Achieved 75.7% mIoU on ScanNet, outperforming previous methods and the "from-scratch" baseline by 3.5%.
  • Instance Segmentation: Reached 27.5 [email protected] on ScanNet200.
  • Object Detection: Improved 3D object detection on ScanNet to 41.1 [email protected], surpassing the previous best by 1.8 points.

Key Insights

  • Emergent Concepts: Learnable prototypes naturally begin to represent real-world concepts like "floors," "chair backrests," or "ceilings" without any human labels.
  • Data Efficiency: In low-label regimes (1% or 5% labeled data), GroupContrast significantly outperforms traditional methods, demonstrating high efficiency.
  • Foundation Model Synergy: The grouping method can be enhanced by integrating masks from 2D foundation models like Segment Anything (SAM), further boosting performance to 75.9% mIoU.

Conclusion

GroupContrast demonstrates that seeking semantic coherence is more effective than simple geometric point discrimination for 3D understanding. By moving beyond point-to-point comparisons, the framework provides a more robust foundation for complex 3D tasks. While currently focused on indoor scenes, future work aims to scale this approach to more diverse datasets and universal 3D foundation models.

Summary updated

Abstract

Self-supervised 3D representation learning aims to learn effective representations from large-scale unlabeled point clouds. Most existing approaches adopt point discrimination as the pretext task, which assigns matched points in two distinct views as positive pairs and unmatched points as negative pairs. However, this approach often results in semantically identical points having dissimilar representations, leading to a high number of false negatives and introducing a "semantic conflict" problem. To address this issue, we propose GroupContrast, a novel approach that combines segment grouping and semantic-aware contrastive learning. Segment grouping partitions points into semantically meaningful regions, which enhances semantic coherence and provides semantic guidance for the subsequent contrastive representation learning. Semantic-aware contrastive learning augments the semantic information extracted from segment grouping and helps to alleviate the issue of "semantic conflict". We conducted extensive experiments on multiple 3D scene understanding tasks. The results demonstrate that GroupContrast learns semantically meaningful representations and achieves promising transfer learning performance.
BibTeX
@inproceedings{wang2024groupcontrast,
  author = {Chengyao Wang and Li Jiang and Xiaoyang Wu and Zhuotao Tian and Bohao Peng and Hengshuang Zhao and Jiaya Jia},
  title = {{GroupContrast: Semantic-Aware Self-Supervised Representation Learning for 3D Understanding}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}
Figure