Perturbed Self-Distillation: Weakly Supervised Large-Scale Point Cloud Semantic Segmentation

Yuan Xie

Abstract

The Perturbed Self-Distillation (PSD) framework addresses the significant "annotation bottleneck" in 3D deep learning. It enables high-quality semantic segmentation of large-scale point clouds using as little as 1% of the data labeled, or even just a single point per class. By enforcing consistency between original and perturbed versions of a point cloud and explicitly modeling the relationships between labeled "anchor" points, PSD achieves results comparable to fully supervised methods while drastically reducing manual labeling effort.

Introduction: The Annotation Bottleneck

Semantic segmentation—the process of assigning a category (e.g., chair, floor, wall) to every point in a 3D scene—is a foundational task for autonomous systems and robotics. However, modern large-scale point clouds contain millions of points, making manual annotation prohibitively expensive. For instance, labeling a single scene in the ScanNet dataset can take over 22 minutes.

While weakly supervised methods aim to reduce this burden, they often struggle with large-scale data. These methods frequently fail to extract discriminative features for the vast majority of unlabeled points and ignore the underlying structural relationships (topology) of the point cloud. This results in "bleeding" errors, where the model fails to distinguish between adjacent objects with similar structures, such as a door and the surrounding wall.

The PSD Framework

The Perturbed Self-Distillation (PSD) framework is designed to be "backbone-agnostic," allowing it to enhance existing efficient architectures like RandLA-Net. It relies on two core innovations:

1. Perturbed Branch and Self-Distillation

PSD processes two versions of the same point cloud: the original and a perturbed version. Perturbations include standard geometric transformations (rotation, jittering) and a novel Attribute Attention layer. This layer learnably weights input features—such as color versus spatial coordinates—allowing the model to prioritize the most reliable information for a given context. By minimizing the Jensen-Shannon divergence between the predictions of both branches, the network learns to propagate labels from the few annotated points to their unlabeled neighbors.

2. Context-Aware Module (CAM)

To leverage the limited labels effectively, PSD treats them as "anchors." The CAM calculates a Point Affinity Matrix to model the relationships between these anchor points. By ensuring that labeled points of the same category have similar feature representations, the module regularizes the feature space and refines the global graph topology of the entire scene.

Key Results

The PSD framework was evaluated on three major benchmarks: S3DIS (indoor), ScanNet-v2 (RGB-D scans), and Semantic3D (outdoor urban environments).

Qualitative comparison between Baseline (RandLA-Net), PSD, and Ground Truth on S3DIS Area-5 scenes.
  • Efficiency with 1% Labels: On S3DIS (Area-5), PSD achieved a mean Intersection over Union (mIoU) of 63.5% with only 1% labels, outperforming previous weakly supervised methods and matching the performance of the fully supervised RandLA-Net (63.0%).
  • The "1pt" Challenge: In an extreme scenario with only one labeled point per category, PSD reached 48.2% mIoU, a 7.5% improvement over the baseline.
  • Outdoor Scalability: On the Semantic3D dataset, PSD showed a 6.9% gain over the baseline, demonstrating its effectiveness in complex, noisy outdoor environments.

Discussion and Insights

The success of PSD highlights several key principles for weakly supervised learning in 3D:

  • Consistency as Supervision: Enforcing prediction consistency across perturbations allows the model to discover the underlying data structure without human intervention.
  • Dynamic Feature Weighting: The Attribute Attention mechanism is crucial for handling noisy data, allowing the model to ignore deceptive features like inconsistent lighting or color.
  • Relational Anchoring: Treating labeled points as relational anchors rather than isolated signals significantly improves the model's holistic understanding of the scene.

Conclusion

While PSD significantly reduces the labeling requirement, it introduces computational complexity due to the memory-intensive nature of the affinity matrix. To mitigate this, the Context-Aware Module is typically applied after the encoder stage. Future research will focus on optimizing these relational modules to further bridge the gap between weak and full supervision in even larger datasets.

Summary updated

Abstract

Large-scale point cloud semantic segmentation has wide applications. Current popular researches mainly focus on fully supervised learning which demands expensive and tedious manual point-wise annotation. Weakly supervised learning is an alternative way to avoid this exhausting annotation. However, for large-scale point clouds with few labeled points, the network is difficult to extract discriminative features for unlabeled points, as well as the regularization of topology between labeled and unlabeled points is usually ignored, resulting in incorrect segmentation results. To address this problem, we propose a perturbed selfdistillation (PSD) framework. Specifically, inspired by selfsupervised learning, we construct the perturbed branch and enforce the predictive consistency among the perturbed branch and original branch. In this way, the graph topology of the whole point cloud can be effectively established by the introduced auxiliary supervision, such that the information propagation between the labeled and unlabeled points will be realized. Besides point-level supervision, we present a well-integrated context-aware module to explicitly regularize the affinity correlation of labeled points. Therefore, the graph topology of the point cloud can be further refined. The experimental results evaluated on three large-scale datasets show the large gain (3.0% on average) against recent weakly supervised methods and comparable results to some fully supervised methods.
BibTeX
@inproceedings{xie2021perturbed,
  author = {Yuan Xie},
  title = {{Perturbed Self-Distillation: Weakly Supervised Large-Scale Point Cloud Semantic Segmentation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=ymCRaL3Hs12},
}
Figure