Label-Guided Knowledge Distillation for Continual Semantic Segmentation on 2D Images and 3D Point Clouds

Ze Yang, Ruibo Li, Evan Ling, Chi Zhang, Yiming Wang, Dezhao Huang, Keng Teck Ma, Minhoe Hur, Guosheng Lin

Introduction

Semantic segmentation is a fundamental task in computer vision, requiring models to assign a class label to every pixel in an image or every point in a 3D cloud. In real-world scenarios, data often arrives sequentially, necessitating Continual Semantic Segmentation (CSS). The goal is to learn new classes over time without suffering from catastrophic forgetting—the tendency of neural networks to abruptly lose previously acquired knowledge when trained on new information.

The Challenge of Background Shift

The primary obstacle in CSS is the "background shift." When a model is trained on a new set of classes at step $t$, the pixels belonging to these new classes were typically labeled as "background" in the previous step $t-1$. Standard Knowledge Distillation (KD) techniques, which encourage a new model to mimic an old one, struggle with this discrepancy.

Conceptual comparison of Standard KD (ILT), Unbiased KD (MiB), and the proposed Label-Guided KD (LGKD). LGKD aligns probabilities by expanding the old model's output and transplanting background signals to novel class labels.

As shown in Figure 1, previous methods like Initial Learning and Tuning (ILT) or Modeling the Background (MiB) either ignore the new classes during distillation or attempt to combine them with the background. This leads to "novel-background confusion," where the model receives conflicting signals: the old model identifies a pixel as background, while the new ground truth identifies it as a specific novel class (e.g., a "bicycle").

Label-Guided Knowledge Distillation (LGKD)

To resolve this conflict, this paper introduces Label-Guided Knowledge Distillation (LGKD). The core innovation is using the ground-truth (GT) labels available at the current training step to re-align the old model's predictions before distillation.

The LGKD mechanism consists of two primary operations:

  1. Expand: The output space of the old model is expanded to include slots for the new classes being learned.
  2. Transplant: For any pixel or point labeled as a novel class in the current step, the probability that the old model assigned to the "background" is moved (transplanted) into the corresponding novel class slot.
Detailed architecture of the LGKD framework. It separates input elements into novel and background sets, using ground-truth labels to guide the distillation process across different modalities (2D images and 3D point clouds).

By treating the old model as a "noisy teacher" and correcting its "vocabulary" using current labels, LGKD ensures that the knowledge transfer is consistent. As illustrated in Figure 2, this approach separates the distillation process into a Novel Set Distillation and a Background Set Distillation, effectively eliminating the ambiguity between old background labels and new class identities.

Experimental Evaluation and Benchmarks

The authors validated LGKD across diverse datasets and modalities:

  • 2D Image Benchmarks: Evaluations were conducted on Pascal-VOC 2012 and the more complex ADE20K dataset.
  • 3D Point Cloud Benchmark: Notably, the authors established the first-ever CSS benchmark for 3D point clouds using the ScanNet dataset. This involves adapting ScanNet into incremental learning scenarios to test how 3D models handle sequential class additions.

Because LGKD operates on output probability distributions rather than internal features, it is modality-agnostic, performing effectively with both 2D CNNs (like DeepLabv3) and 3D Point Cloud architectures (like PointNet++).

Results and Discussion

LGKD achieves state-of-the-art performance across all tested scenarios. On the Pascal-VOC 19-1 benchmark, LGKD improved the mean Intersection over Union (mIoU) of novel classes by a staggering +76% compared to the previous leading method, MiB.

In the 3D domain, the results on the ScanNet benchmark were equally impressive. LGKD improved novel class mIoU by +27% to +29% over the baseline. These results confirm that the background shift is a significant hurdle in 3D CSS and that the label-guided alignment strategy is a robust solution.

Despite these gains, some challenges remain. The authors noted that "long-tail" distributions in 3D scenes make rare classes difficult to learn continually. Additionally, high-level semantic confusion persists in some 2D cases, suggesting that future work could explore more sophisticated semantic distillation techniques.

Conclusion

Label-Guided Knowledge Distillation provides a simple yet powerful solution to the background shift problem in continual semantic segmentation. By using current labels to bridge the gap between old and new model protocols, LGKD prevents catastrophic forgetting and sets a new standard for both 2D and 3D incremental learning tasks.

Summary updated

Abstract

Continual semantic segmentation (CSS) aims to extend an existing model to tackle unseen tasks while retaining its old knowledge. Naively fine-tuning the old model on new data leads to catastrophic forgetting. A common solution is knowledge distillation (KD), where the output distribution of the new model is regularized to be similar to that of the old model. However, in CSS, this is challenging because of the background shift issue. Existing KD-based CSS methods continue to suffer from confusion between the background and novel classes since they fail to establish a reliable class correspondence for distillation. To address this issue, we propose a new label-guided knowledge distillation (LGKD) loss, where the old model output is expanded and transplanted (with the guidance of the ground truth label) to form a semantically appropriate class correspondence with the new model output. Consequently, the useful knowledge from the old model can be effectively distilled into the new model without causing confusion. We conduct extensive experiments on two prevailing CSS benchmarks, Pascal-VOC and ADE20K, where our LGKD significantly boosts the performance of three competing methods, especially on novel mIoU by up to +76%, setting new state-of-the-art. Finally, to further demonstrate its generalization ability, we introduce the first CSS benchmark for 3D point cloud based on ScanNet, along with several re-implemented baselines for comparison. Experiments show that LGKD is versatile in both 2D and 3D modalities without requiring ad hoc design. Codes are available at https://github.com/Ze-Yang/LGKD.
BibTeX
@inproceedings{yang2023label,
  author = {Ze Yang and Ruibo Li and Evan Ling and Chi Zhang and Yiming Wang and Dezhao Huang and Keng Teck Ma and Minhoe Hur and Guosheng Lin},
  title = {{Label-Guided Knowledge Distillation for Continual Semantic Segmentation on 2D Images and 3D Point Clouds}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01705},
}
Figure