Introduction
Semantic segmentation is a fundamental task in computer vision, requiring models to assign a class label to every pixel in an image or every point in a 3D cloud. In real-world scenarios, data often arrives sequentially, necessitating Continual Semantic Segmentation (CSS). The goal is to learn new classes over time without suffering from catastrophic forgetting—the tendency of neural networks to abruptly lose previously acquired knowledge when trained on new information.
The Challenge of Background Shift
The primary obstacle in CSS is the "background shift." When a model is trained on a new set of classes at step $t$, the pixels belonging to these new classes were typically labeled as "background" in the previous step $t-1$. Standard Knowledge Distillation (KD) techniques, which encourage a new model to mimic an old one, struggle with this discrepancy.
As shown in Figure 1, previous methods like Initial Learning and Tuning (ILT) or Modeling the Background (MiB) either ignore the new classes during distillation or attempt to combine them with the background. This leads to "novel-background confusion," where the model receives conflicting signals: the old model identifies a pixel as background, while the new ground truth identifies it as a specific novel class (e.g., a "bicycle").
Label-Guided Knowledge Distillation (LGKD)
To resolve this conflict, this paper introduces Label-Guided Knowledge Distillation (LGKD). The core innovation is using the ground-truth (GT) labels available at the current training step to re-align the old model's predictions before distillation.
The LGKD mechanism consists of two primary operations:
- Expand: The output space of the old model is expanded to include slots for the new classes being learned.
- Transplant: For any pixel or point labeled as a novel class in the current step, the probability that the old model assigned to the "background" is moved (transplanted) into the corresponding novel class slot.
By treating the old model as a "noisy teacher" and correcting its "vocabulary" using current labels, LGKD ensures that the knowledge transfer is consistent. As illustrated in Figure 2, this approach separates the distillation process into a Novel Set Distillation and a Background Set Distillation, effectively eliminating the ambiguity between old background labels and new class identities.
Experimental Evaluation and Benchmarks
The authors validated LGKD across diverse datasets and modalities:
- 2D Image Benchmarks: Evaluations were conducted on Pascal-VOC 2012 and the more complex ADE20K dataset.
- 3D Point Cloud Benchmark: Notably, the authors established the first-ever CSS benchmark for 3D point clouds using the ScanNet dataset. This involves adapting ScanNet into incremental learning scenarios to test how 3D models handle sequential class additions.
Because LGKD operates on output probability distributions rather than internal features, it is modality-agnostic, performing effectively with both 2D CNNs (like DeepLabv3) and 3D Point Cloud architectures (like PointNet++).
Results and Discussion
LGKD achieves state-of-the-art performance across all tested scenarios. On the Pascal-VOC 19-1 benchmark, LGKD improved the mean Intersection over Union (mIoU) of novel classes by a staggering +76% compared to the previous leading method, MiB.
In the 3D domain, the results on the ScanNet benchmark were equally impressive. LGKD improved novel class mIoU by +27% to +29% over the baseline. These results confirm that the background shift is a significant hurdle in 3D CSS and that the label-guided alignment strategy is a robust solution.
Despite these gains, some challenges remain. The authors noted that "long-tail" distributions in 3D scenes make rare classes difficult to learn continually. Additionally, high-level semantic confusion persists in some 2D cases, suggesting that future work could explore more sophisticated semantic distillation techniques.
Conclusion
Label-Guided Knowledge Distillation provides a simple yet powerful solution to the background shift problem in continual semantic segmentation. By using current labels to bridge the gap between old and new model protocols, LGKD prevents catastrophic forgetting and sets a new standard for both 2D and 3D incremental learning tasks.
Summary updated