Abstract
Semantic segmentation of road scenes is a cornerstone technology for autonomous driving perception. While deep Convolutional Neural Networks (CNNs) have demonstrated significant success in this task, state-of-the-art models are often hindered by extreme computational requirements and the need for massive labeled datasets. This paper introduces Spirit Distillation (SD), a novel knowledge distillation framework designed for cross-domain knowledge transference and efficient training under data-scarce conditions. By enabling a compact student network to mimic the intermediate "spirits" (feature maps) of a heavy teacher network, we achieve high-performance real-time segmentation. Furthermore, we propose Enhanced Spirit Distillation (ESD), which incorporates unlabeled images from proximity domains to improve robustness. Experiments on the Cityscapes dataset, using prior knowledge from COCO2017 and KITTI, demonstrate that our method improves mIOU by 1.4% and high-precision accuracy by 8.2% while utilizing only 41.8% of the teacher's FLOPs.
Introduction
In the field of autonomous driving, semantic segmentation—the pixel-level classification of road environments—is essential for safe navigation. However, deploying these models on vehicle hardware presents a dual challenge. First, high-accuracy models like DeepLabV3 are computationally expensive, making real-time execution difficult. Second, high-quality labeled data is notoriously difficult and expensive to produce. While transfer learning (Fine-tuning-based Transfer Learning, or FTT) is a common solution, it often fails to bridge the gap when the target dataset is extremely small or when the model architecture must be significantly compressed for speed.
Methodology
Spirit Distillation (SD)
Traditional knowledge distillation typically involves a student network mimicking the final output (logits) of a teacher network. However, in cross-domain scenarios where the teacher and student are trained on different datasets, the final outputs may not align.
Spirit Distillation shifts the focus from the output to the internal logic of the network. The student network is designed with a compact backbone using efficient group convolutions to reduce computational overhead. During training, the student is tasked with mimicking the teacher’s intermediate feature maps—the "spirits." By learning how a large, pre-trained model (the teacher) extracts general visual features from a massive source domain (like COCO2017), the student develops a robust feature extraction capability that generalizes well to the target road scene, even with limited local labels.
Enhanced Spirit Distillation (ESD)
To further address data insufficiency and prevent overfitting on small datasets, we introduced Enhanced Spirit Distillation. This method leverages a "proximity domain"—a dataset that is visually similar to the target domain but lacks labels (e.g., using KITTI images to supplement Cityscapes).
In ESD, the student network processes a shuffled stream of images from both the target and proximity domains. Because the teacher has already learned generalizable features, it can provide high-quality "spirit" guidance for the proximity images. This forces the student to learn more comprehensive features (such as handling varied lighting or shadow conditions) that may be underrepresented in the small labeled target set.
Experimental Setup
To test the efficacy of SD and ESD in "few-shot" scenarios, we utilized the following datasets:
- Source Domain: COCO2017, used to pre-train the heavy teacher network.
- Target Domain: Cityscapes. To simulate data insufficiency, we used Cityscapes-64, a subset containing only 64 training images.
- Proximity Domain: KITTI, providing unlabeled road scene context for the ESD process.
Results and Discussion
Our experiments demonstrate that Spirit Distillation allows a lightweight student network to significantly outperform standard training regimes:
- Computational Efficiency: The student network achieved its results using only 41.8% of the FLOPs (Floating Point Operations) required by the teacher, making it viable for real-time edge deployment.
- Accuracy Gains: On the Cityscapes-64 benchmark, SD/ESD improved the Mean Intersection over Union (mIOU) by 1.4% and boosted high-precision accuracy by 8.2% compared to traditional fine-tuning.
- Robustness: The ESD method reduced segmentation variance by 21.8%. This indicates that the inclusion of proximity domain data helps the model maintain consistent performance across diverse environmental conditions, effectively mitigating the risks of training on a tiny dataset.
Conclusion
Spirit Distillation represents a pioneering application of knowledge distillation to the problem of few-shot learning in semantic segmentation. By focusing on intermediate feature mimicry and leveraging unlabeled data from similar environments, we can produce models that are both computationally efficient and highly accurate. Future research will explore the automated selection of proximity domains and the application of this framework to other computer vision tasks such as object detection.
Summary updated