Spirit Distillation: Precise Real-time Semantic Segmentation of Road Scenes with Insufficient Data

Zhiyuan Wu, Yu Jiang, Chupeng Cui, Zongmin Yang, Xinhui Xue, Hong Qi

Abstract

Semantic segmentation of road scenes is a cornerstone technology for autonomous driving perception. While deep Convolutional Neural Networks (CNNs) have demonstrated significant success in this task, state-of-the-art models are often hindered by extreme computational requirements and the need for massive labeled datasets. This paper introduces Spirit Distillation (SD), a novel knowledge distillation framework designed for cross-domain knowledge transference and efficient training under data-scarce conditions. By enabling a compact student network to mimic the intermediate "spirits" (feature maps) of a heavy teacher network, we achieve high-performance real-time segmentation. Furthermore, we propose Enhanced Spirit Distillation (ESD), which incorporates unlabeled images from proximity domains to improve robustness. Experiments on the Cityscapes dataset, using prior knowledge from COCO2017 and KITTI, demonstrate that our method improves mIOU by 1.4% and high-precision accuracy by 8.2% while utilizing only 41.8% of the teacher's FLOPs.

Introduction

In the field of autonomous driving, semantic segmentation—the pixel-level classification of road environments—is essential for safe navigation. However, deploying these models on vehicle hardware presents a dual challenge. First, high-accuracy models like DeepLabV3 are computationally expensive, making real-time execution difficult. Second, high-quality labeled data is notoriously difficult and expensive to produce. While transfer learning (Fine-tuning-based Transfer Learning, or FTT) is a common solution, it often fails to bridge the gap when the target dataset is extremely small or when the model architecture must be significantly compressed for speed.

Methodology

Spirit Distillation (SD)

Traditional knowledge distillation typically involves a student network mimicking the final output (logits) of a teacher network. However, in cross-domain scenarios where the teacher and student are trained on different datasets, the final outputs may not align.

Spirit Distillation shifts the focus from the output to the internal logic of the network. The student network is designed with a compact backbone using efficient group convolutions to reduce computational overhead. During training, the student is tasked with mimicking the teacher’s intermediate feature maps—the "spirits." By learning how a large, pre-trained model (the teacher) extracts general visual features from a massive source domain (like COCO2017), the student develops a robust feature extraction capability that generalizes well to the target road scene, even with limited local labels.

Enhanced Spirit Distillation (ESD)

To further address data insufficiency and prevent overfitting on small datasets, we introduced Enhanced Spirit Distillation. This method leverages a "proximity domain"—a dataset that is visually similar to the target domain but lacks labels (e.g., using KITTI images to supplement Cityscapes).

In ESD, the student network processes a shuffled stream of images from both the target and proximity domains. Because the teacher has already learned generalizable features, it can provide high-quality "spirit" guidance for the proximity images. This forces the student to learn more comprehensive features (such as handling varied lighting or shadow conditions) that may be underrepresented in the small labeled target set.

Experimental Setup

To test the efficacy of SD and ESD in "few-shot" scenarios, we utilized the following datasets:

  • Source Domain: COCO2017, used to pre-train the heavy teacher network.
  • Target Domain: Cityscapes. To simulate data insufficiency, we used Cityscapes-64, a subset containing only 64 training images.
  • Proximity Domain: KITTI, providing unlabeled road scene context for the ESD process.

Results and Discussion

Our experiments demonstrate that Spirit Distillation allows a lightweight student network to significantly outperform standard training regimes:

  1. Computational Efficiency: The student network achieved its results using only 41.8% of the FLOPs (Floating Point Operations) required by the teacher, making it viable for real-time edge deployment.
  2. Accuracy Gains: On the Cityscapes-64 benchmark, SD/ESD improved the Mean Intersection over Union (mIOU) by 1.4% and boosted high-precision accuracy by 8.2% compared to traditional fine-tuning.
  3. Robustness: The ESD method reduced segmentation variance by 21.8%. This indicates that the inclusion of proximity domain data helps the model maintain consistent performance across diverse environmental conditions, effectively mitigating the risks of training on a tiny dataset.

Conclusion

Spirit Distillation represents a pioneering application of knowledge distillation to the problem of few-shot learning in semantic segmentation. By focusing on intermediate feature mimicry and leveraging unlabeled data from similar environments, we can produce models that are both computationally efficient and highly accurate. Future research will explore the automated selection of proximity domains and the application of this framework to other computer vision tasks such as object detection.

Summary updated

Abstract

Semantic segmentation of road scenes is one of the key technologies for realizing autonomous driving scene perception, and the effectiveness of deep Convolutional Neural Networks(CNNs) for this task has been demonstrated. State-of-art CNNs for semantic segmentation suffer from excessive computations as well as large-scale training data requirement. Inspired by the ideas of Fine-tuning-based Transfer Learning (FTT) and feature-based knowledge distillation, we propose a new knowledge distillation method for cross-domain knowledge transference and efficient data-insufficient network training, named Spirit Distillation(SD), which allow the student network to mimic the teacher network to extract general features, so that a compact and accurate student network can be trained for real-time semantic segmentation of road scenes. Then, in order to further alleviate the trouble of insufficient data and improve the robustness of the student, an Enhanced Spirit Distillation (ESD) method is proposed, which commits to exploit a more comprehensive general features extraction capability by considering images from both the target and the proximity domains as input. To our knowledge, this paper is a pioneering work on the application of knowledge distillation to few-shot learning. Persuasive experiments conducted on Cityscapes semantic segmentation with the prior knowledge transferred from COCO2017 and KITTI demonstrate that our methods can train a better student network (mIOU and high-precision accuracy boost by 1.4% and 8.2% respectively, with 78.2% segmentation variance) with only 41.8% FLOPs (see Fig. 1).
BibTeX
@misc{wu2021spirit,
  author = {Zhiyuan Wu and Yu Jiang and Chupeng Cui and Zongmin Yang and Xinhui Xue and Hong Qi},
  title = {{Spirit Distillation: Precise Real-time Semantic Segmentation of Road Scenes with Insufficient Data}},
  year = {2021},
  eprint = {2103.13733},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.13733},
}
Figure