Lightweight Depth Completion Network with Local Similarity-Preserving Knowledge Distillation

Yongseop Jeong, Jinsun Park, Donghyeon Cho, Yoonjin Hwang, Seibum B. Choi, In So Kweon

TL;DR

This paper tackles the challenge of accurate and efficient depth estimation for autonomous driving by proposing a novel knowledge distillation technique called Local Similarity-Preserving Knowledge Distillation (LSPKD). LSPKD enables a lightweight "student" network to learn effectively from a more complex "teacher" network, significantly reducing computational costs for both training and real-time inference while maintaining high accuracy.

What Problem Does This Paper Address?

Autonomous driving systems rely heavily on accurate depth perception to navigate safely. While sensors like LiDAR provide precise depth information, they capture only sparse data points. Combining this sparse LiDAR data with dense RGB images is a common approach for "depth completion," aiming to generate a full, dense depth map. However, state-of-the-art depth completion models are often computationally expensive, hindering their deployment in real-time autonomous systems. Existing methods to create lighter models either sacrifice accuracy or fail to leverage the full potential of more powerful, albeit heavier, "teacher" models. This paper addresses the need for lightweight depth completion networks that can achieve high performance without the prohibitive computational burden.

Methods

The core innovation of this paper is the Local Similarity-Preserving Knowledge Distillation (LSPKD). Traditional knowledge distillation (KD) methods aim to transfer knowledge from a large teacher model to a smaller student model. LSPKD focuses on preserving local similarities between neighboring pixels in feature maps.

Here's a breakdown:

  • Teacher-Student Architecture: The authors use a ResNet34-based network as the teacher (capable of high accuracy but computationally heavy) and a ResNet18-based network as the student (lightweight and efficient). Both networks are designed to process RGB and LiDAR inputs.
  • Local Similarity Calculation: For a given feature map, the method calculates a "local similarity" score. This score quantifies how similar a pixel's feature representation is to its immediate neighbors. This is done by taking the dot product of normalized feature vectors of a reference pixel and its neighbors.
  • LSPKD Loss: The proposed distillation loss has two main components:
  • Pixel-level Feature Similarity: This term directly encourages the student's feature maps to match the teacher's feature maps, ensuring that low-level features are transferred. A 1x1 convolution is used to match channel dimensions if they differ.
  • Local Similarity Preservation: This is the novel part. It enforces that the local similarity calculated from the student's feature maps should match the local similarity calculated from the teacher's feature maps. This ensures that the student learns not just the absolute feature values but also the relationships between neighboring features, which is crucial for dense prediction tasks.
  • Overall Loss: The final loss function combines the standard ground-truth loss (for accurate depth prediction), a prediction loss (comparing teacher and student predictions), and the LSPKD loss.

Datasets

The paper evaluates their method on two standard datasets for depth completion:

  • KITTI Depth Completion (KITTI DC): This dataset provides approximately 86K RGB and LiDAR depth images for training and 7K for validation. It's known for its semi-dense ground-truth depth data.
  • NYU Depth V2 (NYUv2): This dataset contains around 50K RGB and depth images for training and 1.5K for evaluation. It's often used for indoor scene depth estimation.

Key Insights and Ideas

  • Local Similarity is Key for Depth: The authors hypothesize and demonstrate that local spatial relationships are more critical for accurate depth completion than global image similarities. This is because depth tends to be continuous within local regions of a scene, except at object boundaries.
  • Efficiency without Sacrificing Accuracy: LSPKD allows a significantly smaller student network (e.g., ~16.53% of the teacher's parameters on KITTI) to achieve performance comparable to, or even better than, larger models or previous distillation methods.
  • Robustness to Sparsity: The method shows remarkable robustness to sparse ground-truth data. Even when trained with extremely sparse supervision (approaching self-supervision), LSPKD still outperforms other distillation techniques, highlighting the strength of learning local relationships.
  • Mid-Level Features Matter Most for Distillation: Ablation studies suggest that distilling knowledge from mid-level encoder and decoder layers yields the best performance improvements, indicating that these layers capture the most relevant information for depth completion.

Key Results

  • KITTI DC: The proposed LSPKD method achieved superior performance compared to other lightweight distillation methods (PROB, ATT) and even outperformed the original teacher network in some metrics when considering the parameter reduction. For instance, the student network with LSPKD achieved an RMSE of 893.0 mm, which is competitive with the teacher's 865.2 mm, while being significantly smaller.
  • NYUv2: On this dataset, LSPKD also demonstrated significant improvements over a student network trained from scratch and was sufficient on its own, unlike other methods that sometimes required combinations with other techniques. The paper notes that LSPKD is sufficient for highly lightweight network distillation.
  • Computational Savings: The method significantly reduces computational costs during both training and inference. The paper highlights that LSPKD requires substantially less GPU memory for distillation compared to global similarity-preserving methods.

Limitations and Future Work

  • Limitations: While the paper demonstrates robustness to sparse supervision, the exact density of the ground truth can still impact performance. The authors also note that the choice of distillation layers can affect performance, and they identified mid-level layers as optimal for their architecture.
  • Future Work: The authors suggest exploring various similarity metrics for local similarity estimation. They also propose that their method could be extended to other computer vision tasks that benefit from local spatial relationships.

Summary updated

Abstract

Depth perception capability is one of the essential requirements for various autonomous driving platforms. However, accurate depth estimation in a real-world setting is still a challenging problem due to high computational costs. In this paper, we propose a lightweight depth completion network for depth perception in real-world environments. To effectively transfer a teacher’s knowledge, useful for the depth completion, we introduce local similarity-preserving knowledge distillation (LSPKD), which allows similarities between local neighbors to be transferred during the distillation. With our LSPKD, a lightweight student network is precisely guided by a heavy teacher network, regardless of the density of the ground-truth data. Experimental results demonstrate that our method is effective to reduce computational costs during both training and inference stages while achieving superior performance over other lightweight networks.
BibTeX
@inproceedings{jeong2022lightweight,
  author = {Yongseop Jeong and Jinsun Park and Donghyeon Cho and Yoonjin Hwang and Seibum B. Choi and In So Kweon},
  title = {{Lightweight Depth Completion Network with Local Similarity-Preserving Knowledge Distillation}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=SJCDJxrBgE},
}
Figure