Boosting LightWeight Depth Estimation via Knowledge Distillation

Junjie Hu, Chenyou Fan, Hualie Jiang, Xiyue Guo, Yuan Gao, Xiangyong Lu, Tin Lun Lam

TL;DR

This paper tackles the accuracy-efficiency trade-off in monocular depth estimation by proposing a lightweight network that achieves state-of-the-art performance with significantly fewer parameters. They achieve this by combining a compact network architecture with knowledge distillation, enhanced by the strategic use of auxiliary unlabeled data to bridge the performance gap between a large teacher model and their small student model.

What Problem Does This Paper Address?

Estimating depth from a single image (monocular depth estimation or MDE) is crucial for many applications like robotics, autonomous driving, and augmented reality. However, existing MDE methods often face a dilemma: either they are highly accurate but computationally expensive, requiring significant processing power and memory, or they are lightweight and efficient but sacrifice accuracy. This makes them unsuitable for real-world applications with limited computational resources. The paper aims to bridge this gap by developing a depth estimation model that is both highly accurate and computationally efficient.

Methods

The core of the proposed solution lies in two main components: a novel lightweight network architecture and an enhanced knowledge distillation (KD) strategy.

  • Lightweight Network Architecture: The authors designed a compact network that significantly reduces model complexity. Unlike traditional encoder-decoder structures, their approach compresses features from multiple encoder layers into a fixed number of channels at each scale, then upsamples them. These compressed features are concatenated and fed into a few convolutional layers to produce the final depth map. This architecture, built upon MobileNet-v2, boasts a mere 1.7 million parameters.
  • Knowledge Distillation with Auxiliary Data: To boost the performance of their lightweight "student" network, they employ knowledge distillation. A larger, pre-trained "teacher" network (e.g., ResNet-34) is used to guide the student's learning. The novelty here is the use of auxiliary unlabeled data to improve the KD process. The intuition is that while the student network has a much smaller capacity, providing it with more diverse data (even if unlabeled) helps it better mimic the teacher's predictions and learn the underlying depth estimation mapping more effectively. They propose two strategies:
  • Using auxiliary unlabeled data alongside the original labeled data for KD.
  • Using auxiliary labeled data to train a more discriminative teacher, which then guides the student via KD.

Datasets

  • Training and Evaluation: The primary dataset used is NYU-v2, a widely used indoor RGB-D dataset.
  • Auxiliary Data:
    • Unlabeled: ScanNet dataset was used as a source of auxiliary unlabeled data. The paper highlights that ScanNet shares similar scene scales and depth histograms with NYU-v2, making it a suitable cross-domain dataset.
  • Labeled: While not the primary focus for the auxiliary data strategy, the paper mentions using ImageNet for pre-training and for preliminary experiments to demonstrate the importance of domain similarity.

Key Insights and Ideas

  • Auxiliary Data is Key for Lightweight KD: The paper strongly suggests that auxiliary data, especially when it shares similar scene characteristics (like scale and depth distribution) with the target domain, is crucial for effective knowledge distillation with lightweight models. It helps bridge the significant capacity gap between a large teacher and a small student.
  • Domain Similarity Matters: Using auxiliary data from a very different domain (like ImageNet for indoor depth estimation) can be detrimental, leading to a significant drop in performance. This underscores the importance of selecting auxiliary data that aligns with the target task's domain.
  • KD Outperforms Supervised Learning for Lightweight Models: Even with auxiliary data, directly training a lightweight model in a supervised manner shows limited improvement compared to using KD. KD, especially when enhanced with auxiliary data, proves more effective in transferring knowledge and boosting performance.
  • Compact Architecture Design is Viable: The paper demonstrates that a carefully designed lightweight architecture can achieve competitive results, challenging the notion that high accuracy inherently requires massive models.

Key Results

  • Parameter Efficiency: The proposed lightweight network uses only 1.7 million parameters, which is about 1% of the parameters used by state-of-the-art large networks (e.g., ResNet-50 with ~60M parameters).
  • Performance: The method achieves comparable performance to state-of-the-art methods on the NYU-v2 dataset, even outperforming some larger models in certain metrics. For instance, their method achieves a $\delta_1$ accuracy of 0.855 using auxiliary labeled data, which is significantly higher than other lightweight methods like Fast-depth (0.775) and Joint-depth (0.790).
  • Inference Speed: The lightweight network achieves very fast inference times, requiring only 11 ms of GPU time per image, making it suitable for real-time applications.
  • Generalizability: The model shows good generalization performance on cross-dataset evaluations on SUNRGBD and TUM datasets, outperforming existing methods.

Limitations and Future Work

  • Auxiliary Data Selection: While the paper shows the effectiveness of auxiliary data, it relies on randomly selecting data from ScanNet. The authors suggest that more carefully selected auxiliary data, similar to the approach in [3], could further improve performance.
  • Fairness of Cross-Dataset Comparison: The paper acknowledges that comparing their method using auxiliary labeled data with other methods on new datasets might not be entirely fair, as their approach leverages more information.
  • Future Work: The paper implicitly suggests exploring more sophisticated ways to select and utilize auxiliary data, potentially including semi-supervised learning techniques that can leverage even larger amounts of unlabeled data. Further research into optimizing the KD process for even larger capacity gaps could also be a direction.

Summary updated

Abstract

Monocular depth estimation (MDE) methods are often either too computationally expensive or not accurate enough due to the trade-off between model complexity and inference performance. In this paper, we propose a lightweight network that can accurately estimate depth maps using minimal computing resources. We achieve this by designing a compact model that maximally reduces model complexity. To improve the performance of our lightweight network, we adopt knowledge distillation (KD) techniques. We consider a large network as an expert teacher that accurately estimates depth maps on the target domain. The student, which is the lightweight network, is then trained to mimic the teacher’s predictions. However, this KD process can be challenging and insufficient due to the large model capacity gap between the teacher and the student. To address this, we propose to use auxiliary unlabeled data to guide KD, enabling the student to better learn from the teacher’s predictions. This approach helps fill the gap between the teacher and the student, resulting in improved data-driven learning. The experiments show that our method achieves comparable performance to state-of-the-art methods while using only 1% of their parameters. Furthermore, our method outperforms previous lightweight methods regarding inference accuracy, computational efficiency, and generalizability.
BibTeX
@inproceedings{hu2023boosting,
  author = {Junjie Hu and Chenyou Fan and Hualie Jiang and Xiyue Guo and Yuan Gao and Xiangyong Lu and Tin Lun Lam},
  title = {{Boosting LightWeight Depth Estimation via Knowledge Distillation}},
  booktitle = {KSEM},
  year = {2023},
  doi = {10.1007/978-3-031-40283-8\_3},
}
Figure