Monocular Depth Estimation via Self-Supervised Self-Distillation

Haifeng Hu, Yuyang Feng, Dapeng Li, Suofei Zhang, Haitao Zhao2024 · DOI

TL;DR

This paper introduces a novel self-supervised monocular depth estimation (SS-MDE) method that leverages self-distillation and a multi-view mask filtering module to improve accuracy in dynamic scenes. By combining pseudo-depth supervision with an iterative teacher-student approach, the method effectively handles occlusions and motion blur, achieving state-of-the-art results on challenging datasets.

What Problem Does This Paper Address?

Estimating depth from a single monocular camera is a fundamental yet challenging task in computer vision, with applications ranging from autonomous driving to augmented reality. Traditional supervised methods require large amounts of manually annotated depth data, which is expensive and time-consuming to acquire. Self-supervised methods offer a solution by learning from the inherent structure of images and geometric relationships, but they struggle with dynamic scenes. Moving objects, occlusions, and motion blur can lead to inconsistent depth predictions and inaccurate estimations. Existing self-supervised approaches often rely on multi-view consistency, which breaks down in the presence of dynamic elements. This paper aims to bridge this gap by developing a self-supervised method that is robust to these dynamic challenges.

Methods

The proposed SS-MDE method integrates several key components:

  • Self-Supervised Depth Estimation Network: A deep network with a multi-scale encoder-decoder architecture and a lightweight pose network is used to predict depth. This network leverages disparity, motion information, and the association between adjacent frames for self-supervision.
  • Pseudo-Depth Supervision: The LeReS network is used to generate pseudo-depth images, providing a rough initial depth estimation without requiring ground truth. This pseudo-depth is used to refine static areas and improve depth estimation accuracy.
  • Forgetting Factor: A dynamically decaying weight is introduced to gradually reduce the model's reliance on pseudo-depth supervision over training epochs, enhancing robustness.
  • Self-Distillation with Multi-View Mask Filtering: A teacher-student paradigm is employed. The student network learns from a teacher network (the student network from the previous epoch). A crucial addition is the multi-view mask filtering module, which filters out unreliable depth and normal information from the teacher model. This module specifically targets outliers and uncertain predictions caused by dynamic factors or noise, allowing the student to focus on more reliable features.
  • Loss Functions: The total loss comprises photometric loss, smoothness loss, pseudo-depth matching loss, and the self-distillation loss (combining depth and normal distillation).

Datasets

The paper evaluates the SS-MDE method on several widely used datasets:

  • KITTI: A benchmark for autonomous driving, containing outdoor scenes with vehicles and pedestrians.
  • NYU-Depth V2: A popular dataset for indoor depth estimation, though it primarily consists of static scenes.
  • BONN: A dataset specifically designed for dynamic indoor environments, focusing on scenarios with significant motion.
  • TUM: A dataset used for robot vision and SLAM research, also featuring dynamic elements.

Key Insights and Ideas

  • Self-Distillation for Dynamic Scenes: The core innovation is applying self-distillation not just for general performance improvement but specifically to enhance robustness in dynamic environments. The teacher-student setup, combined with sophisticated filtering, allows for knowledge transfer that is less susceptible to the noise introduced by motion.
  • Multi-View Mask Filtering is Crucial: The multi-view mask filtering module is highlighted as a key enabler for handling dynamic scenes. By intelligently filtering out unreliable teacher predictions, it prevents the propagation of errors and allows the student to learn more accurate representations.
  • Pseudo-Depth Refinement with Forgetting: Using pseudo-depth from LeReS is a smart way to inject supervision without ground truth. The decaying weight mechanism is essential to prevent over-reliance on potentially noisy pseudo-labels, ensuring the model doesn't become brittle.
  • Balancing Static and Dynamic: The method implicitly aims to balance performance on static regions (where traditional self-supervised methods excel) with dynamic regions (where they struggle), achieving a more generalized solution.

Key Results

The SS-MDE method demonstrates superior performance across multiple datasets:

  • KITTI: Achieved an AbsRel of 0.102 and an accuracy (δ₁) of 0.892, outperforming several state-of-the-art self-supervised methods.
  • NYU-Depth V2: Achieved an AbsRel of 0.111 and an accuracy (δ₁) of 0.873.
  • BONN and TUM: Showed significant improvements in dynamic regions compared to other self-supervised methods, with SS-MDE performing best overall on these datasets.
  • Ablation Studies: Confirmed the effectiveness of each component, including the multi-scale decoder, self-distillation loss, pseudo-depth matching loss, and the decaying weight γt.
  • Backbone Comparison: MPViT backbones, particularly MPViT-S, showed excellent performance, often surpassing ResNet-18 with fewer parameters and lower computational complexity.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Computational Cost: The self-distillation process, while effective, incurs a significant computational cost.
  • Real-World Application Challenges: Multi-view mask filtering may not fully address all real-world scenarios, especially those with extreme lighting changes or very fast-moving objects.
  • Future Work: The authors plan to focus on developing more lightweight models for deployment on resource-constrained platforms. They also aim to further reduce the costs associated with the self-distillation operation.

Summary updated

BibTeX
@misc{hu2024monocular,
  author = {Haifeng Hu and Yuyang Feng and Dapeng Li and Suofei Zhang and Haitao Zhao},
  title = {{Monocular Depth Estimation via Self-Supervised Self-Distillation}},
  year = {2024},
  doi = {10.3390/s24134090},
}
Figure