Enhancing Monocular Depth Estimation with Multi-Source Auxiliary Tasks

Alessio Quercia, Erenus Yildiz, Zhuo Cao, Kai Krajsek, Abigail Morrison, Ira Assent, Hanno Scharr

Abstract

Monocular Depth Estimation (MDE) is a critical task for robotics and augmented reality, yet it remains constrained by the high cost of acquiring dense depth labels. This paper presents a data-efficient training framework that leverages auxiliary tasks—such as semantic segmentation and dense classification—from unrelated datasets to improve MDE performance. By utilizing a shared decoder on a frozen DINOv2 backbone, the proposed method achieves an average 11% improvement in depth quality and demonstrates that auxiliary tasks can reduce the requirement for depth-labeled data by up to 99%.

Introduction

Predicting 3D depth from a single 2D image is a fundamental challenge in computer vision. While foundation models like DINOv2 provide robust feature representations, fine-tuning them for specific tasks like MDE is computationally expensive and often hindered by the scarcity of high-quality depth labels. Traditional Multi-Task Learning (MTL) approaches frequently encounter "task interference," where optimizing for one objective degrades another. This research explores how to effectively harness abundant, non-depth datasets to enhance depth prediction without full model fine-tuning.

Proposed Methodology

The authors introduce a lightweight pipeline designed for maximum data and computational efficiency. The architecture consists of three main components:

  1. Frozen Foundation Backbone: A pre-trained DINOv2 ViT-Giant serves as the feature extractor. By keeping this backbone frozen, the training focus shifts entirely to the task-specific components, drastically reducing resource requirements.
  2. Shared Dense Prediction Transformer (DPT) Decoder: Unlike standard MTL architectures that use separate decoders for different tasks, this method employs a shared decoder. This forces the model to learn a unified feature representation that is beneficial for both depth estimation and auxiliary tasks.
  3. Alternating Training Scheme: The model alternates between MDE and auxiliary task updates. A task-focusing parameter, $\alpha$, regulates the gradient flow, typically prioritizing depth estimation ($\alpha = 0.9$).
The proposed architecture utilizes a frozen backbone and a shared decoder to learn from both depth and auxiliary inputs simultaneously.

A key innovation is the use of Multi-Label Dense Classification (MLDC) as an auxiliary task. Rather than requiring pixel-perfect segmentation masks, MLDC identifies the presence of various classes within a scene. This global semantic understanding provides powerful spatial cues that significantly aid depth perception.

Key Insights and Results

The research yields several surprising findings regarding the relationship between semantic understanding and depth:

  • Classification vs. Segmentation: MLDC often outperforms full semantic segmentation as an auxiliary task. This suggests that for depth estimation, understanding the scene's composition (e.g., "there is a bed in the foreground") is more valuable than precise boundary localization, which can often be noisy.
  • Auxiliary Tasks as Data Augmentation: Auxiliary datasets act as a sophisticated form of augmentation. Learning to identify objects in diverse environments helps the shared decoder internalize spatial relationships that generalize back to the depth task.
  • Exceptional Data Efficiency: Perhaps the most significant result is the reduction in required depth data. By incorporating auxiliary tasks, the model can match or exceed baseline performance using as little as 1% to 20% of the original depth-labeled training set.
Performance comparison across various datasets shows that the proposed method (blue) consistently outperforms the baseline (red), especially when training data is scarce.

Quantitatively, the method achieved an average 11% improvement over the DINOv2 baseline. Significant gains were observed in challenging scenarios, such as a 22.2% improvement on the DIODE Outdoor dataset and a 13.9% gain on NYUv2. Even when applied to state-of-the-art models like "Depth Anything," the approach yielded a 2.5% average performance boost.

Limitations and Future Outlook

The primary constraint identified is the domain gap. While the "MIX6" auxiliary dataset (a union of indoor and general-purpose datasets) significantly improved performance on indoor tasks, it offered no benefit for the KITTI autonomous driving dataset. This indicates that the effectiveness of auxiliary tasks is contingent on the relevance of the auxiliary data to the target domain.

Future research will focus on automating the selection of auxiliary tasks and exploring more diverse datasets to bridge the gap in specialized domains like autonomous driving.

Summary updated

Abstract

Monocular depth estimation (MDE) is a challenging task in computer vision, often hindered by the cost and scarcity of high-quality labeled datasets. We tackle this challenge using auxiliary datasets from related vision tasks for an alternating training scheme with a shared decoder built on top of a pre-trained vision foundation model, while giving a higher weight to MDE. Through extensive experiments we demonstrate the benefits of incorporating various in-domain auxiliary datasets and tasks to improve MDE quality on average by ~11%. Our experimental analysis shows that auxiliary tasks have different impacts, confirming the importance of task selection, highlighting that quality gains are not achieved by merely adding data. Remarkably, our study reveals that using semantic segmentation datasets as Multi-Label Dense Classification (MLDC) often results in additional quality gains. Lastly, our method significantly improves the data efficiency for the considered MDE datasets, enhancing their quality while reducing their size by at least 80%. This paves the way for using auxiliary data from related tasks to improve MDE quality despite limited availability of high-quality labeled data. Code is available at https://jugit.fz-juelich.de/ias-8/mdeaux.
BibTeX
@inproceedings{quercia2025enhancing,
  author = {Alessio Quercia and Erenus Yildiz and Zhuo Cao and Kai Krajsek and Abigail Morrison and Ira Assent and Hanno Scharr},
  title = {{Enhancing Monocular Depth Estimation with Multi-Source Auxiliary Tasks}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00627},
}
Figure