Abstract
Monocular Depth Estimation (MDE) is a fundamental yet ill-posed problem in computer vision, as a single 2D image can correspond to infinite 3D configurations. Existing approaches typically utilize supervised, unsupervised, or weakly supervised learning, but each faces challenges regarding data scarcity, scale ambiguity, or label noise. We introduce SUW-Learn, a framework that jointly optimizes these three objectives. By integrating a novel patch-based semantic loss and accounting for both ego-motion and object-motion, SUW-Learn achieves state-of-the-art results on the KITTI benchmark and exhibits robust generalization across diverse real-world environments.
Introduction
Predicting 3D depth from a single 2D image is critical for applications like autonomous driving and augmented reality. However, the task is inherently ambiguous. Researchers have historically relied on three distinct strategies:
- Supervised Learning (S): Accurate but data-hungry, requiring expensive LiDAR ground truth.
- Unsupervised Learning (U): Uses temporal consistency in video but struggles with absolute scale and moving objects.
- Weakly Supervised Learning (W): Employs ordinal relationships (e.g., "A is closer than B"), which are easier to obtain but often spatially inconsistent.
SUW-Learn addresses these limitations by unifying these paradigms into a single, cohesive training objective that leverages the strengths of each while mitigating their individual weaknesses.
Methodology: The SUW Framework
The core of SUW-Learn is a joint optimization strategy where the total loss is defined as a weighted sum of supervised, unsupervised, and weakly supervised components: $\mathcal{L} = \mathcal{L}_s + \alpha\mathcal{L}_u + \beta\mathcal{L}_w$.
Learning Modules
- Supervised Module: Performs standard depth regression or soft-classification against ground-truth depth data.
- Unsupervised Module: Enhances geometric consistency by modeling both camera (ego-motion) and independent object motion. This prevents the model from misinterpreting moving objects as depth artifacts, a common failure mode in unsupervised MDE.
- Weakly Supervised Module: Introduces a novel "patch-based" semantic loss. Using an off-the-shelf semantic segmentation network, the model identifies regions (e.g., "road," "sky") and enforces depth similarity within classes and ordinal constraints between classes (e.g., the sky must always be farther than trees).
Importantly, the unsupervised and weakly supervised components act as "scaffolding" during training and are discarded during inference, ensuring the final model remains computationally efficient for real-time applications.
Datasets and Evaluation
The model was evaluated on the KITTI dataset, the industry standard for autonomous driving, and a custom M&M Dataset. The M&M dataset combines MegaDepth (internet photos of landmarks) and the Mannequin Challenge (videos of static people), providing a diverse training set that spans wide-angle outdoor vistas and complex human-centric indoor scenes.
Results and Discussion
SUW-Learn demonstrates significant improvements over baseline models. Ablation studies reveal a clear performance trend: $S < SU < SUW$, indicating that the combination of all three learning strategies yields the lowest error rates.
On the KITTI Eigen split, SUW-Learn achieved a Root Mean Square Error (RMSE) of 1.915 meters, significantly outperforming established supervised models like DORN (2.930m) and BTS (2.815m). Furthermore, the model's performance on the MegaDepth test set (scale-invariant RMSE of 0.100) underscores its ability to generalize across domains better than specialized models trained on single datasets.
Conclusion
SUW-Learn proves that combining supervised, unsupervised, and weakly supervised learning leads to a more accurate and robust monocular depth estimator. By leveraging semantic priors and motion modeling, the framework overcomes the traditional "brittleness" of MDE models, paving the way for more reliable depth perception in dynamic, real-world environments.
Summary updated