Can Scale-Consistent Monocular Depth Be Learned in a Self-Supervised Scale-Invariant Manner?

Lijun Wang, Yifan Wang, Linzhao Wang, Yunlong Zhan, Ying Wang, Huchuan Lu

Abstract

Self-supervised monocular depth estimation often suffers from scale inconsistency across video frames due to the inherent ambiguity of single-view geometry. This paper introduces a novel framework that disentangles scene structure from global scale using a two-stream DepthNet. By incorporating a Scale-Aware Geometric (SAG) loss that aligns 3D point clouds between frames, the model learns to maintain a consistent scale without ground truth supervision. Evaluated on the KITTI dataset, the proposed method achieves state-of-the-art accuracy and significantly improves temporal scale stability, benefiting downstream tasks like ego-motion estimation.

Introduction

Monocular depth estimation—predicting 3D depth from a single 2D image—is a fundamental challenge in computer vision due to scale ambiguity. Without a reference, it is impossible to distinguish between a small object nearby and a large object far away. In self-supervised learning, where models learn by reconstructing video frames, this ambiguity often leads to "jittery" scale predictions that vary from frame to frame. While scale-invariant losses can improve relative shape estimation, they fail to ensure that the global scale remains consistent over time, which is critical for applications like autonomous driving and robotics.

Methodology

The proposed framework addresses scale inconsistency through two primary innovations: a disentangled network architecture and a novel geometric loss function.

Two-Stream DepthNet

Instead of a single network predicting absolute depth, the authors split the task into two streams. One stream focuses on predicting normalized depth, capturing the relative geometric structure of the scene. A second, lightweight stream predicts a global scale factor ($\mu$). This separation prevents the model from being confused by fluctuating scales during the learning process.

Scale-Aware Geometric (SAG) Loss

The Scale-Aware Geometric (SAG) loss is the core mechanism for ensuring scale consistency. Unlike standard photometric losses that operate on 2D pixels, the SAG loss works in 3D space. It projects the depth maps of adjacent frames into 3D point clouds and performs an alignment. This alignment process calculates a relative scale factor ($\tau$) between the frames. In a perfectly scale-consistent model, $\tau$ should be 1. Any deviation from 1 is used as an explicit supervision signal to train the scale stream to maintain consistency.

The proposed self-supervised framework featuring the Two-Stream DepthNet and Scale-Aware Geometric (SAG) loss.

Results and Evaluation

The model was evaluated on the KITTI dataset, the standard benchmark for autonomous driving. The results demonstrate that the proposed method achieves state-of-the-art performance across multiple metrics.

Quantitative Performance

On the KITTI Depth Benchmark, the method outperformed previous leading models such as Monodepth2 and PackNet-SfM. It achieved an Absolute Relative Error (Abs Rel) of 0.109, indicating high precision in depth prediction. Furthermore, accuracy metrics ($\delta < 1.25$) showed significant improvements over the baseline, confirming the effectiveness of the disentangled approach.

Qualitative Consistency

Qualitative analysis of the depth maps reveals sharper object boundaries and more stable depth values across sequences. Because the global scale is stabilized, the ego-motion network—which predicts the camera's movement—produces much smoother and more accurate trajectories, rivaling traditional SLAM methods.

Comparison of error, accuracy, and depth map quality between the proposed method and the baseline.

Conclusion

By disentangling scale from structure and introducing the Scale-Aware Geometric loss, this research provides a robust solution to the problem of scale inconsistency in self-supervised monocular depth estimation. The method not only improves per-frame depth accuracy but also ensures temporal stability, making it highly suitable for real-world robotics and autonomous navigation. Future work may involve integrating semantic information to further refine scale inference in dynamic environments with moving objects.

Summary updated

Abstract

Geometric constraints are shown to enforce scale consistency and remedy the scale ambiguity issue in self-supervised monocular depth estimation. Meanwhile, scale-invariant losses focus on learning relative depth, leading to accurate relative depth prediction. To combine the best of both worlds, we learn scale-consistent self-supervised depth in a scale-invariant manner. Towards this goal, we present a scale-aware geometric (SAG) loss, which enforces scale consistency through point cloud alignment. Compared to prior arts, SAG loss takes relative scale into consideration during relative motion estimation, enabling more precise alignment and explicit supervision for scale inference. In addition, a novel two-stream architecture for depth estimation is designed, which disentangles scale from depth estimation and allows depth to be learned in a scale-invariant manner. The integration of SAG loss and two-stream network enables more consistent scale inference and more accurate relative depth estimation. Our method achieves state-of-the-art performance under both scale-invariant and scale-dependent evaluation settings.
BibTeX
@inproceedings{wang2021can,
  author = {Lijun Wang and Yifan Wang and Linzhao Wang and Yunlong Zhan and Ying Wang and Huchuan Lu},
  title = {{Can Scale-Consistent Monocular Depth Be Learned in a Self-Supervised Scale-Invariant Manner?}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01249},
}
Figure