Abstract
Self-supervised monocular methods can efficiently learn depth information of weakly textured surfaces or reflective objects. However, the depth accuracy is limited due to the inherent ambiguity in monocular geometric modeling. In contrast, multi-frame depth estimation methods improve the depth accuracy thanks to the success of Multi-View Stereo (MVS), which directly makes use of geometric constraints. Unfortunately, MVS often suffers from texture-less regions, non-Lambertian surfaces, and moving objects, especially in real-world video sequences without known camera motion and depth supervision. Therefore, we propose MOVEDepth, which exploits the MOnocular cues and VElocity guidance to improve multi-frame Depth learning. Unlike existing methods that enforce consistency between MVS depth and monocular depth, MOVEDepth boosts multi-frame depth learning by directly addressing the inherent problems of MVS. The key of our approach is to utilize monocular depth as a geometric priority to construct MVS cost volume, and adjust depth candidates of cost volume under the guidance of predicted camera velocity. We further fuse monocular depth and MVS depth by learning uncertainty in the cost volume, which results in a robust depth estimation against ambiguity in multi-view geometry. Extensive experiments show MOVEDepth achieves state-of-the-art performance: Compared with Monodepth2 and PackNet, our method relatively improves the depth accuracy by 20% and 19.8% on the KITTI benchmark. MOVEDepth also generalizes to the more challenging DDAD benchmark, relatively outperforming ManyDepth by 7.2%.
Introduction
Self-supervised depth estimation from video sequences typically relies on either monocular or multi-frame (MVS) approaches. Monocular methods excel at learning semantic cues, allowing them to handle challenging areas like reflective surfaces or textureless walls. However, they lack geometric precision. Conversely, multi-frame MVS methods leverage geometric triangulation for high accuracy but often fail in textureless regions, on moving objects, or when camera motion is insufficient to provide a reliable baseline.
MOVEDepth addresses these limitations by identifying a core "matching ambiguity" in MVS. By narrowing the depth search space using monocular priors and adaptive velocity guidance, the framework achieves high geometric precision while maintaining the robustness of monocular semantic understanding.
The MOVEDepth Framework
MOVEDepth integrates three primary innovations to enhance multi-frame depth learning:
- Monocular Depth Priority: Instead of an exhaustive search across 3D space, MOVEDepth uses a monocular depth map as a "geometric anchor." The MVS cost volume is centered around this prior, significantly reducing false geometric matches.
- Velocity-Guided Sampling: The reliability of multi-view geometry is mathematically linked to camera velocity. MOVEDepth adaptively scales the depth search range: at high velocities (large baseline), the range expands for precision; at low velocities (small baseline), it shrinks toward the reliable monocular prior to avoid noise.
- Uncertainty-Based Fusion: The system learns an uncertainty map based on the entropy of the MVS cost volume. In scenarios where geometry is ambiguous (e.g., moving objects), the model leans on monocular cues. When geometry is clear, it prioritizes the MVS result.
Efficiency and Geometric Logic
By utilizing the monocular prior, MOVEDepth only requires 16 depth candidates for its cost volume, compared to the much denser sampling required by traditional MVS. This efficiency allows the model to run at 32 FPS on modern hardware. Furthermore, the "Triangulation Prior" analysis justifies the adaptive sampling strategy, as per-pixel baselines are fundamentally more informative at higher velocities.
Experimental Evaluation
The framework was evaluated on two major autonomous driving benchmarks:
- KITTI: MOVEDepth achieved a 20% improvement in accuracy over Monodepth2 and 19.8% over PackNet, reaching the top of the self-supervised leaderboard.
- DDAD: In high-speed, long-range scenarios (up to 200m), MOVEDepth outperformed ManyDepth by 7.2% in absolute relative error.
Qualitatively, MOVEDepth produces cleaner edges and handles thin structures like poles better than monocular-only methods, while avoiding the artifacts typical of pure MVS approaches in dynamic scenes.
Discussion and Future Work
While MOVEDepth represents a significant advancement, it remains dependent on the initial quality of the monocular prior. If the monocular network fails significantly, the MVS search range may be misaligned. Future work could involve integrating more advanced backbones, such as Vision Transformers (ViTs), to further enhance accuracy. Additionally, while the uncertainty fusion mitigates issues with dynamic objects, extremely complex multi-object motion continues to challenge the underlying geometric assumptions of MVS.
Summary updated