TL;DR
This paper introduces a novel self-supervised learning framework for monocular depth and visual odometry estimation. By incorporating "scale-consistent geometric constraints" that align depth scales between consecutive frames and enforce pose consistency, the method achieves state-of-the-art performance on challenging datasets without requiring ground truth.
What Problem Does This Paper Address?
Estimating depth and camera motion (visual odometry) from monocular (single camera) video is crucial for applications like autonomous driving and augmented reality. While deep learning has made strides, self-supervised methods, which learn without explicit ground truth depth or pose, often struggle with two main issues:
- Photometric Fragility: Relying solely on matching pixel colors (photometric consistency) between frames is unreliable in real-world scenarios due to varying lighting conditions and reflective surfaces.
- Scale Inconsistency: Self-supervised depth estimation inherently suffers from an ambiguous scale. Without a known reference, the predicted depth can be arbitrarily scaled, and this inconsistency propagates to visual odometry, making it difficult to obtain accurate, globally consistent camera poses.
This paper aims to overcome these limitations by introducing new geometric constraints that enforce consistency in both depth and pose estimations.
Methods
The core of this work lies in its novel "scale-consistent geometric constraints" and the framework to implement them.
- Scale-Consistent Depth Reconstruction Loss: Instead of directly comparing predicted depth maps, the authors first align the scales of depth maps from consecutive frames. This is achieved by normalizing them using their mean values. Then, a loss term penalizes the difference between these normalized depth maps, ensuring that the estimated depth maintains a consistent scale across the video sequence.
- Forward-Backward Pose Consistency Loss: This loss leverages the idea that if you predict the relative pose from frame A to B ($T_{A \to B}$) and from frame B to C ($T_{B \to C}$), their composition should ideally match the direct prediction from A to C ($T_{A \to C}$). The authors extend this by considering both forward ($T_{t \to t+1}$) and backward ($T_{t \to t-1}$) predictions, enforcing consistency between them. This helps to stabilize and improve the accuracy of the predicted camera poses.
- Joint Training Framework: The depth and pose estimation networks are trained jointly. The framework uses a standard encoder-decoder architecture for depth prediction (DispNet) and a PoseNet for pose estimation. The proposed geometric losses are integrated alongside traditional photometric and smoothness losses.
Datasets
- KITTI: This is the primary dataset used for both training and evaluation. The authors utilize the Eigen split for monocular depth estimation and the Odometry dataset for visual odometry.
- Make3D: Used to evaluate the generalization ability of the depth estimation model on unseen environments.
- Cityscapes: Used for pre-training the models, which is shown to improve performance on KITTI.
No new benchmarks were introduced; existing splits and datasets were used.
Key Insights and Ideas
- Geometric Constraints are Key: The paper strongly advocates for moving beyond purely appearance-based losses (photometric consistency) and incorporating geometric information for more robust self-supervised depth and VO.
- Scale Alignment is Crucial for Depth and VO: Explicitly aligning depth scales between frames is not just important for depth accuracy but also directly addresses the scale ambiguity that plagues self-supervised visual odometry.
- Forward-Backward Consistency is Powerful: Enforcing consistency in both directions of motion (forward and backward) provides a strong geometric signal for pose estimation.
- Joint Training Benefits Both Tasks: While depth and pose networks are tested independently, their joint training with these new constraints leads to improved performance in both tasks.
Key Results
- State-of-the-Art Depth Estimation: The proposed method achieves state-of-the-art (SOTA) performance on the KITTI Eigen split for monocular depth estimation among self-supervised methods, outperforming previous approaches like SfMLearner and SC-SfMLearner. For example, on the KITTI Eigen test split (80m cap), the proposed method achieves an Abs Rel of 0.126, Sq Rel of 0.902, and RMSE of 5.052.
- SOTA Visual Odometry: The method also achieves SOTA pose estimation results on the KITTI Odometry dataset compared to other self-supervised methods, demonstrating improved accuracy in both translation and rotation errors.
- Improved Qualitative Results: Figure 3 shows qualitative comparisons where the proposed method produces more detailed depth maps with fewer "black holes" (areas where depth is not predicted) compared to baselines. Figure 4 illustrates more accurate and stable visual odometry trajectories.
- Generalization: The model shows good generalization capabilities on the Make3D dataset, even without fine-tuning.
Limitations and Future Work
- Limitations: The paper acknowledges that while their method improves upon previous works, it still relies on photometric consistency, which can be fragile in extreme illumination changes. The scale alignment is done using mean values, which might not be optimal in all scenarios.
- Future Work: The authors suggest exploring the fusion of higher-level semantic features for supervision, which could further enhance robustness. They also propose integrating their self-supervised learning ideas into traditional visual SLAM systems to achieve more robust and accurate pose estimation.
Summary updated