TL;DR
This paper introduces two novel modules, the Inter-task Attention Module (IAM) and the Spatial-Temporal Memory Module (STMM), to improve self-supervised monocular depth estimation by learning richer geometric representations from video. These modules enable better interaction between depth and pose estimation networks and leverage long-range spatial-temporal context, leading to state-of-the-art performance on the KITTI dataset.
What Problem Does This Paper Address?
Estimating depth from a single camera (monocular depth estimation) is a fundamental problem in computer vision with applications ranging from autonomous driving to robotics. While self-supervised methods that jointly learn depth and camera pose from video have shown promise, they often rely on simple photometric losses and don't fully exploit the rich geometric information present in video sequences. Specifically, the paper identifies a gap in learning effective task-specific geometry representations that can mutually benefit depth and pose estimation, and in leveraging long-range spatial and temporal context within video frames to improve perception. Existing methods tend to treat depth and pose estimation networks independently until the final loss calculation, missing opportunities for deeper inter-task collaboration.
Methods
The paper proposes two key novel modules:
- Inter-task Attention Module (IAM): This module aims to capture the geometric correlation between depth and pose estimation. It learns attention maps from depth features to guide the pose estimation network (both rotation and translation branches) on which features to prioritize. This allows the pose network to learn more valuable representations, which, through gradient back-propagation, also benefits the depth estimation network. The IAM uses pooling and convolution to generate these attention maps.
- Spatial-Temporal Memory Module (STMM): Inspired by human perception, this module leverages long-range spatial and temporal dependencies within video sequences. It uses a Non-local network-based attention mechanism to aggregate geometric context from consecutive frames. This helps in disambiguating scenes and achieving more consistent depth estimation, especially for distant objects.
The overall architecture combines these modules with a standard encoder-decoder network for depth estimation (based on ResNet18/50) and a pose estimation network. The IAM is applied to the pose network's features, while the STMM is applied to the depth network's encoded features.
Datasets
- KITTI: Used for both training and evaluation of the proposed method. This is a standard benchmark for autonomous driving datasets.
- Make3D: Used to evaluate the generalization ability of the model to unseen datasets.
- Cityscapes: Also used to evaluate generalization ability, particularly for qualitative comparisons.
The paper also mentions using an "annotated depth map" from the KITTI dataset for evaluation, which tackles tricky cases like occlusion and object motion, differing from the standard reprojected LiDAR points ground truth.
Key Insights and Ideas
- Mutual Benefit of Depth and Pose: The core idea is that depth and pose estimation are not independent tasks. By explicitly learning how depth information can guide pose estimation (and vice-versa through gradients), both tasks can be significantly improved.
- Attention for Inter-Task Guidance: Using attention mechanisms to transfer information between the depth and pose networks is a novel approach. The IAM learns that different spatial regions are important for rotation versus translation, demonstrating a nuanced understanding of geometric cues.
- Memory for Long-Range Context: Mimicking human memory, the STMM effectively captures temporal and spatial relationships across frames, which is crucial for understanding scene structure beyond immediate visual input. This is particularly beneficial for handling distant objects and improving overall scene understanding.
- Lightweight Design: Despite incorporating these advanced modules, the paper emphasizes a lightweight network design, aiming for efficiency.
Key Results
- State-of-the-Art Performance: The proposed method achieves state-of-the-art results on the KITTI dataset for self-supervised monocular depth estimation, outperforming previous methods across various metrics (e.g., Abs Rel, RMSE log). For instance, the R50 model achieves an Abs Rel of 0.105 and RMSE log of 0.181.
- Improved Accuracy for Distant Objects: The STMM module significantly improves performance on distant objects, as shown in ablation studies.
- Effective Generalization: The model demonstrates good generalization ability to unseen datasets like Make3D and Cityscapes.
- Competitive Visual Odometry: The method also shows improved accuracy in visual odometry (pose estimation), achieving results close to traditional SLAM methods.
- Inference Speed: The model achieves a practical inference speed of over 10 fps on a GPU, making it suitable for real-time applications.
Limitations and Future Work
- Limitations:
- The paper acknowledges that while the model is trained on three-frame snippets, inference uses a single frame by repeating features. This might not fully exploit the temporal information during inference.
- CPU inference speed is significantly lower than GPU speed, indicating room for optimization.
- Future Work:
- The authors plan to explore more powerful network architectures, such as Transformers and their corresponding attention mechanisms, to further enhance performance.
- Improving CPU inference speed is also a stated goal.
Summary updated