TL;DR
This paper introduces AVT-SSDepth, a novel transformer-based architecture for supervised surround-view depth estimation. By leveraging an "adjacent-view attention" mechanism, it effectively exchanges information between neighboring camera views, leading to significantly improved depth prediction accuracy on challenging multi-camera datasets.
What Problem Does This Paper Address?
Depth estimation is crucial for robotics and autonomous driving, bridging 2D images with 3D environments. While monocular depth estimation has seen progress, it often relies on front-view cameras and ignores the rich contextual information available from multiple surrounding cameras. Existing multi-view methods either assume large overlaps between cameras (like MVS) or fail to fully exploit the inherent correlations in 360° setups. This paper addresses the gap in supervised surround-view depth estimation for wide-baseline, multi-camera systems with minimal overlap, aiming to build a robust baseline that can predict depth maps for all surrounding views simultaneously.
Methods
The proposed AVT-SSDepth architecture comprises three main components:
- Feature Extraction: A shared encoder that combines Convolutional Neural Networks (CNNs) with transformer layers. This hybrid approach aims to capture both local (CNN) and global (transformer) contextual information, creating enriched multi-scale feature representations.
- Adjacent-view Attention: This is the core innovation. It employs an alternating attention mechanism:
- Self-Attention: Operates within each individual camera view to refine features.
- Adjacent Attention: This novel module allows features to be exchanged between adjacent camera views. This is crucial for handling the limited overlap in surround-view setups, enabling the model to "see" beyond its immediate field of view by leveraging information from neighboring cameras. This process is repeated for multiple layers (Z times) to facilitate deep information exchange.
- Depth Head: A final module that takes the processed features and predicts the depth map for all surround-view cameras simultaneously.
The loss function combines a standard L1 depth loss with a smoothness penalty to encourage locally consistent depth predictions.
Datasets
The paper evaluates AVT-SSDepth on two challenging multi-camera datasets:
- nuScenes [10]: A widely used dataset for autonomous driving tasks, featuring 360° sensor data.
- DDAD [9]: A large-scale dataset specifically designed for long-range and dense depth estimation in multi-camera scenarios.
The authors also establish a benchmark on nuScenes for supervised surround-view depth estimation.
Key Insights and Ideas
- Adjacent-view attention is key: The paper strongly advocates for explicitly modeling interactions between adjacent views, rather than just all views at once. This is particularly effective in wide-baseline scenarios where direct overlap is minimal, allowing information to propagate across the 360° field of view.
- Hybrid CNN-Transformer encoder: Combining CNNs for local feature extraction with transformers for global context proves beneficial for creating rich representations needed for accurate depth estimation.
- Simultaneous multi-view prediction: Predicting all surround-view depth maps concurrently, rather than independently, allows the model to leverage inter-view correlations for improved accuracy.
- Cross-dataset generalization: The proposed method shows strong performance not only on its training datasets but also demonstrates good generalization capabilities in cross-dataset transfer experiments.
Key Results
AVT-SSDepth achieves state-of-the-art performance on both the nuScenes and DDAD datasets.
- On nuScenes, it significantly outperforms previous methods like SurroundDepth [21] and PixelFormer [11], achieving an Abs Rel score of 0.067.
- On DDAD, it also sets a new benchmark, demonstrating its effectiveness in challenging long-range depth estimation scenarios.
- In cross-dataset transfer experiments (DDAD to nuScenes), AVT-SSDepth achieves the lowest Abs Rel (0.292), Sq Rel (4.201), and RMSE (7.481), outperforming the previous best method R3D3 [66].
- Qualitative results show visually sharper and more accurate depth maps compared to other methods, especially in areas with less direct overlap.
Limitations and Future Work
- Computational Cost: While transformers offer advantages, they can be computationally intensive. The paper uses a specific backbone (MPViT-small) and notes the trade-offs in terms of FLOPs and parameters.
- Number of Attention Layers (Z): The ablation study shows that Z=8 is optimal for the adjacent-view attention layers. However, finding the ideal Z might be dataset-dependent.
- Future Work: The authors suggest that their work serves as a strong baseline for supervised surround-view depth estimation, implying that further architectural improvements and exploration of different loss functions could yield even better results. The effectiveness of adjacent-view attention across various backbones and its role in accelerating convergence also suggest avenues for future research.
Summary updated