Adjacent-view Transformers for Supervised Surround-view Depth Estimation

Xianda Guo, Wenjie Yuan, Yunpeng Zhang, Tian Yang, Chenming Zhang, Zheng Zhu, Qin Zou, Long Chen

TL;DR

This paper introduces AVT-SSDepth, a novel transformer-based architecture for supervised surround-view depth estimation. By leveraging an "adjacent-view attention" mechanism, it effectively exchanges information between neighboring camera views, leading to significantly improved depth prediction accuracy on challenging multi-camera datasets.

What Problem Does This Paper Address?

Depth estimation is crucial for robotics and autonomous driving, bridging 2D images with 3D environments. While monocular depth estimation has seen progress, it often relies on front-view cameras and ignores the rich contextual information available from multiple surrounding cameras. Existing multi-view methods either assume large overlaps between cameras (like MVS) or fail to fully exploit the inherent correlations in 360° setups. This paper addresses the gap in supervised surround-view depth estimation for wide-baseline, multi-camera systems with minimal overlap, aiming to build a robust baseline that can predict depth maps for all surrounding views simultaneously.

Methods

The proposed AVT-SSDepth architecture comprises three main components:

  • Feature Extraction: A shared encoder that combines Convolutional Neural Networks (CNNs) with transformer layers. This hybrid approach aims to capture both local (CNN) and global (transformer) contextual information, creating enriched multi-scale feature representations.
  • Adjacent-view Attention: This is the core innovation. It employs an alternating attention mechanism:
  • Self-Attention: Operates within each individual camera view to refine features.
  • Adjacent Attention: This novel module allows features to be exchanged between adjacent camera views. This is crucial for handling the limited overlap in surround-view setups, enabling the model to "see" beyond its immediate field of view by leveraging information from neighboring cameras. This process is repeated for multiple layers (Z times) to facilitate deep information exchange.
  • Depth Head: A final module that takes the processed features and predicts the depth map for all surround-view cameras simultaneously.

The loss function combines a standard L1 depth loss with a smoothness penalty to encourage locally consistent depth predictions.

Datasets

The paper evaluates AVT-SSDepth on two challenging multi-camera datasets:

  • nuScenes [10]: A widely used dataset for autonomous driving tasks, featuring 360° sensor data.
  • DDAD [9]: A large-scale dataset specifically designed for long-range and dense depth estimation in multi-camera scenarios.

The authors also establish a benchmark on nuScenes for supervised surround-view depth estimation.

Key Insights and Ideas

  • Adjacent-view attention is key: The paper strongly advocates for explicitly modeling interactions between adjacent views, rather than just all views at once. This is particularly effective in wide-baseline scenarios where direct overlap is minimal, allowing information to propagate across the 360° field of view.
  • Hybrid CNN-Transformer encoder: Combining CNNs for local feature extraction with transformers for global context proves beneficial for creating rich representations needed for accurate depth estimation.
  • Simultaneous multi-view prediction: Predicting all surround-view depth maps concurrently, rather than independently, allows the model to leverage inter-view correlations for improved accuracy.
  • Cross-dataset generalization: The proposed method shows strong performance not only on its training datasets but also demonstrates good generalization capabilities in cross-dataset transfer experiments.

Key Results

AVT-SSDepth achieves state-of-the-art performance on both the nuScenes and DDAD datasets.

  • On nuScenes, it significantly outperforms previous methods like SurroundDepth [21] and PixelFormer [11], achieving an Abs Rel score of 0.067.
  • On DDAD, it also sets a new benchmark, demonstrating its effectiveness in challenging long-range depth estimation scenarios.
  • In cross-dataset transfer experiments (DDAD to nuScenes), AVT-SSDepth achieves the lowest Abs Rel (0.292), Sq Rel (4.201), and RMSE (7.481), outperforming the previous best method R3D3 [66].
  • Qualitative results show visually sharper and more accurate depth maps compared to other methods, especially in areas with less direct overlap.

Limitations and Future Work

  • Computational Cost: While transformers offer advantages, they can be computationally intensive. The paper uses a specific backbone (MPViT-small) and notes the trade-offs in terms of FLOPs and parameters.
  • Number of Attention Layers (Z): The ablation study shows that Z=8 is optimal for the adjacent-view attention layers. However, finding the ideal Z might be dataset-dependent.
  • Future Work: The authors suggest that their work serves as a strong baseline for supervised surround-view depth estimation, implying that further architectural improvements and exploration of different loss functions could yield even better results. The effectiveness of adjacent-view attention across various backbones and its role in accelerating convergence also suggest avenues for future research.

Summary updated

Abstract

Depth estimation has been widely studied and serves as the fundamental step of 3D perception for robotics and autonomous driving. Though significant progress has been made in monocular depth estimation in the past decades, these attempts are mainly conducted on the KITTI benchmark with only front-view cameras, which ignores the correlations across surround-view cameras. In this paper, we propose an Adjacent-View Transformer for Supervised Surround-view Depth estimation (AVT-SSDepth), to jointly predict the depth maps across multiple surrounding cameras. Specifically, we employ a global-to-local feature extraction module that combines CNN with transformer layers for enriched representations. Further, the adjacent-view attention mechanism is proposed to enable the intra-view and inter-view feature propagation. The former is achieved by the self-attention module within each view, while the latter is realized by the adjacent attention module, which computes the attention across multi-cameras to exchange the multi-scale representations across surroundview feature maps. In addition, AVT-SSDepth has strong crossdataset generalization. Extensive experiments show that our method achieves superior performance over existing state-ofthe-art methods on both DDAD and nuScenes datasets. Code is available at https://github.com/XiandaGuo/SSDepth.
BibTeX
@inproceedings{guo2025adjacent,
  author = {Xianda Guo and Wenjie Yuan and Yunpeng Zhang and Tian Yang and Chenming Zhang and Zheng Zhu and Qin Zou and Long Chen},
  title = {{Adjacent-view Transformers for Supervised Surround-view Depth Estimation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  doi = {10.1109/iros60139.2025.11246758},
}
Figure