A Novel Panorama Depth Estimation Framework for Autonomous Driving Scenarios Based on a Vision Transformer

Yuqi Zhang, Liang Chu, Zixu Wang, Tong He, Jincheng Hu, Jihao LiSensors 2024 · DOI

TL;DR

This paper introduces ViT-Fuse, a novel framework for panoramic depth estimation in autonomous driving. It tackles challenges like sparse 3D data and image distortion using a "Patch Filling" method for data completion and a Vision Transformer-based fusion approach (ViT-Fuse) for improved accuracy and robustness, especially at object edges.

What Problem Does This Paper Address?

Autonomous driving systems rely heavily on accurately perceiving their surroundings, which includes estimating the distance to objects. While LiDAR is useful, its high cost and sparse data limit its widespread adoption. Panoramic cameras offer a wider field of view, but traditional depth estimation methods struggle with the inherent distortions in equirectangular projections and the challenge of obtaining complete depth information from sparse sensor data. This paper aims to bridge this gap by developing a more accurate and robust panoramic depth estimation system for autonomous driving.

Methods

The proposed framework consists of two main components:

  • Patch Filling: This method addresses the issue of incomplete 3D point cloud data. It uses a fixed-size square slider to identify missing depth information in sparse point clouds and fills these gaps using mean interpolation. This process generates a denser and more complete depth map, crucial for accurate ground truth.
  • ViT-Fuse: This is a novel panoramic feature fusion model that leverages the power of Vision Transformers (ViT). It builds upon the UniFuse architecture but incorporates a ViT module in the encoding stage. This allows the model to learn image context and capture global features more effectively, mitigating the distortion issues common in equirectangular panoramic images. The ViT module works in conjunction with a ResNet backbone to extract both local and global features, leading to improved depth estimation, particularly at object edges.

Datasets

The primary dataset used for training and evaluation is the Ford Campus Vision and LiDAR Dataset. This dataset provides panoramic RGB images and raw 3D point cloud data, but not pre-computed panoramic depth maps. The authors generated their ground truth depth maps from the 3D point clouds using the Patch Filling method.

Key Insights and Ideas

  • Patch Filling is Crucial for Sparse Data: The paper strongly emphasizes that even with advanced deep learning models, the quality of the input depth data significantly impacts performance. Patch Filling proves to be an effective way to "heal" sparse point cloud data, making it usable for generating accurate ground truth.
  • Vision Transformers Excel at Global Context: The integration of ViT into the fusion architecture highlights its strength in capturing long-range dependencies and global image features, which is essential for handling the distortions and wide field of view of panoramic images.
  • Edge Details Matter: The authors demonstrate that their ViT-Fuse model significantly improves depth estimation accuracy at object edges, a critical aspect for safe autonomous driving. This is attributed to the combined local (ResNet) and global (ViT) feature extraction.
  • Distortion Mitigation is Key: The paper directly addresses the distortion problem in equirectangular projections, showing how ViT-Fuse, through its feature fusion and context learning, can overcome these limitations better than previous methods.

Key Results

  • The proposed ViT-Fuse model, with the Patch Filling method, achieved a 9.15% average reduction in estimation error compared to baseline methods.
  • Specifically, the Patch Filling method alone led to improvements: MAE reduced by 14.49%, Abs Rel by 18.03%, and RMSE by 11.93% compared to ViT-Fuse without PF.
  • The accuracy metric $\delta < 1.25$ saw a 9.73% increase with Patch Filling.
  • Compared to the UniFuse baseline, ViT-Fuse achieved reductions of 0.0042 in MAE and 0.0075 in RMSE, with a 16.46% improvement in Abs Rel error.
  • Qualitative results show ViT-Fuse producing smoother and more robust depth maps, especially at object edges, outperforming baselines like FCRN, BiFuse, and UniFuse.

Limitations and Future Work

  • Dataset Specificity: The experiments were primarily conducted on the Ford Campus Vision and LiDAR Dataset. The authors acknowledge that this dataset, while comprehensive, may not fully represent the diversity of all real-world driving scenarios, limiting the generalizability of their findings. Future work will involve testing on more diverse datasets.
  • Single Modality Reliance: The current framework relies solely on camera vision. This can be a limitation in challenging driving conditions like poor lighting or adverse weather. Future research aims to integrate other sensing modalities (e.g., radar, ultrasonic sensors) for improved robustness.
  • Computational Efficiency: The authors plan to optimize the computational efficiency of the ViT module to ensure its practical deployment in real-time autonomous driving systems.

Summary updated

Abstract

An accurate panorama depth estimation result is crucial to risk perception in autonomous driving practice. In this paper, an innovative framework is presented to address the challenges of imperfect observation and projection fusion in panorama depth estimation, enabling the accurate capture of distances from surrounding images in driving scenarios. First, the Patch Filling method is proposed to alleviate the imperfect observation of panoramic depth in autonomous driving scenarios, which constructs a panoramic depth map based on the sparse distance data provided by the 3D point cloud. Then, in order to tackle the distortion challenge faced by outdoor panoramic images, a method for image context learning, ViT-Fuse, is proposed and specifically designed for equirectangular panoramic views. The experimental results show that the proposed ViT-Fuse reduces the estimation error by 9.15% on average in driving scenarios compared with the basic method and exhibits more robust and smoother results on the edge details of the depth estimation maps.
BibTeX
@inproceedings{zhang2024novel,
  author = {Yuqi Zhang and Liang Chu and Zixu Wang and Tong He and Jincheng Hu and Jihao Li},
  title = {{A Novel Panorama Depth Estimation Framework for Autonomous Driving Scenarios Based on a Vision Transformer}},
  booktitle = {Sensors},
  year = {2024},
  doi = {10.3390/s24217013},
}
Figure