Dense monocular depth estimation for stereoscopic vision based on pyramid transformer and multi-scale feature fusion

Zhongyi Xia, Tianzhao Wu, Zhuoyan Wang, Man Zhou, Boqi Wu, Chia-Tai Chan, Ling Bing KongScientific 2024 · DOI

TL;DR

This paper introduces SPT-Depth, a novel deep learning architecture that significantly improves monocular depth estimation for stereoscopic vision by combining the strengths of Pyramid Transformers and multi-scale feature fusion. By addressing Transformer limitations like lack of translational invariance and high computational cost, SPT-Depth achieves state-of-the-art accuracy with reduced error metrics.

What Problem Does This Paper Address?

Accurate depth estimation from a single image (monocular depth estimation) is crucial for realistic stereoscopic displays and applications like autonomous driving and 3D reconstruction. While Convolutional Neural Networks (CNNs) have been dominant, they struggle with capturing global context. Transformers, on the other hand, excel at global context but have limitations: they lack translational invariance (making them less robust to spatial shifts) and are computationally expensive, especially for high-resolution images. This paper aims to bridge this gap by developing a Transformer-based model that retains global context while mitigating these drawbacks for improved stereoscopic vision.

Methods

The core of the proposed method is the Stereoscopic Pyramid Transformer-Depth (SPT-Depth). Key innovations include:

  • Pyramid Transformer Backbone: Instead of a standard Transformer, they use a Pyramid Vision Transformer (PVT). PVT progressively downsamples feature maps, reducing the number of tokens and thus computational load, while still maintaining a global receptive field.
  • Multi-Scale Feature Fusion: The model fuses shallow and deep semantic information acquired through stepwise downsampling. This is achieved by upsampling and reassembling feature maps from different stages of the PVT encoder before fusion.
  • Shift and Scale-Invariant Loss (SSI Loss): To address the lack of translational invariance in Transformers, they employ an SSI loss function. This loss is designed to be robust to shifts and scale changes in the input, improving depth prediction accuracy.
  • Edge-Smoothing Function: An additional edge-smoothing loss is incorporated to reduce noise in the predicted depth maps, enhancing robustness and detail preservation.
  • Two-Stage Training (Coarse & Fine Convergence): The training process is split into two stages with distinct strategies and hyperparameters. This approach helps to stabilize training and achieve better overall performance.

Datasets

The authors utilized the following datasets:

  • NYU Depth V2: Used for both pre-training and final evaluation.
  • INRIA: Used for pre-training.
  • Posetrack: Used for pre-training.

The paper notes that while Feng et al. previously used all available samples repeatedly, this can lead to overfitting. Their approach aims to mitigate this.

Key Insights and Ideas

  • Transformers can be adapted for dense prediction: While originally designed for sequential data, Transformers, when architected correctly (like with PVT), can be powerful for dense prediction tasks like depth estimation.
  • Multi-scale fusion is key for depth: Combining features from different levels of abstraction (shallow and deep semantics) is crucial for generating accurate and detailed depth maps.
  • Addressing Transformer weaknesses is critical: The lack of translational invariance and high computational cost are significant hurdles for Transformer adoption in vision tasks. The SSI loss and pyramid architecture directly tackle these issues.
  • Two-stage training offers stability: Separating training into coarse and fine convergence stages with tailored strategies allows for better control over the learning process and improved final results.

Key Results

  • Improved Accuracy: SPT-Depth achieved a 10% reduction in Absolute Relative Error (Abs Rel) and a 36% decrease in Root Mean Square Error (RMSE) compared to a baseline method on the NYU Depth V2 dataset.
  • State-of-the-Art Performance: When compared to other state-of-the-art methods, SPT-Depth showed a 17% reduction in RMSE.
  • Reduced Computational Complexity: The pyramid architecture and token reduction strategy significantly decrease computational load compared to standard Vision Transformers (ViT).
  • Robustness: The inclusion of SSI loss and edge-smoothing functions enhances the model's robustness to noise and scale variations.
  • Visual Quality: Qualitative results (Figures 6-8) demonstrate that SPT-Depth produces sharper object boundaries and more accurate depth details, especially in complex scenes with multiple individuals.

Limitations and Future Work

  • Computational Complexity: While reduced compared to ViT, the computational complexity of Transformers has not been entirely resolved.
  • Artifacts in Complex Regions: The authors acknowledge that in regions where clear and blurred sections converge, artifacts may appear, potentially affecting 3D display quality.
  • Future Work:
    • Bridging the semantic gap between encoder and decoder using self-supervised methods.
  • Incorporating classic computer vision algorithms or end-to-end techniques to further improve rendering and reduce artifacts.
  • Exploring strategies from Swin Transformer to confine attention calculations within windows, further reducing computational demands.

Summary updated

Abstract

Stereoscopic display technology plays a significant role in industries, such as film, television and autonomous driving. The accuracy of depth estimation is crucial for achieving high-quality and realistic stereoscopic display effects. In addressing the inherent challenges of applying Transformers to depth estimation, the Stereoscopic Pyramid Transformer-Depth (SPT-Depth) is introduced. This method utilizes stepwise downsampling to acquire both shallow and deep semantic information, which are subsequently fused. The training process is divided into fine and coarse convergence stages, employing distinct training strategies and hyperparameters, resulting in a substantial reduction in both training and validation losses. In the training strategy, a shift and scale-invariant mean square error function is employed to compensate for the lack of translational invariance in the Transformers. Additionally, an edge-smoothing function is applied to reduce noise in the depth map, enhancing the model's robustness. The SPT-Depth achieves a global receptive field while effectively reducing time complexity. In comparison with the baseline method, with the New York University Depth V2 (NYU Depth V2) dataset, there is a 10% reduction in Absolute Relative Error (Abs Rel) and a 36% decrease in Root Mean Square Error (RMSE). When compared with the state-of-the-art methods, there is a 17% reduction in RMSE.
BibTeX
@inproceedings{xia2024dense,
  author = {Zhongyi Xia and Tianzhao Wu and Zhuoyan Wang and Man Zhou and Boqi Wu and Chia-Tai Chan and Ling Bing Kong},
  title = {{Dense monocular depth estimation for stereoscopic vision based on pyramid transformer and multi-scale feature fusion}},
  booktitle = {Scientific},
  year = {2024},
  doi = {10.1038/s41598-024-57908-z},
}
Figure