SwinDepth: Unsupervised Depth Estimation using Monocular Sequences via Swin Transformer and Densely Cascaded Network

Dongseok Shim, H. Jin Kim

TL;DR

SwinDepth introduces a novel architecture for unsupervised monocular depth estimation that replaces traditional CNN backbones with a hierarchical Swin Transformer and a "Densely Cascaded" decoder. By combining global semantic awareness with dense multi-scale feature fusion, the model sets a new state-of-the-art performance on the KITTI benchmark while maintaining a competitive parameter count.

What Problem Does This Paper Address?

Estimating 3D depth from a single 2D image (monocular depth estimation) is an "ill-posed" problem—mathematically, there are infinite 3D scenes that could produce the same 2D image. While supervised learning can solve this, high-quality "ground truth" depth data (like LiDAR scans) is incredibly expensive and difficult to collect.

Unsupervised learning, which uses video sequences to learn depth by "re-projecting" one frame onto another, is a popular alternative. However, most existing models rely on standard Convolutional Neural Networks (CNNs) in a U-Net structure. CNNs have a limited "receptive field," meaning they struggle to understand the relationship between distant parts of an image. Furthermore, standard decoders often lose important semantic information as they pass features from low-resolution layers to high-resolution ones.

Methods

The authors propose two major architectural shifts:

  1. Swin Transformer Encoder: Instead of a CNN, they use a hierarchical Swin Transformer. Unlike the original Vision Transformer (ViT), the Swin Transformer uses "shifted windows" to calculate self-attention. This allows the model to capture global context (the "big picture") while keeping computational costs manageable.
  2. Densely Cascaded Multi-scale Network (DCMNet): This is the "brain" of the decoder. Rather than using simple skip connections, DCMNet employs top-down cascade addition and dense concatenation. It ensures that every feature map is directly connected to others across different scales. This prevents the "attenuation" of high-level semantic features as the network upsamples to create the final depth map.
  3. Pyramid Pooling Module (PPM): To further bolster the model's understanding of the entire scene, a PPM is added to the first stage of the decoder to provide a global contextual prior.

The model is trained in an unsupervised fashion by jointly estimating depth and camera "ego-motion" (pose) to minimize a photometric reprojection loss between adjacent video frames.

Datasets

  • KITTI 2015: The gold standard for autonomous driving research, consisting of stereo and monocular videos of outdoor driving scenes.
  • Make3D: A smaller dataset used specifically to test how well the model generalizes to new environments without any additional fine-tuning.

Key Insights and Ideas

  • Global Context is King for Depth: Depth estimation isn't just about local textures; it's about understanding the scene layout. Transformers are naturally better at this than CNNs because their attention mechanism can link a car in the foreground to the horizon in the background instantly.
  • Interconnectivity Matters: The authors found that simply having a strong encoder isn't enough. The "Densely Cascaded" approach in the decoder ensures that the fine-grained details (local features) and the broad scene understanding (global features) are effectively fused at every resolution level.
  • Efficiency vs. Power: By using the Swin architecture, the authors achieved better results than massive CNN-based models (like PackNet-SfM) while using significantly fewer parameters (~25M vs ~120M).

Key Results

  • New State-of-the-Art: On the KITTI benchmark, SwinDepth achieved an Absolute Relative (Abs Rel) error of 0.106, outperforming established leaders like Monodepth2 (0.115) and PackNet-SfM (0.111).
  • Sharper Details: Qualitatively, the model produces much cleaner boundaries for thin or complex objects, such as traffic lights, poles, and tree foliage, which are notoriously difficult for unsupervised models to resolve.
  • Zero-Shot Generalization: When tested on the Make3D dataset without any training on that specific data, SwinDepth still outperformed previous unsupervised methods, proving its robustness.

Limitations and Future Work

While the Swin Transformer is more efficient than a standard ViT, it still carries a higher computational overhead than a lightweight ResNet-18, which might limit its use on low-power edge devices in real-time.

The authors also note that their method, like most unsupervised monocular approaches, assumes a "static scene" (where only the camera moves). While their architecture handles dynamic objects (like moving cars) better than predecessors, explicitly modeling object motion remains a promising direction for future research to further reduce depth "holes" and artifacts.

Summary updated

Abstract

Monocular depth estimation plays a critical role in various computer vision and robotics applications such as localization, mapping, and 3D object detection. Recently, learning-based algorithms achieve huge success in depth estimation by training models with a large amount of data in a supervised manner. However, it is challenging to acquire dense ground truth depth labels for supervised training, and the unsupervised depth estimation using monocular sequences emerges as a promising alternative. Unfortunately, most studies on unsupervised depth estimation explore loss functions or occlusion masks, and there is little change in model architecture in that ConvNet-based encoder-decoder structure becomes a de-facto standard for depth estimation. In this paper, we employ a convolution-free Swin Transformer as an image feature extractor so that the network can capture both local geometric features and global semantic features for depth estimation. Also, we propose a Densely Cascaded Multi-scale Network (DCMNet) that connects every feature map directly with another from different scales via a top-down cascade pathway. This densely cascaded connectivity reinforces the interconnection between decoding layers and produces high-quality multi-scale depth outputs. The experiments on two different datasets, KITTI and Make3D, demonstrate that our proposed method outperforms existing state-of-the-art unsupervised algorithms.
BibTeX
@inproceedings{shim2023swindepth,
  author = {Dongseok Shim and H. Jin Kim},
  title = {{SwinDepth: Unsupervised Depth Estimation using Monocular Sequences via Swin Transformer and Densely Cascaded Network}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  url = {https://openreview.net/forum?id=BBrwlJXaiz},
}

Loading related content...

Figure