Lite-Mono: A Lightweight CNN and Transformer Architecture for Self-Supervised Monocular Depth Estimation

Ning Zhang, Francesco Nex, George Vosselman, Norman Kerle

Abstract

Self-supervised monocular depth estimation that does not require ground truth for training has attracted attention in recent years. It is of high interest to design lightweight but effective models so that they can be deployed on edge devices. Many existing architectures benefit from using heavier backbones at the expense of model sizes. This paper achieves comparable results with a lightweight architecture. Specifically, the efficient combination of CNNs and Transformers is investigated, and a hybrid architecture called Lite-Mono is presented. A Consecutive Dilated Convolutions (CDC) module and a Local-Global Features Interaction (LGFI) module are proposed. The former is used to extract rich multi-scale local features, and the latter takes advantage of the self-attention mechanism to encode long-range global information into the features. Experiments demonstrate that Lite-Mono outperforms Monodepth2 by a large margin in accuracy, with about 80% fewer trainable parameters.

Introduction

Monocular depth estimation—the ability to predict a 3D depth map from a single 2D image—is a critical task for autonomous driving, robotics, and augmented reality. While deep learning has significantly advanced this field, most high-performing models rely on "heavy" backbones, such as ResNet-50 or large Vision Transformers, which are often too slow and memory-intensive for deployment on mobile hardware or edge devices.

The primary challenge lies in balancing two architectural strengths:

  1. CNNs are highly efficient at capturing local spatial details but suffer from a limited receptive field, making it difficult to model long-range dependencies across an image.
  2. Transformers excel at modeling global context through self-attention but typically incur quadratic computational complexity, which is prohibitive for high-resolution real-time applications.

Lite-Mono addresses this tension by introducing a lightweight hybrid architecture that is both "globally aware" and "edge-ready," achieving state-of-the-art accuracy with a fraction of the parameters used by standard models like Monodepth2.

Proposed Architecture

Lite-Mono utilizes a four-stage hierarchical encoder-decoder framework designed for efficiency. The core innovation lies in two novel modules within the encoder: the Consecutive Dilated Convolutions (CDC) module and the Local-Global Features Interaction (LGFI) module.

The Lite-Mono architecture overview, showing the hierarchical DepthNet and the PoseNet used for self-supervised training.

Consecutive Dilated Convolutions (CDC)

To expand the receptive field without increasing the parameter count, Lite-Mono employs a series of dilated convolutions. By stacking these convolutions with varying dilation rates, the network can capture multi-scale local features effectively. This allows a relatively shallow network to achieve the representational power of much deeper architectures.

Local-Global Features Interaction (LGFI)

To incorporate global context without the heavy computational burden of standard Transformers, the authors introduce the LGFI module. This module utilizes cross-covariance attention, which computes attention across the channel dimension rather than the spatial dimension. This shift reduces computational complexity from quadratic to linear with respect to the number of pixels, enabling the model to capture long-range dependencies while remaining lightweight.

Self-Supervised Learning

The model is trained in a self-supervised manner, treating depth estimation as a view synthesis problem. By using synchronized video frames, the network learns to predict depth by minimizing the photometric reconstruction error between adjacent frames, eliminating the need for expensive LiDAR ground-truth data.

Experimental Setup

The model was primarily evaluated on the KITTI dataset, the industry standard for autonomous driving benchmarks, using the Eigen split (39,180 training triplets). To test the model's ability to generalize to unseen environments, the authors also evaluated it on the Make3D dataset without any additional fine-tuning.

Results and Discussion

Lite-Mono demonstrates a superior balance of efficiency and accuracy compared to existing self-supervised methods.

Qualitative comparison on the KITTI dataset. Lite-Mono (ours) preserves sharper boundaries for small objects compared to Monodepth2.
  • Efficiency and Speed: The standard Lite-Mono model contains only 3.1 million parameters, roughly 80% fewer than Monodepth2 (ResNet-18). On an NVIDIA Jetson Xavier edge platform, it achieves a real-time inference speed of approximately 50 FPS (20.0ms), significantly outperforming heavier Transformer-based models like MonoViT-tiny.
  • Accuracy: Despite its reduced size, Lite-Mono consistently outperforms Monodepth2 across all standard KITTI metrics. Qualitative results show that Lite-Mono is particularly effective at preserving the boundaries of thin or distant objects, such as cyclists and road signs.
  • Generalization: The model performed best among its peers on the Make3D dataset, indicating that the hybrid CNN-Transformer features are robust and generalize well to diverse outdoor scenes.

Conclusion

Lite-Mono proves that a carefully designed hybrid architecture can achieve high-accuracy monocular depth estimation while remaining suitable for edge deployment. While the authors noted that excessively high dilation rates in the CDC module can eventually degrade performance, the current design finds a highly effective "sweet spot." Future work may explore integrating stereo-pair training or more advanced cross-stage feature fusion to further enhance performance without sacrificing real-time efficiency.

Summary updated

Abstract

Self-supervised monocular depth estimation that does not require ground truth for training has attracted attention in recent years. It is of high interest to design lightweight but effective models so that they can be deployed on edge devices. Many existing architectures benefit from using heavier backbones at the expense of model sizes. This paper achieves comparable results with a lightweight architecture. Specifically, the efficient combination of CNNs and Transformers is investigated, and a hybrid architecture called Lite-Mono is presented. A Consecutive Dilated Convolutions (CDC) module and a Local-Global Features Interaction (LGFI) module are proposed. The former is used to extract rich multi-scale local features, and the latter takes advantage of the self-attention mechanism to encode long-range global information into the features. Experiments demonstrate that Lite-Mono outperforms Monodepth2 by a large margin in accuracy, with about 80% fewer trainable parameters.
BibTeX
@inproceedings{zhang2023lite,
  author = {Ning Zhang and Francesco Nex and George Vosselman and Norman Kerle},
  title = {{Lite-Mono: A Lightweight CNN and Transformer Architecture for Self-Supervised Monocular Depth Estimation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01778},
}
Figure