Depth Estimation maps of lidar and stereo images

Fei Wu, Luoyu Chen

Abstract

This technical report focuses on the evaluation and performance of depth estimation methods based on LiDAR data and stereo imagery (front left and front right). Utilizing datasets provided by Ford, the paper details optimizations for depth estimation performance and provides a rationale for favoring pure mathematical approaches over machine learning in specific cross-domain scenarios. The structure of this report covers performance evaluation of depth maps, methodologies for stereo-based estimation, and LiDAR-based depth completion techniques.

Introduction

Accurate depth estimation is a foundational requirement for autonomous driving and robotics. Currently, the field relies heavily on two modalities: LiDAR, which provides highly accurate but sparse 3D point clouds, and stereo cameras, which provide dense visual data but require complex matching algorithms to infer depth.

Recently, deep learning has dominated stereo depth estimation. However, a significant challenge remains: neural networks trained on standard benchmarks (like KITTI) often overfit to specific data distributions. When applied to new environments, such as the Ford dataset used here, these models frequently fail to produce accurate disparity maps. This paper explores how to optimize traditional, purely mathematical computer vision techniques to achieve robust depth estimation without relying on machine learning.

Methodology

Stereo Depth Estimation

The stereo pipeline utilizes a non-deep-learning approach centered on Semi-Global Block Matching (SGBM). Unlike basic Block Matching, SGBM calculates similarities across traversal windows to create a global energy function, yielding significantly higher precision. To further refine the output, a Weighted Least Squares (WLS) filter is applied to smooth low-gradient areas while preserving sharp edges. Additionally, padding optimization is used to resolve artifacts at the image edges caused by the matching window size.

LiDAR Depth Estimation

LiDAR depth maps are generated by projecting 3D point clouds onto a 2D image plane using coordinate transformations (global to body to camera frame). Because this projection results in sparse data, a neighborhood-weighted averaging technique is employed for depth completion. A critical optimization introduced here is "foreground preservation": sparse, short-distance pixels are dilated before completion to prevent background depths from "swallowing" thin foreground objects like light poles and fences.

Figure 1: Comparison of real images, stereo depth estimation (SBGM + WLS), and LiDAR-based depth maps.

Performance Evaluation

Quantitative and Qualitative Results

The traditional SGBM + WLS approach was evaluated against LiDAR-derived ground truth. Across three sample images, the mathematical approach achieved average per-pixel errors of 5.45m, 12.11m, and 4.93m. The higher error in the second scenario was attributed to complex environmental factors, such as high sunlight and intricate tree shadows, which challenge stereo matching algorithms.

Comparison with Machine Learning Models

When tested on the Ford dataset, pretrained neural networks (including HRSDE and Lac-GwcNet) trained on KITTI showed poor generalization. For instance, the HRSDE model yielded errors ranging from 13.9m to 24.0m on the same images where the mathematical approach performed significantly better. This highlights the robustness of traditional algorithms in novel environments.

Conclusion

While traditional stereo depth estimation remains sensitive to lighting and complex textures, it demonstrates superior generalization compared to overfitted neural networks. By integrating optimizations like WLS filtering and foreground-preserving LiDAR densification, robust depth maps can be generated for autonomous systems. Future work will focus on enhancing stereo matching robustness to extreme lighting conditions, potentially bridging the gap between mathematical generalization and the contextual strengths of machine learning.

Summary updated

Abstract

This paper as technology report is focusing on evaluation and performance about depth estimations based on lidar data and stereo images(front left and front right). The lidar 3d cloud data and stereo images are provided by ford. In addition, this paper also will explain some details about optimization for depth estimation performance. And some reasons why not use machine learning to do depth estimation, replaced by pure mathmatics to do stereo depth estimation. The structure of this paper is made of by following:(1) Performance: to discuss and evaluate about depth maps created from stereo images and 3D cloud points, and relationships analysis for alignment and errors;(2) Depth estimation by stereo images: to explain the methods about how to use stereo images to estimate depth;(3)Depth estimation by lidar: to explain the methods about how to use 3d cloud datas to estimate depth;In summary, this report is mainly to show the performance of depth maps and their approaches, analysis for them.
BibTeX
@misc{wu2022depth,
  author = {Fei Wu and Luoyu Chen},
  title = {{Depth Estimation maps of lidar and stereo images}},
  year = {2022},
  eprint = {2212.11741},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.11741},
}
Figure