Unsupervised confidence for LiDAR depth maps and applications

Andrea Conti, Matteo Poggi, Filippo Aleotti, Stefano Mattoccia

Abstract

In autonomous driving and robotics, the fusion of 3D LiDAR data with 2D RGB images is a fundamental task. However, projecting LiDAR points onto a camera's image plane often introduces "bleeding" outliers—background points incorrectly mapped onto foreground objects due to parallax. This paper introduces the first unsupervised deep learning framework designed to estimate the confidence of these projected depth maps. By leveraging a novel "minimum depth" heuristic and a stable loss function, the proposed method identifies and filters these outliers without requiring expensive ground-truth labels, significantly improving the performance of downstream tasks like depth completion and stereo matching.

Introduction

Reliable depth perception is critical for safe autonomous navigation. While LiDAR sensors provide accurate distance measurements, their data is sparse. To create dense environmental models, researchers typically project these 3D points onto the 2D image plane of a camera. Because the LiDAR and camera are physically offset, they perceive the world from slightly different perspectives. This spatial discrepancy leads to occlusion errors: background points that are visible to the LiDAR but hidden from the camera appear to "bleed" onto foreground objects in the resulting depth map.

Existing methods to mitigate this noise often rely on stereo camera setups for consistency checks or require massive, manually labeled datasets. This paper proposes an unsupervised alternative that learns to predict uncertainty directly from the physical relationship between the sensors.

Methodology

Architecture

The framework utilizes a fast, multi-scale convolutional encoder to extract features from both the RGB image and the sparse LiDAR map. To maintain high efficiency, the model avoids a standard heavy decoder. Instead, it employs a lightweight Multi-Layer Perceptron (MLP) head. This head queries the extracted features only at the specific pixel locations where LiDAR data is present, allowing for real-time processing.

The Minimum Depth Heuristic

The core innovation of the unsupervised training process is the use of "proxy labels." The authors observe that in a local neighborhood, the minimum depth value is statistically the most reliable. Because projection errors almost exclusively involve background points being pulled forward onto the foreground, the nearest point in a cluster is the most likely to be geometrically accurate. This simple yet effective heuristic allows the model to learn confidence scores without human intervention.

Numerical Stability and Loss

To model depth uncertainty, the authors treat depth as a Gaussian distribution. To prevent training instability when the model becomes highly confident (resulting in a very small variance), they introduced a modified negative log-likelihood loss function, denoted as $L_{G^*}$. This ensures numerical stability and smoother convergence during the training phase.

Experimental Results

Performance Benchmarks

The framework was evaluated using the KITTI dataset, specifically the CV split (1,000 images) and the 142 split (high-quality manual ground truth). The proposed method achieved an Area Under the Curve (AUC) of 0.1292, outperforming the previous state-of-the-art unsupervised method (0.7014) and even surpassing supervised NCNN variants (0.1530).

Downstream Impact

The utility of the confidence scores was demonstrated by integrating the filter into existing depth completion frameworks such as PENet and Sparse-to-Dense. By removing low-confidence points before processing, these models achieved higher accuracy without requiring any retraining. Furthermore, the method proved effective for semi-automatic data annotation, reaching lower Root Mean Square Error (RMSE) while preserving more valid data points than traditional stereo-based filters.

Discussion and Conclusion

One of the most significant advantages of this approach is its efficiency; the model runs in approximately 20ms on a modern GPU, making it suitable for real-time robotics applications. However, certain limitations remain. The model is currently specific to the sensor rig it was trained on; significant changes in the physical distance between the camera and LiDAR would necessitate retraining. Additionally, while the model provides a confidence score, users must still define a threshold for outlier removal.

By moving away from a reliance on expensive ground truth and focusing on the inherent geometry of sensor projection, this work provides a scalable and robust solution for cleaning LiDAR data. Future work may involve integrating this confidence estimation directly into end-to-end perception pipelines to enhance robustness against sensor noise from the outset.

Summary updated

Abstract

Depth perception is pivotal in many fields, such as robotics and autonomous driving, to name a few. Consequently, depth sensors such as LiDARs rapidly spread in many applications. The 3D point clouds generated by these sensors must often be coupled with an RGB camera to understand the framed scene semantically. Usually, the former is projected over the camera image plane, leading to a sparse depth map. Unfortunately, this process, coupled with the intrinsic issues affecting all the depth sensors, yields noise and gross outliers in the final output. Purposely, in this paper, we propose an effective unsupervised framework aimed at explicitly addressing this issue by learning to estimate the confidence of the LiDAR sparse depth map and thus allowing for filtering out the outliers. Experimental results on the KITTI dataset highlight that our framework excels for this purpose. Moreover, we demonstrate how this achievement can improve a wide range of tasks.
BibTeX
@inproceedings{conti2022unsupervised,
  author = {Andrea Conti and Matteo Poggi and Filippo Aleotti and Stefano Mattoccia},
  title = {{Unsupervised confidence for LiDAR depth maps and applications}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=BnZapV7Aw0},
}
Figure