Abstract
In autonomous driving and robotics, the fusion of 3D LiDAR data with 2D RGB images is a fundamental task. However, projecting LiDAR points onto a camera's image plane often introduces "bleeding" outliers—background points incorrectly mapped onto foreground objects due to parallax. This paper introduces the first unsupervised deep learning framework designed to estimate the confidence of these projected depth maps. By leveraging a novel "minimum depth" heuristic and a stable loss function, the proposed method identifies and filters these outliers without requiring expensive ground-truth labels, significantly improving the performance of downstream tasks like depth completion and stereo matching.
Introduction
Reliable depth perception is critical for safe autonomous navigation. While LiDAR sensors provide accurate distance measurements, their data is sparse. To create dense environmental models, researchers typically project these 3D points onto the 2D image plane of a camera. Because the LiDAR and camera are physically offset, they perceive the world from slightly different perspectives. This spatial discrepancy leads to occlusion errors: background points that are visible to the LiDAR but hidden from the camera appear to "bleed" onto foreground objects in the resulting depth map.
Existing methods to mitigate this noise often rely on stereo camera setups for consistency checks or require massive, manually labeled datasets. This paper proposes an unsupervised alternative that learns to predict uncertainty directly from the physical relationship between the sensors.
Methodology
Architecture
The framework utilizes a fast, multi-scale convolutional encoder to extract features from both the RGB image and the sparse LiDAR map. To maintain high efficiency, the model avoids a standard heavy decoder. Instead, it employs a lightweight Multi-Layer Perceptron (MLP) head. This head queries the extracted features only at the specific pixel locations where LiDAR data is present, allowing for real-time processing.
The Minimum Depth Heuristic
The core innovation of the unsupervised training process is the use of "proxy labels." The authors observe that in a local neighborhood, the minimum depth value is statistically the most reliable. Because projection errors almost exclusively involve background points being pulled forward onto the foreground, the nearest point in a cluster is the most likely to be geometrically accurate. This simple yet effective heuristic allows the model to learn confidence scores without human intervention.
Numerical Stability and Loss
To model depth uncertainty, the authors treat depth as a Gaussian distribution. To prevent training instability when the model becomes highly confident (resulting in a very small variance), they introduced a modified negative log-likelihood loss function, denoted as $L_{G^*}$. This ensures numerical stability and smoother convergence during the training phase.
Experimental Results
Performance Benchmarks
The framework was evaluated using the KITTI dataset, specifically the CV split (1,000 images) and the 142 split (high-quality manual ground truth). The proposed method achieved an Area Under the Curve (AUC) of 0.1292, outperforming the previous state-of-the-art unsupervised method (0.7014) and even surpassing supervised NCNN variants (0.1530).
Downstream Impact
The utility of the confidence scores was demonstrated by integrating the filter into existing depth completion frameworks such as PENet and Sparse-to-Dense. By removing low-confidence points before processing, these models achieved higher accuracy without requiring any retraining. Furthermore, the method proved effective for semi-automatic data annotation, reaching lower Root Mean Square Error (RMSE) while preserving more valid data points than traditional stereo-based filters.
Discussion and Conclusion
One of the most significant advantages of this approach is its efficiency; the model runs in approximately 20ms on a modern GPU, making it suitable for real-time robotics applications. However, certain limitations remain. The model is currently specific to the sensor rig it was trained on; significant changes in the physical distance between the camera and LiDAR would necessitate retraining. Additionally, while the model provides a confidence score, users must still define a threshold for outlier removal.
By moving away from a reliance on expensive ground truth and focusing on the inherent geometry of sensor projection, this work provides a scalable and robust solution for cleaning LiDAR data. Future work may involve integrating this confidence estimation directly into end-to-end perception pipelines to enhance robustness against sensor noise from the outset.
Summary updated