Abstract
This paper presents a streamlined two-step framework for high-precision depth perception by fusing sparse LiDAR data with stereo RGB images. The approach first predicts a dense depth map using a self-supervised neural network and subsequently refines it through local disparity matching. This method achieves state-of-the-art accuracy on the KITTI dataset while enhancing computational efficiency and model explainability compared to traditional end-to-end fusion models.
Introduction
Reliable 3D perception is the cornerstone of autonomous navigation. Currently, systems rely on two primary sensors: stereo cameras, which provide dense color data but struggle with repetitive patterns, and LiDAR, which offers pinpoint accuracy but produces sparse point clouds. Existing fusion methods typically attempt to integrate LiDAR data directly into traditional stereo matching algorithms, which often increases computational burden and relies on "black box" deep learning models that are difficult to verify for safety. This research proposes a framework that uses LiDAR to guide the stereo matching process rather than merely supplementing it, resulting in a faster, more explainable, and more accurate system.
Proposed Methodology
The authors break the fusion problem into a two-step pipeline designed for both accuracy and efficiency.
Step 1: Disparity Prediction
The first stage utilizes an encoder-decoder network that takes the left RGB image and sparse LiDAR points as input. To train this network without requiring exhaustive ground-truth labels, the framework employs a self-supervised photometric loss. If the network predicts the correct depth (and thus disparity) for a pixel in the left image, that pixel can be "warped" onto the right image to find a match. The difference between this warped image and the actual right image provides the training signal.
Step 2: Disparity Refinement
Traditional stereo matching searches a wide range of possible depths, which is computationally expensive. In this framework, the prediction from Step 1 serves as a high-confidence starting point. The system then performs a local search in a small neighborhood around the predicted value using a Sum of Absolute Differences (SAD) cost function. This ensures the final output is locally optimal and grounded in actual image data.
Experimental Results
The framework was evaluated using the KITTI dataset, a standard benchmark for autonomous driving. The system achieved a Root Mean Squared Error (RMSE) of 916.3 mm, outperforming the GC-Net baseline and remaining highly competitive with other 2019 state-of-the-art fusion methods.
Key Insights
- Reduced Search Space: By using an initial guess from LiDAR-monocular fusion, the search range for disparity is significantly narrowed, saving massive amounts of computation.
- Explainability: The use of a simple SAD cost function in the refinement step makes the process more transparent than pure end-to-end models, allowing researchers to see exactly why specific pixels were matched based on local gradients.
- Self-Supervision: Leveraging the stereo pair for photometric loss reduces the reliance on massive amounts of manually labeled 3D data, allowing the model to learn geometry from the relationship between the two cameras.
Conclusion and Future Work
This two-step approach demonstrates that a simpler, more structured framework can match the performance of complex, computationally heavy architectures. Future work will focus on increasing network depth to improve initial predictions, exploring more sophisticated cost functions to handle lighting changes, and incorporating spatial consistency constraints to ensure smoother, more physically plausible depth maps.
Summary updated