Abstract: Depth completion, the technique of estimating a dense depth image from sparse depth measurements, has a variety of applications in robotics and autonomous driving. However, depth completion faces 3 main challenges: the irregularly spaced pattern in the sparse depth input, the difficulty in handling multiple sensor modalities (when color images are available), as well as the lack of dense, pixel-level ground truth depth labels. In this work, we address all these challenges. Specifically, we develop a deep regression model to learn a direct mapping from sparse depth (and color images) to dense depth. We also propose a self-supervised training framework that requires only sequences of color and sparse depth images, without the need for dense depth labels. Our experiments demonstrate that our network, when trained with semi-dense annotations, attains state-of-the- art accuracy and is the winning approach on the KITTI depth completion benchmark at the time of submission. Furthermore, the self-supervised framework outperforms a number of existing solutions trained with semi- dense annotations.
Introduction
In the world of autonomous driving and robotics, knowing exactly how far away objects are is a life-or-death requirement. While LiDAR sensors provide highly accurate distance measurements, they are notoriously "sparse"—a high-end 64-line LiDAR still only provides data for about 4% of the pixels in a standard camera view. This paper presents a deep learning solution to "fill in the blanks," transforming sparse LiDAR points into dense depth maps without requiring expensive, manually-labeled dense ground truth.
The Depth Completion Challenge
Depth completion is the task of taking a sparse set of depth measurements and predicting a value for every single pixel. This is difficult for three main reasons:
- Irregular Sparsity: LiDAR points are not evenly distributed; they appear in scan lines that get further apart as objects get more distant.
- Sensor Fusion: Combining "color" (RGB) and "depth" (LiDAR) is non-trivial because they represent fundamentally different physical modalities.
- The Label Gap: To train a standard neural network, you need "ground truth" dense depth. However, getting a perfectly dense depth map for a real-world driving scene is nearly impossible and incredibly labor-intensive to annotate.
Methodology
The authors tackle these challenges with a two-pronged approach: a robust architecture and a clever training regime.
1. The Architecture
The model uses an encoder-decoder paradigm based on ResNet-34. To handle sensor fusion, the researchers employed Late Fusion, where sparse depth and RGB images are processed by separate initial convolutional layers before being concatenated. This allows the network to extract features from each modality independently before merging them. Additionally, Skip Connections pass high-resolution features from the encoder directly to the decoder, helping preserve sharp object boundaries like the edges of cars or poles.
2. Self-Supervised Training
The core innovation is a self-supervised framework that uses monocular video sequences instead of dense labels. This relies on several key components:
- Photometric Loss: By predicting a depth map for one frame, the system can use that depth and the camera's movement to mathematically "warp" colors from a subsequent frame back onto the first. If the warped image matches the original, the depth prediction is likely correct.
- Model-Based Pose: Unlike other unsupervised methods that use a neural network to guess camera motion, this approach uses a traditional geometric approach (PnP with RANSAC), providing absolute-scale supervision and greater stability.
- Loss Functions: The total loss combines sparse depth points (identity mapping), photometric consistency (warping), and a smoothness term to ensure depth transitions remain natural.
Key Insights and Results
The researchers found that depth completion error decreases as a power function of the number of input samples. Interestingly, uniform random sampling of points proved more effective for the network than structured LiDAR scan lines.
On the KITTI Depth Completion Benchmark, the supervised RGB-D model was ranked #1 at the time of submission, achieving an RMSE of 814.73 mm. Even more impressively, the self-supervised version achieved an RMSE of 1384.85 mm, outperforming several existing methods that were trained with supervised labels. This proves that temporal consistency in video is a powerful substitute for human-annotated labels.
Limitations and Future Work
Despite its success, the model faces challenges in regions where LiDAR data is absent, such as the sky. Furthermore, the self-supervised warping logic assumes a static world; dynamic objects (like moving cars) can confuse the photometric loss. Future work will focus on explicitly modeling these dynamic objects and improving convergence times, as self-supervised training currently requires significantly more iterations than supervised learning.
Summary updated