Self-Supervised Sparse-to-Dense: Self-Supervised Depth Completion from LiDAR and Monocular Camera

Fangchang Ma, Guilherme Venturelli Cavalheiro, Sertac Karaman

In the world of autonomous driving and robotics, knowing exactly how far away objects are is a life-or-death requirement. While LiDAR sensors provide highly accurate distance measurements, they are notoriously "sparse"—a high-end 64-line LiDAR still only provides data for about 4% of the pixels in a standard camera view.

This paper from researchers at MIT presents a deep learning solution to "fill in the blanks," transforming sparse LiDAR points into dense depth maps. Most importantly, they propose a way to train these models without needing expensive, manually-labeled dense ground truth.

TL;DR

The authors develop a deep encoder-decoder network that maps sparse LiDAR and RGB images to dense depth maps. By using a novel self-supervised framework based on temporal consistency in video, their model achieves state-of-the-art performance on the KITTI benchmark, even outperforming several existing methods that were trained with supervised labels.

What Problem Does This Paper Address?

Depth completion is the task of taking a sparse set of depth measurements and predicting a value for every single pixel. This is difficult for three main reasons:

  1. Irregular Sparsity: LiDAR points are not evenly distributed; they appear in scan lines that get further apart as objects get more distant.
  2. Sensor Fusion: Combining "color" (RGB) and "depth" (LiDAR) is non-trivial because they represent fundamentally different physical modalities.
  3. The Label Gap: To train a standard neural network, you need "ground truth" dense depth. However, getting a perfectly dense depth map for a real-world driving scene is nearly impossible and incredibly labor-intensive to annotate.

Methods

The authors tackle these challenges with a two-pronged approach: a robust architecture and a clever training regime.

1. The Architecture

The model uses an encoder-decoder paradigm based on ResNet-34.

  • Late Fusion: The sparse depth and the RGB image are processed by separate initial convolutional layers before being concatenated. This allows the network to extract features from each modality independently before merging them.
  • Skip Connections: These are used to pass high-resolution features from the encoder directly to the decoder, which helps preserve sharp object boundaries (like the edges of cars or poles).

2. Self-Supervised Training

This is the core innovation. Instead of relying on dense labels, the authors use monocular video sequences.

  • Photometric Loss: If the network predicts an accurate depth map for "Frame 1," we should be able to use that depth (plus the camera's movement) to mathematically "warp" the colors from "Frame 2" back onto "Frame 1." If the warped image matches the original, the depth prediction was likely correct.
  • Model-Based Pose: Unlike other "unsupervised" methods that use a second neural network to guess camera motion, this paper uses a traditional geometric approach (PnP with RANSAC). This provides absolute-scale supervision and makes the system more stable.
  • Loss Functions: The total loss is a combination of the sparse depth points (identity mapping), the photometric consistency (warping), and a smoothness term (to ensure the depth doesn't jitter unnaturally).

Datasets

The primary dataset used is the KITTI Depth Completion Benchmark.

  • It contains roughly 86,000 training images.
  • The "ground truth" provided by KITTI is actually semi-dense (~30% coverage), created by aggregating multiple LiDAR frames over time.
  • The authors use this semi-dense data for their supervised experiments but rely only on raw, sparse LiDAR and RGB sequences for their self-supervised experiments.

Key Insights and Ideas

  • $L_2$ beats $L_1$: While many depth papers suggest $L_1$ loss is better for handling outliers, the authors found that $L_2$ (Mean Squared Error) yielded higher accuracy for this specific architecture and dataset.
  • Self-Supervision is Competitive: The self-supervised model achieved an RMSE of 1384mm. While this is higher than their supervised version (814mm), it actually beat several supervised models from other research groups, proving that temporal consistency is a powerful substitute for human labels.
  • The Power of Sparsity: The researchers found that depth completion error decreases as a power function of the number of input samples. Interestingly, they noted that uniform random sampling of points is much more effective for the network than structured LiDAR scan lines.

Key Results

  • Benchmark Performance: At the time of submission, their supervised RGB-D model was ranked #1 on the KITTI Depth Completion leaderboard.
  • Quantitative Accuracy:
    • Supervised (RGB+D): 814.73 mm RMSE.
  • Self-Supervised (RGB+D): 1384.85 mm RMSE.
  • Qualitative Improvement: The model produces much cleaner and sharper object boundaries compared to previous methods like NConv-CNN or SparseConvs, which often suffer from "bleeding" edges.

Limitations and Future Work

  • The "Sky" Problem: LiDAR sensors have a limited vertical field of view and don't receive returns from the sky or the tops of tall buildings. Consequently, the network struggles to predict depth in the upper regions of an image where there is zero input data.
  • Dynamic Objects: The self-supervised warping logic assumes the world is static. If a car is moving at the same time the camera is moving, the photometric loss becomes "confused." The authors suggest that explicitly modeling dynamic objects is a key next step.
  • Iterative Convergence: The authors noted that self-supervised training takes significantly more iterations to converge compared to supervised training, likely because the RGB signal has a lower initial correlation with depth than the sparse LiDAR points do.

Summary updated

Abstract

Depth completion, the technique of estimating a dense depth image from sparse depth measurements, has a variety of applications in robotics and autonomous driving. However, depth completion faces 3 main challenges: the irregularly spaced pattern in the sparse depth input, the difficulty in handling multiple sensor modalities (when color images are available), as well as the lack of dense, pixel-level ground truth depth labels for training. In this work, we address all these challenges. Specifically, we develop a deep regression model to learn a direct mapping from sparse depth (and color images) input to dense depth prediction. We also propose a self-supervised training framework that requires only sequences of color and sparse depth images, without the need for dense depth labels. Our experiments demonstrate that the self-supervised framework outperforms a number of existing solutions trained with semi-dense annotations. Furthermore, when trained with semi-dense annotations, our network attains state-of-the-art accuracy and is the winning approach on the KITTI depth completion benchmark at the time of submission.
BibTeX
@inproceedings{ma2019self,
  author = {Fangchang Ma and Guilherme Venturelli Cavalheiro and Sertac Karaman},
  title = {{Self-Supervised Sparse-to-Dense: Self-Supervised Depth Completion from LiDAR and Monocular Camera}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2019},
  url = {https://openreview.net/forum?id=nV1P1jpKWuA},
}

Loading related content...

Figure