TL;DR
This paper tackles the challenge of monocular depth estimation (SIDE) by proposing a novel "Sparse-to-Continuous" approach. They leverage Hilbert Maps to densify sparse LiDAR depth data into continuous occupancy maps, which are then used to train deep neural networks, significantly improving depth prediction accuracy without requiring additional sensors or complex network architectures.
What Problem Does This Paper Address?
Estimating depth from a single image (SIDE) is an inherently ill-posed problem because a 2D image can correspond to multiple 3D scenes. While deep learning has made significant strides, a major bottleneck is the quality of training data. Supervised methods often rely on ground-truth depth maps, but these are typically sparse, especially from sensors like LiDAR, leaving large portions of the scene without depth information. This sparsity forces networks to either learn from incomplete data or requires the use of synthetic datasets, which can suffer from a realism gap. The paper aims to improve the quality of monocular depth estimation by addressing the sparsity of training data.
Methods
The core innovation is a "densification" method using the Hilbert Maps framework. Instead of directly using sparse LiDAR point clouds, the authors first create a continuous occupancy map from the 3D points. This continuous map can then be projected into a 2D depth map at any desired resolution. This process effectively "fills in" the missing depth information in a spatially consistent manner.
The densified, continuous depth maps are then used as supervision for training a Fully Convolutional Residual Network (FCRN), inspired by ResNet-50. This network architecture is chosen for its efficiency and ability to handle residual learning for improved accuracy. The authors also employ standard data augmentation techniques like horizontal flips and color distortion, carefully avoiding scaling and translation to preserve world-space geometry. For training, they experiment with three common loss functions: Mean Squared Error (MSE), Eigen's Squared Relative Error, and the Adaptive BerHu Penalty, finding BerHu to be most effective.
Datasets
The primary dataset used is KITTI. The authors differentiate between:
- KITTI Depth (Semi-dense): This is the standard KITTI depth dataset, characterized by sparse depth maps derived from LiDAR scans.
- KITTI Continuous (Dense): This is a new benchmark created by the authors. They generate dense depth maps by applying their Hilbert Maps densification process to the KITTI Raw data. This dataset provides a continuous occupancy map as supervision, effectively filling in the sparse regions of the original KITTI Depth.
Key Insights and Ideas
- Densification is Key: The paper strongly suggests that improving the quality of the training data (making it denser and more continuous) is more impactful than solely relying on more complex network architectures or additional sensors.
- Hilbert Maps for Densification: The Hilbert Maps framework proves to be an effective tool for creating continuous occupancy maps from sparse 3D data, which can then be used for depth map densification.
- Real-World Sparsity is a Major Hurdle: The KITTI Depth dataset, despite being a standard, has a significant percentage of invalid pixels (e.g., for sky or distant objects), which poses a challenge for training robust depth estimation models.
- Continuous Data Outperforms Sparse Data: Training with the densified "KITTI Continuous" dataset consistently yields better results than training with the original sparse "KITTI Depth" dataset, even when using fewer training samples.
- Geometric Consistency is Preserved: The Hilbert Maps approach, by modeling occupancy, helps maintain spatial dependencies and provides a margin of safety around obstacles, unlike some other methods.
Key Results
The paper demonstrates significant improvements in depth estimation accuracy when training with their proposed "Sparse-to-Continuous" method.
- Quantitative Improvements: When evaluated on the Eigen Split of the KITTI Depth dataset, models trained on the "KITTI Continuous" dataset consistently outperformed those trained on the sparse "KITTI Depth" dataset across various metrics (Abs Rel, Sqr Rel, RMSE, RMSE (log), and thresholded accuracy). For instance, the "Ours (KITTI Continuous, dense, berhu)" model achieved an Abs Rel of 0.119 and an RMSE (log) of 0.180 on the 0-50m range, outperforming the sparse counterpart.
- Efficiency Gain: The "KITTI Continuous" dataset required 67.6% fewer training examples than the "KITTI Depth" dataset to achieve superior performance, highlighting the efficiency of using denser supervision.
- Qualitative Improvements: Figures 3 and 4 show that the depth predictions from models trained on continuous data are significantly less blurred, have sharper edges, and are more accurate, especially in areas that were previously sparse.
Limitations and Future Work
- Computational Cost: The authors acknowledge that the preprocessing step of computing continuous depth maps using Hilbert Maps is computationally intensive.
- Reconstructive Capabilities: The effectiveness of the Hilbert Maps framework is bounded by its inherent reconstructive capabilities.
- Future Work: The authors plan to optimize the Hilbert Maps preprocessing method, explore new architectural developments for SIDE, investigate other depth-only in-painting techniques for comparison, and analyze the impact of color distortion on depth estimates.
Summary updated