Abstract
Deep completion which predicts dense depth from sparse depth has important applications in the fields of robotics, autonomous driving and virtual reality. It compensates for the shortcomings of low accuracy in monocular depth estimation. However, the previous deep completion works evenly processed each depth pixel and ignored the statistical properties of the depth value distribution. In this paper, we propose a self-supervised framework that can generate accurate dense depth from RGB images and sparse depth without the need for dense depth labels. We propose a novel attention-based loss that takes into account the statistical properties of the depth value distribution. We evaluate our approach on the KITTI Dataset. The experimental results show that our method achieves state-of-the-art performance. At the same time, ablation study proves that our method can effectively improve the accuracy of the results.
Introduction
In the domains of autonomous driving and robotics, precise spatial awareness is paramount. While LiDAR sensors provide highly accurate depth measurements, the resulting data is inherently sparse, consisting of disconnected points with significant gaps. Depth completion aims to bridge this gap by generating dense depth maps using standard RGB camera images as a guide.
A critical limitation in existing depth completion models is the uniform treatment of all pixels. In typical driving scenarios, the majority of pixels represent nearby objects or road surfaces, while distant objects—crucial for high-speed navigation—are represented by significantly fewer pixels. Standard models often fail to capture these fine details at the horizon. Furthermore, the reliance on dense ground truth labels for training presents a significant bottleneck due to the high cost and difficulty of data acquisition.
Methodology
The proposed framework utilizes a dual-branch CNN architecture operating in a self-supervised manner, eliminating the need for dense ground truth labels. The system integrates two primary pathways:
- Supervised Path: An encoder-decoder network processes RGB images and sparse depth inputs. It uses the available sparse LiDAR points as direct supervision to estimate a dense depth map.
- Self-Supervised Path: This path leverages temporal consistency between consecutive video frames. Using Perspective-n-Point (PnP) and RANSAC algorithms, the system calculates camera pose changes. It then "warps" one image to match another based on the predicted depth. The accuracy of the depth is measured via photometric loss—the difference between the warped and actual images.
Attention-Based Loss
The core innovation is the introduction of an Attention Loss ($L_{attention}$). To address the statistical imbalance of depth values, the authors introduce a weighting factor, $\alpha_D$: $$\alpha_D = 1 + \frac{d^{GT}}{\max(d^{GT})}$$ This formula ensures that the penalty for errors increases as the ground truth depth ($d^{GT}$) increases. This forces the network to prioritize accuracy for distant pixels, which are typically underrepresented in the dataset.
Handling Occlusions and Fusion
To manage dynamic objects and occlusions, a photometric mask is employed to filter out areas that do not follow static background geometry. Additionally, a post-fusion strategy processes RGB and depth data separately before combination, preserving the unique features of both modalities.
Experimental Results
The model was evaluated on the KITTI Dataset, the industry standard for autonomous driving benchmarks.
Quantitative and Qualitative Performance
Experimental results demonstrate that the proposed method achieves state-of-the-art performance among self-supervised depth completion techniques. Qualitative analysis shows that the model produces sharp, dense depth maps with well-defined object boundaries, even at long ranges.
Ablation Study
Ablation studies were conducted to isolate the impact of the attention-based loss. The results confirmed that incorporating distance-aware weighting significantly reduces error compared to standard loss functions, validating the hypothesis that prioritizing distant pixels improves overall scene reconstruction.
Conclusion and Future Work
This research presents a robust self-supervised framework for depth completion that effectively addresses the challenges of data sparsity and statistical imbalance. While the attention-based loss provides a clear accuracy boost, future work should address real-time inference speeds for deployment on autonomous platforms. Additionally, exploring the system's robustness in extreme weather conditions and its generalizability to indoor environments remains a priority.
Summary updated