Abstract
Image-guided depth completion aims at generating a dense depth map from sparse LiDAR data and RGB image. Recent methods have shown promising performance by reformulating it as a classification problem with two sub-tasks: depth discretization and probability prediction. They divide the depth range into several discrete depth values as depth categories, serving as priors for scene depth distributions. However, previous depth discretization methods are easy to be impacted by depth distribution variations across different scenes, resulting in suboptimal scene depth distribution priors. To address the above problem, we propose a progressive depth decoupling and modulating network, which incrementally decouples the depth range into bins and adaptively generates multi-scale dense depth maps in multiple stages. Specifically, we first design a Bins Initializing Module (BIM) to construct the seed bins by exploring the depth distribution information within a sparse depth map, adapting variations of depth distribution. Then, we devise an incremental depth decoupling branch to progressively refine the depth distribution information from global to local. Meanwhile, an adaptive depth modulating branch is developed to progressively improve the probability representation from coarse-grained to fine-grained. And the bi-directional information interactions are proposed to strengthen the information interaction between those two branches (sub-tasks) for promoting information complementation in each branch. Further, we introduce a multi-scale supervision mechanism to learn the depth distribution information in latent features and enhance the adaptation capability across different scenes. Experimental results on public datasets demonstrate that our method outperforms the state-of-the-art methods.
Introduction
Depth completion is a critical task in computer vision, particularly for autonomous robotics and augmented reality. It involves generating a high-resolution, dense depth map from a sparse set of depth measurements—typically from LiDAR—and a corresponding RGB image. This task is inherently challenging due to the ambiguity of 3D space when viewed from a single 2D perspective.
Recent advancements have shifted the paradigm from direct regression to classification-based approaches. These methods discretize the depth range into "bins" and predict the probability of each pixel belonging to a specific bin. However, static discretization often fails to account for the vast variations in depth distributions across different environments. For instance, a model optimized for outdoor driving may perform poorly in a confined indoor setting where the depth range is significantly narrower. This paper introduces a Progressive Depth Decoupling and Modulating (PDDM) network that treats depth completion as a dynamic classification task, adaptively discretizing the depth range based on the specific scene's distribution.
Methodology
The PDDM architecture is built upon a dual-branch structure designed to refine depth information in a coarse-to-fine manner. The core innovation lies in how the model handles depth discretization and probability prediction as interdependent, evolving processes.
Bins Initializing Module (BIM)
Instead of initializing depth bins arbitrarily, the model utilizes the spatial coordinates of valid sparse depth points to create "seed bins." By grounding the initial depth categories in the actual geometry of the scene, the model ensures that the discretization process is relevant to the specific environment from the outset.
Dual-Branch Architecture
- Incremental Depth Decoupling Branch: This branch progressively refines the depth bins using Transformer blocks. It integrates global scene context with local features to narrow down the depth intervals where pixels are most likely to exist. This allows the model to focus its "binning budget" on the most relevant depth ranges.
- Adaptive Depth Modulating Branch: Running in parallel, this branch refines probability predictions. It begins with area-level coarse predictions and sharpens them into pixel-level fine-grained probabilities as the resolution increases through the decoder stages.
Bi-directional Information Interaction
A key feature of the PDDM is the interaction between these two branches. Bin embeddings from the decoupling branch guide the probability predictions, while depth features from the modulating branch help refine the bin partitions. This synergy ensures that the discretization and classification tasks are mutually reinforcing.
Key Insights and Results
The PDDM framework highlights that sparse LiDAR points are more than just targets; they are valuable priors for scene geometry. By focusing on the relative depth range of a specific frame rather than a fixed global range, the model achieves higher efficiency and accuracy.
Performance Benchmarks
The model was evaluated on three major datasets: NYU-Depth-v2 (indoor), ScanNet-v2 (complex indoor), and KITTI (outdoor).
- NYU-Depth-v2: The model achieved a state-of-the-art RMSE of 0.095m.
- KITTI: It reached an RMSE of 1107.29, significantly outperforming established baselines.
- Qualitative Analysis: Visual results demonstrate superior performance in capturing fine details, such as thin object boundaries and small structures, which are typically blurred by standard regression models.
Conclusion
The Progressive Depth Decoupling and Modulating network offers a flexible and robust solution for depth completion. By adaptively discretizing the depth range and refining predictions across multiple scales, it overcomes the limitations of fixed-bin classification methods. While the model shows some sensitivity to extreme sparsity and sensor noise, its ability to capture fine-grained scene geometry sets a new standard for the field. Future work will focus on enhancing robustness to extremely low-density depth samples.
Summary updated