TL;DR
This paper introduces a Progressive Depth Decoupling and Modulating network that treats depth completion as a dynamic classification task. By adaptively discretizing the depth range based on the specific scene's distribution and refining it across multiple scales, the model achieves state-of-the-art accuracy on both indoor and outdoor benchmarks.
What Problem Does This Paper Address?
Depth completion is the task of taking a sparse set of depth measurements (like those from a LiDAR sensor) and a high-resolution RGB image to produce a dense depth map. This is essential for robotics and AR, but it's difficult because monocular images are inherently ambiguous in 3D space.
Recent high-performing methods treat this as a classification problem: they divide the depth range into "bins" (discretization) and predict the probability of a pixel belonging to each bin. However, most existing methods use fixed bins or a single-scale approach. This leads to errors when the depth distribution changes—for example, a model trained on a dataset with a 0–10 meter range might struggle in a small room where everything is between 1 and 3 meters. The authors aim to solve this "one-size-fits-all" binning problem.
Methods
The authors propose a dual-branch architecture that works in a coarse-to-fine manner:
- Bins Initializing Module (BIM): Instead of starting with arbitrary bins, the model uses the spatial coordinates of the valid sparse depth points to create "seed bins." This ensures the initial depth categories are grounded in the actual scene geometry.
- Incremental Depth Decoupling Branch: This branch progressively refines the depth bins. It uses Transformer blocks to integrate global scene information with local features, narrowing down the depth intervals where pixels are most likely to exist.
- Adaptive Depth Modulating Branch: Running in parallel, this branch refines the probability predictions. It starts with area-level coarse predictions and sharpens them into pixel-level fine-grained probabilities as the resolution increases.
- Bi-directional Information Interaction: The two branches "talk" to each other. Bin embeddings guide the probability predictions, and depth features from the modulating branch help refine the bin partitions.
- Multi-scale Supervision: The model doesn't just calculate loss at the final output. It uses a multi-scale loss function (including a novel bin-center density loss) to supervise the depth distribution at every stage of the progressive refinement.
Datasets
The model was evaluated on three standard benchmarks:
- NYU-Depth-v2: A large-scale indoor dataset (464 scenarios).
- ScanNet-v2: An indoor RGB-D video dataset with complex geometry.
- KITTI Depth Completion Dataset: A massive outdoor dataset for autonomous driving.
Key Insights and Ideas
- Sparse Depth as a Prior, Not Just a Target: The authors realized that the sparse LiDAR points themselves contain the "blueprint" of the scene's depth distribution. Using these points to initialize bins (via the BIM) is much more effective than relying solely on the RGB image.
- Relative vs. Absolute Range: The paper highlights that dataset-wide depth ranges (e.g., 0.6m to 10m) are often too broad for specific scenes. By constraining the discretization to the relative range found in the sparse map of a specific frame, the model can allocate its "binning budget" much more efficiently.
- Progressive Refinement: By decoupling the task into multiple stages, the model handles global structure first and then focuses on fine details (like chair legs or faucets), which are often lost in single-scale regression models.
Key Results
The proposed method consistently outperformed state-of-the-art (SOTA) methods across all datasets:
- NYU-Depth-v2: Achieved an RMSE of 0.095, surpassing previous leaders like CostDCNet (0.097) and Lee et al. (0.104).
- KITTI: Achieved an RMSE of 1107.29, a significant improvement over established methods like Nconv (1268.22).
- Qualitative Superiority: Visual results show the model is exceptionally good at capturing "tiny objects" and maintaining sharp object boundaries, which are common failure points for depth completion.
- Efficiency: The BIM module is highly efficient, achieving better results with only 58% of the parameters required by standard convolutional backbones like ResNet34.
Limitations and Future Work
The authors acknowledge a few key limitations:
- Dependency on Sparsity: While the model is robust, its performance still relies on the quality and distribution of the sparse input. In cases of extreme sparsity (e.g., only 5 points), performance degrades toward monocular estimation levels.
- Noise Sensitivity: "Discordant" or noisy points in the sparse depth map can occasionally mislead the model's judgment of depth categories.
- Future Directions: The team plans to investigate discretization methods that are even more robust to extremely low-density depth samples and sensor noise.
Summary updated