Progressive Depth Decoupling and Modulating for Flexible Depth Completion

Zhiwen Yang, Jiehua Zhang, Liang Li, Chenggang Yan, Yaoqi Sun, Haibing Yin

TL;DR

This paper introduces a Progressive Depth Decoupling and Modulating network that treats depth completion as a dynamic classification task. By adaptively discretizing the depth range based on the specific scene's distribution and refining it across multiple scales, the model achieves state-of-the-art accuracy on both indoor and outdoor benchmarks.

What Problem Does This Paper Address?

Depth completion is the task of taking a sparse set of depth measurements (like those from a LiDAR sensor) and a high-resolution RGB image to produce a dense depth map. This is essential for robotics and AR, but it's difficult because monocular images are inherently ambiguous in 3D space.

Recent high-performing methods treat this as a classification problem: they divide the depth range into "bins" (discretization) and predict the probability of a pixel belonging to each bin. However, most existing methods use fixed bins or a single-scale approach. This leads to errors when the depth distribution changes—for example, a model trained on a dataset with a 0–10 meter range might struggle in a small room where everything is between 1 and 3 meters. The authors aim to solve this "one-size-fits-all" binning problem.

Methods

The authors propose a dual-branch architecture that works in a coarse-to-fine manner:

  1. Bins Initializing Module (BIM): Instead of starting with arbitrary bins, the model uses the spatial coordinates of the valid sparse depth points to create "seed bins." This ensures the initial depth categories are grounded in the actual scene geometry.
  2. Incremental Depth Decoupling Branch: This branch progressively refines the depth bins. It uses Transformer blocks to integrate global scene information with local features, narrowing down the depth intervals where pixels are most likely to exist.
  3. Adaptive Depth Modulating Branch: Running in parallel, this branch refines the probability predictions. It starts with area-level coarse predictions and sharpens them into pixel-level fine-grained probabilities as the resolution increases.
  4. Bi-directional Information Interaction: The two branches "talk" to each other. Bin embeddings guide the probability predictions, and depth features from the modulating branch help refine the bin partitions.
  5. Multi-scale Supervision: The model doesn't just calculate loss at the final output. It uses a multi-scale loss function (including a novel bin-center density loss) to supervise the depth distribution at every stage of the progressive refinement.

Datasets

The model was evaluated on three standard benchmarks:

  • NYU-Depth-v2: A large-scale indoor dataset (464 scenarios).
  • ScanNet-v2: An indoor RGB-D video dataset with complex geometry.
  • KITTI Depth Completion Dataset: A massive outdoor dataset for autonomous driving.

Key Insights and Ideas

  • Sparse Depth as a Prior, Not Just a Target: The authors realized that the sparse LiDAR points themselves contain the "blueprint" of the scene's depth distribution. Using these points to initialize bins (via the BIM) is much more effective than relying solely on the RGB image.
  • Relative vs. Absolute Range: The paper highlights that dataset-wide depth ranges (e.g., 0.6m to 10m) are often too broad for specific scenes. By constraining the discretization to the relative range found in the sparse map of a specific frame, the model can allocate its "binning budget" much more efficiently.
  • Progressive Refinement: By decoupling the task into multiple stages, the model handles global structure first and then focuses on fine details (like chair legs or faucets), which are often lost in single-scale regression models.

Key Results

The proposed method consistently outperformed state-of-the-art (SOTA) methods across all datasets:

  • NYU-Depth-v2: Achieved an RMSE of 0.095, surpassing previous leaders like CostDCNet (0.097) and Lee et al. (0.104).
  • KITTI: Achieved an RMSE of 1107.29, a significant improvement over established methods like Nconv (1268.22).
  • Qualitative Superiority: Visual results show the model is exceptionally good at capturing "tiny objects" and maintaining sharp object boundaries, which are common failure points for depth completion.
  • Efficiency: The BIM module is highly efficient, achieving better results with only 58% of the parameters required by standard convolutional backbones like ResNet34.

Limitations and Future Work

The authors acknowledge a few key limitations:

  • Dependency on Sparsity: While the model is robust, its performance still relies on the quality and distribution of the sparse input. In cases of extreme sparsity (e.g., only 5 points), performance degrades toward monocular estimation levels.
  • Noise Sensitivity: "Discordant" or noisy points in the sparse depth map can occasionally mislead the model's judgment of depth categories.
  • Future Directions: The team plans to investigate discretization methods that are even more robust to extremely low-density depth samples and sensor noise.

Summary updated

Abstract

Image-guided depth completion aims at generating a dense depth map from sparse LiDAR data and RGB image. Recent methods have shown promising performance by reformulating it as a classification problem with two sub-tasks: depth discretization and probability prediction. They divide the depth range into several discrete depth values as depth categories, serving as priors for scene depth distributions. However, previous depth discretization methods are easy to be impacted by depth distribution variations across different scenes, resulting in suboptimal scene depth distribution priors. To address the above problem, we propose a progressive depth decoupling and modulating network, which incrementally decouples the depth range into bins and adaptively generates multi-scale dense depth maps in multiple stages. Specifically, we first design a Bins Initializing Module (BIM) to construct the seed bins by exploring the depth distribution information within a sparse depth map, adapting variations of depth distribution. Then, we devise an incremental depth decoupling branch to progressively refine the depth distribution information from global to local. Meanwhile, an adaptive depth modulating branch is developed to progressively improve the probability representation from coarse-grained to fine-grained. And the bi-directional information interactions are proposed to strengthen the information interaction between those two branches (sub-tasks) for promoting information complementation in each branch. Further, we introduce a multi-scale supervision mechanism to learn the depth distribution information in latent features and enhance the adaptation capability across different scenes. Experimental results on public datasets demonstrate that our method outperforms the state-of-the-art methods. The code will be open-sourced at [this https URL](https://github.com/Cisse-away/PDDM).
BibTeX
@article{yang2024progressive,
  author = {Zhiwen Yang and Jiehua Zhang and Liang Li and Chenggang Yan and Yaoqi Sun and Haibing Yin},
  title = {{Progressive Depth Decoupling and Modulating for Flexible Depth Completion}},
  journal = {IEEE Trans. Instrumentation \& Measurement},
  year = {2024},
  doi = {10.1109/tim.2024.3420352},
}

Loading related content...

Figure