Image Guided Sparse to Dense Depth Completion

221 concepts188 papers2 proposals

Active 3D sensors like LiDAR and Time-of-Flight (ToF) cameras provide highly accurate, metric depth measurements. However, they sample the environment selectively, producing sparse point clouds where often fewer than 5%—and sometimes as few as 0.01%—of the pixels contain data. Conversely, standard cameras capture dense, high-resolution RGB images that map every pixel but lack explicit geometric depth.

Sparse-to-dense depth completion is the task of fusing these two complementary modalities. The goal is to take an RGB image and a sparse depth map to reconstruct a continuous, per-pixel depth map of the scene.

[Sparse Depth Map (LiDAR)] ---\
                               +---> [Depth Completion Network] ---> [Dense Depth Map]
[Dense Color Image (RGB)] ----/

The Sparsity Problem: Why Standard Networks Fail

Intuitively, one might expect that feeding a sparse depth map and an RGB image into a standard Convolutional Neural Network (CNN) would allow the network to learn how to fill in the blanks. In practice, standard architectures struggle significantly with highly incomplete spatial signals.

The primary issue is the ambiguity of zero. In a sparse depth tensor, unmeasured pixels are typically represented as zeros. A standard convolutional kernel cannot differentiate between a valid depth measurement of zero (an object touching the camera lens) and a missing data point. As the kernel slides across the image, it averages these structural zeros with the sparse valid points. This dilutes the true depth signal, leading to severe artifacts, systematic depth underestimation, and a failure to propagate the sparse "anchor" points across empty regions.

Furthermore, as noted by Zhao Chen et al. in their work on Estimating Depth from RGB and Sparse Sensing, propagating information effectively becomes increasingly difficult as the density drops. When only 1 in 10,000 pixels contains a valid anchor, traditional localized convolutions rarely overlap with enough valid data points to initiate meaningful propagation.

Guiding Propagation with Visual Affinities

To overcome this, successful depth completion frameworks use the dense RGB image as a structural guide. The core intuition is simple: pixels that share similar visual characteristics (color, texture, semantic context) and are not separated by an edge are highly likely to belong to the same object and share similar depths.

Instead of forcing a network to predict absolute depth values out of thin air, architectures like the Convolutional Spatial Propagation Network (CSPN), introduced by Xinjing Cheng et al. in Depth Estimation via Affinity Learned with Convolutional Spatial Propagation Network, decouple the problem into two parts:

  1. Affinity Learning: The network uses the dense RGB image to learn a localized affinity matrix—a map of how strongly each pixel is related to its neighbors.
  2. Spatial Propagation: The sparse depth points are iteratively diffused across the image grid, guided strictly by these learned affinities.

Because the affinities drop to near zero at object boundaries (where color and texture change abruptly), the depth propagation stops at edges. This prevents the notorious "depth bleeding" or "mixed-depth" artifacts where foreground object boundaries blur into the background.

[ Sparse Depth Anchor ] ---> [ High Affinity Neighbor ] ---> [ High Affinity Neighbor ]
                                      |
                                      v (Low Affinity / Edge)
                             [ Foreground Boundary ]

To prevent errors from propagating over long distances, Jinsun Park et al. proposed the Non-Local Spatial Propagation Network (NLSPN). Rather than restricting propagation to a fixed, local $3\times3$ or $5\times5$ grid, the authors designed a mechanism that dynamically selects non-local pixels across the image. This allows the model to find relevant depth anchors even when the local neighborhood is entirely devoid of sensor measurements.

Bypassing Sparsity via Pre-Densification

An alternative strategy to solving the sparse-to-dense problem is to eliminate the sparsity before the deep neural network even begins processing.

As demonstrated by Zhengxue Wang et al. in Completion as Enhancement: A Degradation-Aware Selective Image Guided Network for Depth Completion, the depth completion task can be reframed as a depth enhancement problem. By utilizing simple, non-learned morphological operations or classical interpolation techniques, the sparse point cloud is first converted into a coarse, fully dense depth map.

This initial densification step resolves the structural ambiguity of zero-valued pixels. The neural network's objective is then simplified from an ill-posed interpolation problem to a residual learning task: correcting the blurry boundaries and quantization errors introduced by the initial coarse interpolation. Similarly, Jiaqi Gu et al. utilized this concept in DenseLiDAR: A Real-Time Pseudo Dense Depth Guided Depth Completion Network, demonstrating that guiding a network with a "pseudo-dense" map significantly reduces outliers and allows the system to run at real-time speeds (up to 50Hz) suitable for autonomous driving.

Capturing Global Context with Transformers

While spatial propagation networks excel at preserving local structures and sharp boundaries, they can struggle with global consistency. If a large region of an image contains no depth anchors whatsoever, local propagation methods can stall or accumulate errors.

To address this, researchers have integrated attention mechanisms to capture long-range dependencies. Kyeongha Rho et al. introduced GuideFormer: Transformers for Image Guided Depth Completion, which utilizes a dual-branch transformer architecture. By leveraging self-attention, the model can look across the entire image frame to find structurally similar objects with valid depth anchors, instantly transferring metric scale information across vast pixel distances. This global context ensures that even when sensor data is highly clustered or missing over large intervals, the overall geometry of the scene remains coherent.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_sparse_to_dense_depth_completion_a5d354a1,
  title = {sparse-to-dense depth completion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sparse-to-dense-depth-completion/rdr_a5d354a197b448aca8b69890ca0b9614},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_sparse_to_dense_depth_completion_a5d354a1_v1,
  title = {sparse-to-dense depth completion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sparse-to-dense-depth-completion/rdr_a5d354a197b448aca8b69890ca0b9614/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake