Guided Upsampling for Sparse Depth Completion

This direction is merged into another direction: sparse-to-dense depth completion. Its last version stays readable and citable here.

Version 1 of 137 papers38 conceptsfrozen 2026-10-04

In autonomous navigation and environmental monitoring, we often face a fundamental hardware trade-off: sensors that measure distance directly, such as LiDAR or spaceborne altimeters, are highly accurate but spatially sparse. A typical LiDAR scan might cover less than 6% of an image's pixels, leaving vast "blind spots" between scan lines. To build a reliable 3D model, we must transform these isolated points into a continuous, dense depth map.

The research direction of sparse depth completion focuses on "filling the blanks" by leveraging the high-resolution structural information present in standard RGB images to guide the interpolation of sparse geometric data.

The Core Intuition: RGB as a Geometric Blueprint

Why does an RGB image help us predict depth? The central insight is that visual boundaries—changes in color, texture, or lighting—frequently coincide with geometric boundaries. If two adjacent pixels in an image belong to the same blue car, they likely share a similar depth; if one pixel is blue and the next is green grass, a depth discontinuity is probable.

Researchers like Yiqi Zhong and Cho-Ying Wu have formalized this by modeling the semantic correlation between modalities. Their work on Deep RGB-D Canonical Correlation Analysis For Sparse Depth Completion demonstrates that the network can "hallucinate" missing depth by identifying shared features between the color and depth domains.

To make this guidance effective, the network must adapt its behavior based on the image content. Jie Tang and Wei Feng introduced Learning Guided Convolutional Network for Depth Completion, which uses the RGB image to dynamically generate spatially-variant convolution kernels. Instead of applying the same filter everywhere, the model changes its weights at every pixel to respect the edges found in the photograph.

Bridging the Gap with Intermediate Representations

Directly mapping sparse points to a dense grid is a massive leap. Many successful approaches introduce an intermediate geometric "bridge" to simplify the task.

  1. Surface Normals: Jiaxiong Qiu and Yinda Zhang developed DeepLiDAR, which first predicts surface normals—the orientation of surfaces—from the RGB image. Because normals are locally smooth and highly correlated with object shapes, they provide a stable geometric guide that constrains how the sparse depth points should be interpolated.
  2. Planar Priors: In man-made environments, surfaces are often flat. Yiran Zhong and Yuchao Dai utilize a Piecewise Planar Model to propagate geometric parameters across neighboring patches. If three sparse points fall on a wall, the model assumes the entire wall follows that plane, filling millions of pixels with a single geometric rule.
  3. Semantic Awareness: Sometimes, color isn't enough. Danish Nazir introduced SemAttNet, which uses semantic segmentation to inform the fusion process. Knowing that a pixel belongs to a "tree" versus a "sidewalk" allows the model to apply different smoothing logic, preventing depth "bleeding" across object boundaries.

Attention and Fusion Strategies

A major challenge in this field is the imbalance between the two inputs: the RGB image is dense and rich in detail, while the LiDAR data is sparse but provides the "ground truth" scale. If the model relies too much on the image, it loses accuracy; if it relies too much on the sparse points, the output remains full of holes.

Modern architectures use attention mechanisms to resolve this conflict. Kevin Christiansen Marsim and Jinwoo Jeon proposed CHADET, which uses cross-hierarchical attention to weigh features from different scales. Similarly, Yingyu Wang developed an attention-based loss function that forces the network to prioritize distant, sparse objects that are often ignored by standard training routines.

From Interpolation to Generation

Recent work has shifted from treating depth completion as a simple interpolation problem to treating it as a generative one. Ming Yuan and Sichao Wang introduced DenseFormer, which uses a conditional diffusion model. Instead of calculating a single depth value, the model iteratively refines a noisy map into a high-fidelity depth image, conditioned on the sparse LiDAR points. This allows the system to produce sharper edges and more realistic textures in areas where sensor data is completely missing.

This evolution—from simple statistical "filling" (Gregory Dudek and Luz Abril Torres-Méndez in early MRF models) to sophisticated generative transformers—has enabled applications ranging from sub-meter canopy height mapping (Jamie Tolan) to real-time 3D object detection for autonomous vehicles (Xiaopei Wu).

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_guided_upsampling_for_sparse_depth_compl_46e4b221,
  title = {Guided Upsampling for Sparse Depth Completion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/guided-upsampling-for-sparse-depth-completion/rdr_46e4b221fcaf4f72966bdb7a0e2fe703},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_guided_upsampling_for_sparse_depth_compl_46e4b221_v1,
  title = {Guided Upsampling for Sparse Depth Completion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/guided-upsampling-for-sparse-depth-completion/rdr_46e4b221fcaf4f72966bdb7a0e2fe703/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}