Guided Depth Super Resolution for Cross Modal Fusion

This direction is split into other directions: sensor resolution disparity. Its last version stays readable and citable here.

Version 1 of 159 papers60 conceptsfrozen 2026-10-04

In modern computer vision, we face a fundamental hardware disparity. High-resolution (HR) RGB cameras are inexpensive and ubiquitous, providing dense 4K textures. However, the sensors that measure actual 3D distance—such as LiDAR, Time-of-Flight (ToF), or Thermal cameras—are physically constrained by high manufacturing costs and power requirements. As noted by Xinni Jiang and Chunle Guo, this creates a "resolution mismatch" where we must align sparse, grainy depth data with sharp, high-definition visual imagery.

Guided Depth Super-Resolution (DSR) is the process of upscaling these low-resolution (LR) depth maps by using the HR RGB image as a structural blueprint. The goal is to "borrow" the sharp edges from the color image to fill in the missing geometric details of the depth map.

The Challenge of Structural Disparity

The most intuitive way to upscale depth is simple interpolation, but this fails because it blurs object boundaries and amplifies sensor noise. To solve this, researchers like Chuhua XIAN and Kun Qian use multi-scale progressive fusion networks. These architectures treat the RGB image as a guide, extracting high-frequency spatial details (like the edge of a table) and injecting them into the depth map at multiple resolutions.

However, a major pitfall in this approach is "texture copying." If a wall has a high-contrast poster on it, a naive model might mistakenly assume the depth changes at the poster's edges. To prevent this, Wuxuan Shi and Bo Du developed uncertainty-aware transmission frameworks. By estimating where RGB textures and depth geometry actually disagree, these models can selectively ignore visual cues that don't correspond to real physical edges.

Correcting Sensor Misalignment

In the real world, the RGB camera and the depth sensor are rarely in the exact same physical position. This creates parallax, where the two images don't perfectly overlap. Guy Shacht and Daniel Cohen-Or addressed this with Cross-Modality Super-Resolution (CMSR), which includes an internal, learnable deformation module. This module automatically warps and aligns the sensors on-the-fly, allowing the system to learn from a single pair of images without external training data.

This alignment problem is even more acute in mobile devices. Andreas Meuleman and Min H. Kim identified the "floating lens" problem caused by Optical Image Stabilization (OIS) in smartphones. Their FloatingFusion system performs per-snapshot calibration to ensure that the low-resolution ToF data correctly maps onto the stabilized high-resolution stereo imagery.

Mathematical and Generative Refinements

Beyond simple feature fusion, researchers are applying rigorous mathematical frameworks to ensure geometric consistency:

  • Anisotropic Diffusion: Nando Metzger and Konrad Schindler introduced DADA, which uses deep learning to guide the mathematical process of diffusion. This allows depth values to "flow" and fill in gaps while being strictly blocked by the sharp boundaries found in the RGB guide.
  • Photometric Integration: Bjoern Haefner and Daniel Cremers demonstrated that by combining low-resolution depth with photometric stereo (analyzing how light hits a surface), one can "resurrect" fine-scale details like fabric wrinkles or facial features that are invisible to the depth sensor but present in the color stream.
  • Generative Models: Recent work by Yuan Shi leverages diffusion models (DSR-Diff) to treat depth upscaling as a generative task. By operating in a compact latent space, these models can synthesize plausible high-resolution geometric structures that match the visual context of the scene.

Why It Works

The intuition behind the success of this research direction is structural redundancy. While the values of RGB pixels (color) and depth pixels (distance) are different, their discontinuities (edges) are highly correlated. By treating the RGB image not as a source of data, but as a source of spatial constraints, these methods bridge the hardware gap. As Haoye Dong shows in the DuCos framework, framing this as a constrained optimization problem allows us to leverage the massive prior knowledge of foundation models to produce depth maps that are both metrically accurate and visually sharp.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_guided_depth_super_resolution_for_cross__ee72eb1b,
  title = {Guided Depth Super Resolution for Cross Modal Fusion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/guided-depth-super-resolution-for-cross-modal-fusion/rdr_ee72eb1bc6dc40fc82225141b5219a0a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_guided_depth_super_resolution_for_cross__ee72eb1b_v1,
  title = {Guided Depth Super Resolution for Cross Modal Fusion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/guided-depth-super-resolution-for-cross-modal-fusion/rdr_ee72eb1bc6dc40fc82225141b5219a0a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}