Monocular Depth Estimation and Completion

151 concepts144 papers1 proposal

The human visual system effortlessly perceives a three-dimensional world from two-dimensional retinal images by combining contextual knowledge—like object sizes, shading, and perspective—with binocular cues. For a computer operating on a single RGB image, however, recovering this third dimension is a fundamentally ill-posed problem. Infinitely many 3D scenes can project onto the exact same 2D pixel grid.

To resolve this ambiguity, the field of computer vision has converged on a powerful paradigm: unifying monocular depth prediction (inferring dense depth from a single image using learned priors) with depth completion (using sparse, physical measurements from sensors like LiDAR, radar, or direct Time-of-Flight to anchor the scale). As demonstrated by Ambrus et al. in their work on unified networks, these two tasks are deeply symbiotic. Sparse geometric points provide the absolute scale boundaries that monocular models lack, while the dense RGB image provides the structural boundaries needed to intelligently interpolate between those sparse points.

[Dense RGB Image]  ───┐
                      ▼
               [Fusion Network] ───► [High-Fidelity Dense Depth Map]
                      ▲
[Sparse Sensor Data] ─┘
(LiDAR / Radar / dToF)

Distilling Knowledge from Depth Foundation Models

One of the most significant shifts in achieving high-fidelity depth estimation is moving away from training models entirely from scratch on small, scene-specific datasets. Instead, modern architectures leverage depth foundation models—large neural networks pre-trained on millions of diverse images that possess a robust, generalized understanding of spatial layouts.

The core intuition here is that a foundation model already knows what the world looks like; the challenge is transferring that knowledge to specific downstream applications without losing fine-grained details.

  • High-Frequency Refinement: Traditional depth estimation losses tend to smooth out sharp edges, treating a crisp object boundary as a gradual slope. To combat this, the HFD-Teacher framework uses a pre-trained foundation model to distill multi-scale, high-frequency structural details into standard depth completion networks, forcing the model to preserve sharp object boundaries.
  • Zero-Shot Generalization: Because these foundation models capture universal geometric principles, they can be deployed in entirely unseen environments without retraining. For example, Li et al. utilize zero-shot depth foundation models paired with graph-based optimization to estimate the heights of skyscrapers from single satellite images. Similarly, Sun et al. show that curriculum learning on massive multi-domain datasets allows a depth model to provide scale-consistent dense mapping for visual odometry in entirely novel outdoor driving environments.

Resolving Ambiguity with Physical and Multi-Modal Cues

While foundation models provide excellent semantic priors, certain surfaces—such as transparent glass, shiny metal, or textureless white walls—break the assumptions of standard visual models. To resolve these ambiguities, researchers inject physical properties of light or alternative sensor modalities directly into the learning process.

Polarization and Defocus Blur

When light reflects off a surface, it becomes polarized based on the surface's orientation and material properties. Ikemura et al. leverage this by introducing polarization prompt fusion tuning. By feeding polarization cues into a pre-trained RGB model, the network can infer the precise geometry of highly reflective or transparent objects that would otherwise be invisible to standard depth sensors. Alternatively, modeling the physics of camera optics—such as camera-independent defocus blur—allows networks to calculate depth based on how light spreads through a lens, bypassing the need to rely purely on semantic patterns.

Sparse Sensor Guidance

Sensors like direct Time-of-Flight (dToF) or radar provide highly accurate depth anchors but suffer from extreme sparsity or noise. Frameworks like DEPTHOR combine noisy dToF signals with monocular depth estimates to filter out sensor artifacts. When dealing with the extreme sparsity of radar, architectures like GET-UP employ attention-enhanced Graph Neural Networks (GNNs) to geometrically upsample the sparse radar points in 3D space before fusing them with 2D image features, ensuring that the sparse guidance aligns perfectly with the visual boundaries of the scene.

Enforcing Structural and Spatio-Temporal Consistency

A major failure mode of independent frame-by-frame depth estimation is temporal flickering and geometric inconsistency. If a robot views a room from two slightly different angles, the estimated 3D structures must match perfectly in 3D space.

Frame t     ──► [Depth Net] ──► Depth t ──┐
                                           ▼
                                  [Warp & Compare] ──► Photometric / Geometric Loss
                                           ▲
Frame t+1   ──► [Pose Net]  ──► Camera Motion ┘

To enforce this, self-supervised methods leverage video sequences by jointly predicting depth and camera ego-motion (pose). By using the predicted camera movement to warp one video frame into the next, the network can use a photometric consistency loss as its supervisor.

To make this constraint even tighter, Fink et al. utilize a multi-view differentiable rendering pipeline. By rendering the estimated depth maps back into neighboring camera views, the network optimizes for true 3D view consistency, outperforming traditional multi-view stereo approaches in complex indoor environments.

Furthermore, capturing these relationships requires architectures that can handle long-range context across space and time:

  • Zhao et al. introduce spatial-temporal memory modules to track geometric features across long video sequences, preventing scale drift over time.
  • Wang et al. combine monocular cues with multi-view geometry and velocity guidance, proving that incorporating physical motion priors can drastically improve self-supervised depth accuracy while maintaining real-time processing speeds.
  • In highly constrained environments like surgical endoscopy, where camera motion is erratic and tissues are deformation-prone, Chen et al. employ a Mamba-based architecture with a "depth-first" scanning mechanism. This allows the network to capture the continuous, smooth structure of internal organs without the prohibitive computational cost of standard global attention mechanisms.

By anchoring deep neural networks with the invariant laws of physics, multi-view geometry, and sparse sensor measurements, modern depth estimation transforms a single 2D image from a flat projection into a highly accurate, structurally coherent gateway to the 3D world.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_state_of_the_art_depth_estimation_b5ca6d88,
  title = {state-of-the-art depth estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/state-of-the-art-depth-estimation/rdr_b5ca6d88cb8840429d1eb4d9f2b89bb5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_state_of_the_art_depth_estimation_b5ca6d88_v1,
  title = {state-of-the-art depth estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/state-of-the-art-depth-estimation/rdr_b5ca6d88cb8840429d1eb4d9f2b89bb5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake