Learning Depth Estimation without Dense Ground Truth

627 concepts489 papers4 proposals

To perceive the 3D world, a computer vision model must map 2D pixels to 3D distances. Traditionally, training such models required dense, pixel-aligned ground-truth depth maps. However, capturing this data is a logistical and financial challenge. High-fidelity LiDAR sensors are expensive, struggle in specific environments like underwater or highly reflective surfaces, and produce sparse or noisy measurements. This creates a severe ground-truth data scarcity bottleneck.

To bypass this constraint, a prominent research direction focuses on training depth estimation models without relying on dense, real-world ground-truth depth maps. Instead of treating depth estimation as a pure supervised regression problem, these approaches leverage geometric constraints, cross-modal distillation, and alternative supervision strategies to learn the underlying 3D structure of the world from readily available, unlabeled data.

The Core Intuition: Geometry as a Proxy for Supervision

How can a model learn absolute or relative distance without ever seeing a depth label? The key insight is that 3D geometry dictates 2D image transformation. If we know how a camera moves, or if we have two cameras looking at the same scene from different angles (stereo vision), the correct depth map is the only one that can perfectly reconstruct one view from another.

This principle underpins unsupervised monocular and stereo depth estimation. For instance, in Unsupervised Monocular Depth Estimation with Left-Right Consistency, the authors enforce a left-right consistency loss during training on stereo image pairs. The network predicts a disparity map for the left image, warps the right image using this disparity, and minimizes the photometric error between the actual left image and the reconstructed one.

[Left Image]  ---> [Depth Network] ---> [Predicted Disparity]
                                                |
[Right Image] -------------------------> [Warping Module] ---> [Reconstructed Left Image]
                                                                        |
                                                              (Minimize Photometric Error)

When moving from stereo pairs to monocular video sequences, the problem becomes harder because both depth and camera motion (pose) are unknown. Chaoyang Wang et al. addressed this in Learning Depth From Monocular Videos Using Direct Methods by integrating a differentiable Direct Visual Odometry (DVO) module. Instead of letting a separate network guess the camera pose loosely, the geometric optimization is embedded directly into the loop, closing the performance gap with supervised methods. To make this matching robust against lighting changes or textureless regions where pixel-to-pixel matching fails, Huangying Zhan et al. introduced a deep feature reconstruction loss in Unsupervised Learning of Monocular Depth Estimation and Visual Odometry With Deep Feature Reconstruction, matching high-level feature representations rather than raw pixel intensities.

Cross-Modal Distillation and Domain Adaptation

Another powerful strategy to combat data scarcity is to transfer knowledge from domains where data or models are abundant to domains where they are practically non-existent.

Consider underwater depth estimation, where traditional sensors like LiDAR fail completely due to water absorption and scattering. Jinghe Yang et al. developed the PUDE framework in Physics-Informed Knowledge Transfer for Underwater Monocular Depth Estimation. Instead of trying to collect impossible underwater ground truth, they took a high-performance model trained on standard "in-air" images and adapted it by embedding a physical underwater imaging model into the training loop. This allowed the network to correct for underwater blur and turbidity implicitly.

Similarly, capturing ground-truth depth for 360-degree panoramic images is exceptionally difficult. Ning-Hsu Wang et al. bypassed this in Depth Anywhere: Enhancing 360 Monocular Depth Estimation via Perspective Distillation and Unlabeled Data Augmentation using a teacher-student distillation process. They leveraged powerful, pre-existing models trained on standard perspective views to generate pseudo-depth labels for massive amounts of unlabeled panoramic data.

When models are trained on synthetic data (where perfect ground truth is cheap) but need to deploy in the real world, they face a severe domain gap. In AdaStereo: An Efficient Domain-Adaptive Stereo Matching Approach, the authors resolved this by aligning synthetic and real domains across multiple levels—input images, internal features, and output predictions—achieving robust real-world depth estimation without requiring real-world annotations.

Scaling Up with Pseudo-Labels and Foundation Models

The ultimate realization of this paradigm is the creation of depth foundation models that scale to tens of millions of images. Lihe Yang et al. demonstrated this with Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data. Rather than relying on specialized hardware to scale up labeled datasets, they trained a model on 62 million unlabeled images. They achieved this by using a data engine that inherits strong semantic priors from a pre-trained foundation model (DINOv2) and enforces a student-teacher framework to ensure zero-shot generalization across diverse, unseen real-world scenes.

Pre-training on massive unlabeled data also fundamentally changes downstream tasks like 3D object detection. Dennis Park et al. showed in Is Pseudo-Lidar needed for Monocular 3D Object detection? that end-to-end monocular 3D detectors can achieve state-of-the-art performance without complex, multi-stage "Pseudo-LiDAR" pipelines, provided they leverage dense depth pre-training on large-scale unlabeled datasets.

Reformulating the Target: Ordinal and Sparse Supervision

If absolute metric depth is too hard to supervise, can we change the target entirely?

Eyke Hüllermeier et al. explored this in Monocular Depth Estimation via Listwise Ranking Using the Plackett-Luce Model. Instead of forcing a network to predict exact numerical distances (e.g., 5.4 meters), they treated depth estimation as an ordering problem (e.g., object A is closer than object B). By training on purely ordinal data, which is far easier to collect or annotate, the model achieved remarkable zero-shot generalization and could even recover metric depth implicitly.

Alternatively, we can combine the scalability of self-supervision with highly sparse, cheap physical measurements. In Robust Semi-Supervised Monocular Depth Estimation with Reprojected Distances, Jie Li demonstrated that a model can achieve state-of-the-art accuracy using as few as 4 LiDAR beams (less than 100 points per image), proving that dense supervision is entirely unnecessary if the underlying geometric reprojection losses are properly formulated.

Finally, Jamie Watson et al. showed in Self-Supervised Monocular Depth Hints that even noisy, off-the-shelf stereo algorithms can be used as "hints" to guide a self-supervised network out of local minima, selectively adopting the hints only when they improve the overall photometric consistency.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 513 papers
  2. Version 1 · 2026-10-04 · 534 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_ground_truth_data_scarcity_24be46b4,
  title = {ground-truth data scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/ground-truth-data-scarcity/rdr_24be46b482274462a0512fbe759ccc61},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_ground_truth_data_scarcity_24be46b4_v2,
  title = {ground-truth data scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/ground-truth-data-scarcity/rdr_24be46b482274462a0512fbe759ccc61/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake