Scaling Depth Estimation with Self-Supervision and Synthetic Transfer

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2534 papers565 conceptsfrozen 2026-10-04

To a human, depth is intuitive. We perceive the world in 3D effortlessly. For a computer, however, an image is just a flat grid of pixels. Teaching a machine to recover the missing third dimension—Monocular Depth Estimation (MDE)—is one of the most significant challenges in computer vision.

The traditional approach is supervised learning: show a model millions of images paired with "ground truth" depth maps (often captured by LiDAR) and let it learn the mapping. But as Lihe Yang and colleagues point out in Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data, high-quality depth labels are incredibly difficult to scale. LiDAR is expensive, sparse, and often fails on reflective or transparent surfaces. This has led to a "data bottleneck" that prevents models from generalizing to the messy, diverse reality of the open world.

Recent research has pivoted toward a more scalable strategy: if we cannot easily label the real world, we must learn from what we already have—unlabeled images, geometric constraints, and synthetic simulations.

Learning from Geometry without Labels

The most elegant solution to the data bottleneck is self-supervision. Instead of a LiDAR sensor, these models use the geometry of the world itself as a teacher.

In a video sequence or a stereo pair, if we know the depth of a pixel and how the camera moved, we can mathematically "warp" one image to look like the other. If the predicted depth is correct, the warped image will match the actual second image perfectly. This "photometric consistency" acts as a supervisory signal. Jamie Watson demonstrated in Self-Supervised Monocular Depth Hints that we can even use "hints" from classic, non-neural stereo algorithms to guide these networks out of local minima, significantly improving their ability to see thin structures.

This principle extends beyond standard cameras. Satoshi Ikehata and his team applied similar self-supervised logic to 360-degree panoramic images in Distortion-Aware Self-Supervised 360° Depth Estimation, overcoming the lack of panoramic LiDAR datasets by accounting for the unique geometric distortions of equirectangular projections.

Bridging the Reality Gap

If real-world labels are scarce, why not use synthetic data? Video games and simulators provide "perfect" depth maps for free. However, a model trained on a clean simulator often fails in the real world due to the "domain gap"—the difference in lighting, textures, and noise.

Researchers are now focusing on learning domain-invariant features. The goal is to force the model to ignore "style" (the color of a wall) and focus on "structure" (the edge of the wall). Hiroyasu Akada and others in Self-Supervised Learning of Domain Invariant Features for Depth Estimation achieved this by using a "Siamese" pre-training stage that encourages the network to produce the same structural representation regardless of the image's visual style. Similarly, Xiaotian Chen developed S2R-DepthNet, which trains on a mix of synthetic data and artistic paintings to ensure the model learns geometry rather than just memorizing photographic textures.

The Rise of Foundation Models and Distillation

The current frontier involves scaling these ideas to "foundation model" proportions. Instead of training on thousands of images, researchers are now using tens of millions.

The core mechanism here is knowledge distillation. In Depth Anywhere, Ning-Hsu Wang and colleagues use a powerful "teacher" model (trained on labeled perspective data) to generate pseudo-labels for massive amounts of unlabeled panoramic data. This allows a "student" model to learn 360-degree depth without ever seeing a single real 360-degree depth label.

This trend culminated in Depth Anything, where Lihe Yang scaled training to 62 million unlabeled images. By using strong semantic priors from models like DINOv2, they created a system that can estimate depth in almost any environment—from underwater scenes to outer space—with zero-shot generalization.

Depth as a Universal Auxiliary Task

Finally, we are seeing that "understanding depth" is useful even when depth isn't the final goal. In MDETrack, Zhenyu Wei showed that teaching a standard object tracker to estimate depth as a "side quest" during training makes it much more robust. Even though the depth branch is discarded at runtime, the model's internal features become "3D-aware," allowing it to better handle occlusions and scale changes.

By moving away from the need for perfect, hand-labeled data and toward self-supervision and synthetic-to-real transfer, the field is transforming depth estimation from a niche laboratory task into a robust, universal component of the modern computer vision stack.

Papers in this version

Showing 60 of 534 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 513 papers
  2. Version 1 · 2026-10-04 · 534 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_ground_truth_data_scarcity_24be46b4,
  title = {ground-truth data scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/ground-truth-data-scarcity/rdr_24be46b482274462a0512fbe759ccc61},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_scaling_depth_estimation_with_self_super_24be46b4_v1,
  title = {Scaling Depth Estimation with Self-Supervision and Synthetic Transfer},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scaling-depth-estimation-with-self-supervision-and-synthetic-transfer/rdr_24be46b482274462a0512fbe759ccc61/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}