The fundamental challenge in depth estimation is not just predicting distance, but doing so reliably when the world changes. A model trained on the sun-drenched highways of Germany (KITTI) often fails catastrophically when placed in a rainy London street or a cluttered indoor office. This "domain gap" exists because traditional models overfit to the specific pixel distributions, camera geometries, and depth ranges of their training sets.
To bridge this gap, researchers are moving away from narrow supervision toward generalizable depth perception. This direction focuses on three pillars: leveraging the "visual common sense" of large-scale foundation models, adapting to new scenes at test-time, and enforcing universal geometric constraints that transcend specific datasets.
The Visual Common Sense of Foundation Models
Recent breakthroughs have shown that models trained on billions of images for synthesis or classification already "understand" geometry. Instead of training a depth network from scratch, researchers like Bingxin Ke and Anton Obukhov in Marigold repurpose the internal representations of generative diffusion models. Because these models must understand object shapes and spatial relationships to generate realistic images, they provide a rich, zero-shot prior for depth.
Similarly, MiDaS v3.1 by Reiner Birkl and colleagues demonstrates that scaling the backbone to massive Vision Transformers (ViTs) like BEiT allows the model to absorb diverse environmental features, leading to a 28% improvement in robustness. The intuition is simple: the more "visual concepts" a model knows, the less likely it is to be surprised by a novel environment.
Bridging the Metric Gap with Test-Time Alignment
While foundation models excel at relative depth (knowing that a chair is behind a table), they often struggle with metric depth (knowing the chair is exactly 2.5 meters away). This is due to "projection ambiguity"—the fact that a 2D image can represent many 3D scales.
To solve this without retraining, Hyoseok Lee and Tae-Hyun Oh propose test-time alignment. They take an affine-invariant (relative) depth prior and "anchor" it to a few sparse metric measurements (like those from a LiDAR or a phone's TOF sensor) during inference. This allows the model to maintain the high-frequency structural detail of the foundation model while snapping the entire scene into the correct physical scale.
Aoran Xiao and Haotian Wang take this further in PacGDC by using foundation models as "scale manipulators." By synthesizing diverse pseudo-geometries, they force the model to learn consistency across different scales, achieving state-of-the-art generalization with less than 1% of the typical training data.
Learning Geometry over Pixels
A recurring theme in robust depth is the shift from learning 2D image features to learning 3D geometric principles. Traditional CNNs often rely on "vertical cues" (objects lower in the frame are closer), which fail on hills or in indoor scenes.
To counter this, Vitor Campagnolo Guizilini and Jiading Fang introduced DeFiNe, which treats geometry as an input feature rather than a hard-coded constraint. By using 3D data augmentations and multi-view consistency, the model learns how points move in space, a principle that remains true whether the camera is in a car or a drone.
For depth completion—filling in the gaps of a sparse LiDAR map—Jia Deng and Yiming Zuo developed OGNI-DC. Instead of a standard black-box neural network, they use an optimization-guided approach. The network predicts a gradient field, which is then integrated through a differentiable optimization layer. This hybrid approach combines the pattern recognition of deep learning with the mathematical stability of classical geometry, ensuring the model doesn't "hallucinate" impossible structures when it encounters a new sensor configuration.
On-the-Fly Adaptation for Robotics
In robotics, a model cannot always wait for a perfect "foundation" prior. Alessio Tonioni and Matteo Poggi pioneered Real-Time Self-Adaptive Deep Stereo. Their framework, MADNet, allows a stereo system to perform unsupervised fine-tuning while it moves. By using the consistency between left and right camera views as a supervisory signal, the model continuously corrects its own errors, effectively "learning" the specific characteristics of a new environment (like lighting or texture) in seconds.
This move toward online adaptation and zero-shot generalization marks a fundamental shift in the field: we are no longer building models that know one environment perfectly, but models that know how to understand any environment they encounter.