Depth estimation—the ability to infer the 3D structure of a scene from 2D images—has long been a bottleneck for autonomous systems. Traditionally, high-fidelity depth maps required either expensive active sensors like LiDAR or computationally heavy optimization algorithms that ran at a fraction of a frame per second.
A significant shift in research has moved toward end-to-end neural architectures that prioritize inference speed and memory efficiency. By rethinking how geometric priors are integrated into neural networks, researchers have enabled high-resolution 3D perception on mobile and embedded hardware.
From Deep Bottlenecks to Shallow Efficiency
Early deep learning models for depth estimation often relied on massive, multi-branch architectures that were too slow for robotics. To solve this, researchers began exploring "shallow but dense" designs. Nantheera Anantrasirichai and colleagues demonstrated that by focusing on near-field resolution and specialized projection losses, a shallow CNN could estimate depth from stereo images up to 10 times faster than state-of-the-art models while maintaining the accuracy needed for view synthesis.
This drive for efficiency is also seen in the work of Tak-Wai Hui, whose RM-Depth framework uses recurrent feature refinement to achieve high accuracy with only 2.97 million parameters. By optimizing for architectural compactness rather than raw scale, these models outperform much larger counterparts in both speed and precision, making them viable for resource-constrained drones and robots.
Learning Geometry Without Ground Truth
A major hurdle in training depth networks is the need for "ground truth" data, which usually requires expensive LiDAR setups. To bypass this, researchers like Clément Godard and Gabriel J. Brostow introduced left-right consistency losses. By training on stereo pairs and forcing the model to predict how one view would look from the perspective of the other, the network learns depth as an internal representation of geometric consistency.
This "learn stereo, infer mono" philosophy was further refined by Matan Goldman and Tal Hassner, who used Siamese networks to exploit image symmetry. These self-supervised methods allow models to reach real-time speeds (exceeding 190 FPS in some cases, as shown by Hanhan Li and Ariel Gordon) without ever seeing a single LiDAR point during training.
Specialized Sensors and End-to-End Reconstruction
While monocular and stereo cameras are common, specialized sensors like Time-of-Flight (ToF) and Light Field (LF) cameras offer unique geometric cues that traditional pipelines struggle to process quickly. Shuochen Su and Wolfgang Heidrich pioneered TOFNET, an end-to-end framework that replaces modular denoising and phase-unwrapping steps with a single network. This approach allows for depth reconstruction at over 400 FPS.
Similarly, in light field imaging, Stefan Heber and Thomas Pock moved from 2D to 3D convolutions to process Epipolar Plane Images (EPIs). By treating the light field as a volume rather than a collection of images, they reduced processing times from hours to sub-second execution, effectively eliminating the "streaking" artifacts that plagued earlier methods.
Scaling to High Resolution and Complex Physics
The latest frontier in this field involves maintaining real-time performance at 2K resolutions and handling "impossible" materials like glass. Gene Chou and Noah Snavely introduced FlashDepth, which uses a Mamba-based temporal alignment module to achieve 24 FPS at 2K resolution. By decoupling global depth calculation from high-resolution structural refinement, they achieve sharp boundaries that traditional batch-processing models cannot match.
For environments where standard light fails, Wen Dong and colleagues developed PolarDepth, which integrates the physics of light polarization. By decoding refractive indices and surface orientations from polarization cues, the system can "see" transparent objects in real-time, a capability essential for robotic manipulation in human environments.
Through these diverse approaches—ranging from state-space models like StereoMamba to gated convolutions for depth completion by Tingxuan Huang—the field is converging on a shared goal: transforming depth estimation from a heavy computational burden into a lightweight, instantaneous utility for autonomous agents.