Real-Time Depth Inference for Autonomous Systems

31 concepts32 papers

Spatial perception—the ability to estimate the distance of objects in a scene—is the foundation of autonomous navigation, robotics, and augmented reality. While high-fidelity depth estimation was once the domain of heavy, offline computations or expensive LiDAR sensors, a new generation of research focuses on real-time depth inference. The goal is to produce dense, accurate depth maps at frame rates exceeding 30 FPS, often using nothing more than a single moving camera or low-cost sensors.

Moving Beyond Computational Bottlenecks

Traditional high-accuracy depth models often rely on 3D convolutional neural networks (CNNs) to regularize cost volumes. While effective, these 3D convolutions are memory-intensive and slow. To achieve real-time performance, Liu et al. demonstrate that replacing these heavy 3D layers with a combination of 2D context networks and temporal transformers can yield a five-fold increase in processing speed without sacrificing accuracy.

Similarly, the transition from multi-step optimization to end-to-end learning has revolutionized specialized sensors. In Time-of-Flight (ToF) imaging, Su et al. replaced modular pipelines with a single CNN (TOFNET), enabling the simultaneous correction of interference and noise at speeds exceeding 400 FPS. This shift suggests that the key to real-time performance is not just faster hardware, but the consolidation of complex geometric reasoning into single-pass neural architectures.

Self-Supervision as a Path to Efficiency

A major hurdle in depth estimation is the need for ground-truth data, which usually requires expensive LiDAR. Researchers have bypassed this by using self-supervised learning, where models learn depth by enforcing geometric consistency between different views.

Godard et al. pioneered this by using "left-right consistency" in stereo pairs during training, allowing the model to predict depth from a single image in just 35 milliseconds at test time. Building on this, Watson and Mac Aodha introduced ManyDepth, which "opportunistically" uses multiple video frames to build a cost volume. By integrating this geometric information into a single forward pass, they achieved results 100 times faster than traditional test-time optimization methods.

Architectural Distillation and Compression

To bring these capabilities to resource-constrained devices like UAVs or mobile phones, the research has moved toward extreme architectural efficiency. Hui showed with RM-Depth that recurrent feature refinement can achieve state-of-the-art accuracy with only 2.97 million parameters—up to 44 times smaller than competing models.

Another powerful strategy is knowledge distillation. Wu et al. proposed DistDepth, which transfers the structural knowledge of large, slow transformer-based models into lightweight convolutional architectures. This allows the efficient CNN to inherit the "global" understanding of the transformer while maintaining the inference speed required for interactive applications like augmented reality.

Handling Dynamic and Sparse Data

Real-world environments are rarely static, and sensors are often noisy. In dynamic scenes, Li et al. and the authors of RM-Depth have developed motion-field estimation strategies that allow models to distinguish between camera movement and moving objects, maintaining real-time speeds (over 190 FPS) even in crowded environments.

When dealing with sparse data, such as mmWave radar, the challenge is densification. Singh et al. demonstrated that fusing a single RGB frame with sparse radar data can achieve high-accuracy depth without the latency of accumulating multiple frames. For LiDAR-based systems, Tang et al. introduced BP-Net, which uses a multi-scale bilateral propagation module to densify sparse points early in the network, ensuring that the refinement stages have high-quality data to work with while remaining within the strict latency limits of autonomous vehicles.

Key Foundational Work

Unsupervised Monocular Depth Estimation with Left-Right Consistency (2017) — contributed 1 concept elements to this direction.
Deep End-to-End Time-of-Flight Imaging (2018) — contributed 1 concept elements to this direction.
Unsupervised Adversarial Depth Estimation Using Cycled Generative Networks (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_real_time_depth_inference_22ffa64b,
  title = {real-time depth inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/real-time-depth-inference/rdr_22ffa64b29e74056953bb66d6566c6f5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_real_time_depth_inference_22ffa64b_v1,
  title = {real-time depth inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/real-time-depth-inference/rdr_22ffa64b29e74056953bb66d6566c6f5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}

See something wrong? Report a mistake