Abstract
Depth perception is paramount to tackle real-world problems, ranging from autonomous driving to consumer applications. For the latter, depth estimation from a single image represents the most versatile solution, since a standard camera is available on almost any handheld device. Nonetheless, two main issues limit its practical deployment: i) the low reliability when deployed in-the-wild and ii) the demanding resource requirements to achieve real-time performance, often not compatible with such devices. Therefore, in this paper, we deeply investigate these issues showing how they are both addressable adopting appropriate network design and training strategies -- also outlining how to map the resulting networks on handheld devices to achieve real-time performance. Our thorough evaluation highlights the ability of such fast networks to generalize well to new environments, a crucial feature required to tackle the extremely varied contexts faced in real applications. Indeed, to further support this evidence, we report experimental results concerning real-time depth-aware augmented reality and image blurring with smartphones in-the-wild.
Introduction
Monocular Depth Estimation (MDE)—predicting a 3D depth map from a single 2D image—is a "holy grail" for mobile computer vision. While active sensors like LiDAR or Time-of-Flight (ToF) exist, they are often expensive, power-hungry, or limited to high-end devices. Software-based MDE offers a versatile alternative, but faces two primary hurdles: generalization to diverse environments and the high computational cost of state-of-the-art models.
Methodology: Distilling Knowledge for the Wild
To overcome these challenges, we propose a "Teacher-Student" training pipeline designed for mobile deployment. Instead of training small models on limited ground-truth data, we leverage Knowledge Distillation. We use MiDaS, a large and accurate model that generalizes exceptionally well, as a teacher to generate "proxy" depth labels for a new, diverse dataset called WILD.
The WILD dataset consists of approximately 447,000 images from COCO and OpenImages, providing the visual variety necessary for "in the wild" performance. We evaluated several compact student architectures, including PyDNet, FastDepth, and DSNet, training them to mimic the teacher's output on this diverse data.
Mobile Optimization and Deployment
Achieving real-time performance requires careful hardware mapping. We outline the process of porting these models to iOS (via CoreML/Metal) and Android (via TensorFlow Lite), leveraging on-device Neural Processing Units (NPUs) and GPUs.
A key finding is the "MAC vs. FPS Gap": the number of Multiply-Accumulate (MAC) operations is not always a perfect predictor of on-device speed. For instance, PyDNet and DSNet have similar MAC counts, but PyDNet is significantly faster on an iPhone XS due to how its specific layers are optimized in hardware.
Experimental Results
Our evaluation shows that a small model's ability to generalize depends more on the diversity of its training data than its parameter count. By distilling knowledge from MiDaS into PyDNet, we created a lightweight model (1.97M parameters) that handles indoor and outdoor scenes far better than larger models trained only on specific benchmarks like KITTI.
PyDNet emerged as the optimal architecture, achieving nearly 60 FPS on an iPhone XS at a resolution of 640x384. Furthermore, we demonstrate that by anchoring the dense monocular map to sparse points from standard AR frameworks (like ARCore), we can recover absolute metric depth, enabling applications like AR occlusion and synthetic bokeh.
Conclusion
This work demonstrates that high-quality, real-time monocular depth estimation is achievable on standard smartphones through effective knowledge distillation and hardware-aware optimization. While student accuracy is currently capped by the teacher's performance, the ability to generalize to unseen environments opens new possibilities for mobile augmented reality and photography.
Summary updated