Real-Time Single Image Depth Perception in the Wild with Handheld Devices

Filippo Aleotti, Giulio Zaccaroni, Luca Bartolomei, Matteo Poggi, Fabio Tosi, Stefano Mattoccia

TL;DR

This paper presents a framework for achieving high-quality, real-time monocular depth estimation on standard smartphones. By using knowledge distillation—training lightweight "student" networks to mimic a massive, state-of-the-art "teacher" model—the authors enable handheld devices to perceive depth "in the wild" at up to 60 FPS, powering applications like augmented reality and synthetic bokeh.

What Problem Does This Paper Address?

Monocular Depth Estimation (MDE)—predicting a 3D depth map from a single 2D image—is a "holy grail" for mobile computer vision. While active sensors like LiDAR or Time-of-Flight (ToF) exist, they are often expensive, power-hungry, or limited to high-end devices.

The authors identify two primary hurdles for software-based MDE on mobile:

  1. Generalization: Models trained on specific datasets (like the driving-focused KITTI) fail miserably when taken "into the wild" (e.g., indoors or in a park).
  2. Efficiency: State-of-the-art models are computationally massive, often requiring high-end desktop GPUs to run, making them incompatible with the thermal and power constraints of a smartphone.

Methods

The authors propose a "Teacher-Student" training pipeline designed for deployment:

  • Knowledge Distillation: Instead of training a small model on limited ground-truth data, they use MiDaS (a very large, accurate model) as a teacher. MiDaS is too slow for mobile but generalizes incredibly well.
  • The WILD Dataset: They curated a new dataset (WILD) consisting of ~447,000 diverse images from COCO and OpenImages. They used MiDaS to generate "proxy" depth labels for these images.
  • Lightweight Students: They evaluated several compact architectures: PyDNet, FastDepth, and DSNet. These models were trained to predict the proxy labels generated by MiDaS.
  • Mobile Optimization: The paper outlines the process of porting these models to iOS (via CoreML/Metal) and Android (via TensorFlow Lite), leveraging on-device Neural Processing Units (NPUs) and GPUs for maximum throughput.

Datasets

  • KITTI: The standard outdoor driving benchmark used for initial performance assessment.
  • WILD (New): A collection of 447k frames from Microsoft COCO and OpenImages. It lacks ground-truth depth but provides the visual diversity needed for "in the wild" generalization.
  • TUM RGBD & NYUv2: Used as "zero-shot" evaluation sets to test how well models trained on WILD generalize to indoor environments they haven't seen before.

Key Insights and Ideas

  • Data Over Architecture: The most striking insight is that a small model's ability to generalize depends more on the diversity of its training data than its parameter count. By distilling knowledge from MiDaS into PyDNet, the authors created a tiny model that handles indoor and outdoor scenes far better than a larger model trained only on KITTI.
  • The MAC vs. FPS Gap: The authors found that the number of Multiply-Accumulate (MAC) operations isn't always a perfect predictor of speed. For example, PyDNet and DSNet have similar MAC counts, but PyDNet is significantly faster on an iPhone XS due to how its specific layers are optimized in hardware.
  • Scale Recovery: Monocular systems typically suffer from "scale ambiguity" (they don't know if a car is small and close or large and far). The authors show that by anchoring the dense monocular map to a few sparse points provided by standard AR frameworks (like ARCore), they can achieve absolute, metric depth.

Key Results

  • Speed: PyDNet emerged as the winner, achieving nearly 60 FPS on an iPhone XS at a resolution of 640x384.
  • Efficiency: PyDNet uses only 1.97M parameters, compared to the 105M+ parameters of the MiDaS teacher, yet retains enough accuracy for consumer applications.
  • Generalization: Models trained using the WILD distillation strategy significantly outperformed those trained with standard self-supervised methods when tested on unseen indoor datasets (TUM/NYUv2).
  • Applications: The authors demonstrated real-time Bokeh effects (blurring the background) and AR occlusion (allowing a virtual object to go behind a real-world chair), both running smoothly on-device.

Limitations and Future Work

  • Teacher Ceiling: The student's accuracy is fundamentally capped by the teacher's performance. Any systematic errors in MiDaS are inherited by the mobile model.
  • Scale Ambiguity: Without external anchors (like ARCore points), the model only provides relative depth.
  • Challenging Surfaces: The paper notes failure cases on highly reflective surfaces (mirrors) or flat surfaces with confusing textures (optical illusions), which remain a challenge for all monocular methods.
  • Future Directions: The authors suggest exploring weight quantization (reducing precision to 8-bit) to further boost speed, though they avoided it here to maintain maximum accuracy for their benchmarks.

Summary updated

Abstract

Depth perception is paramount for tackling real-world problems, ranging from autonomous driving to consumer applications. For the latter, depth estimation from a single image would represent the most versatile solution since a standard camera is available on almost any handheld device. Nonetheless, two main issues limit the practical deployment of monocular depth estimation methods on such devices: (i) the low reliability when deployed in the wild and (ii) the resources needed to achieve real-time performance, often not compatible with low-power embedded systems. Therefore, in this paper, we deeply investigate all these issues, showing how they are both addressable by adopting appropriate network design and training strategies. Moreover, we also outline how to map the resulting networks on handheld devices to achieve real-time performance. Our thorough evaluation highlights the ability of such fast networks to generalize well to new environments, a crucial feature required to tackle the extremely varied contexts faced in real applications. Indeed, to further support this evidence, we report experimental results concerning real-time, depth-aware augmented reality and image blurring with smartphones in the wild.
BibTeX
@inproceedings{aleotti2021real,
  author = {Filippo Aleotti and Giulio Zaccaroni and Luca Bartolomei and Matteo Poggi and Fabio Tosi and Stefano Mattoccia},
  title = {{Real-Time Single Image Depth Perception in the Wild with Handheld Devices}},
  booktitle = {Sensors},
  year = {2021},
  url = {https://openreview.net/forum?id=eJkpVlFhy1v},
}

Loading related content...

Figure