Lightweight Monocular Depth With a Novel Neural Architecture Search Method

Lam Huynh, Phong Nguyen, Jiří Matas, Esa Rahtu, Janne HeikkiläWACV 2022

TL;DR

This paper introduces LiDNAS, a novel neural architecture search (NAS) method specifically designed to create lightweight and accurate monocular depth estimation models. By employing an "Assisted Tabu Search" and a carefully constructed search space, LiDNAS significantly reduces search time and produces models that outperform existing lightweight solutions in accuracy while being substantially more compact.

What Problem Does This Paper Address?

Accurate depth estimation is crucial for many computer vision applications, such as robotics and augmented reality. While deep neural networks (DNNs) have achieved impressive results, they often come with a high computational cost, making them unsuitable for resource-constrained devices like mobile phones. Traditional methods for creating lightweight models, like quantization or pruning, can degrade performance and require extensive manual tuning. Existing neural architecture search (NAS) methods, while promising for automating network design, are notoriously computationally expensive, demanding thousands of GPU hours. This paper aims to bridge this gap by developing an efficient NAS approach that specifically targets the creation of lightweight yet accurate monocular depth estimation models, suitable for deployment on edge devices.

Methods

The core of this paper is the LiDNAS framework, which tackles the challenge of efficient NAS for monocular depth estimation through two main innovations:

  • A Pre-defined Backbone Search Space: Instead of searching for small, reusable "cells" that are then replicated (a common NAS approach that can limit layer diversity), LiDNAS constructs its search space around a pre-defined backbone network. This backbone is structured into multi-scale pyramid networks, each composed of encoder, decoder, refine, downsample, and upsample blocks. The search then focuses on optimizing layers within these blocks, allowing for greater flexibility and layer diversity while managing the overall search space size. The search parameters include the number of resolution scales, the number of layers per block, convolutional operations (type and kernel size), squeeze-and-excitation ratios, skip connections, and output channels.
  • Assisted Tabu Search (ATS): To accelerate the search process, LiDNAS incorporates an "Assisted Tabu Search" (ATS). This method leverages a scoring mechanism that predicts a network's performance before full training, inspired by prior work [40]. By using this score to rank candidate architectures and guide the mutation process, ATS significantly reduces the need for extensive training of every candidate. The search algorithm iteratively mutates parent models, evaluates them using the score and a multi-objective function (balancing accuracy and parameter count), and selects the best performing child model for the next generation.

Datasets

The LiDNAS framework was evaluated on three widely used datasets for monocular depth estimation:

  • NYU-Depth-v2: An indoor dataset containing RGB-D images.
  • KITTI: An outdoor driving dataset.
  • ScanNet: A dataset of richly annotated 3D reconstructions of indoor scenes.

The paper uses standard splits for training and testing on these datasets.

Key Insights and Ideas

  • Search Space Design Matters: The paper highlights that simply replicating small cells can limit architectural diversity, which is crucial for efficient depth estimation. Their approach of using a pre-defined backbone with flexible layer choices within blocks offers a better balance between search space size and architectural flexibility.
  • Predicting Performance is Key to Efficiency: The core idea of using a pre-training performance score to guide NAS is a significant breakthrough. This drastically cuts down the computational cost associated with evaluating candidate architectures, making NAS feasible for complex tasks like depth estimation.
  • Multi-Objective Optimization is Crucial: The paper explicitly balances accuracy and model size (compactness) using a weighted objective function. This is essential for creating models that are not only performant but also deployable on resource-constrained devices.
  • Trade-offs are Inevitable: The exploration of different a values (balancing accuracy and compactness) in Figure 6 and 8 clearly demonstrates the inherent trade-off between model accuracy and size. The optimal choice of a depends on the specific application requirements.

Key Results

LiDNAS-optimized models achieved state-of-the-art results on the NYU-Depth-v2, KITTI, and ScanNet datasets, outperforming existing lightweight baselines.

  • NYU-Depth-v2: LiDNAS-N achieved superior accuracy (REL, RMSE, θ1) compared to lightweight methods like FastDepth and EDA, while being 55% more compact than the best performing NAS-based approach [48]. The search time was also reduced by 90% compared to [48].
  • KITTI: LiDNAS-K yielded competitive results with baselines while being the smallest model, showing noticeable improvements over PyD-Net.
  • ScanNet: LiDNAS-S achieved comparable depth maps using only 20% of the parameters of the best performing baseline.
  • Runtime: On a Pixel 3a phone, LiDNAS models demonstrated competitive runtime performance compared to other lightweight methods, suggesting potential for real-time applications.

Limitations and Future Work

  • Search Time: While significantly reduced, the NAS process still requires approximately 4.3 GPU days to find optimal architectures.
  • Challenging Scenes: The paper acknowledges that even their best models struggle with certain challenging scenes, such as those with Lambertian surfaces.
  • Future Work: The authors suggest that further optimization of the search process and exploration of more complex backbone structures could lead to even better results. The implementation of LiDNAS will be made publicly available, encouraging further research in this area.

Summary updated

BibTeX
@inproceedings{huynh2022lightweight,
  author = {Lam Huynh and Phong Nguyen and Jiří Matas and Esa Rahtu and Janne Heikkilä},
  title = {{Lightweight Monocular Depth With a Novel Neural Architecture Search Method}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure