Latency Estimation Tool and Investigation of Neural Networks Inference on Mobile GPU

Evgeny Ponomarev, Sergey A. Matveev, Ivan V. Oseledets, Valery GlukhovComput 2021 · DOI

TL;DR

This paper introduces LETI, a tool for generating and evaluating neural network inference latency on mobile GPUs. It addresses the critical need for accurate latency prediction beyond FLOPs, demonstrating that existing methods like lookup tables perform poorly on GPUs, while more general machine learning approaches show promise.

What Problem Does This Paper Address?

Deep learning models are increasingly deployed on edge devices like mobile phones. While accuracy is crucial, inference speed (latency) is equally important for real-time applications. Traditionally, researchers optimize neural network architectures based on FLOPs (floating-point operations) as a proxy for latency. However, this proxy is often inaccurate, especially on mobile GPUs, where factors like memory access, hardware specifics, and implementation details significantly impact actual inference time. Existing methods for latency prediction, such as lookup tables (LUTs) that sum layer latencies, work reasonably well for CPUs but fail to provide sufficient precision for GPUs. This gap hinders the efficient design of neural network architectures tailored for mobile GPU inference.

Methods

The core contribution is the LETI (Latency Estimation Tool and Investigation) pipeline. LETI automates the process of:

  • Generating TF.Keras models: It takes a graph-based parametrization (similar to NAS-Bench-101) and generates corresponding TensorFlow Keras models.
  • Converting to TFLite: These models are then converted to TensorFlow Lite format, the standard for mobile deployment.
  • Benchmarking on Devices: LETI facilitates the evaluation of these TFLite models on actual mobile devices, specifically targeting the GPU.

The paper also explores various latency prediction methods:

  • Lookup Table (LUT): The authors implement and evaluate a LUT approach, where the latency of individual layers is measured and summed. This is a common method for CPU latency prediction.
  • Linear Models (RANSAC): They use RANSAC (Random Sample Consensus) regression to model latency based on FLOPs, acknowledging the presence of outliers in the data. They also explore a clustered RANSAC approach based on memory thresholds.
  • Gradient Boosting Methods: XGBoost, CatBoost, and LightGBM are employed to predict latency using a combined feature set including FLOPs, peak memory, and flattened network structure.
  • Graph Convolutional Networks (GCN): Inspired by prior work, they implement a GCN model to predict latency directly from the graph structure of the neural network.

Datasets

The paper utilizes a subset of the NAS-Bench-101 dataset for generating parametrized architectures. They then create a new dataset by evaluating these architectures on mobile GPUs. This dataset consists of 11,055 samples for fully GPU-delegable networks, split into a training set (1000 samples) and a testing set (10,055 samples). The evaluation is performed on two specific mobile devices: Huawei Kirin 970 and Kirin 980.

Key Insights and Ideas

  • LUTs are insufficient for mobile GPUs: The paper strongly demonstrates that the widely used lookup table method, which sums individual layer latencies, yields poor precision for mobile GPU inference. This is a significant finding, as it challenges a common assumption in latency prediction.
  • GPU latency is complex and hardware-specific: The non-linear relationship between FLOPs and latency observed on the Kirin 970, compared to the more linear relationship on the Kirin 980, highlights the impact of GPU architecture (e.g., cache sizes) and implementation details on inference time.
  • General ML models show promise for GPU latency: While LUTs fail, more general machine learning models like gradient boosting (XGBoost, CatBoost, LightGBM) and GCNs, when trained on experimentally derived data, can achieve reasonable latency predictions for mobile GPUs.
  • Data movement and implementation matter: The authors acknowledge that factors beyond FLOPs, such as data movement and software implementation, significantly influence latency, and these are not well-captured by simpler models.

Key Results

  • The LUT method showed high error for layer-wise prediction of neural network latency on a mobile GPU.
  • For CPU latency prediction, the LUT method provided good accuracy (e.g., relative errors around 5-12% for popular models).
  • Gradient boosting methods (XGBoost, CatBoost, LightGBM) achieved accuracy above 80% (Acc@10%) on the Kirin 980 device, with R2 scores around 0.90-0.93.
  • The RANSAC + cluster model showed improved accuracy over simple RANSAC, particularly on the Kirin 970, achieving 71.31% Acc@10%.
  • GCN performance was competitive, achieving 67.16% Acc@10% on the Kirin 970.

Limitations and Future Work

  • Limited Device Coverage: The experiments were conducted on only two specific Huawei Kirin devices. The authors acknowledge that this may not be representative of all mobile GPUs and suggest that a broader range of devices should be tested.
  • GCN Performance: While GCNs show promise, their performance in this study was not as strong as some gradient boosting methods on the tested devices. The authors mention that their GCN implementation might not have been optimal or that the architecture search space might not have been perfectly suited.
  • Future Work: The authors express a desire to address issues like software implementation optimization, data layout, and processing in future research. They also aim to extend their LETI tool to cover more complex scenarios and devices. The potential for using LETI to optimize software implementation for target architectures is also highlighted.

Summary updated

BibTeX
@article{ponomarev2021latency,
  author = {Evgeny Ponomarev and Sergey A. Matveev and Ivan V. Oseledets and Valery Glukhov},
  title = {{Latency Estimation Tool and Investigation of Neural Networks Inference on Mobile GPU}},
  journal = {Comput.},
  year = {2021},
  doi = {10.3390/computers10080104},
}
Figure