TL;DR
This paper introduces LETI, an open-source tool for estimating neural network inference latency on mobile GPUs. It addresses the inadequacy of FLOPs as a proxy for latency on these devices by creating a comprehensive dataset and exploring machine learning models for accurate prediction, paving the way for more efficient mobile AI.
What Problem Does This Paper Address?
Deep learning models are increasingly deployed on mobile devices, where both accuracy and inference speed are critical. While FLOPs (floating-point operations) are often used as a proxy for computational cost, they don't accurately reflect the actual inference latency on mobile hardware, especially on GPUs. This discrepancy is particularly pronounced on mobile GPUs due to their unique architectures and memory access patterns, making it difficult to optimize neural network architectures for real-time performance. Existing methods for latency estimation, like lookup tables, perform poorly on mobile GPUs, necessitating a new approach.
Methods
The authors developed LETI (Latency Estimation Tool), a Python package designed to automate the generation and evaluation of neural network models for latency estimation. The core of LETI involves:
- Parametrization: Representing neural networks using a NAS-Bench-like format, consisting of an adjacency matrix, a list of layers, and a configuration dictionary. This allows for the systematic generation of diverse network architectures.
- Model Generation: Generating TensorFlow Keras models from these parametrizations.
- TFLite Conversion: Converting the Keras models into TensorFlow Lite format, which is optimized for mobile deployment.
- Latency Evaluation: Running the TFLite models on target devices (both desktop GPUs and mobile devices with specific GPUs like Huawei Kirin970's Mali-G72 MP12) and collecting latency measurements. This process is automated and includes robust statistical measures to ensure reliable data.
- Dataset Creation: Building a comprehensive dataset of network architectures, their FLOPs, and measured latencies on various hardware.
- Latency Modeling: Investigating different approaches for predicting latency, including:
- Lookup Tables (LUTs): Evaluating the effectiveness of traditional LUT-based methods, which proved to be inaccurate for mobile GPUs.
- FLOPs-based Regression: Simple linear regression using FLOPs as input.
- Clustering + FLOPs Regression ("Better FLOPs"): A novel approach that clusters models based on peak memory usage to separate "fast" and "slow" domains, then applies FLOPs-based regression within each cluster.
- XGBoost Regression: A more general machine learning approach using FLOPs, peak memory, and flattened adjacency matrices as input features.
Datasets
The paper utilizes and contributes to the following datasets:
- NAS-Bench-101: The authors leverage the parametrization format from NAS-Bench-101 to generate a large number of neural network architectures. They generate a subset of 10k cells from this dataset for detailed analysis on mobile GPUs.
- Custom Latency Datasets: The primary contribution is the creation of novel latency datasets generated by LETI. These datasets contain measured inference latencies for various neural network architectures on:
- Desktop GPU: Tesla V100 (for TF.Keras models).
- Mobile GPU: Huawei Kirin970 (Mali-G72 MP12), Huawei Kirin 980 (Mali-G76), and Samsung Exynos 9810 (for TFLite models).
Key Insights and Ideas
- FLOPs are a poor proxy for mobile GPU latency: The most striking finding is that FLOPs alone do not correlate well with inference latency on mobile GPUs. Even networks with similar FLOP counts can exhibit vastly different latencies, highlighting the importance of hardware-specific profiling.
- Mobile GPU latency is multi-modal: The latency distributions on mobile GPUs often show distinct clusters (e.g., "fast" and "slow" domains), suggesting that simple regression models based solely on FLOPs are insufficient.
- Peak memory is a crucial factor for mobile GPU latency: The "Better FLOPs" approach demonstrates that peak memory usage is a strong indicator for separating latency clusters on mobile GPUs. This suggests that memory access patterns and management play a significant role.
- LUTs fail on mobile GPUs: Traditional lookup-table methods, effective for CPUs, yield poor precision on mobile GPUs, underscoring the need for more sophisticated modeling techniques.
- LETI as a versatile tool: The developed LETI tool is presented as a valuable asset for researchers in neural architecture search (NAS) and for anyone needing to evaluate and model latency on mobile hardware.
Key Results
- LUTs on CPU: The lookup-table method achieved good accuracy for latency prediction on mobile CPUs, with relative errors around 5-12% for popular models like ResNet50.
- LUTs on GPU: The same LUT method resulted in "extremely low precision" for mobile GPU latency prediction.
- "Better FLOPs" approach: By clustering based on peak memory and then applying FLOPs-based linear regression, this method achieved "nice accuracy" using only 5% of the data for training, with high accuracy scores (e.g., 92.5% at ±10% tolerance).
- XGBoost Regression: This general machine learning model, using FLOPs, peak memory, and network structure as input, also showed promising results, achieving 92.4% accuracy at ±10% tolerance with a 5% training split.
- Dataset Generation: The LETI tool successfully generated datasets of hundreds of thousands of TF.Keras models and tens of thousands of TFLite models for evaluation.
Limitations and Future Work
- Limited GPU Operations: The paper notes that not all operations are supported for full GPU delegation in TFLite, requiring workarounds (e.g., substituting SLICE operations). This limits the scope of models that can be fully evaluated on the GPU.
- Specific Hardware: The experiments were conducted on specific mobile GPUs (Mali-G72 MP12, Mali-G76) and CPUs. Generalizing these findings to a wider range of hardware would require further evaluation.
- Future Work: The authors suggest that LETI can be used for more extensive latency evaluation and that the generated datasets can serve as a foundation for developing more advanced latency prediction models. They also imply that the tool can be integrated with optimization frameworks for direct latency optimization.
Summary updated