Rethinking Pareto Frontier for Performance Evaluation of Deep Neural Networks

Vahid Partovi Nia, Alireza Ghaffari, Mahdi Zolnouri, Yvon SavariaarXiv 2022 · arXiv

TL;DR

This paper introduces a more realistic way to benchmark deep learning models by using a stochastic, multi-dimensional Pareto frontier. By accounting for training costs, hardware-specific latencies, and the inherent randomness of performance metrics, the authors provide a framework to identify models that are truly "efficient" across diverse real-world constraints.

What Problem Does This Paper Address?

In the race to build better AI, we usually rank models based on a simple trade-off: accuracy versus speed (or FLOPs). However, this approach has two major flaws:

  1. It ignores the "bill": The cost and resources required to train a model are often overlooked, even though they are critical for organizations with limited budgets.
  2. Performance is noisy: Accuracy isn't a fixed number; it varies depending on the random seed used during training (Stochastic Gradient Descent). Similarly, inference latency fluctuates based on hardware interrupts and background tasks.

Standard "Pareto frontiers"—the set of optimal solutions where you can't improve one metric without hurting another—treat these values as deterministic points. This paper argues that because performance is stochastic, we need a "shaded band" rather than a sharp line to evaluate models fairly.

Methods

The authors propose a framework that moves beyond 2D plots into multi-dimensional space. Their approach includes:

  • Relative Efficiency Evaluation: They use a linear optimization framework to calculate a model's efficiency relative to its peers. Instead of just looking at one input (like latency), they can look at a vector of inputs (e.g., training cost + cloud latency + edge latency) against an output (accuracy).
  • Stochastic Pareto Frontier: To handle noise, they introduce a Parametric Bootstrap method. They assume performance metrics follow a distribution (like a Gaussian), sample from these distributions using Monte Carlo methods, and then calculate the distribution of efficiency scores.
  • Stochastic Dominance: This allows researchers to rank models even when their performance ranges overlap. A model "stochastically dominates" another if it is statistically likely to be more efficient across various scenarios.
Figure 1. (a) A two-dimensional Pareto frontier with random varia- tions of a point, leading to a random Pareto frontier highlighted in green. (b) A three-dimensional Pareto variant obtained by adding another input to (a).

Datasets

The study evaluates a wide range of popular image classification architectures (including the ResNet family, MobileNets, ShuffleNets, and InceptionV3) using the ImageNet1K dataset. Performance was measured across different hardware targets:

  • Cloud: NVIDIA V100 GPU.
  • Edge: ARM Kirin 970 CPU.

Key Insights and Ideas

  • Training Cost is a Game Changer: When training cost is included as a metric, the "best" models change. For instance, while MobileNetV3 is great for inference, ShuffleNetV2 becomes a top contender when you factor in how much it costs to get the model ready for deployment.
  • The "Shaded" Frontier: Traditional benchmarks provide a false sense of certainty. By visualizing the Pareto frontier as a stochastic band, the authors show that many models are statistically indistinguishable in certain hardware setups.
  • Hardware Specificity: A model that sits on the Pareto frontier for a cloud GPU might be far from optimal for an edge device. The multi-dimensional approach allows for a "universal" efficiency score that considers multiple hardware targets simultaneously.
Figure 2. (a) Univariate Pareto frontier for top-1 accuracy versus CPU latency in milli-seconds, (b) Univariate Pareto frontier for top-1 accuracy versus training cost in dollars.

Key Results

  • Top Performers: MobileNetV3 and SqueezeNetV1.1 were identified as highly efficient when focusing on inference. However, MobileNetV1's efficiency jumped significantly when training cost was added to the mix, proving it is "easier to train" than its successors.
  • Efficiency Distributions: The authors used boxplots to show the stability of model efficiency. Smaller networks like ShuffleNetV2 and MobileNetV3 showed very narrow whiskers near 100% efficiency, meaning they are consistently dominant.
  • The "Losers": Large models like DualPathNet92 exhibited the worst relative efficiency, often falling below a 20% efficiency threshold when compared to optimized tiny models.

Limitations and Future Work

  • Gaussian Assumption: The current stochastic model assumes a Gaussian (normal) distribution for noise. The authors acknowledge that other distributions might better represent hardware latency spikes or accuracy variations.
  • Scope of Tasks: The evaluation was limited to image classification. The authors suggest that future work should apply this multi-dimensional ranking to other domains like Natural Language Processing (NLP) or object detection.
  • Metric Selection: While they included training cost and latency, other factors like power consumption (Watts) or memory footprint could be integrated into the same linear optimization framework for an even more holistic view.

Summary updated

Abstract

Performance optimization of deep learning models is conducted either manually or through automatic architecture search, or a combination of both. On the other hand, their performance strongly depends on the target hardware and how successfully the models were trained. We propose to use a multi-dimensional Pareto frontier to re-define the efficiency measure of candidate deep learning models, where several variables such as training cost, inference latency, and accuracy play a relative role in defining a dominant model. Furthermore, a random version of the multi-dimensional Pareto frontier is introduced to mitigate the uncertainty of accuracy, latency, and throughput of deep learning models in different experimental setups. These two complementary methods can be combined to perform objective benchmarking of deep learning models. Our proposed method is applied to a wide range of deep image classification models trained on ImageNet data. Our method combines competing variables with stochastic nature in a single relative efficiency measure. This allows ranking deep learning models that run efficiently on different hardware, and combining inference efficiency with training efficiency objectively.
BibTeX
@misc{nia2022rethinking,
  author = {Vahid Partovi Nia and Alireza Ghaffari and Mahdi Zolnouri and Yvon Savaria},
  title = {{Rethinking Pareto Frontier for Performance Evaluation of Deep Neural Networks}},
  year = {2022},
  eprint = {2202.09275},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.09275},
}
Figure