ML-EXray: Visibility into ML Deployment on the Edge

Hang Qiu, Ioanna Vavelidou, Jian Li, Evgenya Pergament, Pete Warden, Sandeep Chinchali, Zain Asgar, Sachin Katti

Abstract

Benefiting from expanding cloud infrastructure, deep neural networks (DNNs) today have increasingly high performance when trained in the cloud. Researchers spend months of effort competing for an extra few percentage points of model accuracy. However, when these models are actually deployed on edge devices in practice, very often, the performance can abruptly drop over 10% without obvious reasons. The key challenge is that there is not much visibility into ML inference execution on edge devices, and very little awareness of potential issues during the edge deployment process. We present ML-EXray, an end-to-end framework, which provides visibility into layer-level details of the ML execution, and helps developers analyze and debug cloud-to-edge deployment issues. Evaluations show that ML-EXray can effectively catch deployment issues, such as pre-processing bugs, quantization issues, suboptimal kernels, etc. Using ML-EXray, users need to write less than 15 lines of code to fully examine the edge deployment pipeline. Eradicating these issues, ML-EXray can correct model performance by up to 30%, pinpoint error-prone layers, and guide users to optimize kernel execution latency by two orders of magnitude.

The Visibility Gap in Edge AI

There is a significant "visibility gap" in edge AI. A model might achieve state-of-the-art accuracy in a cloud training environment, but once deployed on a smartphone or IoT device, its performance often plummets—sometimes by more than 10%—without throwing a single error. This phenomenon, termed "silent failure," occurs because the model continues to process data and produce outputs, but with significantly reduced confidence and accuracy due to underlying mismatches in the deployment pipeline.

The fundamental issue lies in a disconnect between the researchers who train models and the engineers who deploy them. While training focuses on maximizing accuracy, deployment prioritizes efficiency and hardware compatibility. This leads to subtle, undocumented mismatches in the "data plumbing":

  • Preprocessing Bugs: Differences in image resizing algorithms (e.g., bilinear vs. area-averaging), color channel ordering (RGB vs. BGR), or normalization ranges (e.g., [-1, 1] vs. [0, 1]).
  • Quantization Issues: Converting 32-bit floats to 8-bit integers for edge efficiency can introduce numerical drift or functional bugs in specific hardware kernels.
  • Hardware Heterogeneity: Kernels optimized for cloud GPUs may perform poorly or exhibit buggy behavior on specialized mobile DSPs or NPUs.

Currently, debugging these issues is a manual, laborious process involving logging individual tensors and comparing them by hand, which is often impractical for complex models.

ML-EXray: An End-to-End Debugging Framework

ML-EXray introduces a systematic three-pillar approach to bridge this visibility gap and automate the validation process.

ML-EXray system architecture overview, illustrating the instrumentation, reference pipeline, and debugging report components.
  1. Instrumentation & Logging APIs: ML-EXray provides a cross-platform library (supporting C++, Java, and Python) that allows developers to instrument their edge applications with minimal effort. It logs critical "telemetry data," including layer-level inputs and outputs, performance metrics like latency and memory usage, and peripheral sensor data such as device orientation or lighting conditions.
  2. Reference Pipeline: To establish a baseline, ML-EXray runs a "gold standard" version of the model in a controlled cloud or workstation environment. By using the same input data as the edge device, it creates a reliable point of comparison.
  3. Deployment Validation Framework: This core component analyzes the logs from both the edge and reference pipelines. It performs:
  4. Accuracy Validation: A high-level check to see if the final model output on the edge matches the reference.
  5. Per-layer Validation: If a discrepancy is found, the framework uses normalized Root Mean Square Error (rMSE) to pinpoint exactly where the numerical drift begins. A sudden "jump" in rMSE after a specific layer or operation immediately flags the source of the error.
  6. Root-Cause Analysis: The framework employs built-in or user-defined "assertions" to test hypotheses. For example, it might check if applying a 90-degree rotation or swapping BGR to RGB channels fixes the accuracy drop, thereby identifying the specific bug.

Experimental Evaluation

The authors evaluated ML-EXray across a diverse set of tasks and popular models to demonstrate its broad applicability:

  • Image Classification: Using the ImageNet dataset with models like MobileNet (v1, v2, v3), ResNet50, Inception v3, and DenseNet 121.
  • Object Detection: Using the COCO dataset with SSD-MobileNet and FasterRCNN.
  • Image Segmentation: Evaluated with DeepLab v3.
  • Speech Recognition: Using the Google Speech Commands dataset with Plain CNN and Yamnet.
  • Text Classification: Using IMDB movie reviews with MobileBert and NNLM embeddings.

Key Results and Insights

ML-EXray proved highly effective in identifying and resolving deployment issues that would otherwise remain hidden.

Impact of various deployment issues on image classification accuracy across different models.
  • Accuracy Recovery: By pinpointing and fixing preprocessing bugs—such as incorrect resizing methods or orientation mismatches—ML-EXray helped correct model performance by up to 30%.
  • Bug Discovery: The tool successfully identified previously unknown bugs in official TensorFlow Lite quantization kernels, specifically within depthwise convolutions.
  • Minimal Developer Effort: The framework significantly reduces the complexity of debugging. The code required for instrumentation and assertions is typically fewer than 15 lines, representing an 80-95% reduction in lines of code compared to manual logging.
  • Production-Ready Performance: Instrumentation adds negligible overhead. On a mobile GPU, it introduces only about 3ms of latency per frame with a tiny memory footprint (~4MB), making it suitable for real-world testing environments.
  • Kernel Optimization: Beyond accuracy, ML-EXray's performance metrics revealed that unoptimized kernels could be two orders of magnitude slower than their optimized counterparts, providing a clear path for latency improvements.

Conclusion and Future Work

ML-EXray provides a much-needed "X-ray" into the black box of edge ML deployment. By automating the comparison between edge execution and a reference baseline, it empowers developers to resolve complex deployment bugs with minimal effort. Future work includes expanding the framework to provide automated "auto-fixes" based on assertion results and improving debugging for dynamic, transient environmental conditions. The authors plan to release ML-EXray as an open-source library to support the broader AI deployment community.

Summary updated

Abstract

Benefiting from expanding cloud infrastructure, deep neural networks (DNNs) today have increasingly high performance when trained in the cloud. Researchers spend months of effort competing for an extra few percentage points of model accuracy. However, when these models are actually deployed on edge devices in practice, very often, the performance can abruptly drop over 10% without obvious reasons. The key challenge is that there is not much visibility into ML inference execution on edge devices, and very little awareness of potential issues during the edge deployment process. We present ML-EXray, an end-to-end framework, which provides visibility into layer-level details of the ML execution, and helps developers analyze and debug cloud-to-edge deployment issues. More often than not, the reason for sub-optimal edge performance does not only lie in the model itself, but every operation throughout the data flow and the deployment process. Evaluations show that ML-EXray can effectively catch deployment issues, such as pre-processing bugs, quantization issues, suboptimal kernels, etc. Using ML-EXray, users need to write less than 15 lines of code to fully examine the edge deployment pipeline. Eradicating these issues, ML-EXray can correct model performance by up to 30%, pinpoint error-prone layers, and guide users to optimize kernel execution latency by two orders of magnitude. Code and APIs will be released as an open-source multi-lingual instrumentation library and a Python deployment validation library.
BibTeX
@inproceedings{qiu2022ml,
  author = {Hang Qiu and Ioanna Vavelidou and Jian Li and Evgenya Pergament and Pete Warden and Sandeep Chinchali and Zain Asgar and Sachin Katti},
  title = {{ML-EXray: Visibility into ML Deployment on the Edge}},
  booktitle = {MLSys},
  year = {2022},
  url = {https://openreview.net/forum?id=Psk2o7y0Ojc},
}
Figure