ML-EXray: Visibility into ML Deployment on the Edge

Hang Qiu, Ioanna Vavelidou, Jian Li, Evgenya Pergament, Pete Warden, Sandeep Chinchali, Zain Asgar, Sachin Katti

TL;DR

ML-EXray is a lightweight, end-to-end framework designed to debug the "silent" performance drops that occur when moving machine learning models from the cloud to edge devices. By providing layer-level visibility and automated comparisons against a reference pipeline, it allows developers to fix deployment bugs with less than 15 lines of code, potentially recovering up to 30% in model accuracy.

What Problem Does This Paper Address?

There is a significant "visibility gap" in edge AI. A model might achieve state-of-the-art accuracy in a cloud training environment, but once deployed on a smartphone or IoT device, its performance often plummets—sometimes by more than 10%—without throwing a single error.

The authors identify a fundamental disconnect between the researchers who train models and the engineers who deploy them. Training focuses on accuracy, while deployment prioritizes efficiency. This leads to subtle, undocumented mismatches in the "data plumbing":

  • Preprocessing Bugs: Differences in image resizing algorithms (bilinear vs. area-averaging), color channel ordering (RGB vs. BGR), or normalization ranges ([-1, 1] vs. [0, 1]).
  • Quantization Issues: Converting 32-bit floats to 8-bit integers can introduce numerical drift or even functional bugs in specific hardware kernels.
  • Hardware Heterogeneity: A kernel that runs fast on a cloud GPU might be disastrously slow or buggy on a specific mobile DSP or NPU.

Currently, debugging these issues is a manual, "laborious and cumbersome" process of logging individual tensors and comparing them by hand.

Methods

ML-EXray introduces a systematic three-pillar approach to validation:

  1. Instrumentation & Logging APIs: A cross-platform library (C++, Java, Python) that allows developers to instrument their edge apps. It logs "telemetry data," including layer-level inputs/outputs, performance metrics (latency, memory), and peripheral sensor data (orientation, lighting).
  2. Reference Pipeline: ML-EXray runs a "gold standard" version of the model in a controlled cloud/workstation environment using the same input data to establish a baseline.
  3. Deployment Validation Framework: This is the "brain" of the system. It performs:
  4. Accuracy Validation: Checks if the final output matches the reference.
  5. Per-layer Validation: If accuracy is low, it uses normalized Root Mean Square Error (rMSE) to pinpoint exactly which layer the numerical drift begins.
  6. Root-Cause Analysis: Uses built-in or custom "assertions" (e.g., checking if a 90-degree rotation or a BGR-to-RGB swap fixes the error) to identify the specific bug.

Datasets

The authors evaluated ML-EXray across a wide variety of tasks and popular models to prove its generality:

  • Image Classification: ImageNet dataset using MobileNet (v1, v2, v3), ResNet50, Inception v3, and DenseNet 121.
  • Object Detection: COCO dataset using SSD-MobileNet and FasterRCNN.
  • Image Segmentation: DeepLab v3.
  • Speech Recognition: Google Speech Commands dataset using Plain CNN and Yamnet.
  • Text Classification: IMDB movie reviews using MobileBert and NNLM embeddings.

Key Insights and Ideas

  • The "Silent Failure" is the Norm: The paper highlights that many deployment issues don't cause crashes; they cause "washed-out" data or slightly off-center crops that the model processes without complaint, but with much lower confidence.
  • rMSE as a Diagnostic Tool: The authors found that rMSE, when normalized by the layer's output scale, is a highly reliable indicator of where a pipeline starts to diverge. A "jump" in rMSE after a specific operation (like a quantization step) immediately flags a buggy kernel.
  • Minimal Developer Effort: By abstracting the logging and comparison logic, the framework reduces the code needed for debugging from hundreds of lines of manual logging to fewer than 15 lines of instrumentation and assertions.
  • Kernel Optimization: Beyond accuracy, ML-EXray revealed that unoptimized reference kernels can be two orders of magnitude slower than optimized ones, providing a roadmap for latency debugging.

Key Results

  • Accuracy Recovery: By identifying and fixing preprocessing bugs (like incorrect resizing or orientation), ML-EXray helped correct model performance by up to 30%.
  • Bug Discovery: The tool successfully pinpointed previously undiscovered bugs in official TensorFlow Lite quantization kernels (specifically in depthwise convolutions).
  • Low Overhead: The framework is production-ready. On a mobile GPU, the instrumentation adds only about 3ms of latency per frame and has a tiny memory footprint (~4MB), making it feasible to run during real-world testing.
  • Efficiency: Compared to manual debugging ("W/O ML-EXray"), the lines of code (LoC) required to debug quantization or preprocessing were reduced by 80-95%.

Limitations and Future Work

  • Reference Dependency: The system relies on the existence of a "correct" reference pipeline. If the reference itself is flawed or if the training code is unavailable, the baseline might be misleading.
  • Dynamic Environments: While the paper addresses sensor variations, debugging issues that only appear under specific, transient environmental conditions (like rapid motion blur) remains a challenge.
  • Future Directions: The authors plan to release ML-EXray as an open-source library. Future work could involve automated "auto-fixes" where the system suggests the correct preprocessing parameters based on the assertion results.

Summary updated

Abstract

Benefiting from expanding cloud infrastructure, deep neural networks (DNNs) today have increasingly high performance when trained in the cloud. Researchers spend months of effort competing for an extra few percentage points of model accuracy. However, when these models are actually deployed on edge devices in practice, very often, the performance can abruptly drop over 10% without obvious reasons. The key challenge is that there is not much visibility into ML inference execution on edge devices, and very little awareness of potential issues during the edge deployment process. We present ML-EXray, an end-to-end framework, which provides visibility into layer-level details of the ML execution, and helps developers analyze and debug cloud-to-edge deployment issues. More often than not, the reason for sub-optimal edge performance does not only lie in the model itself, but every operation throughout the data flow and the deployment process. Evaluations show that ML-EXray can effectively catch deployment issues, such as pre-processing bugs, quantization issues, suboptimal kernels, etc. Using ML-EXray, users need to write less than 15 lines of code to fully examine the edge deployment pipeline. Eradicating these issues, ML-EXray can correct model performance by up to 30%, pinpoint error-prone layers, and guide users to optimize kernel execution latency by two orders of magnitude. Code and APIs will be released as an open-source multi-lingual instrumentation library and a Python deployment validation library.
BibTeX
@inproceedings{qiu2022ml,
  author = {Hang Qiu and Ioanna Vavelidou and Jian Li and Evgenya Pergament and Pete Warden and Sandeep Chinchali and Zain Asgar and Sachin Katti},
  title = {{ML-EXray: Visibility into ML Deployment on the Edge}},
  booktitle = {MLSys},
  year = {2022},
  url = {https://openreview.net/forum?id=Psk2o7y0Ojc},
}

Loading related content...

Figure