Abstract
The same machine learning model running on different edge devices may produce highly-divergent outputs on a nearly-identical input. Possible reasons for the divergence include differences in the device sensors, the device's signal processing hardware and software, and its operating system and processors. This paper presents the first methodical characterization of the variations in model prediction across real-world mobile devices. We demonstrate that accuracy is not a useful metric to characterize prediction divergence, and introduce a new metric, instability, which captures this variation. We characterize different sources for instability, and show that differences in compression formats and image signal processing account for significant instability in object classification models. Notably, in our experiments, 14-17% of images produced divergent classifications across one or more phone models. We evaluate three different techniques for reducing instability, demonstrating that fine-tuning techniques can reduce instability by 75%.
Introduction: The Problem of Model Instability
When evaluating computer vision models, researchers typically rely on static, gold-standard datasets like ImageNet. However, in real-world deployments, a model does not process a "perfect" tensor; it processes data captured and transformed by a specific device's hardware and software stack.
This study identifies a phenomenon termed "model instability": the situation where two different mobile devices photograph the same object under identical conditions, yet the model produces contradictory labels (e.g., "water bottle" on a Samsung vs. "bubble" on an iPhone). While the resulting images may appear identical to the human eye, the underlying pixel-level differences—introduced by sensors and signal processors—are sufficient to derail model consistency. This instability represents a significant hurdle for the reliable deployment of AI at the edge.
Methodology
To quantify and address this issue, the researchers developed a rigorous experimental framework.
Defining the Instability Metric
The authors argue that standard accuracy is a deceptive metric for edge deployment. A model might achieve 60% accuracy on both an iPhone and a Samsung, but it may not be correct on the same 60% of images. To capture this, they defined instability as the probability that a model returns a correct class on one device or environment but an incorrect one on another for the same input.
Controlled Hardware Lab
To isolate hardware and software variables, the team constructed a specialized rig. Five smartphones (Samsung S10, LG K10, HTC Desire 10, Moto G5, and iPhone XR) were mounted in a dark room with light-blocking curtains. Images were displayed on a high-quality monitor, and each phone photographed the display to ensure the "ground truth" input remained as consistent as possible.
Isolation Experiments
The researchers isolated potential sources of noise through several targeted tests:
- ISP Variation: Running raw images through different software Image Signal Processors (e.g., Adobe vs. ImageMagick).
- Compression Formats: Comparing the impact of JPEG, PNG, WebP, and HEIF.
- Computational Variance: Running inference on different mobile CPUs and operating systems using identical bit-streams to check for floating-point errors.
Key Insights: Root Causes of Divergence
The investigation revealed that the primary drivers of instability are located in the image capture and saving phase, rather than the computation phase.
- The ISP as a Hidden Variable: The Image Signal Processor—responsible for color correction and noise reduction—is a major source of instability. Differences in ISP implementations accounted for approximately 14% of the observed instability.
- Compression Impact: Switching between formats like JPEG and HEIF, or even varying JPEG quality levels, introduced 7.6% to 9.6% instability, even when overall accuracy remained stable.
- OS and CPU Neutrality: Interestingly, running the exact same image file across different mobile processors and operating systems resulted in negligible instability (0.64%), suggesting that modern mobile hardware is computationally consistent for inference.
Mitigation and Stability Training
To combat these variations, the authors adapted "stability training" techniques. This involves augmenting the training dataset with noise that simulates device-specific variations. They utilized specialized loss functions—specifically Relative Entropy (KL Divergence) and Embedding Distance—to force the model to produce similar internal representations for both clean and "noisy" versions of an image.
Results and Conclusion
The study's findings underscore the prevalence of hardware-induced variance:
- Quantified Instability: In end-to-end tests, roughly 15% of images yielded inconsistent results across different phone models.
- Mitigation Success: By employing stability training with an Embedding Distance loss, the researchers reduced model instability from a baseline of 7.22% to 3.91%.
- Calibration Efficiency: The researchers found that models can be made robust with minimal data; augmenting a model with as few as 10 photos per class from a target device during fine-tuning significantly improves consistency.
While this study focused on a subset of classes and devices, it provides a vital foundation for understanding edge AI reliability. Future work will need to address more complex variables, such as physical lens variations and dynamic lighting conditions, to further bridge the gap between lab performance and real-world stability.
Summary updated