Abstract
Deep learning models often experience unexpected performance degradation when transitioning from high-end training environments to diverse deployment hardware and software stacks. This paper introduces SysNoise, a systematic study of "system inconsistency"—the subtle discrepancies in data handling across different system implementations. Through a comprehensive benchmark of over 20 models, we demonstrate that these inconsistencies can lead to significant accuracy drops (up to 10%) and that standard robustness techniques are often insufficient.
Introduction
In the standard deep learning workflow, models are typically trained in optimized cloud environments using specific hardware (e.g., NVIDIA GPUs) and software libraries (e.g., DALI). However, real-world deployment occurs on a heterogeneous landscape of edge devices, DSPs, and ASICs, each utilizing different software backends like TensorRT, SNPE, or CANN. While researchers have extensively studied adversarial attacks and natural corruptions, the impact of varying system implementations—termed SysNoise—has been largely overlooked. This paper formalizes SysNoise as a critical threat to model reliability.
The System Inconsistency Problem
SysNoise arises because even with identical model weights, the underlying system implementation changes during deployment. Tiny variations in image decoding, coordinate calculation, or operator logic introduce noise that can undermine performance. For instance, switching from a cloud-based decoder to an edge-optimized one might seem trivial, but the resulting pixel-level differences can lead to misclassification.
Categorizing SysNoise
The authors categorize SysNoise into three primary stages of the inference pipeline:
- Pre-processing: Inconsistencies in image decoding (e.g., libjpeg vs. turbojpeg) and resizing algorithms (e.g., Pillow-Bilinear vs. OpenCV-Nearest).
- Model Inference: Variations in operator implementations, such as the "Ceil mode" flag in pooling layers, and the effects of reduced data precision (FP32 to INT8).
- Post-processing: Differences in final logic, such as Non-Maximum Suppression (NMS) implementations or bounding box coordinate offsets.
Key Insights and Benchmarking
The study evaluates SysNoise across tasks including image classification (ImageNet-1K), object detection (MS COCO), instance segmentation (Cityscapes), and NLP. Several key findings emerged:
- Architecture Sensitivity: Lightweight models like MobileNetV2 are more vulnerable than larger architectures. Vision Transformers (ViTs) show robustness to decoder noise but sensitivity to color mode changes.
- The "Ceil Mode" Trap: A simple boolean choice in max-pooling (ceil vs. floor) can cause profound mismatches between training and deployment.
- Failure of Standard Defenses: Common robustness techniques like data augmentation and adversarial training (PGD) show limited effectiveness against SysNoise, sometimes even increasing sensitivity.
Key Results
The cumulative effect of mismatches across the pipeline is significant. While a single noise source might cause a minor drop, the total degradation can exceed 10%.
Specifically, SysNoise caused up to a 9.97% drop in classification accuracy and a 10.67% drop in detection mAP. Quantization to INT8 also introduced noticeable noise, particularly in NLP tasks. To mitigate these effects, the authors propose Mix Training, where models are exposed to multiple decoders and resizing methods during training. This strategy significantly reduced performance variance across systems; for ResNet-50, the standard deviation of accuracy across decoders dropped from 0.36 to 0.06.
Conclusion
SysNoise represents a silent but significant barrier to reliable model deployment. By benchmarking these inconsistencies and proposing mitigation strategies like Mix Training, this work provides a foundation for building system-invariant deep learning models. Future research will expand this benchmark to specialized fields like audio processing and automate the detection of system mismatches.
Summary updated