SysNoise: Exploring and Benchmarking Training-Deployment System Inconsistency

Yan Wang, Yuhang Li, Ruihao Gong, Aishan Liu, Jian Hu, Yongqiang Yao, Yunchen Zhang, Tianzi Xiao, Fengwei Yu, Xianglong Liu

Abstract

Deep learning models often experience unexpected performance degradation when transitioning from high-end training environments to diverse deployment hardware and software stacks. This paper introduces SysNoise, a systematic study of "system inconsistency"—the subtle discrepancies in data handling across different system implementations. Through a comprehensive benchmark of over 20 models, we demonstrate that these inconsistencies can lead to significant accuracy drops (up to 10%) and that standard robustness techniques are often insufficient.

Introduction

In the standard deep learning workflow, models are typically trained in optimized cloud environments using specific hardware (e.g., NVIDIA GPUs) and software libraries (e.g., DALI). However, real-world deployment occurs on a heterogeneous landscape of edge devices, DSPs, and ASICs, each utilizing different software backends like TensorRT, SNPE, or CANN. While researchers have extensively studied adversarial attacks and natural corruptions, the impact of varying system implementations—termed SysNoise—has been largely overlooked. This paper formalizes SysNoise as a critical threat to model reliability.

The System Inconsistency Problem

SysNoise arises because even with identical model weights, the underlying system implementation changes during deployment. Tiny variations in image decoding, coordinate calculation, or operator logic introduce noise that can undermine performance. For instance, switching from a cloud-based decoder to an edge-optimized one might seem trivial, but the resulting pixel-level differences can lead to misclassification.

Categorizing SysNoise

The authors categorize SysNoise into three primary stages of the inference pipeline:

  1. Pre-processing: Inconsistencies in image decoding (e.g., libjpeg vs. turbojpeg) and resizing algorithms (e.g., Pillow-Bilinear vs. OpenCV-Nearest).
  2. Model Inference: Variations in operator implementations, such as the "Ceil mode" flag in pooling layers, and the effects of reduced data precision (FP32 to INT8).
  3. Post-processing: Differences in final logic, such as Non-Maximum Suppression (NMS) implementations or bounding box coordinate offsets.
The inference pipeline and potential sources of system inconsistency across pre-processing, model inference, and post-processing stages.

Key Insights and Benchmarking

The study evaluates SysNoise across tasks including image classification (ImageNet-1K), object detection (MS COCO), instance segmentation (Cityscapes), and NLP. Several key findings emerged:

  • Architecture Sensitivity: Lightweight models like MobileNetV2 are more vulnerable than larger architectures. Vision Transformers (ViTs) show robustness to decoder noise but sensitivity to color mode changes.
  • The "Ceil Mode" Trap: A simple boolean choice in max-pooling (ceil vs. floor) can cause profound mismatches between training and deployment.
  • Failure of Standard Defenses: Common robustness techniques like data augmentation and adversarial training (PGD) show limited effectiveness against SysNoise, sometimes even increasing sensitivity.

Key Results

The cumulative effect of mismatches across the pipeline is significant. While a single noise source might cause a minor drop, the total degradation can exceed 10%.

Cumulative impact of various SysNoise sources on object detection performance (mAP), showing a total drop of 10.67%.

Specifically, SysNoise caused up to a 9.97% drop in classification accuracy and a 10.67% drop in detection mAP. Quantization to INT8 also introduced noticeable noise, particularly in NLP tasks. To mitigate these effects, the authors propose Mix Training, where models are exposed to multiple decoders and resizing methods during training. This strategy significantly reduced performance variance across systems; for ResNet-50, the standard deviation of accuracy across decoders dropped from 0.36 to 0.06.

Conclusion

SysNoise represents a silent but significant barrier to reliable model deployment. By benchmarking these inconsistencies and proposing mitigation strategies like Mix Training, this work provides a foundation for building system-invariant deep learning models. Future research will expand this benchmark to specialized fields like audio processing and automate the detection of system mismatches.

Summary updated

Abstract

Extensive studies have shown that deep learning models are vulnerable to adversarial and natural noises, yet little is known about model robustness on noises caused by different system implementations. In this paper, we for the first time introduce SysNoise, a frequently occurred but often overlooked noise in the deep learning training-deployment cycle. In particular, SysNoise happens when the source training system switches to a disparate target system in deployments, where various tiny system mismatch adds up to a non-negligible difference. We first identify and classify SysNoise into three categories based on the inference stage; we then build a holistic benchmark to quantitatively measure the impact of SysNoise on 20+ models, comprehending image classification, object detection, instance segmentation and natural language processing tasks. Our extensive experiments revealed that SysNoise could bring certain impacts on model robustness across different tasks and common mitigations like data augmentation and adversarial training show limited effects on it. Together, our findings open a new research topic and we hope this work will raise research attention to deep learning deployment systems accounting for model performance. We have open-sourced the benchmark and framework at https://modeltc.github.io/systemnoise_web.
BibTeX
@misc{wang2023sysnoise,
  author = {Yan Wang and Yuhang Li and Ruihao Gong and Aishan Liu and Yanfei Wang and Jian Hu and Yongqiang Yao and Yunchen Zhang and Tianzi Xiao and Fengwei Yu and Xianglong Liu},
  title = {{SysNoise: Exploring and Benchmarking Training-Deployment System Inconsistency}},
  year = {2023},
  eprint = {2307.00280},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.00280},
}
Figure