Understanding Challenges in Deploying Deep Learning Based Software: An Empirical Study

Zhenpeng Chen, Yanbin Cao, Yuanqiang Liu, Haoyu Wang, Tao Xie, Xuanzhe Liu

Abstract

This study provides a systematic analysis of the challenges encountered during the deployment phase of deep learning (DL) software. By analyzing over 3,000 Stack Overflow posts, we demonstrate that deployment is significantly more difficult and time-consuming than model development. We present a taxonomy of 72 specific challenges across cloud, mobile, and browser platforms, highlighting critical gaps in documentation and tool support.

Introduction

The "last mile" of AI—transitioning a model from a research notebook to a production environment—remains a significant hurdle for software engineers. While research often focuses on model training, the deployment phase involves adapting models for diverse hardware, including cloud clusters, mobile devices, and web browsers. This transition often requires complex model conversion and environment configuration that are poorly documented.

Overview of the DL software lifecycle, distinguishing between development and deployment phases.

Methodology

The researchers analyzed a decade of Stack Overflow data (2008–2019), extracting 70,669 DL-related questions. Through keyword filtering and tag analysis, they identified 3,023 posts specifically addressing deployment issues across Server/Cloud (1,325), Mobile (1,533), and Browser (165) platforms. A representative subset of 769 posts underwent a manual open-coding process (requiring approximately 450 man-hours) to construct a hierarchical taxonomy of 72 distinct challenge categories.

Quantifying the Deployment Difficulty

The study reveals that deployment is a major bottleneck in the DL lifecycle. Deployment-related questions are statistically harder to resolve than development ones: 70.7% of deployment questions lacked an accepted answer, compared to 62.7% for non-deployment topics.

Comparison of median response times for deployment vs. non-deployment questions on Stack Overflow.

As shown in the figure above, the median response time for deployment issues is 404.9 minutes—nearly three times longer than the 145.8 minutes observed for general DL development questions. This gap underscores the frustration developers face when navigating cryptic errors and fragmented framework landscapes.

Taxonomy of Deployment Challenges

The qualitative analysis yielded a taxonomy of 72 leaf categories of challenges. Key platform-specific findings include:

  • Mobile Platforms: The primary hurdle is model conversion (e.g., quantization) and dealing with "unsupported operators" that mobile frameworks cannot execute.
  • Cloud/Server: Challenges are dominated by environment configuration and efficient request handling (e.g., batching).
  • Browser: Developers frequently struggle with the asynchronous nature of JavaScript during model loading and confusion regarding local storage versus URL-based loading.
  • Silent Failures: 53% of "unclear" questions involved unexpected runtime errors where the root cause, such as a version mismatch between training and serving environments, was completely hidden from the developer.

Conclusion and Future Directions

The findings suggest a critical need for better developer support in the DL deployment phase. Future research should prioritize automated fault localization tools and configuration checkers that can validate environments before production. Furthermore, framework vendors must improve documentation to bridge the gap between high-level API references and practical, end-to-end deployment tutorials for specific hardware targets.

Summary updated

BibTeX
@inproceedings{chen2020understanding,
  author = {Zhenpeng Chen and Yanbin Cao and Yuanqiang Liu and Haoyu Wang and Tao Xie and Xuanzhe Liu},
  title = {{Understanding Challenges in Deploying Deep Learning Based Software: An Empirical Study}},
  booktitle = {ESEC/SIGSOFT FSE},
  year = {2020},
  doi = {10.1145/3368089.3409759},
}
Figure