Understanding performance problems in deep learning systems

Junming Cao, Bihuan Chen, Chao Sun, Longjie Hu, Shuaihong Wu, Xin Peng

Abstract: Deep learning (DL) has been widely applied to many domains. Unique challenges in engineering DL systems are posed by the programming paradigm shift from traditional systems to DL systems, and performance is one of the challenges. Performance problems (PPs) in DL systems can cause severe consequences such as excessive resource consumption and financial loss. While bugs in DL systems have been extensively investigated, PPs in DL systems have hardly been explored. To bridge this gap, we present the first comprehensive study to i) characterize symptoms, root causes, and introducing and exposing stages of PPs in DL systems developed in TensorFLow and Keras, with 224 PPs collected from 210 StackOverflow posts, and to ii) assess the capability of existing performance analysis approaches in tackling PPs, with a constructed benchmark of 58 PPs in DL systems. Our findings shed light on the implications on developing high-performance DL systems, and detecting and localizing PPs in DL systems. To demonstrate the usefulness of our findings, we develop a static checker DeepPerf to detect three types of PPs. It has detected 488 new PPs in 130 GitHub projects. 105 and 27 PPs have been confirmed and fixed.

Introduction

Deep learning has shifted the programming paradigm from logic-based code to data-driven systems, introducing unique performance challenges. For instance, training a model like GPT-3 costs millions of dollars; a minor inefficiency can result in staggering financial losses and wasted hardware cycles. Despite the high stakes, developers often struggle to diagnose why their DL systems are slow or memory-hungry. While functional bugs (e.g., crashes) are well-documented, performance-specific symptoms and root causes have remained largely anecdotal until now.

Methodology

The researchers followed a multi-step empirical approach to characterize performance problems (PPs) in TensorFlow and Keras:

  1. Data Collection: Mined 210 StackOverflow posts to identify 224 distinct performance problems.
  2. Taxonomy Construction: Manually labeled these PPs to characterize their symptoms, root causes, and the stages where they were introduced versus where they manifested.
  3. Benchmark Reproduction: Built a benchmark of 58 reproduced PPs using Docker to ensure environment consistency.
  4. Tool Assessment: Evaluated the effectiveness of the TensorFlow Profiler, the XLA compiler, and official documentation in identifying these issues.
  5. Tool Development: Created DeepPerf, a static analysis tool that uses Abstract Syntax Trees (AST) and type inference to detect common DL performance anti-patterns.

Characterizing Performance Problems

The study provides a detailed taxonomy of how performance issues manifest and what causes them.

Taxonomy of symptoms for performance problems in DL systems.

As illustrated in Figure 1, the majority of symptoms relate to execution time (56%) and memory issues (25%), such as Out-of-Memory (OOM) errors or memory leaks.

Taxonomy of root causes for performance problems in DL systems.

Figure 2 reveals that API misuse is the dominant root cause, accounting for over 51% of performance problems. This often occurs when developers "reinvent the wheel" with manual Python loops instead of using optimized library features like the tf.data pipeline.

Key Insights and Results

  • The "Graph Confusion" Trap: Approximately 12% of PPs arise from developers treating DL graphs like imperative code—for instance, repeatedly adding nodes to a computation graph inside a training loop, causing the system to slow down exponentially over time.
  • Localization Challenges: In 40% of cases, the stage where a bug is introduced (e.g., data preparation) is different from where it is exposed (e.g., training). This "distance" makes manual debugging exceptionally difficult.
  • Tool Fragility: Existing profiling tools are often brittle. Ironically, many PPs cause OOM errors that prevent the TensorFlow Profiler itself from running, creating a "catch-22" for developers.
  • DeepPerf Success: The static checker identified 488 new PPs across 130 GitHub projects. Fixing these identified issues resulted in average execution time improvements of 35.6% for certain API misuse cases.

Conclusion

This study highlights that performance problems in DL systems are frequently caused by API misuses and architectural misunderstandings. While existing tools struggle to address these issues, static analysis tools like DeepPerf show significant promise. Future work should focus on automated rule mining to learn new performance anti-patterns directly from developer discussions and improving static type inference for Python to make checkers more robust.

Summary updated

Abstract

Deep learning (DL) has been widely applied to many domains. Unique challenges in engineering DL systems are posed by the programming paradigm shift from traditional systems to DL systems, and performance is one of the challenges. Performance problems (PPs) in DL systems can cause severe consequences such as excessive resource consumption and financial loss. While bugs in DL systems have been extensively investigated, PPs in DL systems have hardly been explored. To bridge this gap, we present the first comprehensive study to i) characterize symptoms, root causes, and introducing and exposing stages of PPs in DL systems developed in TensorFLow and Keras, with 224 PPs collected from 210 StackOverflow posts, and to ii) assess the capability of existing performance analysis approaches in tackling PPs, with a constructed benchmark of 58 PPs in DL systems. Our findings shed light on the implications on developing high-performance DL systems, and detecting and localizing PPs in DL systems. To demonstrate the usefulness of our findings, we develop a static checker DeepPerf to detect three types of PPs. It has detected 488 new PPs in 130 GitHub projects. 105 and 27 PPs have been confirmed and fixed.
BibTeX
@inproceedings{cao2022understanding,
  author = {Junming Cao and Bihuan Chen and Chao Sun and Longjie Hu and Shuaihong Wu and Xin Peng},
  title = {{Understanding performance problems in deep learning systems}},
  booktitle = {ESEC/SIGSOFT FSE},
  year = {2022},
  url = {https://openreview.net/forum?id=az9IZIxGIb},
}
Figure