TL;DR
This paper presents the first comprehensive study to characterize performance problems (PPs) in deep learning (DL) systems, identifying their symptoms, root causes, and stages of occurrence. The authors also developed a static checker, DEEPPERF, which successfully detected hundreds of new PPs in real-world DL projects, demonstrating the prevalence and impact of these issues.
What Problem Does This Paper Address?
Deep learning (DL) systems are increasingly used across various domains, but their unique data-driven programming paradigm introduces new engineering challenges. While traditional software bugs in DL systems have been studied, performance problems (PPs) – which can lead to excessive resource consumption, financial loss, and poor user experience – have been largely overlooked. This paper aims to fill this gap by systematically understanding PPs in DL systems, specifically those built with TensorFlow and Keras, and evaluating existing tools' ability to address them.
Methods
The researchers employed a multi-faceted approach:
- Data Collection: They mined 210 Stack Overflow posts, identifying 224 performance problems (PPs) related to TensorFlow and Keras.
- Manual Analysis & Characterization: Two authors manually analyzed these PPs to categorize their symptoms (e.g., slow execution, out of memory), root causes (e.g., API misuse, model configuration), and the stages in the DL pipeline where they were introduced and exposed.
- Benchmark Construction: A benchmark of 58 PPs was created by reproducing these issues to assess the effectiveness of existing performance analysis tools.
- Tool Assessment: The capabilities of TensorFlow Profiler, XLA (Accelerated Linear Algebra), and TensorFlow documentation were evaluated against the benchmark.
- Tool Development: Based on their findings, the authors developed DEEPPERF, a static checker designed to detect three specific types of PPs.
Datasets
- Stack Overflow Posts: 210 posts were used to collect 224 performance problems.
- Custom Benchmark: A benchmark of 58 PPs was constructed from the collected data for evaluating existing tools.
- GitHub Projects: 1,108 GitHub repositories were used to evaluate the developed DEEPPERF tool.
Key Insights and Ideas
- PPs are Pervasive and Distinct: Performance problems in DL systems are common and exhibit unique symptoms and root causes compared to general software bugs, necessitating specialized study.
- API Misuse is a Major Culprit: A significant portion of PPs stem from incorrect or inefficient use of DL library APIs, highlighting the need for better developer understanding and guidance.
- Data Preparation is a Buggy Stage: The data preparation stage is identified as both the most frequent source of PP introduction and a significant stage for PP exposure, suggesting it's a critical area for optimization and developer attention.
- Existing Tools Fall Short: Standard performance analysis tools like TensorFlow Profiler and XLA have limited applicability and effectiveness in detecting and resolving PPs in DL systems.
- Static Analysis Shows Promise: The development of DEEPPERF demonstrates that rule-based static analysis can effectively detect a substantial number of PPs in DL code.
Key Results
- 224 PPs Characterized: The study identified and categorized 224 PPs, revealing that over half slow down DL systems, and nearly a third involve extreme resource consumption.
- Root Causes Identified: API misuse was the most common root cause (51.3%), followed by model-related issues (22.3%) and data processing problems (9.4%).
- Stage Analysis: Data preparation was the most bug-prone stage for introduction, while training and data preparation were the most bug-affecting stages for exposure. Notably, nearly 40% of PPs were not exposed in their introducing stage, making localization challenging.
- Tool Assessment: TensorFlow Profiler was applicable to only 25.9% of benchmark PPs and solved few. XLA was applicable to 41.4% but solved even fewer. TensorFlow documentation was applicable to only 20.7%.
- DEEPPERF Performance: The static checker DEEPPERF detected 488 new PPs across 130 GitHub projects, with 105 confirmed and 27 fixed by developers.
Limitations and Future Work
- Framework Specificity: The study focused on TensorFlow and Keras, and findings may not directly generalize to other DL frameworks like PyTorch.
- Data Source: Reliance on Stack Overflow posts might introduce biases, and exploring GitHub for PPs is suggested.
- Manual Effort: The manual analysis and benchmark construction were time-consuming, limiting the scale of the initial study.
- DEEPPERF Scope: DEEPPERF currently detects only three types of PPs. Future work includes expanding its rule set and exploring automated rule learning.
- Generalizability of PPs: The authors acknowledge that the benchmark size is not very large and are continuously working to expand it.
- Future Directions: The authors suggest developing intelligent techniques for high-performance DL system development, enhancing PP detection tools, and creating more sophisticated PP localization techniques, including automated log analysis and API usage mining.
Summary updated