An exploratory study on the introduction and removal of different types of technical debt in deep learning frameworks

Jiakun Liu, Qiao Huang, Xin Xia, Emad Shihab, David Lo, Shanping Li

TL;DR

This paper investigates how developers "borrow from the future" by analyzing Self-Admitted Technical Debt (SATD) across seven major deep learning frameworks like TensorFlow and PyTorch. The researchers found that while design debt is the most frequently introduced, requirement debt is the most likely to be paid off, and developers are surprisingly quick to fix their own design shortcuts compared to other types of technical debt.

What Problem Does This Paper Address?

In the rush to ship features, developers often take shortcuts—writing "quick and dirty" code with the intention of fixing it later. This is Technical Debt (TD). When a developer explicitly acknowledges this shortcut in a code comment (e.g., // TODO: This is a hack, fix it later), it’s called Self-Admitted Technical Debt (SATD).

While we know TD is detrimental to long-term software health, we don't fully understand the "lifecycle" of different types of debt. Do developers prioritize fixing buggy logic over missing documentation? Does the fast-paced world of Deep Learning (DL) frameworks handle debt differently than traditional software? This paper fills that gap by tracking the introduction and removal of seven specific categories of debt across the history of the world's most popular DL libraries.

Methods

The authors employed a multi-step pipeline to mine and analyze debt:

  1. Project Selection: They chose seven high-profile, open-source DL frameworks (TensorFlow, Keras, PyTorch, MXNet, CNTK, Caffe, and DL4J).
  2. Comment Extraction: They mined the entire commit history of these projects, using tools like srcML and Python’s tokenize module to isolate comments from code.
  3. SATD Identification: They used a specialized NLP-based "SATD-detector," re-trained on manually labeled data, to distinguish between regular comments and those admitting to technical debt.
  4. Categorization: Using a "card-sorting" method, they classified 24,032 SATD instances into seven types: Design, Defect, Documentation, Requirement, Test, Algorithm, and Compatibility.
  5. Survival Analysis: To measure how "fast" debt is removed, they used Kaplan-Meier estimators and Cox proportional hazards models—statistical techniques typically used in medical research to track patient outcomes, but here applied to the "lifespan" of a code comment.

Datasets

The study utilized the version control history of seven open-source GitHub repositories:

  • TensorFlow, Keras, PyTorch, MXNet, CNTK, Caffe, and DL4J.
  • The raw data consisted of 445,149 distinct comments.
  • The final analyzed dataset included 24,032 SATD instances introduced at least one year before the projects' latest stable releases to avoid "right-censoring" bias (debt that hasn't had time to be removed yet).

Key Insights and Ideas

  • Design Debt is the Default Shortcut: Roughly 65% of all admitted debt is "Design Debt" (sub-optimal architecture or misplaced code). Developers seem most comfortable admitting they’ve compromised on the "how" of the implementation.
  • The "Self-Cleaning" Phenomenon: Most debt is "self-removed," meaning the person who introduced the shortcut is the one who eventually fixes it. This suggests that SATD acts as a personal "sticky note" for developers.
  • Prioritization Paradox: Even though design debt is the most common, Requirement Debt (incomplete features) has the highest removal rate. Developers are under more pressure to finish a feature than to clean up the architecture.
  • The Speed of DL: Technical debt in DL frameworks is removed significantly faster than in traditional software (like the Eclipse IDE or Apache projects studied in prior work). The high stakes and massive user bases of DL libraries likely put more pressure on maintainers to "pay back" their debt quickly.

Key Results

  • Prevalence: Design debt is the most common (65.35%), followed by Requirement debt (12.81%) and Algorithm debt (7.05%).
  • Removal Rates: Requirement debt is removed the most (76.3% removal rate), while Compatibility debt is the hardest to get rid of (only 48.1% removed).
  • Pace of Removal: Design debt is removed the fastest, with an average "half-life" of about 101.5 days. In contrast, Documentation and Test debt are the "slow movers," often lingering for much longer.
  • Who Cleans Up? High-activity developers (the "power users" of the repo) are significantly more likely to be the ones fixing complex Compatibility and Documentation debt, whereas Design and Requirement debt are usually handled by the original author.

Limitations and Future Work

  • Comment-Code Gap: The study relies on comments. While prior research suggests comments and code change together 72-91% of the time, there is still a risk that a developer fixed the code but forgot to delete the // TODO comment.
  • Subjectivity: Manual classification of debt types is inherently subjective, though the authors mitigated this with high inter-rater agreement scores (Cohen’s Kappa of 0.85).
  • Scope: The findings are specific to DL frameworks. The authors suggest future work should compare these patterns against other domains like IoT or traditional web services.
  • The "Why": The paper identifies what happens, but not always why. Future qualitative studies (like developer interviews) could explain why certain types of debt, like documentation, are consistently neglected.

Summary updated

Abstract

To complete tasks faster, developers often have to sacrifice the quality of the software. Such compromised practice results in the increasing burden to developers in future development. The metaphor, technical debt, describes such practice. Prior research has illustrated the negative impact of technical debt, and many researchers investigated how developers deal with a certain type of technical debt. However, few studies focused on the removal of different types of technical debt in practice. To fill this gap, we use the introduction and removal of different types of self-admitted technical debt (i.e., SATD) in 7 deep learning frameworks as an example. This is because deep learning frameworks are some of the most important software systems today due to their prevalent use in life-impacting deep learning applications. Moreover, the field of the development of different deep learning frameworks is the same, which enables us to find common behaviors on the removal of different types of technical debt across projects. By mining the file history of these frameworks, we find that design debt is introduced the most along the development process. As for the removal of technical debt, we find that requirement debt is removed the most, and design debt is removed the fastest. Most of test debt, design debt, and requirement debt are removed by the developers who introduced them. Based on the introduction and removal of different types of technical debt, we discuss the evolution of the frequencies of different types of technical debt to depict the unresolved sub-optimal trade-offs or decisions that are confronted by developers along the development process. We also discuss the removal patterns of different types of technical debt, highlight future research directions, and provide recommendations for practitioners.
BibTeX
@inproceedings{liu2021exploratory,
  author = {Jiakun Liu and Qiao Huang and Xin Xia and Emad Shihab and David Lo and Shanping Li},
  title = {{An exploratory study on the introduction and removal of different types of technical debt in deep learning frameworks}},
  booktitle = {Empir. Softw. Eng},
  year = {2021},
  url = {https://openreview.net/forum?id=vCImjWk4Xh},
}

Loading related content...

Figure