An Exploratory Study on the Introduction and Removal of Different Types of Technical Debt

Jiakun Liu, Qiao Huang, Xin Xia, Emad Shihab, David Lo, Shanping Li

Abstract

To complete tasks faster, developers often have to sacrifice the quality of the software. Such compromised practice results in the increasing burden to developers in future development. The metaphor, technical debt, describes such practice. Prior research has illustrated the negative impact of technical debt, and many researchers investigated how developers deal with a certain type of technical debt. However, few studies focused on the removal of different types of technical debt in practice. To fill this gap, we use the introduction and removal of different types of self-admitted technical debt (i.e., SATD) in 7 deep learning frameworks as an example. This is because deep learning frameworks are some of the most important software systems today due to their prevalent use in life-impacting deep learning applications. Moreover, the field of the development of different deep learning frameworks is the same, which enables us to find common behaviors on the removal of different types of technical debt across projects. By mining the file history of these frameworks, we find that design debt is introduced the most along the development process. As for the removal of technical debt, we find that requirement debt is removed the most, and design debt is removed the fastest. Most of test debt, design debt, and requirement debt are removed by the developers who introduced them. Based on the introduction and removal of different types of technical debt, we discuss the evolution of the frequencies of different types of technical debt to depict the unresolved sub-optimal trade-offs or decisions that are confronted by developers along the development process. We also discuss the removal patterns of different types of technical debt, highlight future research directions, and provide recommendations for practitioners.

Introduction

In the high-pressure environment of modern software development, teams often face a trade-off between speed and quality. To meet deadlines, developers frequently take shortcuts—a practice known as Technical Debt (TD). While these shortcuts allow for faster initial delivery, they create a "debt" that must eventually be repaid with interest in the form of increased maintenance effort.

A specific and measurable subset of this phenomenon is Self-Admitted Technical Debt (SATD), where developers explicitly acknowledge their shortcuts in code comments (e.g., // TODO: fix this hack). This study explores the lifecycle of SATD within the domain of Deep Learning (DL) frameworks, which are critical components of today's technology stack.

Methodology and Dataset

The researchers analyzed the version control history of seven major open-source DL frameworks. These projects represent a diverse cross-section of the ecosystem, varying in size, age, and primary programming languages.

Overview of the seven deep learning frameworks analyzed in the study.

The study employed a multi-step pipeline to mine and analyze debt:

  1. Extraction: Mining 445,149 comments from the entire commit history of the selected projects.
  2. Identification: Using an NLP-based SATD detector to isolate 24,032 instances of admitted debt.
  3. Categorization: Classifying these instances into seven types: Design, Defect, Documentation, Requirement, Test, Algorithm, and Compatibility.
  4. Survival Analysis: Applying Kaplan-Meier estimators and Cox proportional hazards models to determine the "lifespan" of different debt types.

Key Findings: The Lifecycle of Debt

The study reveals distinct patterns in how different types of debt are introduced and eventually resolved.

Prevalence and Removal Rates

Design debt is overwhelmingly the most common shortcut, accounting for approximately 65.35% of all SATD. This suggests that developers most frequently compromise on internal architecture and code structure to maintain velocity. However, when it comes to repayment, Requirement Debt (incomplete features) takes priority, boasting the highest removal rate at 76.3%.

Prevalence and removal statistics for different types of Self-Admitted Technical Debt (SATD).

The Speed of Resolution

Not all debt is paid back at the same pace. Design Debt is resolved the fastest, with a median survival time of approximately 101.5 days. In contrast, Test Debt and Documentation Debt tend to linger much longer, often remaining in the codebase for hundreds of days. This indicates a prioritization of functional and structural fixes over verification and explanatory tasks.

The "Self-Cleaning" Phenomenon

A significant finding is that most technical debt is "self-removed." The original author of the shortcut is typically the one who eventually fixes it. This suggests that SATD often serves as a personal reminder or "sticky note" for developers rather than a communication tool for the broader team. However, for more complex issues like Compatibility or Documentation debt, high-activity "power users" often step in to assist with the cleanup.

Discussion and Implications

The study highlights that technical debt in DL frameworks is generally removed faster than in traditional software domains studied previously. The rapid evolution and high stakes of the DL field likely drive this increased urgency.

For practitioners, these findings suggest that while design shortcuts are common, they are also actively managed. However, the persistence of Test and Documentation debt remains a concern, as these areas are often neglected in favor of feature completion and architectural fixes.

Limitations

While the study provides a comprehensive look at SATD, it relies on the assumption that code comments accurately reflect the state of the codebase. There is a possibility of "stale" comments where the code was fixed but the comment remained. Additionally, the findings are specific to the DL domain and may not generalize to all types of software development.

Summary updated

Abstract

To complete tasks faster, developers often have to sacrifice the quality of the software. Such compromised practice results in the increasing burden to developers in future development. The metaphor, technical debt, describes such practice. Prior research has illustrated the negative impact of technical debt, and many researchers investigated how developers deal with a certain type of technical debt. However, few studies focused on the removal of different types of technical debt in practice. To fill this gap, we use the introduction and removal of different types of self-admitted technical debt (i.e., SATD) in 7 deep learning frameworks as an example. This is because deep learning frameworks are some of the most important software systems today due to their prevalent use in life-impacting deep learning applications. Moreover, the field of the development of different deep learning frameworks is the same, which enables us to find common behaviors on the removal of different types of technical debt across projects. By mining the file history of these frameworks, we find that design debt is introduced the most along the development process. As for the removal of technical debt, we find that requirement debt is removed the most, and design debt is removed the fastest. Most of test debt, design debt, and requirement debt are removed by the developers who introduced them. Based on the introduction and removal of different types of technical debt, we discuss the evolution of the frequencies of different types of technical debt to depict the unresolved sub-optimal trade-offs or decisions that are confronted by developers along the development process. We also discuss the removal patterns of different types of technical debt, highlight future research directions, and provide recommendations for practitioners.
BibTeX
@inproceedings{liu2021exploratory,
  author = {Jiakun Liu and Qiao Huang and Xin Xia and Emad Shihab and David Lo and Shanping Li},
  title = {{An Exploratory Study on the Introduction and Removal of Different Types of Technical Debt}},
  booktitle = {Empir. Softw. Eng},
  year = {2021},
  url = {https://openreview.net/forum?id=vCImjWk4Xh},
}
Figure