An Empirical Study on the Effectiveness of Large Language Models for SATD Identification and Classification

Mohammad Sadegh Sheikhaei, Yuan Tian, Shaowei Wang, Bowen Xu

Abstract

Self-Admitted Technical Debt (SATD) represents sub-optimal software development choices documented by developers within code comments. While managing SATD is critical for software maintainability, its rarity within large codebases makes manual identification impractical. This study investigates the efficacy of Large Language Models (LLMs), specifically the Flan-T5 family, in identifying and classifying SATD. We evaluate these models across various usage settings, including fine-tuning with specialized classification heads and in-context learning (ICL). Our results show that fine-tuned LLMs outperform traditional baselines in SATD identification, with the Flan-T5-XL model achieving a 7.2% improvement in F1 score over the best non-LLM baseline. In classification tasks, we find that while LLMs lead, traditional CNN models remain highly competitive. Furthermore, we demonstrate that incorporating contextual information, such as surrounding code, significantly benefits larger models while acting as noise for smaller ones.

Introduction

In software engineering, "Technical Debt" describes the eventual cost of choosing an easy or quick solution now instead of a better approach that would take longer. When developers are aware of these shortcuts, they often document them in comments—a phenomenon known as Self-Admitted Technical Debt (SATD). Examples include comments like // TODO: This is a hack, fix it later or // FIXME: This needs a more robust implementation.

Identifying and categorizing SATD is vital for long-term project health, yet it is a "needle in a haystack" problem; SATD typically accounts for only 0.5% to 4% of all code comments. Traditional keyword-based searches (e.g., looking for "TODO") often fail to capture the nuance of complex debt. This paper explores whether LLMs can automate two primary tasks: Identification (binary classification of whether a comment is SATD) and Classification (categorizing the debt into types such as Design, Requirement, Defect, or Test debt).

Methodology

Model Selection and Usage Settings

We evaluated several LLMs, primarily focusing on the Flan-T5 family, which ranges from "Small" (77M parameters) to "XXL" (11B parameters). We compared these against established baselines, including BERT, CodeBERT, and a traditional Convolutional Neural Network (CNN). The models were tested in three distinct settings:

  1. Fine-tuning: Models were trained specifically on SATD datasets. For the Flan-T5 models, we experimented with replacing the standard text-generation layer with a dedicated classification head (a dropout layer followed by a linear layer) to optimize for classification rather than sequence generation.
  2. In-Context Learning (ICL): We utilized the 11B parameter Flan-T5-XXL without additional training. Instead, we provided prompts containing task definitions, category descriptions, and few-shot examples.
  3. Contextual Integration: For classification, we tested the impact of providing the model with surrounding information, such as the file path, method signature, or the entire body of the function containing the comment.

Datasets

The study utilized two benchmark datasets:

  • Maldonado-62k: 62,275 comments from 10 open-source Java projects, used primarily for the identification task.
  • OBrien Dataset: 789 entries from machine-learning repositories, including rich metadata (file paths, commit hashes) used for the six-category classification task.

Key Results

SATD Identification

In the binary identification task, all fine-tuned LLMs outperformed the best non-LLM baseline (CNN). The Flan-T5-XL achieved the highest F1 score of 0.839, a 7.2% improvement over traditional methods. While the 11B parameter XXL model using zero-shot ICL was competitive, it still performed 6.4% to 9.2% worse than the smaller, fine-tuned models, suggesting that fine-tuning remains the superior strategy for simple identification.

SATD Classification

The classification task proved more challenging. The fine-tuned Flan-T5-XL led with an accuracy of 0.620. However, the traditional CNN baseline remained surprisingly robust, surpassing four out of the six LLMs tested. This indicates that for specific multi-class software engineering tasks, massive parameter counts do not automatically guarantee dominance over specialized traditional architectures.

Interestingly, for classification, the few-shot ICL approach (incorporating category descriptions and examples) allowed the XXL model to surpass the fine-tuned "Small" and "Base" Flan-T5 models, showing that semantic guidance in prompts is highly effective for complex categorization.

Discussion and Insights

The "Context" Threshold

A significant finding of this research is the relationship between model size and context. We observed a "context threshold": only the larger models (Flan-T5-Large and XL) successfully utilized extra context like method bodies to improve performance. For smaller models, this additional information acted as noise, leading to a decrease in accuracy. This suggests that the ability to filter relevant signals from surrounding code is an emergent property of larger LLMs.

Architectural Tweaks

Our experiments confirmed that substituting the T5's default generation layer with a classification head provided a noticeable performance boost. This architectural modification is particularly effective when training data is scarce, as it allows the model to focus specifically on the classification boundary.

Conclusion

LLMs represent a significant step forward in managing technical debt. While fine-tuning remains the most effective method for identification, the ability of large models to leverage context and semantic prompts makes them powerful tools for complex classification. Future work should address the limitations of current datasets, as the small size of existing classification benchmarks likely caps the potential of the largest models. Furthermore, exploring decoder-only models (like Llama or GPT-4) may yield even higher reasoning capabilities for nuanced technical debt analysis.

Summary updated

Abstract

Self-Admitted Technical Debt (SATD), a concept highlighting sub-optimal choices in software development documented in code comments or other project resources, poses challenges in the maintainability and evolution of software systems. Large language models (LLMs) have demonstrated significant effectiveness across a broad range of software tasks, especially in software text generation tasks. Nonetheless, their effectiveness in tasks related to SATD is still under-researched. In this paper, we investigate the efficacy of LLMs in both identification and classification of SATD. For both tasks, we investigate the performance gain from using more recent LLMs, specifically the Flan-T5 family, across different common usage settings. Our results demonstrate that for SATD identification, all fine-tuned LLMs outperform the best existing non-LLM baseline, i.e., the CNN model, with a 4.4% to 7.2% improvement in F1 score. In the SATD classification task, while our largest fine-tuned model, Flan-T5-XL, still led in performance, the CNN model exhibited competitive results, even surpassing four of six LLMs. We also found that the largest Flan-T5 model, i.e., Flan-T5-XXL, when used with a zero-shot in-context learning (ICL) approach for SATD identification, provides competitive results with traditional approaches but performs 6.4% to 9.2% worse than fine-tuned LLMs. For SATD classification, few-shot ICL approach, incorporating examples and category descriptions in prompts, outperforms the zero-shot approach and even surpasses the fine-tuned smaller Flan-T5 models. Moreover, our experiments demonstrate that incorporating contextual information, such as surrounding code, into the SATD classification task enables larger fine-tuned LLMs to improve their performance.
BibTeX
@inproceedings{sheikhaei2024empirical,
  author = {Mohammad Sadegh Sheikhaei and Yuan Tian and Shaowei Wang and Bowen Xu},
  title = {{An Empirical Study on the Effectiveness of Large Language Models for SATD Identification and Classification}},
  booktitle = {Empirical},
  year = {2024},
  doi = {10.1007/s10664-024-10548-3},
}
Figure