Automating the removal of obsolete TODO comments

Zhipeng Gao, Xin Xia, David Lo, John C. Grundy, Thomas Zimmermann

TL;DR

Researchers have developed TDCleaner, a deep-learning-based tool designed to automatically identify obsolete TODO comments in source code. By leveraging BERT-based encoders to analyze code changes, commit messages, and the comments themselves, TDCleaner achieves an F1-score of approximately 85%, significantly outperforming traditional keyword-matching methods and helping developers maintain cleaner, more reliable codebases.

What Problem Does This Paper Address?

Software developers frequently use TODO comments to flag pending tasks. However, in the fast-paced world of software evolution, these comments are often forgotten even after the task is completed. These obsolete TODO comments become a form of "technical debt."

They create several problems:

  1. Confusion: They mislead team members about the actual state of the project.
  2. Bugs: Developers might try to "fix" something that has already been addressed, potentially introducing new errors.
  3. Loss of Trust: When a codebase is littered with stale comments, developers lose confidence in the reliability of the documentation.

The challenge in automating their removal lies in the "lexical gap." Code is formal and structured, while comments are natural language. Simply looking for overlapping words isn't enough because a developer might resolve a TODO using different terminology than what was used in the comment.

Methods

The authors propose TDCleaner (TODO comment Cleaner), which treats the problem as a binary classification task: Is this TODO resolved or unresolved?

The core of TDCleaner is a "triplet" architecture using three specialized BERT-based encoders:

  • Code Change Encoder: Captures the semantics of added and removed lines in a commit.
  • TODO Comment Encoder: Processes the natural language text of the TODO.
  • Commit Message Encoder: Analyzes the developer's stated intent for the change, which often provides the "missing link" between the code and the comment.

These three encoders transform their respective inputs into high-dimensional vectors. These vectors are then "fused" and fed into a Multi-Layer Perceptron (MLP). The MLP learns the complex correlations between the three inputs to predict the likelihood that the code change actually resolved the specific task mentioned in the TODO.

Datasets

The researchers built a massive, first-of-its-kind dataset by mining the top 10,000 Python and Java repositories on GitHub.

  • Scale: They identified over 410,000 TODO-related commits for Python and 350,000 for Java.
  • Labeling: They used a clever heuristic for automatic labeling. If a TODO was removed in a commit (LinesRemoved), it was labeled as a Positive Sample (resolved). If the code changed but the TODO remained (LinesEqual), it was labeled as a Negative Sample (unresolved).
  • Final Count: The resulting dataset included roughly 74,000 labeled samples for Python and 66,000 for Java.

Key Insights and Ideas

  • The Power of Intent: One of the most important findings is that commit messages are vital. Sometimes the code change alone is too ambiguous to determine if a TODO is resolved, but the commit message provides the necessary context to bridge the gap.
  • Beyond Keywords: Traditional tools (like IRSC or keyword overlap) fail when developers use synonyms. Because TDCleaner uses BERT, it understands semantics. For example, it can link a TODO about "cleaning up" to a code change that "refactors" a function.
  • Co-evolution is Key: The paper emphasizes that TODOs and code must be viewed as co-evolving entities. You cannot judge a TODO's status in isolation; you must look at the history of how the code around it changed.

Key Results

TDCleaner was tested against several baselines, including keyword-matching (TCO/TMO) and information retrieval methods (IRSC).

  • Performance: TDCleaner achieved an F1-score of 84.7% for Python and 85.3% for Java, outperforming the best baseline (IRSC) by over 20 percentage points.
  • Ablation Study: Removing the BERT component dropped performance significantly (F1 fell by ~17-19%), proving that deep semantic understanding is essential for this task.
  • In-the-Wild Validation: The authors reported 18 obsolete TODOs identified by the tool to real-world GitHub developers. 9 of these were immediately confirmed and removed by the maintainers, with one developer even asking, "I wonder how did you find this? Just browsing the code at leisure time? :)"

Limitations and Future Work

While highly effective, the authors acknowledge a few hurdles:

  • Multiple TODOs: The current model filters out commits containing multiple TODOs to avoid noise during training. Future versions will need to handle these more complex scenarios.
  • Complex Changes: If a TODO is resolved across many different files or sub-modules, the current "diff-based" approach might struggle to capture the full context.
  • Language Expansion: The study focused on Python and Java; the researchers plan to adapt TDCleaner for other languages and other types of task comments like FIXME or HACK.

Summary updated

Abstract

TODO comments are very widely used by software developers to describe their pending tasks during software development. However, after performing the task developers sometimes neglect or simply forget to remove the TODO comment, resulting in obsolete TODO comments. These obsolete TODO comments can confuse development teams and may cause the introduction of bugs in the future, decreasing the software's quality and maintainability. Manually identifying obsolete TODO comments is time-consuming and expensive. It is thus necessary to detect obsolete TODO comments and remove them automatically before they cause any unwanted side effects. In this work, we propose a novel model, named TDCleaner, to identify obsolete TODO comments in software projects. TDCleaner can assist developers in just-in-time checking of TODO comments status and avoid leaving obsolete TODO comments. Our approach has two main stages: offline learning and online prediction. During offline learning, we first automatically establish <code_change, todo_comment, commit_msg> training samples and leverage three neural encoders to capture the semantic features of TODO comment, code change and commit message respectively. TDCleaner then automatically learns the correlations and interactions between different encoders to estimate the final status of the TODO comment. For online prediction, we check a TODO comment's status by leveraging the offline trained model to judge the TODO comment's likelihood of being obsolete. We built our dataset by collecting TODO comments from the top-10,000 Python and Java Github repositories and evaluated TDCleaner on them. Extensive experimental results show the promising performance of our model over a set of benchmarks. We also performed an in-the-wild evaluation with real-world software projects, we reported 18 obsolete TODO comments identified by TDCleaner to Github developers and 9 of them have already been confirmed and removed by the developers, demonstrating the practical usage of our approach.
BibTeX
@inproceedings{gao2021automating,
  author = {Zhipeng Gao and Xin Xia and David Lo and John C. Grundy and Thomas Zimmermann},
  title = {{Automating the removal of obsolete TODO comments}},
  booktitle = {ESEC/SIGSOFT FSE},
  year = {2021},
  url = {https://openreview.net/forum?id=ghDcTrw3eY4o},
}

Loading related content...

Figure