Abstract
TODO comments are very widely used by software developers to describe their pending tasks during software development. However, after performing the task developers sometimes neglect or simply forget to remove the TODO comment, resulting in obsolete TODO comments. These obsolete TODO comments can confuse development teams and may cause the introduction of bugs in the future, decreasing the software's quality and maintainability. In this work, we propose a novel model, named TDCleaner (TODO comment Cleaner), to identify obsolete TODO comments in software projects. TDCleaner can assist developers in just-in-time checking of TODO comments status and avoid leaving obsolete TODO comments. Our approach has two main stages: offline learning and online prediction. During offline learning, we first automatically establish <code_change, todo_comment, commit_msg> training samples and leverage three neural encoders to capture the semantic features of TODO comment, code change and commit message respectively. TDCleaner then automatically learns the correlations and interactions between different encoders to estimate the final status of the TODO comment. For online prediction, we check a TODO comment's status by leveraging the offline trained model to judge the TODO comment's likelihood of being obsolete. We built our dataset by collecting TODO comments from the top-10,000 Python and Java Github repositories and evaluated TDCleaner on them. Extensive experimental results show the promising performance of our model over a set of benchmarks. We also performed an in-the-wild evaluation with real-world software projects, we reported 18 obsolete TODO comments identified by TDCleaner to Github developers and 9 of them have already been confirmed and removed by the developers, demonstrating the practical usage of our approach.
Introduction
Software developers frequently use TODO comments to flag pending tasks or future improvements. However, as codebases evolve, these comments are often forgotten even after the associated tasks are completed. These "obsolete TODO comments" represent a form of technical debt that can mislead developers, introduce bugs, and erode trust in project documentation.
The primary challenge in automating the removal of these comments is the "lexical gap" between structured source code and natural language comments. A developer might resolve a TODO using terminology that differs significantly from the words used in the comment itself. To address this, we introduce TDCleaner, a deep-learning-based approach that understands the semantic relationship between code changes and developer intent.
Methodology: The TDCleaner Architecture
TDCleaner treats the identification of obsolete TODOs as a binary classification task. The model determines whether a specific code change has resolved the task described in a TODO comment.
The core of TDCleaner is a "triplet" architecture consisting of three specialized BERT-based encoders:
- Code Change Encoder: Captures the semantics of added and removed lines within a commit.
- TODO Comment Encoder: Processes the natural language text of the TODO comment.
- Commit Message Encoder: Analyzes the developer's stated intent, providing a crucial link between the code and the comment.
These encoders transform inputs into high-dimensional vectors, which are then fused and passed through a Multi-Layer Perceptron (MLP). The MLP learns to identify the complex correlations between these three inputs to predict the likelihood that a TODO is obsolete.
Data Collection and Labeling
To train TDCleaner, we constructed a large-scale dataset by mining the top 10,000 Python and Java repositories on GitHub. We identified over 410,000 TODO-related commits for Python and 350,000 for Java.
We employed an automatic labeling heuristic to create a ground truth for the model:
- Positive Samples: If a TODO comment was removed in a commit (
LinesRemoved), it was labeled as resolved/obsolete. - Negative Samples: If the surrounding code changed but the TODO comment remained (
LinesEqual), it was labeled as unresolved.
The final dataset consisted of approximately 74,000 labeled samples for Python and 66,000 for Java.
Experimental Results
TDCleaner was evaluated against several baseline methods, including keyword-matching and information retrieval techniques.
- Superior Performance: TDCleaner achieved an F1-score of 84.7% for Python and 85.3% for Java, significantly outperforming traditional methods by over 20 percentage points.
- Importance of Semantics: An ablation study revealed that removing the BERT component led to a significant performance drop (F1 decreased by ~17-19%), confirming that deep semantic understanding is vital for bridging the lexical gap.
- Real-World Impact: In an "in-the-wild" evaluation, TDCleaner identified 18 obsolete TODOs in active GitHub projects. Nine of these were confirmed and removed by maintainers, validating the tool's practical utility.
Conclusion
TDCleaner demonstrates that deep learning, specifically BERT-based semantic analysis of code changes and commit messages, can effectively identify obsolete TODO comments. By automating this process, TDCleaner helps developers maintain cleaner codebases and reduce technical debt. Future work will focus on expanding the tool to support more programming languages and other types of task-related comments like FIXME and HACK.
Summary updated