TL;DR
Researchers have developed TDCleaner, a deep-learning-based tool designed to automatically identify obsolete TODO comments in source code. By leveraging BERT-based encoders to analyze code changes, commit messages, and the comments themselves, TDCleaner achieves an F1-score of approximately 85%, significantly outperforming traditional keyword-matching methods and helping developers maintain cleaner, more reliable codebases.
What Problem Does This Paper Address?
Software developers frequently use TODO comments to flag pending tasks. However, in the fast-paced world of software evolution, these comments are often forgotten even after the task is completed. These obsolete TODO comments become a form of "technical debt."
They create several problems:
- Confusion: They mislead team members about the actual state of the project.
- Bugs: Developers might try to "fix" something that has already been addressed, potentially introducing new errors.
- Loss of Trust: When a codebase is littered with stale comments, developers lose confidence in the reliability of the documentation.
The challenge in automating their removal lies in the "lexical gap." Code is formal and structured, while comments are natural language. Simply looking for overlapping words isn't enough because a developer might resolve a TODO using different terminology than what was used in the comment.
Methods
The authors propose TDCleaner (TODO comment Cleaner), which treats the problem as a binary classification task: Is this TODO resolved or unresolved?
The core of TDCleaner is a "triplet" architecture using three specialized BERT-based encoders:
- Code Change Encoder: Captures the semantics of added and removed lines in a commit.
- TODO Comment Encoder: Processes the natural language text of the TODO.
- Commit Message Encoder: Analyzes the developer's stated intent for the change, which often provides the "missing link" between the code and the comment.
These three encoders transform their respective inputs into high-dimensional vectors. These vectors are then "fused" and fed into a Multi-Layer Perceptron (MLP). The MLP learns the complex correlations between the three inputs to predict the likelihood that the code change actually resolved the specific task mentioned in the TODO.
Datasets
The researchers built a massive, first-of-its-kind dataset by mining the top 10,000 Python and Java repositories on GitHub.
- Scale: They identified over 410,000 TODO-related commits for Python and 350,000 for Java.
- Labeling: They used a clever heuristic for automatic labeling. If a TODO was removed in a commit (
LinesRemoved), it was labeled as a Positive Sample (resolved). If the code changed but the TODO remained (LinesEqual), it was labeled as a Negative Sample (unresolved). - Final Count: The resulting dataset included roughly 74,000 labeled samples for Python and 66,000 for Java.
Key Insights and Ideas
- The Power of Intent: One of the most important findings is that commit messages are vital. Sometimes the code change alone is too ambiguous to determine if a TODO is resolved, but the commit message provides the necessary context to bridge the gap.
- Beyond Keywords: Traditional tools (like IRSC or keyword overlap) fail when developers use synonyms. Because TDCleaner uses BERT, it understands semantics. For example, it can link a TODO about "cleaning up" to a code change that "refactors" a function.
- Co-evolution is Key: The paper emphasizes that TODOs and code must be viewed as co-evolving entities. You cannot judge a TODO's status in isolation; you must look at the history of how the code around it changed.
Key Results
TDCleaner was tested against several baselines, including keyword-matching (TCO/TMO) and information retrieval methods (IRSC).
- Performance: TDCleaner achieved an F1-score of 84.7% for Python and 85.3% for Java, outperforming the best baseline (IRSC) by over 20 percentage points.
- Ablation Study: Removing the BERT component dropped performance significantly (F1 fell by ~17-19%), proving that deep semantic understanding is essential for this task.
- In-the-Wild Validation: The authors reported 18 obsolete TODOs identified by the tool to real-world GitHub developers. 9 of these were immediately confirmed and removed by the maintainers, with one developer even asking, "I wonder how did you find this? Just browsing the code at leisure time? :)"
Limitations and Future Work
While highly effective, the authors acknowledge a few hurdles:
- Multiple TODOs: The current model filters out commits containing multiple TODOs to avoid noise during training. Future versions will need to handle these more complex scenarios.
- Complex Changes: If a TODO is resolved across many different files or sub-modules, the current "diff-based" approach might struggle to capture the full context.
- Language Expansion: The study focused on Python and Java; the researchers plan to adapt TDCleaner for other languages and other types of task comments like
FIXMEorHACK.
Summary updated