TL;DR
This paper tackles the problem of identifying uninformative "redundant" comments in software by introducing the concept of "comment entailment." They developed a deep learning tool called CRAIC that uses sequence-to-sequence models to score how well a comment can be inferred from its corresponding code, effectively flagging comments that merely restate what the code already says. This work offers a practical way to improve code documentation quality and provides insights into the informativeness of Javadoc tags.
What Problem Does This Paper Address?
Software comments are crucial for maintenance and understanding, but there's a lack of practical tools to help developers write useful comments. While advice exists to avoid comments that just repeat code, there's little quantitative support for identifying them. This paper addresses this gap by focusing on "redundant" comments – those whose meaning can be easily inferred directly from the code itself. Such comments, while not always outright "bad," can clutter codebases, become maintenance burdens, and give a false sense of understanding by hiding deeper design rationales. The authors aim to provide a method to automatically detect these uninformative comments, encouraging developers to write more valuable explanations.
Methods
The core of the paper introduces the concept of comment entailment, defined as the degree to which a comment's semantics can be inferred from the code. Instead of a binary "entailed/not entailed," they propose an entailment score, where lower scores indicate higher entailment (i.e., more easily inferred and potentially redundant).
To achieve this, they leverage deep sequence-to-sequence (seq2seq) learning, a technique popular in machine translation. The approach involves two main stages:
- Training a comment generator model: A seq2seq model is trained to predict a comment sentence given a code snippet. This model learns to map the code's representation to the comment's natural language.
- Obtaining entailment predictions: The trained model is used to calculate the probability of a comment sentence given the code. This probability is then converted into a perplexity score, which serves as the entailment score. Comments with low perplexity (high probability) are considered highly entailed and thus potentially redundant.
The authors explore different ways to represent the input code for the seq2seq model, including using just the method signature, a combination of the method's beginning and end, or an identifier-based representation.
Datasets
The primary dataset used is a large collection of Java methods paired with their comments, derived from the GitHub Java Corpus. This corpus contains over 3 million method-comment pairs. For training, a random subset of 3 million pairs was used, with 5,000 for validation and 5,000 for testing.
Additionally, a subset of 1352 method-comment sentence pairs from the work of Pascarella and Bacchelli [44] was used to compare their model's predictions with pre-existing comment categories.
For human evaluation, a sample of 45 projects was selected from the Github corpus, and 500 method-comment sentence pairs were collected and annotated by five M.Sc. students.
Key Insights and Ideas
- Comment Entailment as a Proxy for Redundancy: The core idea is that comments easily inferred from code are likely to be redundant and less informative. This reframes the problem from subjective "usefulness" to a more objective measure of predictability from code.
- Seq2Seq Models Excel at Code-Comment Prediction: The paper demonstrates that deep seq2seq models, when trained on code-comment pairs, significantly outperform unimodal language models (trained only on comments) in predicting comment sentences. This highlights the value of incorporating code context.
- Javadoc Tags are Less Informative: A surprising finding is that common Javadoc tags (like
@param,@return,@throws) are, on average, more than twice as predictable from code as free-form comments. This suggests these tags might often be used to restate code elements rather than provide deeper insights. - Human Judgments Align with Model Scores: The study shows a good correlation between the model's perplexity scores and human annotators' judgments of comment entailment, validating the approach.
Key Results
- The best performing seq2seq model (s2s-begin-end) achieved perplexity scores significantly lower than a unimodal language model trained on comments, indicating effective use of code context.
- The model's predictions showed good agreement with human judgments, with entailed examples having lower average perplexity than partly entailed or non-entailed examples.
- Sentences within common Javadoc tags had an average perplexity of around 6-15, while non-Javadoc sentences averaged around 15.58, suggesting Javadoc tags are more easily inferred.
- In a qualitative analysis, the model successfully identified examples of highly redundant comments (low perplexity) and comments that were harder to infer (high perplexity), aligning with human intuition.
Limitations and Future Work
- Definition of Entailment: The authors acknowledge that a fully formal definition of comment entailment is challenging due to the complexities of natural language semantics. Their definition relies on the understanding of a professional programmer.
- Subjectivity in Human Annotation: While striving for objectivity, the human annotation process still involved some subjectivity, particularly in distinguishing between "entailed" and "partly entailed," and some annotators showed disagreement.
- Focus on Method-Level Comments: The current framework is primarily focused on method-level comments. Extending it to class-level or other types of comments is a potential future direction.
- Beyond Redundancy: While CRAIC effectively detects redundant comments, future work could aim to identify other types of uninformative comments with broader coverage and minimal annotation.
- Integration into Workflows: The authors suggest integrating CRAIC into code review processes, perhaps using a heatmap visualization, to provide real-time feedback to developers.
- Automated Comment Generation: The findings suggest the possibility of using deep learning models to automatically generate more informative comments, allowing developers to focus on comments requiring deeper understanding.
Summary updated