Learning to Update Natural Language Comments Based on Code Changes

Sheena Panthaplackel, Pengyu Nie, Milos Gligoric, Junyi Jessy Li, Raymond J. Mooney

Abstract

We formulate the novel task of automatically updating an existing natural language comment based on changes in the body of code it accompanies. We propose an approach that learns to correlate changes across two distinct language representations, to generate a sequence of edits that are applied to the existing comment to reflect the source code modifications. We train and evaluate our model using a dataset that we collected from commit histories of open-source software projects, with each example consisting of a concurrent update to a method and its corresponding comment. We compare our approach against multiple baselines using both automatic metrics and human evaluation. Results reflect the challenge of this task and that our model outperforms baselines with respect to making edits.

Introduction

Software maintenance is a constant cycle of refactoring and feature additions. While developers are diligent about updating source code, they often neglect the accompanying comments. This inconsistency—known as "comment rot"—leads to time-wasting confusion for other developers and can even mask logic bugs.

Existing research has focused on detecting when a comment is out of date or generating a brand-new comment from a code snippet. However, generating a comment from scratch often discards the specific stylistic nuances and salient information present in the original documentation. This paper introduces the first automated approach to update existing comments based on code changes, using a novel encoder-decoder model that learns to generate specific edit actions rather than rewriting comments from scratch.

Methodology

The authors propose an Edit Model based on a dual-encoder-decoder architecture designed to mimic how humans work: identifying what changed in the code and applying surgical edits to the existing comment.

Model Architecture

The model uses two separate Gated Recurrent Unit (GRU) encoders. One encodes the code change (represented as a "diff" sequence of old and new code), and the other encodes the existing comment.

Architecture of the proposed Edit Model featuring dual encoders and an edit sequence decoder.

Instead of predicting words directly, the decoder predicts a sequence of edit actions: Keep, Insert, Delete, and Replace. This forces the model to focus on how to change the text rather than just what the text should be. To help the model bridge the gap between code and natural language, lexical features like Java keywords, Part-of-Speech (POS) tags, and "anchor" tokens are incorporated to help the model locate exactly where an edit should occur.

Reranking and Dataset

After generating candidate updates, the system uses a reranker to score candidates based on model confidence, generation likelihood (ensuring fluency), and similarity to the original comment to encourage minimal, precise edits. The researchers constructed a new dataset by mining GitHub commit histories from popular open-source Java projects, focusing specifically on @return Javadoc tags. The final dataset consists of 7,239 examples where both a method's body and its comment were updated in the same commit.

Results

The Edit Model was compared against several baselines, including a rule-based "Return type substitution" and a standard "Generation" model trained on over 100,000 comment pairs.

Quantitative Evaluation

The Reranked Edit Model achieved a SARI score of 45.48 and a GLEU score of 46.11, significantly outperforming the Generation model (SARI: 21.16, GLEU: 17.35). SARI and GLEU are specifically designed to measure the quality of text edits.

Performance comparison of the Edit Model against baselines across various automatic metrics.

Human Evaluation

In a user study with Java developers, the Edit Model's suggestions were preferred 30.2% of the time, compared to only 12.4% for the Generation model and 18.4% for the rule-based baseline. However, users selected "None of the above" 55% of the time, revealing that many code changes are internal refactorings that don't actually require a comment update.

Discussion and Conclusion

The core insight of this work is that treating comment updating as a sequence-to-edit task is more effective than a standard sequence-to-sequence task. By predicting edits, the model naturally preserves the parts of the comment that remain valid.

Despite these gains, limitations exist. The model struggles when an update requires knowledge outside the immediate method context. Furthermore, when a method is completely rewritten, the "Edit" approach can become cluttered, suggesting that a hybrid approach might be necessary. Future work could expand this to other types of documentation, such as method descriptions or parameter tags, and include a classifier to decide if an update is needed at all.

Summary updated

Abstract

We formulate the novel task of automatically updating an existing natural language comment based on changes in the body of code it accompanies. We propose an approach that learns to correlate changes across two distinct language representations, to generate a sequence of edits that are applied to the existing comment to reflect the source code modifications. We train and evaluate our model using a dataset that we collected from commit histories of open-source software projects, with each example consisting of a concurrent update to a method and its corresponding comment. We compare our approach against multiple baselines using both automatic metrics and human evaluation. Results reflect the challenge of this task and that our model outperforms baselines with respect to making edits.
BibTeX
@inproceedings{panthaplackel2020learning,
  author = {Sheena Panthaplackel and Pengyu Nie and Milos Gligoric and Junyi Jessy Li and Raymond J. Mooney},
  title = {{Learning to Update Natural Language Comments Based on Code Changes}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=TEj8i2WEEPW},
}
Figure