Learning to Update Natural Language Comments Based on Code Changes

Sheena Panthaplackel, Pengyu Nie, Milos Gligoric, Junyi Jessy Li, Raymond J. Mooney

Abstract

We formulate the novel task of automatically updating an existing natural language comment based on changes in the body of code it accompanies. We propose an approach that learns to correlate changes across two distinct language representations, to generate a sequence of edits that are applied to the existing comment to reflect the source code modifications. We train and evaluate our model using a dataset that we collected from commit histories of open-source software projects, with each example consisting of a concurrent update to a method and its corresponding comment. We compare our approach against multiple baselines using both automatic metrics and human evaluation. Results reflect the challenge of this task and that our model outperforms baselines with respect to making edits.

Introduction

Software documentation frequently suffers from "rotting," where natural language comments become obsolete as the underlying source code evolves. This inconsistency creates significant technical debt, leading to developer confusion and potential bugs. While previous research has focused on detecting these inconsistencies or generating comments from scratch, neither approach effectively maintains existing documentation. Generating a new comment often discards valuable, manually written context that should be preserved.

Example of a code change requiring a comment update to maintain synchronization.

This paper addresses the challenge of automatically propagating specific code changes into their corresponding natural language comments, ensuring that documentation remains in sync with the implementation.

Methodology

The authors treat comment updating as a sequence-to-sequence problem but shift the focus from raw text generation to the generation of edit actions (insert, delete, replace).

Architecture

The proposed model utilizes a two-encoder, one-decoder GRU-based architecture:

  • Code Edit Encoder: Represents the code change as a "diff" sequence ($M_{edit}$) that explicitly identifies added or removed tokens.
  • Comment Encoder: Encodes the existing comment ($C_{old}$).
  • Edit Decoder: Employs a pointer network to output a sequence of edit actions, allowing the model to either generate new tokens or copy from the input.
The model architecture featuring dual encoders for code edits and existing comments, feeding into an edit sequence decoder.

To ensure the updates are placed correctly, the model uses a "Modified Comment Edit Lexicon" to bind edits to specific "anchor" words in the original text. Furthermore, a reranking scheme is applied during beam search to balance the edit score with the naturalness of the resulting comment.

Dataset and Experimental Setup

The researchers constructed a specialized dataset by mining commit histories from open-source Java projects on GitHub. They focused on @return Javadoc tags due to their structured relationship with method return statements. The final dataset includes 7,239 examples of concurrent code and comment updates across over 1,000 projects. Heuristics were applied to filter out purely stylistic changes, ensuring the model learns meaningful semantic updates.

Results and Analysis

The "Edit" model was evaluated against rule-based baselines and large-scale generation models using metrics such as SARI (measuring word additions, deletions, and retentions) and Exact Match (xMatch).

  • Quantitative Performance: The Edit model achieved an 18.4% exact match rate, significantly higher than the 1.1% achieved by standard generation models. Its SARI score of 45.5 more than doubled the generation baseline's score of 21.2.
  • Human Evaluation: In a developer study, the reranked Edit model's suggestions were preferred 30.2% of the time, compared to 12.4% for the generation model.

Key insights revealed that explicit "diff" sequences are crucial; the model performs better when it can explicitly "see" the change in the code rather than just comparing old and new versions side-by-side.

Conclusion and Future Work

While the edit-based approach represents a significant advancement over traditional generation, the task remains challenging. Human evaluators selected "None" (indicating no suggestion was adequate) 55% of the time, highlighting the difficulty of capturing external knowledge not present in the immediate method body. Future work will explore incorporating broader code context and utilizing more sophisticated structural encoders like Tree-LSTMs or Graph Neural Networks to further improve update accuracy.

Summary updated

Abstract

We formulate the novel task of automatically updating an existing natural language comment based on changes in the body of code it accompanies. We propose an approach that learns to correlate changes across two distinct language representations, to generate a sequence of edits that are applied to the existing comment to reflect the source code modifications. We train and evaluate our model using a dataset that we collected from commit histories of open-source software projects, with each example consisting of a concurrent update to a method and its corresponding comment. We compare our approach against multiple baselines using both automatic metrics and human evaluation. Results reflect the challenge of this task and that our model outperforms baselines with respect to making edits.
BibTeX
@inproceedings{panthaplackel2020learning,
  author = {Sheena Panthaplackel and Pengyu Nie and Milos Gligoric and Junyi Jessy Li and Raymond J. Mooney},
  title = {{Learning to Update Natural Language Comments Based on Code Changes}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=TEj8i2WEEPW},
}
Figure