A Neural Architecture for Generating Natural Language Descriptions from Source Code Changes

Pablo Loyola, Edison Marrese-Taylor, Yutaka Matsuo

Abstract

We propose a model to automatically describe changes introduced in the source code of a program using natural language. Our method receives as input a set of code commits, which contains both the modifications and message introduced by an user. These two modalities are used to train an encoder-decoder architecture. We evaluated our approach on twelve real world open source projects from four different programming languages. Quantitative and qualitative results showed that the proposed approach can generate feasible and semantically sound descriptions not only in standard in-project settings, but also in a cross-project setting.

1. Introduction

Software development is an evolutionary process characterized by a continuous stream of incremental changes. Understanding the rationale behind these changes—the "why" of a code diff—is a critical but time-consuming task for developers. While static code summarization is well-established, it often fails to capture the dynamic intent of a modification. Currently, documentation relies on human-written commit messages, which are frequently inconsistent, vague, or missing entirely. This paper addresses the challenge of automating this documentation by proposing a neural architecture capable of generating semantically sound natural language summaries from source code modifications.

2. Methodology

The authors frame the generation of commit messages as a sequence-to-sequence (Seq2Seq) task, analogous to neural machine translation. This approach leverages the "naturalness" hypothesis, which posits that source code follows similar distributional regularities to natural language.

2.1 Architecture

The core model is an encoder-decoder architecture enhanced with a global attention mechanism.

  • Encoder: A lookup layer processes the sequence of tokens within a code change (the diff).
  • Decoder: A dropout-regularized Long Short-Term Memory (LSTM) network generates natural language descriptions word by word.
  • Attention Mechanism: Given the structural divergence between code and natural language, the attention mechanism allows the decoder to focus on specific, relevant segments of the code change during the generation process. This allows the model to map specific code tokens (like add or variable names) to meaningful words in the summary.

2.2 Inference

To optimize the generated descriptions, the model employs beam search with a size of 10, exploring multiple candidate word sequences to identify the one with the highest overall probability.

3. Experimental Setup

The model was evaluated using a curated dataset from 12 prominent open-source GitHub projects across four programming languages: Python (Theano, Keras, youtube-dl), JavaScript (Node.js, Angular, React), C++ (OpenCV, CNTK, Bitcoin), and Java (CoreNLP, Elasticsearch, Guava). The data was categorized into "Atomic" commits (single-file changes) and "Full" commits (multi-file changes). A "Uni-action" subset of Python commits was also used for cross-project evaluation to test the model's ability to generalize beyond project-specific jargon.

4. Results and Evaluation

The neural approach was compared against MOSES, a standard phrase-based statistical machine translation baseline.

4.1 Quantitative Results

The neural model demonstrated superior performance, outperforming the MOSES baseline in 11 out of 12 projects. On average, the model achieved a gain of approximately 5 BLEU points. Notably, performance was strongest on Java projects like CoreNLP and Guava, likely due to the increased context provided by longer Java diffs which the encoder can better digest.

Quantitative comparison of the neural model vs. MOSES baseline across projects.

4.2 Qualitative Results and Insights

Qualitative analysis confirms that the model can generate semantically sound descriptions. In projects with diverse commit styles, the model exhibited an "emergence of rephrasing capabilities," often selecting general, appropriate terms over overly specific ones. This suggests the model is learning the logic of changes rather than just memorizing patterns.

Qualitative examples comparing reference commit messages with model-generated descriptions.

5. Discussion and Limitations

A significant finding is the model's feasibility in cross-project settings. In Python experiments, it achieved BLEU scores of 14.6 for additions and 18.9 for removals, indicating it learns the underlying logic of changes. However, several limitations remain:

  • Sequence Length: The model is constrained to 100 tokens, making large "megadiffs" difficult to process.
  • Structural Understanding: Treating code as a flat sequence ignores hierarchical structures like Abstract Syntax Trees (ASTs).
  • Contextual Awareness: The model lacks access to broader project context or execution traces, which are often necessary to fully grasp the intent of a change.
  • Vocabulary: Handling rare or unique variable names (represented as UNK tokens) remains a challenge in software engineering contexts.

6. Conclusion

This research demonstrates that neural encoder-decoder architectures with attention are a promising path for automating code change documentation. By treating diffs as a translation problem, the model generates summaries that are not only accurate within a project but also generalize across different software ecosystems, paving the way for more consistent and automated software maintenance.

Summary updated

Abstract

We propose a model to automatically describe changes introduced in the source code of a program using natural language. Our method receives as input a set of code commits, which contains both the modifications and message introduced by an user. These two modalities are used to train an encoder-decoder architecture. We evaluated our approach on twelve real world open source projects from four different programming languages. Quantitative and qualitative results showed that the proposed approach can generate feasible and semantically sound descriptions not only in standard in-project settings, but also in a cross-project setting.
BibTeX
@inproceedings{loyola2017neural,
  author = {Pablo Loyola and Edison Marrese-Taylor and Yutaka Matsuo},
  title = {{A Neural Architecture for Generating Natural Language Descriptions from Source Code Changes}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2017},
  url = {https://openreview.net/forum?id=rJWbxag_Zr},
}
Figure