Abstract
We propose a model to automatically describe changes introduced in the source code of a program using natural language. Our method receives as input a set of code commits, which contains both the modifications and message introduced by an user. These two modalities are used to train an encoder-decoder architecture. We evaluated our approach on twelve real world open source projects from four different programming languages. Quantitative and qualitative results showed that the proposed approach can generate feasible and semantically sound descriptions not only in standard in-project settings, but also in a cross-project setting.
Introduction
Software development is an inherently incremental process. To maintain a clear understanding of a system's evolution, developers rely on commit messages—short natural language descriptions of code changes. However, writing high-quality messages is time-consuming. Consequently, many projects suffer from descriptions that are cryptic, inconsistent, or missing entirely.
While previous research has focused on static code summarization, this work addresses the dynamic challenge of code change summarization. The goal is to build a tool that automatically explains what was modified and why, easing the documentation burden and helping developers navigate complex version histories.
Methodology
The problem is framed as a sequence-to-sequence task, analogous to Neural Machine Translation (NMT). The proposed system utilizes an encoder-decoder architecture with a global attention mechanism.
- Encoder: A lookup layer that processes the sequence of tokens in a code "diff" (the lines added or removed).
- Decoder: An LSTM (Long Short-Term Memory) network that generates the natural language description word by word.
- Attention Mechanism: Given that code changes can be long and heterogeneous, attention allows the model to focus on the most "expressive" parts of the code snippet when generating specific words in the description.
- Atomicity: To simplify initial learning, the study focuses on "atomic" commits—those modifying only a single file—ensuring a direct relationship between code changes and the message.
- Inference: At test time, the model employs beam search (size 10) to identify the most likely sequence of words.
Experimental Setup
The model was evaluated using a diverse dataset crawled from 12 popular open-source GitHub projects across four programming languages:
- Python: Theano, Keras, youtube-dl
- JavaScript: Node.js, Angular, React
- C++: OpenCV, CNTK, Bitcoin
- Java: CoreNLP, Elasticsearch, Guava
The resulting dataset comprised over 150,000 commits, including full histories and diff files.
Key Results and Discussion
The model's performance was measured using BLEU-4 and METEOR scores, standard metrics in machine translation.
Quantitative Performance
The neural model outperformed the baseline (MOSES, a phrase-based statistical MT system) in 11 out of 12 projects. On average, the neural approach provided a gain of 5 BLEU points over the baseline, with the most significant improvements observed in Java projects like CoreNLP and Guava.
Qualitative Success
In many instances, the model generated descriptions that were more concise or clearer than the original human-written messages. For example, a complex Theano commit was correctly rephrased as "Better warning message."
Insights
- Code as Language: The work supports the "naturalness of software" hypothesis, suggesting code follows distributional regularities similar to natural language.
- Cross-Project Generalization: Models trained on one set of projects could generate semantically sound descriptions for entirely different projects, indicating that the "language" of code changes transcends specific codebases.
- The "Rigidity" Trap: In projects like Node.js where commit messages follow rigid patterns, the model tended to memorize templates rather than learning logic, which can hinder generalization.
Limitations and Future Work
Current limitations include structural blindness, as the model treats code as a flat sequence rather than a hierarchical structure. Future iterations could incorporate Abstract Syntax Trees (ASTs). Additionally, the model lacks broader context, such as project architecture or bug tracker issues. The authors suggest that incorporating functional data, like execution traces or test results, could further improve the model's understanding of a change's impact.
Summary updated