CommitBERT: Commit Message Generation Using Pre-Trained Programming Language Model

Tae Hwan JungACL 2021

TL;DR

This paper tackles the challenge of automatically generating commit messages, which are crucial for understanding code changes but often time-consuming for developers to write. The authors introduce CommitBERT, a novel approach that leverages the pre-trained CodeBERT model and a carefully curated dataset of code modifications and their corresponding commit messages to significantly improve generation quality.

What Problem Does This Paper Address?

In software development, commit messages serve as concise natural language summaries of code changes. They are vital for code comprehension, collaboration, and maintaining a clear project history. However, writing effective commit messages requires significant effort and time, leading many developers to neglect this task or produce low-quality summaries. While automatic commit message generation has been an active research area, existing methods often struggle with:

  • Lack of curated datasets: Previous work often relied on limited or noisy datasets, especially across multiple programming languages.
  • Contextual representation gap: Bridging the semantic gap between programming language (PL) code and natural language (NL) descriptions has been a persistent challenge. Models often failed to capture the nuanced relationship between code modifications and their textual explanations.
  • Inefficient input representation: Treating entire git diff outputs as input can be noisy and less effective than focusing on the actual code additions and deletions.

This paper aims to address these limitations by proposing a more robust dataset and a model architecture that better captures the PL-NL relationship for commit message generation.

Methods

The authors propose a two-pronged approach:

  1. Dataset Curation: They collected a large-scale dataset of 345,000 code modification and commit message pairs from 52,000 GitHub repositories across six programming languages (Python, PHP, Go, Java, JavaScript, and Ruby). Crucially, this dataset focuses on the added and deleted lines of code from git diff, rather than the entire diff, to reduce noise and improve focus. This dataset is made publicly available.
  2. CommitBERT Model Architecture:
    • Encoder-Decoder Framework: They adopt a standard encoder-decoder architecture, similar to Neural Machine Translation (NMT) models.
  3. CodeBERT as Initial Model: To bridge the PL-NL gap, they leverage CodeBERT, a pre-trained language model specifically designed for code. This provides a strong initial representation of code.
  4. Input Representation: For the encoder, they concatenate added and deleted code tokens, separated by special [SEP] tokens. This structured input aims to provide a clearer representation of code changes.
  5. Pre-training Enhancement: They further fine-tune CodeBERT on a "Code-to-NL" task (summarizing code snippets into natural language) before using it as the initial weight for commit message generation. This step is hypothesized to further reduce the PL-NL gap.

Datasets

  • Newly Introduced Dataset: The paper introduces a new dataset of 345,000 code modification (added/deleted lines) and commit message pairs, collected from 52,000 GitHub repositories across six programming languages. This dataset is a significant contribution, offering a more focused and multilingual resource compared to previous efforts.
  • Existing Datasets for Evaluation: For specific experiments, they also utilize data from Loyola et al. (2017) and CodeSearchNet.

Key Insights and Ideas

  • Focusing on Changed Lines is Key: The most significant insight is that using only the added and deleted lines of code as input, rather than the entire git diff, leads to substantially better commit message generation performance. This suggests that extraneous information in the diff can actually hinder the model's learning process.
  • Pre-trained Models are Essential: Leveraging pre-trained models like CodeBERT is crucial for bridging the PL-NL gap. Simply training from scratch on this task is insufficient.
  • Code-to-NL Pre-training Boosts Performance: Further pre-training CodeBERT on a related task like "Code-to-NL" summarization before fine-tuning for commit message generation yields even better results, indicating the value of task-specific pre-training.
  • Multilingualism Matters: The dataset and model are designed to handle multiple programming languages, addressing a limitation of prior work that often focused on a single language.

Key Results

  • Improved BLEU-4 Scores: CommitBERT, using only changed lines as input and initialized with CodeBERT pre-trained on Code-to-NL, achieved the highest BLEU-4 scores across various languages compared to other initial weights (Random, RoBERTa, CodeBERT). For instance, in Table 2, the "CodeBERT + Code-to-NL" setting consistently outperforms others.
  • Input Type Matters: Table 3 demonstrates that using only changed lines (Ours) significantly outperforms using all code modifications for both RoBERTa and CodeBERT initial weights, with BLEU-4 scores increasing from 10.91 to 12.52 (RoBERTa) and 11.77 to 13.32 (CodeBERT).
  • Reduced Perplexity (PPL): The "CodeBERT + Code-to-NL" initialization also resulted in the lowest perplexity (PPL) on the validation set, indicating a better fit of the model to the data.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while their approach improves performance, there's still a gap in contextual representation between PL and NL.
  • The current model doesn't explicitly understand code syntax structure.
  • Future Work:
    • The authors suggest converting the model to use Abstract Syntax Trees (AST) before feeding code modifications to the encoder. This would allow the model to better grasp the structural aspects of code.
  • Further research into understanding code syntax structure is highlighted as a promising direction.

Summary updated

BibTeX
@inproceedings{jung2021commitbert,
  author = {Tae Hwan Jung},
  title = {{CommitBERT: Commit Message Generation Using Pre-Trained Programming Language Model}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure