AUGER: Automatically Generating Review Comments with Pre-training Models

Lingwei Li, Li Yang, Huaxi Jiang, Jun Yan, Tiejian Luo, Zihan Hua, Geng Liang, Chun Zuo

Abstract

Code review is one of the best practices as a powerful safeguard for software quality. In practice, senior or highly skilled reviewers inspect source code and provide constructive comments, considering what authors may ignore, for example, some special cases. The collaborative validation between contributors results in code being highly qualified and less chance of bugs. However, since personal knowledge is limited and varies, the efficiency and effectiveness of code review practice are worthy of further improvement. In fact, it still takes a colossal and time-consuming effort to deliver useful review comments. This paper explores a synergy of multiple practical review comments to enhance code review and proposes AUGER (AUtomatically GEnerating Review comments): a review comments generator with pre-training models. We first collect empirical review data from 11 notable Java projects and construct a dataset of 10,882 code changes. By leveraging Text-to-Text Transfer Transformer (T5) models, the framework synthesizes valuable knowledge in the training stage and effectively outperforms baselines by 37.38% in ROUGE-L. 29% of our automatic review comments are considered useful according to prior studies. The inference generates just in 20 seconds and is also open to training further. Moreover, the performance also gets improved when thoroughly analyzed in case study.

Introduction

Modern software engineering relies heavily on code review to maintain high standards of quality and security. However, this process is often bottlenecked by human limitations: inconsistency in reviewer expertise and the significant time investment required from senior engineers. While static analysis tools effectively catch syntax errors, they lack the nuance to provide constructive feedback on design patterns or specific edge cases.

Figure 1: Mapping GitHub pull request data to structured code changes and review comments.

To address these challenges, we introduce AUGER (AUtomatically GEnerating Review comments), a deep-learning framework designed to automate the generation of natural language review comments. By treating code review as a translation task—converting problematic code snippets into helpful natural language explanations—AUGER aims to provide near-instant, human-like feedback to developers.

Methodology

The AUGER framework consists of four primary stages: data preparation, review line tagging, cross pre-training, and fine-tuning for comment generation.

Figure 2: The AUGER framework architecture, including data preparation, line tagging, and T5-based generation.

1. Data Preparation and Augmentation

Raw data is fetched from GitHub and undergoes rigorous cleaning to remove noise such as bot comments or simple "LGTM" (Looks Good To Me) replies. To improve model robustness, data augmentation techniques are applied, expanding the initial dataset nine-fold.

2. Review Lines Tagging

A critical innovation in AUGER is the "Review Line Tagging" approach. Instead of processing entire files, the system inserts a special <review_tag> at the specific line where a human reviewer previously suggested a change. This focuses the model's attention on the relevant context, reducing "deviated comprehension" where irrelevant code segments might confuse the generator.

3. Cross Pre-training

AUGER utilizes the Text-to-Text Transfer Transformer (T5) architecture. We begin with a model pre-trained on Java documentation (CodeTrans) and perform "Cross Pre-training" using a Masked Language Model (MLM) objective. This step helps the model learn the semantic correlations between Java source code and English review comments.

4. Fine-tuning

The final stage involves fine-tuning the model to take a tagged code block as input and generate a specific, context-aware review comment as output.

Experimental Setup

The framework was evaluated using a high-quality dataset constructed from 11 prominent Java projects on GitHub, including ElasticSearch, Apache Flink, and Kafka. Out of 174,955 initial pull requests, we filtered for 10,882 code changes where a review comment directly led to a code revision, ensuring the training data consisted of "useful" feedback. The dataset was split into 80% training, 10% validation, and 10% testing sets.

Results and Discussion

AUGER significantly outperforms existing baselines, including LSTM, COPYNET, and CodeBERT.

  • Text Similarity: AUGER achieved a ROUGE-L score of 22.97%, representing a 37.38% improvement over the best-performing baseline.
  • Perfect Prediction: The model achieved a 4.32% perfect match rate, which is 14 times higher than previous state-of-the-art methods.
  • Efficiency: Inference is highly efficient, generating comments in approximately 20 seconds, making it suitable for integration into CI/CD pipelines.
  • Human Usefulness: In a manual evaluation of 100 samples, 29% of AUGER's comments were rated as "Useful" (likely to trigger a code change), and an additional 22% were deemed "Somewhat Useful."

Conclusion and Future Work

AUGER demonstrates the potential of pre-trained transformer models to automate the nuanced task of code review. By focusing on specific code lines and leveraging transfer learning, it provides constructive feedback that mirrors human intent. Future work will focus on making AUGER language-agnostic beyond Java and developing autonomous methods for the model to identify where a comment is needed without manual tagging.

Summary updated

Abstract

Code review is one of the best practices as a powerful safeguard for software quality. In practice, senior or highly skilled reviewers inspect source code and provide constructive comments, considering what authors may ignore, for example, some special cases. The collaborative validation between contributors results in code being highly qualified and less chance of bugs. However, since personal knowledge is limited and varies, the efficiency and effectiveness of code review practice are worthy of further improvement. In fact, it still takes a colossal and time-consuming effort to deliver useful review comments. This paper explores a synergy of multiple practical review comments to enhance code review and proposes AUGER (AUtomatically GEnerating Review comments): a review comments generator with pre-training models. We first collect empirical review data from 11 notable Java projects and construct a dataset of 10,882 code changes. By leveraging Text-to-Text Transfer Transformer (T5) models, the framework synthesizes valuable knowledge in the training stage and effectively outperforms baselines by 37.38% in ROUGE-L. 29% of our automatic review comments are considered useful according to prior studies. The inference generates just in 20 seconds and is also open to training further. Moreover, the performance also gets improved when thoroughly analyzed in case study.
BibTeX
@inproceedings{li2022auger,
  author = {Lingwei Li and Li Yang and Huaxi Jiang and Jun Yan and Tiejian Luo and Zihan Hua and Geng Liang and Chun Zuo},
  title = {{AUGER: Automatically Generating Review Comments with Pre-training Models}},
  booktitle = {ESEC/SIGSOFT FSE},
  year = {2022},
  url = {https://openreview.net/forum?id=IWSsM84qeFU},
}
Figure