AUGER: automatically generating review comments with pre-training models

Lingwei Li, Li Yang, Huaxi Jiang, Jun Yan, Tiejian Luo, Zihan Hua, Geng Liang, Chun Zuo

Code review is the "seatbelt" of software engineering—essential for safety, but often a bit of a hassle. While it’s one of the best ways to catch bugs and share knowledge, it’s also a massive time-sink for senior engineers. What if an AI could look at a pull request and offer the same kind of constructive, line-specific feedback a human mentor would?

In their paper presented at ESEC/FSE ’22, researchers Lingwei Li and colleagues introduce AUGER (AUtomatically GEnerating Review comments), a framework that leverages the power of the T5 transformer model to automate the "human" side of code review.

TL;DR

AUGER is a deep-learning framework that automatically generates natural language review comments for specific code changes. By using a specialized T5 model and a novel "line-tagging" approach, it outperforms existing baselines by over 37% in text similarity and produces comments that human-centric metrics deem "useful" nearly 30% of the time.

What Problem Does This Paper Address?

Modern code review is plagued by two main issues: inconsistency and high cost. Reviewers are human; they get tired, have varying levels of expertise, and often take hours (or days) to respond to a pull request.

While we have static analysis tools (like Linters) to catch syntax errors, they can't provide the "nuanced" feedback that a human does—like suggesting a better design pattern or pointing out a missing edge case. Previous attempts at automating this either focused on simply predicting if a comment was needed or tried to automatically fix the code. AUGER fills the gap by generating the natural language explanation of why a piece of code might be problematic.

Methods

The researchers treated code review as a "translation" task: translating a "buggy" code snippet into a "helpful" natural language comment.

  1. Data Preparation & Augmentation: They cleaned raw GitHub data by removing noise (like bot comments or "LGTM" replies) and used data augmentation techniques to expand their dataset nine-fold, ensuring the model had enough variety to learn from.
  2. Review Lines Tagging: This is the "secret sauce." Instead of feeding the model an entire function, they inserted a special <review_tag> at the exact line where a human reviewer had previously suggested a change. This forces the model to focus its "attention" on the specific context of the problem.
  3. Cross Pre-training: They started with a T5 model already pre-trained on Java documentation (CodeTrans). They then performed "Cross Pre-training" using a Masked Language Model (MLM) objective to help the model understand the inner correlations between Java code and English review comments.
  4. Fine-tuning: Finally, the model was fine-tuned to take a code block with a tag and generate a specific review comment.

Datasets

The authors constructed a high-quality dataset from 11 notable Java projects on GitHub, including heavy hitters like ElasticSearch, Apache Flink, and Kafka.

  • Initial Pull Requests: 174,955
  • Final Filtered Dataset: 10,882 code changes where a review comment actually led to a code revision (ensuring the comments were "useful").
  • Training Split: 80% training, 10% validation, 10% testing.

Key Insights and Ideas

  • The Power of the Tag: The <review_tag> acts as a pointer. The authors found that narrowing the scope to the function level and tagging the specific line significantly reduced "deviated comprehension," where the model gets confused by irrelevant parts of the code.
  • Transfer Learning Works: By starting with a model that already "knew" Java (CodeTrans) and then teaching it how humans talk about code, AUGER achieved much higher fluency than models trained from scratch.
  • Semantic vs. Exact Match: The researchers noted that even when AUGER didn't match the human's comment word-for-word (the "Perfect Prediction" metric), it often captured the same intent. For example, if a human said "use a hotfix here," and AUGER said "this should be a separate hotfix," it’s a win for the developer, even if the ROUGE score isn't 100%.

Key Results

AUGER significantly outperformed baselines like LSTM, COPYNET, and CodeBERT across all metrics:

  • ROUGE-L (Text Similarity): Achieved 22.97%, a 37.38% improvement over the best baseline.
  • Perfect Prediction: While still a difficult task, AUGER achieved 4.32%, which was 14 times higher than some existing methods.
  • Efficiency: The model generates a comment in roughly 20 seconds, making it viable for near-instant feedback in a CI/CD pipeline.
  • Human Usefulness: In a manual study of 100 samples, 29% of AUGER's comments were rated as "Useful" (meaning they would trigger a code change) and 22% were "Somewhat Useful."

Limitations and Future Work

  • The "Where" Problem: Currently, AUGER requires a human (or another tool) to place the <review_tag>. It knows what to say, but it doesn't yet autonomously decide where to say it.
  • Language Barrier: The study was limited to Java. While the T5 architecture is language-agnostic, the model would need to be re-trained for Python, C++, or Go.
  • Context Window: Like all transformer models, there is a limit to how much code context it can "see" at once. Very large functions or cross-file dependencies might still baffle the model.

The authors suggest that future work will focus on making the tool "language-agnostic" and integrating it more deeply into professional developer workflows.

Summary updated

Abstract

Code review is one of the best practices as a powerful safeguard for software quality. In practice, senior or highly skilled reviewers inspect source code and provide constructive comments, consider- ing what authors may ignore, for example, some special cases. The collaborative validation between contributors results in code being highly qualified and less chance of bugs. However, since personal knowledge is limited and varies, the efficiency and effectiveness of code review practice are worthy of further improvement. In fact, it still takes a colossal and time-consuming effort to deliver useful review comments. This paper explores a synergy of multiple practical review comments to enhance code review and proposes AUGER (AUtomatically GEnerating Review comments): a review comments generator with pre-training models. We first collect empirical review data from 11 notable Java projects and construct a dataset of 10,882 code changes. By leveraging Text-to-Text Transfer Transformer (T5) models, the framework synthesizes valuable knowledge in the training stage and effectively outperforms baselines by 37.38% in ROUGE-L. 29% of our automatic review comments are considered useful according to prior studies. The inference generates just in 20 seconds and is also open to training further. Moreover, the performance also gets improved when thoroughly analyzed in case study.
BibTeX
@inproceedings{li2022auger,
  author = {Lingwei Li and Li Yang and Huaxi Jiang and Jun Yan and Tiejian Luo and Zihan Hua and Geng Liang and Chun Zuo},
  title = {{AUGER: automatically generating review comments with pre-training models}},
  booktitle = {ESEC/SIGSOFT FSE},
  year = {2022},
  url = {https://openreview.net/forum?id=IWSsM84qeFU},
}

Loading related content...

Figure