AutoPRTitle: A Tool for Automatic Pull Request Title Generation

Ivana Clairine Irsan, Ting Zhang, Ferdian Thung, David Lo, Lingxiao Jiang

Abstract

With the rise of the pull request mechanism in software development, the quality of pull requests has gained more attention. Prior works focus on improving the quality of pull request descriptions and several approaches have been proposed to automatically generate pull request descriptions. As an essential component of a pull request, pull request titles have not received a similar level of attention. To further facilitate automation in software development and to help developers in drafting high-quality pull request titles, we introduce AutoPRTitle. AutoPRTitle is specifically designed to automatically generate pull request titles. AutoPRTitle can generate a precise and succinct pull request title based on the pull request description, commit messages, and the associated issue titles. AutoPRTitle is built upon a state-of-the-art text summarization model, BART, which has been pre-trained on large-scale English corpora. We further fine-tuned BART in a pull request dataset containing high-quality pull request titles. We implemented AutoPRTitle as a stand-alone web application. We conducted two sets of evaluations: one concerning the model accuracy and the other concerning the tool usability. For model accuracy, BART outperforms the best baseline by 24.6%, 40.5%, and 23.3%, respectively. For tool usability, the evaluators consider our tool as easy-to-use and useful when creating a pull request title of good quality.

Introduction

In collaborative software development, the Pull Request (PR) is the primary gatekeeper for code quality. While significant research has focused on automating PR descriptions, the PR title is often overlooked. A poor title—such as "Fixes #145340" or "Update code"—provides zero context to reviewers, forcing them to navigate through commits and descriptions just to understand the PR's intent. This adds cognitive load and slows the development cycle.

A high-quality, succinct title is essential for efficient bug triaging and merge decisions. However, developers often lack the time or inclination to draft them manually. AutoPRTitle addresses this "lazy PR title" problem by automatically generating concise, descriptive titles that capture the intent of the changes.

Methodology

The core of AutoPRTitle is BART (Bidirectional and Auto-Regressive Transformers), a pre-trained sequence-to-sequence model known for its excellence in text summarization. The researchers treated PR title generation as a one-sentence summarization task.

To provide the model with sufficient context, the system concatenates three sources of information into a single input string:

  1. The PR description (if available).
  2. All commit messages within the PR.
  3. The titles of any linked issues (which often contain the "why" behind the changes).
The data flow and processing steps of AutoPRTitle, illustrating how various sources are concatenated to generate a title.

The model was fine-tuned on a specialized dataset to learn the specific "dialect" of software engineering titles. The final system is implemented as a web application built with Streamlit and Docker, allowing developers to paste a GitHub URL and receive a suggested title via the GitHub API.

Datasets and Experimental Setup

The study utilized the PRTiger dataset, which includes 43,816 pull requests from 495 popular GitHub repositories across diverse programming languages including JavaScript, Python, Java, C, and C++. The dataset specifically targets high-quality PRs from top-starred and top-forked repositories to ensure the model learns from "gold standard" examples. The data was divided into an 8:1:1 ratio for training, validation, and testing.

Key Results

The researchers evaluated AutoPRTitle using both automatic metrics and human feedback.

Model Accuracy

Using ROUGE scores (a standard for summarization), BART significantly outperformed existing baselines such as PRSummarizer and iTAPE. It achieved a ROUGE-1 score of 47.22, which is 24.6% higher than the best-performing baseline.

Quantitative comparison of AutoPRTitle (BART) against baseline models using ROUGE metrics.

Usability and Qualitative Impact

In a manual evaluation with seven experienced developers (each with 4+ years of GitHub experience), the tool received high marks for ease-of-use (4.3/5.0) and usefulness (4.4/5.0). Qualitatively, the tool demonstrated the ability to transform uninformative titles like "Fixes #145340" into descriptive ones like "fix inactive view for jupyter notebook," significantly reducing the time required for reviewers to understand the PR's purpose.

Conclusion and Future Work

AutoPRTitle represents a significant step forward in automating software development documentation. By leveraging the power of pre-trained models like BART and incorporating multi-source context (commits, descriptions, and issues), it provides a practical solution for improving PR quality.

Future work will explore incorporating code diffs and comments directly into the model to gain a deeper understanding of logic changes. Additionally, the authors suggest that deeper integration, such as a GitHub Action or IDE plugin, would further lower the barrier to adoption for busy engineers.

Summary updated

Abstract

With the rise of the pull request mechanism in software development, the quality of pull requests has gained more attention. Prior works focus on improving the quality of pull request descriptions and several approaches have been proposed to automatically generate pull request descriptions. As an essential component of a pull request, pull request titles have not received a similar level of attention. To further facilitate automation in software development and to help developers in drafting high-quality pull request titles, we introduce AutoPRTitle. AutoPRTitle is specifically designed to automatically generate pull request titles. AutoPRTitle can generate a precise and succinct pull request title based on the pull request description, commit messages, and the associated issue titles. AutoPRTitle is built upon a state-of-the-art text summarization model, BART, which has been pre-trained on large-scale English corpora. We further fine-tuned BART in a pull request dataset containing high-quality pull request titles. We implemented AutoPRTitle as a stand-alone web application. We conducted two sets of evaluations: one concerning the model accuracy and the other concerning the tool usability. For model accuracy, BART outperforms the best baseline by 24.6%, 40.5%, and 23.3%, respectively. For tool usability, the evaluators consider our tool as easy-to-use and useful when creating a pull request title of good quality. Source code: https://github.com/soarsmu/Auto-PR-Title Video demo: https://tinyurl.com/AutoPRTitle
BibTeX
@inproceedings{irsan2022autoprtitle,
  author = {Ivana Clairine Irsan and Ting Zhang and Ferdian Thung and David Lo and Lingxiao Jiang},
  title = {{AutoPRTitle: A Tool for Automatic Pull Request Title Generation}},
  booktitle = {ICSME},
  year = {2022},
  doi = {10.1109/icsme55016.2022.00058},
}
Figure