AutoPRTitle: A Tool for Automatic Pull Request Title Generation

Ivana Clairine Irsan, Ting Zhang, Ferdian Thung, David Lo, Lingxiao Jiang

TL;DR

AutoPRTitle is a new tool designed to solve the "lazy PR title" problem by automatically generating concise, descriptive titles for GitHub pull requests. By fine-tuning the state-of-the-art BART language model on a massive dataset of real-world contributions, the researchers created a system that outperforms existing baselines by over 24% in accuracy while remaining highly intuitive for developers to use.

What Problem Does This Paper Address?

In collaborative software development, the Pull Request (PR) is the primary gatekeeper for code quality. While much research has focused on automating PR descriptions, the PR title is often overlooked.

A poor title—like "Fixes #145340" or "Update code"—provides zero context to reviewers, forcing them to dig through commits and descriptions just to understand the PR's intent. This adds cognitive load and slows down the development cycle. The authors argue that a high-quality, succinct title is essential for efficient "bug triaging" and merge decisions, yet developers often lack the time or inclination to draft them manually.

Methods

The core of AutoPRTitle is BART (Bidirectional and Auto-Regressive Transformers), a pre-trained sequence-to-sequence model known for its excellence in text summarization.

The researchers treated PR title generation as a one-sentence summarization task. To provide the model with enough context, they concatenated three sources of information into a single input string:

  1. The PR description (if available).
  2. All commit messages within the PR.
  3. The titles of any linked issues (which often contain the "why" behind the changes).

The team fine-tuned BART on a specialized dataset to teach it the specific "dialect" of software engineering titles. They then wrapped this model in a user-friendly web application built with Streamlit and Docker, allowing developers to simply paste a GitHub URL and receive a suggested title via the GitHub API.

Datasets

The study utilized the PRTiger dataset, which the authors introduced in their previous work.

  • Scope: 43,816 pull requests.
  • Source: 495 popular GitHub repositories across diverse programming languages (JavaScript, Python, Java, C, and C++).
  • Selection: The dataset specifically targets high-quality PRs from top-starred and top-forked repositories to ensure the model learns from "gold standard" examples.
  • Split: The data was divided into an 8:1:1 ratio for training, validation, and testing.

Key Insights and Ideas

  • Context is King: A title isn't just a summary of code; it’s a summary of intent. By including linked issue titles, AutoPRTitle captures the problem the developer was trying to solve, not just the changes they made.
  • The Power of Pre-training: Unlike previous specialized models (like PRSummarizer or iTAPE) that use pointer-generator networks, leveraging a massive, general-purpose model like BART provides a much stronger linguistic foundation for generating natural-sounding titles.
  • Tooling over Theory: The authors emphasize that a model is only useful if it fits into a workflow. By providing a web interface that handles the API calls to GitHub automatically, they lower the barrier to adoption for busy engineers.

Key Results

The researchers evaluated AutoPRTitle using both automatic metrics and human feedback:

  • Model Accuracy: Using ROUGE scores (a standard for summarization), BART significantly outperformed baselines. It achieved a ROUGE-1 score of 47.22, which is 24.6% higher than the best-performing baseline (PRSummarizer).
  • Usability: In a manual evaluation with seven experienced developers (each with 4+ years of GitHub experience), the tool received high marks:
  • Ease-of-use: 4.3 / 5.0
    • Usefulness: 4.4 / 5.0
  • Qualitative Impact: The paper demonstrates that AutoPRTitle can turn an uninformative title like "Fixes #145340" into a descriptive one like "fix inactive view for jupyter notebook," saving reviewers significant time.

Limitations and Future Work

While AutoPRTitle is a significant step forward, the authors acknowledge a few areas for growth:

  • Information Sources: The current model relies on text (commits, descriptions, issues). Future iterations could incorporate code comments or the actual code diffs to gain a deeper understanding of the logic changes.
  • Integration: While the web app is functional, the authors suggest that deeper integration (perhaps as a GitHub Action or IDE plugin) would make the tool even more seamless for developers.
  • Dataset Evolution: As coding styles evolve, the model will need to be updated with more recent PR data to remain relevant.

Summary updated

Abstract

With the rise of the pull request mechanism in software development, the quality of pull requests has gained more attention. Prior works focus on improving the quality of pull request descriptions and several approaches have been proposed to automatically generate pull request descriptions. As an essential component of a pull request, pull request titles have not received a similar level of attention. To further facilitate automation in software development and to help developers in drafting high-quality pull request titles, we introduce AutoPRTitle. AutoPRTitle is specifically designed to automatically generate pull request titles. AutoPRTitle can generate a precise and succinct pull request title based on the pull request description, commit messages, and the associated issue titles. AutoPRTitle is built upon a state-of-the-art text summarization model, BART, which has been pre-trained on large-scale English corpora. We further fine-tuned BART in a pull request dataset containing high-quality pull request titles. We implemented AutoPRTitle as a stand-alone web application. We conducted two sets of evaluations: one concerning the model accuracy and the other concerning the tool usability. For model accuracy, BART outperforms the best baseline by 24.6%, 40.5%, and 23.3%, respectively. For tool usability, the evaluators consider our tool as easy-to-use and useful when creating a pull request title of good quality. Source code: https://github.com/soarsmu/Auto-PR-Title Video demo: https://tinyurl.com/AutoPRTitle
BibTeX
@inproceedings{irsan2022autoprtitle,
  author = {Ivana Clairine Irsan and Ting Zhang and Ferdian Thung and David Lo and Lingxiao Jiang},
  title = {{AutoPRTitle: A Tool for Automatic Pull Request Title Generation}},
  booktitle = {ICSME},
  year = {2022},
  doi = {10.1109/icsme55016.2022.00058},
}

Loading related content...

Figure