DeepRelease: Language-agnostic Release Notes Generation from Pull Requests of Open-source Software

Huaxi Jiang, Jie Zhu, Li Yang, Geng Liang, Chun Zuo

Abstract

The release note is an essential software artifact of open-source software that documents crucial information about changes, such as new features and bug fixes. With the help of release notes, both developers and users could have a general understanding of the latest version without browsing the source code. However, it is a daunting and time-consuming job for developers to produce release notes. Although prior studies have provided some automatic approaches, they generate release notes mainly by extracting information from code changes. This will result in language-specific and not being general enough to be applicable. Therefore, helping developers produce release notes effectively remains an unsolved challenge. To address the problem, we first conduct a manual study on the release notes of 900 GitHub projects, which reveals that more than 54% of projects produce their release notes with pull requests. Based on the empirical finding, we propose a deep learning based approach named DeepRelease (Deep learning based Release notes generator) to generate release notes according to pull requests. The process of release notes generation in DeepRelease includes the change entries generation and the change category (i.e., new features or bug fixes) generation, which are formulated as a text summarization task and a multi-class classification problem, respectively. Since DeepRelease fully employs text information from pull requests to summarize changes and identify the change category, it is language-agnostic and can be used for projects in any language. We build a dataset with over 46K release notes and evaluate DeepRelease on the dataset. The experimental results indicate that DeepRelease outperforms four baselines and can generate release notes similar to those manually written ones in a fraction of the time.

Introduction

Release notes are critical for communicating software changes, such as new features and bug fixes, to both developers and users. However, manually producing these notes is a "daunting and time-consuming" task, often requiring hours of effort per release. Existing automated approaches primarily extract information from source code changes (e.g., AST diffs), which introduces language dependency and often misses the high-level "why" behind a change.

DeepRelease addresses these challenges by shifting the focus from source code to natural language descriptions found in GitHub Pull Requests (PRs). An empirical study of 900 GitHub projects revealed that over 54% of projects already utilize PRs to generate their release notes. By leveraging this existing developer practice, DeepRelease provides a language-agnostic solution that captures semantic intent.

Methodology

DeepRelease employs a multi-stage deep learning pipeline to transform raw PR data into formatted Markdown release notes. The architecture consists of four primary modules:

  1. Pull Request Collector: Gathers PR titles, descriptions, and commit messages via the GitHub API.
  2. Entry Summarizer: Uses a Pointer-Generator Network to generate descriptive text. This model combines a Sequence-to-Sequence (Seq2Seq) architecture with an attention mechanism, allowing it to either generate new words or "copy" specific tokens (like function names or CVE IDs) from the input.
  3. Category Discriminator: A FastText-based classifier that assigns each PR to one of four categories: Issues fixed, New features, Non-functional requirements, or Documentation.
  4. Text Generator: Formats the summarized entries and categories into a final Markdown document.
The overall architecture of the DeepRelease pipeline, showing the flow from GitHub data collection to final release note generation.

Dataset and Language Agnosticism

To train and evaluate the model, the authors curated a dataset of 46,656 release note entries from 400 non-trivial GitHub repositories. A key strength of DeepRelease is its language-agnostic nature. Because the model processes natural language descriptions rather than source code, it is applicable to projects written in any programming language. The dataset reflects this diversity, spanning nine major languages.

Distribution of projects across nine programming languages in the DeepRelease dataset, demonstrating its language-agnostic capabilities.

Experimental Results

DeepRelease was evaluated against several baselines, including LeadCM and PR-Title.

  • Summarization Performance: DeepRelease achieved a ROUGE-L score of 73.51, outperforming the PR-Title baseline. The model's ability to condense and rephrase information resulted in more readable and professional notes compared to raw titles.
  • Classification Accuracy: The Category Discriminator reached F1 scores of up to 88%, significantly more accurate than simple keyword-based heuristics.
  • Efficiency: The system is highly efficient, generating release note entries in an average of 29.7 milliseconds per entry on modern hardware.

Conclusion and Future Work

DeepRelease demonstrates that Pull Requests are a rich source of semantic information for automating software documentation. By focusing on text rather than code, it achieves high performance across diverse programming ecosystems. Future work will focus on reducing redundancy when multiple PRs address the same feature and expanding the classification categories to include internal or miscellaneous changes.

Reference: Jiang, H., et al. (2022). DeepRelease: Language-agnostic Release Notes Generation from Pull Requests of Open-source Software. arXiv:2201.06720v1 [cs.SE].

Summary updated

Abstract

The release note is an essential software artifact of open-source software that documents crucial information about changes, such as new features and bug fixes. With the help of release notes, both developers and users could have a general understanding of the latest version without browsing the source code. However, it is a daunting and time-consuming job for developers to produce release notes. Although prior studies have provided some automatic approaches, they generate release notes mainly by extracting information from code changes. This will result in language-specific and not being general enough to be applicable. Therefore, helping developers produce release notes effectively remains an unsolved challenge. To address the problem, we first conduct a manual study on the release notes of 900 GitHub projects, which reveals that more than 54% of projects produce their release notes with pull requests. Based on the empirical finding, we propose a deep learning based approach named DeepRelease (Deep learning based Release notes generator) to generate release notes according to pull requests. The process of release notes generation in DeepRelease includes the change entries generation and the change category (i.e., new features or bug fixes) generation, which are formulated as a text summarization task and a multi-class classification problem, respectively. Since DeepRelease fully employs text information from pull requests to summarize changes and identify the change category, it is language-agnostic and can be used for projects in any language. We build a dataset with over 46K release notes and evaluate DeepRelease on the dataset. The experimental results indicate that DeepRelease outperforms four baselines and can generate release notes similar to those manually written ones in a fraction of the time.
BibTeX
@inproceedings{jiang2021deeprelease,
  author = {Huaxi Jiang and Jie Zhu and Li Yang and Geng Liang and Chun Zuo},
  title = {{DeepRelease: Language-agnostic Release Notes Generation from Pull Requests of Open-source Software}},
  booktitle = {APSEC},
  year = {2021},
  url = {https://openreview.net/forum?id=MYxchpDOtO5},
}
Figure