DeepRelease: Language-agnostic Release Notes Generation from Pull Requests of Open-source Software

Huaxi Jiang, Jie Zhu, Li Yang, Geng Liang, Chun Zuo

TL;DR

DeepRelease is a deep learning-based approach that automatically generates release notes by summarizing the natural language content within GitHub Pull Requests (PRs). By focusing on text descriptions rather than source code changes, it provides a language-agnostic solution that outperforms traditional baselines and generates human-like summaries in milliseconds.

What Problem Does This Paper Address?

Release notes are vital for communicating new features and bug fixes to users, yet they are notoriously "daunting and time-consuming" to produce. Previous studies suggest that manually curating these notes can take a maintainer up to eight hours per release.

While automated tools exist, they typically rely on analyzing source code changes (e.g., AST diffs). This creates two major hurdles:

  1. Language Dependency: A tool built for Java won't work for Python or C++.
  2. Missing Context: Code changes show what changed, but often fail to capture the why or the high-level impact on the user.

The authors of DeepRelease observed that many developers already use Pull Requests as a primary source for release notes. They set out to bridge the gap by treating release note generation as a natural language processing (NLP) task rather than a code analysis task.

Methods

DeepRelease utilizes a multi-stage deep learning pipeline to transform raw PR data into formatted Markdown release notes.

  • Pull Request Collector: This module uses the GitHub API to gather the PR title, description, and commit messages for all PRs merged since the last release.
  • Entry Summarizer: To generate the actual descriptive text for each change, the authors employed a Pointer-Generator Network. This is a specialized Sequence-to-Sequence (Seq2Seq) model with an attention mechanism. Its "killer feature" is the ability to either generate new words from a vocabulary or "copy" specific words directly from the input. This is crucial for software documentation, which is often filled with "Out-of-Vocabulary" (OOV) tokens like specific function names, module identifiers, or CVE numbers.
  • Category Discriminator: Not all changes are equal. DeepRelease uses a FastText-based classifier to categorize each PR into one of four types: Issues fixed, New features, Non-functional requirements, or Documentation.
  • Text Generator: Finally, the system pairs the summarized entries with their predicted categories and renders them into a clean Markdown format.

Datasets

Because no public dataset existed for this specific task, the authors built their own.

  • The Corpus: They collected 46,656 release note entries from 400 non-trivial GitHub repositories.
  • Diversity: The dataset spans nine programming languages (C, Python, Java, C++, C#, JavaScript, PHP, Ruby, and Go), ensuring the model is truly language-agnostic.
  • Preprocessing: They implemented rigorous cleaning, including removing PR templates, filtering out "noise" (like emojis, signatures, and URLs), and unifying disparate category labels (e.g., merging "Fix" and "Bug fixes" into a single "Issues fixed" category).

Key Insights and Ideas

  • The PR-Centric Shift: The most significant "aha moment" was the empirical finding that 54% of projects already link their release notes to PRs. This validated the authors' hypothesis that PRs contain enough semantic information to replace code-heavy analysis.
  • Language Agnosticism is a Feature, Not a Bug: By ignoring the source code and focusing on the developer's description of the code, the tool becomes universally applicable across the entire software ecosystem.
  • Handling Technical Jargon: The use of the Pointer-Generator network addresses a common failure point in NLP for software engineering: the "OOV problem." By allowing the model to "point" to specific technical terms in the PR title and copy them into the summary, the notes remain technically accurate.

Key Results

DeepRelease was evaluated against common baselines like LeadCM (using the first few tokens of commit messages) and PR-Title (using the PR title as the entry).

  • Summarization Quality: DeepRelease achieved a ROUGE-L score of 73.51, outperforming the PR-Title baseline by roughly 5%. While 5% sounds modest, the authors note that DeepRelease's ability to rephrase and condense information makes the notes much more readable than raw titles.
  • Classification Accuracy: The Category Discriminator achieved F1 scores as high as 88% for certain categories, significantly beating keyword-based heuristics.
  • Efficiency: In a head-to-head against the "eight-hour manual process," DeepRelease generated entries in an average of 29.7 milliseconds on a single NVIDIA RTX 3090.

Limitations and Future Work

The authors identify a few areas for improvement:

  • Dependency on PR Quality: If a developer writes a cryptic PR title (e.g., "Update x.py"), DeepRelease will struggle. It assumes a baseline of "good practice" in PR documentation.
  • Redundancy: Currently, the tool generates one entry per PR. If multiple PRs address the same feature, the release notes might feel repetitive. Future work involves using NLP to merge similar entries.
  • Category Breadth: The model currently only handles the top four most common categories. Expanding this to include "Internal changes" or "Miscellaneous" is a goal for future versions.

Reference: Jiang, H., et al. (2022). DeepRelease: Language-agnostic Release Notes Generation from Pull Requests of Open-source Software. arXiv:2201.06720v1 [cs.SE].

Summary updated

Abstract

The release note is an essential software artifact of open-source software that documents crucial information about changes, such as new features and bug fixes. With the help of release notes, both developers and users could have a general understanding of the latest version without browsing the source code. However, it is a daunting and time-consuming job for developers to produce release notes. Although prior studies have provided some automatic approaches, they generate release notes mainly by extracting information from code changes. This will result in language-specific and not being general enough to be applicable. Therefore, helping developers produce release notes effectively remains an unsolved challenge. To address the problem, we first conduct a manual study on the release notes of 900 GitHub projects, which reveals that more than 54% of projects produce their release notes with pull requests. Based on the empirical finding, we propose a deep learning based approach named DeepRelease (Deep learning based Release notes generator) to generate release notes according to pull requests. The process of release notes generation in DeepRelease includes the change entries generation and the change category (i.e., new features or bug fixes) generation, which are formulated as a text summarization task and a multi-class classification problem, respectively. Since DeepRelease fully employs text information from pull requests to summarize changes and identify the change category, it is language-agnostic and can be used for projects in any language. We build a dataset with over 46K release notes and evaluate DeepRelease on the dataset. The experimental results indicate that DeepRelease outperforms four baselines and can generate release notes similar to those manually written ones in a fraction of the time.
BibTeX
@inproceedings{jiang2021deeprelease,
  author = {Huaxi Jiang and Jie Zhu and Li Yang and Geng Liang and Chun Zuo},
  title = {{DeepRelease: Language-agnostic Release Notes Generation from Pull Requests of Open-source Software}},
  booktitle = {APSEC},
  year = {2021},
  url = {https://openreview.net/forum?id=MYxchpDOtO5},
}

Loading related content...

Figure