TL;DR
DeepRelease is a deep learning-based approach that automatically generates release notes by summarizing the natural language content within GitHub Pull Requests (PRs). By focusing on text descriptions rather than source code changes, it provides a language-agnostic solution that outperforms traditional baselines and generates human-like summaries in milliseconds.
What Problem Does This Paper Address?
Release notes are vital for communicating new features and bug fixes to users, yet they are notoriously "daunting and time-consuming" to produce. Previous studies suggest that manually curating these notes can take a maintainer up to eight hours per release.
While automated tools exist, they typically rely on analyzing source code changes (e.g., AST diffs). This creates two major hurdles:
- Language Dependency: A tool built for Java won't work for Python or C++.
- Missing Context: Code changes show what changed, but often fail to capture the why or the high-level impact on the user.
The authors of DeepRelease observed that many developers already use Pull Requests as a primary source for release notes. They set out to bridge the gap by treating release note generation as a natural language processing (NLP) task rather than a code analysis task.
Methods
DeepRelease utilizes a multi-stage deep learning pipeline to transform raw PR data into formatted Markdown release notes.
- Pull Request Collector: This module uses the GitHub API to gather the PR title, description, and commit messages for all PRs merged since the last release.
- Entry Summarizer: To generate the actual descriptive text for each change, the authors employed a Pointer-Generator Network. This is a specialized Sequence-to-Sequence (Seq2Seq) model with an attention mechanism. Its "killer feature" is the ability to either generate new words from a vocabulary or "copy" specific words directly from the input. This is crucial for software documentation, which is often filled with "Out-of-Vocabulary" (OOV) tokens like specific function names, module identifiers, or CVE numbers.
- Category Discriminator: Not all changes are equal. DeepRelease uses a FastText-based classifier to categorize each PR into one of four types: Issues fixed, New features, Non-functional requirements, or Documentation.
- Text Generator: Finally, the system pairs the summarized entries with their predicted categories and renders them into a clean Markdown format.
Datasets
Because no public dataset existed for this specific task, the authors built their own.
- The Corpus: They collected 46,656 release note entries from 400 non-trivial GitHub repositories.
- Diversity: The dataset spans nine programming languages (C, Python, Java, C++, C#, JavaScript, PHP, Ruby, and Go), ensuring the model is truly language-agnostic.
- Preprocessing: They implemented rigorous cleaning, including removing PR templates, filtering out "noise" (like emojis, signatures, and URLs), and unifying disparate category labels (e.g., merging "Fix" and "Bug fixes" into a single "Issues fixed" category).
Key Insights and Ideas
- The PR-Centric Shift: The most significant "aha moment" was the empirical finding that 54% of projects already link their release notes to PRs. This validated the authors' hypothesis that PRs contain enough semantic information to replace code-heavy analysis.
- Language Agnosticism is a Feature, Not a Bug: By ignoring the source code and focusing on the developer's description of the code, the tool becomes universally applicable across the entire software ecosystem.
- Handling Technical Jargon: The use of the Pointer-Generator network addresses a common failure point in NLP for software engineering: the "OOV problem." By allowing the model to "point" to specific technical terms in the PR title and copy them into the summary, the notes remain technically accurate.
Key Results
DeepRelease was evaluated against common baselines like LeadCM (using the first few tokens of commit messages) and PR-Title (using the PR title as the entry).
- Summarization Quality: DeepRelease achieved a ROUGE-L score of 73.51, outperforming the PR-Title baseline by roughly 5%. While 5% sounds modest, the authors note that DeepRelease's ability to rephrase and condense information makes the notes much more readable than raw titles.
- Classification Accuracy: The Category Discriminator achieved F1 scores as high as 88% for certain categories, significantly beating keyword-based heuristics.
- Efficiency: In a head-to-head against the "eight-hour manual process," DeepRelease generated entries in an average of 29.7 milliseconds on a single NVIDIA RTX 3090.
Limitations and Future Work
The authors identify a few areas for improvement:
- Dependency on PR Quality: If a developer writes a cryptic PR title (e.g., "Update x.py"), DeepRelease will struggle. It assumes a baseline of "good practice" in PR documentation.
- Redundancy: Currently, the tool generates one entry per PR. If multiple PRs address the same feature, the release notes might feel repetitive. Future work involves using NLP to merge similar entries.
- Category Breadth: The model currently only handles the top four most common categories. Expanding this to include "Internal changes" or "Miscellaneous" is a goal for future versions.
Reference: Jiang, H., et al. (2022). DeepRelease: Language-agnostic Release Notes Generation from Pull Requests of Open-source Software. arXiv:2201.06720v1 [cs.SE].
Summary updated