RNSum: A Large-Scale Dataset for Automatic Release Note Generation via Commit Logs Summarization

Hisashi Kamezawa, Noriki Nishida, Nobuyuki Shimizu, Takashi Miyazaki, Hideki NakayamaACL 2022 · OpenReview

Introduction

Writing release notes is often considered the "documentation debt" of software development. While essential for keeping users and developers informed about changes, manually distilling hundreds of commit messages into a concise summary is a labor-intensive task. Previous attempts to automate this process were hampered by data scarcity—with datasets often containing fewer than 5,000 examples—and a reliance on specific external tools like Jira, which limits their applicability across the broader GitHub ecosystem.

To address these challenges, researchers have introduced RNSum, the first large-scale dataset designed for classwise summarization of commit logs into release notes. By treating the problem as a categorized summarization task, the authors demonstrate that modern transformer-based models can produce organized updates that rival human clarity.

The RNSum Dataset

The primary contribution of this work is the RNSum dataset, which significantly expands the available resources for this task. It consists of approximately 82,000 English release notes paired with their corresponding commit messages, sourced from over 7,000 GitHub repositories.

Comparison of RNSum with existing datasets showing its significantly larger scale.

Unlike previous datasets that were limited in scope, RNSum organizes notes into four key functional classes:

  • Features (F)
  • Improvements (I)
  • Bug Fixes (B)
  • Deprecations+ (D) (including removals and breaking changes)

This categorization allows models to learn the specific linguistic styles and information requirements of different types of software updates.

Methodology

The researchers explored two primary architectural strategies based on the BART (Bidirectional and Auto-Regressive Transformers) model:

  1. Classwise Extractive-then-Abstractive Summarization (CEAS): This "filter-then-distill" approach uses a classifier (BERT or CodeBERT) to identify relevant commits for a specific category before passing them to a generator for summarization.
  2. Classwise Abstractive Summarization (CAS): This is an end-to-end approach. The authors tested both a single model for all categories (CAS-Single) and separate models specialized for each category (CAS-Multi).

To overcome the lack of manual labels for every commit, the team employed pseudo-labeling, using common commit message prefixes (e.g., "fix:") to automatically categorize hundreds of thousands of messages for training.

Results and Insights

The proposed models demonstrated a substantial performance leap over existing baselines such as Glyph and simple clustering methods. On the "Cleaned" test set, the CEAS(BERT) model achieved a ROUGE-L score of 36.5, more than doubling the performance of previous state-of-the-art methods.

Performance comparison of various summarization models on the RNSum test sets.

Beyond automatic metrics, human evaluations revealed that the CAS-Multi model was particularly effective. Interestingly, it received higher "Noise" scores than human-written references, indicating it was often more concise and better at filtering out irrelevant "chores" or internal refactors.

A key insight from the study is the "Hidden Info Problem." Human-written release notes frequently include details found in Pull Request descriptions or Issue trackers rather than the commit messages themselves. This explains the remaining "coverage gap" where AI models struggle to capture information not explicitly present in the input logs.

Conclusion and Future Work

While RNSum represents a major step forward, challenges remain. The dataset exhibits a natural class imbalance, with "Bug Fixes" being far more common than "Deprecations." Furthermore, the "context gap" suggests that future research should integrate auxiliary data sources like PR descriptions and Issue trackers to fully capture the "why" behind software changes.

The RNSum dataset and associated code have been made publicly available, providing a robust new benchmark for the software engineering and NLP communities to advance the state of automated documentation.

This paper was presented at the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022).

Summary updated

BibTeX
@inproceedings{kamezawa2022rnsum,
  author = {Hisashi Kamezawa and Noriki Nishida and Nobuyuki Shimizu and Takashi Miyazaki and Hideki Nakayama},
  title = {{RNSum: A Large-Scale Dataset for Automatic Release Note Generation via Commit Logs Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=psdtsQkyYJe},
}
Figure