GIRT-Data: Sampling GitHub Issue Report Templates

Nafiseh Nikeghbal, Amir Hossein Kargaran, Abbas Heydarnoori, Hinrich SchützeMSR 2023 · DOI · OpenReview

TL;DR

This paper introduces GIRT-DATA, the first and largest dataset of GitHub Issue Report Templates (IRTs), along with a tool to collect them. This resource aims to democratize research on IRTs, which are currently underutilized despite their potential to improve issue report quality and streamline developer workflows.

What Problem Does This Paper Address?

GitHub, hosting millions of open-source projects, relies heavily on issue reports for software development. However, the quality and structure of these reports can vary significantly, leading to increased developer effort in understanding and addressing them. To combat this, GitHub introduced Issue Report Templates (IRTs) to pre-fill issue descriptions with guidance. Despite their introduction in 2016 and positive reception, IRTs are used in less than 5% of repositories. Existing research on IRTs is limited, often analyzing only a small subset of popular projects. This lack of comprehensive data hinders a deeper understanding of IRT effectiveness, usage patterns, and potential for improvement. The paper addresses this gap by creating a large-scale dataset and a tool to facilitate research in this area.

Methods

The authors developed a crawler to collect IRTs and associated repository data from GitHub. The process involved several key steps:

  • Target Repository Selection: They started with a large set of repositories from the GHS project, filtering for those with at least 10 stars to ensure a reasonable level of quality and manageability.
  • Data Extraction: The crawler utilizes the GitHub API (via PyGithub) and XPath queries on repository landing pages to extract 29 repository characteristics. This includes metrics like stars, forks, contributors, and issue counts.
  • IRT Collection: For repositories identified as having IRTs (indicated by the presence of a .github/ISSUE_TEMPLATE directory), the crawler downloads the template files.
  • IRT Characteristic Parsing: For Markdown IRTs, the authors parse them to extract structured information such as the presence of an initial table, name, description, default title, labels, assignees, and the raw and anonymized body. They also extract headlines. YAML IRTs are stored in their raw format, as they are more structured and can be easily converted to other formats.

Datasets

The primary contribution is the GIRT-DATA dataset.

  • Size: It contains data from 1,084,300 GitHub repositories, with 50,032 of these repositories supporting IRTs.
  • Content: The dataset includes 29 repository characteristics and detailed characteristics for both Markdown and YAML IRTs.
  • Formats: IRTs are provided in both YAML and Markdown formats.
  • Availability: A stable version of the dataset and the crawler tool are available on Zenodo [13] and GitHub, respectively.
  • Relation to Existing Datasets: This dataset is novel in its focus on IRTs and its scale, going beyond previous studies that analyzed much smaller subsets of repositories.

Key Insights and Ideas

  • Underutilization of IRTs: Despite their potential benefits, IRTs are surprisingly underutilized, with only about 5% of eligible repositories employing them. This highlights a significant opportunity for improvement in developer practices.
  • IRT Structure Matters: The paper distinguishes between Markdown and YAML IRTs, noting that Markdown is more popular but less structured, leading to more detailed parsing for this format. This suggests that the format itself might influence adoption and utility.
  • Repository Characteristics Correlate with IRT Usage: Preliminary analysis (Figure 1) suggests that as certain repository characteristics (like stars, forks, and issue counts) increase, the likelihood of a repository using an IRT also tends to increase. This implies that more established or active projects might be more inclined to adopt such features.
  • Need for a Comprehensive Dataset: The authors emphasize that existing research is hampered by a lack of large-scale, comprehensive datasets of IRTs, which GIRT-DATA aims to address.

Key Results

  • Dataset Size: GIRT-DATA is the first and largest dataset of GitHub IRTs, encompassing 1,084,300 repositories and 50,032 repositories with IRTs.
  • IRT Formats: The dataset includes IRTs in both YAML and Markdown formats.
  • Repository Characteristics: 29 repository characteristics are collected, providing context for IRT usage.
  • Preliminary Analysis: A positive correlation was observed between several repository metrics (e.g., stargazers, forks) and the adoption of IRTs, though contributors and commits showed less clear patterns.

Limitations and Future Work

  • Limitations:
    • The crawler's ability to collect data is dependent on the GitHub API and repository landing pages, which can have limitations.
  • The analysis of IRT usage is preliminary, and deeper investigations are needed.
  • Future Work:
    • The authors plan to support more repository characteristics and IRT/PRT characteristics to further enrich the dataset.
  • They aim to encourage more research by providing this comprehensive dataset and tool, enabling studies on various aspects of IRTs, such as their impact on issue resolution time, contributor behavior, and their effectiveness in generating better issue reports.

Summary updated

Abstract

GitHub's issue reports provide developers with valuable information that is essential to the evolution of a software development project. Contributors can use these reports to perform software engineering tasks like submitting bugs, requesting features, and collaborating on ideas. In the initial versions of issue reports, there was no standard way of using them. As a result, the quality of issue reports varied widely. To improve the quality of issue reports, GitHub introduced issue report templates (IRTs), which pre-fill issue descriptions when a new issue is opened. An IRT usually contains greeting contributors, describing project guidelines, and collecting relevant information. However, despite of effectiveness of this feature which was introduced in 2016, only nearly 5% of GitHub repositories (with more than 10 stars) utilize it. There are currently few articles on IRTs, and the available ones only consider a small number of repositories. In this work, we introduce GIRT-Data, the first and largest dataset of IRTs in both YAML and Markdown format. This dataset and its corresponding open-source crawler tool are intended to support research in this area and to encourage more developers to use IRTs in their repositories. The stable version of the dataset contains 1,084,300 repositories and 50,032 of them support IRTs. The stable version of the dataset and crawler is available here: https://github.com/kargaranamir/girt-data
BibTeX
@inproceedings{nikeghbal2023girt,
  author = {Nafiseh Nikeghbal and Amir Hossein Kargaran and Abbas Heydarnoori and Hinrich Schütze},
  title = {{GIRT-Data: Sampling GitHub Issue Report Templates}},
  booktitle = {MSR},
  year = {2023},
  doi = {10.1109/msr59073.2023.00026},
}

Loading related content...

Figure