TL;DR
This paper introduces GIRT-DATA, the first and largest dataset of GitHub Issue Report Templates (IRTs), along with a tool to collect them. This resource aims to democratize research on IRTs, which are currently underutilized despite their potential to improve issue report quality and streamline developer workflows.
What Problem Does This Paper Address?
GitHub, hosting millions of open-source projects, relies heavily on issue reports for software development. However, the quality and structure of these reports can vary significantly, leading to increased developer effort in understanding and addressing them. To combat this, GitHub introduced Issue Report Templates (IRTs) to pre-fill issue descriptions with guidance. Despite their introduction in 2016 and positive reception, IRTs are used in less than 5% of repositories. Existing research on IRTs is limited, often analyzing only a small subset of popular projects. This lack of comprehensive data hinders a deeper understanding of IRT effectiveness, usage patterns, and potential for improvement. The paper addresses this gap by creating a large-scale dataset and a tool to facilitate research in this area.
Methods
The authors developed a crawler to collect IRTs and associated repository data from GitHub. The process involved several key steps:
- Target Repository Selection: They started with a large set of repositories from the GHS project, filtering for those with at least 10 stars to ensure a reasonable level of quality and manageability.
- Data Extraction: The crawler utilizes the GitHub API (via
PyGithub) and XPath queries on repository landing pages to extract 29 repository characteristics. This includes metrics like stars, forks, contributors, and issue counts. - IRT Collection: For repositories identified as having IRTs (indicated by the presence of a
.github/ISSUE_TEMPLATEdirectory), the crawler downloads the template files. - IRT Characteristic Parsing: For Markdown IRTs, the authors parse them to extract structured information such as the presence of an initial table, name, description, default title, labels, assignees, and the raw and anonymized body. They also extract headlines. YAML IRTs are stored in their raw format, as they are more structured and can be easily converted to other formats.
Datasets
The primary contribution is the GIRT-DATA dataset.
- Size: It contains data from 1,084,300 GitHub repositories, with 50,032 of these repositories supporting IRTs.
- Content: The dataset includes 29 repository characteristics and detailed characteristics for both Markdown and YAML IRTs.
- Formats: IRTs are provided in both YAML and Markdown formats.
- Availability: A stable version of the dataset and the crawler tool are available on Zenodo [13] and GitHub, respectively.
- Relation to Existing Datasets: This dataset is novel in its focus on IRTs and its scale, going beyond previous studies that analyzed much smaller subsets of repositories.
Key Insights and Ideas
- Underutilization of IRTs: Despite their potential benefits, IRTs are surprisingly underutilized, with only about 5% of eligible repositories employing them. This highlights a significant opportunity for improvement in developer practices.
- IRT Structure Matters: The paper distinguishes between Markdown and YAML IRTs, noting that Markdown is more popular but less structured, leading to more detailed parsing for this format. This suggests that the format itself might influence adoption and utility.
- Repository Characteristics Correlate with IRT Usage: Preliminary analysis (Figure 1) suggests that as certain repository characteristics (like stars, forks, and issue counts) increase, the likelihood of a repository using an IRT also tends to increase. This implies that more established or active projects might be more inclined to adopt such features.
- Need for a Comprehensive Dataset: The authors emphasize that existing research is hampered by a lack of large-scale, comprehensive datasets of IRTs, which GIRT-DATA aims to address.
Key Results
- Dataset Size: GIRT-DATA is the first and largest dataset of GitHub IRTs, encompassing 1,084,300 repositories and 50,032 repositories with IRTs.
- IRT Formats: The dataset includes IRTs in both YAML and Markdown formats.
- Repository Characteristics: 29 repository characteristics are collected, providing context for IRT usage.
- Preliminary Analysis: A positive correlation was observed between several repository metrics (e.g., stargazers, forks) and the adoption of IRTs, though contributors and commits showed less clear patterns.
Limitations and Future Work
- Limitations:
- The crawler's ability to collect data is dependent on the GitHub API and repository landing pages, which can have limitations.
- The analysis of IRT usage is preliminary, and deeper investigations are needed.
- Future Work:
- The authors plan to support more repository characteristics and IRT/PRT characteristics to further enrich the dataset.
- They aim to encourage more research by providing this comprehensive dataset and tool, enabling studies on various aspects of IRTs, such as their impact on issue resolution time, contributor behavior, and their effectiveness in generating better issue reports.
Summary updated