Is It Enough to Recommend Tasks to Newcomers? Understanding Mentoring on Good First Issues

Xin Tan, Yiran Chen, Haohua Wu, Minghui Zhou, Li Zhang

TL;DR

This paper investigates the crucial role of mentoring for newcomers in open-source software (OSS) projects, moving beyond just recommending "good first issues" (GFIs). By analyzing over 48,000 GFIs, the study reveals that while expert involvement significantly boosts the chances of a newcomer successfully resolving an issue, it surprisingly correlates negatively with their long-term retention, suggesting that strong initial skills and willingness are key drivers for continued contribution.

What Problem Does This Paper Address?

Open-source software (OSS) projects rely heavily on a continuous influx of new contributors to thrive. To attract and onboard these newcomers, many projects use "good first issues" (GFIs) as a way to recommend manageable tasks. However, simply pointing newcomers to these issues might not be enough. There's a significant gap in understanding how effective mentoring actually is in OSS communities, what forms it takes, and its true impact on both immediate success and long-term contributor retention. This paper aims to fill this gap by delving into the daily mentoring practices within OSS projects.

Methods

The researchers employed a mixed-methods approach:

  • Quantitative Analysis: They analyzed the resolution process of 48,402 GFIs from 964 repositories. This involved examining comment data, participant interactions, and response times.
  • Network Analysis: To understand mentorship structures, they built bipartite graphs representing collaboration networks between newcomers and experts, identifying four distinct mentorship types (centralized, decentralized, collaborative, distributed).
  • Thematic Analysis: They performed a deep dive into the comments of "highly discussed" GFIs to identify common newcomer challenges and the types of mentoring provided by experts.
  • Email Survey: A survey was conducted with newcomers and experts to gather their perspectives on the frequency and importance of identified challenges and mentoring strategies.
  • Generalized Linear Models (GLMs): Statistical models were used to assess the relevance of expert involvement and other factors on both the successful resolution of GFIs by newcomers and their subsequent retention.

Datasets

The study utilized data from the GHTorrent database, specifically focusing on repositories that actively use the GFI mechanism. The constructed dataset comprised:

  • 964 repositories
  • 48,402 closed GFIs
  • 286,371 comments (including issue and PR/commit comments)

The researchers also conducted a pilot survey and a main survey with newcomers and experts, collecting 11 and 32 responses respectively.

Key Insights and Ideas

  • Expert involvement is common, but not always deep: While ~70% of GFIs see some form of expert participation, the engagement is often limited to one expert making a couple of comments.
  • Mentorship isn't one-size-fits-all: OSS communities exhibit diverse mentorship structures, with "collaborative mentoring" (multiple experts working together) being the most prevalent.
  • Newcomers face a broad spectrum of challenges: Beyond just technical hurdles, newcomers struggle with understanding issue assignments, reproducing issues, and navigating complex project workflows.
  • Experts offer more than just technical help: Mentoring includes clarifying community regulations, providing task recommendations, and offering a warm, encouraging attitude.
  • The "Good First Issue" is just the start: Recommending tasks is a good first step, but active expert guidance is crucial for newcomers to actually solve these issues.
  • The paradox of expert involvement: While experts help newcomers succeed in their first contribution, their involvement seems to decrease the likelihood of those newcomers becoming long-term contributors.

Key Results

  • Expert Involvement: Approximately 70% of GFIs have expert participation. Experts typically provide their first comment within 8.5 hours of a newcomer's comment.
  • Mentorship Structures: Four types were identified: centralized (29.09%), decentralized (14.91%), collaborative (36.00%), and distributed (20.00%). Collaborative mentoring is the most common.
  • Discussed Topics: 14 newcomer challenges and 18 expert mentoring content areas were identified. Newcomers found "understanding" and "implementation" the most difficult challenges, while experts rated "clarification of the issue" and "pointing out better solutions" as very important mentoring actions.
  • GFI Resolution: Expert involvement positively correlates with newcomers successfully resolving GFIs (a 71% increase in success probability). Newcomers leaving more comments and GFIs in popular projects also increase resolution chances.
  • Newcomer Retention: Expert involvement negatively correlates with newcomer retention. Factors like newcomers' strong initial skills and willingness appear to be more significant drivers for long-term contribution.

Limitations and Future Work

  • External Validity: The study focused on the top 1% of repositories with the most GFIs, which might not represent all OSS communities, especially those with fewer GFIs.
  • Causality: The study identifies correlations, but proving direct causal relationships, especially for retention, requires further investigation.
  • Measuring Ability/Willingness: Future work should focus on directly measuring newcomer ability and willingness to better understand retention factors and reduce the interference of expert involvement.
  • Mentorship Structure Impact: The impact of mentorship structures on onboarding was not a significant predictor in the models, possibly due to limited labeled data. Future research could explore this further with more sophisticated classification algorithms.
  • Tool Development: Designing automatic tools to facilitate mentoring based on the identified newcomer challenges is a promising direction.

Summary updated

Abstract

Newcomers are critical for the success and continuity of open source software (OSS) projects. To attract newcomers and facilitate their onboarding, many OSS projects recommend tasks for newcomers, such as good first issues (GFIs). Previous studies have preliminarily investigated the effects of GFIs and techniques to identify suitable GFIs. However, it is still unclear whether just recommending tasks is enough and how significant mentoring is for newcomers. To better understand mentoring in OSS communities, we analyze the resolution process of 48,402 GFIs from 964 repositories through a mix-method approach. We investigate the extent, the mentorship structures, the discussed topics, and the relevance of expert involvement. We find that ~70% of GFIs have expert participation, with each GFI usually having one expert who makes two comments. Half of GFIs will receive their first expert comment within 8.5 hours after a newcomer comment. Through analysis of the collaboration networks of newcomers and experts, we observe that community mentorship presents four types of structure: centralized mentoring, decentralized mentoring, collaborative mentoring, and distributed mentoring. As for discussed topics, we identify 14 newcomer challenges and 18 expert mentoring content. By fitting the generalized linear models, we find that expert involvement positively correlates with newcomers' successful contributions but negatively correlates with newcomers' retention. Our study manifests the status and significance of mentoring in the OSS projects, which provides rich practical implications for optimizing the mentoring process and helping newcomers contribute smoothly and suecessfully,
BibTeX
@misc{tan2023is,
  author = {Xin Tan and Yiran Chen and Haohua Wu and Minghui Zhou and Li Zhang},
  title = {{Is It Enough to Recommend Tasks to Newcomers? Understanding Mentoring on Good First Issues}},
  year = {2023},
  eprint = {2302.05058},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.05058},
}
Figure