Understanding code snippets in code reviews: a preliminary study of the OpenStack community

Liming Fu, Peng Liang, Beiqi Zhang

Abstract

Code review is a fundamental practice in modern software engineering, yet the nuances of how developers communicate within this process are still being explored. This study investigates the use of code snippets in review comments within the OpenStack community. By analyzing over 10,000 comments, the researchers found that while code snippets appear in only 5.8% of reviews, they are highly effective communication tools. Reviewers are the primary authors of these snippets, using them to suggest improvements that are accepted by developers approximately 70% of the time.

Introduction

Modern code review is often an asynchronous, text-based process. While reviewers frequently include snippets of code to clarify their feedback or suggest alternatives, the empirical impact of this practice has remained largely unquantified. This research addresses this gap by examining the prevalence, intent, and effectiveness of code snippets in the OpenStack community. The goal is to understand the "information needs" of developers and provide insights that could lead to better-integrated review tools.

Methodology

The study employed a Goal Question Metric (GQM) approach to investigate three primary areas: prevalence, purpose, and acceptance of code snippets.

Data Collection and Processing

The researchers focused on the OpenStack community, specifically the Nova (provisioning) and Neutron (networking) projects. Using the Gerrit REST API, they collected closed code changes from 2020. To ensure consistency, the dataset was filtered to include only human-authored comments on Python files. This resulted in a final dataset of 10,790 review comments.

Analysis Techniques

To identify and categorize snippets, the team used a multi-step process:

  1. Detection: The Guesslang tool was used to automatically detect code fragments within comments.
  2. Qualitative Labeling: The researchers used the Constant Comparison method to iteratively categorize the snippets, resulting in a stable taxonomy of use cases.
  3. Validation: A Cohen’s Kappa coefficient of 0.86 was achieved, indicating high inter-rater reliability during the manual labeling process.

Key Findings

1. Prevalence and Authorship

Code snippets are a specialized tool rather than a ubiquitous one. They appeared in 5.8% (626) of the analyzed review comments. Notably, the study found that snippets are primarily a tool for reviewers; 87.4% of snippets were authored by reviewers rather than the original code submitters. This suggests that snippets serve as a "show, don't just tell" mechanism for providing constructive feedback.

2. A Taxonomy of Snippet Usage

The researchers categorized snippet usage into two high-level categories:

  • Suggestion: Proactive advice intended to change the code (e.g., fixing errors or improving implementation).
  • Citation: Reactive references to existing code used to provide context or elaborate on a specific point.

Within these categories, the most common motivation for including a snippet was to suggest a better way to implement a feature, accounting for 56.1% of all instances.

3. Impact and Acceptance Rates

When a reviewer provides a code snippet as a suggestion, it carries significant weight. The study found an overall acceptance rate of 68.1% for snippet-based suggestions.

Category Count Acceptance Rate
Following Code Style 74 75.7%
Correcting Code 52 73.1%
Complementing Implementation 40 67.5%
Improving Implementation 307 65.5%

Interestingly, suggestions regarding "Code Style" saw the highest acceptance rate, indicating that developers are particularly receptive to snippets that help them align with community standards.

Discussion and Limitations

While the high acceptance rate demonstrates the value of code snippets, the study is limited by its focus on a single community (OpenStack) and a single programming language (Python). The researchers also noted that while they tracked acceptance, the specific reasons why approximately 32% of suggestions were rejected remain a subject for future qualitative inquiry.

Conclusion

This preliminary study highlights the effectiveness of code snippets as a medium for technical communication in code reviews. Although they are used in a minority of comments, their high acceptance rate suggests they are a powerful tool for reviewers to influence code quality. Future work will involve expanding this analysis to larger, multi-language datasets and exploring the potential for tools that can automatically suggest relevant snippets to reviewers based on the context of the change.

Summary updated

BibTeX
@inproceedings{fu2022understanding,
  author = {Liming Fu and Peng Liang and Beiqi Zhang},
  title = {{Understanding code snippets in code reviews: a preliminary study of the OpenStack community}},
  booktitle = {ICPC},
  year = {2022},
  doi = {10.1145/3524610.3527884},
}
Figure