An Industrial Case Study on Shrinking Code Review Changesets through Remark Prediction

Tobias Baum, Steffen Herbold, Kurt SchneiderarXiv 2018 · arXiv

Introduction

In modern software engineering, change-based code review is the primary mechanism for ensuring code quality and knowledge sharing. However, as software systems grow in complexity, reviewers are increasingly confronted with a "wall of code"—massive changesets that lead to cognitive overload. When a changeset becomes too large, the quality of the review often diminishes, allowing critical bugs to slip through the cracks while reviewers spend time on boilerplate or trivial modifications.

This paper presents an industrial case study focused on providing "cognitive support" to reviewers. The researchers explore whether machine learning can be used to automatically identify and hide low-risk code changes that are unlikely to trigger reviewer comments (remarks). By filtering out these unimportant sections, reviewers can focus their limited mental energy on complex logic where defects are most likely to reside.

The Challenge of Remark Tracing

To build a predictive model for code reviews, researchers first need to understand which historical code changes actually prompted reviewer feedback. While algorithms like SZZ are commonly used in defect prediction to link bugs to commits, the authors found them insufficient for the specific "review-fix" cycles of industrial development.

Standard SZZ often fails to account for the iterative nature of developer tickets. To address this, the authors developed a novel tracing algorithm that links review remarks back to their specific implementation triggers. By expanding the search scope from individual lines to blocks and methods, the algorithm accurately identifies the source of a remark. The study found that for 60% of review remarks, standard SZZ would have produced different—and likely incorrect—results compared to this specialized tracing approach.

Methodology: Human-in-the-Loop Rule Mining

Rather than deploying an opaque "black box" model, the researchers prioritized transparency and explainability to gain developer trust. They utilized a multi-objective meta-heuristic (GRASP-PR) to mine simple, logic-based "skip rules."

Feature Selection and Rule Generation

The model incorporated 52 distinct features, including:

  • Standard Metrics: File age, author experience, and change size.
  • Novel Metrics: "Code entropy," which measures how surprising or unusual a new code snippet is relative to the existing codebase.

To ensure the rules were practical, the researchers kept humans in the loop. Developers reviewed and provided feedback on the generated rules (e.g., “Skip if the change is only whitespace AND the file is a test data file”). Interestingly, the study revealed a significant preference for simple syntactic rules over complex statistical ones, even when the latter were mathematically sound.

Experimental Dataset

The study was conducted using a comprehensive dataset from a medium-sized German software company spanning five years (2013–2018). The dataset included:

  • Approximately 6,000 developer tickets.
  • Over 700,000 "change part" records.
  • A primary focus on Java, with additional data from XML and TypeScript files.

The authors have released an anonymized version of this dataset and their algorithm implementations to facilitate further research in the community.

Key Results and Insights

The implementation of the "SESSION" ruleset yielded significant improvements in review efficiency with minimal risk:

  • Workload Reduction: The system successfully identified and skipped 25.2% of change parts and 23.2% of Java source lines.
  • High Precision: This reduction came at the cost of missing only 0.6% of actual review remarks. Even when using a log-transformed metric to account for remark severity, the miss rate remained low at 1.3%.
  • The 80/20 Rule: The researchers discovered that 84% of the total effort savings were derived from just three simple rules: ignoring changes that were exclusively package/import updates, whitespace-only changes, or modifications to binary files.
  • Economic Viability: A "break-even" analysis suggested that the approach is profitable as long as the cost of missing a single remark is less than approximately 726 times the cost of reviewing a single code hunk.

Discussion and Developer Perspectives

The study highlighted a philosophical divide within the development team. While many saw the tool as a pragmatic way to manage workload, others viewed the hiding of code as "dangerous," fearing it might create a false sense of security. This skepticism underscores the importance of explainability; developers consistently rated rules based on "entropy" or "surprisingness" lower than those based on clear logic.

Limitations and Future Directions

Several hurdles remain for the widespread adoption of remark prediction:

  1. Data Noise: Distinguishing between a lack of remarks due to "perfect code" versus a lack of remarks due to reviewer fatigue remains difficult.
  2. Self-Fulfilling Prophecies: If a tool consistently hides certain code types, reviewers may stop scrutinizing them, leading to a lack of training data for future bugs in those areas.
  3. The Understandability Hull: Future work aims to develop algorithms that don't just show "important" code, but also automatically include the minimum necessary context (the "hull") required for a human to comprehend the change.

Conclusion

By combining a novel tracing algorithm with human-centric rule mining, this case study demonstrates that it is possible to significantly reduce the code review workload without compromising software quality. The findings suggest that simple, explainable rules are the most effective path toward gaining developer adoption and improving the sustainability of the code review process.

Summary updated

Abstract

Change-based code review is used widely in industrial software development. Thus, research on tools that help the reviewer to achieve better review performance can have a high impact. We analyze one possibility to provide cognitive support for the reviewer: Determining the importance of change parts for review, specifically determining which parts of the code change can be left out from the review without harm. To determine the importance of change parts, we extract data from software repositories and build prediction models for review remarks based on this data. The approach is discussed in detail. To gather the input data, we propose a novel algorithm to trace review remarks to their triggers. We apply our approach in a medium-sized software company. In this company, we can avoid the review of 25% of the change parts and of 23% of the changed Java source code lines, while missing only about 1% of the review remarks. Still, we also observe severe limitations of the tried approach: Much of the savings are due to simple syntactic rules, noise in the data hampers the search for better prediction models, and some developers in the case company oppose the taken approach. Besides the main results on the mining and prediction of triggers for review remarks, we contribute experiences with a novel, multi-objective and interactive rule mining approach. The anonymized dataset from the company is made available, as are the implementations for the devised algorithms.
BibTeX
@misc{baum2018industrial,
  author = {Tobias Baum and Steffen Herbold and Kurt Schneider},
  title = {{An Industrial Case Study on Shrinking Code Review Changesets through Remark Prediction}},
  year = {2018},
  eprint = {1812.09510},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.09510},
}
Figure