Introduction
In modern software engineering, change-based code review is the primary mechanism for ensuring code quality and knowledge sharing. However, as software systems grow in complexity, reviewers are increasingly confronted with a "wall of code"—massive changesets that lead to cognitive overload. When a changeset becomes too large, the quality of the review often diminishes, allowing critical bugs to slip through the cracks while reviewers spend time on boilerplate or trivial modifications.
This paper presents an industrial case study focused on providing "cognitive support" to reviewers. The researchers explore whether machine learning can be used to automatically identify and hide low-risk code changes that are unlikely to trigger reviewer comments (remarks). By filtering out these unimportant sections, reviewers can focus their limited mental energy on complex logic where defects are most likely to reside.
The Challenge of Remark Tracing
To build a predictive model for code reviews, researchers first need to understand which historical code changes actually prompted reviewer feedback. While algorithms like SZZ are commonly used in defect prediction to link bugs to commits, the authors found them insufficient for the specific "review-fix" cycles of industrial development.
Standard SZZ often fails to account for the iterative nature of developer tickets. To address this, the authors developed a novel tracing algorithm that links review remarks back to their specific implementation triggers. By expanding the search scope from individual lines to blocks and methods, the algorithm accurately identifies the source of a remark. The study found that for 60% of review remarks, standard SZZ would have produced different—and likely incorrect—results compared to this specialized tracing approach.
Methodology: Human-in-the-Loop Rule Mining
Rather than deploying an opaque "black box" model, the researchers prioritized transparency and explainability to gain developer trust. They utilized a multi-objective meta-heuristic (GRASP-PR) to mine simple, logic-based "skip rules."
Feature Selection and Rule Generation
The model incorporated 52 distinct features, including:
- Standard Metrics: File age, author experience, and change size.
- Novel Metrics: "Code entropy," which measures how surprising or unusual a new code snippet is relative to the existing codebase.
To ensure the rules were practical, the researchers kept humans in the loop. Developers reviewed and provided feedback on the generated rules (e.g., “Skip if the change is only whitespace AND the file is a test data file”). Interestingly, the study revealed a significant preference for simple syntactic rules over complex statistical ones, even when the latter were mathematically sound.
Experimental Dataset
The study was conducted using a comprehensive dataset from a medium-sized German software company spanning five years (2013–2018). The dataset included:
- Approximately 6,000 developer tickets.
- Over 700,000 "change part" records.
- A primary focus on Java, with additional data from XML and TypeScript files.
The authors have released an anonymized version of this dataset and their algorithm implementations to facilitate further research in the community.
Key Results and Insights
The implementation of the "SESSION" ruleset yielded significant improvements in review efficiency with minimal risk:
- Workload Reduction: The system successfully identified and skipped 25.2% of change parts and 23.2% of Java source lines.
- High Precision: This reduction came at the cost of missing only 0.6% of actual review remarks. Even when using a log-transformed metric to account for remark severity, the miss rate remained low at 1.3%.
- The 80/20 Rule: The researchers discovered that 84% of the total effort savings were derived from just three simple rules: ignoring changes that were exclusively package/import updates, whitespace-only changes, or modifications to binary files.
- Economic Viability: A "break-even" analysis suggested that the approach is profitable as long as the cost of missing a single remark is less than approximately 726 times the cost of reviewing a single code hunk.
Discussion and Developer Perspectives
The study highlighted a philosophical divide within the development team. While many saw the tool as a pragmatic way to manage workload, others viewed the hiding of code as "dangerous," fearing it might create a false sense of security. This skepticism underscores the importance of explainability; developers consistently rated rules based on "entropy" or "surprisingness" lower than those based on clear logic.
Limitations and Future Directions
Several hurdles remain for the widespread adoption of remark prediction:
- Data Noise: Distinguishing between a lack of remarks due to "perfect code" versus a lack of remarks due to reviewer fatigue remains difficult.
- Self-Fulfilling Prophecies: If a tool consistently hides certain code types, reviewers may stop scrutinizing them, leading to a lack of training data for future bugs in those areas.
- The Understandability Hull: Future work aims to develop algorithms that don't just show "important" code, but also automatically include the minimum necessary context (the "hull") required for a human to comprehend the change.
Conclusion
By combining a novel tracing algorithm with human-centric rule mining, this case study demonstrates that it is possible to significantly reduce the code review workload without compromising software quality. The findings suggest that simple, explainable rules are the most effective path toward gaining developer adoption and improving the sustainability of the code review process.
Summary updated