This summary explores the paper "Anticipating Future Build Results with BuildPredictor" by Christian Macho and Katharina Stengg, which introduces a machine learning approach to streamline Continuous Integration (CI) workflows.
TL;DR
BuildPredictor is a machine learning model designed to forecast Continuous Integration (CI) build outcomes before they are executed. By analyzing fine-grained code changes and developer history, the model achieves high predictive accuracy (median MCC of 0.86), offering a way to reduce resource waste and developer "waiting time" in modern software engineering.
What Problem Does This Paper Address?
Continuous Integration is a cornerstone of modern software development, providing rapid feedback by automatically building and testing code after every commit. However, this convenience comes at a cost. Large projects often suffer from long build durations, leading to "context switching"—where developers lose focus while waiting for results. Furthermore, the massive computational resources required for frequent builds translate into significant financial and environmental costs.
While previous research has attempted to predict build outcomes to mitigate these issues, existing models often rely on coarse-grained data or lack the interpretability needed for developers to trust them. The authors aim to bridge this gap by creating a more detailed, "holistic" prediction model that works across different projects without extensive fine-tuning.
Methods
The authors propose BuildPredictor, a Random Forest-based classifier. The core innovation lies in its feature engineering. Rather than looking at commits as simple "black boxes," the model leverages eight categories of features:
- Commit Metadata: Day of the week, time since the last commit, and contributor type.
- Historical Build Outcomes: Results of the most recent builds.
- Experience History: Fine-grained metrics of a developer's experience with specific code changes.
- Relative Code Complexity: Changes in metrics like Depth of Inheritance Tree (DIT) or Response for a Class (RFC).
- Fine-grained Changes: Specific counts of added/deleted/modified files and source code structures (e.g., method body changes) extracted via tools like ChangeDistiller and BuildDiffer.
The researchers evaluated the model using two strategies: Intra-project (training and testing on the same project) and Holistic (combining data from multiple projects). They also tested a variant called BuildPredictor-NoHist, which intentionally excludes historical build data to see if the model can still perform well using only code and developer attributes.
Datasets
The authors curated a new dataset of 25 Java projects using the Apache Maven build tool. They filtered for active, non-toy projects with a substantial commit history. The final dataset comprises 24,198 commits.
A key characteristic of this dataset is its imbalance: approximately 82% of builds succeed, which reflects real-world conditions but makes prediction more challenging.
Key Insights and Ideas
- The "Naïve Parrot" Effect: The strongest predictors of a build's success are often the results of the previous builds. However, relying solely on history creates a "Naïve Parrot" that just repeats the last result. While accurate, this doesn't help developers understand why a new change might break the build.
- Holistic Power: One of the most significant findings is that a holistic model (trained on many projects) outperforms models trained on isolated projects. This suggests that there are universal patterns in how code changes lead to build failures, regardless of the specific project.
- The MCC Advantage: The authors advocate for using the Matthews Correlation Coefficient (MCC) over standard metrics like F1-score or AUC. In imbalanced datasets (where most builds pass), F1-scores can look deceptively high even if the model fails to catch actual breakages. MCC provides a more robust measure of true predictive power.
- The Ramp-up Phase: When evaluated in a realistic "time-wise" sequence, the model requires a "warm-up" period of roughly 300 to 500 commits before its performance stabilizes and reaches its peak.
Key Results
- High Accuracy: In the holistic setting, BuildPredictor achieved a median MCC of 0.86 and an F1-score of 0.98, significantly outperforming previous state-of-the-art models like BuildFast and RavenBuild-CR.
- Interpretability vs. Performance: When historical features were removed (BuildPredictor-NoHist), the MCC dropped to approximately 0.55–0.60. While lower, this version is far more interpretable, revealing that developer experience and the frequency of changes to specific files become the most critical predictors.
- Stability: The holistic model showed very low variance across different data folds, indicating it is highly stable and reliable across diverse project types.
Limitations and Future Work
The current study is limited to Java projects using Maven. Future work is needed to determine if these patterns hold for other languages (like C++ or Python) and build systems (like Gradle or Bazel).
Additionally, the authors suggest that incorporating "softer" data—such as developer sentiment extracted from commit messages or code smells—could further refine the model's accuracy. Finally, while the model anticipates results, the next logical step is to integrate it directly into CI pipelines to automatically "skip" builds that have a near-100% predicted success rate, thereby directly saving time and money.
Summary updated