TL;DR
This paper introduces VeriCI, a novel tool that statically analyzes Continuous Integration (CI) configuration files to predict build failures before a commit is pushed. By learning from historical build data, VeriCI achieves 83% accuracy in predicting failures and provides explanations, significantly reducing developer time wasted on debugging CI issues.
What Problem Does This Paper Address?
Continuous Integration (CI) is a cornerstone of modern software development, enabling rapid feedback on code changes. However, CI systems rely on complex configuration files that, when incorrect, can lead to lengthy build failures. Debugging these configuration errors is particularly challenging because developers often lack direct access to the diverse environments where CI builds run. The only recourse has been to push a commit and wait for the CI server's verdict, a process that can consume valuable developer time and delay releases. This paper tackles the problem of proactively identifying and explaining CI configuration errors before they cause build failures.
Methods
The core of VeriCI is a machine learning approach that learns a model of CI configuration correctness. The key insight is that historical build data within a repository (commits, configuration files, and build statuses) provides a rich source of information.
VeriCI employs a Misclassification Guided Abstraction Refinement (MiGAR) loop, inspired by the CEGAR (Counterexample-Guided Abstraction Refinement) loop used in formal verification. Unlike CEGAR which uses counterexamples, MiGAR uses misclassifications to drive the refinement of feature extraction. This process iteratively improves the set of "code features" (representations of code and configuration) used to train the model.
The system uses decision tree learning to build a classifier. Decision trees are chosen for their transparency, allowing VeriCI to provide justifications for its predictions. The learning process is divided into two stages:
- Global Learning: Learns general rules applicable across multiple repositories.
- Local Learning: Learns repository-specific rules, adapting to unique project constraints.
A crucial component is Abstraction Based Relabelling (ABR), which addresses the "impurity" of CI builds (failures due to external factors like network issues). ABR relabels build statuses to ensure that identical code features consistently map to the same build outcome, thereby reducing noise in the training data.
Datasets
The primary dataset used is the TravisTorrent dataset. This dataset comprises historical build logs from a large number of open-source projects hosted on GitHub and using TravisCI. The data is naturally ordered by commit history, providing the time-aware repository build status information that VeriCI leverages. The authors also mention evaluating VeriCI on 33 large repositories from GitHub.
Key Insights and Ideas
- CI build history is a goldmine for learning configuration correctness: The paper's central insight is that the time-series data of commits, configurations, and build outcomes within a repository is sufficient to learn what constitutes a "correct" CI configuration.
- Misclassifications are powerful drivers for learning: Adapting the CEGAR concept, using misclassifications to refine feature extraction is more effective than relying solely on explicit counterexamples for learning CI configuration patterns.
- Global and Local learning are complementary: Some CI errors are common across projects (e.g., library version conflicts), while others are highly project-specific. VeriCI's dual-stage learning approach captures both, leading to more robust predictions.
- Abstraction Based Relabelling (ABR) is key to handling noisy CI data: CI builds can fail for reasons outside the code's direct control (network issues, dependency problems). ABR helps clean the training data by ensuring that similar code states lead to consistent build outcomes, making the learning process more reliable.
- Decision trees offer explainability: The choice of decision trees allows VeriCI to not only predict failures but also provide understandable justifications, guiding developers on why a failure is predicted.
Key Results
- 83% accuracy in predicting build failures: VeriCI achieves a significant accuracy rate in predicting whether a CI build will fail before it actually runs.
- Explainable predictions: VeriCI provides justifications for its predictions, identifying keywords and locations in the code that are likely contributing to the failure.
- Error message accuracy: In 38%-48% of cases, the justification provided by VeriCI matched the actual fix made by the developer, indicating that the tool can pinpoint the root cause of errors.
- Low false positive rate: The paper reports a false positive rate of 1:20, which is crucial for user adoption as it minimizes unnecessary alarms.
- Scalability: The learning process scales roughly linearly with the size and complexity of the training set, suggesting it can handle large datasets.
Limitations and Future Work
- Impure Errors: VeriCI cannot detect errors caused by external factors like network failures or dependency issues, as its analysis is limited to the code and configuration within the repository.
- False Positives: While the false positive rate is low, it's still an area for improvement, and the authors plan to conduct user studies to determine acceptable tolerance levels.
- Optimization: The current implementation prioritizes correctness over speed. Future work will focus on optimizing training times to handle even larger datasets and potentially millions of commits.
- Integration into Workflow: The authors plan to deploy VeriCI in long-term studies to understand its impact on developer workflows and investigate the best ways to present error messages.
- Combining with Metadata: Future work could explore combining VeriCI's code-feature-based predictions with metadata-based approaches for even higher accuracy.
Summary updated