Statically Verifying Continuous Integration Configurations

Mark Santolucito, Jialu Zhang, Ennan Zhai, Ruzica PiskacarXiv 2018 · arXiv · OpenReview

TL;DR

This paper introduces VeriCI, a novel tool that statically analyzes Continuous Integration (CI) configuration files to predict build failures before a commit is pushed. By learning from historical build data, VeriCI achieves 83% accuracy in predicting failures and provides explanations, significantly reducing developer time wasted on debugging CI issues.

What Problem Does This Paper Address?

Continuous Integration (CI) is a cornerstone of modern software development, enabling rapid feedback on code changes. However, CI systems rely on complex configuration files that, when incorrect, can lead to lengthy build failures. Debugging these configuration errors is particularly challenging because developers often lack direct access to the diverse environments where CI builds run. The only recourse has been to push a commit and wait for the CI server's verdict, a process that can consume valuable developer time and delay releases. This paper tackles the problem of proactively identifying and explaining CI configuration errors before they cause build failures.

Methods

The core of VeriCI is a machine learning approach that learns a model of CI configuration correctness. The key insight is that historical build data within a repository (commits, configuration files, and build statuses) provides a rich source of information.

VeriCI employs a Misclassification Guided Abstraction Refinement (MiGAR) loop, inspired by the CEGAR (Counterexample-Guided Abstraction Refinement) loop used in formal verification. Unlike CEGAR which uses counterexamples, MiGAR uses misclassifications to drive the refinement of feature extraction. This process iteratively improves the set of "code features" (representations of code and configuration) used to train the model.

The system uses decision tree learning to build a classifier. Decision trees are chosen for their transparency, allowing VeriCI to provide justifications for its predictions. The learning process is divided into two stages:

  1. Global Learning: Learns general rules applicable across multiple repositories.
  2. Local Learning: Learns repository-specific rules, adapting to unique project constraints.

A crucial component is Abstraction Based Relabelling (ABR), which addresses the "impurity" of CI builds (failures due to external factors like network issues). ABR relabels build statuses to ensure that identical code features consistently map to the same build outcome, thereby reducing noise in the training data.

Datasets

The primary dataset used is the TravisTorrent dataset. This dataset comprises historical build logs from a large number of open-source projects hosted on GitHub and using TravisCI. The data is naturally ordered by commit history, providing the time-aware repository build status information that VeriCI leverages. The authors also mention evaluating VeriCI on 33 large repositories from GitHub.

Key Insights and Ideas

  • CI build history is a goldmine for learning configuration correctness: The paper's central insight is that the time-series data of commits, configurations, and build outcomes within a repository is sufficient to learn what constitutes a "correct" CI configuration.
  • Misclassifications are powerful drivers for learning: Adapting the CEGAR concept, using misclassifications to refine feature extraction is more effective than relying solely on explicit counterexamples for learning CI configuration patterns.
  • Global and Local learning are complementary: Some CI errors are common across projects (e.g., library version conflicts), while others are highly project-specific. VeriCI's dual-stage learning approach captures both, leading to more robust predictions.
  • Abstraction Based Relabelling (ABR) is key to handling noisy CI data: CI builds can fail for reasons outside the code's direct control (network issues, dependency problems). ABR helps clean the training data by ensuring that similar code states lead to consistent build outcomes, making the learning process more reliable.
  • Decision trees offer explainability: The choice of decision trees allows VeriCI to not only predict failures but also provide understandable justifications, guiding developers on why a failure is predicted.

Key Results

  • 83% accuracy in predicting build failures: VeriCI achieves a significant accuracy rate in predicting whether a CI build will fail before it actually runs.
  • Explainable predictions: VeriCI provides justifications for its predictions, identifying keywords and locations in the code that are likely contributing to the failure.
  • Error message accuracy: In 38%-48% of cases, the justification provided by VeriCI matched the actual fix made by the developer, indicating that the tool can pinpoint the root cause of errors.
  • Low false positive rate: The paper reports a false positive rate of 1:20, which is crucial for user adoption as it minimizes unnecessary alarms.
  • Scalability: The learning process scales roughly linearly with the size and complexity of the training set, suggesting it can handle large datasets.

Limitations and Future Work

  • Impure Errors: VeriCI cannot detect errors caused by external factors like network failures or dependency issues, as its analysis is limited to the code and configuration within the repository.
  • False Positives: While the false positive rate is low, it's still an area for improvement, and the authors plan to conduct user studies to determine acceptable tolerance levels.
  • Optimization: The current implementation prioritizes correctness over speed. Future work will focus on optimizing training times to handle even larger datasets and potentially millions of commits.
  • Integration into Workflow: The authors plan to deploy VeriCI in long-term studies to understand its impact on developer workflows and investigate the best ways to present error messages.
  • Combining with Metadata: Future work could explore combining VeriCI's code-feature-based predictions with metadata-based approaches for even higher accuracy.

Summary updated

Abstract

Continuous Integration (CI) testing is a popular software development technique that allows developers to easily check that their code can build successfully and pass tests across various system environments. In order to use a CI platform, a developer must include a set of configuration files to a code repository for specifying build conditions. Incorrect configuration settings lead to CI build failures, which can take hours to run, wasting valuable developer time and delaying product release dates. Debugging CI configurations is challenging because users must manage configurations for the build across many system environments, to which they may not have local access. Thus, the only way to check a CI configuration is to push a commit and wait for the build result. To address this problem, we present the first approach, VeriCI, for statically checking for errors in a given CI configuration before the developer pushes a commit to build on the CI server. Our key insight is that the repositories in a CI environment contain lists of build histories which offer the time-aware repository build status. Driven by this insight, we introduce the Misclassification Guided Abstraction Refinement (MiGAR) loop that automates part of the learning process across the heterogeneous build environments in CI. We then use decision tree learning to generate constraints on the CI configuration that must hold for a build to succeed by training on a large history of continuous integration repository build results. We evaluate VeriCI on real-world data from GitHub and find that we have 83% accuracy of predicting a build failure.
BibTeX
@misc{santolucito2018statically,
  author = {Mark Santolucito and JIALU ZHANG and Ennan Zhai and Ruzica Piskac},
  title = {{Statically Verifying Continuous Integration Configurations}},
  year = {2018},
  eprint = {1805.04473},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1805.04473},
}

Loading related content...

Figure