Categorizing Defects in Infrastructure as Code

Akond Rahman, Sarah Elder, Faysal Hossain Shezan, Vanessa Frost, Jonathan Stallings, Laurie A. WilliamsarXiv 2018 · OpenReview

TL;DR

This paper presents a comprehensive analysis of defects found in Infrastructure as Code (IaC) scripts, categorizing them using the Orthogonal Defect Classification (ODC) framework. The study reveals that "assignment" defects are the most prevalent, suggesting a need for increased focus on code inspection and unit testing in IaC development.

What Problem Does This Paper Address?

Infrastructure as Code (IaC) is becoming increasingly vital for automating software deployment and management. However, IaC scripts can be complex and prone to defects, leading to significant system disruptions. Despite the growing popularity of IaC, there's a lack of systematic understanding regarding the types of defects that occur in these scripts. This paper aims to fill that gap by categorizing IaC defects and providing actionable recommendations for improving IaC development processes.

Methods

The authors employed a qualitative analysis approach to categorize defects. They collected commit messages and issue reports related to IaC scripts from four organizations. These messages were then processed into "extended commit messages" (XCMs). A team of 89 raters, with software engineering experience, categorized these XCMs using the defect type attribute of the Orthogonal Defect Classification (ODC) framework. This framework includes eight standard defect categories, plus "No Defect" and "Other" categories. To ensure reliability, each XCM was reviewed by at least two raters, and disagreements were resolved by the first author. The study also compared the defect category distribution of IaC scripts with that of 26 non-IaC software systems from existing literature.

Datasets

The study utilized datasets from four organizations: Mirantis, Mozilla, Openstack, and Wikimedia Commons. These datasets were constructed from open-source repositories containing Puppet scripts.

  • Mirantis: 20 repositories, 165 Puppet scripts, 1,021 commits.
  • Mozilla: 2 repositories, 580 Puppet scripts, 3,074 commits.
  • Openstack: 61 repositories, 1,383 Puppet scripts, 7,808 commits.
  • Wikimedia Commons: 11 repositories, 296 Puppet scripts, 972 commits.

The paper also references a collection of 26 non-IaC software systems from prior research for comparative analysis.

Key Insights and Ideas

  • Assignment Defects Dominate: The most striking finding is the overwhelming prevalence of "assignment" defects in IaC scripts. These defects, related to syntax errors and incorrect configuration values, account for a significant majority of issues across all datasets.
  • IaC vs. Non-IaC Differences: IaC scripts exhibit a different defect category distribution compared to non-IaC software. Specifically, assignment-related defects are more frequent in IaC.
  • Consistent Temporal Trend: Unlike hardware systems that often show a "bathtub curve" of defects, IaC defects do not exhibit a significant temporal trend (increasing or decreasing) over time, suggesting a consistent level of defect introduction or resolution.
  • Challenges in Categorization: While raters agreed on whether a commit was defect-related, they sometimes disagreed on the specific defect category. This highlights the potential ambiguity in commit messages and the need for more detailed information.

Key Results

  • Defect Density: The defect density (defects per 1000 lines of code) varied across organizations, with Mirantis showing the highest at 27.6 DDKLOC and Openstack the lowest at 16.2 DDKLOC.
  • Dominant Defect Category: "Assignment" defects were the most frequent, ranging from 36.5% (Mozilla) to 62.7% (Wikimedia).
  • Temporal Trend: Cox-Stuart tests indicated consistent temporal trends for defects per month across all datasets, meaning defect frequency did not significantly change over time.
  • Practitioner Agreement: The study found substantial to almost perfect agreement between the raters' categorization and practitioners' feedback, validating the methodology.

Limitations and Future Work

  • Subjectivity of Qualitative Analysis: The reliance on human raters for categorization introduces subjectivity, though this was mitigated by using multiple raters and a resolver.
  • Commit Message Information: The study acknowledges that commit messages may not always provide sufficient context for accurate defect categorization, leading to a higher "Other" category in some instances.
  • ODC Trigger Attribute: The "trigger" attribute of ODC (e.g., design conformance, side effects) was not used due to a lack of information in the data sources.
  • Declarative vs. Imperative IaC: The study focused on Puppet, a declarative language. Future work could investigate imperative IaC languages to see if defect patterns differ.
  • Automation of Categorization: The authors suggest future research into automating the ODC categorization process, which is currently manual and labor-intensive.
  • Causal Analysis: Further investigation into the root causes of observed IaC defects is recommended.

Summary updated

BibTeX
@misc{rahman2018categorizing,
  author = {Akond Rahman and Sarah Elder and Faysal Hossain Shezan and Vanessa Frost and Jonathan Stallings and Laurie A. Williams},
  title = {{Categorizing Defects in Infrastructure as Code}},
  year = {2018},
  url = {https://openreview.net/forum?id=ZvkkDfXLdS},
}

Loading related content...

Figure