Empirical assessment of generating adversarial configurations for software product lines

Paul Temple, Gilles Perrouin, Mathieu Acher, Battista Biggio, Jean-Marc Jézéquel, Fabio RoliEmpir. Softw. Eng 2021 · DOI · OpenReview

TL;DR

This paper introduces adversarial machine learning (advML) techniques to the domain of Software Product Lines (SPLs) to improve the quality assurance of ML-based SPLs. By generating "adversarial configurations" that fool ML classifiers, the authors demonstrate a novel way to identify weaknesses in SPLs and enhance their robustness.

What Problem Does This Paper Address?

Software Product Line (SPL) engineering aims to create a vast array of customized products from a common set of features. However, the sheer size of the configuration space makes it impossible to test every possible product. Machine learning (ML) is used to approximate the set of "acceptable" products, but ML classifiers can make errors, leading to wasted resources on non-acceptable products. This paper addresses the challenge of improving the reliability of ML-based SPLs by using adversarial machine learning to identify and diagnose these prediction errors.

Methods

The authors adapt adversarial machine learning (advML) techniques, specifically evasion attacks, to the SPL domain. They develop two adversarial configuration generators: a dedicated algorithm and one based on the secML library. These generators aim to create configurations that are misclassified by an ML model while still adhering to the logical constraints of the SPL's variability model. The core idea is to "attack" the ML classifier by generating configurations that are just on the wrong side of the decision boundary, thus revealing potential weaknesses.

Datasets

The study uses two case studies:

  • MOTIV video generator: An industrial video synthesizer with over 100 configuration options, resulting in approximately 10^314 possible configurations.
  • JHipster: An open-source web application configurator with 58 features and 90,210 variants.

Key Insights and Ideas

  • advML is applicable to SPLs: The paper demonstrates that advML techniques, originally developed for domains like image recognition, can be effectively adapted to the SPL context.
  • Adversarial configurations reveal classifier weaknesses: By generating configurations that fool the ML classifier, the authors can pinpoint areas where the classifier is uncertain or makes errors, leading to the generation of non-acceptable products.
  • Constraints are crucial: A key challenge is ensuring that generated adversarial configurations remain valid according to the SPL's variability model. The authors developed methods to enforce these constraints, ensuring that the generated configurations are not only adversarial but also logically sound.
  • Improved quality assurance: The work lays the groundwork for a new quality assurance framework for ML-based SPLs, enabling developers to identify and address potential issues more effectively.
  • Impact on training data: Adding adversarial configurations to the training set can, in some cases, improve classifier performance, suggesting a potential for "adversarial training" in SPLs.

Key Results

  • 100% Misclassification Rate: The adversarial generators achieved up to a 100% misclassification rate on acceptable configurations, meaning they could consistently fool the classifier into labeling them as non-acceptable.
  • Valid Configurations: Crucially, all generated adversarial configurations were valid with respect to the SPL's logical constraints.
  • Effectiveness of Evasion Attacks: Evasion attacks were found to be significantly more effective than random modifications in generating misclassified configurations.
  • Impact on Training: Adding a small number of adversarial configurations (25) to the training set could improve classifier accuracy by up to 2-3% in balanced datasets, though the impact was less pronounced in already high-performing classifiers.

Limitations and Future Work

  • Parameter Tuning: Finding optimal parameter values for advML techniques (step size, displacement limits) requires careful tuning and can impact computation time.
  • Constraint Handling: While the authors developed ad-hoc solutions for constraint handling, a more systematic approach using constraint solvers is suggested.
  • Number of Adversarial Configurations: The impact of adding adversarial configurations to the training set was observed with a small number (25). Future work could explore the effect of larger numbers and the risk of degrading performance.
  • Other Attack Types: The study focused on evasion attacks; exploring other types of adversarial attacks could provide further insights.
  • Generalization: While two case studies were used, further investigation with more diverse SPLs would strengthen the generalization of the findings.
  • Adversarial Training: The authors suggest exploring other defense mechanisms and adapting advML knowledge for "friendly" attacks that improve learning accuracy rather than just triggering misclassifications.
  • Testing Oracle Improvement: Adversarial configurations can be used to identify weaknesses in testing oracles, leading to their improvement.
  • Variability Model and Implementation: The work suggests that adversarial configurations can also provide insights into improving the variability model and its implementation.

Summary updated

Abstract

Software product line (SPL) engineering allows the derivation of products tailored to stakeholders’ needs through the setting of a large number of configuration options. Unfortunately, options and their interactions create a huge configuration space which is either intractable or too costly to explore exhaustively. Instead of covering all products, machine learning (ML) approximates the set of acceptable products (e.g., successful builds, passing tests) out of a training set (a sample of configurations). However, ML techniques can make prediction errors yielding non-acceptable products wasting time, energy and other resources. We apply adversarial machine learning techniques to the world of SPLs and craft new configurations faking to be acceptable configurations but that are not and vice-versa. It allows to diagnose prediction errors and take appropriate actions. We develop two adversarial configuration generators on top of state-of-the-art attack algorithms and capable of synthesizing configurations that are both adversarial and conform to logical constraints. We empirically assess our generators within two case studies: an industrial video synthesizer (MOTIV) and an industry-strength, open-source Web-app configurator (JHipster). For the two cases, our attacks yield (up to) a 100% misclassification rate without sacrificing the logical validity of adversarial configurations. This work lays the foundations of a quality assurance framework for ML-based SPLs.
BibTeX
@article{temple2021empirical,
  author = {Paul Temple and Gilles Perrouin and Mathieu Acher and Battista Biggio and Jean-Marc Jézéquel and Fabio Roli},
  title = {{Empirical assessment of generating adversarial configurations for software product lines}},
  journal = {Empir. Softw. Eng.},
  year = {2021},
  doi = {10.1007/s10664-020-09915-7},
}
Figure