Test case selection and prioritization using machine learning: a systematic literature review

Rongqi Pan, Mojtaba Bagherzadeh, Taher Ahmed Ghaleb, Lionel C. Briand

TL;DR

This paper presents a systematic literature review of machine learning (ML) techniques for test case selection and prioritization (TSP). It reveals that while ML-based TSP shows promise, current research faces challenges in standardization, reproducibility, and the use of realistic datasets, hindering its practical adoption in continuous integration environments.

What Problem Does This Paper Address?

Regression testing, the process of re-running tests after code changes to ensure no new bugs are introduced, is crucial for software quality. With the rise of Continuous Integration (CI), where builds and tests run frequently, traditional "run-all" regression testing becomes a significant bottleneck due to time and resource constraints. Test Case Selection and Prioritization (TSP) techniques aim to alleviate this by intelligently choosing and ordering test cases to provide faster feedback. While various TSP methods exist, this paper focuses on the growing trend of using Machine Learning (ML) for TSP (ML-based TSP), seeking to understand its current state, effectiveness, and limitations.

Methods

The authors conducted a systematic literature review (SLR) following standard procedures: defining research questions, establishing a search strategy across multiple digital libraries, applying inclusion/exclusion criteria, and synthesizing data. They analyzed 29 primary studies published between 2006 and 2020. The SLR addressed five key research questions: (1) ML techniques used and their pros/cons, (2) features employed, (3) evaluation methods, (4) performance of techniques, and (5) repeatability/reproducibility of studies.

Datasets

The paper discusses the use of various subjects (software projects) for evaluation. It notes that about half of the studies used publicly available datasets like the Software-artifact Infrastructure Repository (SIR), ABB Robotics & GSDTSR, and Defects4J. However, many of these datasets are considered old and may not reflect current CI practices. The authors highlight a need for more recent benchmarks collected from CI contexts. No new datasets were introduced in this paper.

Key Insights and Ideas

  • ML for TSP is a growing trend, especially in CI: The number of studies has significantly increased, particularly after 2018, driven by the need for efficient regression testing in CI.
  • Feature engineering is crucial, but often limited: While code complexity, execution history, and coverage information are commonly used features, researchers are not fully exploiting other potentially rich sources like textual data or user inputs.
  • Reproducibility is a major hurdle: A significant portion of the reviewed studies lack the necessary details (datasets, scripts, clear experimental setups) to be reproduced or even repeated, hindering the scientific community's ability to build upon existing work.
  • Evaluation metrics are inconsistent: The lack of standardized metrics and calculation methods makes direct comparison of TSP techniques across studies challenging. APFD (Average Percentage of Fault Detection) is the most consistently used metric.
  • Real-world applicability is questionable: Many studies are evaluated on subjects with very low failure rates and short execution times, which don't accurately reflect the challenges faced in typical CI environments.

Key Results

  • ML Techniques: The review identified four main categories of ML techniques used for TSP: Supervised Learning (SL), Unsupervised Learning (UL), Reinforcement Learning (RL), and Natural Language Processing (NLP)-based. RL and NLP-based techniques showed promise for CI due to their adaptability.
  • Feature Impact: Statistical analysis indicated that code complexity, coverage information, and execution history features have a statistically significant and large effect on performance (measured by NAPFD). Execution history, in particular, showed a strong positive association with higher performance.
  • Reproducibility: Only 5 out of 29 studies were deemed reproducible, and only 2 were repeatable, highlighting a significant gap in research quality.
  • Performance: Comparing performance across studies is difficult due to varying metrics, datasets, and feature sets. However, studies using execution history features generally reported higher performance.

Limitations and Future Work

The authors acknowledge several limitations:

  • Search Scope: The search was limited to specific online repositories.
  • Subjectivity: Study inclusion and data extraction can involve subjective judgment.
  • Data Extraction Errors: Manual data extraction might lead to missing or misinterpreted information.
  • Dataset Age: Many used datasets are outdated and may not represent current CI practices.
  • Low Failure Rates: Subjects often have very low failure rates, making them unsuitable for robust evaluation.

Future work directions suggested include:

  • Developing comprehensive benchmarks: Creating up-to-date datasets from CI contexts with higher failure rates and longer execution times.
  • Standardizing evaluation: Establishing common evaluation procedures, metrics, and benchmarks.
  • Exploring advanced features: Investigating the use of more comprehensive feature sets, including advanced NLP techniques (e.g., BERT, CodeBERT) for similarity analysis.
  • Improving reproducibility: Encouraging researchers to provide detailed guidelines, scripts, and datasets to facilitate replication.
  • Investigating RL and NLP: Further exploring RL for its adaptability in CI and NLP for its potential in feature extraction and similarity-based TSP.

Summary updated

BibTeX
@article{pan2022test,
  author = {Rongqi Pan and Mojtaba Bagherzadeh and Taher Ahmed Ghaleb and Lionel C. Briand},
  title = {{Test case selection and prioritization using machine learning: a systematic literature review}},
  journal = {Empir. Softw. Eng.},
  year = {2022},
  doi = {10.1007/s10664-021-10066-6},
}
Figure