Test Pair Selection for Test Case Prioritization in Regression Testing for WS-BPEL Programs

Lijun Mei, Yan Cai, Changjiang Jia, Bo Jiang, W. K. ChanInt. J. Web Serv. Res 2013 · DOI

TL;DR

This paper introduces a novel approach to regression testing for WS-BPEL web services by prioritizing test cases based on the structural similarity of their associated XML artifacts. By analyzing WSDL specifications, XML messages, and workflow modules, their techniques aim to more effectively detect faults earlier in the testing cycle, leading to more efficient and robust web service evolution.

What Problem Does This Paper Address?

Web services, particularly those built with WS-BPEL, are complex and dynamic. They communicate using XML messages, and their behavior can change through updates to their interface specifications (like XML Schemas). This dynamism makes regression testing – ensuring that changes haven't broken existing functionality – a significant challenge. Traditional test case prioritization methods often treat test cases generically, failing to leverage the rich structural information embedded within the XML artifacts that define these services. This can lead to less effective fault detection and a slower feedback loop for developers. The authors aim to address this by developing prioritization techniques that specifically exploit the structural similarities within these XML-based components.

Methods

The core of this paper lies in its novel pairwise test case selection strategy for prioritization. Instead of considering test cases individually, they analyze pairs of test cases to determine their similarity. This similarity is measured across three levels:

  1. WSDL Specification Similarity (W-I): Compares the WSDL documents associated with test cases.
  2. WSDL Specification and XML Message Similarity (W-II): Extends W-I by also considering the XML messages exchanged during test case execution.
  3. WSDL Specification, XML Message, and Workflow Module Similarity (W-III): The most comprehensive level, incorporating WSDL, XML messages, and the workflow modules (e.g., BPEL activities) exercised by the test cases.

They define a W3-Set to represent the information (WSDL, messages, workflow modules) associated with a test case. Using a Geometric Mean (GM), they calculate similarity scores between pairs of test cases for each of the three levels. Based on these similarity scores, they propose six prioritization techniques (M1-M6). These techniques group test case pairs by similarity and then select pairs in a specific order (e.g., most similar first, least similar first, or iteratively). They also compare their methods against traditional coverage-based techniques (C1-C4).

Datasets

The authors use a set of eight WS-BPEL applications as benchmarks. These applications were chosen because they have served as benchmarks in previous research and are representative of typical WS-BPEL programs. For each application, they generated random test suites that cover all workflow branches, XRG branches, and WSDL elements of the original version. They also used a set of known faults introduced in modified versions of these applications to evaluate the effectiveness of their prioritization techniques.

Key Insights and Ideas

  • Pairwise similarity is key: The paper argues that analyzing pairs of test cases for structural similarity provides richer information than individual test case analysis, especially for complex XML-based systems.
  • XML structure matters: Simply looking at static WSDL elements isn't enough to distinguish between test cases that might behave identically in practice. Incorporating dynamic data (XML messages) is crucial for effective differentiation.
  • Levels of similarity yield different results: The effectiveness of prioritization techniques can vary significantly depending on which level of XML artifact similarity (WSDL only, WSDL+messages, or WSDL+messages+workflow) is used.
  • Iterative and refined selection strategies show promise: Techniques that iteratively select test case pairs (M3-M6) and consider relationships between selected pairs tend to outperform simpler selection strategies (M1-M2) and traditional coverage-based methods.
  • Combining static and dynamic information is beneficial: The most comprehensive similarity metric (W-III), which includes WSDL, XML messages, and workflow modules, generally leads to better prioritization results.

Key Results

The empirical evaluation demonstrated that their proposed pairwise selection techniques (M3-M6) can be more effective than conventional coverage-based techniques (C1-C4) and random ordering in terms of Average Percentage of Faults Detected (APFD). Specifically:

  • Techniques M3-M6 using the most comprehensive similarity metric (W-III) showed a significant upward trend in APFD.
  • The pairwise selection strategy, especially when combined with richer similarity metrics (W-II and W-III), led to better fault detection rates compared to traditional methods.
  • The results indicated that incorporating both static (WSDL) and dynamic (XML messages) information significantly improves prioritization effectiveness.
  • The proposed techniques were found to be robust even when a small percentage of XML messages were lost.

Limitations and Future Work

  • Computational Complexity: The pairwise selection strategy, while effective, is noted as being NP-complete, suggesting that further optimization or approximation approaches could be beneficial.
  • Generalizability: While the study uses WS-BPEL programs as a representative example, the authors suggest that their techniques can be applied to other applications that manipulate XML documents. However, they acknowledge that the results might not generalize to scenarios with strict casual constraints between test cases.
  • Test Case Generation: The study relies on generated test suites. Future work could explore how their techniques perform with real-world, manually created test suites.
  • Self-Adaptation Cycle: The current work addresses a part of the self-adaptation cycle for web services. A more comprehensive study incorporating other aspects of self-adaptation is needed.
  • N-way Selection: The paper focuses on pairwise selection. Exploring n-way selection strategies could be a future direction.

Summary updated

BibTeX
@article{mei2013test,
  author = {Lijun Mei and Yan Cai and Changjiang Jia and Bo Jiang and W. K. Chan},
  title = {{Test Pair Selection for Test Case Prioritization in Regression Testing for WS-BPEL Programs}},
  journal = {Int. J. Web Serv. Res.},
  year = {2013},
  doi = {10.4018/jwsr.2013010104},
}

Loading related content...

Figure