Whitening SOA testing

Cesare Bartolini, Antonia Bertolino, Sebastian G. Elbaum, Eda Marchetti

TL;DR

This paper introduces "Service Oriented Coverage Testing" (SOCT), a novel approach to enable white-box testing of Service-Oriented Architectures (SOAs) without compromising their inherent flexibility and loose coupling. SOCT allows integrators to gain visibility into the internal workings of external web services, facilitating more thorough testing and fault detection.

What Problem Does This Paper Address?

Service-Oriented Architectures (SOAs) are increasingly popular for building complex applications by composing independent web services. While this offers flexibility, it poses a significant challenge for testing. Service integrators typically only have access to the service interface, making it difficult to understand how their application interacts with external services and to ensure the overall system's quality. Traditional white-box testing, which relies on internal code structure, is not directly applicable to these black-box external services. This gap leaves integrators unable to detect faults arising from unexercised internal paths or unexpected behavioral changes in third-party services.

Methods

The core of SOCT is the concept of "whitening" external web services. This involves adding an intermediate "coverage service" that allows testers to gain insights into the service's execution without direct access to its source code.

Key components and concepts include:

  • Testable Service: A web service instrumented with "probes" to collect coverage data. These probes are special instructions inserted at targeted locations.
  • SOCT Tester: The service integrator who initiates testing sessions and invokes the Testable Service.
  • TCov (Test Coverage Provider): An organization that deploys services to collect and report coverage information.
  • Testing Session: A bounded set of interactions between the SOCT Tester and the Testable Service for which coverage is collected. Each session is assigned a unique identifier (SID).
  • Test Scope: A mechanism to define a subset of relevant WSDL operations for a Testing Session, allowing coverage measures to be tailored to the specific functionality being tested. This refines the coverage domain to be more meaningful to the tester.
  • Relative Coverage Domain/Measure: A refined notion of coverage that considers how a service is actually used by the client, rather than just the total potential coverage of the entire service.

The paper outlines two primary interaction scenarios for SOCT, differing in how the TCov provider is engaged and whether there's a pre-existing contract between the service provider and TCov.

Datasets

Not discussed in the paper. The paper focuses on a conceptual approach and a preliminary study using a specific web service.

Key Insights and Ideas

  • Bridging the Black-Box/White-Box Divide: SOCT ingeniously allows for white-box testing insights (coverage) into black-box external services, a seemingly contradictory goal.
  • "Whitening" as a Metaphor: The term "whitening" effectively captures the idea of making opaque services more transparent for testing purposes.
  • Test Scope for Meaningful Coverage: The introduction of "Test Scope" is crucial. It moves beyond measuring coverage of the entire service to measuring coverage relevant to the specific operations the integrator is actually using, making the results much more actionable.
  • Decoupling Instrumentation and Testing: The approach separates the responsibility of instrumenting services (by the provider) from the act of testing (by the integrator), allowing both to evolve independently.
  • Flexibility and Loose Coupling Preserved: SOCT is designed to work within the existing SOA paradigm, maintaining flexibility, dynamism, and loose coupling.

Key Results

The paper presents preliminary results from an exploratory study involving the XMLIndexer web service.

  • Demonstrated Feasibility: The study successfully instantiated SOCT and used it to collect block coverage information for test cases executed against the XMLIndexer service.
  • Actionable Insights: The results showed that SOCT could help testers identify redundant tests (e.g., T4 contributing little over T3) and areas where additional testing might be beneficial (e.g., increasing coverage by 16% after adding robustness tests).
  • Relative Coverage Value: The study highlighted that relative coverage provided a more precise estimate (16% higher) compared to absolute coverage, emphasizing the importance of the Test Scope.
  • Identification of Limitations: The study also revealed that even with SOCT, certain blocks (e.g., within catch blocks) might remain difficult to reach without deeper service introspection or specific input generation strategies.

Limitations and Future Work

  • Limitations:
    • The effectiveness of SOCT relies on the willingness of service providers to instrument their services.
  • Discovering faults in deeply nested or exceptionally handled code paths might still be challenging without direct source code access or more advanced instrumentation capabilities.
  • The study was preliminary and involved a single, relatively small web service.
  • Future Work:
    • Conduct larger-scale studies with more complex orchestrated services and industrial collaborators to better understand costs, performance, and overall impact.
  • Refine coverage domain estimation and explore various reachability analyses for relative coverage.
  • Investigate efficient collection of coverage information, potentially leveraging existing work on minimizing probe overhead.
  • Explore how SOCT can be integrated with test case generation methodologies.

Summary updated

BibTeX
@inproceedings{bartolini2009whitening,
  author = {Cesare Bartolini and Antonia Bertolino and Sebastian G. Elbaum and Eda Marchetti},
  title = {{Whitening SOA testing}},
  booktitle = {ESEC/SIGSOFT FSE},
  year = {2009},
  doi = {10.1145/1595696.1595721},
}

Loading related content...

Figure