Automatic Generation of System Test Cases from Use Case Specifications: an NLP-based Approach

Chunhui Wang, Fabrizio Pastore, Arda Goknil, Lionel C. BriandarXiv 2019 · arXiv

TL;DR

This paper introduces UMTG, an approach that automatically generates executable acceptance test cases from natural language use case specifications. By leveraging NLP and constraint solving, UMTG significantly reduces the manual effort and cost associated with testing safety-critical embedded systems, while ensuring better test coverage.

What Problem Does This Paper Address?

Acceptance testing is crucial for safety-critical systems, ensuring they meet functional requirements. However, generating comprehensive acceptance test cases from natural language specifications (like use cases) is a manual, time-consuming, and error-prone process. Existing automated approaches often require complex upfront modeling (e.g., UML diagrams) or restricted natural languages, which are not practical in many industrial settings. This paper addresses the challenge of automating test case generation directly from commonly used, less formal use case specifications, without demanding extensive additional modeling.

Methods

UMTG employs a multi-stage approach:

  • Use Case Elicitation and Modeling: It starts with use case specifications written in a semi-structured format called Restricted Use Case Modeling (RUCM). A domain model (UML class diagram) is also required.
  • NLP for Information Extraction: A Natural Language Processing (NLP) pipeline, including tokenization, part-of-speech tagging, and Semantic Role Labeling (SRL), is used to extract key information from the use case specifications. This includes identifying domain entities, keywords, and conditions.
  • OCL Constraint Generation: UMTG automatically generates Object Constraint Language (OCL) constraints from the extracted natural language conditions. This is a core innovation, leveraging SRL and semantic similarity detection to translate informal requirements into formal constraints.
  • Use Case Test Model (UCTM) Generation: The extracted information and OCL constraints are used to build a UCTM, a directed graph that explicitly represents the control flow of the use case.
  • Scenario and Test Input Generation: UMTG generates use case scenarios (paths through the UCTM) and associated test inputs by solving the OCL constraints using the Alloy analyzer. It supports various coverage criteria (branch, def-use, subtype) to guide scenario generation.
  • Executable Test Case Generation: Finally, UMTG generates executable test cases by mapping high-level operation descriptions and test inputs to concrete test driver functions, using a provided mapping table.

Datasets

The approach was evaluated using two industrial case studies from the automotive domain:

  • BodySense: A safety-critical automotive software for child seat detection and airbag deployment.
  • Hands Off Detection (HOD): An embedded system that monitors driver's hands on the steering wheel.

These systems provided real-world use case specifications and domain models, allowing for evaluation in a practical industrial context.

Key Insights and Ideas

  • NLP for Practicality: The paper demonstrates that advanced NLP techniques, particularly Semantic Role Labeling (SRL), can bridge the gap between informal natural language requirements and formal test specifications, making automation feasible without overly restricting language.
  • OCL as the Intermediate Language: Generating OCL constraints from natural language is a crucial step, providing a formal representation that can be effectively used for test data generation via constraint solving.
  • Coverage-Driven Scenario Generation: The use of distinct coverage criteria (branch, def-use, subtype) allows for tailored test suite generation, ensuring that different aspects of the system's behavior are exercised, and can even uncover scenarios missed by manual testing.
  • Subtype Coverage for Incomplete Specifications: The subtype coverage criterion proved particularly valuable when use case specifications were incomplete, ensuring that various error conditions and input partitions were systematically tested.
  • Industrial Relevance: The approach is designed to integrate with existing industrial practices, relying on commonly used artifacts like use case specifications and domain models, and producing output compatible with test drivers.

Key Results

  • High OCL Constraint Accuracy: UMTG automatically and correctly generated 95% of the required OCL constraints, with a precision of 0.99 and a recall of 0.96 after model improvements.
  • Improved Test Coverage: UMTG test cases exercised more use case scenarios and input partitions than manually generated test cases, especially for bounded/global alternative flows and when specifications were incomplete.
  • Reduced Effort and Time: The automated generation process significantly reduced the effort and time required for test case generation compared to manual methods, which took days of human effort, while UMTG required hours of computation and minimal manual review.
  • Scalability: Despite theoretical worst-case complexities, the approach proved feasible in practice for realistic industrial systems due to the nature of use case specifications and modern computing power.

Limitations and Future Work

  • Threat to Validity (Generalizability): The authors acknowledge that the study's generalizability might be limited by its focus on two automotive embedded systems. However, they argue that the systems represent diverse functionalities and were developed with different practices.
  • OCL Constraint Limitations: The approach primarily handles simple OCL constraints related to system state. More complex constraints or those involving intricate temporal logic might require manual intervention.
  • NLP Heuristics: The reliance on NLP means that the approach is heuristic-based, and while empirically validated, a formal proof of soundness is not possible.
  • Future Work: The authors suggest extending UMTG to handle different types of testing problems (e.g., security testing), improving scalability for very large systems, and exploring advanced solving approaches like SMT solvers and answer set programming. They also aim to investigate the use of model transformation by example for learning mapping tables.

Summary updated

BibTeX
@misc{wang2019automatic,
  author = {Chunhui Wang and Fabrizio Pastore and Arda Goknil and Lionel C. Briand},
  title = {{Automatic Generation of System Test Cases from Use Case Specifications: an NLP-based Approach}},
  year = {2019},
  eprint = {1907.08490},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.08490},
}

Loading related content...

Figure