Abstract
This paper presents KITSS (Knowledge-based Interactive Test Scripting System), a framework designed to bridge the gap between informal English requirements and formal, executable test scripts for complex reactive systems. By integrating a trainable statistical parser with a logic-based "story understander," KITSS automates the translation of real-world test scenarios. Evaluation on a corpus of telephony scenarios demonstrates a 70% success rate, significantly reducing the manual effort required for test script generation from hours to minutes.
Introduction
In the development of complex reactive systems, such as Private Branch Exchanges (PBX), engineers typically define requirements using "scenarios." These are stylized English prose descriptions of intended system behavior—for example, "Station A calls Station B, but B is busy, so the call goes to coverage." While intuitive for humans, these scenarios are informal, often underspecified, and frequently contain errors.
To validate software, these scenarios must be converted into rigid, executable test scripts. Traditionally, this manual conversion is a significant bottleneck, often requiring several hours of a test engineer's time per scenario. KITSS was developed to automate this translation by bridging the wide abstraction gap between human intent and machine execution.
System Architecture
KITSS employs a two-stage pipeline to transform English prose into executable logic:
1. Translation to Interlingua
The first stage involves converting natural language into a logic-based interlingua known as WIL.
- Statistical Chart-Parsing: Unlike traditional systems that rely on hand-crafted grammars, KITSS utilizes a trainable statistical parser. This parser learns from example "sentence diagrams" provided by engineers, allowing for easier maintenance and adaptation by domain experts who may lack formal linguistics training.
- Rule-Based Phrase Conversion: The parser's output is processed to handle "normalization" (e.g., ensuring "Station B1" and "the first station" refer to the same entity) and to map specific verbs to discrete system actions.
2. Scenario Analysis (The "Story Understander")
The core innovation of KITSS is its ability to handle the underspecification inherent in human language. A direct translation is often insufficient because scenarios frequently omit necessary intermediate steps.
- Theorem Proving: Using a background theory of telephony consisting of 250 logic axioms, the system audits each step to ensure it is logically possible within the system's state space.
- Plan Recognition: The system compares the scenario against a library of approximately 70 stereotypical plans (e.g., "making a call"). If a scenario is missing steps—such as forgetting to pick up a handset before dialing—the planner "fleshes out" the missing initialization conditions and intermediate actions required to create a functional test.
Experimental Evaluation
The researchers evaluated KITSS using a graduated corpus of 100 behavior scenarios. These scenarios covered seven advanced calling features for a PBX and were authored by five different engineers to ensure a variety of styles and vocabularies. These documents represented actual "informal contracts" used within AT&T engineering workflows.
Key Results
- Success Rate: KITSS successfully translated 70% of the test corpus into executable scripts with zero or minimal manual intervention.
- Efficiency: For successful translations, the process was reduced from several hours to just a few minutes.
- Parser Accuracy: The statistical parser demonstrated a 77% probability of correctly parsing novel sentences and a 96% success rate in selecting the correct parse when a valid parse was available.
- Domain Modeling: The project resulted in a comprehensive, machine-interpretable formal model of modern telephony, addressing the complexities of features like call forwarding and bridging.
Discussion and Insights
The development of KITSS revealed that the abstraction gap in reactive systems is remarkably wide; a single phrase like "Place a call" involves multiple discrete steps (off-hook, mapping extensions, choosing a dialing method) that the system must infer.
To maintain usability, the authors implemented a "credulous mode" for the logic analyzer. When the system encounters a scenario it cannot fully prove or understand, it temporarily suspends auditing to accept the user's input until it can resynchronize with a known plan. This keeps the human engineer in the loop rather than stalling the automation process.
Limitations and Future Work
Despite its success, KITSS faced a "semantic bottleneck." While the statistical parser was robust, the phrase converter required new rules for almost every new idiom encountered, limiting its scalability. Additionally, the logic analyzer proved to be the primary computational bottleneck and the source of most fatal bugs. Future research is required to scale reasoning about actions and states for industrial-sized problems and to better handle the variability in human input when manual corrections are necessary.
Conclusion
KITSS demonstrates that combining statistical natural language processing with deep domain-specific logic can effectively automate the generation of test scripts. By reducing the time required for scenario translation from hours to minutes, the system provides a scalable path for verifying the behavior of complex, reactive software systems.
Summary updated