Model-Enhanced LLM-Driven VUI Testing of VPA Apps

Suwan Li, Lei Bu, Guangdong Bai, Fuman Xie, Kai Chen, Chang YuearXiv 2024 · DOI · arXiv

Abstract

Voice Personal Assistant (VPA) applications, such as Amazon Alexa skills, present unique testing challenges due to their "invisible" Voice User Interfaces (VUIs). Traditional testing methods often struggle with semantic redundancy or lack the structured memory needed for deep exploration. This paper introduces Elevate, a framework that integrates the semantic reasoning of Large Language Models (LLMs) with the structural rigor of Model-Based Testing (MBT). By dynamically constructing a Finite State Machine (FSM) through LLM-interpreted interactions, Elevate achieves 15% higher state-space coverage and significantly improved testing efficiency over previous state-of-the-art tools.

Introduction

The proliferation of VPA apps has made ensuring their security, privacy, and quality a critical task. However, testing VUIs is notoriously difficult. Existing approaches typically fall into two categories: chatbot-style testing, which lacks a formal map of explored states, and traditional Model-Based Testing (MBT), which often fails to bridge the "semantic gap." Traditional MBT tools might treat semantically identical responses (e.g., "How can I help?" vs. "What do you want to do?") as distinct states, leading to state-space explosion and inefficient exploration.

Figure 3: Semantic redundancy and repeated states in VUI interactions often lead to inefficient testing in traditional models.

The Elevate Framework

Elevate addresses these challenges by using LLMs to interpret the meaning behind voice interactions, allowing for precise and efficient exploration of an app's logic. The framework operates in three recurring phases:

  1. States Extraction: Upon receiving an app response, an LLM determines if it represents a new functional state or is semantically equivalent to an existing one, effectively "compressing" the search space.
  2. Input Events Generation: The LLM generates context-relevant user inputs based on the current state.
  3. State Space Exploration: The LLM acts as a navigator, selecting inputs most likely to discover unexplored states based on the constructed model and transition history.
Figure 4: The architecture of the Elevate framework, highlighting the three-phase loop of state extraction, input generation, and exploration.

A key component of Elevate is its automated feedback loop. To mitigate LLM hallucinations or invalid inputs, "checkers" validate suggestions. If an input is misunderstood by the app or leads to repetitive states, a "Feedback Prompt" is issued to refine the LLM's strategy.

Evaluation and Results

The researchers evaluated Elevate using a large-scale dataset of 4,000 real-world Amazon Alexa skills and an intensive benchmark of 50 stable skills. Elevate was compared against Vitas (a state-of-the-art model-based tester) and a standalone GPT-4 chatbot.

Key findings include:

  • Superior Coverage: Elevate achieved 15% higher state-space coverage than Vitas across the 4,000-app dataset.
  • High Efficiency: In intensive benchmarks, Elevate reached over 80% average semantic state coverage within 20 interaction rounds, while Vitas plateaued at 68% and the pure GPT-4 chatbot reached only 45%.
  • Model Agnostic: The framework proved effective using both GPT-4 and Llama2-70b, demonstrating its versatility across different LLM architectures.

Discussion

Elevate demonstrates that voice app state spaces are often smaller than they appear when semantic compression is applied. Furthermore, it highlights that even powerful LLMs require a formal "map" (the FSM) to avoid getting lost in conversational loops and to reach deep functionalities.

While effective, Elevate faces challenges common to LLM-based systems, including non-determinism and occasional persistent hallucinations. Additionally, the cost and latency associated with high-end LLMs suggest that future work should focus on distilling these capabilities into smaller, specialized models to enhance scalability and reduce overhead.

Summary updated

Abstract

The flourishing ecosystem centered around voice personal assistants (VPA), such as Amazon Alexa, has led to the booming of VPA apps. The largest app market Amazon skills store, for example, hosts over 200,000 apps. Despite their popularity, the open nature of app release and the easy accessibility of apps also raise significant concerns regarding security, privacy and quality. Consequently, various testing approaches have been proposed to systematically examine VPA app behaviors. To tackle the inherent lack of a visible user interface in the VPA app, two strategies are employed during testing, i.e., chatbot-style testing and model-based testing. The former often lacks effective guidance for expanding its search space, while the latter falls short in interpreting the semantics of conversations to construct precise and comprehensive behavior models for apps. In this work, we introduce Elevate, a model-enhanced large language model (LLM)-driven VUI testing framework. Elevate leverages LLMs' strong capability in natural language processing to compensate for semantic information loss during model-based VUI testing. It operates by prompting LLMs to extract states from VPA apps' outputs and generate context-related inputs. During the automatic interactions with the app, it incrementally constructs the behavior model, which facilitates the LLM in generating inputs that are highly likely to discover new states. Elevate bridges the LLM and the behavior model with innovative techniques such as encoding behavior model into prompts and selecting LLM-generated inputs based on the context relevance. Elevate is benchmarked on 4,000 real-world Alexa skills, against the state-of-the-art tester Vitas. It achieves 15% higher state space coverage compared to Vitas on all types of apps, and exhibits significant advancement in efficiency.
BibTeX
@misc{li2024model,
  author = {Suwan Li and Lei Bu and Guangdong Bai and Fuman Xie and Kai Chen and Chang Yue},
  title = {{Model-Enhanced LLM-Driven VUI Testing of VPA Apps}},
  year = {2024},
  eprint = {2407.02791},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.02791},
}
Figure