Abstract
Automated mobile application testing is often hindered by the fragility of test scripts and the high cost of cross-platform maintenance. This paper investigates the efficacy of Large Language Models (LLMs), specifically ChatGPT, in automating test script generation and migration. By shifting the focus from rigid UI coordinates to semantic business logic, the study explores how LLMs can generate executable scripts from natural language, migrate tests between Android and iOS, and transfer logic across different applications within the same domain. Our findings suggest that while LLMs offer significant potential for understanding intent and self-correcting during script execution, challenges remain regarding context window limitations and API accuracy.
Introduction
Mobile application testing is a complex, moving target. Developers must navigate a fragmented landscape of devices and operating systems while maintaining rapid release cycles. Traditional automation relies on test scripts—predefined instructions that interact with specific UI elements. However, these scripts are notoriously brittle; a minor change in a button's resource ID or a layout shift can render a script useless. Furthermore, migrating tests between platforms (e.g., Android to iOS) or similar apps (e.g., Outlook to Gmail) typically requires manual rewriting due to differences in underlying code and UI structures.
This research explores whether LLMs can bridge this gap by acting as "intelligent testers" that understand the underlying intent of a test case rather than just following hard-coded coordinates.
Methodology
The investigation utilized OpenAI’s gpt-3.5-turbo model to perform three primary automation tasks:
- Scenario-based Generation: Converting natural language descriptions (e.g., "log in and send an email") into executable Appium scripts.
- Cross-platform Migration: Adapting a script written for one platform (e.g., Android) to function on another (e.g., iOS) for the same application.
- Cross-app Migration: Transferring test logic between different applications with similar functional domains (e.g., migrating a "search flight" test from Ctrip to Fliggy).
To manage the complexity of these tasks, the researchers implemented a Dialogue-based Approach. Rather than using a single, monolithic prompt, the interaction was partitioned into three distinct phases:
- Initiation: Establishing the testing context, goals, and environment.
- Exploration: An iterative process where the LLM analyzes the XML structures of the app's current state and determines the next logical action.
- Summarization: Synthesizing the interaction history into a final, clean, and executable test script.
Experimental Setup
The study evaluated the model using popular, real-world mobile applications across two major categories:
- Email Services: Outlook, QQ Mail, and NetEase Mail.
- Travel and Booking: Fliggy, Ctrip, and Mafengwo.
These subjects were selected for their functional complexity and their availability across multiple app stores (Google Play and HUAWEI AppGallery), providing a robust environment for cross-platform and cross-app testing.
Key Insights and Capabilities
Business Logic over UI Structure
The primary advantage of LLMs in this context is their ability to prioritize semantics over syntax. Unlike traditional tools that search for specific element IDs, LLMs can recognize that a button labeled "Submit" and an icon of a checkmark likely perform the same logical function. This semantic understanding allows for more resilient automation.
Multi-to-Multi Event Mapping
Traditional migration tools often struggle when a single action in one app requires multiple steps in another. LLMs handle these "multi-to-multi" mappings gracefully because they view the test script as a holistic sequence of logical events rather than a 1:1 mapping of commands.
Self-Correction and Reasoning
During the dialogue-based exploration phase, the LLM demonstrated an ability to reason through obstacles. For example, it could identify and click a "Terms of Service" checkbox that was not present in the original script but was required to proceed in the target application.
Results and Discussion
RQ1: Generation Success
ChatGPT successfully generated syntactically correct Appium scripts. While the model occasionally utilized deprecated APIs, the resulting scripts were logically sound and accurately reflected the intended test operations.
RQ2: Cross-platform Seamlessness
Migration between Android and iOS was found to be predominantly seamless. By providing the LLM with differential information—such as device names or specific XPath changes—the model effectively updated the scripts to maintain functionality across platforms.
RQ3: Cross-app Complexity
Cross-app migration proved to be the most challenging task. The LLM's performance remained consistent, provided it was given sufficient context. In these scenarios, the original script essentially served as a "few-shot" example, guiding the LLM to generate a new script for the target application based on the logic of the first.
Limitations and Future Directions
Despite the promising results, several bottlenecks were identified:
- Context Memory: Complex applications with numerous screens can cause the LLM to "lose the thread" as the conversation history exceeds the model's context window.
- API Hallucinations: The model occasionally generates deprecated or non-existent Appium APIs, necessitating human-in-the-loop verification.
- Manual Effort: Crafting initial prompts and extracting XML structures still requires significant manual intervention, which can offset the time saved by automation.
Future research should focus on Multi-modal Information Fusion. By enabling LLMs to process GUI images alongside code and logs, the gap between AI-driven automation and human-like testing intuition can be further narrowed.
Summary updated