GUITester: Enabling GUI Agents for Exploratory Defect Discovery

Yifei Gao, Jiang Wu, Xiaoyi Chen, Yifan Yang, Zhe Cui, Tianyi Ma, Jiaming Zhang, Jitao Sang

TL;DR

This paper introduces GUITester, a novel multi-agent framework designed to empower GUI agents to autonomously discover defects during exploratory testing. By decoupling navigation from verification and proactively probing for anomalies, GUITester significantly outperforms existing agents, paving the way for more robust and efficient GUI quality assurance.

What Problem Does This Paper Address?

Ensuring software quality through exploratory GUI testing is crucial but incredibly labor-intensive. While modern AI agents, particularly those powered by Multimodal Large Language Models (MLLMs), have become adept at navigating user interfaces, they struggle to autonomously identify and report software defects. This failure stems from two key challenges:

  1. Goal-Oriented Masking: Agents are primarily trained to successfully complete tasks. When they encounter a defect (like a non-responsive button), their instinct is to find an alternative path to the goal rather than flagging the defect itself. This "success-at-all-costs" behavior effectively hides the problem.
  2. Execution-Bias Attribution: In the absence of explicit test oracles, agents must distinguish between their own operational errors (e.g., misclicks) and genuine software bugs. Due to the inherent stochasticity of LLMs, agents tend to blame themselves, misclassifying system defects as their own execution imprecision.

These limitations prevent current GUI agents from effectively performing the critical task of exploratory defect discovery, leaving a significant gap in automated software quality assurance.

Methods

The authors propose GUITester, a multi-agent framework that tackles these challenges by decoupling navigation from defect verification. It comprises two core modules:

  1. Planning-Execution Module (PEM): This module is responsible for planning navigation tasks and embedding "testing intents." Instead of just aiming for task completion, the PEM guides the agent to proactively probe for potential boundary behaviors and anomalies. This is achieved through three test intent patterns: "Alternative Paths" (exploring different entry points), "Boundary Conditions" (testing edge cases like empty or special character inputs), and "State Validation" (verifying UI element responsiveness).
  2. Hierarchical Reflection Module (HRM): This module handles anomaly detection and attribution.
  3. Monitor Agent: Observes the environment's response after each action, capturing whether an anomaly occurs without immediately attributing its cause. If a FAIL state is detected, it signals the Reflector.
  4. Reflector Agent: Analyzes the execution trajectory and interaction history to distinguish between agent errors and genuine GUI defects. It uses visual attribution (marking interaction points on screenshots) to aid in this analysis. If an agent error is identified and within a retry limit, the Executor attempts self-correction; otherwise, the system may replan.

Datasets

The paper introduces GUITestBench, the first interactive benchmark specifically designed for evaluating exploratory GUI defect discovery. It consists of:

  • 26 real-world defects collected from public GitHub issues.
  • 143 navigation tasks across 5 diverse application domains.
  • Defects are categorized into UI functional defects and user experience (UX) defects, further classified into three fault modes: Operation No Response (ONR), Unexpected Task Result (UTR), and Navigation Logic Error (NLE).
  • Tasks are synthesized with controlled guidance levels: "Defect-Oriented Tasks" (manually crafted to guide the agent to the defect) and "Exploration-Oriented Tasks" (where encountering the defect is a necessary waypoint during exploration).

Key Insights and Ideas

  • Proactive Exploration is Key: Simply navigating to complete tasks isn't enough for defect discovery. Agents need to be explicitly guided to "poke around" and test boundary conditions to uncover hidden bugs.
  • Decoupling is Crucial: Separating the act of detecting an anomaly from attributing its cause is vital. This allows for more accurate identification of genuine defects versus agent mistakes.
  • The "Wait-or-Miss" Dilemma: Real-world environments are noisy. Agents need robust mechanisms to distinguish between genuine delays and actual defects, a challenge that remains open.
  • Domain Knowledge Matters: The Monitor's effectiveness is limited by its understanding of application-specific behaviors. Integrating more domain knowledge could reduce false positives.

Key Results

GUITester significantly outperforms state-of-the-art baselines on the GUITestBench:

  • Overall Performance (Pass@3): GUITester achieved an F1-score of 48.90%, a substantial improvement over the best baseline's 33.35%.
  • Defect Type Performance: GUITester showed significant gains across all defect types (ONR, UTR, NLE).
  • Task Type Performance: While baselines struggled in the "Exploration-Oriented" setting (where defect locations are less predictable), GUITester maintained consistent performance across both "Defect-Oriented" and "Exploration-Oriented" tasks.
  • Single-Action vs. Multi-Action Defects: GUITester showed substantial improvements on single-action defects and notable gains on multi-action defects.

Limitations and Future Work

The authors acknowledge several limitations:

  • The Wait-or-Miss Dilemma: Distinguishing environmental delays (network fluctuations, server lag) from true anomalies remains a challenge.
  • Monitor Capability Boundaries: The Monitor's effectiveness is limited by its domain knowledge. Integrating application-specific knowledge or more advanced vision-language models could improve precision.
  • Small Action Space, Narrow Testing Scope: Current GUI agents have limited action spaces, preventing them from exploring richer interactions (e.g., complex gestures).
  • Broader Defect Coverage: The current work focuses on interactive defects. Expanding coverage to include layout issues (overlapping elements, misaligned text) is a future direction.

Future work will focus on developing robust strategies for handling environmental noise, enhancing the HRM's domain understanding, enriching action spaces for broader testing scenarios, and expanding defect coverage.

Summary updated

Abstract

Exploratory GUI testing is essential for software quality but suffers from high manual costs. While Multi-modal Large Language Model (MLLM) agents excel in navigation, they fail to autonomously discover defects due to two core challenges: \textit{Goal-Oriented Masking}, where agents prioritize task completion over reporting anomalies, and \textit{Execution-Bias Attribution}, where system defects are misidentified as agent errors. To address these, we first introduce \textbf{GUITestBench}, the first interactive benchmark for this task, featuring 143 tasks across 26 defects. We then propose \textbf{GUITester}, a multi-agent framework that decouples navigation from verification via two modules: (i) a \textit{Planning-Execution Module (PEM)} that proactively probes for defects via embedded testing intents, and (ii) a \textit{Hierarchical Reflection Module (HRM)} that resolves attribution ambiguity through interaction history analysis. GUITester achieves an F1-score of 48.90\% (Pass@3) on GUITestBench, outperforming state-of-the-art baselines (33.35\%). Our work demonstrates the feasibility of autonomous exploratory testing and provides a robust foundation for future GUI quality assurance~\footnote{Our code is now available in~\href{https://github.com/ADaM-BJTU/GUITestBench}{https://github.com/ADaM-BJTU/GUITestBench}}.
BibTeX
@inproceedings{gao2026guitester,
  author = {Yifei Gao and Jiang Wu and Xiaoyi Chen and Yifan Yang and Zhe Cui and Tianyi Ma and Jiaming Zhang and Jitao Sang},
  title = {{GUITester: Enabling GUI Agents for Exploratory Defect Discovery}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.946},
}

Loading related content...

Figure