TL;DR
This paper introduces INREACT, a novel framework for training multimodal GUI agents that can understand user instructions and interact with graphical interfaces. INREACT uses a two-stage training approach: curriculum learning to build foundational skills and reinforcement learning with a custom reward function to refine precise actions, significantly improving performance on complex GUI tasks.
What Problem Does This Paper Address?
Interacting with graphical user interfaces (GUIs) is a fundamental aspect of using digital devices. Developing intelligent GUI agents that can execute user instructions to perform tasks on these interfaces is a challenging but valuable goal. While recent advancements in multimodal large language models (MLLMs) have shown promise, existing GUI agents often struggle with two key issues: precisely locating small or visually similar elements within a dense interface, and performing complex, multi-step reasoning to achieve a goal. This paper aims to address these limitations by proposing a structured training framework that enhances both the precision of element localization and the agent's reasoning capabilities.
Methods
INREACT employs a two-stage training framework:
- Curriculum Learning: This stage systematically builds the agent's core capabilities in a progressive manner. It's divided into three sub-stages:
- Perception: Focuses on understanding GUI layouts, spatial awareness, and element recognition through tasks like GUI-based Question Answering and Interface Description Generation.
- Grounding: Teaches the agent to connect user instructions with specific interface elements by predicting coordinates from instructions and generating descriptions from coordinates.
- Reasoning: Introduces an explicit "observe-think-act" process, moving beyond single-step predictions to support multi-step decision-making.
- Reinforcement Learning (RL): This stage refines the agent's action accuracy and grounding robustness. INREACT uses a GRPO-based optimization framework with a novel, rule-based reward function. This reward function is designed to provide dense, informative feedback by combining:
- Action-Type Reward: A binary reward for correctly predicting the action type (e.g., click, type).
- Distance-Aware Coordinate Reward: This component includes an accuracy reward (checking if the predicted point is within the target bounding box) and a precision reward that penalizes the normalized Euclidean distance to the center of the target box. This encourages fine-grained localization.
The overall agent workflow follows a ReAct-like pattern, integrating observation, thinking, and acting.
Datasets
The paper evaluates INREACT on several datasets for both GUI grounding and navigation tasks:
- GUI Grounding:
- ScreenSpot (Cheng et al., 2024): A dataset with common interface elements across mobile, desktop, and web platforms.
- ScreenSpot-Pro (Li et al., 2025b): A more challenging dataset featuring 23 professional applications with high-resolution and complex layouts.
- GUI Navigation:
- Mind2Web (Deng et al., 2023): A web-based dataset for evaluating generalist agents performing complex tasks via language instructions.
- AndroidControl (Li et al., 2024): A mobile dataset for testing multi-step task execution in realistic Android environments.
The paper also mentions various sources for training data, including GUIChat, ScreenQA, MultiUI, AutoGUI, SeeClick, GUIEnv, Aguvis-stage2, and OmniAct.
Key Insights and Ideas
- Curriculum Learning is Crucial: The staged approach, moving from basic perception to grounding and then reasoning, is essential. Removing any stage significantly degrades performance, highlighting the importance of building skills progressively.
- Reinforcement Learning with a Smart Reward: The proposed piecewise reward function, which combines action-type correctness with distance-aware spatial precision, is more effective than simple binary or linear rewards for improving fine-grained localization in GUIs.
- Structured Reasoning Matters: The "think" component in the observe-think-act cycle is critical for formulating effective plans and making robust decisions, even more so than the observation phase itself in complex scenarios.
- Real-World GUI Logic: The piecewise reward function's design, with discrete steps and a focus on central clicks, better reflects the practical nature of GUI interactions than simpler reward schemes.
Key Results
- Superior Performance: INREACT consistently outperforms existing methods across four datasets for both GUI grounding and navigation tasks.
- Grounding: On ScreenSpot, INREACT achieves an average accuracy of 77.8%, surpassing strong baselines. On the more challenging ScreenSpot-Pro, it achieves an overall accuracy of 20.7%.
- Navigation: On Mind2Web, INREACT shows strong performance across cross-task, cross-website, and cross-domain generalization splits. On AndroidControl, it achieves 83.1 on Low-Level and 68.1 on High-Level tasks, outperforming prior baselines.
- Ablation Studies: Removing curriculum learning stages led to significant performance drops. Reinforcement learning with the proposed reward function improved performance over binary or linear rewards. Removing the "thought" component had a more pronounced negative impact than removing the "observation" component.
Limitations and Future Work
- Limited Action Space Coverage: The current action space (click, type, scroll) is insufficient for the full diversity of real-world GUI operations (e.g., drag-and-drop, multi-touch gestures). Future work could extend this action space.
- Inference Speed and Efficiency: The structured reasoning process can introduce computational overhead, potentially affecting real-time interactions. Optimizing inference speed through model compression or caching is suggested.
- Handling Dynamic Interfaces: The framework primarily works with static interfaces. Incorporating temporal modeling or environment change detection would be valuable for handling dynamic GUI updates.
- Reward Function Tuning: The thresholds for the piecewise reward function were chosen empirically and may not be optimal; further tuning is left for future work.
Summary updated